Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
Authors: Kazuki Fujii
Taishi Nakamura
Mengsay Loem
Hiroki Iida
Masanari Ohi
Kakeru Hattori
Hirai Shota
Sakae Mizuki
Rio Yokota
Naoaki Okazaki
Venue: COLM (2024)
Keywords: Japanese, Continual Pre-Training, Llama 2, cross-lingual LLM adaptation, vocabulary expansion, parallel corpus
TL;DR: Cross-lingual continual pre-training effectively adapts LLMs across languages (in our case Japanese), while vocabulary expansion and parallel corpora enhance the token efficiency and translation ability, respectively.

Abstract: Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.

Published as a conference paper at COLM 2024
Continual Pre-Training for Cross-Lingual LLM Adaptation:
Enhancing Japanese Language Capabilities
Kazuki Fujii* Taishi Nakamura to Mengsay Loem* Hiroki lida* Masanari Ohi
Kakeru Hattori Hirai Shota* Sakae Mizukit Rio Yokotat Naoaki Okazaki*
t "Department of Computer Science, School of Computing, Tokyo Institute of Technology GGoobal Scientific Information and Computing Center, Tokyo Institute of Technology (kazuki. (kazuki.fujii@rrz www.ifiioo.gggiii.
taishi nww.marakammarii.ooom gsic, mengsay. 1oem@n1p..c hiroki. .irdai. c, masanari.ohionlpp kakeru. hathorittiil Cnlp.c, shota. hww.iinir..., sakae. miliae..zzki@nnlp.c. rioyokotadrio. gsic, okazakiλc). sitech.ac.jp
Abstract
Cross-lingual continual pre-training of large language models (LLMs) ini- tially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B to- kens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-traine teeiniiinng revealed that it was par- ticularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expan- sion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.
Introduction
Large language models (LLMs) such as ChatGPT have attracted significant attention by demonstrating human-level language understanding and generation capabilities, as well as generalizability to various fields. However, many LLMs, including Llama 2 Touvron et al., 2023), are primarily trained on English corpora, and their performance in other languages, especially in those with syntactic structures and writing systems greatly different from English, is decreased (OpenAl et al. 2023). Motivated by this performance gap, establishing methods to build LLMs that excel in Japanese (hereafter referred to as Japanese LLMs) is an important research issue in Japan. In particular, since English language resources are outstanding in terms of quality and quantity, insights on effectively utilizing both Japanese and English language resources are in high demand. For example, is estimated that there
are approximately nine times more English web pages than Japanese web pages] However, pre-training from scratch using both Japanese and English language data requires enormous computational resources, making it difficult to acquire insights in timely manner.
"qqual contribution.
Statistics of Common Crawl Monthly Archives: httpe://ccmmoncracl. github.
cithub.io/cc-cnnee wttciii//ccaa cracl--rrallaaaa statistica

---

Published as a conference paper at COLM 2024
Japanese tasks
English tasks
JCQA
OBQA
Swallow-7b
JEMHQA
Swallow-133
TrQA
NIILC
Swallow-70b
JSQUAD
HS
XL-Sum
SQuAD2
Nog
MGSM
JMT20
GSMBK
20
0
20
40
60
80
20
0
20
40
60
80
Relative change [%)] in performance
Relative change [96] in performance
Figure 1: Relative change in performance of Swallow compared to Llama Japanese tasks (left, see Table 2 for task details) improved by up to approximately 70%.
Therefore, we choose to conduct Japanese continual pre-training from English LLMs, aiming to save computational resources, and obtain insights how transfer the knowledge and abilities learned by English LLMs to Japanese. Recently, there has been an increasing number of attempts to adapt LLMs other languages using continual pre-training (Gupta et al., 2023 Cui et al. 2023 Pires et al. 2023 (hu Zhu et al. 2023 Zhao et al. 2024), but a comprehensive investigation on the effectiveness of continual pre-training has not been conducted. For example, the relationship between the amount of Japanese data used for continual pre- training and model performance, and the impact of the model size on this relationship, are unclear. We construct Swallou an LLM with enhanced Japanese capabilities, by performing continual pre-train traininiiin on Llama 2's 7b, 13b, and 70b models using a Japanese corpus. We then evaluate the performance on six types of tasks including question answering and machine translation in both Japanese and English to analyze changes in knowledge and abilities in both languages. As a result, we demonstrated performance improvements in almost all Japanese tasks for all model sizes (Figure I.
Furthermore, to identify efficient methodologies for cross-lingual continual pre-training, we also investigate the impact of vocabulary expansion and the effectiveness of Japanese- English parallel corpora. Vocabulary expansion has the effect of shorteming token sequence lengths and improving the learning and generation efficiency of Japanese text by adding Japanese characters and words. However, previous studies have not sufficiently analyzed the extra cost of optimizing parameters for the added vocabulary or the impact on per- formance due to the increase in the amount of learnable text. Therefore, in this study, we conducted a comprehensive evaluation of the impact of vocabulary expansion on the performance of both Japanese and English. Vocabulary expansion improved Japanese text generation efficiency by up to 78,, thanks to a 56.2% reduction in tokens in the Swallow Corpus. This enhancement did not compromise downstream task accuracy, except for summarization.
Parallel corpora known to have the effect of promoting cross-lingual transfer in multilin- gual models (Chi et al., 2022 Hu et al., 2020 Feng et al. 2022). However, the effectiveness of parallel corpora in continual pre-training settings of the target language has not been studied in detail. Our experimental results reveal that simply mixing parallel corpora with plain text corpora improves the accuracy of machine translation tasks.
The contributions of this study are as follows:
Swallow achieved the highest performance in Japanese among all models devel- oped in Japan (as of December 2023). We demonstrate that cross-lingual continual pre-training can achieve higher performance with fewer computational resources compared to Japanese LLMs trained from scratch.
2The I.LMs were released with different official name, but this code name for the review.
2

---

Published as a conference paper at COLM 2024
We show that continual pre-training is effective for improving Japanese abilities, especially question answering tasks that require Japanese knowledge.
We provide evidence that the Japanese performance of language models improves monotonically as the amount of Japanese training data increases.
We show that vocabulary expansion does not affect performance in most tasks, and only degrades performance in automatic summarization.
We reveal that using parallel corpora together enhances translation ability without affecting the performance of other tasks.
Related work
2.1 Continual pre-training
Continual pre-training is form of domain adaptation where additional pre-training tasks are performed on a pre-trained language model using text from the target downstream
task before fine-tuning the model on that task (Lee al. 2019 Beltagy et al. 2019 Sung
et al. 2019). Since the emergence of open and high-pertormance English LLMs, there have been an increasing number of attempts perform continual pre-training adapt LLMs to other tasks and languages [Gupta et al. [2023] Cui et al. 2023 Pires et al. 2023 Zhu et al.
2023 Zhao et al. 2024). However, comprehensive invesligation of the effects of continual pre-training for various model sizes and training data sizes has not been conducted.
2.2 Vocabulary expansion
Vocabulary expansion is method to increase the vocabulary of trained LLM. The Japanese writing system differs from English, where kanji characters tend to be converted into UTF-- byte sequences. For example, a Japanese single-character noun ži (cat) is represented by three byte-level tokens <0xE7> <0x8C> <0xAB>, which do not provide any semantic meaning. This treatment not only looks unreasonable as Japanese representation but also increases the sequence length of Japanese text well as the cost of text generation (Ahia al., 2023). Adding Japanese characters and words to the vocabulary can reduce the number of tokens required to represent Japanese text and alleviate this problem. In domain adaptation of text embedding models, it is known that adding vocabulary from the target domain can improve performance (Sachidananda et al. 2021 Yao et al. 2021). In contrast, the main motivation for vocabulary expansion in continual pre-training of LLMs is to improve training and generation efficiency in the target language, and there little knowledge about its impact performance.
Examples of vocabulary expansion in building Japanese LLMs are limited to evaluating Japanese language ability, and do not evaluate the English language ability. Examples of vocabulary expansion in Chinese LLMs (Cui et al. 2023) only report the performance with vocabulary expansion, and do not compare with a base model. In multilingual LLMs (Ahujaa et al. 2023), it was shown that the vocabulary size for each language correlates with perfor- mance, but these studies train longer as the vocabulary is expanded, so the isolated impact of vocabulary size remains unclear.
2.3 Parallel corpus
In multilingual text embedding models, it has been reported that pre-training with the language modeling objective using parallel corpora mitigates language-specific hidden state vectors and promotes lingull transfer (Chi al. 2022 Hu al. al. 2020 Feng et al.) 2022). In applications to LLMs, it has been reported that using parallel sentences for instruction tuning can improve translation ability more efficiently than using the same amount of multilingual corpora (Zhu et al.) 2023 Ranaldi et al. 2023). However, the effectiveness of combining continual pre-training with parallel corpora remains unclear.
JJapanese Stable L.M Beta: htttp:///aaaa....// wtw.seaatttaaaaablooo/aaa./ staeeeettaaeeeeaeeeeeeeeeeeeeeee 1m-beta
3

---

Published as a conference paper at COLM 2024
Params
mmndel
Heads
Layers
Context length
GQA
Tooees
LR
7B
4096
32
32
4096
X
100B
1.0 x 10
13B
5120
40
4096
10DB
1.0 10 10-4
70B
8192
64
80
4096
100B
5.0 x 10-5
Table 1: Architecture and hyperparameters of the Swallow models
3
Experimental settings for continual pre-training
The continual pre-training of Swallow involves expanding the vocabulary of a pre-trained Llama 2 model and then performing continual pre-training using corpora primarily consist- ing of Japanese. The models are evaluated on six types of tasks in Japanese and English. The evaluation results are compared with LLMs developed in English-speaking regions and in Japan, to demonstrate the comparative effectiveness of our continual pre-training approach. In this section, we describe the training settings, training corpus, vocabulary expansion method, and evaluation method.
3.1 Training details
Table I shows the hyperparameters of the Swallow models. Preliminary experiments were conducted with reference to Llemma (Azerbayev et al. 2024) and Code Llama (Rozière al. 2023) to determine the parameters. Since continual pre-training is bound by the model architecture of the base model, Swallow adopts the same Transformer decoder as Llama 2. The hidden size, number of attention heads, number of layers, and context length are the same as Ll ama 2. To maintain consistency with the pre-training phase, batch size of 1024 was used for all model sizes in Swallow, matching the global batch size of 4M tokens in Llama 2.
The AdamW optimizer [Loshchilov & Hutter 2019 was employed for training the models, with hyperparameters 81 = 0.9, B2 0.95, 1.0 x 10 8 A cosine learning rate scheduler was used, and the learning rate was set to reach its maximum value at 1,000 warmup steps and finally decays to 1/30 of that value. Additionally, a weight decay of 0.1 and gradient clipping of 1.0 were used. Furthermore, Flash Attention (Dao 2023) was adopted
for improved computational efficiency and memory footprints. Reter to Figure 8 in the appendix for the training loss curve.
3.2 Training corpora
When continually pre-training with full parameters, forgetting previously learned knowl- edge is concern (Jin et al., 2022). One method to prevent forgetting is the Experience Replay technique (Chaudhry et al. 2019). This method involves reusing a portion the data previously used tor training the language model during continual pre-training (Scialom) let al. 2022). Following this approach, our study incorporates a portion of the English corpus in addition to the target Japanese corpus for continual pre-training.
The corpus used for continual pre-training includes the Swallow Corpus, which is explained in Appendix A Japanese Wikipedia and for English, the RefinedWeb (Penedo et al. 2023 and The Pile (Gao et 20200. From These corpora, approximately 100B tokens were sampled for the training data of continual pre-training. The sampling was configured so that 5% of the English text comes from RefinedWeb, another 5% from English arXiv paper texts within The Pile, and the remaining 90% from Japanese texts. The Japanese text comprises about 1.6B tokens from Japanese Wikipedia, with the rest from the Swallow Corpus. The ratio of Japanese to English data in the training set was decided based on preliminary experiments (see Appendix B for details).
Dump dated March 20, 2023.

---

Published as a conference paper at COLM 2024
3.3 Vocabulary expansion
In this study, we aim to adapt LLMs to languages with different writing systems from the Latin alphabet. For this purpose, we performed vocabulary expansion. Vocabulary expansion refers to the post-hoc addition of vocabulary to an existing LLM. This increases the amount of text that can be trained and generated within the same computational budget, thereby improving computational efficiency. In the vocabulary expansion adopted in Swallow, we constructed Japanese vocabulary, initialized vectors, and added string preprocessing.
We conducted preliminary experiments to determine the optimal vocabulary size by com- paring sequence lengths with 16k, 32k, and 48k Japanese vocabularies. The sequence lengths were 56.2%, 54.1%, and 53.2%, respectively, compared to using original LLaMA vocabulary. Since there was no significant performance difference in Japanese tasks among these sizes, we selected the 16k vocabulary for its advantage in training speed. The construction of Japanese vocabulary involved creating a vocabulary (up to 16k) using the BPE algorithm the Swallow Corpus segmented by MeCab and the UniDic dictionary We then merged the Japanese vocabulary (subwords) with the original LLaMA vocabulary, resulting in total vocabulary size of 43,176. Refer Appendix E.1 for further details on the vocabulary expansion procedure.
The vectors for the embedding and output layers of the added subwords were initialized with the average of the vectors of the subwords segmented by the LLaMA tokenizer, i.e., the subwords trained by Llama 2, following previous research (Yao al. 2021). This initialization method was adopted because we confirmed that random initialization led to corrupted outputs.
String preprocessing enhanced by adding NFKC normalization to utilize the pre-trained knowledge of alphanumeric characters and symbols in the ASCII code range.
3.4 Evaluation method
The evaluation methods for Japanese and English are shown in Tables and respectively. The dataset consists of five types of Japanese and four types of English tasks, with few-shot settings for question answering (QA), reading comprehension (RC), automatic summariza- tion (AS), arithmetic reasoning (AR), commonsense reasoning (CR) and machine translation (MT). For details, see Appendix G We evaluate these downstream tasks under few-shot In-Context Learning without fine-tuning the model. Additionally, the evaluation datasets were not included in our continual pre-training corpora to ensure an unbiased assessment of the model's performance. For all fasks, higher scores indicate better performance.
The selection of tasks was based on discussions in LLM-jp (Han et al. 2024) and the methodology of the Llama paper (Touvron al. 2023), with an active adoption of tasks related to inference and text generation. The natural language inference task in I1m-jp- eval was excluded from evaluation due to unstable scores the 7b and 13b models (see Appendix
Results
4.1 Effects of continual pre-training
shoows the evaluation results of Swallow and its base model Llama on Japanese and English tasks. Figure displays the increase or decrease rate of Swallow 's scores com- pared to Llama The average score of Swallow on Japanese tasks surpasses Llama by approximately 7 points. On the other hand, the English scores are 2-5 points lower, but the performance drop tends to be smaller the model size increases. When looking at indi-
lactps:/ttaku900 io/mecabbb/bbbbbboo
https///ddrd..injall.........ppp
5

---

Published as a conference paper at COLM 2024
Benchmark Eval. task
Ilm-jp-eval (Han et al. 2024)
(v1.0.0)
JP LM Evaluation Harness (commit #9b42d41)
Question Answering
RC
AS
AR
Machine Translation
Dataset
JCQA
JEMHQA
NILCC JSQuAD
XL-Sum
MGSM WMT WMT'200j-Ean
Instances Few-shots
1,119
120
198
4,442
766
250
1,000
993
4
4
4
Eval. metric
EM acc.
Char-F1
Char-F1 Char-F1 ROUGE-2 EM acc.
BLEU
Table Japanese datasets. Acc. stands for accuracy, EM for exact match. Evaluation datasets
include ICQA for [CommonsenseQA (Kurihara et al. 2022), JEMHQA for JEMHopQA (Ishi
et )023), NIIL (Sekine ) JSQuAD (Kurihara et al.) 2022), XL-Sum (Hasan al.
2021
MGSM (Shi et al. 2023), and WMT'00 (Barrault et al. 2020).
Benchmark Eval. task
LM Evaluation Harness
(Gao et al.
2022
(v0.3.0)
QA
RC
CK
AR GSM8K
Dataset
OBQA
TrQA
SQuAD2
HS
XW
Instances
500
17,944
11,873
10,042
2,325
1,319 8
Few-shots
8
Eval. metric
acc.
EM acc.
EM acc. acc.
acc.
EM acc.
Table 3: English datasets. OBQA for Open (Mihaylov et al. 2018), TrQA for Trivi- aQA (Joshi et al. 2017), HS for HellaSwag (Zellers et al. 2019), XW for XWINO (Tikhonov & Ryabinin 2021)
AQuAD2 (Rajpurkar et 2018), and GSM8K (Cobbe et al. 2021).
Japanese question answering (JCQA, JEMHQA, NIILC) shows significant improvement of up 75,, and arithmetic reasoning (MGSM) improves by 36-63%. In contrast, English question answering (TrQA) and arithmetic reasoning (GSMSK) degrade by 6-23%. The change in automatic summarization (XL-Sum) is less than 5%. Machine translation shows contrasting results depending on the direction, with 15-41% improve- ment in Japanese reading comprehension (JSQuAD) has limited room for improvement as Llama score is above 0.8, resulting in less than 10% improvement.
vidual taske
(En-Ja) and 4-13% degradation in Japanese-to- to-English (Ja-En).
We analyze the impact of continual pre-training Japanese abilities and knowledge. In the case of arithmetic reasoning (Ja: MGSM, En: GSMSK), while Llama 2 shows superiority in English (GSM8K > MGSM), Swallow improves MGSM, but GSM8K degrades to similar level, suggesting that the reasoning ability in English has not been fully transferred to Japanese. Considering the reports of reasoning ability transfer in instruction tuning (Ye) et al. 2023), combining instruction datasets could be promising strategy to bring Japanese arithmetic reasoning to the same level observed in English.
Regarding knowledge, the significant improvement in question answering suggests that the acquisition of Japanese knowledge has progressed. Figure 2 illustrates the impact of continual pre-training on the scoring of each question in the NILC QA dataset. The dense color in the upper left corner indicates that many questions have shifted from incorrect to correct answers, while the opposite is uncommon. This trend suggests that continual pre-training successfully incorporated new knowledge and corrected inaccurate responses.
4.2 Comparison with full-scratch models
Table 5 shows the evaluation results of the Swallow and major Japanese LLMs trained from scratch (see Table 12 in the appendix for model references). Note that these models are all general-purpose language models without instruction tuning. Compared to the LLMs trained from scratch Japan (calm-7b, l1m-jp-13b-v1.0, PLaMo-13b) (see Appendix[L]] for de- tails of each model), the average score of Swallow is 8,4 to 17.4 points higher, demonstrating the usefulness of continual pre-training.
We assess performance changes relative scale, considering the differences in task score levels.

---

Published as a conference paper at COLM 2024
Evaluation in Japanese NIICC JSQuAD XL-Sum
Model
JCQA
JEMHQA
MGSM
En-Ja
Ja-En
Avg.
Llama 2-7b
38.5
42.4
34.1
79.2
19.1
7.6
17.8
17.4
32.0
Swallow-7b
48.1
50.8
59.7
85.7
18.3
12.4
25.1
15.1
39.4
Llama 2-13b
70.0
44.2
41.7
85.3
21.4
13.2
21.5
19.8
39.6
Swallou-132 13b
78.4
50.6
64.0
90.1
21.7
20.4
27.2
17.7
46.3
Llama 2-70b
86.9
46.6
52.6
90.8
23.6
35.6
26.4
24.0
48.3
Swallow-70b
93.5
62.9
69.6
91.8
22.7
48.4
30.4
23.0
55.3
Evaluation English SQuAD2
Model
[BBQ
TrQA
H5
XW
GSM8K
Avg
Llama 2-7b
35.8
62.7
58.6
32.1
90.5
14.1
49.0
Swallow-7b 7b
31.8
48.4
53,1
31.3
88.2
11.3
44.0
Llama 2-13b
37.6
72.6
61.5
36.8
91.4
24.0
54.0
Swallow-13b
35.0
58.5
56.6
34.1
90.8
20.4
49.2
Llama 2-70b
42.8
82,4
67.4
37.7
92.9
52.8
62.7
Swallow-70b
42.2
77.6
64.6
37.5
92.0
48.7
60.4
Table 4: Evaluation of Swallow and its pre-training source, Llama 2, in Japanese and English.
0.55
1.0
Japanese Score (mean)
0.8
Swallow-77 7
0.6
0.45
0.4
0.40
Swellow 7h
0.2
0.35
Swellow-
Swellow- 70h
0.0
0.0
0.2
0.4
0.6
0.8 1.0
Llama-2-7b
Billions of tokens
Figure 3: Scalability continual pre-training Japanese tasks. Score at OB tokens corre- sponds to the baseline performance of the Llama 2 model.
Figure Joint distribution of Llama (x- axis) and Swallow (y-axis) scores (character F1, with 1.0 representing an exact match) for NIILC questions.
4.3 Scalability to training tokens
the relationship between the volume of training data (number of tokens) used for continual pre-training and the average performance score on the Japanese bench- mark. We evaluated the performance of Swallow 7b, 13b, and 70b at each interval of approximately 20B tokens of the training data. The figure reveals monotonic upward trend in average scores in correlation with the augmentation of Japanese training data for continual pre-training. Notably, the largest performance increase occurs in the initial training stage with 20B tokens, with subsequent gains diminishing. However, performance consistently improves as the amount of training data increases, suggesting that the perfor- mance improvement has not saturated even with approximately 100B tokens of continual pre-training.
7

---

Published as a conference paper at COLM 2024
Evaluation in Japanese NIILC JSQuAD XL-Sum
Model
|JCQA
JEMHQA
MGSM
En-Ja
Ja-En
Avg
calm2-7b
22.0
50.5
50.7
78.0
2.3
6.0
23.5
15.0
31.0
Swallow -7b
48.1
50.8
59.7
85.7
18.3
12.4
25.1
15.1
39.4
1m-jp-11bb-1.00
22.6
47.9
38.6
77.4
10.8
2.4
19.6
11.9
28.9
PLaMo-13b
22.7
51.9
41.4
76.2
10.3
3.6
15.8
12.0
29.2
Swallow -13b
78.4
50.6
64.0
90.1
21.7
20.4
27.2
17.7
46.3
Table Comparison of evaluation results between Swallow and models trained from scratch on Japanese tasks.
5 Analysis
5.1
Vocabulary expansion
5.1.1
Experimental setup and motivation for vocabulary expansion
To investigate the impact of vocabulary expansion (VE) specifically, the addition of Japanese words and characters we compare models pre-trained with and without VE. The model pre-trained without VE is denoted VVE hereafter. Incorporating VE can shorten token sequences, thereby enhancing the efficiency of both training and generating Japanese text. However, the impact of vocabulary expansion on performance is unclear. Intuitively, if the added vocabulary is poorly optimized, it may degrade performance. Yet, studies on domain adaptation scenarios report that VE enhances performance (Sachidananda al., 2021 Yao et al.) 2021). Additionally, being able to train with more text within the same computational budgets could positively influence performance.
5.1.2 Impact of vocabulary expansion
the performance of Swallow relative Swallow-VE, where the vocabulary expansion was not applied to the latter. Regarding Japanese language capabilities, the overall impact of vocabulary expansion on performance is minor. Appendix H provides the absolute scores and more detailed analysis. When examining specific tasks, observe the performance change in question answering approximately +10%, yet there is no consistent degradation across different model sizes. Therefore, the
trend of either improvement increase in the amount of training text brought from the vocabulary expansion does not directly affect performance. Additionally, from Figure 7 we did not observe any significant difference in convergence characteristics in the performance curve of Japanese question- answering task. In contrast, automatic summarization (XL-Sum) degraded by about 5-15% with vocabulary expansion for all model sizes This suggests that the impact of vocabulary expansion may be more apparent in tasks that involve processing longer contexts.
5.2 Parallel corpus
5.2.1
Experimental setup for parallel corpus
In the experiments investigate the effectiveness of parallel corpora, we use Swallox-VE the baseline to separate the impact of vocabulary expansion, and examine the performance when incorporating parallel corpora in the continual pre-training, We used JlaarCrrwl 3.0 (Morishita et al. 2022) corpus, which contains approximately 22 million Japanese-Englin parallel sentences extracted from the web.
We investigate three different settings to utilize parallel corpus, as shown in Table These methods vary by training sequence and task format. The training sequence can be "two-
9This trend remained unchanged after instruction tuning, so it does not seem to be problem with instruction following ability.
8

---

Published as a conference paper at COLM 2024
JCQA
JCQA
JEMHQA
JEMHQA
NIILC
JSQuAD
JSQuAD
XL-Sum
XL-Sum
MGSM
MGSM
pM
EM
WMT20
WMT20 Fm
20
-15
-10
-5
5
10
15
20
OBQA
Swellow-7b
OBQA
Settings
TrQA
Swellow-13b
TrQA
NTP Two-staged
HS
Swellow
H5
NTP Mixed
SQuAD2
SQuAD2
BBC & Two-staged
XW
XW
GSMBK
6SM8K
-20
-15 -10
20
-20
20
40
60
Relative change [96] in performance
Relative change [96] in performance
Figure 4: Relative change in performance with versus without vocabulary expansion (Swallow vs. Swallow-VE).
Figure 5: Relative change in performance when using parallel corpus compared to Svallow VE.
0.30
Swallow-/b
avg.
Swallow-/e /b
0.26
Swallow-13b
0.7
Swallow-13b
R
Swallow-70b
Swallow-70b
0.22
/o-Vt-VV
0.6
Swallow /b-VE
Swallow 136-VE
Swallow-130 13b-VE
0.19
Swaaloww TVE
Swallow- T0b--V
Uama-2-7b
Uama-2-7b
0.14
Uama-2-13bb
Lama-2213b
0
20
10
60
80
100
Uammr2270b
0
20
10
60
90
100
Uama-22-06
Num, of training tokens [x10
Num. of training Lokens [x10")
Figure 6: Performance curve of Swallow-VE and Swallow in automatic summarization (XL-Sum).
Figure 7: Performance curve of Swallow-VE and Swallow in the average score of question answering task (JCQA, JEMHQA, NIIIC).
staged", where the parallel corpus is used first and followed by the multilingual corpus, or "mixed", where it is combined with the multilingual corpora from the start. The intention behind the two-staged setting is the assumption that parallel sentences could facilitate the switch from English-centric to Japanese-centric training.
For task formats, we have "next token prediction" (NTP), which involves concatenating parallel sentences, and "translation instruction" (TI), where the target sentence is predicted based on the source sentence and a translation instruction. Both formats are applied in two directions, Ja->En and En- for each pair of parallel sentences. The following templates show how parallel sentences are converted into NTP format (top) and TI format (bottom). Note that for the TI format, only the target language sentence is the target for training.
[Japanese sentence] [English sentence]
[English sentence] [Japanese sentence]
Please translate the following Japanese text into English. [Japanese sentence] [English sentence]
Please translate the following English text into Japanese. [English sentence] [Japanese sentence]

---

Published as a conference paper at COLM 2024
Task format
Training seq.
Number of training tokens [x 10
Next Token Prediction (NTP)
Two-staged
5.6
Next Token Prediction (NTP)
Mixed
5.6
Translation Instruction (TI)
Two-staged
2.8
Table 6: List of experimental settings in the use of the parallel corpus.
5.2.2 Effectiveness of parallel corpus
Figure shows the rate of increase or decrease in scores when using a parallel corpus in conjunction with continual pre-training, with Swallow-VE without vocabulary expansion as the baseline. Translation performance improved by 9-24% for En-Ja and 14-51% for Ja-En. The improvement in Ja-En is particular to the parallel corpus. Regarding the usage of the corpus, we found that the next token prediction format with a "mixed" setting the translation instruction format in "two-staged" setting was effective. In other words, we proved that simply mixing parallel sentences into the multilingual corpus and conducting continual pre-training can effectively improve translation performance. This finding is consistent with the claim that the translation capabilities of LLMs stem from parallel sentences incidentally present in plain text corpora (Briakou al. 2023).
The relative change in scores for tasks other than translation was within =15%, without showing consistent improvement or decline across different model sizes. Therefore, no evidence was obtained that the parallel corpus promotes cross-lingual transfer and improves abilities other than translation.
Conclusions
In this study, we developed Swallow, leveraging Llama models for enhanced Japanese language performance through continual pre-training on Japanese datasets, and analyzed their performance to address the lack of comprehensiveness in model size, training data size, and evaluation methods in previous studies. Through our evaluation, we found that continual pre-training significantly boosts Japanese abilities, particularly in knowledge- intensive question answering task. Consequently, we demonstrated that continual pre- training is an efficient approach for achieving high performance, as the continual pre-trained models outperform Japanese LLMs that are trained from scratch. We also observed that performance improves in line with increases in training data. Furthermore, in our quest for more efficient methodology, we investigated how expanding the vocabulary and incorporating parallel corpora into continual pre-training affect performance. We revealed that while the vocabulary expansion improves computational efficiency, it has little impact performance except for summarization. Additionally, found that simply integrating parallel sentences into plain text corpora improves translation performance, particularly for Japanese-Engliste but does not hurt the performance of other tasks. The experimental results and analyses in this study provide important insights into the effectiveness of continual pre-training, the impact of vocabulary expansion, and the effects of using parallel corpora in the development of LLMs for non-Engllish languages.
10

---

Published as a conference paper at COLM 2024
7 Ethical considerations
Swallow is subject to the same well-recognized limitations of other LLMs, including the inability to update information after the pretraining phase, the potential for non-factual generation such as unqualified advice, and a propensity towards hallucinations.
Similar other large language models, Swallow may produce content that is harmful, offensive, or biased due to being trained on datasets derived from publicly available on- line sources. We commit ongoing fine-tuning and plan to release updated versions of Swallow as we make further advancements in resolving these issues.
Reproducibility statement
All models developed in this study (continual pre-training on Llama 7B, 13B, 70B) have already been released on Hugging Face. The benchmark datasets used in this study are also publicly available. Therefore, it is straightforward to reproduce our experimental results reported in Table
Acknowledgments
This work was supported by the ABCI Large-scale Language Model Building Support Program of the Al Bridging Cloud Infrastructure (ABCI), built and operated by the National Institute of Advanced Industrial Science and Technology (AIST). We would like to thank Mr. Takuya Akiba of Sakana Al for providing valuable advice on training. We also received advice on vocabulary expansion methods from Mr. Tatsuya Hiraoka of MBZUAI.
In the evaluation experiments of the trained LLMs, we utilized data and insights developed and made publicly available by LLM-jp. This research was supported by JST, JPMJCR2112.
References
Orevaoghene Ahia, Sachin Kumar, Hila Gonen, Jungo Kasai, David Mortensen, Noah Smith, and Yulia Tsvetkov. Do all languages cost the same? Tokenization in the era of commercial language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing (EMNLP), pp. 9904-9923, 2023. URL httphh:/cclaanthllloooo org/ 2023. 1115.emmlppmaii 614
Kabir Ahuja, Harshita Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Mohamed Ahmed, Kalika Bali, and Sunayana Sitaram. MEGA: Multilingual evaluation of generative Al. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP), Pp. URL /ttp:
4232-4267, 2023.
org/ 2023. emnlp-man main. 258
Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. Llemma: An open
open language model for mathematics. In the Twelfth International Conference on Learning Representations (ICLR), 2024. httpss
Loic Barrault, Magdalena Biesialska, Ondłej Bojar, Marta R. Costa-jussà, Christian Feder- mann, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Matthias Huck, Eric Joanis, Tom Kocmi, Philipp Koehn, Chi-kiu Lo, Nikola Ljubešić, Christof Monz, Makoto Mor- ishita, Masaaki Nagata, Toshiaki Nakazawa, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2020 conference on machine translation (WMT20). In Proceedings of WMT, wwwp////........o.oo 2020 wttp://222w.mt1.... wmt-1.1
pp. 1-55, 2020. URL htttp::/aclanthhol.
Iz Beltagy, Kyle Lo, and Arman Cohan. SciBERT: A pretrained language model for scientific text. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
pp. 3615-36210 2019. URL htttps://acllattholloo/.. 1tw.llogy.oro//9--117713
11

---

Published as a conference paper at COLM 2024
Eleftheria Briakou, Colin Cherry, and George Foster. Searching for needles in haystack: On the role of incidental bilingualism in PaLM's translation capability, In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 9432-9452, 2023. Utthttps::/ h/ap://hoolll org/2023. acl aal-long... 524.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokanía, Philip H. S. Torr, and Marc' Aurelio Ranzato. On tiny episodic memories in continual learning, arXiv:1902.10486, 2019.
Zewen Chi, Shaohan Huang, Li Dong, Shuming Ma, Bo Zheng, Saksham Singhal, Payal Bajaj, Xia Song, Xian-Ling Mao, Heyan Huang, and Furu Wei. XLM-E: Cross-lingual language model pre-training via ELECTRA. In Proceedings of the 60th Annual Meeling of the Association for Computational Linguistics (Volume Long Papers), pp. 1170-6182, 2022.
URL https://aclanthoooggy org/ a0g/2022...lloong 427
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.1416, 2021.
Yiming Cui, Ziqing Yang, and Xin Yao. Efficient and effective text encoding for Chinese LLaMA and Alpaca. arXiv:2304.08177 2023,
Tri Dao. Ftestottt Faster attention with better parallelism and work partitioning. arXiv:2307.08691 2023.
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language- agnostic BERT sentence embedding. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics (ACL), pp. 878-891, 2022. URL
org/2022. acl-1acclnnngg 62
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The Pile: 800GB dataset of diverse text for language modeling. arXiv:2101.00027, 2020.
Leo Jonathan Tow, Stella Biderman, Charles Lovering, Jason Phang, Anish Thite, Fazz, Niklas Muennighoff, Thomas Wang, sdtblck, tttyuntian, researcher2, Zdeněk Kasner, Khalid Almubarak, Jeffrey Hsu, Pawan Sasanka Ammanamanchi, Dirk Groeneveld, Eric Tang, Charles Foster, kkawamul, xagi dev, uyhcire, Andy Zou, Ben Wang, Jordan Clive, igor0, Kevin Wang, Nicholas Kross, Fabrizio Milo, and silentvOx. Fleuther /Im- evaluation-harness: v0.3.0, 2022. URL httpps://oi...... http:///.o..or////// 10lgg/1.... wtt.5521//i/oo 7413426
Kshitij Gupta, Benjamin Thérien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Bugene Belilovsky, Irina Rish, and Timothée Lesort. Continual pre-training of large language models: How to re-warm your model? In Workshop on Efficient Systems for Foundation Models @ ICML2023, 2023. URL https://openrevier wtt.:riiiiii.eetttoorniiiai Pg7PUJe0TI
Namgi Han, Nobuhiro Ueda, Masatoshi Otake, Satoru Katsumata, Keisuke Kamata, Hi- rokazu Kiyomaru, Takashi Kodama, Saku Sugawara, Bowen Chen, Hiroshi Matsuda, Yusuke Miyao, Yugo Miyawaki, and Koki Ryu. llm-jp-eval: Automatic evaluation tool for Japanese large language models [llm-jp-eval: A ソD B 2 fili
J (in Japanese). In the 30th Annual Meeting of Japanese Association for Natural Language Processing (NLP2024), 2024. URLhttps: / //www//www/../ anlp pptpprccediiggs//////ppppp gamual_meeting! 2024/pdf _ilfffdir//A di--dd-2rr//.e.. A.../A8-2.110 pdf
Tahmid Hasan, Abhik Bhattacharjee, Md. Saiful Islam, Kazi Mubasshir, Yuan-Fang Li, and Rifat Shahriyar. XL-sum: Large-scale multilingual abstractive summarization for 44 languages. In Findings of the Association for Computational Linguistics (ACL), pp. 4693-4703, 2021. tttpssttts//// httphooooooyooog/211111/// http://22.....iinnnaa.......ee
12

---

Published as a conference paper at COLM 2024
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A massively multilingual multi-task benchmark for evaluating cross- lingual generalisation. In Proceedings of the 37th International Conference Machine Learning, pp. 14.44411442 2020. URL htt.ttt::/ppccceeeeee
mlr. www.srresssveeee
v1 wwtp////19/....ee//// html
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, Hyoukjoong Lee, Jiquan Ngiam, Quoc Le, Yonghui Wu, and zhifeng Chen. GPipe: Efficient training of giant networks using pipeline parallelism. In Advances in Neural Information Processing Systems, volume 32, 2019. URL httpsttt/////rrcdddddda neerips pdf
cc/paper_ cc/paper_files/ppeeeeee wtt.t///wwaeoo////////////////a/
Ai Ishi,, Naoya Inoue, and Satoshi Sekine. Construction of Japanese multi-hop QA dataset for QA systems capable of explaining the rationale [ & BH & 7
// J 4 tz 8 0) A il A ト z y 7qa 2 t y } fis x
(in Japanese). In the 29th
Annul Meeting of Japanese Association for Natural Language Processing (NI.P2023), pp. 2088- 2093, 2023. https: / //www/ anlp. http://oceeenngs///eeppppp annual/ http:/aanaaaaaaaianagee thetinng/22230 2/ _dir/
Q8-14.pdff
Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. Lifelong pretraining: Continually adapting language models to emerging corpora. In Proceedings of BigScience Episode #5 Workshop on Challenges & Perspectioes in Creating Large Language Models, pp- 1-16, 2022.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 1601-1611, 2017. URL ptttts://acllaee fttt:::://caaathologg.o www.atthhooooooooo/oo1--o.. org/p17- 1147
Ai Kawazoe, Ribeka Tanaka, Koji Mineshima, and Daisuke Bekki. An inference problem set for evaluating semantic theories and semantic processing systems for Japanese [H A |u f } -t y } o) []] (in Japanese). In the 21st Annual Meeting of Japanese Association for Natural Language Processing (NLP2015), pp. 704-707, 2015. URL https httppooeeeiigg//////ooopop/sssee aacpiinggg/nnnnaanaaa aaaaaannneetiinneeeeaaa www.mmmetttinnneeee http://i/////////ppddddd/////11-
Vijay Anand Korthikanti, Jared Casper, Sangkug Lym, Lawrence McAfee, Michael Andersch, Mohammad Shoeybi, and Bryan Catanzaro. Reducing activation recomputation in large Transformer models. Proceedings of the 6th Machine Learning and Systems Conference, 2023.
Taku Kudo and John Richardson. Sentenceliece: A simple and language independent subword tokenizer and detokenizer for neural text processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 66-71, 2018. URL
Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. JGLUE: Japanese general language understanding evaluation. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pp. 2957-2966, 2022. URL
wrw.:e...rr/0/00 2022
1rec-1.317
Jinhyuk Lee, Wonjin Yoon, Sungdong Kim, Donghyeon Kim, Sunkyu Kim, Chan Ho So, and Jaewoo Kang. BioBERT: a pre-trained biomedical language representation model for biomedical text mining. Bioinformatics, 36(4):1234-124) 1240, 2019. URL http:tttt////oorr/////
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httplhpp:s/gggpeeeeeeeeeeeeee//e
Bkg6
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381,2399,, 2018.
U http://aclaaattooooooooo//
13

---

Published as a conference paper at COLM 2024
Makoto Morishita, Katsuki Chousa, Jun Suzuki, and Masaaki Nagata. JParaCrawl v3.0: A large-scale English--apanese parallel corpus. In Proceedings of the Thirteenth Language Resources and Evuluation Conference, pp. 6704-6710, 2022. URL https://aclaattheee
org/2022.1rec-1..777
Deepak Narayanan, Amar Phanishayee, Kaiyu Shi, Xie Chen, and Matei Zaharia. Memory- efficient pipeline-paralle DNN training. In Proceedings of the 38th International Conference on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 7937-7947, mlr, press vttp//111///aa// mar.:aannnaaa html
2021. URL /tRp:
Op Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapirg, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brak- man, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, I'amela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Fran- cis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, lan Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, llya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thomp- son, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sher- win Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. GPT-4 technical report, 2023.
14

---

Published as a conference paper at COLM 2024
Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. The WefinedWeb dataset for Falcon LLM: Outperforming curated corpora with web data, and web data only. arXiv:2306.01116, 2023.
Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira. Sabiá: Por- tuguese large language models. arXiv:2304.07880, 2023.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Perey Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, 2016. URL https://
oclanthology.org///. 1264
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (ACL), pp. 784-789, 2018. URL htttt:::/aclaathlllllo/ooo org/
P18-2124/
Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. Empowering Cross-lingual Abilities of Instruction-tune tuned Large Language Models by pranslation-follwwiii demonstrations. arXiv:2308.144 14186, 2023.
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve. Code Llama: Open foundation models for code. arXiv:2308. 12950, 2023.
Vin Sachidananda, Jason Kessler, and Yi-An Lai. Efficient domain adaptation of language models via adaptive tokenization. In Proceedings of the Second Workshop on Simple and Efficient Natural Language Processing, pp. 155-165, 2021. URL https://acllanthology org//022.sustaannlpp.
Thomas Scialom, Tuhin Chakrabarty, and Smaranda Muresan. Fine-tuned language models are continual learners. In Proceedings of the 2022 Conference on Empirical Methods in Nat- ural Language Processing, pp. 6122, 2022. URL htttps://aclanthologgg hwwpcehhhooooooog//00000 2022
emnlp-main. 410
Satoshi Sekine. Development of a question answering system focused on an encyclopedia L 5 A A [ (in Japanese). In the 9th Annual Meeting of Japanese Association for Natural Language Processing (NLP2003), pp. 637-640, 2003. URL
https://www, anlp jp/ pww.eedincss//////oo hww.anlp.pp/prroceedng////////// anaualnnnaaaa meeting//003//pittt. 2ttp:/000000ppp01000006--16 pdf
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain chain-of-thought reasoners. In the Eleventh International Conference on Learning Representations, 2023. /ttLittts//ppeerevieee formm?iid
[RRwGCX= IXp
Tomoki Sugimoto, Yasumasa Onoe, and Hitomi Yanaka. Jamp: Controlled Japanese tem- poral inference dataset for evaluating generalization capacity of language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), pp. 57-65, 2023. URL htLnttts///ccaaahollllo
org/ 2023. acl-srw.
Chul Sung, Tejas Dhamecha, Swarnadeep Saha, Tengfei Ma, Vinay Reddy, and Rishi Arora. Pre-training BERT on domain resources for short answer grading. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 6071-6075, 2019.
URL https: URLhttps:
15

---

Published as a conference paper at COLM 2024
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. CommonsenseeQ: A
question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4149 4158, 2019. URL org/N19-1421
https://acclantollogy
Alexey Tikhonov and Max Ryabinin. It's all in the heads: Using attention heads as baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computational Linguistics (ACL), pp. 3534-3546, 2021. httphtts::/ org/ /0// 18653/
v1/2021. findings-ncl......
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. arXiv:2307 .09288, 2023.
Hitomi Yanaka and Koji Mineshima. Assessing the generalization capacity of pre-trained language models through Japanese adversarial natural language inference. In Proceedings of the Fourth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP, pp. 337-349, 2021. URL htttt:://aalaattto/t/t
org/ www.kboxmlpp-1.2222
Hitomi Yanaka and Koji Mineshima. Compositional evaluation on Japanese textual en- tailment and similarity. Transactions of the Association for Computational Linguistics, 10: 1266-1284, 2022. URL htt.h: htap::::///llaahhooogg.. org/ 2022..tccl. 1.73
Yunzhi Yao, Shaohan Huang, Wenhui Wang, Li Dong, and Furu Wei. Adapt-and-distill: Developing small, fast and effective pretrained language models for domains. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pp. 460-470, 2021. URL https://aclanthhlle.
Jiacheng Ye, Xijia Tao, and Lingpeng Kong. Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability. arXiv:2306.06688, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag:
Can a machine really finish your sentence? Proceedings of the 57th Annual Meel- ing of the Association for Compulational Linguistics, pp. 4791-4800, 2019. URL https
/aaclanthology. .wt.
Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond English: An empirical study on language capability transfer. arXiv:2401.01055,
2024.
Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. Extrapolating large language models non-english by aligning languages. ArXiv:2308.049488 2023.
A Swallow corpus
Swallow Corpus was constructed by extracting and refining Japanese text from the Common Crawl archives, which consist of 21 snapshots collected between 2020 and 2023, comprising approximately 63.4 billion pages. The resulting corpus contains about 312.1 billion characters (approximately 173 million pages).
16

---

Published as a conference paper at COLM 2024
B Ratio of training data
The ratio of mixing Japanese and Englishh texts was determined through preliminary experi- ments. We tested two experimental settings with a JA.EN ratio of 5:5 and 9:1, and compared the average performance on Japanese tasks after training on approximately 20B tokens. In the preliminary experiments, the 9:1 setting slightly outperformed the 5:5 setting in terms of the average score on Japanese tasks, hence we adopted the 9:1 JA:EN ratio. While the goal was to construct powerful LLM specialized for the target language, transferring knowledge from English was also one of the objectives. It has been shown that including 1% of the pre-trained data in training can prevent catastrophic forgetting (Scialom et al. 2022), but due to the unavailability of Llama 2's training data and budgetary constraints, the preliminary experiments were conducted only with the 5:5 and 9:1 ratios.
C Training loss curves
3.0
Swallow-7b
2.8
Swallow-13b
2.6
Swallow-70b
Training loss
2.4
2.2
2.0
1.8
1.6
1.4
1.2 OB
20B
40B
60B
80B
100B
Billions of tokens
Figure 8: Training loss curves (number of tokens trained and loss values) for Swallow 7b,
13b, and 70b
D Distributed parallel training of models
Training large language models on single GPU is challenging due to both GPU memory constraints and the time required for training. In terms of GPU memory, even using the latest H100 80GB, it is difficult to train the 7B model used in this study. Moreover, even if the model parameters, gradients, and optimizer states could fit on a single GPU, training on single GPU would require unrealistic amount of time to complete. Therefore, in this study, we adopted distributed parallel training, combining data parallelism and model parallelism.
D.1 Training environment
We utilized the Al Bridging Cloud Infrastructure (ABCI) of the National Institute of Ad- vanced Industrial Science and Technology, Japan for training. We employed mixed precision (bfloati6) and used multiple NVIDIA A100 nodes for distributed parallel training. Each node equipped with eight NVIDIA A100 40GB GPUs, and the nodes are interconnected via InfiniBand HDR.
17

---

Published as a conference paper at COLM 2024
D.2 Distributed training
Number of Parameters
DP
TP
PP
SP
Distributed Optimizer
7B
16
2
v
13B
8
2
4
V
70B
8
V
Table 7: Distributed training settings. DP, TP, PP, and SP represent Data Parallelism, Tensor Parallelism, Pipeline Parallelism, and Sequence Parrallelsmm respectively.
To efficiently perform the training process, adopted 3D parallelism, which integrates data parallelism, tensor parallelism, and pipeline parallelism, aiming for high computa- tional efficiency and efficient memory utilization. We used the Megatron-LMM library for training. Table 7able?shove shows the distributed training settings for each model size. In addition, we incorporated the following techniques:
Efficient Memory Consumption By using the Distributed Optimizer in Megatron-LM, we distributed the optimizer states across data-parallel processes and eliminated redun- dancy, reducing the required memory usage. The Distributed Optimizer efficiently communicates using Reduce Scatter and All Gather, enabling memory reduction with the same communication cost as regular data parallelism.
Topology-aware 3D Mapping In 3D parallelism, Transformer blocks are distributed across multiple GPUs using pipeline parallelism, and the parameters within each layer are distributed using tensor parallelism. As proposed in Megatron-LMM (Narayanan) al. 2021), we placed the workers requiring more communication (tensor parallel workers) within the same node. This is because intra-node communication using NVLink is faster than inter-node communication. Additionally, considering the communication for gradient averaging in data parallelism, placed data-pate data-parallel workers within the same node as much as possible. Pipeline parallelism requires less communication compared to other parallelization methods, using P2P (Point- to-Point) communication. Therefore, we placed pipeline stages across nodes.
Adoption of 1F1B for Memory Efficiency By using PipeDream-
(Narayanan et al. 2021), 1F1B (one forward pass followed by one backward pass) pipeline par- allelism, we ensured that only number micro-batches less than or equal to the number of pipeline stages require activation, improving memory efficiency compared to GPipe (Huang et al. 2019).
Parallelization
using Sequence Parallelism Tensor parallelism parallelizes Self-Attention and MLP blocks, but not Layer-Norms and Dropouts, resulting in redundant mem- ory usage of these components across tensor processes. To improve ef- ficiency, we utilized Sequence Parallelism (Korthikanti et al.) 2023). By using Se- quence Parallelism in conjunction with tensor parallelism, memory efficiency can be achieved without the overhead of communication costs.
D.3 Computational efficiency
Table the computational performance in TFLOPS per GPU during the actual training process. In terms of execution efficiency, the 70B model achieved over 50%, indicating that the training was conducted efficiently
latttpa:/giithu.. http://hhh....emm/////on//ttaaaa
The theoretical values (312 IFLOPS) are extracted from the specifications of the NVIDIA A100 BFIOOATI6 Tensor Core (without sparsity):
https://www.nvin wwwiia..iiiiiiiaaa com un-uusdata- cen.etrracntt///000/
18

---

Published as a conference paper at COLM 2024
Params
Nodes
TELOPS/GPU
Execution efficiency
7B
4
134
43.0 %
13B
143
45.8 %
70B
32
158
50.6 %
Table 8: TFLOPS/GPU for each model parameter
D.4 Computation required for training
The training of Swallow 7b, 13b, and 70b required approximately 5.0 1021 FLOPS, 9.4 x 10a1 FLOPS, 5.0 1022 FLOPS of computation, respectively. 12
E Details of experimental setup
E.1 Vocabulary expansion method
The Japanese vocabulary constructed using randomly sampled subset (1.5B tokens) of the word-segmented Swallow Corpus. However, to treat symbols as independent subwords, words containing symbols were split at both ends of the symbols. Following Llama 2, we used the BPE algorithm implemented in Sentence post-processing steps were applied to the subword vocabulary constructed by SentencePiece. Firstly, the whitespace escape characters added by Sentence were removed. This is because, unlike during the vocabulary construction phase, word segmentation using MeCab is not performed in tokenization during training and inference. Secondly, the size of the additional vocabulary was intentionally adjusted to be a multiple of 8. This a measure to facilitate distributed parallel training of the model.
(Kudo & Richardson, 2018). Two
The scores of the added subwords were used as-is from the BPE results output by Senten- cePiece. This decision made based on the judgment that conflicts between the merge rules of the original vocabulary and the added vocabulary are extremely rare. In fact, the original vocabulary of the Llama 2 tokenizer does not contain Japanese subwords of two or more characters.
Issues in evaluating natural language inference tasks
For the natural language inference datasets included in Ilm-jp-eval, more specifically, Jamp (Sugimoto et al. (023), JaNLI (anakka & Mineshimal 2021), JNLI (Krrihara et al. 2022), JSeM (Kawazoe et al. 2015),, and JSICK (Yanaka Mineshima, 2022), we contirmed score fluctuations due to the class imbalance in multiple models. First, Figure 9 shows the class distributions both the ground truth and the predictions made by Swallow-7b. Both the ground truth and predictions are greatly imbalanced, with the most frequent class accounting for over 95% of all predictions three datasets. As result, scores significantly whether the most frequent classes in predictions and ground truth align by chance. Figure shows the learning curve of Swallow-7b on these datasets. While fluctuations of about 40 points occurred in two datasets, this was due to transitions in the most frequent class of predictions. The imbalance in prediction was not unique to Swallow but was also observed in other 7b and 13b models (Figure [1). From these obser- vations, we concluded that it is misleading to discuss natural language inference abilities based solely on scores, leading to the exclusion from our evaluation. For a fair and stable evaluation of this task, addressing the imbalance in ground truth classes is recommended.
fluctuate depending
12The calculation of FLOPs is based on the formula presented in APPENDIX: FLOATING-POINT OPERATIONS of (Narayanan al. 2021, with modifications made to adapt to the laama architecture.
19

---

Published as a conference paper at COLM 2024
1.0
Class labels
Class distribution
0.8
entailment
neutral
0.6
contradiction
0.4
non-entailment
yes
0.2
no
0.0
unknown
Jamp (3)
JaNLI (2)
JNLI (3)
JSeM (4) JSICK (3)
30000
undef
Datasets
Figure 9: Class distributions in natural language inference task datasets. For each dataset, the left bar presents the ground truth, and the right bar does the prediction by Swallow-7b. The numbers parentheses indicate the number of classes.
Datasets
0.6
Jamp
Accuracy
JaNLI
0.4
JNLI
JSeM
JSICK
0.2
20
40
60
80
100
Number of training tokens [x109]
Figure 10: Accuracy curve Swallov-7b on natural language inference during training.
G Evaluation methods for Japanese and English
Tables and present the evaluation benchmarks for Japanese and English, respectively.
The Japanese benchmarks include llm-jp-eval (Han et al. 2024) for a comprehensive eval- uation of language models Japanese, and the JP Language Model Evaluation Harness, which is designed for more focused assessment of language model capabilities. The datasets cover a range of tasks including multiple-choice and open-ended question an- swering (JCQA (Kurihara et al. 2022), JEMHQA (Ishii al. (023), NIILC (Sekine 2003)), machine reading comprehension ([SQuAD (Kurihara et al. (022)), automatic summarization (XL-Sum (Hasan et al. 2021)) ), arithmetic reasoning (MGSM (Shi et al.) 2023)), and machine translation (WM1'20 En-Ja and Ja-En (Barrault et al. 2020 Below, we describe six notable task datasets included in these benchmarks.
JCQA JCommonsenseeA (JCQA) is a Japanese question-answering dataset in a five-choice format that assesses common sense knowledge. is the Japanese version of Common- created to evaluate common sense reasoning ability, and consists of pairs of question sentence and answer choices. The knowledge covered by JCQA has good balance of Japanese linguistic knowledge and common sense
senseQA (Talmor et al., 2019), which
20

---

Published as a conference paper at COLM 2024
Majority class proportion
1.00
Models
JSLM Beta 7B
0.75
Llama 2 7B
0.50
Swallow 7B
LLM-jp-13B v1.0
0.25
Llama 2 13B
Swallow 13B
0.00
Jamp
JaNLI
JNLI Datasets
JSeM
JSICK
Figure 11: The proportion of the majority class in predictions by various LLMs.
knowledge. Moreover, since the incorrect answers generated by language models, the dataset contains questions that cannot be solved just by avoiding semantically unrelated choices.
JEMHQA JEMHopQA (JMHQA)) is a Japanese open-ended question answering dataset that was originally designed multi-hop QA task. In our task setting, the dataset is used to evaluate a model's ability to generate answers directly from the given questions. The questions in JEMHQA are created using information from the Japanese version of Wikipedia, ensuring diverse range of topics and entities. JEMHQA serves as an important benchmark for evaluating the extent of knowledge and ability to answer questions through reasoning with that knowledge.
JSQuAD JSQuAD Japanese machine reading comprehension dataset, which is the Japanese version of SQuAD (Rajpurkar et 2016). It focuses on question-answering style, which involves reading document and a question, and then extracting a segment of text in the document as the answer. JSQuAD utilizes article paragraphs from the Japanese version of Wikipedia as its source documents.
NILC NIILC is a dataset created for the purpose of developing question-answering systems in Japanese, featuring relatively straightforward questions that can be answered by referencing an encyclopedia. This makes it valuable for evaluating the encyclopedic knowledge of LLMs. We use only the question and answer pairs from the dataset, although the original dataset includes additional metadata like question types and evidence sentences.
XL-Sum XL-Sum Japanese version dataset created by extracting the Japanese portion from XL-Sum, large-scale summarization dataset collected from BBC News articles, and further filtering it into subset suitable for abstractive summarization. The dataset was filtered by calculating the 15-gram overlap rate between articles and summaries, selecting pairs with low overlap rates. Consequently. this dataset necessitates the ability to paraphrase and abstract information, rather than simply extracting sentences.
MGSM MGSM is a dataset created by selecting 250 elementary school arithmetic word problems from GSMBK (Cobbe et al., 2021) and manually translating them. As it deals with Japanese arithmetic word problems, it requires both Japanese linguistic knowledge and mathematical reasoning ability.
21

---

Published as a conference paper at COLM 2024
Evaluation Japanese
Model
Tokens
JCQA JEMHQA NIILC JSQuAD XL-Sum MGSM
En-Ja
Ja-En
Avg
20
39.2
55.4
51.2
83.5
14.7
7.2
22.4
14.6
36.0
40
45.8
52.4
52.6
84.6
15.9
8.8
23.6
14.4
37.2
Swallou-7B-VE
60
42.2
51.2
58.3
84.3
18.4
10.4
24.6
14.7
38.0
80
49.2
49.5
60.3
86.0
18.4
9.6
24.9
15.0
39.1
100
48.1
50.8
59.7
85.7
18.3
12.4
25.1
15.1
39.4
20
45.8
52.2
48.7
84.6
19.0
8.0
21.8
13.6
36.7
40
39.1
54.1
52.7
84.7
18.2
22.7
14.7
36.7
Swallov-7---V
60
49.2
54.9
57.2
86.4
20.4
9.2
23.6
15.2
39.5
80
52.3
55.4
58.6
87.1
21.8
8.8
23.8
15.1
40.4
100
54.3
54.3
57.3
86.8
21.2
12.0
24.1
15.1
40.6
20
76.6
51.8
58.6
88.5
18.9
12.0
24.4
16.0
43.3
40
75.3
52.0
59.4
88.8
19.5
15.6
25.9
16.3
44.1
Swallox-13B-VE
60
50.5
27.0
17.3
44.7
80
75.8
48.9
63.5
89.7
21.2
22.8
27.1
17.5
45.8
100
78.4
50.6
64.0
90.1
21.7
20.4
27.2
17.7
46.3
20
72.9
46.7
54.0
88.6
22.7
12.0
23.7
14.5
41.9
40
69.4
50.5
60.0
89.2
21.0
12.8
25.8
16.8
43.2
Swallow-135
60
76.4
53.9
61.6
89.9
23.1
16.0
26.7
17.4
45.6
80
51.4
90.0
23.2
26.8
17.3
46.6
100
77.1
54.4
63.5
90.3
22.9
21.2
27.4
18.2
46.9
20
90.8
59.9
65.3
92.0
22.2
38.0
28.4
22.3
52.4
40
92.1
62.0
69.2
92.0
22.6
40.0
29.8
22.1
53.7
Swallox-70B-V6
60
92.2
61.2
67.8
91.8
22.7
48.0
29.7
22.7
54.5
80
93.4
60.4
91.8
22.5
30.6
23.1
54.8
100
93.5
62.9
69.6
91.8
22.7
48.4
30.4
23.0
55.3
20
91..3
57.1
63.1
38.0
28.7
22.2
52.4
40
92.8
54.6
64.5
91.8
24.7
40.8
29.8
23.0
52.7
Swallou-70B--10 VE
60
93.5
58.1
68.5
92.2
26.4
42.8
30.0
23.0
54.3
80
93.9
58.6
70.0
92.3
27.4
48.4
30.2
22.6
55.4
100
57.6
92.5
27.6
30.4
23.2
55.4
Table 9: Evaluation of Swallow models in Japanese tasks with and without vocabulary expansion. "Tokens" represent the number of training tokens.
H Detail analysis of vocabulary expansion impact
Table and show the absolute performance scores for Swallow with and without vocab- ulary expansion (Swallow -VE). While Figure Souure/showe shows relative changes, this table provides comprehensive view of the absolute scores across different evaluation tasks.
Regarding Japanese language capabilities, we analyzed the overall impact of vocabulary expansion on performance across various tasks and model sizes. Excluding the XL-Sum task, the only dataset where we observed relative change of more than 10% is JCQA for the Swallox-7B-VE model. This kind of change was not observed in other model sizes or evaluation tasks. Table indicates that the training trajectory of JCQA score are not stable. Unlike the XL-Sum task, where the model without vocabulary expansion (-VE) consistently outperforms the model with vocabulary expansion (VE), JCQA does not exhibit such consistent trend. For instance, at 40B training tokens, the VE model outperforms the -VE model in JCQA. Based these evaluation results and the observed instability, we conclude that the impact vocabulary expansion on overall performance is minor except for XL-Sum task.
22

---

Published as a conference paper at COLM 2024
Evaluation English
Model
Tokens
OBQA
TrQA
HS
SQuAD2
XW
GSM8K
Avg.
20
29.8
46.5
53.1
28.6
88.6
7.8
42.4
40
31.8
43.8
52.3
30.5
88.2
42.4
Swallou-7B-VE
60
31.2
46.1
52.4
31.3
88.3
9.2
43.1
80
31.4
47.9
52.7
30.4
88.7
9.7
43.5
100
31.8
48.4
53.1
31.3
88.2
11.3
44.0
20
30.6
48.2
52.9
27.6
87.8
7.9
42.5
40
30.6
46.2
52.2
29.2
88.0
42.4
Swallow-7B--VE
60
30.0
48.6
52.9
31.2
88.7
9.7
43.5
80
32,0
49.6
53.0
29.8
88.5
8.5
43.6
100
31.8
50.8
53.3
29.2
88.2
9.9
43.9
20
33.6
56.8
56.3
30.5
89.6
14.8
46.9
40
33.0
53.4
55.5
33.4
89.6
15.9
46.8
Swallow-13B-VE
60
34.0
55.5
55.7
32.4
89.9
47.5
80
33.8
58.2
56.2
33.0
90.3
19.8
48.5
100
35.0
58.5
56.6
34.1
90.8
20.4
49.2
20
35.2
56.8
56.2
31.1
89.9
13.7
47.1
40
33.6
55.2
55.6
33.4
89.9
15.6
47.2
Swallow-13B
60
33.4
57.5
56.7
34.5
89.9
16.8
48.1
80
35.2
34.5
90.0
17.0
48.8
100
34.6
60.3
57.0
34.8
90.1
17.5
49.0
20
41.6
78.2
64.1
32.7
92.4
39.9
58.1
40
41.0
76.0
64.1
35,9
92.2
43.5
58.8
Swallou-70B-VE
60
42.2
76.8
64.2
36.3
92.1
45.4
59.5
80
42.6
37.4
92.2
48.3
60.4
100
42.2
77.6
64.6
37.5
92.0
48.7
60.4
20
40.8
78.1
64.2
35.1
92.3
41.1
58.6
40
41.6
76.8
64.0
31.0
91.9
43.4
58.1
Swallox-70B--VE
60
41.6
77.1
64.2
34.0
92.1
46.9
59.3
80
43.6
78.0
64.6
33.7
92.4
49.1
60.2
100
42.0
78.3
64.7
33.3
92.2
60.0
Table 10: Evaluation of Swallow models in English tasks with and without vocabulary expansion. "Tokens" represent the number of training tokens.
I Comparison of models trained from scratch and other continual pre-training models
Table 11 shows the evaluation results of pre-trained models and continual pre-training models, with the majority developed in Japan (see Table in the appendix for the sources of the models). Some LLMs trained from scratch outside of Japan (Mistral v0.1, Qwen- 7B, Qwen-14B) show higher Japanese performance than Llama 2. The models that are continually pre-trained from these LLMs (japanese-saabellm-baa www--base-gama-77bb nekomata-7b, nekomata-14p show higher average scores than Swallow, suggesting that the performance differences in the base models are reflected.
L.1 Details of models trained from scratch
calm2-7b model with the Llama architecture trained on 1.3T tokens of Japanese and English corpora.
11m-jp-13b-v1.0 A model with the GPT-2 architecture trained on 300B tokens of Japanese and English corpora.
PLaMo-13b A model trained on 180B tokens of Japanese, 1.32T tokens of English, totaling 1.5T tokens.
23

---

Published as a conference paper at COLM 2024
Model
Japanese evaluation dataset
JCQA
JEMHQA
NII.C JSQuAD XI-Sum
MGSM
En-Ja
Ja-En
Avg
calm2-7b
22,0
50.5
50.7
78.0
2.3
6.0
23.5
15.0
31.0
Stable LM B 7b
36.1
44.8
44.3
83.2
22.0
7.2
19.5
12.3
33.7
ELYZA-7b
57.9
47.0
40.2
82.3
13.1
6.0
18.0
12.9
34.7
youri-7b
46.2
47.8
50.0
85.1
19.6
26.7
19.7
37.7
Mistral v0.1 7B
73.0
42.5
27.2
85.6
20.1
17.6
14.1
17.3
37.2
Stable LM 7b
73.6
46.4
55.7
89.1
22.9
16.8
23.9
15.6
43.0
Qwen-7B
77.1
42.3
23.8
85.9
13.7
21.6
16.9
18.0
37.4
nekomata-7b
74.2
49.3
87.1
16.8
12.4
26.7
Llama-2-7b
38.5
42.4
34.1
79.2
19.1
7.6
17.8
17.4
32.0
Swallow-7b
48.1
50.8
59.7
85.7
18.3
12.4
25.1
15.1
39.4
11m-jp-13b-v1.0
22.6
47.9
38.6
77.4
10.8
2.4
19.6
11.9
28.9
PLaMo-13b
22.7
51.9
41.4
76.2
10.3
15.8
12.0
ELYZA-13b
74.0
42.6
46.8
87.2
14.1
3,2
22.0
14.9
38.1
Qwen-14B
88.3
42.4
32.2
89.8
18.5
38,8
22.2
22.2
44.3
nekomata-14b
91.7
57.8
61.1
91.5
21.3
35.6
29.9
23.1
51.5
Llama-2-13b
70.0
44.2
41.7
85.3
21.4
13.2
21.5
19.8
39.6
Swallow-138
78.4
50.6
90.1
21.7
20.4
46.3
Stable LM / 70b
91.2
49.3
60.4
91.9
25.7
41.6
27.7
23.4
51.4
Llama-2-70b
86.9
46.6
52.6
90.8
23.6
35.6
26.4
24.0
48.3
Swallow-70b
93.5
62.9
69.6
91.8
22.7
48.4
30.4
23.0
55.3
Table 11: Evaluation results Japanese evaluation datasets.
J Sources of evaluated models
Table 12 lists the names of the models used in the experiments and their corresponding Hugging Face Model Names. The abbreviations in the "Training detail" column of Table 12 defined follows: VE for vocabulary expansion, CT for continual pre-training, and PT for pre-training from scratch.
The models with "Ja & En" in their Training detail were trained both Japanese and English data, while those with "Zh & En" were trained using Chinese and English data. Models with "Ja" were trained solely on Japanese data, and those with "En" were primarily trained on English data.
Note that the "Training detail" column presents the major data source used for training. However, some models may also use small amounts of other types of data in their training data, for example, source code, mathematical texts, and corpora in other languages.
24

---

Published as a conference paper at COLM 2024
Model
Training detail (PT, Ja & En)
Model name on Hugging Face
calm2-7b
cyberagent/calmmg calm2-7b
Stable LM 7b
CT, Ja & En)
ELYZA-7b
(TT, Ja)
Alyaa/BL772A-jppnne--lamm--21 rinna/youri-7b
youri-7b
(CT, Ja & En)
Mistral v0.1 (7B)
(PT, En)
mistrali//Mistrrl-BB-.111
Stable LM y 7b
(CT, Ja & En)
Qwen-7B
(PT, Zh & En)
Qwen/Qwen-7B
nekomata-7b
(CT, a En)
rinna/nekomata-7b
Llama-2-7b
(PT,
meta-laama/llama-22bbhff
Swallow-7b
VE, VE,
En)
tokyotech-limm/Swalloowhhh
11m-j1-13b-v1.00
(PT, & En)
Ilm-jp
1mmip/1Imm-p-113-v11..
PLaMo-13b
(a & En)
pfnet/plamin plaeoo/3lm--13bbb
ELYZA-13b
(CT,
elyza
Qwen- 14B
(PT, Zh & En)
Qwen/Qwen-14B
nekomata-14b
(CT, Ja & En)
rinna/nekomata-14h
Llama-2-13b
(PT, En)
met-llama/Llama-2113hhff
Swallow-13h
(CT, VE, Ja En)
folyoiechhlmm/SSwalloww--hhf
J Stable LM 70b
(CT, Ja & En)
sablitaai/ppnneesssbebbmbeelle--
Llama-2-70b
(PT, En)
meta-llama/Llama-2200b-f (olyoted-llmm/Svallo--0hh-
Swallow-70b
(CT, VE, Ja & En)
Table 12: Evaluated models and their distribution URLs
25


Reviews:
[{"review_id": "xFUW70RInq", "reviewer": "Reviewer_rjny", "summary": "The authors present work that extends baseline Llama-2 large language models to include (more) Japanese-language data through continued pre-training on up to 100B additional tokens of Japanese data. By employing model vocabulary expansion in conjunction with the additional data, the authors improve generation efficiency by up to 78% due to the introduction of Japanese-specific vocabulary terms. Experimental results show marked improvements on Japanese performance on six evaluation tasks in addition to improvements in English-Japanese machine translation (MT). Some performance degradation is noted in English-based evaluation tasks as well as Japanese-English MT performance. Lastly, the authors compare their continued pre-training approach against similarly-sized Japanese language models trained from scratch.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n> Elastic Weight Consolidation\n\nWe agree with your suggestion. Perhaps due to the capacity of the model, there tends to be a decline in the original English proficiency during continual pre-training. Indeed, mitigating this decline is crucial, and we have been implementing a strategy to prevent catastrophic forgetting. We also appreciate your suggestion to explore Elastic Weight Consolidation as another potential approach.\n\n> Language-specific LLM pretraining has been explored in many other works, but the combination of examination of model size and effects of vocabulary expansion may help mitigate this fact.\n\nWe investigated the impact of vocabulary expansion comprehensively across various model sizes, along with the effectiveness of incorporating parallel corpora. We believe this is particularly beneficial for developing large language models for non-English speaking regions to enhance large language models for targeted languages.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The contributions in this paper are systematically explored to show effects of the two major contributions. Vocabulary expansion is demonstrated to increase generation efficiency and generally has little effect on task evaluation (with the exception of a minor performance regression on summarization.) Continued training with additional Japanese monolingual and Japanese-English parallel data improves performance on Japanese evaluation tasks. The authors also demonstrate that pretraining with a strong English baseline model outperforms similar-sized models that were trained from scratch. This pretraining approach is tested on the three common Llama-2 model sizes 7B, 13B and 70B to show that this tactic scales with differing model sizes.", "reasons_to_reject": "Language-specific LLM pretraining has been explored in many other works, but the combination of examination of model size and effects of vocabulary expansion may help mitigate this fact.", "questions_to_authors": "It would be interesting to see results of other ways of mitigating 'catastrophic forgetting' such as Elastic Weight Consolidation in place of Experience Replay, though the implementation details may be tricky.", "extra_scores": {}}, {"review_id": "aVMEinD7cp", "reviewer": "Reviewer_ATKc", "summary": "This works presents a new version of LLAMA2, adapted for Japanese. First, LLAMA's vocabulary is expanded with a Japanese vocabulary, created from the new training data through MeCab, Unidic and BPE. Then, the model is trained by performing continual pre-training. Finally, the new model is tested on several downstream tasks, achieving satisfying results.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review and comments.\n\n> paper relies too much on the appendixes \n\nTo meet the page limit, we carefully selected the content to include in the main text and the appendices. In the camera-ready version, we plan to add one more page, allowing us to move the vocabulary expansion method and the usage of parallel corpora into the main text, which should enhance readability and understanding.\n\n> Despite its importance, the experimental setup is missing. \n\nWe have described our experimental setups in sections 3 of the manuscript. To provide an overview: we use Llama 2 as base models,  expand vocabulary for Japanese, and conduct continual pre-training using a multilingual corpus. To improve clarity, we will rename Section 3 to \"Experimental settings of continual pre-training\" and restructure Section 3.1 and 3.2 to clearly distinguish between the experimental settings and the motivations/rationale behind them.\n\n> the evaluation lacks context\n\nWe would like to clarify that the evaluation tasks and metrics are detailed in Section 3.4 and Tables 2 and 3 of our paper. These tables specify the datasets used and the evaluation metrics. To further clarify, we use few-shot In-Context Learning to solve downstream tasks without fine-tuning, and we have not (intentionally) included evaluation data in the pre-training datasets.In the camera-ready version, we will explicitly state that higher scores indicate better results. Additionally, we will include more information on our approach of using few-shot In-Context Learning without fine-tuning and confirm that the evaluation datasets were not used in the pre-training process.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work describes the construction of a new model and an analysis which could benefit the community.", "reasons_to_reject": "As it is right now, the paper relies too much on the appendixes and is very hard to read and understand. Despite its importance, the experimental setup is missing. Similarly, the evaluation lacks context: It is mentioned which downstream task are being used for the evaluation, but there is no description even of which metrics and corpora are being used. Thus, result tables are not informative enough--we don't even know if greater values means better results or if the corpora could have already been seen at training time!\n\nWhile I find the work interesting, I believe it would benefit from a revision. The main details should be described on the document's body instead of relaying so much on the appendixes.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9gW1yu958y", "reviewer": "Reviewer_Uybi", "summary": "This paper presents an effort in continuing pre-training an LLM to improve Japanese language capabilities. Authors applied and analysed vocabulary expansion and parallel corpus. Finally llama-2 at 7B, 30B, and 70B are continue-pre-trained with more text in Japanese. Through language-oriented tests, it is found that Japanese test scores mostly increase, but English scores drop. These models achieve better scores in comparison with LLMs trained from scratch in Japanese.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.\n\n>  lacks novelty and depth\n\nWe confirmed the training scalability and effectiveness of continual pre-training across different model sizes. This fact has not been investigated comprehensively in the previous studies (§2.1). Our analysis of vocabulary expansion (VE) in §5.1.2 showed that the effect of VE varies across different tasks, which was a non-trivial finding.\n\nAdditionally, we released SoTA Japanese LLMs (as of submission), which also make a non-trivial finding for developing Japanese foundation models. These contributions align with the \"LM for everyone\" topic in the CFP.\n\n> English test scores do drop\n\nTeaching new facts to pre-trained LLMs can degrade the original capability [Gudibande et al., 2024; Gekhman et al., 2024]. Therefore, inspired by experience replay [Chaudhry et al., 2019], we took measures to mitigate the negative impact on English by using a multilingual corpus of Japanese and English (§3.2). Table 4 shows the effectiveness of these mitigations, especially for larger models (70B). In addition, we want to emphasize that the primary goal of this study is to develop a Japanese LLM (§1). \n\n> what \"efficiency\" means here?\n\n\"Efficiency\" refers to the reduction of the number of tokens required to encode/decode the same text. \n\n> \"猫 (cat)\"\n\nYour argument may be true for computers. However, the three byte-level tokens do not correspond to the morphology, phonology, and semantics of 猫.\n\n> Japanese vocabulary size \"up to 16k\"\n\nA preliminary experiment (§E.1) showed that the sequence lengths with 16k, 32k, and 48k Japanese vocabularies were similar (43.8%, 45.9%, and 46.8%, respectively, of that with the original vocabulary), and concluded that 16k was sufficient. We will include this in §3.3 of the camera-ready version.\n\n> new embeddings are \"initialized with the average\"\n\nNew embeddings were not trained at all compared to the existing ones. We followed Yao et al. (2021) (§3.3). Additionally, we confirmed that random initialization caused corrupted generation. We will include this in §3.3.\n\n> Sec 5.1.2 and Figure 4\n\nCorrect. Figure 4 shows the relative change using \"Llama 2-JA without VE\" as the reference point. We will add the absolute score changes to the appendix. \n\n> \"impact of vocabulary expansion on performance is minor.\"\n\nAmong the Japanese evaluation tasks, except for XL-Sum, we described the results as \"minor\" due to the inconsistent positive and negative impacts across different model sizes (§5.1.2).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- the technical points are detailed and easy-to-follow. This work can be consulted by other researchers and engineers looking to continual pre-train a base LLM.\n- Without any doubt, if released, the trained checkpoints will be adopted by practitioners to deal with Japanese texts.\n- the incorporation of parallel corpus into continued pre-training is novel. Note that I have seen it somewhere else [1], but I agree, that both works are rather concurrent.\n\n[1] TowerLLM; https://arxiv.org/abs/2402.17733", "reasons_to_reject": "- I find this work lacks novelty and depth from a research perspective. Continued pre-training has been done and reported by some researchers; the technique of vocabulary expansion has been widely used. I did do not find many new insights in this paper.\n- It is observed that English test scores do drop after continued pre-training, and the paper does not discuss or propose any techniques to mitigate the issues. Is it possible to do continued pre-training with both English and Japanese as opposed to the current setting?\n- There are some confusions or concerns about soundness of the arguments or claims, which I detail in the questions section below.", "questions_to_authors": "I would like to clarify with the authors some points below which I think can affect the soundness of the work:\n\n1. Abstract: \"*The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance*\". I wonder what \"*efficiency*\" means here?\n2. Sec 2.2: \"*For example, a Japanese single-character noun 猫 (cat) is represented by three byte-level tokens <0xE7> <0x8C> <0xAB>, which do not provide any semantic meaning*\"\n    - I understand that if a string is not in the original tokenizer, it will be split up in a more fine-grained way.\n    - However, I might not agree with the statement because it is discussed from the perspective of human readers.\n    - If the three byte-level tokens appear together and often enough during LLM training, they do provide semantic information when being encoded by the model. In other words, I do not see why \"猫 (cat)\" provides more semantic meaning.\n3. Sec 3.3: As a research paper, I'd ask about the reason behind the decision of a Japanese vocabulary of \"*(up to 16k)*\", and why the new embeddings are \"*initialized with the average of the vectors of the subwords segmented by the LLaMA tokenizer*\".\n4. Sec 5.1.2 and Figure 4: Is \"Llama 2-JA without VE\" the reference point, and do you have individual absolute scores for it? I also find the statement of \"*overall impact of vocabulary expansion on performance is minor.*\" to be a bit hand waving because for some tasks and gain/loss are more than 10%, translating to significant absolute differences in test scores.", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper describes an approach to continual pre=train LLaMA for Japanese using 100 billion additional tokens. 

Strengths:
1. a detailed description of the procedure, including vocabulary expansion, continual pre-training step. 
2. Evaluation on a variety of model sizes show its effectiveness. 

Weakness:
1. Limited technical novelty, compared to other work. 
2. Description of important experiment setup is missing. 

Overall, this work made an important contribution despite limited technical novelty. I would be worthwhile to include in COLM.
Decision: Accept