Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities

Authors: kazuki  fujii
taishi  nakamura
mengsay  loem
hiroki  iida
masanari  ohi
kakeru  hattori
hirai  shota
sakae  mizuki
rio  yokota
naoaki  okazaki
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 16:24:07

Abstract: Cross-lingual continual pre-training of large language models (LLMs) initially trained on English corpus allows us to leverage the vast amount of English language resources and reduce the pre-training cost. In this study, we constructed Swallow, an LLM with enhanced Japanese capability, by extending the vocabulary of Llama 2 to include Japanese characters and conducting continual pre-training on a large Japanese web corpus. Experimental results confirmed that the performance on Japanese tasks drastically improved through continual pre-training, and the performance monotonically increased with the amount of training data up to 100B tokens. Consequently, Swallow achieved superior performance compared to other LLMs that were trained from scratch in English and Japanese. An analysis of the effects of continual pre-training revealed that it was particularly effective for Japanese question answering tasks. Furthermore, to elucidate effective methodologies for cross-lingual continual pre-training from English to Japanese, we investigated the impact of vocabulary expansion and the effectiveness of incorporating parallel corpora. The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance, except for the summarization task, and that the combined use of parallel corpora enhanced translation ability.

Main paper: 
 introduction: 
 Large language models (LLMs) such as ChatGPT have attracted significant attention by demonstrating human-level language understanding and generation capabilities, as well as generalizability to various fields. However, many LLMs, including Llama 2 (Touvron et al., 2023) , are primarily trained on English corpora, and their performance in other languages, especially in those with syntactic structures and writing systems greatly different from English, is decreased (OpenAI et al., 2023) . Motivated by this performance gap, establishing methods to build LLMs that excel in Japanese (hereafter referred to as Japanese LLMs) is an important research issue in Japan. In particular, since English language resources are outstanding in terms of quality and quantity, insights on effectively utilizing both Japanese and English language resources are in high demand. For example, it is estimated that there are approximately nine times more English web pages than Japanese web pages 1 . However, pre-training from scratch using both Japanese and English language data requires enormous computational resources, making it difficult to acquire insights in a timely manner. 2 for task details) improved by up to approximately 70%. Therefore, we choose to conduct Japanese continual pre-training from English LLMs, aiming to save computational resources, and obtain insights on how to transfer the knowledge and abilities learned by English LLMs to Japanese. Recently, there has been an increasing number of attempts to adapt LLMs to other languages using continual pre-training (Gupta et al., 2023; Cui et al., 2023; Pires et al., 2023; Zhu et al., 2023; Zhao et al., 2024) , but a comprehensive investigation on the effectiveness of continual pre-training has not been conducted. For example, the relationship between the amount of Japanese data used for continual pretraining and model performance, and the impact of the model size on this relationship, are unclear. We construct Swallow2 , an LLM with enhanced Japanese capabilities, by performing continual pre-training on Llama 2's 7b, 13b, and 70b models using a Japanese corpus. We then evaluate the performance on six types of tasks including question answering and machine translation in both Japanese and English to analyze changes in knowledge and abilities in both languages. As a result, we demonstrated performance improvements in almost all Japanese tasks for all model sizes (Figure 1 ). Furthermore, to identify efficient methodologies for cross-lingual continual pre-training, we also investigate the impact of vocabulary expansion and the effectiveness of Japanese-English parallel corpora. Vocabulary expansion has the effect of shortening token sequence lengths and improving the learning and generation efficiency of Japanese text by adding Japanese characters and words. However, previous studies have not sufficiently analyzed the extra cost of optimizing parameters for the added vocabulary or the impact on performance due to the increase in the amount of learnable text. Therefore, in this study, we conducted a comprehensive evaluation of the impact of vocabulary expansion on the performance of both Japanese and English. Vocabulary expansion improved Japanese text generation efficiency by up to 78%, thanks to a 56.2% reduction in tokens in the Swallow Corpus. This enhancement did not compromise downstream task accuracy, except for summarization.Parallel corpora are known to have the effect of promoting cross-lingual transfer in multilingual models (Chi et al., 2022; Hu et al., 2020; Feng et al., 2022) . However, the effectiveness of parallel corpora in continual pre-training settings of the target language has not been studied in detail. Our experimental results reveal that simply mixing parallel corpora with plain text corpora improves the accuracy of machine translation tasks.The contributions of this study are as follows:• Swallow achieved the highest performance in Japanese among all models developed in Japan (as of December 2023). We demonstrate that cross-lingual continual pre-training can achieve higher performance with fewer computational resources compared to Japanese LLMs trained from scratch.• We show that continual pre-training is effective for improving Japanese abilities, especially question answering tasks that require Japanese knowledge.• We provide evidence that the Japanese performance of language models improves monotonically as the amount of Japanese training data increases.• We show that vocabulary expansion does not affect performance in most tasks, and only degrades performance in automatic summarization.• We reveal that using parallel corpora together enhances translation ability without affecting the performance of other tasks. 
continual pre-training: 
 Continual pre-training is a form of domain adaptation where additional pre-training tasks are performed on a pre-trained language model using text from the target downstream task before fine-tuning the model on that task (Lee et al., 2019; Beltagy et al., 2019; Sung et al., 2019) . Since the emergence of open and high-performance English LLMs, there have been an increasing number of attempts to perform continual pre-training to adapt LLMs to other tasks and languages (Gupta et al., 2023; Cui et al., 2023; Pires et al., 2023; Zhu et al., 2023; Zhao et al., 2024) . However, a comprehensive investigation of the effects of continual pre-training for various model sizes and training data sizes has not been conducted.Vocabulary expansion is a method to increase the vocabulary of a trained LLM. The Japanese writing system differs from English, where kanji characters tend to be converted into UTF-8 byte sequences. For example, a Japanese single-character noun 猫 (cat) is represented by three byte-level tokens <0xE7> <0x8C> <0xAB>, which do not provide any semantic meaning. This treatment not only looks unreasonable as Japanese representation but also increases the sequence length of Japanese text as well as the cost of text generation (Ahia et al., 2023) .Adding Japanese characters and words to the vocabulary can reduce the number of tokens required to represent Japanese text and alleviate this problem. In domain adaptation of text embedding models, it is known that adding vocabulary from the target domain can improve performance (Sachidananda et al., 2021; Yao et al., 2021) . In contrast, the main motivation for vocabulary expansion in continual pre-training of LLMs is to improve training and generation efficiency in the target language, and there is little knowledge about its impact on performance.Examples of vocabulary expansion in building Japanese LLMs3 are limited to evaluating Japanese language ability, and do not evaluate the English language ability. Examples of vocabulary expansion in Chinese LLMs (Cui et al., 2023) only report the performance with vocabulary expansion, and do not compare with a base model. In multilingual LLMs (Ahuja et al., 2023) , it was shown that the vocabulary size for each language correlates with performance, but these studies train longer as the vocabulary is expanded, so the isolated impact of vocabulary size remains unclear.In multilingual text embedding models, it has been reported that pre-training with the language modeling objective using parallel corpora mitigates language-specific hidden state vectors and promotes cross-lingual transfer (Chi et al., 2022; Hu et al., 2020; Feng et al., 2022) .In applications to LLMs, it has been reported that using parallel sentences for instruction tuning can improve translation ability more efficiently than using the same amount of multilingual corpora (Zhu et al., 2023; Ranaldi et al., 2023) . However, the effectiveness of combining continual pre-training with parallel corpora remains unclear. The AdamW optimizer (Loshchilov & Hutter, 2019) was employed for training the models, with hyperparameters β 1 = 0.9, β 2 = 0.95, ϵ = 1.0 × 10 -8 . A cosine learning rate scheduler was used, and the learning rate was set to reach its maximum value at 1,000 warmup steps and finally decays to 1/30 of that value. Additionally, a weight decay of 0.1 and gradient clipping of 1.0 were used. Furthermore, Flash Attention 2 (Dao, 2023) was adopted for improved computational efficiency and memory footprints. Refer to Figure 8 in the appendix for the training loss curve. 
training corpora: 
 When continually pre-training with full parameters, forgetting previously learned knowledge is a concern (Jin et al., 2022) . One method to prevent forgetting is the Experience Replay technique (Chaudhry et al., 2019) . This method involves reusing a portion of the data previously used for training the language model during continual pre-training (Scialom et al., 2022) . Following this approach, our study incorporates a portion of the English corpus in addition to the target Japanese corpus for continual pre-training.The corpus used for continual pre-training includes the Swallow Corpus, which is explained in Appendix A, Japanese Wikipedia4 , and for English, the RefinedWeb (Penedo et al., 2023) and The Pile (Gao et al., 2020) . From these corpora, approximately 100B tokens were sampled for the training data of continual pre-training. The sampling was configured so that 5% of the English text comes from RefinedWeb, another 5% from English arXiv paper texts within The Pile, and the remaining 90% from Japanese texts. The Japanese text comprises about 1.6B tokens from Japanese Wikipedia, with the rest from the Swallow Corpus. The ratio of Japanese to English data in the training set was decided based on preliminary experiments (see Appendix B for details).In this study, we aim to adapt LLMs to languages with different writing systems from the Latin alphabet. For this purpose, we performed vocabulary expansion. Vocabulary expansion refers to the post-hoc addition of vocabulary to an existing LLM. This increases the amount of text that can be trained and generated within the same computational budget, thereby improving computational efficiency. In the vocabulary expansion adopted in Swallow, we constructed Japanese vocabulary, initialized vectors, and added string preprocessing.We conducted preliminary experiments to determine the optimal vocabulary size by comparing sequence lengths with 16k, 32k, and 48k Japanese vocabularies. The sequence lengths were 56.2%, 54.1%, and 53.2%, respectively, compared to using original LLaMA vocabulary.Since there was no significant performance difference in Japanese tasks among these sizes, we selected the 16k vocabulary for its advantage in training speed. The construction of Japanese vocabulary involved creating a vocabulary (up to 16k) using the BPE algorithm on the Swallow Corpus segmented by MeCab5 and the UniDic dictionary6 . We then merged the Japanese vocabulary (subwords) with the original LLaMA vocabulary, resulting in a total vocabulary size of 43,176. Refer to Appendix E.1 for further details on the vocabulary expansion procedure.The vectors for the embedding and output layers of the added subwords were initialized with the average of the vectors of the subwords segmented by the LLaMA tokenizer, i.e., the subwords trained by Llama 2, following previous research (Yao et al., 2021) . This initialization method was adopted because we confirmed that random initialization led to corrupted outputs.String preprocessing was enhanced by adding NFKC normalization to utilize the pre-trained knowledge of alphanumeric characters and symbols in the ASCII code range.The evaluation methods for Japanese and English are shown in Tables 2 and 3 , respectively. The dataset consists of five types of Japanese and four types of English tasks, with few-shot settings for question answering (QA), reading comprehension (RC), automatic summarization (AS), arithmetic reasoning (AR), commonsense reasoning (CR) and machine translation (MT). For details, see Appendix G. We evaluate these downstream tasks under few-shot In-Context Learning without fine-tuning the model. Additionally, the evaluation datasets were not included in our continual pre-training corpora to ensure an unbiased assessment of the model's performance. For all tasks, higher scores indicate better performance.The selection of tasks was based on discussions in LLM-jp (Han et al., 2024) and the methodology of the Llama 2 paper (Touvron et al., 2023) , with an active adoption of tasks related to inference and text generation. The natural language inference task in llm-jpeval was excluded from evaluation due to unstable scores in the 7b and 13b models (see Appendix F). 
effects of continual pre-training: 
 Table 4 shows the evaluation results of Swallow and its base model Llama 2 on Japanese and English tasks. Figure 1 (Kurihara et al., 2022) , JEMHQA for JEMHopQA (Ishii et al., 2023) , NIILC (Sekine, 2003) , JSQuAD (Kurihara et al., 2022) , XL-Sum (Hasan et al., 2021) , MGSM (Shi et al., 2023) , and WMT'20 (Barrault et al., 2020) . (Joshi et al., 2017) , HS for HellaSwag (Zellers et al., 2019) , XW for XWINO (Tikhonov & Ryabinin, 2021) , SQuAD2 (Rajpurkar et al., 2018) , and GSM8K (Cobbe et al., 2021) .vidual tasks8 , Japanese question answering (JCQA, JEMHQA, NIILC) shows a significant improvement of up to 75%, and arithmetic reasoning (MGSM) improves by 36-63%. In contrast, English question answering (TrQA) and arithmetic reasoning (GSM8K) degrade by 6-23%. The change in automatic summarization (XL-Sum) is less than 5%. Machine translation shows contrasting results depending on the direction, with a 15-41% improvement in English-to-Japanese (En-Ja) and a 4-13% degradation in Japanese-to-English (Ja-En). Japanese reading comprehension (JSQuAD) has limited room for improvement as Llama 2 's score is above 0.8, resulting in less than 10% improvement.We analyze the impact of continual pre-training on Japanese abilities and knowledge. In the case of arithmetic reasoning (Ja: MGSM, En: GSM8K), while Llama 2 shows superiority in English (GSM8K > MGSM), Swallow improves MGSM, but GSM8K degrades to a similar level, suggesting that the reasoning ability in English has not been fully transferred to Japanese. Considering the reports of reasoning ability transfer in instruction tuning (Ye et al., 2023) , combining instruction datasets could be a promising strategy to bring Japanese arithmetic reasoning to the same level observed in English.Regarding knowledge, the significant improvement in question answering suggests that the acquisition of Japanese knowledge has progressed. Figure 2 illustrates the impact of continual pre-training on the scoring of each question in the NIILC QA dataset. The dense color in the upper left corner indicates that many questions have shifted from incorrect to correct answers, while the opposite is uncommon. This trend suggests that continual pre-training successfully incorporated new knowledge and corrected inaccurate responses.Table 5 shows the evaluation results of the Swallow and major Japanese LLMs trained from scratch (see Table 12 in the appendix for model references). Note that these models are all general-purpose language models without instruction tuning. Compared to the LLMs trained from scratch in Japan (calm-7b, llm-jp-13b-v1.0, PLaMo-13b) (see Appendix I.1 for details of each model), the average score of Swallow is 8.4 to 17.4 points higher, demonstrating the usefulness of continual pre-training.  
analysis: 
 5.1 Vocabulary expansionTo investigate the impact of vocabulary expansion (VE)specifically, the addition of Japanese words and characterswe compare models pre-trained with and without VE. The model pre-trained without VE is denoted as ¬VE hereafter. Incorporating VE can shorten token sequences, thereby enhancing the efficiency of both training and generating Japanese text. However, the impact of vocabulary expansion on performance is unclear. Intuitively, if the added vocabulary is poorly optimized, it may degrade performance. Yet, studies on domain adaptation scenarios report that VE enhances performance (Sachidananda et al., 2021; Yao et al., 2021) . Additionally, being able to train with more text within the same computational budgets could positively influence performance.Figure 4 shows the performance of Swallow relative to Swallow¬VE, where the vocabulary expansion was not applied to the latter. Regarding Japanese language capabilities, the overall impact of vocabulary expansion on performance is minor. Appendix H provides the absolute scores and a more detailed analysis. When examining specific tasks, we observe the performance change in question answering approximately ±10%, yet there is no consistent trend of either improvement or degradation across different model sizes. Therefore, the increase in the amount of training text brought from the vocabulary expansion does not directly affect performance. Additionally, from Figure 7 , we did not observe any significant difference in convergence characteristics in the performance curve of Japanese questionanswering task. In contrast, automatic summarization (XL-Sum) degraded by about 5-15% with vocabulary expansion for all model sizes9 . This suggests that the impact of vocabulary expansion may be more apparent in tasks that involve processing longer contexts.In the experiments to investigate the effectiveness of parallel corpora, we use Swallow¬VE as the baseline to separate the impact of vocabulary expansion, and examine the performance when incorporating parallel corpora in the continual pre-training. We used JParaCrawl 3.0 (Morishita et al., 2022) corpus, which contains approximately 22 million Japanese-English parallel sentences extracted from the web.We investigate three different settings to utilize a parallel corpus, as shown in Table 6 . These methods vary by training sequence and task format. The training sequence can be "two- staged", where the parallel corpus is used first and followed by the multilingual corpus, or "mixed", where it is combined with the multilingual corpora from the start. The intention behind the two-staged setting is the assumption that parallel sentences could facilitate the switch from English-centric to Japanese-centric training.For task formats, we have "next token prediction" (NTP), which involves concatenating parallel sentences, and "translation instruction" (TI), where the target sentence is predicted based on the source sentence and a translation instruction. Both formats are applied in two directions, Ja→En and En→Ja, for each pair of parallel sentences. The following templates show how parallel sentences are converted into NTP format (top) and TI format (bottom). Note that for the TI format, only the target language sentence is the target for training.[ Figure 5 shows the rate of increase or decrease in scores when using a parallel corpus in conjunction with continual pre-training, with Swallow¬VE without vocabulary expansion as the baseline. Translation performance improved by 9-24% for En-Ja and 14-51% for Ja-En. The improvement in Ja-En is particular to the parallel corpus. Regarding the usage of the corpus, we found that the next token prediction format with a "mixed" setting or the translation instruction format in a "two-staged" setting was effective. In other words, we proved that simply mixing parallel sentences into the multilingual corpus and conducting continual pre-training can effectively improve translation performance. This finding is consistent with the claim that the translation capabilities of LLMs stem from parallel sentences incidentally present in plain text corpora (Briakou et al., 2023) .The relative change in scores for tasks other than translation was within ±15%, without showing consistent improvement or decline across different model sizes. Therefore, no evidence was obtained that the parallel corpus promotes cross-lingual transfer and improves abilities other than translation. 
conclusions: 
 In this study, we developed Swallow, leveraging Llama 2 models for enhanced Japanese language performance through continual pre-training on Japanese datasets, and analyzed their performance to address the lack of comprehensiveness in model size, training data size, and evaluation methods in previous studies. Through our evaluation, we found that continual pre-training significantly boosts Japanese abilities, particularly in knowledgeintensive question answering task. Consequently, we demonstrated that continual pretraining is an efficient approach for achieving high performance, as the continual pre-trained models outperform Japanese LLMs that are trained from scratch. We also observed that performance improves in line with increases in training data. Furthermore, in our quest for a more efficient methodology, we investigated how expanding the vocabulary and incorporating parallel corpora into continual pre-training affect performance. We revealed that while the vocabulary expansion improves computational efficiency, it has little impact on performance except for summarization. Additionally, we found that simply integrating parallel sentences into plain text corpora improves translation performance, particularly for Japanese-English, but it does not hurt the performance of other tasks. The experimental results and analyses in this study provide important insights into the effectiveness of continual pre-training, the impact of vocabulary expansion, and the effects of using parallel corpora in the development of LLMs for non-English languages. 
ethical considerations: 
 Swallow is subject to the same well-recognized limitations of other LLMs, including the inability to update information after the pretraining phase, the potential for non-factual generation such as unqualified advice, and a propensity towards hallucinations.Similar to other large language models, Swallow may produce content that is harmful, offensive, or biased due to being trained on datasets derived from publicly available online sources. We commit to ongoing fine-tuning and plan to release updated versions of Swallow as we make further advancements in resolving these issues. 
reproducibility statement: 
 All models developed in this study (continual pre-training on Llama 2 7B, 13B, 70B) have already been released on Hugging Face. The benchmark datasets used in this study are also publicly available. Therefore, it is straightforward to reproduce our experimental results reported in Table 4 , 5, 11. 
 
 Appendix: Swallow Corpus was constructed by extracting and refining Japanese text from the Common Crawl archives, which consist of 21 snapshots collected between 2020 and 2023, comprising approximately 63.4 billion pages. The resulting corpus contains about 312.1 billion characters (approximately 173 million pages).The ratio of mixing Japanese and English texts was determined through preliminary experiments. We tested two experimental settings with a JA:EN ratio of 5:5 and 9:1, and compared the average performance on Japanese tasks after training on approximately 20B tokens. In the preliminary experiments, the 9:1 setting slightly outperformed the 5:5 setting in terms of the average score on Japanese tasks, hence we adopted the 9:1 JA:EN ratio. While the goal was to construct a powerful LLM specialized for the target language, transferring knowledge from English was also one of the objectives. It has been shown that including 1% of the pre-trained data in training can prevent catastrophic forgetting (Scialom et al., 2022) , but due to the unavailability of Llama 2's training data and budgetary constraints, the preliminary experiments were conducted only with the 5:5 and 9:1 ratios. Training large language models on a single GPU is challenging due to both GPU memory constraints and the time required for training. In terms of GPU memory, even using the latest H100 80GB, it is difficult to train the 7B model used in this study. Moreover, even if the model parameters, gradients, and optimizer states could fit on a single GPU, training on a single GPU would require an unrealistic amount of time to complete. Therefore, in this study, we adopted distributed parallel training, combining data parallelism and model parallelism.We utilized the AI Bridging Cloud Infrastructure (ABCI) of the National Institute of Advanced Industrial Science and Technology, Japan for training. We employed mixed precision (bfloat16) and used multiple NVIDIA A100 nodes for distributed parallel training. Each node is equipped with eight NVIDIA A100 40GB GPUs, and the nodes are interconnected via InfiniBand HDR. To efficiently perform the training process, we adopted 3D parallelism, which integrates data parallelism, tensor parallelism, and pipeline parallelism, aiming for high computational efficiency and efficient memory utilization. We used the Megatron-LM10 library for training. Table 7 shows the distributed training settings for each model size. In addition, we incorporated the following techniques:Efficient Memory Consumption By using the Distributed Optimizer in Megatron-LM, we distributed the optimizer states across data-parallel processes and eliminated redundancy, reducing the required memory usage. The Distributed Optimizer efficiently communicates using Reduce Scatter and All Gather, enabling memory reduction with the same communication cost as regular data parallelism.Topology-aware 3D Mapping In 3D parallelism, Transformer blocks are distributed across multiple GPUs using pipeline parallelism, and the parameters within each layer are distributed using tensor parallelism. As proposed in Megatron-LM (Narayanan et al., 2021) , we placed the workers requiring more communication (tensor parallel workers) within the same node. This is because intra-node communication using NVLink is faster than inter-node communication. Additionally, considering the communication for gradient averaging in data parallelism, we placed data-parallel workers within the same node as much as possible. Pipeline parallelism requires less communication compared to other parallelization methods, using P2P (Pointto-Point) communication. Therefore, we placed pipeline stages across nodes.Adoption of 1F1B for Memory Efficiency By using PipeDream-Flush (Narayanan et al., 2021) , a 1F1B (one forward pass followed by one backward pass) pipeline parallelism, we ensured that only a number of micro-batches less than or equal to the number of pipeline stages require activation, improving memory efficiency compared to GPipe (Huang et al., 2019) .Tensor parallelism parallelizes Self-Attention and MLP blocks, but not Layer-Norms and Dropouts, resulting in redundant memory usage of these components across tensor-parallel processes. To improve efficiency, we utilized Sequence Parallelism (Korthikanti et al., 2023) . By using Sequence Parallelism in conjunction with tensor parallelism, memory efficiency can be achieved without the overhead of communication costs. The Japanese vocabulary was constructed using a randomly sampled subset (1.5B tokens) of the word-segmented Swallow Corpus. However, to treat symbols as independent subwords, words containing symbols were split at both ends of the symbols. Following Llama 2, we used the BPE algorithm implemented in SentencePiece (Kudo & Richardson, 2018) . Two post-processing steps were applied to the subword vocabulary constructed by SentencePiece. Firstly, the whitespace escape characters added by SentencePiece were removed. This is because, unlike during the vocabulary construction phase, word segmentation using MeCab is not performed in tokenization during training and inference. Secondly, the size of the additional vocabulary was intentionally adjusted to be a multiple of 8. This is a measure to facilitate distributed parallel training of the model.The scores of the added subwords were used as-is from the BPE results output by Senten-cePiece. This decision was made based on the judgment that conflicts between the merge rules of the original vocabulary and the added vocabulary are extremely rare. In fact, the original vocabulary of the Llama 2 tokenizer does not contain Japanese subwords of two or more characters.For the natural language inference datasets included in llm-jp-eval, more specifically, Jamp (Sugimoto et al., 2023) , JaNLI (Yanaka & Mineshima, 2021) , JNLI (Kurihara et al., 2022) , JSeM (Kawazoe et al., 2015) , and JSICK (Yanaka & Mineshima, 2022) , we confirmed score fluctuations due to the class imbalance in multiple models. First, Figure 9 shows the class distributions of both the ground truth and the predictions made by Swallow-7b.Both the ground truth and predictions are greatly imbalanced, with the most frequent class accounting for over 95% of all predictions in three datasets. As a result, scores significantly fluctuate depending on whether the most frequent classes in predictions and ground truth align by chance. Figure 10 shows the learning curve of Swallow-7b on these datasets. While fluctuations of about 40 points occurred in two datasets, this was due to transitions in the most frequent class of predictions. The imbalance in prediction was not unique to Swallow but was also observed in other 7b and 13b models (Figure 11 ). From these observations, we concluded that it is misleading to discuss natural language inference abilities based solely on scores, leading to the exclusion from our evaluation. For a fair and stable evaluation of this task, addressing the imbalance in ground truth classes is recommended. Tables 2 and 3 present the evaluation benchmarks for Japanese and English, respectively.The Japanese benchmarks include llm-jp-eval (Han et al., 2024) for a comprehensive evaluation of language models in Japanese, and the JP Language Model Evaluation Harness, which is designed for a more focused assessment of language model capabilities. The datasets cover a range of tasks including multiple-choice and open-ended question answering (JCQA (Kurihara et al., 2022) , JEMHQA (Ishii et al., 2023) , NIILC (Sekine, 2003) ), machine reading comprehension (JSQuAD (Kurihara et al., 2022) ), automatic summarization (XL-Sum (Hasan et al., 2021) ), arithmetic reasoning (MGSM (Shi et al., 2023) ), and machine translation (WMT'20 En-Ja and Ja-En (Barrault et al., 2020) ). Below, we describe six notable task datasets included in these benchmarks.JCQA JCommonsenseQA (JCQA) is a Japanese question-answering dataset in a five-choice format that assesses common sense knowledge. It is the Japanese version of Common-senseQA (Talmor et al., 2019) , which was created to evaluate common sense reasoning ability, and consists of pairs of question sentence and answer choices. The knowledge covered by JCQA has a good balance of Japanese linguistic knowledge and common sense Table 9 and 10 show the absolute performance scores for Swallow with and without vocabulary expansion (Swallow ¬VE). While Figure 4 shows relative changes, this table provides a comprehensive view of the absolute scores across different evaluation tasks.Regarding Japanese language capabilities, we analyzed the overall impact of vocabulary expansion on performance across various tasks and model sizes. Excluding the XL-Sum task, the only dataset where we observed a relative change of more than 10% is JCQA for the Swallow-7B-VE model. This kind of change was not observed in other model sizes or evaluation tasks. The LLMs were released with a different official name, but we use this code name for the review.Japanese Stable LM Beta: https://ja.stability.ai/blog/japanese-stable-lm-betahttps://dumps.wikimedia.org/other/cirrussearch/20230320: Dump dated March 20, 2023.https://taku910.github.io/mecab/https://clrd.ninjal.ac.jp/unidic/https://github.com/Stability-AI/lm-evaluation-harnessWe assess performance changes on a relative scale, considering the differences in task score levels.This trend remained unchanged after instruction tuning, so it does not seem to be a problem with instruction following ability.https://github.com/NVIDIA/Megatron-LMThe theoretical values (312 TFLOPS) are extracted from the specifications of the NVIDIA A100 BFLOAT16 Tensor Core (without sparsity): https://www.nvidia.com/en-us/data-center/a100/.The calculation of FLOPs is based on the formula presented in APPENDIX: FLOATING-POINT OPERATIONS of(Narayanan et al., 2021), with modifications made to adapt to the Llama architecture. 


Decision: This paper describes an approach to continual pre=train LLaMA for Japanese using 100 billion additional tokens. 

Strengths:
1. a detailed description of the procedure, including vocabulary expansion, continual pre-training step. 
2. Evaluation on a variety of model sizes show its effectiveness. 

Weakness:
1. Limited technical novelty, compared to other work. 
2. Description of important experiment setup is missing. 

Overall, this work made an important contribution despite limited technical novelty. I would be worthwhile to include in COLM.

Reviews: review: paper_summary: paper_summary: The authors present work that extends baseline Llama-2 large language models to include (more) Japanese-language data through continued pre-training on up to 100B additional tokens of Japanese data. By employing model vocabulary expansion in conjunction with the additional data, the authors improve generation efficiency by up to 78% due to the introduction of Japanese-specific vocabulary terms. Experimental results show marked improvements on Japanese performance on six evaluation tasks in addition to improvements in English-Japanese machine translation (MT). Some performance degradation is noted in English-based evaluation tasks as well as Japanese-English MT performance. Lastly, the authors compare their continued pre-training approach against similarly-sized Japanese language models trained from scratch.
strength_weakness: strength_weakness: Language-specific LLM pretraining has been explored in many other works, but the combination of examination of model size and effects of vocabulary expansion may help mitigate this fact.
questions: questions: It would be interesting to see results of other ways of mitigating 'catastrophic forgetting' such as Elastic Weight Consolidation in place of Experience Replay, though the implementation details may be tricky.
paper_summary: paper_summary: This works presents a new version of LLAMA2, adapted for Japanese. First, LLAMA's vocabulary is expanded with a Japanese vocabulary, created from the new training data through MeCab, Unidic and BPE. Then, the model is trained by performing continual pre-training. Finally, the new model is tested on several downstream tasks, achieving satisfying results.
strength_weakness: strength_weakness: As it is right now, the paper relies too much on the appendixes and is very hard to read and understand. Despite its importance, the experimental setup is missing. Similarly, the evaluation lacks context: It is mentioned which downstream task are being used for the evaluation, but there is no description even of which metrics and corpora are being used. Thus, result tables are not informative enough--we don't even know if greater values means better results or if the corpora could have already been seen at training time!

While I find the work interesting, I believe it would benefit from a revision. The main details should be described on the document's body instead of relaying so much on the appendixes.
paper_summary: paper_summary: This paper presents an effort in continuing pre-training an LLM to improve Japanese language capabilities. Authors applied and analysed vocabulary expansion and parallel corpus. Finally llama-2 at 7B, 30B, and 70B are continue-pre-trained with more text in Japanese. Through language-oriented tests, it is found that Japanese test scores mostly increase, but English scores drop. These models achieve better scores in comparison with LLMs trained from scratch in Japanese.
strength_weakness: strength_weakness: - I find this work lacks novelty and depth from a research perspective. Continued pre-training has been done and reported by some researchers; the technique of vocabulary expansion has been widely used. I did do not find many new insights in this paper.
- It is observed that English test scores do drop after continued pre-training, and the paper does not discuss or propose any techniques to mitigate the issues. Is it possible to do continued pre-training with both English and Japanese as opposed to the current setting?
- There are some confusions or concerns about soundness of the arguments or claims, which I detail in the questions section below.
questions: questions: I would like to clarify with the authors some points below which I think can affect the soundness of the work:

1. Abstract: "*The results showed that the efficiency gained through vocabulary expansion had no negative impact on performance*". I wonder what "*efficiency*" means here?
2. Sec 2.2: "*For example, a Japanese single-character noun 猫 (cat) is represented by three byte-level tokens <0xE7> <0x8C> <0xAB>, which do not provide any semantic meaning*"
    - I understand that if a string is not in the original tokenizer, it will be split up in a more fine-grained way.
    - However, I might not agree with the statement because it is discussed from the perspective of human readers.
    - If the three byte-level tokens appear together and often enough during LLM training, they do provide semantic information when being encoded by the model. In other words, I do not see why "猫 (cat)" provides more semantic meaning.
3. Sec 3.3: As a research paper, I'd ask about the reason behind the decision of a Japanese vocabulary of "*(up to 16k)*", and why the new embeddings are "*initialized with the average of the vectors of the subwords segmented by the LLaMA tokenizer*".
4. Sec 5.1.2 and Figure 4: Is "Llama 2-JA without VE" the reference point, and do you have individual absolute scores for it? I also find the statement of "*overall impact of vocabulary expansion on performance is minor.*" to be a bit hand waving because for some tasks and gain/loss are more than 10%, translating to significant absolute differences in test scores.

Comments: title: Acknowledgement of Rebuttal
Initiating the discussion
Reviewer Uybi's response after rebuttal
comment: Thank you for addressing my review comments - as discussed by other reviewers, the paper may benefit from some portions of the appendix moved into the main paper if afforded an extra page upon acceptance.
Thank you for your clarifications. I think that the extra page for the camera-ready will benefit the readability and understanding of the paper.
We sincerely thank the reviewer for the valuable suggestions. We hope our response has thoroughly addressed all concerns and that the reviewer can consider improving their score based on our response. There is nothing we want to add to the response, but we are also happy to discuss with the reviewer if they still have a concern.
>  I find the current justification (relying on semantics) unsupported.

We appreciate the reviewer's insights and understanding of the concerns regarding the use of "morphology" and "phonology" in the context of LLMs where tokens are mapped to indices and vectors. Our justification for stating that byte-level tokens "do not provide any semantic meaning" is based on three primary reasons: semantic integrity, prevention of generating non-existent characters, and improved tokenization efficiency and contextualization.


1. Semantic Integrity:
Byte-level tokens can result in shared tokens for characters with vastly different meanings. For instance, "猟" (hunt) is encoded as <0xE7> <0x8C> <0x9F>, and "猫" (cat) is encoded as <0xE7> <0x8C> <0xAB>, sharing with the identical first two bytes despite their different meanings. This overlap can dilute the semantic integrity of Japanese kanji letters, which are ideograms unlike most of other languages. It is possible to argue that English also suffers from the same problem, e.g., “cat” and “car” are totally different, but the treatment of English tokenization is much better than that of Japanese because the tokenizer of the base model was trained on a corpus with a lot of English text and little Japanese text. In other words, it is unlikely for the base tokenizer to split them into “cat” and “car” into <c> <a> <t> and <c> <a> <r>. While LLMs derive semantics from context, starting with more semantic Japanese tokens is reasonable because Japanese kanji letter are ideograms.


2. Prevention of Generating Non-existent Characters:
Byte-level tokenization can produce byte sequences that do not correspond to any valid characters in the target language. For instance, [it is known](https://okumuralab.org/~okumura/misc/230611.html) in Japan that ChatGPT generated the non-existent Japanese word "視覴," allegedly caused by the impossible combination of byte tokens from "覚(perception)" and "聴(listen)." Such occurrences are less likely if we treat a Japanese kanji as a single token, which respects the boundaries of valid characters and words. This reduces the possibility of generating meaningless or incorrect text and thereby improves the reliability and quality of the model's output.


3. Improved Tokenization Efficiency and Contextualization:
Representing a kanji letter as a single token provides a more consistent and semantically relevant input to the token embeddings, enhancing the model's ability to understand and generate coherent text. For example, "猫"(cat) = <0xE7> <0x8C> <0xAB> shares the first two tokens with 17 other kanji in byte-level tokenization, which potentially increases the semantic ambiguity and degrades its performance. By adding kanji letters to the vocabulary, we can ensure that each token has a clear and distinct meaning, which improves the model's ability to contextualize and generate accurate representations.


>  I feel that the statement "impact of vocabulary expansion on performance is minor" is not accurate.

Thank you for your feedback. we would like to clarify our position regarding the statement, "the impact of vocabulary expansion on performance is minor."

Excluding the XL-Sum task, the only dataset where we observed a relative change of more than 10% is JCQA for the Llama-2-JA-7b model. This kind of change was not observed in other model sizes or tasks, as we noted in §5.1.2.

To provide more evidence, we present an additional analysis of the JCQA scores, isolated from the averaged numbers reported in Figure 7. The scores for 7b models’ JCQA with and without vocabulary expansion (VE and $\neg$VE, respectively) are as follows:
|Num. of training tokens (B)|JCQA(VE)|JCQA($\neg$VE)|
|---|---|---|
|0|38.52|38.52|
|20|39.23|45.76|
|40|45.76|39.14|
|60|42.18|49.15|
|80|49.15|52.28|
|100|48.08|54.33|
With 40B tokens used for training, the model with vocabulary expansion (VE) outperforms the one without it ($\neg$VE). This indicates that the JCQA scores are not stable. In contrast, Figure 6 shows that the XL-Sum scores consistently show better performance for the model without vocabulary expansion, exhibiting a different trend.

Furthermore, the average and standard deviation of JCQA scores after training 40B tokens (40,60,80,100 B Tokens) are:

- Vocabulary Expansion (VE): 46.29 ± 3.09
- No Vocabulary Expansion ($\neg$VE): 48.73 ± 6.74

The observed 6.25-point (11.5%) difference of the JCQA scores between the vocabulary expansion (VE) and non-vocabulary expansion ($\neg$VE) of the Llama-2-JA-7b models does not indicate a substantial impact because of the instability and large standard deviation on the dataset.

Therefore, considering these evaluation results and the observed instability, we regard that the impact of vocabulary expansion on overall performance is minor. 

In the camera-ready version, we will include these elaborations in the Appendix.
Thank you to the authors for the explanations and clarifications. I re-read the paper while referencing to your rebuttal and related works. I feel that I have a better understanding of the contribution of this paper now. While most points are understood, I feel that I still have reserved opinions about the two statement below.

**1.** Re: intact token v. byte-level tokens: whether the former provides more semantics when fed to an LLM. 

The original statement and the explanation are both centred around human interpretation of words/characters. After rebuttal, I feel unsure about whether/how "morphology" or "phonology" are used by an LLM if the tokens are mapped to indices and then vectors. LLM's embeddings (so-called semantics) are derived by contextualizing on the input itself and surrounding tokens. Note that I completely agree with the choice of expanding the vocabulary for extra language support. It improves *tokenizer fertility*. I find the current justification (relying on semantics) unsupported.

**2.** I feel that the statement "impact of vocabulary expansion on performance is minor" is not accurate.

I would say that if some changes are noticeably positive while some are negative (10% relative change is a lot), it actually means that the impact on test performance is significant rather than "minor"?