Enhancing Translation Accuracy of Large Language Models through Continual Pre-Training on Parallel Data (2024)

Abstract: In this paper, we propose a two-phase training approach where pre-trained large language models are continually pre-trained on parallel data and then supervised fine-tuned with a small amount of high-quality parallel data. To investigate the effectiveness of our proposed approach, we conducted continual pre-training with a 3.8B-parameter model and parallel data across eight different formats. We evaluate these methods on thirteen test sets for Japaneseto-English and English-to-Japanese translation. The results demonstrate that when utilizing parallel data in continual pre-training, it is essential to alternate between source and target sentences. Additionally, we demonstrated that the translation accuracy improves only for translation directions where the order of source and target sentences aligns between continual pretraining data and inference. In addition, we demonstrate that the LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoderdecoder models. We also show that the highest accuracy is achieved when the data for continual pre-training consists of interleaved source and target sentences and when tags are added to the source sentences.

heading: 1 Introduction
text: In machine translation, transformer encoderdecoder models (Vaswani et al., 2017), such as NLLB-200 (NLLB Team et al., 2022), mT5 (Xue et al., 2021), and mBART (Liu et al., 2020) predominate. The emergence of pre-trained Large Language Models (LLMs) composed solely of the transformer decoder, such as GPT series (Brown et al., 2020; OpenAI, 2023), has prompted the development of pre-trained LLMs, including, PaLM (Chowdhery et al., 2022), and LLaMA (Touvron et al., 2023). When translating with these LLMs, it is common to use in-context few-shot
learning. According to Hendy et al. (2023), GPT-3 demonstrates comparable or superior accuracy to WMT-best for high-resource languages. Furthermore, as reported by Kocmi et al. (2023), GPT4’s 5-shot surpasses WMT-best’s accuracy in most translation directions. However, Zhu et al. (2024) noted that in 8-shot scenarios, relatively small-scale LLMs (e.g., 7B parameters) exhibit lower accuracy than supervised encoder-decoder models. Therefore, it is necessary to investigate methods capable of achieving translation accuracy equivalent to existing translation models with relatively small-scale LLMs.
On the other hand, in models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019), which consist solely of transformer encoders, the effectiveness of continual pre-training, where pretrained models are further trained on task-specific data such as classification to improve the accuracy of the task, has been reported (Jin et al., 2022; Ke et al., 2022). In the context of LLMs, continual pre-training has been reported to transfer models primarily pre-trained in English, such as LLaMA, to other languages (Cui et al., 2023). Additionally, when building LLM-based translation models, the effectiveness of conducting continual pre-training with either monolingual data, parallel data, or both, followed by supervised fine-tuning, has been reported, mainly when basing the model on primarily English pre-trained models such as LLaMA-2 (Xu et al., 2024a; Alves et al., 2024; Guo et al., 2024).
Although those recent publicaion in the context of LLMs are closely related to our study, this paper presents research conducted independently of those latest LLM-based translation studies such as Xu et al. (2024a); Alves et al. (2024); Guo et al. (2024). This paper proposes a two-phase training approach: continual pre-training on parallel data crawled from the web and supervised fine-tuning using a small amount of high-quality parallel data created by professional translators. To comprehen-
251
sively investigate methods for improving translation accuracy through continual pre-training, we conduct continual pre-training across eight data formats for Japanese-to-English (Ja ⇒ En) and English-to-Japanese (En ⇒ Ja) translations using a 3.8B-parameter LLM. We evaluate the translation accuracy on 13 test sets. Our paper’s novelty compared to Xu et al. (2024a); Alves et al. (2024); Guo et al. (2024) lies in the following aspects.
• When conducting continual pre-training on data where source and target sentences appear alternately, the direction of language in which accuracy improves varies depending on the order of source and target sentences.
• LLM-based translation model is more robust in translating spoken language and achieves higher accuracy with less training data compared to supervised encoder-decoder models.
• When indicating the translation direction with tags (“<2en>” etc.) on data for continual pre-training, higher accuracy is achieved compared to simply concatenating source and target sentences.

heading: 2 Related Work
text: Parallel Data in Pre-Training from Scratch When pre-training LLMs from scratch, it is expected to use monolingual data. However, some reports incorporating bilingual data, such as parallel data, into the pre-training dataset can enhance the accuracy of downstream tasks. Briakou et al. (2023) show that incorporating parallel data into pre-training 1B and 8B parameter LLMs enhances translation accuracy in zero- and five-shot. Separate studies further show that including parallel data in the pre-training of the encoder-decoder model also improved performance in downstream multilingual and cross-lingual tasks (Kale et al., 2021; Schioppa et al., 2023).
LLMs-Based Translation Models Zhang et al. (2023) demonstrated that fine-tuning 15 multilingual LLMs using QLoRA for French-to-English translation surpasses the accuracy of both incontext few-shot learning and models trained from scratch. Conversely, Xu et al. (2024a) demonstrated that models predominantly pre-trained on English data, such as LLaMA-2, suffer reduced translation accuracy when translating into nonEnglish target languages. Addressing this issue,
they introduced ALMA, a method that employs fine-tuning monolingual data in the first stage, followed by supervised fine-tuning with a small quantity of high-quality parallel data in the second stage. Furthermore, there exists a report on improving translation accuracy by employing Contrastive Preference Optimization (CPO) for the second stage of supervised fine-tuning in ALMA (Xu et al., 2024b). In addition, the effectiveness of utilizing monolingual and parallel data in the first stage has been reported (Alves et al., 2024; Guo et al., 2024).
LLM-based translation models have only been evaluated on test data from the WMT General Machine Translation Task (Kocmi et al., 2022, 2023) and Flores-200 (NLLB Team et al., 2022). Therefore, their effectiveness compared to conventional supervised encoder-decoder models has not been sufficiently validated across various types of data. Additionally, the impact of continual pre-training data on translation accuracy remains unclear. Our study aims to address these two points.

heading: 3 Continual Pre-Training and Supervised Fine-Tuning with Parallel Data
text: We introduce a two-phase training to enhance the accuracy of translation of LLMs. In the first phase, we perform continual pre-training using parallel data crawled from the web, such as ParaCrawl (Bañón et al., 2020). Then, in the second phase, we conduct supervised fine-tuning with a small amount of high-quality parallel data. In LLM fine-tuning, the importance of data quality has been reported (Xu et al., 2024a; Zhou et al., 2023). However, it has also been reported that parallel data crawled from the web may have low data quality (Thompson et al., 2024). Therefore, we used data created by professional translators as high-quality parallel data.

heading: 3.1 Continual Pre-Training
text: Continual pre-training involves training on data where the source and target sentences appear alternately. Let the source sentences be denoted as {x1, . . . , xn} and the target sentences as {y1, . . . , yn}, creating a dataset {x1, y1, . . . , xn, yn}. With the tokens of the created dataset represented as z = {z1, z2, . . . , zm}, we train the model parameters θ to minimize the following loss:
L1(θ) = − ∑
t
logP (zt|zt−c, . . . , zt−1; θ) (1)
where c is the number of context lengths representing the maximum input length of LLMs. L1(θ) is a standard causal language modeling loss, which predicts the next word based on previous words (Radford et al., 2018). Therefore, we train by extracting (zt−c, . . . , zt−1) from z in increments of c tokens, such that the source and target sentences alternate to predict the next word for each token. Extracting c tokens may result in the input’s start and end being in the middle of the source or target sentence.
In pre-trained models such as LLaMA-2, primarily pre-trained in English, it has been reported that the effectiveness of utilizing monolingual data in continual pre-training, in addition to parallel data, is significant (Guo et al., 2024; Alves et al., 2024). The rationale behind continual pre-training with monolingual data is to acquire the generative ability in languages other than English. Therefore, in models where pre-training with monolingual data has been sufficiently conducted from scratch or where continual pre-training with monolingual data has been conducted, it is optional to conduct continual pre-training with monolingual data.

heading: 3.2 Supervised Fine-Tuning
text: After continual pre-training, we perform supervised fine-tuning with a small amount of high-quality parallel data. Let the source sentence be denoted by x, the target sentence corresponding to x by y, and the prompt by I(x). We train the model parameters to minimize the following loss:
L2(θ) = − T∑
t=1
logP (yt|y<t, I (x) ; θ) (2)
where T represents the number of tokens in the target sentence, and yt is the t-th token of the target sentence. While L2(θ) is also standard causal language modeling loss, it computes the loss only for the output of the target sentence (Xu et al., 2024a; Zhang et al., 2024). Therefore, we combine the prompt and target sentence (e.g., Translate “Good morning” into Japanese: おはよう) and input it into the model. The model predicts the next word for all input words, including the prompt portion. However, this portion is not used during inference and hence excluded from the loss.

heading: 4 Experiments
text: We conduct experiments on two NVIDIA RTX A6000 GPUs. Due to severely limited com-
putational resources, we use a 3.8B parameters LLM, rinna/bilingual-gpt-neox-4b (rinna4b)1, which is already pre-trained on Japanese and English data, totaling 524B tokens, with 173B tokens in Japanese and 293B tokens in English. Since rinna-4b has undergone sufficient pre-training from scratch on monolingual data for both Japanese and English, as stated in Section 3, we believe that continual pre-training with monolingual data is unnecessary. Given that the model we employ is pretrained on Japanese and English, we experiment with Japanese-to-English and English-to-Japanese translation tasks. All experiments utilizing rinna-4b are conducted using the open-source huggingface transformers library.2

heading: 4.1 Dataset

heading: 4.1.1 Continual Pre-Training
text: We utilize JParaCrawl v3.0 (Morishita et al., 2022) as the web-based parallel data comprising 21.8M parallel sentence pairs, the largest and newest dataset of English-Japanese parallel data available. From this dataset of 21.8M parallel sentence pairs, we sample 20.8M sentence pairs using LEALLAlarge3 (Mao and Nakagawa, 2023) for train data. Details on sampling are provided in Appendix A. For dev data, we use the dev and test data from WMT20 (Barrault et al., 2020) and the test data from WMT21 (Akhbardeh et al., 2021).

heading: 4.1.2 Supervised Fine-Tuning
text: We utilize the dev and test data of WMT20 and Flores-200 (NLLB Team et al., 2022), along with the train data from KFTT (Neubig, 2011) as train data, all created by professional translators. The train data for KFTT utilized in experiments consists of 10k instances randomly sampled from 440k samples. The resulting train data comprise 15k samples for both En ⇒ Ja and Ja ⇒ En. For dev data, we utilize the WMT21 test data. We use the prompts written in the following source language, based on the report by Xu et al. (2024a).4
En ⇒ Ja Translate this from English to Japanese: English: {source sentence} Japanese:
1https://huggingface.co/rinna/ bilingual-gpt-neox-4b
2https://github.com/huggingface/transformers 3https://huggingface.co/setu4993/LEALLA-large 4The reason for writing prompts in the source sentence’s language is that it is more natural to create translation prompts in the source sentence’s language when translating.
Ja ⇒ En これを日本語から英語に翻訳してください : 日本語 : {source sentence} 英語 :

heading: 4.1.3 Test Sets
text: We use the test sets employed by Morishita et al. (2022) to evaluate translation accuracy. Since we include the test data from WMT20 and WMT21 in the train and dev data for continual pre-training and supervised fine-tuning, we exclude these and add the test data from WMT22. As a result, there are 13 test sets: 5 for the En ⇒ Ja direction, 3 for the Ja ⇒ En direction, and 5 for both the En ⇒ Ja and En ⇒ Ja directions. For detailed information on the test sets, please refer to Table 6 of Appendix D.

heading: 4.2 Models

heading: 4.2.1 Baseline Models
text: We establish two baseline models as described below. The train data consists of the data described in Section 4.1.1 and Section 4.1.2, and the dev data is the WMT21 test data. Note that the data from JParaCrawl v3.0 is created by randomly sampling 10.4M parallel sentences, which is 50% of the total 20.8M parallel sentences, to be used respectively as the train data for En ⇒ Ja and Ja ⇒ En.
Transformer This model is a 1B-parameter transformer trained from scratch. The model architecture is based on mT5-large5, with two modifications: reducing the vocab_size from 250,112 to 65,536, matching that of rinna-4b, and increasing the feed-forward network dimension from 2,816 to 4,096. As a result, the model has 24 layers each for the encoder and decoder, a model dimension of 1,024, 16 attention heads, a feed-forward network with GeGLU activation (Shazeer, 2020), and a dropout (Srivastava et al., 2014) of 0.1. The tokenizer is newly created using the sentencepiece library6 (Kudo and Richardson, 2018) with the subword method set to unigram, character coverage to 0.9995, and byte-fallback enabled. Training is conducted with a total batch size of 4,096 for 15 epochs (38,160 steps), with validation every 1,000 steps, and it is terminated if the validation loss does not improve for three consecutive validations. We use AdamW optimizer (Loshchilov and Hutter, 2019), with β1 = 0.9, β2 = 0.98, ϵ = 1.0 × 10−8. We set the weight decay and label smoothing (Szegedy
5https://huggingface.co/google/mt5-large 6https://github.com/google/sentencepiece
et al., 2016) to 0.1, and gradient clipping (Pascanu et al., 2013) to 1.0. The peak learning rate is set to 1.0 × 10−3, with a warmup ratio 0.1 and an inverse square root scheduler applied. Additionally, bfloat16, gradient checkpointing (Chen et al., 2016), and the deepspeed7 (Rasley et al., 2020) ZeRO stage 2 are applied during training. Training is conducted on two NVIDIA RTX A6000 GPUs with these settings, taking 17 days.
Direct-SFT Direct-SFT consists of the rinna-4b directly supervised fine-tuning with parallel data, using LoRA tuning (Hu et al., 2022). We conduct supervised fine-tuning of this model using the prompts mentioned in Section 4.1.2. Furthermore, to approximate conditions for full-weight tuning, we apply LoRA to the linear layers of selfattention’s query, key, value, and output, as well as the linear layers of the feed-forward network. We set the rank of LoRA to 16, resulting in 25.9M trainable parameters, which constitutes 0.68% of the parameters in rinna-4b.

heading: 4.2.2 Source and Target Sentences Ordering in Continual Pre-Training
text: We conduct continual pre-training with 4 patterns, varying the order in which source and target sentences. After continual pre-training with these 4 orders, we undergo supervised fine-tuning using the data and prompts described in Section 4.1.2 with full fine-tuning and LoRA tuning.
Mono As stated in Section 3.1, instead of alternating between source and target sentences, the approach involves sequences such as (x1, . . . , xn), (y1, . . . , yn), where only the source or target sentences appear consecutively. Therefore, either Japanese-only or English-only sentences appear consecutively.
En-Ja Concatenating a Japanese translation immediately after each English sentence, making it parallel data only in the En ⇒ Ja.
Ja-En Concatenating an English translation immediately after each Japanese sentence, making it parallel data only in the Ja ⇒ En.
Mix Randomly sampling 10.4M, which is 50% from the total of 20.8M, from the En-Ja and Ja-En without duplication.
7https://github.com/microsoft/DeepSpeed
green .

heading: 4.3 Hyperparameters

heading: 4.3.1 Continual Pre-Training
text: We use the AdamW optimizer, with β1 = 0.9, β2 = 0.95, ϵ = 1.0×10−8. The context length is 2048, the same as when pre-training rinna-4b from scratch, and training is conducted for 1 epoch. We perform validation every 100 training steps. We use a cosine learning rate schedule with a warmup ratio of 1% and a peak learning rate of 1.5× 10−4. We use a weight decay of 0.1 and gradient clipping of 1.0. We utilize two NVIDIA RTX A6000 GPUs, processing 1 batch on each GPU with a gradient accumulation step of 128, achieving an adequate batch size 256. During training, bfloat16 precision, gradient checkpointing, and deepspeed ZeRO stage 2 are employed. With these configurations, it takes 10 days.

heading: 4.3.2 Supervised Fine-tuning
text: We perform supervised fine-tuning on the model that achieves the minimum validation loss in continual pre-training. We change the AdamW optimizer’s parameter used in Section 4.3.1 only β2 = 0.95 to β2 = 0.999. Weight decay and gradient clipping are the same as Section 4.3.1. The peak
learning rate is set to 3.0×10−5 for full fine-tuning and 2.0× 10−4 for LoRA tuning, with a warmup ratio of 1% using an inverse square schedule. For LoRA, we set r = 16, α = 32, and dropout to 0.05, applying to the linear layers of query, key, and value in the multi-head attention, resulting in approximately 6.4M trainable parameters corresponding to 0.17% of the rinna-4b’s parameters. We conduct validation every 10% of the total training steps for Direct-SFT only, with 1 epoch and a batch size of 256. For all other cases, validation is performed every 100 training steps, with 5 epoch and the batch size of 64.

heading: 4.3.3 Inference
text: All models use the one with the minimum validation loss for inference, applying bfloat16. The Transformer, which has fewer parameters than the rinna-4b, employs beam search with a beam size of 4 due to its smaller number of parameters. At the same time, the rinna-4b-based models use greedy decoding with the prompt described in Section ?? for inference.

heading: 4.4 Metrics
text: We use BLEU8 (Papineni et al., 2002) and COMET9 (Rei et al., 2022) as evaluation metrics. We use Unbabel/wmt22-comet-da as COMET model.

heading: 5 Results

heading: 5.1 The Impact of Source and Target Sentences Order
text: Table 1 presents the results of baseline models compared to models continually pre-trained with three orders described in Section 4.2.2 and then supervised fine-tuning. All results of Table 1 can be found in Table 7 and Table 8 of Appendix D. DirectSFT, which is directly fine-tuned, and Mono, which was pre-trained with parallel data treated as monolingual data, exhibit lower accuracy than the Transformer. On the other hand, continual pre-training improves accuracy only in the translation direction aligned with the parallel data. Therefore, continual pre-training with data where source and target sentences appear alternately is necessary to achieve high accuracy. Despite data in both the En ⇒ Ja and Ja ⇒ En translation directions, Mix exhibits improved accuracy, even though the input data’s translation direction is inconsistent. This result suggests that LLMs can leverage the knowledge of the translation direction matching the order of the source and target sentences, and they can utilize the knowledge acquired from parallel sentences mixed in the training data.
8https://github.com/mjpost/sacrebleu 9https://github.com/Unbabel/COMET

heading: 5.2 Accuracy Comparison Across Test Sets
text: Figure 1 shows a radar chart of the COMET score of a model in which the Transformer and rinna-4b are continually pre-trained as a Mix, followed by supervised fine-tuning with full weight. In particular, the LLM-based translation model significantly outperforms the Transformer on the WMT19, 20 Robustness Task for the Reddit domain, and on the TED (tst2015), IWSLT21 En-Ja Dev, and JESC for the TED Talk and movie subtitles domains. This result suggests that the LLM-based translation model is more robust than the traditional encoder-decoder model regarding data containing spoken language.

heading: 6 Discussion

heading: 6.1 Data Format in Continual Pre-Training
text: Mixing data from two translation directions, as in the case of Mix, improves accuracy for both translation directions, allowing one model to be used for both. However, the accuracy is lower than continual pre-training with data from only one translation direction. Therefore, we investigate methods to enhance translation accuracy by explicitly indicating the translation direction for the data used in continual pre-training. Drawing inspiration from studies incorporating parallel data during pre-training from scratch, we conduct experiments on the following four formats.
Interleaved Translations This format directly concatenates the source and target sentences (Briakou et al., 2023), identical to the Mix described in Section 4.2.2.
Prefixed This format involves inserting the prefix written in the source sentence’s language before the
source sentence, followed by the concatenation of the target sentence (Kale et al., 2021). For En ⇒ Ja, the prefix “translate to Japanese: ” is used, while for Ja ⇒ En, “英語に翻訳してください: ” is employed.
Tagged This format involves inserting a tag before the source sentence that indicates the target sentence’s language, such as “<2en>” and “<2ja>” (Schioppa et al., 2023).
JSON The JSON format is {“L1”: {source}, “L2”: {target}}, where “source” represents the source sentence, “target” represents the target sentence, and “L1”, “L2” are the names of the source and target sentence’s languages written in the
source sentence’s language.10
We conduct continual pre-training with these four formats and perform supervised fine-tuning under the same conditions as Section 4. All formats are conducted in the Mix format described in Section 4.2.2, with the continual pre-training data for En ⇒ Ja and Ja ⇒ En fixed to be the same. Table 2 presents the results of the Transformer and models continually pre-trained with the four formats. Among the four formats, Prefix and Tagged showed significant differences in BLEU and COMET metrics compared to the Transformer, and the models are continually pre-trained in the in-
10Given that the pre-training data for rinna-4b includes source code, this format aims to transfer the knowledge obtained from the source code to translation.
terleaved translations format. This result suggests that the prefixed or tagged format demonstrates higher accuracy than interleaved translation, where source and target sentences are concatenated, indicating that from the convenience perspective, the Tagged format can achieve the highest accuracy most easily. Whether these formats can be applied to other translation directions and models remains a matter of our future work.

heading: 6.2 Effectiveness of Continual Pre-Training and Supervised Fine-Tuning
text: As an ablation study, we experiment with all combinations of continual pre-training and supervised fine-tuning. When supervised fine-tuning is conducted, the inference is made with a zero-shot, while for other cases, the inference is performed with a five-shot. We randomly sample five translation examples from the WMT21 test data for fiveshot, and the same set of five samples is fixed for all inferences. Continual pre-training is conducted in the Tagged format as described in Section 6.1, and all inferences utilize the prompts described in Section 4.1.2 and employ bfloat16 precision and greedy decoding. Table 3 presents the results, while all results are shown in Table 10 of Appendix D. These results suggest that achieving high accuracy is most feasible when both continual pre-training and supervised fine-tuning are conducted while achieving high accuracy solely through continual pre-training or supervised fine-tuning alone is chal-
lenging.

heading: 6.3 How Much Parallel Data is Needed?
text: Figure 2 presents the data curves for these three models at 10%, 20%, 30%, 50%, and 100% data usage on the WMT22 test data. For the Transformer model, only the sampling rate from JParaCrawl v3.0 varies, while other settings remain the same as described in Section 4.2.1. As mentioned in Section 4.3, Direct-SFT performs supervised finetuning for one epoch, and Mix also performs continual pre-training for one epoch. Therefore, for these two models, the proportion of training data is equivalent to the proportion of checkpoints, and we report the accuracy for the checkpoints at 10%, 20%, 30%, 50%, and 100%. The Transformer shows very low accuracy, up to 10% and 20%, but there is a significant improvement in accuracy at 30%, after which the increase becomes gradual. When at 20%, the accuracy decreased compared to at 10%, possibly due to the instability in learning caused by the smaller data. On the other hand, Direct-SFT and Mix demonstrate significantly better accuracy at 10% and 20% compared to the Transformer, and like the Transformer, the accuracy increases gradually from 30% onwards. These results suggest that LLM-based translation models can achieve higher accuracy with less training data than supervised encoder-decoder models. Additionally, COMET scores for all three models show a gradual increase in accuracy from 30%, while BLEU scores con-
tinue to improve even after 30%. This suggests that at least 3M sentence pairs are needed for the translation model to output sentences containing the same meaning as the reference, whereas more parallel data than the 10.4M sentence pairs is required to output sentences containing the exact words as the reference.

heading: 6.4 Specific Results of Spoken Language
text: To analyze the differences in translation between the LLM-based model and the encoder-decoder model for spoken language, Table 4 presents En ⇒ Ja translation examples from two test sets comprising TED Talks domain. In these two examples, the LLM-based translation model has achieved higher COMET scores than the Transformer. In Table 4a, the source sentence contains the phrase "a thing," which the reference translates as "社会現象" (a social phenomenon). In contrast, the Transformer translates "a thing" literally as "もの", and the LLM-based model translates it as "物議をかもすもの" (a controversial thing). Additionally, in Table 4b, the source sentence includes the word "dive," which the reference translates as "始めましょう" (let us start). The Transformer translates "dive" literally as "飛び込む" (jump into), whereas the LLM-based model correctly translates it as "始めよう" (let us start). These results suggest that the LLM-based translation model can perform free translation better than the traditional encoderdecoder model.

heading: 7 Conclusion
text: We propose a two-phase training approach comprising continual pre-training with interleaved source and target sentence data, followed by supervised fine-tuning using a small amount of high-quality parallel data. Our investigation comprehensively explores methods for enhancing translation accuracy through continual pre-training across eight data formats. Evaluation across 13 test sets reveals that models trained with continual pre-training followed by supervised fine-tuning outperform those supervised fine-tuned solely on parallel data. Furthermore, we observe variations in language direction accuracy improvement during continual pretraining based on the order of source and target sentences. We also demonstrate that LLM-based translation models are more robust in translating sentences containing spoken language, and achieve higher accuracy with less training data, compared to traditional encoder-decoder models. Additionally, augmenting source sentences with tags or using prefixes yields higher accuracy than simple concatenation. In larger LLMs than the rinna-4b model we utilized, such as LLaMA-2 7B and 13B, LoRA enables training with fewer computational resources. LoRA has been reported to be effective in translation tasks (Zhang et al., 2023; Guo et al., 2024). Therefore, it is essential to experiment with LoRA in the future to determine if similar results can be achieved and to investigate if similar results can be obtained with other LLMs.

heading: 8 Limitations
text: Our experiments and conclusions are based only on two translation directions (Englishto-Japanese, Japanese-to-English) and rinna/bilingual-gpt-neox-4b, which is an LLM pre-trained in English and Japanese. Evaluation for other translation directions and LLMs has yet to be conducted. While in Section 6.3, we demonstrated that continual pre-training requires 3M parallel data, we anticipate that this may vary depending on the translation direction and model. Whether our approach applies to LLMs primarily pre-trained in English, such as XGLM and LLaMA, remains unverified, especially in low resource languages is challenging. Additionally, all the experiments are conducted using only the parameters described in Section 4.3, and an optimal hyperparameter search still needs to be performed. Especially in Direct-SFT, it should be noted that the importance of hyperparameters has been highlighted by Dettmers et al. (2023), and whether full fine-tuning and LoRA tuning demonstrate the same performance varies depending on the model, hyperparameters, and task.

heading: 9 Ethical statement
text: We have not conducted verification on significant risks associated with our research. While we propose a method that may enhance translation accuracy using LLMs, it is worth noting that Zhu et al. (2024) have reported GPT-4’s 8-shot translation accuracy to be comparable to or below that of existing methods such as supervised encoder-decoder models. Therefore, even if the proposed method is applied to other LLMs, we do not think that there is a potential risk that the proposed method achieves too high translation accuracy so that it is to be abused.
This study uses a dataset from Morishita et al. (2022), available only for research and development purposes, inheriting potential biases from their datasets. We utilize open-source pre-trained LLM, and our experimental codes also leverage open-source libraries, as mentioned in Section 4. Therefore, this study’s models, data, and tools adhere to the intended usages of those models, data, and tools.

heading: A Sampling of JParaCrawl v3.0
text: We use 20.8M parallel sentences from JParaCrawl v3.0, initially consisting of 21.8M parallel sentences. We sample sentence pairs using cosine similarity scores between 0.4 and 0.95 based on sentence vector embeddings obtained from LEALLAlarge. Parallel sentences with a similarity score below 0.4 are excluded, as a visual inspection revealed a significant presence of inappropriate samples, such as Japanese and English sentences with disproportionate lengths. Additionally, parallel sentences with similarity scores of 0.95 or higher are also excluded, as they consist of Japanese and English sentences that were nearly identical. This sampling results in 1.8B tokens when tokenized with the rinna-4b tokenizer.

heading: B Comparison with ALMA
text: We compared with ALMA by using ALMA-7BJa-V211, which was trained similarly to ALMA with LLaMA-2 7B but with Russian replaced by Japanese among the languages experimented with ALMA. We compared against ALMA-Ja-V2 using the BLEU and COMET averages of 12 test sets, as shown in the Table 5. From these results, it is evident that the 3.8B LLM-based translation model outperforms the LLaMA-2-based ALMA7B-Ja-V2. This result aligns with reports suggesting higher accuracy when using parallel data for continual pre-training (Alves et al., 2024; Guo et al., 2024) and the consistency with reports indicating that the influence of parallel data increases with fewer parameters (Kale et al., 2021; Briakou et al., 2023).

heading: C Analyzing Catastrophic Forgetting
text: We conducted continual pre-training on En-Ja and then observed catastrophic forgetting by conducting continual pre-training on data, which was in the reverse direction. Based on reports suggesting preventing catastrophic forgetting by mixing data from tasks that should not be forgotten (Scialom et al., 2022), we mixed 1% of En-Ja data. This model is named En-Ja2Mix. Figure 3 shows the data curves for BLEU and COMET scores at each 10% checkpoint for En-Ja2Mix on the WMT22 test data, demonstrating En ⇒ Ja. As an ablation study, we also show the data curves for a scenario where the 1% of En-Ja data added to En-Ja2Mix is removed, and continual pre-training is conducted entirely with Ja-En data. From these data curves, it is observed that when conducting continual pretraining with En-Ja data and subsequently with data in the reverse direction, mixing 1% of the first continual pre-training data can mitigate the degradation in accuracy for En ⇒ Ja. Therefore, this suggests that in the continual pre-training of LLMs, incorporating a small proportion of data that one does not wish to be forgotten can suppress catastrophic forgetting.
D Detailed Tables
11https://huggingface.co/webbigdata/ ALMA-7B-Ja-V2
scores in each line, and scores that surpass the Transformer are emphasized in green . “*” indicates significant differences compared to Transformer,“ # Sig.” indicates the number of test sets showing significant differences from Transformer (p < 0.05).

heading: Future Work
text: Therefore, it is essential to experiment with LoRA in the future to determine if similar results can be achieved and to investigate if similar results can be obtained with other LLMs