LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data Augmentation (2024)

Abstract: Adapting English-based large language models (LLMs) to other languages has become increasingly popular due to the efficiency and potential of cross-lingual transfer. However, existing language adaptation methods often overlook the benefits of cross-lingual supervision. In this study, we introduce LEIA, a language adaptation tuning method that utilizes Wikipedia entity names aligned across languages. This method involves augmenting the target language corpus with English entity names and training the model using left-to-right language modeling. We assess LEIA on diverse question answering datasets using 7B-parameter LLMs, demonstrating significant performance gains across various non-English languages.1

heading: 1 Introduction
text: While large language models (LLMs) are emerging as foundational technology (Brown et al., 2020), their data hungriness restricts their application to a few resource-rich languages, with English being the most dominant among them (Joshi et al., 2020). A promising strategy to broaden their scope is language adaptation tuning (Müller and Laurent, 2022; Yong et al., 2023), where an already-pretrained LLM is further trained on a corpus of a language of interest. The underlying motivation is that the model can leverage the knowledge acquired during pretraining to the target language.
However, this typical approach overlooks the potential benefits of incorporating cross-lingual supervision. Although language models can learn cross-lingual knowledge from a mix of monolingual corpora (Conneau et al., 2020), knowledge sharing between languages is limited, and significant performance gaps still exist between English and non-English languages (Ahuja et al., 2023; Etxaniz et al., 2023; Huang et al., 2023).
1The source code is available at https://github. com/leia-llm/leia.
In this work, we propose a language adaptation tuning method, LEIA (Lightweight Entity-based Inter-language Adaptation), that explicitly exploits cross-lingual supervision. We focus on Wikipedia as a source of target language corpus, as it offers high-quality text data in a wide range of languages and the text contains hyperlinks to entities (i.e., Wikipedia articles) that are aligned across different languages via inter-language links. In our tuning phase, we insert an English entity name beside the corresponding entity in the text (see Figure 1), and train the model using the left-to-right language modeling objective. This simple modification enables the model to extract and apply its English knowledge about the entities within the target language text during training, which we hypothesize to facilitate cross-lingual knowledge transfer.
We assess the effectiveness of LEIA through experiments using 7B-parameter LLMs, LLaMA 2 (Touvron et al., 2023) and Swallow (Fujii et al., 2024), and a diverse set of question answering datasets. The results demonstrate that through our fine-tuning, LLMs benefit from knowledge transfer from English and significantly outperform the base models and those fine-tuned without LEIA.
7029

heading: 2 Method
text: Our method involves fine-tuning on a pretrained LLM using an augmented corpus derived from the target language edition of Wikipedia. Specifically, for each hyperlink in the Wikipedia corpus, we insert the English name of the referred entity next to the hyperlink (Figure 1). The English name is enclosed within special <translate> and </translate> tokens, allowing the model to identify its boundaries. The English name of an entity is extracted from the title of the corresponding English Wikipedia page, which is identified using the interlanguage links. We ignore any hyperlinks pointing to entities not present in the English Wikipedia.2 Further details are available in Appendix A. We fine-tune a pretrained LLM using the language modeling objective. We train the model on the corpus of a target language and evaluate it using datasets in the same language. The aforementioned special tokens are added to the vocabulary. To prevent the model from generating these special tokens during inference, we block loss propagation when predicting these tokens during training.

heading: 3 Experiments with LLaMA 2
text: We start with experiments using the LLaMA 2 7B model (Touvron et al., 2023). Since it is primarily trained for English, it possesses a substantial amount of English knowledge that could be transferred to other languages. Furthermore, its training corpus, containing approximately 38B non-English language tokens (Touvron et al., 2023), fosters competitive multilingual performance (Etxaniz et al., 2023). This makes LLaMA 2 a good candidate for investigating the effectiveness of our language adaptation method from English to other languages.

heading: 3.1 Setup
text: Training We conduct experiments across seven languages: Arabic (ar), Spanish (es), Hindi (hi), Japanese (ja), Russian (ru), Swahili (sw), and Chinese (zh). These languages are selected from five distinct language families (Appendix B). We finetune the model using up to 200 million tokens following Yong et al. (2023). We use a batch size of 4 million tokens, following Touvron et al. (2023), resulting in 20 training steps for Swahili and 50
2Across all the languages we experimented with, we successfully resolved over 80% of the hyperlinks to their corresponding English Wikipedia pages using inter-language links.
steps for other languages.3 The further details of the training are available in Appendix A. Datasets We evaluate the model using two multiple-choice question answering datasets, XCODAH and X-CSQA (Lin et al., 2021), which require commonsense knowledge to solve. We present 0-shot results for X-CODAH and 4-shot results for X-CSQA. Detailed information about these tasks is available in Appendix E. Baselines As our primary baseline, we use a model fine-tuned under the same training settings as LEIA, using the original Wikipedia corpus without the insertion of English names (denoted as LLaMA2+FT). Comparison with this baseline confirms that performance gains stem from the insertion of English names, not just from fine-tuning on the Wikipedia corpus. We also use the random baseline and the LLaMA 2 model without fine-tuning. Method configurations We test three strategies to add the English name: (1) left: inserting the name before the hyperlink, (2) right: inserting the name after the hyperlink, and (3) replace: replacing the original entity text with the name. To reduce the train-test discrepancy, we randomly omit the insertion with a probability of pskip. The example in Figure 1 adopts the right strategy with pskip = 0.0. Due to our limited computational resources, we test only pskip ∈ {0.0, 0.5}.

heading: 3.2 Results
text: We initially present the average accuracy across all languages for different method configurations in Table 1. Overall, the choice of strategy has a minimal impact on performance. Additionally, models with pskip = 0.5 consistently outperform their counterparts with pskip = 0.0 on both datasets. To reduce computational costs, we exclusively use
3The fewer training steps for Swahili are due to the significantly smaller size of the Swahili Wikipedia corpus.
the right strategy with pskip = 0.5 in subsequent experiments and refer to this setting as LEIA.
Table 2 shows our main results. LEIA outperforms all baseline models in all languages on X-CODAH and in 5 out of the 7 languages on X-CSQA. Furthermore, LEIA outperforms the LLaMA2+FT baseline in all languages on both datasets. These results demonstrate that LEIA effectively enhances cross-lingual transfer.
Furthermore, all models, including LEIA, fail to surpass the random baseline in Hindi and Swahili on X-CSQA. It appears that the models struggle to handle few-shot tasks in these two languages, likely due to the very limited presence of these languages in the pretraining corpora of LLaMA 2 (Touvron et al., 2023). Additionally, LLaMA2+FT does not outperform LLaMA 2 in several languages on both datasets. We believe this decline could be due to Wikipedia’s uniform, clean, and formal style. Overfitting to this style might result in poor model performance on texts of different styles, such as casual, informal, and question-style texts.
Additional results based on different numbers of few-shot examples are available in Appendix D.

heading: 4 Experiments with Swallow
text: In this section, we examine if bilingual language models that already possess substantial knowledge not only in English but also in the target language can benefit from knowledge transfer from LEIA. We focus on Japanese, a language with a variety of benchmark datasets, and experiment with the state-of-the-art English-Japanese LLM, Swallow 7B (Fujii et al., 2024).4 This model was developed through continual pretraining on LLaMA 2 with vocabulary extension (Cui et al., 2023; Nguyen et al., 2023; Zhao et al., 2024), using bilingual corpora consisting of 90B Japanese tokens and 10B English tokens. We show that even after the adaptation with a massive target language corpus, LEIA can further boost the performance of the model.

heading: 4.1 Setup
text: Training We fine-tune the Swallow 7B model using the Japanese Wikipedia corpus, following the same training setup described in Section 3.1. Datasets In addition to X-CODAH and X-CSQA, we use four question answering datasets available in two tools for evaluating Japanese LLMs:
4https://huggingface.co/tokyotech-llm/ Swallow-7b-hf
JEMHopQA (Ishii et al., 2023) and NIILC (Sekine, 2003) in llm-jp-eval (Han et al., 2024),5 and JCommonsenseQA (Kurihara et al., 2022) and JAQKET (Suzuki et al., 2020) in the JP Language Model Evaluation Harness.6 We present 4-shot results obtained with these tools. We use accuracy for JCommonsenseQA and JAQKET, and characterbased F-measure for JEMHopQA and NIILC. Further details are available in Appendix E. Baselines We denote the model fine-tuned using the plain Wikipedia corpus as Swallow+FT. We also evaluate Swallow without fine-tuning.

heading: 4.2 Results
text: Table 3 shows that LEIA significantly outperforms all baseline models on all datasets. This demonstrates the effectiveness of LEIA when the base model has already been trained with a massive corpus of the target language. Furthermore, similar to the experimental results with LLaMA 2 (§3.2), we observe the performance degradation of Swallow+FT compared to Swallow without fine-tuning.

heading: 5 Analysis
text: Qualitative analysis We present five random predictions of LEIA and LLaMA2+FT (§3) from the Japanese X-CODAH dataset, where LEIA answered correctly but the LLaMA2+FT failed in Table 4. They demonstrate that LEIA effectively acquires commonsense knowledge (e.g., the sea cannot be boiled) and factual knowledge (e.g., the Eiffel Tower is in Paris) via cross-lingual knowledge transfer from English. How does LEIA facilitate transfer? The English names inserted into the corpus can enhance the training in two ways: (1) names as labels: serving as labels to predict based on the preceding tokens, and (2) names as contexts: providing context for the subsequent tokens. Both aspects can facilitate cross-lingual transfer, allowing the model to apply knowledge from one language to another. To determine which causes the performance improvements, we remove the effect of using names as labels by blocking loss propagation when predicting tokens in English entity names.
The results in Table 5 show that preventing loss propagation from the English entity tokens has a minimal impact on performance. This indicates
5https://github.com/llm-jp/llm-jp-eval 6https://github.com/Stability-AI/
lm-evaluation-harness/tree/jp-stable
that LEIA’s performance enhancement is mainly attributed to using names as contexts in training. Effects of special tokens To investigate the effects of the special <translate> and </translate> tokens during training, we conduct the training on LLaMA 2 without using these tokens when inserting English names.
The results in Table 6 show that performance consistently declines on both the X-CODAH and X-CSQA datasets when these special tokens are not used during training. This suggests that these special tokens enable the model to identify the boundaries of inserted English names and play a crucial role in training.

heading: 6 Related Work
text: Language adaptation A common domain adaptation technique for language models is training on a domain-specific corpus (Gururangan et al., 2020), and when different languages are considered as different domains, it can be used for language adaptation. This strategy is shown to be effective in various models including encoder-decoder models (Neubig and Hu, 2018), bidirectional language models (Han and Eisenstein, 2019; Wang et al., 2020; Chau et al., 2020), and auto-regressive language models (Müller and Laurent, 2022; Yong et al., 2023). However, when adapting to a new language, knowledge transfer from one language to an-
other can be insufficient due to discrepancies in the surface forms. To facilitate the sharing of internal knowledge across languages, our proposed method leverages cross-lingually aligned entity names. Cross-lingual supervision for language models To enhance cross-lingual transfer, incorporating cross-lingual supervision is effective. This supervision can come from various sources, including bilingual dictionaries and bitext (Conneau and Lample, 2019; Kale et al., 2021; Reid and Artetxe, 2022; Wang et al., 2022). Wikipedia hyperlinks, which can be considered a special case of words or phrases aligned via a bilingual dictionary, have also been shown to be effective (Jiang et al., 2022; Ri et al., 2022). Exploring the potential of Wikipedia is promising as its high-quality formal text and the continual expansion of data across many languages. Our study showcases the benefits of using crosslingually aligned entity names in continual training of language models.

heading: 7 Conclusion
text: We introduced LEIA, a method to facilitate crosslingual knowledge transfer via fine-tuning language models using Wikipedia text augmented with English entity names. We applied LEIA to the English LLM, LLaMA 2, and the English-Japanese LLM, Swallow, demonstrating significant improvements on various non-English question answering tasks.
Future research will investigate LEIA’s effectiveness using an arbitrary text corpus with entity annotations generated through entity linking, instead of the Wikipedia corpus, and employing the augmented corpus during the pretraining of bi- or multi-lingual LLMs, rather than relying on posthoc fine-tuning.

heading: 8 Limitations
text: Our evaluation focused on question answering tasks, on the basis of the assumption that knowl-
edge transferred from entity names mainly includes commonsense and world knowledge. While this type of knowledge has the potential to benefit a wider range of tasks, broader evaluation is left for future research.
The data source for our method is Wikipedia, limiting our coverage to languages represented therein. However, our approach is adaptable to incorporate other forms of cross-lingual supervision such as bilingual dictionaries. Such extensions could enhance the applicability of our proposed framework to additional languages not currently covered by Wikipedia.

heading: Appendix for “LEIA: Facilitating Cross-lingual Knowledge Transfer in Language Models with Entity-based Data
text: Augmentation”

heading: A Details of Training
text: Preprocessing The training corpus is derived from the target language edition of October 2023 Wikipedia dump.7 We extract text and hyperlinks using the open-source WikiExtractor tool.8 Each entity referenced by a hyperlink is mapped to its English equivalent using the inter-language link database obtained from the October 2023 Wikidata dump.9 We filter out entities whose English names begin with the following prefixes, denoting special Wikipedia entities:
• Book:
• Category:
• Draft:
• File:
• Help:
• List of
• MediaWiki:
• Portal:
• Special:
• Talk:
• Template:
• User:
• Wikipedia:
• WikiProject:
Additionally, we remove suffix strings enclosed in parentheses from entity names, e.g., “(state)” in “Washington (state)”. The entity names are enclosed in the special <translate> and </translate> tokens, and inserted into the Wikipedia text as described in §2. Training We train our models using the AdamW optimizer (Loshchilov and Hutter, 2017) with β1 = 0.99 and β2 = 0.95, following Touvron et al. (2023). We use a cosine learning rate schedule with an initial learning rate of 5e-6. The model checkpoint corresponding to the last training step
7https://dumps.wikimedia.org/ 8https://github.com/attardi/ wikiextractor 9https://dumps.wikimedia.org/ wikidatawiki/
is used as the final model. The detailed hyperparameters are shown in Table 7. To enhance training efficiency, we create an input sequence by concatenating multiple short Wikipedia articles until the maximum token length is reached.
The special tokens, i.e., <translate> and </translate>, are added to the vocabulary of the model and their embeddings are initialized using the mean embedding derived from all token embeddings.
We use a machine with eight Nvidia A100 40GB to train the model. The LLaMA 2 (Touvron et al., 2023) and Swallow (Fujii et al., 2024) models, both available under the LLaMA 2 Community License10, are used as our base models. The training approximately takes 2 hours for Swahili in the experiments with LLaMA 2 and 5 hours for the other models. We adopt data parallelism with sharding parameters across GPUs using DeepSpeed Zero Redundancy Optimizer (Rajbhandari et al., 2019), mixed precision training with bfloat16, and FlashAttention-2 (Dao, 2023) to reduce computational costs.
10https://ai.meta.com/llama/license/

heading: B Details of Languages Used in LLaMA 2 Experiments
text: Our experiments with LLaMA 2 (§3) are conducted in seven languages from five diverse language families shown in Table 8.

heading: C Full Experimental Results
text: The detailed per-language results of comparing different method configurations of LEIA are presented in Table 9. The Japanese sentences corresponding to the English sentences in Table 4 can be found in Table 10. The detailed results for our models with loss propagation enabled and disabled when predicting tokens in English entity names are provided in Table 11.

heading: D Results with Varied Number of Few-shot Examples
text: The experimental results with LLaMA 2 using fourand seven-shot examples are available in Table 12. LEIA consistently outperforms the baseline models in both settings.

heading: E Details of Experiments
text: We evaluate our models using two multilingual question answering datasets, X-CODAH and XCSQA (Lin et al., 2021), along with four Japanese question answering datasets: JEMHopQA, NIILC, JCommonsenseQA, and JAQKET. While inputs of X-CODAH consist of single texts, inputs for
the other datasets are divided into questions and answers. This necessitates specifying the input format for the model, leading us to use a few-shot setting for datasets other than X-CODAH, instead of a zero-shot setting.
For the JEMHopQA and NIILC datasets, which do not provide answer candidates, the model generates textual answers, and its performance is measured using a character-based F-measure, following Han et al. (2024). For other tasks, we input each answer candidate into the models, and select the one with the highest probability. We use the llm-jp-eval tool for JEMHopQA and NIILC, and the JP Language Model Evaluation Harness for JCommonsenseQA and JAQKET. The prompts for our experiments are presented in Figures 2–6, with {question}, {answer}, and {choiceX} replaced by the actual question, answer, and answer candidates, respectively. The prompts shown in Figures 3–4 and Figures 5–6 are the default prompts in llm-jpeval and JP Language Model Evaluation Harness, respectively.
The detailed descriptions of the datasets used in our experiments are provided as follows:
• X-CODAH (Lin et al., 2021) is a four-way multiple-choice, multilingual question answer-
ing dataset created by translating the English CODAH dataset (Chen et al., 2019). We use the validation set, consisting of 300 examples, as the test set labels are not publicly accessible. We do not use a hand-crafted prompt and simply input the original text. This dataset is obtained from the corresponding Hugging Face repository11 licensed under the MIT license.
• X-CSQA (Lin et al., 2021) is a five-way multiple-choice, multilingual question answering dataset, translated from the CommonsenseQA dataset (Talmor et al., 2019). Due to the unavailability of test set labels, we use the validation set, which comprises 1,000 instances. Few-shot examples are randomly selected from the same set. The prompt for this
11https://huggingface.co/datasets/xcsr
dataset is shown in Figure 2. This dataset is obtained from the same repository as the X-CODAH dataset.
• JEMHopQA (Ishii et al., 2023) is a Japanese question answering dataset with 120 input questions and their ideal answers. Few-shot examples are selected from its dedicated set. The prompt for this dataset is shown in Figure 3. This dataset is licensed under the Creative Commons CC BY-SA 4.0 license.
• NIILC (Sekine, 2003) is a Japanese question answering dataset comprising 198 input questions and their ideal answers. Few-shot examples are selected from its dedicated set. The
文章と質問と回答の選択肢を入力として受 け取り、選択肢から質問に対する回答を選 択してください。なお、回答は選択肢の番 号(例:0)でするものとします。
質問：{question} 選択肢:0.{choice0},1.{choice1},1.{choice1}, 2.{choice2},3.{choice3},4.{choice4} 回答:{answer}

heading: Future Work
text: While this type of knowledge has the potential to benefit a wider range of tasks, broader evaluation is left for future research. The data source for our method is Wikipedia, limiting our coverage to languages represented therein. However, our approach is adaptable to incorporate other forms of cross-lingual supervision such as bilingual dictionaries. Such extensions could enhance the applicability of our proposed framework to additional languages not currently covered by Wikipedia