Venue: NAACL 2024

# Improving In-context Learning of Multilingual Generative Language

Models with Cross-lingual Alignment

Chong Li, Shaonan Wang, Jiajun Zhang1 , Chengqing Zong

State Key Laboratory of Multimodal Artificial Intelligence Systems,

Institute of Automation, CAS, Beijing, China

School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, China

lichong2021@ia.ac.cn,

{shaonan.wang, jjzhang, cqzong}@nlpr.ia.ac.cn

Corresponding author.

###### Abstract

Multilingual generative models obtain remarkable cross-lingual in-context learning capabilities through pre-training on large-scale corpora. However, they still exhibit a performance bias toward high-resource languages and learn isolated distributions of multilingual sentence representations, which may hinder knowledge transfer across languages. To bridge this gap, we propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning and aligns outputs by following cross-lingual instructions in the target language. Experimental results show that even with less than \(0.1\)% of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models and mitigates the performance gap. Further analyses reveal that it results in a better internal multilingual representation distribution of multilingual models. 1

Footnote 1: Our code is available at [https://github.com/chongli17/CrossLingualAlignment_Proceedings](https://github.com/chongli17/CrossLingualAlignment_Proceedings) of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers)_, pages 8058–8076

June 16-21, 2024 02024 Association for Computational Linguistics

## 1 Introduction

Multilingual generative language models achieve impressive universality across many languages by pre-training on large-scale unsupervised multilingual corpora Liu et al. (2020); Xue et al. (2021); Lin et al. (2022); Scao et al. (2022); Soltan et al. (2022); OpenAI (2022). However, models still show a strong language bias toward high-resource languages Asai et al. (2023), even the state-of-the-art multilingual generative models like GPT-4, exhibiting a 27.5% relative performance gap between English and Telugu in MMLU OpenAI (2023). This challenge partly arises from the significant linguistic resource imbalance among languages, which is hard to address solely through corpus scaling or balancing. Given such a model with language bias and the huge cost of re-training, how can we improve its cross-lingual capabilities and alleviate the language bias using limited data?

Previous work focused on scaling multilingual instructions Muennighoff et al. (2023); Zhu et al. (2023), ignoring internal alignment and knowledge transfer between languages in the multilingual generative model. Through visualizing the sentence representations in the multilingual generative model by mean pooling, we find that there is a distinct gap between the sentence representation distributions for different languages like Figure 1(a) (the multilingual ones are shown in Appendix B.3). This is similar to learning representations for each language separately in the model, which is more challenging for multilingual models to transfer the knowledge learned from other languages. Thus, it is important to investigate whether the cross-lingual ability of models will be promoted by learning a better-aligned representation distribution.

To address the above issues, we propose a cross-lingual alignment framework named Align After Pre-training (**AFP**), which aims to exploit translation pairs to narrow the gap between languages

Figure 1: (a, b) Our method aligns the internal EN-ZH sentence representations of XGLM\({}_{56\text{MM}}\), which are shown in t-SNE. (c) It also mitigates the performance gap on XNLI.



in the multilingual generation model. To be specific, our method can be divided into the following two modules: 1) **Multilingual Contrastive Learning (MCL)** on internal representations: we treat a translation sentence pair as the positive example for contrastive learning, and pull the sentence representations in two languages to be closer within the multilingual generated model. This module intends to reduce the differences between languages from the internal representations of the model. 2) **Cross-lingual Instruction Following (CIF)** on the outputs: models must learn to answer in the target language given a prompt from the source language. It aims at enhancing semantic coherence and knowledge transfer across languages in the model.

After extensive experiments, it can be found that AFP greatly improves the performance of multilingual generative models in cross-lingual natural language inference, multilingual reasoning, and other tasks using less than 1M parallel samples. The performance gap between languages is narrowed, e.g., the relative performance gap of XGLM\({}_{564\text{M}}\) reduces 6.53% on XNLI between English and Chinese (Figure 1(c)). Our method also advances the performance on unseen languages for models, e.g., the Chinese performance of Llama, which is pre-trained on the corpus mainly in English Touvron et al. (2023); [7]. Further analyses reveal that the representation gap has been mitigated as illustrated in Figure 1(b) after training with AFP. In addition, experimental results show that the cross-lingual instruction following task is better than the multilingual instruction tuning task in promoting cross-lingual ability with the same parallel corpus.

To sum up, our main contributions are as follows:

* We propose a simple yet effective cross-lingual alignment framework, including the internal representation alignment (MLL) and external output alignment (CIF), to exploit the parallel corpus for multilingual generative models.
* Experimental results demonstrate that our method greatly improves the cross-lingual ability of generative models, including multilingual ones and models pre-trained on English corpus, by using less than 1M samples.
* Further analyses reveal that AFP promotes the alignment and uniformity of internal multilingual representation distributions. Ablation study shows that using the internal representation alignment of AFP alone cannot boost multilingual generative models.

## 2 Related Work

### Multilingual Generative Language Model

Through unsupervised pre-training on the large-scale multilingual corpus, generative language models obtain impressive multilingual abilities, e.g., multilingual machine translation Liu et al. (2020); He et al. (2021); Wang et al. (2022); Lu et al. (2023), cross-lingual natural language understanding Xue et al. (2021) and cross-lingual in-context learning Lin et al. (2022); Scao et al. (2022); Wei et al. (2023); Anil et al. (2023). Most of them extended the pre-training method developed for the monolingual corpus Lewis et al. (2020); Raffel et al. (2020) and relied on a balanced sampling method across languages, while a significant performance gap between high-resource languages and low-represented languages persists in the pretrained model Asai et al. (2023). Different from the unsupervised pre-training on the multilingual corpus, this work attempts to alleviate the performance gap across languages by cross-lingual alignment using parallel samples.

### Multilingual Instruction Tuning

Large language models show better zero-shot multilingual performance and language generalization results after multilingual instruction tuning Muennighoff et al. (2023); Zhang et al. (2023); Zhu et al. (2023); Ranaldi et al. (2023). Our cross-lingual instruction following task requires the model to respond in the target language and is different from multilingual instruction tuning, where prompt and answer in the same language for each sample.

### Contrastive Learning in Natural Langauge Processing

Most of the work in NLP adopted contrastive learning to improve the representation of sentences in the language model Reimers and Gurevych (2019); Pan et al. (2021); Gao et al. (2021); Yang et al. (2021); Pan et al. (2022); Ni et al. (2022); Sherborne et al. (2023). Specifically, contrastive learning is often applied to the sentence representations of encoder Cao et al. (2020); Fang et al. (2020); Wu et al. (2020); Pan et al. (2021); Chi et al. (2021); Wei et al. (2021). However, it is less explored how to promote the representation of Transformer decoder models 

(Vaswani et al., 2017; Zhao et al., 2023). In this work, we try to improve the internal multilingual representation of the decoder models by multilingual contrastive learning rather than the one of encoder Wang et al. (2021); Qin et al. (2022).

## 3 Method

As shown in Figure 2, our framework AFP contains the following two modules: 1) Multilingual contrastive learning (Section 3.1), which aims to align the internal representations of models across different languages. 2) Cross-lingual instruction following (Section 3.2), which requires models to align the outputs between different languages.

### Multilingual Contrastive Learning

To align the internal multilingual representation of models, we exploit the contrastive learning method, which is generally found effective in aligning the representations from different modalities in multimodal work Radford et al. (2021); Xu et al. (2021); Liang et al. (2022). Hence, translation pairs are regarded as positive instances with closely aligned semantics in multilingual contrastive learning, and we pull their internal representations closer. The other sentences in the same batch are taken as the negative samples for the translation pair.

Formally, to align the \(l\)-th layer of model \(f(\theta)\), the sentence representations \((h_{i},h^{+}_{i})\) is calculated as follows:

\[h_{i}=g(f_{l}(s_{i};\theta)),~{}h^{+}_{i}=g(f_{l}(s^{+}_{i};\theta)) \tag{1}\]

where \(f_{l}(\cdot)\) represents the output from the \(l\)-th layer, \(g(\cdot)\) is the pooling method to obtain the sentence representation for decoder models, e.g., mean pooling or max pooling, and \((s_{i},s^{+}_{i})\) is a parallel sample from \(\mathcal{D}=\left\{(s_{1},s^{+}_{1}),...,(s_{n},s^{+}_{n})\right\}\). We determine the specific layer to align according to the performance of models on the dev set and find that the first layer after embedding comes to better performance (please refer to Section 4.2.2 for more details). Then, the training objective of Multilingual Contrastive Learning (MCL) is:

\[\small\mathcal{L}_{\textit{MCL}}(\theta)=\mathbb{E}_{(s_{i},s^{+}_{i})\sim \mathcal{D}}\left[-\log\left(\frac{e^{\text{sim}(h_{i},h^{+}_{i})/\tau}}{\sum_ {j}e^{\text{sim}(h_{i},h_{j})/\tau}}\right)\right] \tag{2}\]

where sim\((\cdot)\) is used to determine the similarity between representations, which is cosine similarity in this work, \(h_{j}\) is the sentence representation of \(s_{j}\) in the mini-batch containing \((s_{i},s^{+}_{i})\), and \(\tau\) is a temperature hyper-parameter.

### Cross-lingual Instruction Following

To further align the output of multilingual generative models, we introduce a method named Cross-lingual Instruction Following (CIF), which imposes models to respond in the target language given the source language as the context. It is more difficult than the multilingual instruction tuning task, which prompts and answers in the same language for each sample, and requires a better cross-lingual understanding and generation ability for multilingual generative models.

Specifically, given a pair of context and response \((c^{a}_{i},r^{a}_{i})\) from a Dataset \(\mathcal{D}^{a}\) in the same language \(a\), e.g., an English instruction tuning dataset like FLAN or Alpaca Wei et al. (2022); Wang et al. (2023); Taori et al. (2023), response \(r^{a}_{i}\) is first translated into the target language \(b\) by the translator \(t^{a\to b}(\cdot)\). We append a prompt \(p^{b}\) informing the target language \(b\), e.g., "Answer in German" in Figure 2, at the end of context to construct the training sample \(\left(c^{a\to b}_{i}=c^{a}_{i}+p^{b},r^{b}_{i}=t^{a\to b}( r^{a}_{i})\right)\) for CIF. Therefore, the loss function of CIF for the multilingual generative model \(f(\theta)\) comes to:

\[\small\mathcal{L}_{\textit{CIF}}(\theta)=\mathbb{E}_{(c^{a}_{i},r^{a}_{i}) \sim\mathcal{D}^{a}}\left[\sum_{j}-\log\left(\mathsf{P}(r^{b}_{ij}|c^{a\to b }_{i},r^{b}_{i,<j};\theta)\right)\right] \tag{3}\]

where the target language \(b\) has the possibility \(p_{\textit{src}}\in[0,1]\) to be set the same as the source language \(a\), which is a hyper-parameter and investigated in Section 4.2.3. When the target language is always the source language of the context (\(p_{\textit{src}}=1\)), it degenerates into the vanilla multilingual instruction tuning method.

With the two modules of aligning methods mentioned before, Multilingual Contrastive Learning (MCL) and Cross-lingual Instruction Following (CIF), we obtain the following loss function of our alignment framework AFP:

\[\small\mathcal{L}_{\textit{AFP}}(\theta)=\mathcal{L}_{\textit{MCL}}(\theta)+ \alpha\mathcal{L}_{\textit{CIF}}(\theta) \tag{4}\]

where \(\alpha\in\mathbb{R}^{+}_{0}\) is a hyper-parameter to balance the two alignment methods.

## 4 Experiments

### Experiments Settings

Parallel CorporaTo cover more parallel samples from different domains and languages, we adopt a multilingual instruction tuning dataset named Bactrian-X Li et al. (2023), which is translated into 52 languages from Alpaca Taori et al. ( 

2023) and Dolly (Conover et al., 2023) by Google Translate, and a multilingual machine translation dataset, OPUS-100 (Zhang et al., 2020), to align the models evaluated. Only 100k parallel samples are selected from OPUS-100 in our experiments to match the amount of Bactrian-X, which contains 67k samples for each language. The number of tokens used is about 20M, which is nearly 0.05 % of tokens used in the pre-training of BLOOM (Scao et al., 2022).

Language modelsWe apply AFP on two multilingual generative model structures, XGLM (Lin et al., 2022) and BLOOM (Scao et al., 2022), across three different parameter amounts. The models fine-tuning with multilingual instruction tuning, "+MIT" or BLOOMZ (Muennighoff et al., 2023), are taken as the baseline. Llama (Touvron et al., 2023), which is mainly pre-trained on English corpus, is also included for comprehensive evaluation. Training settings and hyperparameters are reported in Appendix A.

Multilingual TasksWe evaluate the performance of models on the following benchmarks:

* **Natural Language Inference** We use XNLI (Conneau et al., 2018) in this task.
* **Paraphrase Detection** PAWS-X (Yang et al., 2019) is evaluated for this task.
* **Reasoning** We adopt XCOPA (Ponti et al., 2020), XStoryCloze (Lin et al., 2022) and XWinograd (Tikhonov and Ryabinin, 2021) in this task.
* **Machine Translation** For this task, we use FLORES-101 (Goyal et al., 2022).

The detailed descriptions and prompt formats for each task during evaluation are presented in Appendix C. We keep the same prompt formats across all multilingual generation models for a fair comparison.

### Bilingual Results and Analyses

To make a comprehensive analysis of the influence on performance and representations in models, we first conduct bilingual alignment experiments in English and Chinese. Then we extend to the condition of multilingual alignment (Section 4.3).

Table 1 shows the experimental alignment results on EN-ZH parallel samples. These generative models, including three architectures with different amounts of parameters, are consistently improved by our method. The average improvement is up to 3.31% using only 167k parallel samples, and the models with 7B parameters surpass the GPT-3 with comparable parameters after alignment. Specifically, models improve 4.28% on the first two natural language understanding tasks (XNLI and PAWS-X), and 2.67% on the other three reasoning tasks. After alignment using AFP, BLOOM shows a better performance than the BLOOMZ model with the same amount of parameters, which is fine-tuned on 78M multilingual instructions (Muennighoff et al., 2023).

It is interesting to find that the model Llama pre-trained on mainly English corpus, also obtains improvement after bilingual alignment using AFP. The performance on the unseen language Chinese is even comparable with the one pre-training on an

Figure 2: Illustration of how to align the internal representations and outputs of multilingual generative models with AFP. (I) Given a translation parallel sample as the positive sample, multilingual contrastive learning pulls their representations together and pushes apart the ones from other samples. (II) Multilingual generative models are required to answer in the target language to align the outputs across languages.

 

additional 20GB Chinese corpus Cui et al. (2023). This result further proves the effectiveness of our method. We assume that this performance gain may benefit from better-aligned multilingual representations in models, which promotes the transfer of knowledge learned in the English corpus.

In addition to cross-lingual understanding and reasoning abilities, the multilingual generation ability of models has been improved. The bilingual translation results of XGLM models are reported in Table 2. Models not only obtain a better cross-lingual generation ability, but also show a more balanced generation performance than the vanilla ones between both directions. It is interesting to find that the average performance of models in the zero-shot condition improves from 27.3 to 61.2 COMET on average, which may come from the response in the target language format used in cross-lingual instruction following is similar to the one in the machine translation task.

#### 4.2.1 AFP Brings Better Bilingual Representations

**Visualization of sentence representations.** Given \(1\mathrm{k}\) EN-ZH translation parallel samples, we visualize the sentence representations of XGLM\({}_{564\text{M}}\) and BLOOM\({}_{560\text{M}}\), which are obtained by the mean pooling method using the representation for each token in one sentence. In the vanilla models, there is a distinct separation between sentence representations from different languages (Figure 1(a) and 3(a)). However, the ones using

\begin{table}
\begin{tabular}{c|c c c|c c c|c c c|c c} \hline \hline  & \multicolumn{3}{c}{**EN\(\rightarrow\)ZH**} & \multicolumn{3}{c}{**ZHI\(+\)EN**} & \multicolumn{3}{c}{**Avg**} \\ \cline{2-19}
**Model** & 0 & 1 & 5 & 10 & 0 & 1 & 5 & 10 & 0 & 1 & 5 & 10 \\ \hline XGLM\({}_{560\text{M}}\) & 25.3 & 31.6 & 62.2 & 63.3 & 26.7 & 67.4 & 69.8 & 70.8 & 26.0 & 49.5 & 66.0 & 67.1 \\ +AFP & **52.7** & **62.8** & **65.4** & **67.6** & **65.9** & **70.9** & **71.8** & **72.3** & **59.3** & **66.9** & **68.6** & **69.9** \\ XGLM\({}_{58\text{L}}\) & 28.1 & 79.3 & 79.8 & 80.1 & 29.1 & 81.6 & 81.8 & 82.2 & 28.6 & 80.4 & 80.8 & 81.2 \\ +AFP & **57.4** & **80.4** & **80.8** & **81.0** & **68.8** & **81.7** & **81.8** & **82.4** & **63.1** & **81.1** & **81.3** & **81.7** \\ \hline \end{tabular}
\end{table}
Table 2: Translation results of COMET Rei et al. (2020) on FLORES-10I devet set.

\begin{table}
\begin{tabular}{c|c c|c c|c c|c c|c c|c c} \hline \hline  & \multicolumn{3}{c}{**XNLJ**} & \multicolumn{3}{c}{**PANS-X**} & \multicolumn{3}{c}{**XCOPA**} & \multicolumn{3}{c}{**XStoryClore**} & \multicolumn{3}{c}{**XWinograd**} \\ \cline{2-19}
**Model** & **EN-u/5** & **ZH-u/5** & **EN-u/5** & **ZH-u/5** & **EN-u/5** & **ZH-u/5** & **EN-u/5** & **EM-u/5** & **EN-u/5** & **ZH-u/5** & **Arg** \\ \hline GPT-3\({}_{67\text{m}}\) & 55.3/52.8 & 42.4/45.9 & 60.6/59.7 & 53.2/54.1 & 73.6/74.5 & 55.0/57.7 & 73.6/74.5 & 55.9/54.5 & 64.6/68.1 & 71.5/72.2 & 61.0 \\ \hline XGLM\({}_{560\text{M}}\) & 45.5/41.2 & 47.3/65.5 & 50.4/46.6 & 50.9/47.8 & 56.4/69.6 & 52.8/52.2 & 59.6/60.8 & 54.3/52.9 & 54.8/56.7 & 67.1/66.9 & 52.5 \\ +MIT & 46.6/43.9 & 37.5/41.6 & 53.5/53.1 & 52.3/51.0 & 57.6/51.0 & 57.2/55.4 & 61.1/61.3 & 57.5/54.5 & 54.8/57.7 & 66.7/65.3 & 54.4 \\ +APF & **48.1/65** & **41.6/42** & **54.2/38** & **53.2/51.8** & **52.6** & **62.0**/62.2** & **59.0**/88.8** & **62.2/62.5** & **56.3**/56.1** & **55.6**/59.0** & **67.5**/**67.3** & **56.1** \\ \(-\)**+APF & **54.1/49.9** & 45.4/42.4 & 58.9/56.3 & 52.9/55.8 & 69.4/74.6 & 62.4/63.2 & 69.2/73.7 & 59.5/59.2 & 62.8/66.4 & 73.8/73.72 & 61.2 \\ +MIT & 54.3/54.1 & 47.8/44.8 & 63.1/57.3 & 54.4/55.0 & 69.4/75.0 & 63.2/64.6 & 71.1/74.3 & 60.1/61.7 & 64.5/67.5 & 74.4/73.4 & 62.5 \\ +APP & **55.0/4** & **48.0/8.8** & **64.1/2 

AFP come to be more aligned between languages and uniform (Figure 1(b) and 3(b)), which means our method promotes the representation of the model to be better-aligned from a qualitative point of view.

Alignment and uniformity.The distribution of multilingual representations is quantified by the two metrics, **alignment** and **uniformity** proposed by Wang and Isola (2020), for further analysis. Specifically, the alignment score measures the expected distance between the representations of positive samples, which are translation parallel samples for multilingual generative models, and is calculated as follows:

\[{\ell_{\text{align}}}\triangleq\operatorname*{\mathbb{E}}_{(x,x^{+})\sim \mathcal{D}_{pos}}\big{\|}f(x)-f(x^{+})\big{\|}^{2} \tag{5}\]

where \(\mathcal{D}_{pos}\) is the distribution of positive samples.

In contrast, uniformity reflects the degree of uniformly distributed for representations:

\[{\ell_{\text{uniform}}}\buildrel\triangle\over{=}\text{log}\operatorname*{ \mathbb{E}}_{x,y^{i}\cdot\iota d\cdot\mathcal{D}}e^{-2\lVert f(x)-f(y) \rVert^{2}} \tag{6}\]

where \(x\) and \(y\) are randomly sampled from the distribution \(\mathcal{D}\). Therefore, the smaller \({\ell_{\text{align}}}\) and \({\ell_{\text{uniform}}}\) are, the better representations models learn.

Figure 4 illustrates the deviation of \({\ell_{\text{align}}}\) and \({\ell_{\text{uniform}}}\) for XGLM\({}_{564\text{M}}\) using different training methods on the same training data. The initial 5000 steps are visualized, with one point for every 500 steps. We can find that the metrics are both decreasing using AFP, while the bilingual pre-training only improves the uniformity of representations. The results further prove that our method improves the multilingual representation distributions within the multilingual generative models.

#### 4.2.2 Multilingual Contrastive Learning on Bottom Layer Performs Better

Figure 5(a) presents the impact of different layers applied by contrastive learning on the 5 cross-lingual datasets (XNLI, PAWS-X, XCOPA, XS-toryCloze, and XWinograd). The average performance of models shows a trend of decreasing first and then increasing, which changes at the 10th layer for XGLM\({}_{564\text{M}}\) or the 17th layer for BLOOM\({}_{560\text{M}}\). And the first transformer layer is better for both models when using multilingual contrastive learning. As a result, multilingual contrastive learning is applied to the first layer after the embedding layer by default.

#### 4.2.3 Cross-lingual Instruction Following or Multilingual Instruction Tuning?

As shown in Figure 5(b), multilingual instruction tuning (\(p_{src}=1\)) is inferior to cross-lingual instruction following (\(p_{src}<1\)) for the models evaluated. Moreover, the result becomes suboptimal when all samples are transferred into the cross-lingual format (\(p_{src}=0\)). We empirically set the \(p_{src}\) to 0.5 in the cross-lingual instruction following task.

### Multilingual Results and Analyses

In addition to the bilingual alignment, AFP can be applied to align the models in multilingual conditions. English is first chosen as the pivot language of alignment for the dominance performance in multilingual generative models. That is, the input parallel samples of AFP are selected from the EN-XX corpus, e.g., EN-ZH and EN-TH, to pull the representations and outputs of models in other languages closer to the ones in English. We also investigate the other alignment methods like pairwise alignment in Section 4.3.1, which shows an inferior performance.

Table 3 reports the results of alignment between 5 languages from different language families, where the performance of models on the NLI and

Figure 4: The deviation of \({\ell_{\text{uniform}}}\)–\({\ell_{\text{align}}}\) for XGLM\({}_{564\text{M}}\) during training process with different multilingual training methods. The smaller these two metrics are, the better representations models learn. “BPre” and “BIT” denote bilingual pre-training and bilingual instruction tuning, respectively.

 

reasoning tasks is improved by 3.72% from high-resource languages to the less-represented language Swahili.

Moreover, models with AFP obtain a more balanced performance distribution. Taking XGLM models as an example, the variance of performance across 5 languages decreases from 3.44% to 2.96% on average. It is noted that AFP advances the performance of BLOOM in the two unseen languages, Thai (TH, +3.9%) and Turkish (TR, +3.92%).

Multilingual generative models also obtain a performance gain (+0.75 BLEU) in the multilingual machine translation task after alignment (Table 4). It can also find a more balanced performance distribution across languages, where the average variance reduction is 0.4% for the models evaluated.

#### 4.3.1 English as a pivot language or Pairwise Alignment?

Besides adopting English as a pivot language to align multilingual representations, we also investi 

gate the pairwise alignment policy, which is aligned by languages in pairs. For example, assuming to align the representations of English (EN), Chinese (ZH), and Thai (TH), the former policy comes to two parallel samples for input, which are EN-ZH and EN-TH, while the latter contains three parallel samples: EN-ZH, EN-TH, and ZH-TH.

The results of five languages alignment experiments on XNLI and XCOPA are reported in Table 5. The pairwise alignment policy performs consistently better in the low-resource language Swahili, although its average improvement is inferior to that when adopting English as a pivot language.

#### 4.3.2 Combination with Other Cross-lingual Methods

After alignment, multilingual generative models can use other cross-lingual methods for further improvement. We take a method named semantic alignment for an example, which is able to promote the cross-lingual ability using semantic aligned demos in prompt Tanwar et al. (2023). As shown in Table 6, models obtain a further 0.4% improvement in the multilingual NLI and reasoning tasks on average.

### Extended to Alignment in 52 Languages

Based on the above analyses, we extend the alignment to all 52 languages in the Bactrian-X dataset by adopting English as a pivot language (information about all languages involved is reported in Appendix D). As shown in Table 7, models obtain a 2.6% improvement in 5 multilingual tasks on average, and mitigate the variance across languages. It is also noted that the performance of BLOOM\({}_{7.1\text{B}}\) on unseen languages among 5 datasets is improved by 2.8% using only parallel samples via our alignment framework, which may arise from the knowledge transferred from other languages after alignment.

### Ablation Study

To take a deep look into the improvements contributed by AFP, we conduct an ablation study on the 5 datasets of bilingual tasks using XGLM\({}_{564\text{M}}\) (Table 8).

The in-context learning abilities of the models decrease when only multilingual contrastive learning (MLC) is used. It may arise from the next word prediction ability of the model is affected by the MCL. Using the same data, both multilingual instruction tuning (MIT, +1.3%) and cross-lingual instruction following (CIF, +2.1%) can improve multilingual generative models, while the latter can promote it more. In addition, the performance of the models can be further improved after combining MCL and CIF, which is the proposed alignment framework AFP.

## 5 Conclusion and Future Work

In this paper, we proposed a simple yet effective multilingual alignment framework, including internal multilingual representations alignment and cross-lingual outputs alignment methods. Experimental results show that this framework improves both the internal representations and cross-lingual capabilities of generative models across various scales.

Beyond aligning different languages, our framework can be extended to align the internal representations and outputs across different modalities in the multi-modal generative models by replacing parallel samples. However, it is noted that the current framework relies on labeled training data for

\begin{table}
\begin{tabular}{c c|c c c c c c c c c} \hline \hline  & \multicolumn{2}{c}{**XNLI**} & \multicolumn{2}{c}{**PAVS-X**} & \multicolumn{2}{c}{**SCOPA**} & \multicolumn{2}{c}{**XStoryClore**} & \multicolumn{2}{c}{**XWinograd**} \\ \cline{2-9}
**Model** & **0-shot** & **5-shot** & **0-shot** & **5-shot** & **0-shot** & **5-shot** & **0-shot** & **5-shot** & **Avg** \\ \hline XGLM\({}_{56}\) & 43.6\(\pm\)3.4 & 43.6\(\pm\)3.1 & 55.47\(\pm\)3.1 & 55.1\(\pm\)1.6 & 58.9\(\pm\)5.0 & 60.4\(\pm\)5.7 & 60.6\(\pm\)3.9 & 60.5\(\pm\)5.0 & 63.9\(\pm\)5.1 & 64.7\(\pm\)4.2 & 55.3\(\pm\)8.5 \\ +AFP & **47.5\(\pm\)3.3** & **47.7\(\pm\)3.0** & **57.7\(\pm\)3.3** & **57.5\(\pm\)1.4** & **Gl3.4\(\pm\)5.5** & **62.4\(\pm\)5.7** & **62.4\(\pm\)3.7** & **63.5\(\pm\)4.9** & **65.5\(\pm\)4.8** & **66.7\(\pm\)4.1** & **57.8\(\pm\)8.0** \\ \hline BLOOMZ\({}_{7.18}\) & 44.1\(\pm\)4.0 & 43.5\(\pm\)6.0 & 57.8\(\pm\)2.6 & **56.6\(\pm\)2.9** & 53.1\(\pm\)3.5 & 54.6\(\pm\)5.5 & 58.9\(\pm\)6.7 & 61.0\(\pm\)4.7 & 60.0\(\pm\)4.9 & 60.4\(\pm\)5.0 & 54.2\(\pm\)8.3 \\ BLOOM\({}_{7.18}\) & 43.3\(\pm\)5.5 & 42.5\(\pm\)4.7 & 54.5\(\pm\)3.1 & 53.5\(\pm\)3.6 & 52.3\(\pm\)4.7 & 53.3\(\pm\)4.0 & 57.3\(\pm\)6.2 & 59.2\(\pm\)7.2 & 59.0\(\pm\)6.2 & 59.2\(\pm\)5.2 & 52.0\(\pm\)8.2 \\ +AFP & **45.4\(\pm\)**4.5 & **45.9\(\pm\)**3.9 & **58.1\(\pm\)**2.6 & 56.1\(\pm\)3.1 & **58.0\(\pm\)**3.7 & **55.1\(\pm\)**3.9 & **61.3\(\pm\)**6.0 & **62.5\(\pm\)**7.2 & **61.1\(\pm\)**5.8 & **60.5\(\pm\)**5.2 & **54.7\(\pm\)**8.0 \\ \hline \hline \end{tabular}
\end{table}
Table 7: In-context learning results of models on 5 datasets across all languages. The variance of performance across languages is marked in the subscript. All results are reported in Appendix B.4.

\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **0-shot** & **3-shot** & **5-shot** \\ \hline XGLM\({}_{564\text{M}}\) & 52.94\(\pm\)0.54 & 51.71\(\pm\)0.90 & 52.03\(\pm\)0.89 \\ \hline w/ MCI. & 50.23\(\pm\)0.43 & 48.66\(\pm\)0.51 & 48.60\(\pm\)0.49 \\ w/ MIT & 54.25\(\pm\)0.49 & 53.54\(\pm\)0.75 & 52.93\(\pm\)0.68 \\ w/ CIF & 55.31\(\pm\)0.55 & 54.02\(\pm\)0.63 & 53.58\(\pm\)0.64 \\ w/ AFP & **55.97\(\pm\)**0.48 & **55.50\(\pm\)**0.55 & **56.15\(\pm\)**0.43 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Ablation study of different training methods on 5 datasets for XGLM\({}_{564\text{M}}\).

 

alignment. Future works can focus on the unsupervised multilingual alignment method for language models.

## Limitations

Firstly, although our cross-lingual framework boosts the cross-lingual ability of multilingual generative language models using only a small amount of parallel samples, it is noted that the proposed framework relies on labeled training data for alignment, which is unavailable for languages without parallel samples.

In addition, due to limited computation resources, our framework is constrained to multi-lingual generative language models with less than or equal to 7.5B parameters.

Lastly, there is an error propagation problem from the involved machine translation system, which may result in inferior performance.

## Ethical Considerations

Since our alignment framework is applied to the pre-trained multilingual generative language models, the model aligned may inherit the potential risk and bias in the vanilla language model Tamkin et al. (2021). The cultural bias and offensive response in English may be incorporated into other languages due to the alignment policy used, which adopts English as a pivot language. Future explorations are needed to mitigate the risk and cultural bias in multilingual generative language models.

## Acknowledgements

We thank anonymous reviewers for their insightful comments and suggestions. The research work was supported by the National Science Foundation of China (No. 62036001 and No. 62122088).

## References

* Anil et al. (2023) Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al. 2023. Palm 2 technical report. _arXiv preprint arXiv:2305.10403_.
* Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023. Buffet: Benchmarking large language models for few-shot cross-lingual transfer. _arXiv preprint arXiv:2305.14857_.
* Cao et al. (2020) Steven Cao, Nikita Kitaev, and Dan Klein. 2020. Multilingual alignment of contextual word representations. In _International Conference on Learning Representations_.
* Chi et al. (2021) Zewen Chi, Li Dong, Furu Wei, Nan Yang, Saksham Singhal, Wenhui Wang, Xia Song, Xian-Ling Mao, Heyan Huang, and Ming Zhou. 2021. InfoXLM: An information-theoretic framework for cross-lingual language model pre-training. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3576-3588, Online. Association for Computational Linguistics.
* Conneau et al. (2018) Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. 2018. XXLI: Evaluating cross-lingual sentence representations. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2475-2485, Brussels, Belgium. Association for Computational Linguistics.
* Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. Free dolly: Introducing the world's first truly open instruction-tuned llm. _daturicks_.
* Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and effective text encoding for chinese llama and alpaca. _arXiv preprint arXiv:2304.08177_.
* Fang et al. (2020) Hongchao Fang, Sicheng Wang, Meng Zhou, Jiayuan Ding, and Pengtao Xie. 2020. Cert: Contrastive self-supervised learning for language understanding. _arXiv preprint arXiv:2005.12766_.
* Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021. SimCSE: Simple contrastive learning of sentence embeddings. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6894-6910, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Goyal et al. (2022) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaune Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzato, Francisco Guzman, and Angelia Fan. 2022. The Flores-101 evaluation benchmark for low-resource and multilingual machine translation. _Transactions of the Association for Computational Linguistics_, 10:522-538.
* He et al. (2021) Hao He, Qian Wang, Zhipeng Yu, Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2021. Synchronous interactive decoding for multilingual neural machine translation. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 35, pages 12981-12988.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training 

for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 7871-7880. External Links: Link Cited by: SS2.
* H. Li, F. Koto, M. Wu, A. Fikri Aji, and T. Baldwin (2023)Bactrian-x: a multilingual replicable instruction-following model with low-rank adaptation. arXiv preprint arXiv:2305.15011. Cited by: SS4.
* V. Heixin Liang, Y. Zhang, Y. Kwon, S. Yeung, and J. Y. Zou (2022)Mind the gap: understanding the modality gap in multi-modal contrastive representation learning. In Advances in Neural Information Processing Systems, Vol. 35, pp. 17612-17625. External Links: Link, Document Cited by: SS2.
* X. Victoria Lin, T. Mihaylov, M. Artetxe, T. Wang, S. Chen, D. Simig, M. Ott, N. Goyal, S. Bhosale, J. Du, R. Pasunuru, S. Shleifer, P. Swih Koura, V. Chaudhry, B. O'Horo, J. Wang, L. Zettlemoyer, Z. Kozareva, M. Diab, V. Stoyanov, and X. Li (2022)Few-shot learning with multilingual generative language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 9019-9052. External Links: Link, Document Cited by: SS2.
* Y. Liu, J. Gu, N. Goyal, X. Li, S. Edunov, M. Ghazvininejad, M. Lewis, and L. Zettlemoyer (2020)Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics8, pp. 726-742. External Links: Link, Document Cited by: SS2.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In International Conference on Learning Representations, Cited by: SS2.
* J. Lu, Y. Lu, and J. Zhang (2023)Take a closer look at multilinguality! improve multilingual pre-training using monolingual corpora only. In Findings of the Association for Computational Linguistics: EMNLP 2023, Singapore, pp. 2891-2907. External Links: Link, Document Cited by: SS2.
* P. Micikevicius, S. Narang, J. Alben, G. Diamos, E. Elsen, D. Garcia, B. Ginsburg, M. Houston, O. Kuchaiev, G. Venkatesh, et al. (2018)Mixed precision training. In International Conference on Learning Representations, Cited by: SS2.
* N. Muennighoff, T. Wang, L. Sutawika, A. Roberts, S. Biderman, T. Le Scao, M. Saiful Bari, S. Shen, Z. Yong, H. Schoelkopf, X. Tang, D. Radev, A. Kiri Aji, K. Almubarak, S. Albanie, Z. Alyafaei, A. Webson, E. Raff, and C. Raffel (2023)Crosslingual generalization through multitask finetuning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 15991-16111. External Links: Link, Document Cited by: SS2.
* J. Ni, G. Hernandez Abrego, N. Constant, J. Ma, K. Hall, D. Cer, and Y. Yang (2022)Sentence-t5: scalable sentence encoders from pretrained text-to-text models. In Findings of the Association for Computational Linguistics: ACL 2022, Dublin, Ireland, pp. 1864-1874. External Links: Link, Document Cited by: SS2.
* OpenAI (2022)Introducing chatgpt. OpenAI blog. External Links: Link Cited by: SS2.
* D. Carlini, C. Hang, H. Qi, A. Shah, S. Potdar, and Mo Yu (2021)Multilingual BERT post-pretraining alignment. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 210-219. External Links: Link, Document Cited by: SS2.
* L. Pan, C. Hang, A. Sil, and S. Potdar (2022)Improved text classification via contrastive adversarial training. Proceedings of the AAAI Conference on Artificial Intelligence36 (10), pp. 11130-11138. External Links: Link, Document Cited by: SS2.
* X. Pan, M. Wang, L. Wu, and L. Li (2021)Contrastive learning for many-to-many multilingual neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 244-258. External Links: Link, Document Cited by: SS2.
* E. Morria Ponti, G. Glavas, Q. Majewska, Q. Liu, I. Vulic, and A. Korhonen (2020)XCOPA: a multilingual dataset for causal commonsense reasoning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2362-2376. External Links: Link, Document Cited by: SS2.
* L. Qin, Q. Chen, T. Xie, Q. Li, J. Lou, W. Che, and M. Kan (2022)GL-CLeF: a global-local contrastive learning framework for cross-lingual spoken language understanding. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Dublin, Ireland, pp. 2677-2686. External Links: Link, Document Cited by: SS2.
* A. Radford, J. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever (2021)Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, Vol. 139 of Proceedings of Machine Learning Research, pp. 8748-8763. External Links: Link, Document Cited by: SS2.

 

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research21 (1), pp. 5485-5551. Cited by: SS3.
* L. Randaldi, G. Pucci, and A. Freitas (2023)Empowering cross-lingual abilities of instruction-tuned large language models by translation-following demonstrations. arXiv preprint arXiv:2308.14186. External Links: Link Cited by: SS1, SS2.
* J. Rasley, S. Rajbhandari, O. Ruwase, and Y. He (2020)DeepSpeed: system optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD '20, New York, NY, USA, pp. 3505-3506. External Links: ISBN 978-1-4503-387X, Document, Link Cited by: SS1, SS2.
* R. Rei, C. Stewart, A. Farinha, and A. Lavie (2020)COMET: a neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 2685-2702. External Links: Link, Document Cited by: SS1, SS2.
* N. Reimers and I. Gurevych (2019)SentenceBERT: sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3982-3992. External Links: Link, Document Cited by: SS1, SS2.
* M. Roemmele, C. Adrian Bejan, and A. S Gordon (2011)Choice of plausible alternatives: an evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, Cited by: SS1, SS2.
* T. Le Scao, A. Fan, C. Akiki, E. Pavlick, S. Ilic, D. Hesslow, R. Castagne, A. Sasha Luccioni, F. Yvon, M. Galle, et al. (2022)BIOOM: a 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100. Cited by: SS1, SS2.
* T. Sherborne, T. Hosking, and M. Lapata (2023)Optimal transport posterior alignment for cross-lingual semantic parsing. Transactions of the Association for Computational Linguistics11, pp. 1432-1450. External Links: ISSN 0018-9209, Link, Document Cited by: SS1, SS2.
* S. Soltan, S. Ananthakrishnan, J. FitzGerald, R. Gupta, W. Hamza, H. Khan, C. Peris, S. Rawls, A. Rosenbaum, A. Rumshisky, et al. (2022)Alexatm 20b: few-shot learning using a large-scale multilingual seq2seq model. arXiv preprint arXiv:2208.01448. Cited by: SS1, SS2.
* A. Tamkin, M. Brundage, J. Clark, and D. Ganguli (2021)Understanding the capabilities, limitations, and societal impact of large language models. arXiv preprint arXiv:2102.02503. Cited by: SS1, SS2.
* E. Tanwar, S. Dutta, M. Borthakur, and T. Chakraborty (2023)Multilingual LLMs are better cross-lingual in context learners with alignment. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Toronto, Canada, pp. 6292-6307. External Links: Link, Document Cited by: SS1, SS2.
* R. Taori, Ishaan Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P. Liang, and T. B. Hashimoto (2023)Stanford alpaca: an instruction-following llama model. Note: [https://github.com/tatsu-lab/stanford_lapacA](https://github.com/tatsu-lab/stanford_lapacA) Cited by: SS1, SS2.
* A. Tikhonov and M. Ryabinin (2021)It's all in the heads: using attention heads as a baseline for cross-lingual transfer in commonsense reasoning. In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, Online, pp. 3534-3546. External Links: Link, Document Cited by: SS1, SS2.
* H. Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothee Lacroix, Baptiste Roziere, Naman Goyal, Eric Hambro, Faisal Azhar, et al. (2023)LIMA: open and efficient foundation language models. arXiv preprint arXiv:2302.13971. Cited by: SS1, SS2.
* H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, P. Bhargava, S. Bhosale, et al. (2023)Llama 2: open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288. Cited by: SS1, SS2.
* A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin (2017)Attention is all you need. In Advances in Neural Information Processing Systems, Vol. 30, pp. . Cited by: SS1, SS2.
* L. Wang, W. Zhao, and J. Liu (2021)Aligning cross-lingual sentence representations with dual momentum contrast. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. . Cited by: SS1, SS2.
* Q. Wang, J. Zhang, and C. Zong (2022)Synchronous inference for multilingual neural machine translation. IEEE/ACM Transactions on Audio, Speech, and Language Processing30, pp. 1827-1839. External Links: Link, Document Cited by: SS1, SS2.
* T. Wang and P. Isola (2020)Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In Proceedings of the 37th International Conference on Machine Learning, Vol. 119 of Proceedings of Machine Learning Research, pp. 9929-9939. External Links: Link, Document Cited by: SS1, SS2.
* Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi (2023)Self-instruct: aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for 

Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wei et al. [2022] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022. Finetuned language models are zero-shot learners. In _International Conference on Learning Representations_.
* Wei et al. [2023] Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, et al. 2023. Polylm: An open source polyglot large language model. _arXiv preprint arXiv:2307.06018_.
* Wei et al. [2021] Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2021. On learning universal representations across languages. In _International Conference on Learning Representations_.
* Wu et al. [2020] Zhuofeng Wu, Sinong Wang, Jiatao Gu, Madian Khabsa, Fei Sun, and Hao Ma. 2020. Clear: Contrastive learning for sentence representation. _arXiv preprint arXiv:2012.15466_.
* Xu et al. [2021] Hu Xu, Gargi Ghosh, Po-Yao Huang, Dmytro Okhonko, Armen Aghajanyan, Florian Metze, Luke Zettlemoyer, and Christoph Feichtenhofer. 2021. VideoCLIP: Contrastive pre-training for zero-shot videotext understanding. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 6787-6800, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Xue et al. [2021]Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. mT5: A massively multilingual pre-trained text-to-text transformer. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 483-498, Online. Association for Computational Linguistics.
* Yang et al. [2021] Nan Yang, Furu Wei, Binxing Jiao, Daxing Jiang, and Linjun Yang. 2021. xMoCo: Cross momentum contrastive learning for open-domain question answering. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 6120-6129, Online. Association for Computational Linguistics.
* Yang et al. [2019] Yinfei Yang, Yuan Zhang, Chris Tar, and Jason Baldridge. 2019. PAWS-X: A cross-lingual adversarial dataset for paraphrase identification. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 3687-3692, Hong Kong, China. Association for Computational Linguistics.
* Zhang et al. [2020] Biao Zhang, Philip Williams, Ivan Titov, and Rico Sennrich. 2020. Improving massively multilingual neural machine translation and zero-shot translation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 1628-1639, Online. Association for Computational Linguistics.
* Zhang et al. [2023] Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangtong Gui, Yunji Chen, Xilin Chen, et al. 2023. Bayling: Bridging cross-lingual alignment and instruction following through interactive translation for large language models. _arXiv preprint arXiv:2306.10968_.
* Zhao et al. [2023] Yang Zhao, Jiajun Zhang, and Chengqing Zong. 2023. Transformer: A general framework from machine translation to others. _Machine Intelligence Research_, 20(4):514-538.
* Zhu et al. [2023] Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating large language models to non-english by aligning languages. _arXiv preprint arXiv:2308.04948_.

## Appendix A Hyperparameters

To align the representations and outputs of multilingual generative models, we adopt AdamW (Loshchilov and Hutter, 2019) optimizer, where \(\beta_{1}=0.9\) and \(\beta_{2}=0.999\), and a learning rate of 1e-5. The temperature \(\tau\) is set to 0.05 in the multi-lingual contrastive learning task. Mixed precision training and ZeRO are applied to speed up the training process and save memory used (Micikevicius et al., 2018; Rasley et al., 2020). The number of training steps is empirically set to 10k with a batch size of 128. All experiments are conducted on a GPU server with 8*A100 80GB RAM.

## Appendix B Additional Results

### Pooling Methods

Given representations for each token in the sentence, there are three general methods, the last token representation, max pooling and mean pooling, to obtain the representation of this sentence. Figure 6 illustrates the results of XGLM\({}_{\rm 564M}\) under different pooling methods using AFP. It can be found that the last token and mean pooling perform better, and our method is less sensitive to the pooling method chosen. Thus, these two methods are used in AFP and are selected according to the performance of the development set 



### Weight of Cross-lingual Instruction Following

We find that the weight \(\alpha\) of cross-lingual instruction following in Eq. (4) affects the multilingual performance of models. The average performance of XGLM\({}_{564\text{M}}\) on 5 datasets with different \(\alpha\) is presented in Figure 6, where models perform better than the other values evaluated when \(\alpha\) is set to 1.5. Therefore, we only consider a limited hyperparameter sweep for each multilingual generative model with \(\alpha\in\{1,1.5,2\}\).

### Distribution of multilingual representations

Figure 7 illustrates the distributions of 5 languages sentence representations from the vanilla XGLM models and the aligned ones via t-SNE. Similar to the bilingual distribution, we can find that there are distinct gaps between the sentence representations from different languages in the vanilla models (Figure 7-7). After training with AFP, the multilingual sentence distributions of models are better aligned between languages across different scales (Figure 7-7). The alignment of multilingual sentence representations in XGLM\({}_{7.5\text{B}}\) is not as good as the two smaller models, which may arise from the limited parallel samples used.

### Performance on multilingual datasets

All results of XGLM\({}_{7.5\text{B}}\), BLOOMZ\({}_{7.1\text{B}}\), and BLOOM\({}_{7.1\text{B}}\) on the 5 multilingual datasets are reported in Tabel 9-13.

## Appendix C Task Descriptions and Prompt Templates

To comprehensively evaluate our models, six datasets across four tasks are adopted in this work. Table 14 shows the statistics of all datasets used. It is noted that the original COPA dataset Roemmele et al. (2011) in English is also included in the evaluation. Most templates of prompt follow the ones in Lin et al. (2022).

Natural Language InferenceThis task aims to determine the semantic relationship between the premise and hypothesis. Table 15 illustrates the template and 3-shot example used in our evaluation for this task.

Paraphrase DetectionModels need to evaluate whether the second sentence is a paraphrase of the first sentence in this task. The template and 3-shot example adopted are reported in Table 16.

ReasoningThree popular multilingual reasoning datasets are applied in this task category. Given candidate sentences or pronouns mentioned above, models have to select the best one with semantic coherence and comply with the rules of the physics world. The detailed templates and examples are presented in Table 17 (XCOPA), Table 18 (XStoryCloze) and Table 19 (XWinogrande).

Multilingual Machine TranslationGiven sentences in the source language, models for this task have to generate the corresponding sentences in the target language. Table 20 illustrates the template and 3-shot example used in our evaluation for FLORES-101.

Figure 6: Results of different pooling methods (a) and weights of CIF (b) on 5 EN-ZH datasets using XGLM\({}_{564\text{M}}\).





\begin{table}
\begin{tabular}{l|c|c c c c c c c c c|c c c c|c} \hline \hline  & \multicolumn{6}{c}{**High**} & \multicolumn{6}{c}{**Medium**} & \multicolumn{6}{c}{**Low**} \\

- \\ XGLM\({}_{7:58}\) +APP & 0 & 54.8 & 44.6 & 41.9 & 51.3 & 48.4 & 50.6 & 48.8 & 48.8 & 47.2 & 48.6 & 47.4 & 44.7 & 44.4 & 45.3 & 42.6 & **47.5** \\  & 5 & 51.7 & 48.2 & 50.7 & 51.0 & 47.1 & 48.9 & 47.5 & 47.3 & 48.8 & 49.3 & 45.5 & 50.2 & 49.3 & 43.7 & 40.2 & **47.7** \\ \hline BLOOMZ\({}_{7:18}\) & 0 & 51.1 & 43.7 & 41.4 & 48.0 & 42.3 & 49.7 & 48.2 & 40.3 & 39.7 & 40.9 & 39.8 & 48.7 & 45.6 & 39.2 & 42.9 & 44.1 \\  & 5 & 52.0 & 45.3 & 43.2 & 50.2 & 41.6 & 48.0 & 45.9 & 41.3 & 38.0 & 37.6 & 36.1 & 47.7 & 45.4 & 39.7 & 40.1 & 43.5 \\  & 0 & 54.0 & 39.2 & 41.5 & 51.7 & 41.3 & 48.1 & 47.4 & 37.8 & 36.3 & 39.3 & 38.9 & 49.7 & 47.7 & 37.9 & 43.3 \\ BLOOM\({}_{7:18}\) & 5 & 48.7 & 43.5 & 42.8 & 50.3 & 39.1 & 47.5 & 45.7 & 40.7 & 35.1 & 37.4 & 35.0 & 48.0 & 44.1 & 38.9 & 40.4 & 42.5 \\  &
- \\ BLOOM\({}_{7:18}\) +APF & 0 & 55.0 & 42.2 & 43.8 & 52.7 & 42.4 & 48.1 & 50.0 & 40.3 & 40.4 & 42.3 & 41.6 & 50.0 & 45.3 & 42.0 & 45.1 & **45.4** \\  & 5 & 53.3 & 44.5 & 44.1 & 51.9 & 44.1 & 49.8 & 49.2 & 42.4 & 42.3 & 41.9 & 41.3 & 50.7 & 47.0 & 42.4 & 44.1 & **45.9** \\ \hline \hline \end{tabular}
\end{table}
Table 10: In-context learning results on PAWS-X across all languages. “**High**” and “**Medium**” denotes the available amount of linguistic resources. \({}^{\dagger}\) denotes the unseen language in the pre-training corpus of BLOOM.

\begin{table}
\begin{tabular}{l|c|c c c c c c c|c c c c|c} \hline \hline  & \multicolumn{6}{c}{**High**} & \multicolumn{6}{c}{**Medium**} & \multicolumn{6}{c}{**Low**} \\
**Model** & **\#shot** & EN & DE\({}^{\dagger}\) & ES & FR & KH & JA\({}^{\dagger}\) & kO\({}^{\dagger}\) & **Avg** \\ \hline XGLM\({}_{7:58}\) & 0 & 58.9 & 58.0 & 57.3 & 54.0 & 52.9 & 50.3 & 51.6 & 54.7 \\  & 5 & 56.3 & 56.1 & 56.6 & 55.8 & 55.8 & 53.3 & 52.2 & 55.1 \\  & 0 & 61.4 & 58.2 & 58.2 & 59.4 & 57.5 & 56.2 & 53.4 & **57.7** \\  & 5 & 59.4 & 59.0 & 57.4 & 58.1 & 57.6 & 56.3 & 54.9 & **57.5** \\ \hline BLOOMZ\({}_{7:18}\) & 0 & 63.6 & 57.9 & 58.5 & 57.4 & 56.9 & 55.7 & 54.8 & 57.8 \\  & 5 & 62.2 & 56.9 & 57.3 & 57.1 & 56.1 & 54.7 & 51.8 & **56.6** \\  & 0 & 59.9 & 54.7 & 57.7 & 54.0 & 53.2 & 52.0 & 50.3 & 54.5 \\  & 5 & 60.4 & 54.4 & 54.9 & 54.9 & 54.1 & 50.4 & 48.5 & 53.5 \\  & 0 & 62.4 & 59.3 & 59.6 & 59.2 & 56.2 & 56.1 & 54.2 & **58.1** \\  & 5 & 62.3 & 56.2 & 55.8 & 58.0 & 53.2 & 54.9 & 52.5 & 56.1 \\ \hline \hline \end{tabular}
\end{table}
Table 9: In-context learning results on XNLI across all languages. “**High**", “**Medium**” and “**Low**” denotes the available amount of linguistic resources. \({}^{\dagger}\) denotes the unseen language in the pre-training corpus of BLOOM.





\begin{table}
\begin{tabular}{c|r c|c c c c|c c c|c c c|c} \hline \hline  & \multicolumn{10}{c}{**High**} & \multicolumn{4}{c}{**Medium**} \\ \cline{3-12}




## Appendix D Additional Information about Language Code

Table 21 presents more information about the language codes involved in this work.

Figure 7: Distribution of multilingual sentence representations in XGLM.(Vanilla:(a)-(c), Aligned:(d)-(f), shown in t-SNE)



\begin{table}
\begin{tabular}{l c|c} \hline \hline
**Template** & **Candidate Verbalizer** \\ \hline \{Sentence 1\(\}\)_, right?_ [Label], \{Sentence 2\(\}\) & True\(\rightarrow\)Yes, False\(\rightarrow\)No \\ \multicolumn{3}{c}{_3-shot Example in English_} \\ Write anywhere , run once, _right?_ No, Write anywhere , once run
 \\ It was Easipower that said :, _right?_ Yes, It said that Easipower was ,
 \\ In 1951 , he died and retired in 1956 _., right?_ No, He died in 1951 and retired in 1956 _.
 \\ Green took over Park ’s No _., right?_ Yes, Park Green took over No . \\ \hline \hline \end{tabular}
\end{table}
Table 16: Examples of 3-shot demonstrations used in the evaluation of PAWS-X. Connectors are indicated in _italics_. The label for each example is underlined. The red text is the prediction from the model evaluated.

\begin{table}
\begin{tabular}{l l} \hline \hline
**Template** & **Candidate Verbalizer** \\ \hline \{Premise\(\}\)_, right?_ [Label], \{Hypothesis\(\}\) & Entailment\(\rightarrow\)Yes, Neural\(\rightarrow\)Also, Contradiction\(\rightarrow\)No \\ \multicolumn{3}{c}{_3-shot Example in English_} \\ We ask every nation to join us., _right?_ Also. We need at least 10 countries to join us. ¡/s> \\ One of the benefits we get of course is travel., _right?_ Yes, Traveling is one perk we get.
 \\ Serious crime down, but murders increase., _right?_ Yes, There has been a rise in murders. ¡/s> \\ So I’m not really sure why., _right?_ No, I am certain as to the reason why. \\ \hline \hline \end{tabular}
\end{table}
Table 17: Examples of 3-shot demonstrations used in the evaluation of XCOPA. Connectors are indicated in _italics_. The label for each example is underlined. The red text is the prediction from the model evaluated.

\begin{table}
\begin{tabular}{l c} \hline \hline
**Template** & **Candidate Verbalizer** \\ \hline \{Sentence 1\(\}\)_, right?_ [Label], \{Sentence 2\(\}\) & True\(\rightarrow\)Yes, False\(\rightarrow\)No \\ \multicolumn{3}{c}{_3-shot Example in English_} \\ Write anywhere , run once, _right?_ No, Write anywhere , once run
 \\ It was Easipower that said :, _right?_ Yes, It said that Easipower was ,
 \\ In 1951 , he died and retired in 1956 _., right?_ No, He died in 1951 and retired in 1956 _.
_ \\ Green took over Park ’s No _., right?_ Yes, Park Green took over No . \\ \hline \hline \end{tabular}
\end{table}
Table 15: Template and example of 3-shot demonstrations used in the evaluation of XNLI. Connectors are indicated in _italics_. The label for each example is underlined. The red text is the prediction from the model evaluated.





\begin{table}
\begin{tabular}{l c|c} \hline \hline
**Template** & **Candidate Verbalizer** \\ \hline \{Sentence 1) \{Sentence 2) \{Sentence 3) \{Sentence 4) \Label} \}  Identity \\ _3-shot Example in English_ \\ Ava started to notice wrinkles by her eyes. She bought an expensive wrinkle cream. She applied it every night. After a month she checked her eyes out carefully. She was happy to see her wrinkles were gone.
 \\ Jenny wanted to learn how to ride a horse. She went to a local horse farm. After a quick lesson, she mounted the horse. A feeling of joy enveloped her as she rode the horse around a ring. She decided to come back soon for another fun lesson.
 \\ Rick liked eating chocolate oatmeal. But his friend suggested that he use higher quality cocoa powder. Rick was tight about money. But he decided to buy more expensive cocoa powder just once. The taste was worth the price.
 \\ Gordon bought his son a remote control car for Christmas. But he realized that it needed AA batteries. Gordon could not find any. So the next day, he went to the toy store where he bought the car. He bought a big package of AA batteries. \\ \hline \hline \end{tabular}
\end{table}
Table 18: Examples of 3-shot demonstrations used in the evaluation of XStoryCloze. Connectors are indicated in _italics_. The label for each example is underlined. The red text is the prediction from the model evaluated.





\begin{table}
\begin{tabular}{c c l|c} \hline \hline
**ISO 639-1** & **Language** & **Family** \\ \hline AF & Arikkans & Indo-European \\ AR & Arabic & Afro-Asiatic \\ AZ & Azerbaijan & Turkic \\ BG\({}^{\dagger}\) & Bulgarian & Indo-European \\ BN & Bengali & Indo-European \\ CS & Czech & Indo-European \\ DE & German & Indo-European \\ EL,\({}^{\dagger}\) & Greek, Modern & Indo-European \\ EN\({}^{*}\) & English & Indo-European \\ ES & Spanish & Indo-European \\ ET & Estonian & Un trailer \\ EU\({}^{\dagger}\) & Basque & Language Isolate \\ FA & Persian & Indo-European \\ FI & Finnish & Unlike \\ FR & French & Indo-European \\ GL & Galicia & Indo-European \\ GU & Gujarati & Indo-European \\ HE & Hebrew & Afro-Asiatic \\ HI & Hindi & Indo-European \\ HR & Croatian & Indo-European \\ HT\({}^{\dagger}\) & Haitan Creole & French Creole \\ ID & Indonesian & Austronesian \\ IT & Italian & Indo-European \\ JA & Japanese & Japonic \\ KA & Georgian & Kartivalian \\ KK & Kazakh & Turkic \\ KM & Khmer & Austrosiatic \\ KO & Korean & Koreanic \\ \hline \end{tabular} 
\begin{tabular}{l l} \hline \hline
* \\ RO & Romanian & Indo-European \\ RU & Russian & Indo-European \\ SI & Sinhula & Indo-European \\ SL & Slovene & Indo-European \\ SV & Swedish & Indo-European \\ SW\({}^{*}\) & Swahili & Niger-Congo \\ TA & Tamil & Dravidian \\ TE & Telugu & Dravidian \\ TH\({}^{*}\) & Thai & Kra-Dai \\ TL & Tagalog & Austronesian \\ TR\({}^{*}\) & Turkish & Turkic \\ UK & Ukrainian & Indo-European \\ UR & Urdu & Indo-European \\ VI & Vietnamese & Australasatic \\ XH & Xhoa & Niger-Congo \\ ZH\({}^{*}\) & Chinese & Sino-Tibetan \\ \hline \end{tabular}
\end{table}
Table 20: Examples of 3-shot demonstrations used in the evaluation of FLORES-101. Connectors are indicated in _italics_. The label for each example is underlined. The red text is the prediction from the model evaluated.