Venue: NAACL 2025

# PROM: Pivoted and Regulated Optimization for

Multilingual Instruction Learning

Jaeseong Lee1, Seung-won Hwang\({}^{1*}\), Hojin Lee2, Yunju Bak2, Changmin Lee2

Footnote 1: Corresponding author

###### Abstract

Large language models (LLMs) have become standard for natural language generation tasks, with instruction-tuning enhancing their capabilities. However, the lack of instruction-tuning datasets in languages other than English limits their application to diverse languages. To address this, researchers have adapted English-centric LLMs to other languages by appending English tuning data with its translated pair. However, we observe negative interference between the two. To resolve this, our contribution is identifying English as an internal pivot language, which disentangles the use of English and target language data. Moreover, to better generalize for under-represented languages, we regulate the proposed objective. Experiments across 9 different languages demonstrate the effectiveness of our approach on multiple benchmarks. The code is publicly available for further exploration.1

Footnote 1: [https://github.com/thknightfly/PROM](https://github.com/thknightfly/PROM)

+Computer Science and Engineering, Seoul National University

Kakao Corp.

{tbvj5914,seungwohn}@snu.ac.kr

{lambda.xprime,juliet.bak,louie.m}@kakaocorp.com

## 1 Introduction

Recently, large language models (LLMs) became a de-facto standard for various natural language generation tasks OpenAI (2023); Touvron et al. (2023); Jiang et al. (2024). Moreover, careful instruction-tuning Wang et al. (2023) improves the LLMs to be more powerful.

However, due to the lack of instruction tuning datasets in other languages, most of instruction-tuned LLMs remain English-centric, hindering the application to 6500+ existing languages Austin and Sallabank (2011). Existing solutions thus propose to adapt English-centric LLMs into a monolingual target language model: Instructions in the target language are either unseen, or under-represented in pretraining, for which the existing solution translates a high-quality English instruction tuning, to pair with its translation in the target language Zhu et al. (2023); Ranaldi et al. (2023).

Despite the expected performance gain from expanding the training set, our first contribution is observing otherwise, that negative interference Conneau et al. (2020); Wang et al. (2020) exists between the original and translated pair. (Section 3.2).

To overcome this, we devise a _pivoted_ objective that disentangles English and target language data in training, to alleviate such interference. Specifically, we are inspired by a recent finding that English-centric LLMs generate in English first and then convert the output into the target language Wendler et al. (2024); Zhao et al. (2024). This implies that we can design two separate objectives, the first objective using English data for generating the representation corresponding to the English version of the next token at the middle of the layers, then another objective using target data for gradually converting into the representation for the target language.

While such disentangled objectives are effective in many languages, we find they fail to generalize well to under-represented languages, where we observe the pivoted behavior reported by Wendler et al. (2024) may not hold. To illustrate, Figure 1 contrasts language where pivoted assumption holds (left) and not (right), selected for illustration from our empirical studies reported in Appendix: Following Wendler et al. (2024), the x-axis in the

Figure 1: Language on the left shows pivoted behavior, as argued in Wendler et al. (2024). However, we find that argument does not hold in some languages (right).



figure represents layer index, from each of which, the y-axis shows the probability (according to logits) of correct target language next token (blue) or English as pivot (orange). While the left figure shows English pivot probability higher than the target token in Greek, such behavior is not observed in the right (Arabic). Inspired, we propose a regulated version, classifying between the two cases, to selectively apply pivoted objective.

Our proposed method, PROM (Pivoted and Regulated Optimization) is shown to be effective on MGSM, XQuAD, MLQA, IndicQA across 9 languages. PROM dominates the baselines in most cases, improving the QA exact match score by 50% overall. The code is publicly available.1

Footnote 2: Our observation of negative interference is consistently supported in Section 4.

## 2 Pivoted and Regulated Optimization

**Preliminaries: Adapting LLM to the Target Language** We first formalize the training of LLM architecture as follows:

\[h_{0}=f(s),s\in S \tag{1}\] \[h_{i}=L_{i}(h_{i-1}) \tag{2}\]

where \(L_{i}\) is the \(i\)th transformer layer in LLM, and \(f\) is the embedding layer, \(S\) is the set of given inputs. For instruction tuning, typically, only English instruction tuning data sample \(s_{e}\) constructs the input \(S\). The final hidden representation \(h_{N}\) is used for updating the model, where \(N\) is the total number of layers.

To enhance the set \(S\) for adaptation to the target language, we typically augment each existing English instruction and response \(s_{e}\in S\) with its translated counterpart \(s_{t}\). Moreover, an additional English to target language translation task sample \(s_{e\to t}\) can be added to further align English and the target language Zhu et al. (2023); Ranaldi et al. (2023); Kuulmetts et al. (2024).

### Motivation: Negative Interference

While 'bigger is better' is commonly believed, that adding English instruction tuning samples \(s_{e}\) along with other samples (\(s_{t},s_{e\to t}\)) to construct \(S\) is expected to be beneficial Zhu et al. (2023); Ranaldi et al. (2023), our observation in Section 3.2 indicates the contrary. To explain, we analyze negative interference between two languages, in the latter layers, especially the last layer, which is most relevant to generating the target language Wendler et al. (2024); Zhao et al. (2024).

Specifically, negative interference Wang et al. (2020) is quantified using cosine similarity between gradients from two batches composed of different languages Wang et al. (2020).2 When such similarity is low, negative interference is considered high, indicating that the gradients are conflicting and pointing in opposite directions.

Footnote 2: Our observation of negative interference is consistently supported in Section 4.

Figure 2 (blue vs. green) demonstrates that appending English data to the target language results in high negative interference, i.e., low cosine similarity between gradients from two batches. We attribute the suboptimality of appending English data (Section 3.2) to this negative interference.

Our goal is to benefit from English data while avoiding negative interference (orange line in Figure 2). The following subsection introduces how we achieve this.

### Pivoted Objective

We first disentangle the roles of English and target language data. According to Wendler et al. (2024), when generating in non-English using an English-centric LLM, English serves as a pivot language. In other words, forwarding \(h_{n}\) through the LM head for some \(n<N\) generates the English version of the next token. This implies that English data is crucial for semantics in the pivot language, while target language data is essential for generating output in the target language.

Next, we devise separate training objectives for each role. To retain semantics while utilizing English data, we design a loss function that considers English as a pivot language. Specifically, we use \(h_{n}\) passed through an LM head for instruction tuning with English data and denote the loss for this as \(\mathcal{L}_{n,e}\). Since we are not aiming for exact gen

Figure 2: Gradient similarity in the last layer. Lower gradient similarity implies higher negative interference. Ours shows low interference while utilizing both English and the translated data.

 

eration, we apply label smoothing with \(\alpha\) to \({\cal L}_{n,e}\). For language generation using target language data, we use \(h_{N}\) passed through another LM head for instruction tuning and denote the loss for this as \({\cal L}_{N,t}\). Finally, we optimize the weighted sum of the two objectives:

\[{\cal L}=\lambda\cdot{\cal L}_{n,e}+{\cal L}_{N,t} \tag{3}\]

### Regulated Objective for Under-represented Languages

While the effectiveness of the objective \({\cal L}_{n,e}\) depends on the validity of Wendler et al. (2024), recall that their assertion does not apply universally, particularly for under-represented languages in Figure 1, contrasting the scenarios following pivoted assumption (left; Greek) and not (right; Arabic). 3

Footnote 3: We translated the clozse task in Wendler et al. (2024) for this analysis. We ran in a 5-shot manner. See our results for all languages in Appendix

We propose to classify such cases by setting \(\lambda=0\) if \(\overline{P_{n,e}}<\overline{P_{n,t}}\), where -- denotes the average, \(e\) denotes English, and \(t\) denotes the target language. \(P_{n,l}\) denotes the probability of the language \(l\) version of the next token in the \(n\)th layer, following the definition by Wendler et al. (2024).

## 3 Experiments

### Experimental Settings

We use LLaMA2-7B (Touvron et al., 2023) as the representative English-centric LLM.

**Tasks and Datasets**   For the English-centric instruction tuning data, we use the Alpaca dataset (Taori et al., 2023). We use Google Translate API to obtain the target language counterpart. For the parallel data for the translation task instruction tuning, we use the WMT23 development dataset,4 the NTEX (Federmann et al., 2022) and the FLORES (Goyal et al., 2021). We only use these high-quality parallel data, since only high-quality parallel dataset guarantees the performance increase for diverse tasks (Kuulmets et al., 2024).

Footnote 4: [https://www2.statmt.org/wmt23/translation-task.html](https://www2.statmt.org/wmt23/translation-task.html)

We evaluate our model on lm-evaluation-harness Gao et al. (2021). We use the available multilingual generative tasks: MSGM (Shi et al., 2023), MLQA (Lewis et al., 2020), and XQuAD (Artetxe et al., 2020). We additionally implement IndicQA (Doddapaneni et al., 2023) evaluation. For QA evaluation, we use the extended version of lm-evaluation-harness.5

Footnote 5: [https://github.com/OpenGPTX/lm-evaluation-harness](https://github.com/OpenGPTX/lm-evaluation-harness)

Language SelectionTotal 9 languages are available in the given datasets:6Arabic (ar), Bengali (bn), Greek (el), Malayalam (ml), Marathi (mr), Swahili (sw), Tamil (ta), Telugu (te), and Thai (th).

Footnote 6: We use languages whose task performance improves by the baseline adaptation method.

Implementation DetailsTo perform instruction tuning, we largely follow the setting from Alpaca (Taori et al., 2023).7 We use learning rate of 2e-5; warmup for 3% of total steps; and train for 3 epochs. We use batch size of 32, sequence length of 1024 or 2048, depending on the GPU consumption. We use \(n=24,\alpha=0.1,\lambda=0.1\).8 Training is done on 8 A100-80GB, taking less than six hours. We evaluate the LLMs with a batch size of 8, in a zero-shot manner. We use the prompts given in the target languages. Evaluation is conducted on an A100, which takes less than two hours.

ComparisonsWe compare the following methods: a) _LLaMA2_: The baseline English-centric LLM. b) _Bacterian+(t)_: Use the target language data only (Li et al., 2023), enhanced with translation data (Kuulmets et al., 2024), i.e., \(S\) consists of \(s_{t},s_{c\to t}\). c) _xLLAMA2(t+e)_: Add english language data (Zhu et al., 2023), i.e., \(S\) consisting of \(s_{e},s_{t},s_{e\to t}\), d) _PROM_: Our proposed method.

Footnote 7: [https://github.com/tatsu-lab/stanford_alpac](https://github.com/tatsu-lab/stanford_alpac)

Footnote 8: We describe the hyperparameter choice in the Appendix.

### Experimental Results

Negative Interference Drops PerformanceThe final row of Table 3 highlights the positive impact of excluding English instruction tuning data from xLLAMA2(_t+e_). Across all 11 cases of MGSM and QA evaluation, its exclusion results in superior performance in 8 instances. This supports our claim that naively appending translated instruction tuning data incurs negative interference, thereby impairing performance.

Superiority of PROMTables 1 and 2 show that PROM successfully outperforms the baseline, xLLAMA2(_t+e_). For example, overall, the exact match score of QA increases by about 50% compared with the baseline. Additionally, as depicted in Table 3, xLLAMA2(_t+e_) never outperforms PROM, implying PROM is a reliable method for leveraging English instruction tuning data.

Importance of Pivoted ObjectiveThe third row in Table 3, identical to the removal of \({\cal L}_{n,e}\) entirely, emphasizes the beneficial nature of the proposed \({\cal L}_{n,e}\) when contrasted with the first row.

 

Importance of Regulated ObjectiveA comparison between the first and second rows in Table 3 highlights the necessity of regulation.

### Analysis

In this analysis, we show that PROM also deepens the English-pivoting behavior of the LLM. Applying PROM soars up the probability of the English-version of the next token as depicted in the right of Figure 3. This means PROM not only mitigates negative interference, but also improves the pivoting behavior -resulting in a performance increase (Table 1,2).

## 4 Related Work

Instruction-tuned LLMs for Non-EnglishTo extend the capabilities of instruction-tuned LLMs to languages other than English, early attempts involved human annotation of instruction-tuning datasets Zhang et al. (2023), which lacks scalability.

Wei et al. (2023); Li et al. (2023) leverage LLMs to generate synthetic data for instruction-tuning, however the quality would plummet as the generation ability of LLM for that language decreases than English.

Alternatively, machine-translated instruction-tune datasets Chen et al. (2023); Holmstrom and Doostmohammadi (2023); Santilli and Rodola (2023); Cui et al. (2023) paired with higher-quality English instruction-tune data Zhu et al. (2023); Ranaldi et al. (2023) gained popularity.

Our distinction is observing a possible negative interference between English and target data, and mitigating it by disentangling the roles of the two.

**English as a Pivot LanguageWendler et al. (2024) explicitly observed pivoting behavior in LLaMA2, an English-centric LLM that the LLM first generates representations for the next token in English at the middle layer before converting them to representations of the target language at the final layer. Our work is inspired by this observation but goes beyond passive observation by (1) recognizing the limitations of their findings for under-represented languages and (2) extending into optimization objectives to mitigate negative interferences.**

## 5 Conclusion

In this paper, we found that appending the English instruction sets along with its translated pairs is not always beneficial, for instruction-tuning in multiple languages. To overcome this, we proposed PROM, where we devised pivoted objective and regulated objective. Experimental results across 9 languages

\begin{table}
\begin{tabular}{l|c c c c c|c c c c c c c c|c c c c} \hline  & \multicolumn{6}{c|}{XQuAD} & \multicolumn{6}{c|}{MLQA} & \multicolumn{6}{c|}{IndicQA} & \\  & \multicolumn{3}{c}{th} & \multicolumn{3}{c}{ar} & \multicolumn{3}{c|}{el} & \multicolumn{3}{c}{ar} & \multicolumn{3}{c|}{ta} & \multicolumn{3}{c|}{mr} & ml & avg \\  & em & f1 & em & f1 & em & f1 & em & f1 & em & f1 & em & f1 & em & f1 & em & f1 & em & f1 \\ \hline PROM & **10.7** & **22.5** & 3.4* & **16.8*** & **5.3** & **22.9** & 2.5* & **16.6*** & **0.7*** & **4.0*** & **1.3** & **12.3** & **3.7*** & **13.9*** & **3.9** & **15.6** \\ xLLAMA2(\(t+e\)) & 2.4 & 14.9 & **4.2** & 16.6 & 4.1 & 20.5 & **3.1** & 16.4 & 0.3 & 3.4 & 0.3 & 11.7 & 3.3 & 13.1 & 2.5 & 13.8 \\ LLaMA2 & 1.6 & 9.8 & 0.1 & 5.2 & 1.8 & 11.4 & 1.0 & 7.1 & 0.0 & 0.7 & 0.2 & 4.3 & 0.0 & 0.8 & 0.7 & 5.6 \\ \hline \end{tabular}
\end{table}
Table 1: Exact match and F1 score of diverse QA benchmarks. (*: \(\lambda=0\) for under-represented languages.)

Figure 3: Pivoting behavior (en probability) before (left) and after (right) applying PROM.

 

show the effectiveness of our proposal.

### Limitation

We conducted our experiment on only one English-centric LLM, LLaMA2 (Touvron et al., 2023). However, we are following the convention of previous studies Zhao et al. (2024); Zhu et al. (2023); Kew et al. (2023) that focus on LLaMA for studying English-centric LLMs. We leave applying PROM to other English-centric LLMs, such as Mistral Jiang et al. (2023), as a future work.

## Acknowledgements

This research was partially supported by the MSIT (Ministry of Science and ICT), Korea, under the ITRC (Information Technology Research Center) support program (IITP-2025-2020-0-01789) supervised by the IITP (Institute for Information & Communications Technology Planning & Evaluation), MSIT/IITP grant (2022-0-00995, 2022-0-00077/RS-2022-II220077, AI Technology Development for Commonsense Extraction, Reasoning, and Inference from Heterogeneous Data).

## References

* Aretxe et al. (2020) Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. 2020. On the Cross-lingual Transferability of Monolingual Representations. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4623-4637, Online. Association for Computational Linguistics.
* Austin and Sallabank (2011) Peter K Austin and Julia Sallabank. 2011. _The Cambridge Handbook of Endangered Languages_. Cambridge University Press.
* Chen et al. (2023) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Barry Haddow, and Kenneth Heafield. 2023. Monolingual or Multilingual Instruction Tuning: Which Makes a Better Alpaca.
* Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised Cross-lingual Representation Learning at Scale. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8440-8451, Online. Association for Computational Linguistics.
* Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023. Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca.
* Doddapaneni et al. (2023) Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. 2023. Towards Leaving No Indic Language Behind: Building Monolingual Corpora, Benchmark and Models for Indic Languages. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 12402-12426, Toronto, Canada. Association for Computational Linguistics.
* News Test References for MT Evaluation of 128 Languages. In _Proceedings of the First Workshop on Scaling Up Multilingual Evaluation_, pages 21-24, Online. Association for Computational Linguistics.
* Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021. A framework for few-shot language model evaluation. Zenodo.
* Goyal et al. (2021) Naman Goyal, Cynthia Gao, Vishrav Chaudhary, Peng-Jen Chen, Guillaume Wenzek, Da Ju, Sanjana Krishnan, Marc'Aurelio Ranzzato, Francisco Guzman, and Angela Fan. 2021. The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation. _arXiv:2106.03193 [cs]_.
* Holmstrom and Doostmohammadi (2023) Oskar Holmstrom and Ehsan Doostmohammadi. 2023. Making Instruction Finetuning Accessible to NonEnglish Languages: A Case Study on Swedish Models. In _Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)_, pages 634-642, Torshavn, Faroe Islands. University of Tartu Library.
* Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mengsch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulinier, Lelong Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2023. MIST 7B.
* Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lello Renard Lavaud, Lucile Saulinier, Mariene Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Symon Antoniak, Teven Le Scao, Theophile Gervet, Thibaut Lavril, Thomas Wang, Timothee Lacroix, and William El Sayed. 2024. Mixtral of Experts.
* Kew et al. (2023) Tannon Kew, Florian Schottmann, and Rico Sennrich. 2023. Turning English-centric LLMs Into Polyglots: How Much Multilinguality Is Needed?
* Kuulmets et al. (2024) Hele-Andra Kuulmets, Taido Purason, Agnes Luhtaru, and Mark Fishel. 2024. Teaching Ilama a new language through cross-lingual knowledge transfer 

Patrick Lewis, Barlas Oguz, Ruty Rinott, Sebastian Riedel, and Holger Schwenk. 2020. MLQA: Evaluating Cross-lingual Extractive Question Answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7315-7330, Online. Association for Computational Linguistics.
* Li et al. (2023) Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptation.
* OpenAI (2023) OpenAI. 2023. GPT-4 Technical Report.
* Ranaldi et al. (2023) Leonardo Ranaldi, Giulia Pucci, and Andre Freitas. 2023. Empowering Cross-lingual Abilities of Instruction-tuned Large Language Models by Translation-following demonstrations.
* Santilli and Rodola (2023) Andrea Santilli and Emanuele Rodola. 2023. Camoscio: An Italian Instruction-tuned LLaMA.
* Shi et al. (2023) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2023. Language models are multilingual chain-of-thought reasoners. In _The Eleventh International Conference on Learning Representations_.
* Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsuneri B. Hashimoto. 2023. Stanford alpaca: An instruction-following LLaMA model.
* Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almhaiiri, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esibou, Jude Fernandes, Jeremy Fu, Wenvin Fu, Brian Fuller, Cynthia Gao, Vedantuj Goswami, Namam Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inanc, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Likovich, Yinghai Lu, Yunqing Mao, Xavier Martinez, Todor Mihalov, Pushkar Mishra, Igor Molbyog, Yixin Nie, Andrew Poulton, Jeremy Rezenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Purxin Xu, Zheng Yan, Iilyau Zarov, Yuchen Zhang, Angela Fan, Melanie Kambandatur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Sciamon. 2023. Llama 2: Open Foundation and Fine-Tuned Chat Models.
* Wang et al. (2023) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-Instruct: Aligning Language Models with Self-Generated Instructions. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 13484-13508, Toronto, Canada. Association for Computational Linguistics.
* Wang et al. (2020) Zirui Wang, Zachary C. Lipton, and Yulia Tsvetkov. 2020. On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 4438-4450, Online. Association for Computational Linguistics.
* Wei et al. (2023) Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, Tianxiang Hu, Shangjie Li, Biyuan Hui, Bowen Yu, Dayijheng Liu, Baosong Yang, Fei Huang, and Jun Xie. 2023. PolyLM: An Open Source Polylgot Large Language Model.
* Wendler et al. (2024) Chris Wendler, Venimim Veselovsky, Giovanni Monea, and Robert West. 2024. Do Llamas Work in English? On the Latent Language of Multilingual Transformers.
* Zhang et al. (2023) Shaolei Zhang, Qingkai Fang, Zhuocheng Zhang, Zhengrui Ma, Yan Zhou, Langlin Huang, Mengyu Bu, Shangntong Gui, Yunji Chen, Xilin Chen, and Yang Feng. 2023. BayLing: Bridging Cross-lingual Alignment and Instruction Following through Interactive Translation for Large Language Models.
* Zhao et al. (2024) Yiran Zhao, Wenzuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024. How do Large Language Models Handle Multilingualism?
* Zhu et al. (2023) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023. Extrapolating Large Language Models to Non-English by Aligning Languages.

 



## Appendix A Appendix

### Full Results for Figure 1

Figure 4 reports our results for nine languages, with and without pivoted behaviors.

### The Choice of Hyperparameters

We tuned \(N,\alpha,\lambda\) on thai language as Table 4. Only our setting is on par or outperform the best baseline, Bactrian+(\(t\)). Note that removing the thai columns from Table 1, 2 does not change the trend or analysis.