Language Imbalance Driven Rewarding for Multilingual Self-improving
Authors: Wen Yang
Junhong Wu
Chen Wang
Chengqing Zong
Jiajun Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Language Model
Self-Improving
Multilinguality
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper proposes Language Imbalance Driven Rewarding, which leverages the inherent imbalance in LLMs as a reward signal to bootstrap LLMs’ multilingual capabilities in a self-improving manner.

Abstract: Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46\% win rate on the X-AlpacaEval leaderboard and 13.9\% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs.

Published as a conference paper at ICLR 2025
LANGUAGE IMBALANCE DRIVEN REWARDING FOR MULTILINGUAL SELF-IMPROVING
Wen Yang Junhong Wu ; Chen Wang! Chengging , Jiajun Zhang
School of Artificial Intelligence, University of Chinese Academy of Sciences
2 Institute of Automation, Chinese Academy of Sciences
3 Wuhan AI Research Shanghai Artificial Intelligence Laboratory, Shanghai, China
[yangwen2023 wu wu.junhong2021, wangchen2020 ac. en
[cazong, Bizhang)@n1pp) www.r.ia..c....mm cn
ABSTRACT
Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly ben- efited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportu- nity to bootstrap the multilingual capabilities of LLM in self-improving manner. Thus, we propose Language Imbalance Driven Rewarding, where the inherent imbalance between dominant and non-dominant languages within LLMs lever- aged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's s capacity, thereby yielding iterative reward signal. Fine-tuning Meta-I --Stssntrtuttt over two iterations of this approach re- sults in continuous improvements in multilingual performance across instruction- following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46% win rate the X-AlpacaBval leaderboard and 13.9% accuracy the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvemment of LLMs. The code is available at hhtpp:: t/u. Homm/NNPP/aa com///NiPP/anguagee- Imba ance Driven: Reen rdin
INTRODUCTION
Large Language Models (LLMs) have revolutionized the field of Natural Language Processing (NLP) with superior performance across numerous tasks. However, existing studies show that due to the imbalance of pre-training and fine-tuning data across languages, existing LLMs have predominately benefited a few "first-class" languages, particularly English and Chinese, thereby overlooking wide range of other languages (Qin et al. 2024). Given that LLMs are used world- wide, such language imbalance presents significant risks for users who operate in less dominant languages (Deshpande al. 2023). To this end, enhancing the multilingual performance of LLMs has gained increasing attention.
Previous research predominantly frames this imbalance as an issue to be resolved, often addressing it through multilingual training and lingul alignment. The first approach aims to improve multilingual performance by incorporating additional multilingual data (Wei et al. 2023 Dang et al. 2024). However, high-quality multilingual instruction tuning and preference data, particularly for low-resource languages, remain scarce and expensive (Boubdir al.) 2023 Chaudhari et al. 2024). The second approach seeks to bridge the performance gap between languages by aligning non-
dominant and dominant ones (Li al. 2023a Chen al. 2023b Chai al.) 2024 Zhang al.
2024). which are often bottlenecked by the performance of the dominant language.
Equal contribution
Corresponding author

---

Published as a conference paper at ICLR 2025
This work takes different perspective, positing that language imbalance, while still issue, cre- ates a natural preference ranking between dominant and non-dominant languages, which can be leveraged as reward signal. As the preference ranking is mutual, the reward signal benefits both dominant and non-dominant languages, enabling their simultaneous optimization. Consequently, reliance on human-authored data eliminated, and the performance ceiling for dominant languages is surpassed.
We thus introduce Language Imbalance Driven Rewarding, which leverages the reward gener- ated from inherent language imbalance to enhance the multilingual capabilities of LLM in a self- improving manner. Specifically. our approach adopts an Iterative Direct Preference Optimization (DPO) (Rafailov et al., (024) similar to previous works (Yuan et al. (024). As shown in Figure starting from any instruction model with basic multilingual capabilities, responses are generated by the model for multilingual prompts and are then mutually translated by that same model. This translation process largely preserves the original preference rankings yielding from language im- balance (discussed in Section 3.2). allowing for the construction of preference dataset where sponses the dominant language are treated as preferred and those in the non-dominant language rejected. Subsequently,
approach employs variant of the DPO that incorporates negative log-likelihood (NLL) loss term for the chosen labels and has been demonstrated to be crucial for per- formance in Pang al. (2024). The DPO training is executed both dominant and non-dominant languages, enhancing their performance simultaneously. The model trained with DPO is capable of continuously providing reward signals in proceeding iterations.
Self Multilingual Preference Pair Generation
Multilingual Preference Optimization
Generate
Translate
Muttilingual preference pairs
Muirilingual prompts
D3
y4
y/3
{a;
en
Vi on
9 :
von
DPOANLI Training
lang
:
M2
M
M
de
d
-de
y!
y
{a yi
ruy
z r
en
Next Iteration Model
Figure 1: Language Imbalance Driven Rewarding. Our method consists of two steps: (i) Self multilingual preference pair generation: Multilingual prompts are used to generate multilingual responses from MM. respectively. Then, M, is utilized to perform mutual translations between re- sponses dominant language (c.g., en) and non-dominant languages (c.g., es, de, ru). Finally, the inherent language imbalance in LLMSs is leveraged to construct multilingual preference pairs. (ii) Multilingual preference optimization: Multilingual preference pairs are constructed by M, itself, which are used for training via a DPO+NLL objective, resulting in model M,+1. The whole process iteratively repeated, enhancing the model's multilingual abilities across all languages in each sub- sequent iteration, until optimization saturates.
In experiments, begin with Reta--llama-3-88 SB-Instruc (Meta 2024) as the seed model and perform two rounds of iteration. Results demonstrate that multilingual preference optimization not only significantly enhances the wottuncttooooolowwiiin abilities of non-dominant languages compared to the seed model but also improves the performance of the dominant language. This means that during training, the model is not constrained by the initial performance of the dominant language, which is crucial for iterative self-improvemen ually saturate as the performance gap between languages narrows, it presents an intriguing oppor- tunity to bootstrap the multilingual performance of LLMs across all languages without the need for authn-authored datasets.
within our approach. Although this effect will grad-
LANGUAGE IMBALANCE DRIVEN REWARDING
Our approach first assumes access to an instruction-followin language model, and a set of mul- tilingual training prompts. Starting from any instruction model that possesses basic multilingual generation capabilities, each iteration consists of two steps, (i) Self multilingual preference pair generation and (ii) Multilingual preference optimization, shown in Figure 1. For the th iteration,
2

---

Published as a conference paper at ICLR 2025
the current model Mr, with the seed model denoted as My. Step (i) generates multilingual preference pairs data for DPO training in step (ii). After training, the updated model M,+1 is utilized as the initial weight in the next iteration.
Initialization Given an instruction-tuned model Mg and a set of parallel multilingual instruction prompts X, where X' includes both the dominant language and non-dominant languages participat- ing in the self-improving process, the model is updated iteratively, resulting in a sequence of models M1, My,..., Mr.
Self' Multilingual Preference Pair Generation The current model Mr generates corresponding responses vf Ma(a.) for the instruction zt in any language supported by the model.
Mi(2.) for all if X
Let di and nl represent any dominant and non-dominant language supported by the model, respec- tively. After generating the corresponding responses, we utilize the self-translation capability of LLM to facilitate translation between dominant and non-dominant language responses, the self- translation prompt is given in Appendix H.4 Specifically, for the dominant language response vta Mj is utilized to translate it into any non-dominant language nl, resulting in y Similarly, we randomly select a response in any non-dominant language ná for each prompt, denoted as y, and translate it into the dominant language, resulting in gddl+
Due to the inherent differences in the multilingual capabilities of the model itself, and translation does not alter language biases. The following preference ranking holds true the dominant lan- guage dl and any non-dominant language nd supported by the model:
For the same instruction in dominant language
Vi u al
Vi
For the same instruction in non-dominant language ar!a
- > y/"
(3)
Thus, the preference ranking between the dominant language and non-dominant languages is utilized to construct multilingual preference pair dataset in all languages supported by the model.
Da {x,² 3// Vb -d- N i==1
(4)
D {2771 2// dn ml i=1
(5)
The Dai and D'd combined form the multilingual preference pair dataset, which is denoted as D = (Da, }.
Multilingual Preference Optimization Given multilingual preference pair far, Bwiin Moose from D, a variant of DPO is employed to maximize the probability of the chosen output ywen and minimize that of the undesirable output Ylose. Specifically, negative log-likelihood (NLL) loss term for the chosen labels is incorporated into the vanilla DPO (Rafailov et al., 2024) formulation to improve alignment performance. The optimization objective is formulated as:
Ma (yuwin (r) Blog Me Mylossel (x) }
Ma MMRoee (x)
LDPo = - -E (e.wwan Vwnn Miose Ma log (Blog
(6)
M1 (V
log Mo (ywin [x)]
LNCL B(z.eenn)
|uwil
Overall,
== LDPO + aLNcc
(8)
Where M-Ix)x) is the policy model to be optimized, Me(-(zr) (r) is the reference model kept unchanged during training. The parameters 0 are initialized from model M,, a is the sigmoid function. Note that the Lvce term is normalized by the response length, while DPO loss is not.
After the DPO training, our next model is updated as M = My. which will be utilized to construct new multilingual preference pairs data for the next iteration.
3

---

Published as a conference paper at ICLR 2025
Iterative Training Our overall procedure starts from cnstruction-folloiinn model Mu and in- struction prompts X. training a series of models M1. My, .... Mr. The models and corresponding training data used are defined follows: (1) Mp; Base LLM; mnstruction-followwe
model. (2) Mi: Initialized with Ma, using Mo and X' to generate Do. then conduct multilingual preference
optimization on the Do- (3) M2: Initialized with M1, using M1 and X' to generate D1, then conduct multilingual preference optimization on the D1.
3 DISCUSSION
The insight behind our proposed method is to leverage the inherent differences the multilingual capabilities of LLMs provide rewards for DPO training. Therefore, two key questions remain to be addressed:
3.1 Do LLMS EXIIBIT SIGNIFICANT DIFFERENCES IN MULTILINGUAL CAPABILITIES?
While the differences in multilingual capabilities have been evidenced by many prior works (Ranaldi) & Pucci 2023 Yuan et al. [2023] Zhao et al. (024), we further validate the disparity in multilingual capabilities in Llama-3.
The average quality of responses across different languages for parallel multilingual in- structions. Note that Llama-3-8B- --8B-Instruct is subject to the off-target issue in certain languages (e.g., ja and ru).
Table 1:
GPT-4 Score (0-10)
Model
en
es
fr
it
de
Llama-3-8B-Instruct
9.60
8.34
6.43
6.66
4.69
0.76
2.21
Specifically, we randomly selected 100 multilingual Alpagasus (Chen al.) (2023)) instructions and evaluated the response quality across different languages using GPT-4 score. Based on the tech- nical report for LLaMA 3 (Metal 2024), English is selected as the dominant language, while the other languages considered non non-dominam languages. As shown in Table [ significant differ- ence in response quality remains between the dominant language (en) and non-dominant languages, demonstrating inherent imbalance exists in the multilingual capabilities within the model.
3.2 DOES TRANSLATION PRESERVE THE RANKING OF RESPONSE PREFERENCES?
As shown in Discussion 3.1 Given an English prompt x and non-dominant language prompt a! model M consistently produces a better response for the English prompt: M (yf""(z"" (z(") >
M (x))) However, tnff-translation is employed by our method to convert the English response ofth into non-dominant language ni, and vice versa. Therefore, key question arises: Do the trans- lated responses preserve the preference ranking in Equation and
As translation will largely preserve the semantics and the structure of the sentence, it is reason- able to believe that the preference ranking stemming from the quality difference of the response is largely preserved. To verify our assumption, the GPT-4 score is utilized to assess the quality of self-translated responses and compare it with original responses. As shown in Table 2 the GPT-4 score of the translated response dd-- is lower than the original response yd However, a sub- stantial gap remains between the translated response and the original response in non-dominant languages, which is consistent with the original preference ranking. This conclusion also holds for the dominant language (English), where the original response is superior to the response sampled from non-dominant languages and self-translated into English. Overall, the self-translation process does preserve the ranking of response preference.
To observe the final preference ranking, multilingual preference pairs are constructed between the original and translated responses, sampling 100 pairs from each language. Reward accuracy (Win Rate) was then assessed through head-to-head comparisons by GPT-4. As shown Table 3 lan- guage imbalance provides a positive reward (>0.50). Moreover, the strength of reward signals varies across languages, ranging from 0.57 in es to 0.79 ja. In line with the GPT-4 score Table [ this indicates that languages with weaker performance in LLMss tend to exhibit stronger reward signals.
4

---

Published as a conference paper at ICLR 2025
Table The average quality of self-translated responses. We selected the same responses discussed in Section 3.1 5.Iland and assessed the self-translate quality using GPT-4 score. The underlined scores rep- resent the sell-translation responses sampled from other languages into English (y (ym(=en), while the bold scores indicate the self-translation of English responses into other languages (y
GPT-4 Score (0-10)
Type
fr
it
de
ja
Self Generation
9.60
8.34
6,43
6.66
4.69
0.76
2.21
Self Translation
8.03
9.32
9.17
8.72
8.96
7.89
7.75
Based empirical observations, languages classified with rewards of 0.60 the threshold into low-reward ones (es, ft; it) and high-reward ones (de, ja, ru).
Table 3: The reward accuracy of multilingual preference pairs.
Reward Accuracy (0-1)
Model
fr
it
de
ja
1lama-3-8-IIntruus 0.72
0.57
0.60
0.57
0,70
0.79
0.74
4 GENERAL INSTRUCTION FOLLOWING
4.1 EXPERIMENTAL SETUP
Base Models In our experiments, we use a widely adopted instruction-follloinng model as our base model Mo, namely Lamm----B-IInsreeee (Meta 2024). Bamm-3--8B-Instmeee an English-centric LLM, often encounters off-target issues when handling non-English requests.
Languages English is chosen the common dominant language, and non-dominant languages in- clude high-reward languages (German, Russian) and low-reward languages (Spanish, French). Ad- ditionally, Chinese selected as unseen language observe the generalization of our approach. Note that unseen language means that not included in the training data.
Datasets The Alpagasus dataset (Chen et al. 2023a) includes 9,000 high-quality instruction- following examples filtered from the original 52,000 in the Alpaca dataset (Taori et al. (202)). We sample 1,000 prompts from the Alpagasus dataset and translate them into other languages using the Google Translate API to obtain multilingual prompts.
Implementation Details Models are trained for one epoch in each iteration across all experiments.
More details described Appendix F.4
Evaluation and Metries
Head-to-head performance: Hcad-to-head performance is evaluated between base model and the iterative models using GPT-4 Turbo as an evaluator (Liu al. (023) over 805 test prompts in X-AlpacaEval (Zhang et al. 2023) The detailed setup can be found in Appendix F.I
X-AlpacaEval leaderboard: We extend the existing AlpacaEval 2.0 toolkit (Li et al. 2023d) from an English-only framework to multilingual one and compare both propri- etary and open-source models on their multilingual caatuctioo-followinn abilities.
Multilingual MT-Bench: Results additionally reported multilingual MT-Bench. MT-Bench (Zheng et al. 2024a) consists of a series of open-ended questions that evaluate the multi-turn conversational and instuctiionfoollwiing abilities, which GPT-4 Turbo to grade the model responses on a scale of 10.
Multilingual NLP benchmarks: To assess the alignment tax of our method, we further evaluate the performance of our model on multilingual versions of the MMLJ (Hendrycks)
5

---

Published as a conference paper at ICLR 2025
et al. 2020), HellaSwag (Zellers et al. 2019), ARC Challenge (Clark al. 2018 and TruthfulQA (Lin et al. 2021) benchmarks.
Fair Evaluation Appendix D.I explains how to prevent language bias in ALM-as-a-Judge and D.2 highlights GPT--4's multilingual judging capabilities, aligning with the advanced GPT-4o. D.3 dis- cusses avoiding translationese bias in evaluation.
4.2
HEAD-TO- HEAD PERFORMANCE
The head-to-head win rates of our models on the X-AlpacaEval dataset are shown in Figure
Wini
Tie
Lowe
English
Spanish
Russian
M2 vs K0
M2 vs MD
M2 vs MD
MO
30.1
15.2
14.8
M1 MD
412
41.0
17.8
M1 MO
70.9
186 9.4
vs M1
M2 M1
M2
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
Chinese
M2 vs M/
70.4
17a 13.8
M2 vs MD
27.6
16.5
M2 vs MD
54.7
MO
69.2
11.6
37.4
57.4
M2 vs M1
25.1
M2 vs M1
20.2
M2 vs M1
35.7
40
00
20
40
60
100
60
80
100
Figure 2: Multilingual Instruction following ability improves with Language Imbalance Driven
Rewarding on Alama-3-8B-Instnut model.
Findings 1: Language Imbalance Driven Rewarding is effective. The head-to-head perfor- mance shows that Mj achieves notable win rates against Mo across all five training languages and one unseen language. For these five training languages, M2 demonstrates a significant improvement, with W-I values for each language ranging from 15.3% (en) to 61.5% (ru) compared to the base model. Upon comparing different languages, high-reward languages (rus de) gain larger improve- ments than low-reward languages (es, fr) in Iteration (M, vs. Mo), but in Iteration (M2 vs. Mi). the gains across all languages diminish and converge. We hypothesize that the reward signals across different languages gradually weaken and align over iterations.
The dominant language also benefits from Language Imbalance Driven Reward- ing. Since the responses for non-dominant languages are translated from English, it is natural for these languages
Findings 2:
see improvements. However, English also achieves better performance com- pared to the reference model. As shown in Figure 2 English shows a 15.3% increase in AW-L for Mi vs. Mo. and 14.6% increase in AW- for M2 vs. Mi. These results indicate that the dominant language also benefits from rejected responses translated from non non-dominant languages, highlighting the value of incorporating negative samples in preference pair construction, aligning with observation in Duan et al. (2024).
Findings Iterative training possible and effective. Findings reveals that English benefits from language imbalance driven rewarding, which lays the foundation for iterative optimization. Specifically, the enhancement English makes possible to generate higher-quality responses in the next iteration of training, enabling continual self-improving. In Figure [ consistent gain is observed Iteration (M1 vs. Mo) and Iteration 2 (M₂ vs. Mi) in all languages. Compared to Iteration I, the gains for all training languages (except for English) in Iteration 2 become more consistent and convergent. This demonstrates that our approach are capable of iteratively aligning all languages until reaching saturation.
Findings 4: Multilingual optimization can generalize to unseen languages. For the unseen language, the gains (A W-L) in Chinese are 32.2% for Iteration 1 (M, vs. Ma) and 22.4% for Iter-

---

Published as a conference paper at ICLR 2025
ation (M2 Mi), respectively. These results indicate that multilingual preference optimization facilitates tross-lingual transfer, which is consistent with observations in Dang et al. (0224).
Table 4: The X-AlpacaEval Leaderboard, which shows the win rate over GPT-4 Turbo evaluated by GPT-4.
Win Rate
Model
Avg
en
es
ru
de
fr
Language Imbalance Driven Rewarding
Meta-I.lama-3-81 ---aam------ssruueee (MO)
24.90%
18.08%
7.81%
8.65%
14.18%
14.72%
Iteration (M1)
30.11%
21.82%
18.01%
16.87%
17.51%
20.86%
Iteration 2 (M2)
34.09%
21.21%
19.25%
16.02%
20.34%
22.18%
Multillinuual Alignment
Meta-Llama-3 --ttstttuttttt (SFT)
21.88%
18.98%
15.90%
16.68%
18.15%
18.32%
SOTA Multilingual Models
GPT-4o-mini
45.17%
44.63%
47.03%
44.2%
44.93%
45.19%
GPT-4-0613
15.61 %
18.18%
16.82%
16.00%
15.23%
16.37%
GPT-3.5-turho0025
11.96%
14.42%
13.74%
12.41%
12.70%
13.05%
Qwwn2-72B-instruet
37.72%
24.73%
27.15%
23.93%
24.63%
27.63%
39.74%
32.58%
9.14%
9.48%
25.20%
23.23%
Internl Necmm--M..5Chht-2200
31.77%
16.62%
11.10%
11.56%
13.61 %
16.93%
22.15%
20.63%
12.02%
16.05%
18.55%
17.88%
Meta-Llama-2213BB-Be
8.84%
5.31%
0.93%
1.19%
1.36%
3.53%
PolyLM- Chat-13B
3,81%
3.61%
2.27%
2.79%
3.56%
3.21%
Aya-23-8B
15.26%
16.68%
17.95%
18.50%
14.70%
16.62%
Owen2-7B-Instruce
24.39%
13.89%
14.33%
11.45%
15.97%
16.01
Mistral-78-Innstuut-00.33
21.46%
13.36%
13.75%
11.91%
13.28%
14.75%
4.3 X-ALPACAEVAL LEADERBOARD
The X-AlpacaEval leaderboard, shown in Table demonstrates high degree of consistency with head-to-head evaluations. After two rounds of iteration, Llama-3-8B-Insrruet achieves average improvements of 7,46% in win rates GPT-4 Turbo across five languages. Additionally, eval- uate the performance of sate-offthheart multilingual models on X-AlpacaEval, including OpenAl's
GPT-4o, GPT-4 (Achiam et al. 2023). along with Qwen series (Bai al. [2023 Yang al. 2024) Llama series (Touvron et al. 2023b; Meta 2024). InternL.22 (Cai et al.) 2024). Aya-23 (Ustün et al. 2024). Mistral (Jiang et al., 2023) and PolyLM (Wei et al., (023). Our method based on Llama-3- 8B-Instruct outperforms both 7B and 14B-level models, achieving comparable performance to the 70B-level models.
Moreover, a comparative experiment multilingual alignment is conducted, which performed supervised fine-tuning by self-translating model responses from the dominant language to non- dominant languages under the same experimental conditions. Multilingual alignment utilizes the performance of the dominant language as an anchor to align the capabilities between dominant and non-dominant languages. While there is improvement in performance guages, a significant gap remains compared to our method, with 18.32% vs. 22.18% Llama3. Notably, multilingual alignment places excessive focus non-dominant languages during the SFT process, resulting in degradation of performance in English (-3.02%). In contrast, our approach improves English performance (49.19%). This improvement is crucial for enabling iteration in our method, whereas the performance decline in English seen with multilingual alignment hinders fur- ther iteration.
non-dominant lan-
4.4 PERFORMANCE ON MULTILINGUAL MT-BENCH
reports the multilingual MT- results on a scale of score 10. A significant performance improvement MT-Bench in Llama3 is observed across the training iterations, increasing from

---

Published as a conference paper at ICLR 2025
6.80 in My to 7.51 in M2. This is because Llama3 initially exhibits strong reward signal in Mo;
however, this signal weakens as iterations progress. A detailed analysis is provided in Section 4.6
Table 5: The Multilingual MT-Bench Benchmark.
Training Languages
Unseen zh
Model
Avg
de
fr
Meta-LLama-33 3- 3B-IInstrcc (MO)
8,20
7.51
5.86
6.36
7.21
5.64
6,80
Iteration (MI)
8.22
7.55
7.12
7.46
7.56
5.87
7.30
Iteration (M2)
8.30
7.59
7.37
7.62
7.93
6.22
7.51
4.5 ALIGNMENT TAX ON MULTILINGUAL NLP BENCHMARKS
Previous studics have shown that instruction tuning and RLHF can lead to forgetting, also known the alignment tax (Ouyang et al. 2022). The changes in world knowledge and commonsense reasoning abilities are examined throughout the iterative process by evaluating its performance on multilingual NLP benchmarks.
Table O presents the average results across five training languages (English, Spanish, Russian, Ger- man, French) and one unseen language (Chinese) four benchmarks, with detailed results provided in the Appendix G.T Overall, during the iteration process, the performance on the bench- marks only exhibits no significant degradation compared the base models but also shows slight improvement. These results indicate that the multilingual preference optimization process did not introduce any alignment tax.
Table 6: The Multilingual NLP Benchmarks.
Multilingual
Multilingual HellaSwag
Multilingual
Mulrilingual TruthfilQA MCI
Model
MMLU
ARC challenge
MC2
Lota-Llama-3-88 3-88-Inntruct (MO) 0.5666110003
0..024=00.051
0.4228 20.0144
0.3417
0.5076
Iteration (M1)
0.56872200043
=0.005)
0.4312:20014
0.3464
0.5169 10.0157
Iteration 2 (M2)
0.5687+4.004
0,4763
0,4321 +0.0144
0.3472
0.5165 +0.017
4.6 THE REWARD SIGNAL CHANGES OVER ITERATIONS, GETTING STRONGER OR WEAKER?
The reward signal strength on pairwise data was analyzed at the beginning of each itera- tion, as outlined in Section 3.2 Figure FigureBssows the change in reward accuracy training lan- guages (en, es, ru, de, fr) and unseen lan- guages (it, ja) across iterations. For the train- ing languages, high-reward languages, except English, gradually shift to lower-reward status after Iteration 1. As English capabilities im- prove through iterations, low-reward languages remain in the lower-reward range with some fluctuations, enabling the self-improving pro- cess to continue iteratively.
0.8
MI
0.6
Reward Acc
0.4
0.0
de
Languages
Figure 3: The reward accuracy over iterations.
For unseen languages, reward accuracy also steadily increases (it) as English capabilities improve continuously. However, during the DPO train- ing, certain preferences, such as controlling off tarett responses, are transferred to unseen languages (ja). This is evident from the sharp drop in Japanese reward accuracy after Iteration 1, which corre- sponds reduction in off-target responses.
4.7 SCALING AND GENERALIZING
In Appendix C we scale our method to Qwen2-7B-Be (oen2-BB-Instruee (Yang et al. 2024). In Appendix E| we extend our approach extreme scenarios: using weaker base model, Llama2-7B-Chan (Touvron
8

---

Published as a conference paper at ICLR 2025
et al. 2023b in E.1 addressing lower-resouree languages (bn, sw, th) in E.2 and relaxing the self- improvement paradigm through the use of an external translation system in
5 ARITHMETIC REASONING
Arithmetic reasoning is task where language models often struggle (Ahn et al. 2024), and while they are considered language-agnostec (Brannon 2005), existing LLMs demonstrate inconsistent reasoning capabilities across different languages. We scale our method to arithmetic reasoning to enhance reasoning performance across languages.
5.1 EXPERIMENTAL. SETUP
Experiments Settings The arithmetic reasoning task is conducted on Llama-3-8P 1-8B-Instruct, start- ing with multilingual GSM8K (Cobbe et al., 202T) prompts. Performance was measured using the MGSM benchmark (Shi et al. 2022), which consists of 250 manually translated GSM8K problems ten languages. We report reasoning accuracy (Acc) across five training and five unseen languages, and assess the off-target rate (Off-tag) of the reasoning responses using the LangDetect library. The implementation details are described in Appendix F22
Compared Methods We first compared approach to multilingual alignment, where English responses were self-translated into other languages for SFT training. Additionally, we focus on comparing reasoning task performance with MAPO (She et al., 2024). To ensure fair comparison with MAPO, we considered two variants: MAPO uses the same sampling count as our method, while MAPO* f uses MAPO's sampling configuration but with training data size consistent with ours. We used MAPO's official code and hyperparameters for sampling and trained all preference pairs under identical training conditions. We report MAPO's best results, achieved after two iterations.
Table 7: Model performances on MGSM benchmark on LLama-3-88 --aaa--------nnttt base model. The subscript values represent the relative change in performance compared to the base model Mo for each language. Improvements are indicated green, and declines red.
ft
Training Languages
Acc(1)
Off-tug ()ce(t))
Off- (Ace(h)
OfT-tag Aex(())
Off-tug( Ave(t)
OfT-tag(
00.456
0.012
0..488
0.076
Multilinuual Aligument
0.680
0.604
0
0.592
0
0.552
0.008
0.540
MAPO!
0.668
0.600
0
0.608
0.012
0.560
0.028
0.524
0.004
MAPO#
0.716 11.5%
0.668 112.
0
0.620
1328
0.036
0.508
14.0%
0.028
0.592 11289
0.02
0.712 00
0.616 4s
0.004
0.596 +13
0.720 1200
00 00.640 13.00
0.620
13.09
0.004
0.570
102e
0.6088 14.00
ja
Unseen Languuges
Acc(1)
Off- Acelt))
Ofr-tag( Ace(())
OfT-tagi Aeet(()
Off-tag( Ace(t)
OII-tag(
MO
0.284
0,280
0.192
0.204
0.324
0.124
0.464
0.336
0.328
0.024
Muitilngual All Allgument
0.356 27.55
0.020
0.216 42.5
0.032
0.436 -11.5
0.004
0.520
.5.0
0.124
0.308 -2.0%
MAPO!
0.348 -6
0.292 Joam
0.044
0.508 -18.
0.040
0.172
a.rs
MAPO
0,384 +00.0
0.072
0.308
0.016
0.472
+1.39
0.084
0.540
27.0
0.104
+500
0.004
MI
0.428
0.008
0.320
0.016
0.524
0
0.552
0.076
0.464
13399
M2
0.476 110.75
0.004
0.328 11.6
0.002
0.536
.21.0
0
0.592
0.064
0.472
5.2
RESULTS
The results are presented in Table 7 In training languages, our approach outperforms multilingual alignment, with M2 improving by T1.6% on average, compared to 7.8% for multilingual alignment. While multilingual alignment struggles to balance English and non-dominant languages, leading to a decline in English performance (-2.0%). In unseen languages, our method (M1) achieves 16.2% average improvement, exceeding the 11.6% gain observed in training languages. This suggests our approach effectively leverages language imbalance to learn language-agnoss to superior generalization. In contrast, traditional multilingual alignment tends to overemphasize training languages to align English capabilities, resulting in much poorer generalization on unseen languages compared to our method (average 4.9% 16.2%)
reasoning, leading

---

Published as a conference paper at ICLR 2025
With the same sampling effort, MAPOř underperforms across all languages, likely due to its lim- ited preference data. This highlights the efficiency our method in directly leveraging language imbalance. Using the same training data, MAPO also performs worse than our approach in most languages, further demonstrating the superior effectiveness of our reward mechanism.
6 RELATED WORK
LLM Self-Improving The goal of LLM self-improvement is to enhance its capability by leverag- ing the knowledge embedded within the model itself. Self-improvement two categories: self-syuthetic and self-critical. Self-syuthetic involves generating synthetic training data using the model itself. For example. Self-Instruct (Bai et al. 2022 Wang et al. 2022) is tech- nique for generating prompts and responses independently. which can be utilized to enhance a base language model. Instruction backtranslation (Li et 2023c) similarly augments and curates train- ing data by augmenting it through back-translation from web documents to generate instructions. Self-critical (Dubois et al., 2024) Saha al. 2023 Bai et al. 2024) refers to using I.L.Mas---JJuee to assess the quality of the data. Self-rewarding Yuan et al. 2024) involves using the model itself, via LLM-as-a-Judge prompting, to provide its own reward mechanism. However, existing self- improvement methods primarily focus enhancing the overall capabilities of language models, without addressing the potential for self-improvemeet across different languages within the model. This is the key insight of our work.
be broadly divided into
Multilingual LLMs Contemporary LLMs Touvron et al. 2023a b Team et al. 2024; Bai et al., 2023 Achiam et al. 2023) are predominantly trained on multilingual corpora. However, the lan- guage distribution in the data primarily focuses on English and Chinese. The imbalanced data dis- tribution above has led to significant limitations in the capabilities of LLMs across most languages. To enhance the multilingual capabilities, one straightforward approach is multilingual fraining, us-
ing multilingual data during the pre-training (Conneau & Lample| 2019) Le Scao et al. (023) instruction-followwe (Li et al. 2023b] Muennighoff et al. (022) and post training (Dang et al. 2024]. However, high-quality multilingual data, particularly for low-resource languages, remains scarce and expensive. The second approach is cross-lingual alignment, which seeks to bridge the performance gap by aligning non-dominant and dominant languages. This approach utilizes tech- niques such cross-lingual transfer (Exxaniz et al.] [0033] |Huang al., [202]] Ranaldi & Pucci,
2023 Qin et al. (023), cross-lingual instruction tuning (Schuster et al.
2019 Wen- Yi & Mimno,
2023 and self-distillatio (Zhang et al. 2024)..
The most similar work, MAPO (She et al. 2024), uses an off-the-shelf translation model as a reward model assess lross-language consistency as the preference for optimization, focusing on aligning non-dominant languages with dominant ones in reasoning task. However, MAPO may struggle with consistency due to the limited context size in the translator, which makes it primarily suitable for reasoning tasks. In contrast, our approach relies on the LLM itself for translation, constructs pref- erence pairs directly based on language imbalance, and supports both dominant and non-dominant languages. Our work enables iterative self-improvement across all languages for general tasks. The comparisons with MAPO on reasoning task are presented Section
7 CONCLUSION
This paper introduces Language Imbalance Driven Rewarding, which leverages the inherent imbal- ance between dominant and non-dominant languages in LLMs reward signal to bootstrap LLMs' multilingual capabilitics in a self-improving manner. Starting from any with basic multilingual capabilities, this approach generates and self-translates the responses be- tween dominant and non-dominant languages within LLMss, constructing preference ranking and adopting an Iterative DPO for training. This approach not only enhances LLM performance in non- dominant languages but also improves the dominant language's capacity. Experiments on Llama-3- 8B-Instruct demonstrate significant improvements in instruction-following and arithmetic reasoning tasks. While much remains to be explored, this work paves the way for developing models capable of enhancing their multilingual abilities autonomously across all languages.
model
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This work is supported by the National Key R&D Program of China 20222D0160602. We would like to thank the anonymous reviewers for their helpful diseussions and valuable comments.
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303. 087744, 2023.
Janice Ahn, Rishu Verma, Renze Lou, Di Liu, Rui Zhang, and Wenpeng Yin. Large language models for mathematical reasoning: Progresses and challenges. arXiv preprint arXiv:2402.00157, 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:23009. 16609, 2023.
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harm- lessness from ai feedback. arXiv preprint urXiv:2212.08077: 2022.
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, al. Benchmarking foundation models with language-mon -aotag---ode------nnne axxann.x. Advances in Neural Information Processing Systems, 36, 2024.
Meriem Boubdir, Edward Kim, Beyza Ermis, Marzieh Fadaee, and Sara Hooker. Which prompts make the difference? data prioritization for efficient human Ilm evaluation. arXiv preprint AXXv:23100.14224, 2023.
Elizabeth M Brannon.
The independence of language and mathematical reasoning. Proceedings of the National Academy of Sciences, 10279):1177-317, 2005.
Zheng Cai, Maosong Cao, Haojiong Chen, Kai Chen, Keyu Chen, Xin Chen, Xun Chen, Zehui Chen, Zhi Chen, Pei Chu, et al. Internlm2 technical report. arXiv preprint arXiv:2403.17297, 2024.
Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, et al. xcot: Cross-lingual instruction tuning for cross-lingual thain-tf-thouuhtt reasoning. arXiv preprint arXiv:2401.07033. 2024.
Shreyas Chaudhari, Pranjal Aggarwal, Vishvak Murahari, Tanmay Rajpurohit, Ashwin Kalyan, Karthik Narasimhan, Ameet Deshpande, and Bruno Castro da Silva. Rlhf deciphered: A critical analysis of reinforcement learning from human feedback for 1lms. arXiv preprint urXiv:24044005555 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training a better alpaca with fewer data. arXiv preprini arxiv:2307 087001 2023a.
Nuo Chen, Zinan Zheng, Ning Wu, Linjun Shou, Ming Gong, Yangqiu Song, Dongmei Zhang, and Jia Li. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. arXiv preprint arX1v22110.02246, 2023b.
Pinzhen Chen, Zhicheng Guo, Barry Haddow, and Kenneth Heafield. Iterative translation refinement with large language models. arXiv preprint arXiv:2306.03855 2023c.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 15X3::880300444 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Rejichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1466. 2021.
11

---

Published as a conference paper at ICLR 2025
Alexis Conneau and Guillaume Lample. Cross-lingua language model pretraining. Advances in neural information processing systems, 32, 2019.
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. Rlhf can speak many languages: Unlocking multilingual preference optimization for llms. arXiv preprint 4XX1:22477.02552, 2024.
Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. arXiv preprint urXiv:2307.0869/1 2023.
Ameet Deshpande, Vishvak Murahari, Tammay Rajpurohit, Ashwin Kalyan, and Karthik Narasimhan. Toxicity in chatgpt: Analyzing persona-assigned language models. arXiv preprint S3Kv:22004.05335. 2023.
Shitong Duan, Xiaoyuan Yi. Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, and Ning Gu. Negating negatives: Alignment with human negative samples via distributional dispreference op- timization. In Findings of the Association for Computational Linguistics: EMNLP 2024, pp- 1012-1042, 2024.
Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Guirajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Alpacafarm: A simulation framework for methods that learn from human feedback. Advances in Neural Information Processing Systems, 36, 2024.
Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. Do multilin- gual language models think better in english? arXiv preprint arXiv:2308..011 002.0022,,1,,,,. 2023.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang. Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lin- tang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL /tttsttts:// 12608602
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan. Accelerate: Training and inference scale made sim- ple, efficient and adaptable. https: /
le lerate] 2022.
Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. Are large language bosedl-base evaluators the solution to scaling up multilingual evaluation? arXiv preprint arXiv:2309.07480 079466,, 2023.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint arXiv:2009.0333 030000003300,. 2020.
Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. Not all languages are created equal in 1lms: Improving multilingual capability by cross- lingual-thought prompting. arXiv preprint 23V::2305.00004 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, al. Mistral 7b, arXiv preprint arXiv:2310.0683 2023.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization, arXiv preprint arXiv:1412.6900 2014.
Maria Kunilovskaya, Koel Dutta Chowdhury, Heike Przybyl, Cristina España-Bor Bonntt, and Josef Gen- abith. Mitigating translationese with gpt-4: Strategies and performance. In Proceedings of the 25th Annual Conference of the European Association for Machine Translation (Volume 1), pp- 41 11-430, 2024.
12

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Viet Dac Lai, Chien Van Nguyen, Nghia Trung Ngo, Thuat Nguyen, Franck Demoncourt, Ryan A Rossi, and Thien Huu Nguyen. Okapi: Instruction-tuned large language models in multiple lan- guages with reinforcement learning from human feedback. arXiv preprint arXiv:2307 16039, 2023.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023.
Chong Li, Shaonan Wang, Jiajun Zhang, and Chengqing Zong. Align after pre-train: Improving multilingual generative models with cross-lingual alignment. arXiv preprint arXiv:2311.0808, 2023a.
Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation. arXiv preprint arXiv:2305.. 2501::300515011, 2023b.
Xian Li, Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, arXiv preprint arXiv:2308.06255, 2023c.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of foolooii models. https: /tt::///ittiie wttp://tttttttttaaaaaaa..aaa 1 5 2023d.
_evll,
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07955 2021.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu, G-eval: Nlg evaluation using gpt-4 with better human alignment. arXiv preprint 1036643333.6664, 2023.
AI Meta. Introducing meta llama 3: The most capable openly available Ilm to date. Meta AI, 2024.
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, et al. Crosslingual gen- eralization through multitask finetuning. arXiv preprint arXiv:2211.0/ 2022.
Long Ouyang. Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to fol- low instructions with human feedback. Advances in neural information processing systems, 35: 27730-27744, 2022.
Richard Yuanzhe Pang. Weizhe Yuan, Kyunghyun Cho, He He, Sainbayar Sukhbaatar, and Jason Weston. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733, 2024.
Libo Qin, Qiguang Chen, Fuxuan Wei, Shijue Huang, and Wanxiang Che. Cross-lingual prompting: Improving zero-shot chain-of-thought reasoning across languages. arXiv preprint arXiv:2310. 147999 2023.
Libo Qin, Qiguang Chen, Yuhang Zhou, Zhi Chen, Yinghui Li, Lizi Liao, Min Li, Wanxiang Che, and Philip S Yu. Multilingual large language model: A survey of resources, taxonomy and fron- tiers. arXiv preprint urXiv:2404.064 2444044944 2024.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024.
13

---

Published as a conference paper at ICLR 2025
Samyam Rajbhandari, Olatunji Ruwase, Jeff Rasley, Shaden Smith, and Yuxiong He. Zero-infinity: Breaking the gpu memory wall for extreme scale deep learning. In Proceedings of the interna- tional conference for high performance computing. networking. storage and analysis, pp. 1-14, 2021.
Leonardo Ranaldi and Giulia Pucci. Does the english matter? elicit cross-lingual abilities of large language models. In Proceedings of the 3rd Workshop on Multi-lingual Representation Learning (MRL), pp. 173-183, 2023.
Swamnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-solve-mee improves large language model evaluation and generation. arXiv preprint 151:23330..1133, 2023.
Tal Schuster, Oni Ram, Regina Barzilay, and Amir Globerson. Cross-lingual alignment of con- textual word embeddings, with applications to zero-shot dependency parsing. arXiv preprint 10X,::/902.09492,, 2019.
Shuaijie She, Shujian Huang, Wei Zou, Wenhao Zhu, Xiang Liu, Xiang Geng, and Jiajun Chen. Mapo: Advancing multilingual reasoning through multilingual piignmeet-as-prefeeemen optimiza- tion. arXiv preprint Stiv:2401.068:: 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. Language models are multi- lingual chain-of-thought reasoners. arXiv preprint arXiv:2210.03057 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followinn llama model, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.0829) 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2377,
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2xiv:2307.09288, 2023b.
Ahmet Üstün, Viraat Aryabumi, Zheng-Xin Xin Yong, Wei-Yin Ko, Daniel D'souza, Gbemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, et al. Aya model: An in- struction finetuned open-access multilingual language model. arXiv preprint a72.72240..07827. 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprini 22Xiv:2212.105600 2022.
Xiangpeng Wei, Haoran Wei, Huan Lin, Tianhao Li, Pei Zhang, Xingzhang Ren, Mei Li, Yu Wan, Zhiwei Cao, Binbin Xie, al. Polylm: An open source polyglot large language model. arXiv preprint arXiv:2307.06018 2023.
Andrea W Wen-Yi and David Mimno. Hyperpolyglot 1mms: Cross-lingual interpretability in token embeddings. arXiv preprint arXiv:231 18.004 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407. 10671, 2024.
Fei Yuan, Shuai Yuan, Zhiyong Wu, and Lei Li. How multilingual is multilingual llm? arXiv preprint 1:Xiv:2311.0007/ 2023.
14

---

Published as a conference paper at ICLR 2025
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho. Sainbayar Sukhbuatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.10020, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? arXiv preprint WwXiv:1955007885 2019.
Yuanchi Zhang, Yile Wang. Zijun Liu, Shuo Wang, Xiaolong Wang, Peng Li, Maosong Sun, and Yang Liu. Enhancing multilingual capabilities of large language models through self-distillation from resource- languages. arXiv preprint arXiv:2402. 12204, 2024.
Zhihan Zhang, Dong-Ho Lee, Yuwei Fang. Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. Plug: Leveraging pivot language in cross-lingual instruction tuning. arXiv preprint 1rXiv:231.08711, 2023.
Jun Zhao, Zhihao Zhang, Qi Zhang, Tao Gui, and Xuanjing Huang. Llama beyond english: An empirical study on language capability transfer. arXiv preprint arXiv:2401 Stt.:2401.000555, 2024.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances Neural Information Processing Systems, 36, 2024a.
Yaowei Zheng, Richong Zhang, Junhao Zhang. Yanhan Ye, and Zheyan Luo. Llamafactory: Unified efficient fine-tuning of 100+ language models. arXiv preprint 13374 2024b.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
Limitations and Future Work
17
Reproducibility Statement
17
Scaling on Model: Qwen2
17
C.1
Hcad-to-head performance
17
C.2
X-AlpacaEval Leaderboard
17
C Multilingual MT-Bench
18
C.4
Multilingual NLP Benchmarks
18
Discussion on Fair Evaluation
19
D.1 How to avoid Language Bias in LLM-ast as-Maa-----udge
19
D.2 Aligning with Advanced Model: Using GPT-4o Judge
19
D.3 How to avoid Translationese bias multilingual benchmarks evaluation
20
Generalizing to extreme scenarios
20
E,
Performance Weaker Model: Llama2
20
E.2
Performance Lower-resource languages: bn, th
20
E,3
Relaxation of the Self-improvement paradigm under extreme scenarios
20
F
Implementation Details
21
F.1
Experimental Details General Instruction-following
21
F2
Experimental Details on Arithmetic Reasoning
22
F3
Experiments Environments
22
F4
Hyperparameters
22
Detailed Results and Analysis
22
G.I
Multilingual NLP Benchmarks
22
H
Prompts Template
24
H.1
GPT-4 Score Prompt
24
H.2
Head-to-head Comparison Prompt
24
H.3
X-AlpacaEival Prompt
25
H.4
Self Translation Prompt
25
H.5
Multilingual Reasoning Prompt
26
16

---

Published as a conference paper at ICLR 2025
A LIMITATIONS AND FUTURE WORK
Our work has certain limitations. The reward signal is derived from the inherent language imbalance within LLMs, which provides a coarse-graines signal. Developing more refined and accurate reward signals for multilingual self eeffimprrveemet is an our approach relies on the LLM to self-translate the multilingual responses. Although LLMs out- perform traditional machine translation systems, the translated responses may still exhibit artifacts, which hinders the response quality.
plan explore in future work. Additionally,
B REPRODUCIBILITY STATEMENT
Codes and model weights have been made public after review to advocate future research. For eval- uation, primarily greedy decoding to ensure reproducibility, except where specific genera- tion configurations are mandated by certain benchmark tools. Note that evaluations on instruction- following abilities (AlpacaEval and MT-Bench) rely OpenAl's API. The randomness of API responses may have little impact on the reproducibility of these benchmarks.
C SCALING MODEL: QWEN2
Qwen2-7B-Instreee Yang et al., 2024) exhibits stronger multilingual capabilities and seldom pro- duces off-target responses. We believe scaling experiments to a multilingual LLM enhances the comprchensiveness of the evaluation. Following the experimental setup outlined in Section 4 Qwen2- 7B Instruct was chosen the base model validate the generalizability of Language Im- balance Driven Rewarding.
C.I HEAD-TO-HEAD PERFORMANCE
Win
Tie
Lose
English
Spanish
$2.4
$2.0
19.8
32.5
M1 MO
33.9
55.7
10.4
M1 MD
64.0
43.2
12.8
M1 MD
48.4
34.3
17.3
20
40
60
80
100
20
40
60
80
100
20
40
60
80
100
French
MO
40.9
39.3
12.9
MD
39.3
43.7
16.5
M1
33.9
5399
12.2
60
20
60
80
Figure 4: Multilingual Instruction following ability improves with Language Imbalance Driven Rewarding Bree2-7B-Instruuee model.
Figure 3 iillustrates Qwen2's head-to-head performance, which highly consistent with the results from the Llama3 performance.
For the training languages, Mj demonstrates a significant improvement, with AW-L ranging from 21.0% to 31.2% compared the base model. It demonstrates that Language Imbalance Driven Rewarding is effective. For the dominant language, English in M, gains 23.5% AW-L compared with Mo. while English M2 gains 7.3% A compared with M,. These results demonstrate that incorporating negative samples in preference pair construction significantly enhances the model's performance.
C.2 X-ALPACAEVAL LEADERBOARD
The X-AlpacaEval leaderboard Qwen2 as the base model is shown in Table After two rounds of iterations, Qwen2-78 Instrett achieved average improvements of 5.84% win rates over GPT-4 Turbo across five languages, demonstrating performance comparable to 70B-level models.
17

---

Published as a conference paper at ICLR 2025
Table 8: The X-AplacaEval Leaderboard On Qwen2-7B-Instruct which shows the win rate over GPT-4 Turbo evaluated by GPT-4.
Win Rate
Model
Avg
dc
fr
Language Imbalance Driven Rewarding
Qwen2-7B-Instnee (MO)
24.39%
13.89%
14.33%
11.45%
15.97%
16.01
Iteration (M1)
32.1
18.40%
18.61%
14.36%
18.47%
20.39%
Iteration 2 (M2)
34.84%
18.99%
20.28%
14.11%
21.03%
21.85%
Multilingual Aligument
Cennt--B- Qwen2-7B Instruct (SFT)
20.79%
18.27%
19.36%
12.61% %
17.39%
17.68%
Compared to multilingual alignment, our approach shows significantly better performance, with Win Rate of 21 88%% versus 17.68% for multilingual alignment. We analyze that multilingual alignment overemphasizes non-dominant languages during supervised fine-tuning, resulting in a 3.6% decline in English performance.
contrast, our method effectively utilizes language imbalance as a re- ward signal, capturing the partial order relationships among all languages, including the dominant one. This strategy results in a significant improvement in English performance, with an increase of 10.54%.
C.3 MULTILINGUAL MT-BENCH
In Table Swen2-7B-Instrue initially achieved a high score of 8.05, reflecting its robust mul- tilingual capabilities. Despite the strong performance reducing the effectiveness of the language imbalance-driven reward signal, Qwen2 improved its average score to 8.20 after two training itera- tions. This shows that even with high initial scores, our approach continues to improve performance through iterative refinement.
Table 9: The Multilingual MT-Bench Benchmark On Qwen2-7B-Instre
Training Languages
Unseen
Model
Avg
en
es
nu
de
fr
zh
Qwen2-7B- Instrructt (MO)
8.35
7.87
7.81
7.99
7.92
8.39
8.05
Iteration 1 (MI)
8.39
8.00
7.90
8.03
7.99
8.42
8.12
Iteration 2 (M2)
8.46
8.10
7.94
8.00
8.19
8.54
8.20
C.4 MULTILINGUAL NLP BENCHMARKS
Table shows average performance across five training languages and Chinese on four benchmarks based on Qwen2, detailed Appendix [..I] Slight performance improvements are observed in multilingual optimization iterations compared to the base models, indicating that the multilingual alignment process does not incur any alignment tax.
Table 10: The Multilingual NLP Benchmark On Qwen2-7B-Instree
Multilingual MMLU
Multilingual HellaSwag
Multilingual
Multilingual TruthfulQA
Model
ARC challenge
MC1
MC2
Qwen2-7B-Instruct (M0)
0.5139 0.6387
0.4321 +00144
0.3731
0.5395-0am0
Iteration (M1)
1.0001
0.5143 o.0052
1.4316
0.3744 10074-
0.5418 0.54182...1.9
Iteration (M2)
0.5130=a.0us 4.6403
0.3740
0.5410=p.m19
18

---

Published as a conference paper at ICLR 2025
D DISCUSSION ON FAIR EVALUATION
D.1 How TO AVOID LANGUAGE BIAS LLM-AS-A-JUDGE
The prior work (Hada et al. 2023) discussed that GPT-4's scores across languages may not be tirely consistent, as its evaluation capabilities can vary depending on the languages. In Table [] evaluate the GPT score for individual responses in different languages, e.g.. y or y² This approach may introduce language bias, as scores could vary depending on the language in which the response is generated. To mitigate this potential issue, we perform more detailed pairwise com- parisons within the same language in Tables 2 and 3 thereby avoiding cross- coos-lingual scoring bias. Importantly, this language bias Table not affect our conclusion, as we focus on constructing pairwise comparisons between responses within the same language for Tables and
Specifically, in each column of Table 2 both responses are in the same language. Take ru as an example. The 'Self Generation' GPT score calculated on y"" and the 'Self Translation' GPT score is calculated on y y As both responses are in it will not introduce potential language bias of L.M-as-a-Judgee providing fair comparison. Similarly, in Table 3 the reward accuracy is evaluated on preference pairs consisting of y"" and y which also avoids the cross-lingual comparison issue.
Both Table 2 and 3 evaluate GPT score based on pairwise comparisons within the same language. This methodology inherently avoids the cross-lingual comparison issue, ensuring a fairer and more consistent assessment since all evaluations are conducted within the same language.
D.2 ALIGNING WITH ADVANCED MODEL: USING GPT-40 AS A JUDGE
To investigate this potential bias, we employed the more advanced GPT-4o model (gpt-4o-202400--05 for evaluation. Specifically, we used the same responses as those in Table [ and evaluated their quality using GPT-4o. The results in Table [I] highlight an inherent imbalance in the model's multilingual capabilities, which aligns with the findings from GPT-4 sspt-4-110-ppees (rreveew evaluations in Table [ The consistent results from GPT-4 (Table I and GPT-4o (Table IT] across different evaluation models indicate the robustness our findings, even in the presence of potential cross-linguise evaluation biases.
Table 11: The average quality of responses across different languages for parallel multilingual instructions, evaluated with GPT-4o.
GPT-4o Score (0-10)
Model
cm
cs
fr
it
de
ja
ru
Llama-3-8B 3-8BBInsttrct
9.68
8.99
7.64
7.65
6.40
2.97
4.51
Morcover, we evaluated win rates on the X-AlpacaEval benchmark using GPT-4o. It is worth noting that the existing AlpacaEval repository does offer a GPT-4o evaluator configuration with human- alignment calibration. As a result, we had to use GPT-4o with a configuration calibrated for GPT-4 Turbo, which may introduce some bias.
The results in Table [2 remain consistent with Table evaluated by GPT-4 Turbo. These findings demonstrate consistent multilingual performance improvements and validate the robustness of our approach, despite potential minor evaluation bias.
Table 12:
The X-AplacaEval Leaderboard On Meta--Llama-3-8B-Innst evaluated with GPT-
4o.
Win Rate
Model
Avg
cn
CS
ru
de
fr
MO
35.12%
26.60%
8.35%
9.93%
19.21%
19.84%
M1
38.44%
34.28%
25.79%
26.34%
30.13%
31.00%
M2
39.78%
33.68%
28.37%
26.67%
31.49%
32.00%
19

---

Published as a conference paper at ICLR 2025
By adopting these measures, believe have adequately addressed the uncertainty surrounding GPT-4's reliability as a multilingual evaluator, ensuring that the assessments are as fair and consis- tent possible.
D.3 HOw TO AVOID TRANSLATIONESE BIAS IN MULTILINGUAL BENCHMARKS EVALUATION
Due to the expense and scarcity of multilingual benchmarks, most benchmarks in multilingual- related work, including both open-ended and structured tests, predominantly succhine-translate from English into other languages. Since the preference data is also constructed using translation, there is possibility that "translationese bias" could be exploited. However, our approach leverages LLMs for self-translation to construct training data, which offers key advantages to avoid transla- tionese bias:
(I) Different Data Distributions: Our method uses LLM self-tramslation to construct training data, while multilingual benchmarks are derived from machine translation of English datasets. This en- sures that the training data and benchmark data have different distributions, effectively minimizing the risk of translationese bias influencing evaluation.
(2) Reduction of Translationese Artifacts: L.LM self-translation significantly reduces translationese effects, producing fluent and natural translations that align closely with native text. This is supported by prior works (Chen et al., 2023c Kunilovskaya et al. 2024), which highlights the high-quality outputs of LLMs.
E GENERALIZING TO EXTREME SCENARIOS
E.1 PERFORMANCE ON WEAKER MODEL:: LLAMA2
Table 13 demonstrates that even when starting with model with weaker multilingual capabilities, such as Llama2-7BB -B-Cht, which exhibits extremely low performance in languages like Russian (ru), Genman (de), and French (fr) on the X-AlpacaEval, significant improvements can be achieved. By leveraging language imbalance-drived rewarding for self melf-muutilingual optimization across two iter- ations, the model shows substantial enhancement across all training languages, particularly in those where the original model's performance was initially weaker.
Table 13: The X-AplacaEvaa Leaderboard On Llama-2-7B- evaluated with GPT-4o.
Win Rate
Model
Avg
en
es
ru
MO
11.60%
3.40%
0.32%
0.87%
0.69%
3.38%
MI
14.62%
5.30%
1.91%
1.89%
2.68%
5.28%
M2
14.86%
6.62%
4.51%
3.62%
6.35%
7.19%
E.2 PERFORMANCE ON LOWER-RESOURE LANGUAGES: bn, sw, th
Llama3-8h Sb-Inttruct demonstrates weak performance in low-resource languages such as Bengali (bn), Swahili (sw), and Thai (th). It is important note that the effectiveness of post-training on these low-resource languages inherently limited. The model's s multilingual capabilities are primarily developed during the pre- pre-training phase, where it learns from diverse and extensive mul- tilingual corpus. As such, the gains from post- tostinraining are incremental and cannot fully overcome the limitations of the pre-training data for low-resource languages.
To assess the impact of our approach these languages, conducted experiments using Llama3- 8b-Instruct as the base model. Table 14 shows that even though the model performs weakly in these languages, approach remains effective in low-resone resource settings and can iteratively improve the model's 's performance across all languages.
E.3 RELAXATION OF THE DELF-IMPROVEMENT PARADIGM UNDER EXTREME SCENARIOS
Our approach is designed as self-improving paradigm, where the model iteratively refines its ca- pabilities. The primary goal of using self-translation is to preserve the integrity of a self-improving paradigm. However, in cases where the model's generation capabilities are particularly limited for certain low-resource languages, relaxing this constraint and using an external translator is also a viable solution.
20

---

Published as a conference paper at ICLR 2025
Table 14: The X-AplacaEval Leaderboard On Metta-Llmaa---B-Innttree in lower-resource lan- guages with self-translation, evaluated with GPT-4o.
Win Rate
Model
Avg
bn
sw
th
MO
35.12%
1.05%
1.01%
2.79%
9.99%
M1
38.22%
4.06%
1.15%
23.27%
16.68%
M2
39.27%
4.49%
2.07%
28.07%
18.48%
Table demonstrates Google Translate as external translation systems can be leveraged for mutual translation between dominant and low-resource languages bootstrap performance.
Compared with self-translation in Table 14 the external Google translation system provides higher- quality data for low-resource languages, enhancing the model's capabilities in these languages dur- ing optimization due to the model's initially weaker generation capabilities in these languages. How- more effectively improves the performance of English because avoids intro-
ever, Self-translatin ducing external translations, maintaining consistent generation space. This prevents disruption English's 's established capabilities, leading to better performance improvements.
The X-AplacaEval Leaderboard On Llama-3--B--nssteeeee in lower-resource lan- guages with Google Translate System, evaluated with GPT-4o.
Table 15:
Win Rate
Model
Avg
bn
th
MO
35.12%
1.05%
1.01% 101%
2.79%
9.99%
M1
38.35%
4.32%
2.98%
26.94%
18.15%
M2
38.38%
5.99%
3.45%
29.10%
19.23%
F IMPLEMENTATION DETAILS
F.1 EXPERIMENTAL DETAILS ON GENERAL INSTRICTTIN-FFLLOWWNNS
Head-to-hen sead- Performance Considering the excellent multilingual understanding ability of GPT- 4, we use GPT-4" as judge to conduct the automatic evaluation. GPT-4 as an evaluator has a higher correlation with human judgements (Liu al. 2023 Li et al.) 2023d). Specifically,
pairwise evaluation, asking GPT-4 to determine the better response between (r1, 72) from different models, given instruction x,. During the evaluation, GPT-4 assigns a score from 0 to 10 based on the prompt in Appendix H..2 GPT-4
evaluator, exhibits a significant positional bias, showing preference for responses that appear earlier (Zheng et al. 2024a). To mitigate this bias, we first request GPT-4 to evaluate (r1, 72), then switch the position to (r2, for the second evaluation. The better response is the one that wins twice or wins once and draws once.
X-AlpacaEval The X-AlpacaEval leaderboard lists the win rates of various models over GPT-4 Turbo evaluated against GPT-4. Based the we weithheed.... a paca eval gpt4 turbo config used in AlpacaEval 2, we modified the prompt to enable the model better evaluate multilingual responses. The modified prompt is provided in the Appendix H.3
Multilingual MT-Bench MT-Bench (Zheng et al.) 2024a) is a challenging multi-turn English question set designed evaluate the conversational and mnstuuttionfollloinng ability of LLMs. In our experimental setup, we collect multilingual MT- BrTcBench in German, French, Russian, and Chinese from Githut In addition, we translate the English data into Spanish by Google Translate API.
We use 1 API during the hcad-to-hcad evaluation.
https:://ii thub. wtt.c/cmmm////// highbllggthhlooo ue-thhhhmettte//m /mu ingual bench
21

---

Published as a conference paper at ICLR 2025
We examine the changes in world knowledge and commonsense reasoning abilities throughout the iterative process by evaluating it on the multilingual versions the
Multilingual NLP Benchmarks
MMLU (Hendrycks al. |020 HellaSwag (Zellers et al. 2019 | ARC Challenge (Clark et al,
2018 S
and TruthfulQA (Lin et al. 2021 benchmarks. We utilized the multilingual benchmarks
provided by Okapi (Lai et al. 2023). which were translated from the original benchmarks using
imeevaluuaiion ChatGPT, and conducted evaluations under the 1m-eval invvallation harness
(Gao et al.) 2024
framework.
F.2 EXPERIMENTAL DETAILS ON ARITHMETIC REASONING
Datasets We start from the GSM8K (Cobbe et al. 2021) dataset, which consists of 8,5K high- quality grade school math problems created by human problem writers English. We utilize the instructions from the 7,473 training examples and translate them into multiple languages using the Google Translate API construct the multilingual GSM8K instructions.
We input the multilingual GSM8K instructions into the model and explicitly constrain the model's response language in the prompt, as detailed in Appendix H.5 for both training and inference. We believe that by providing instructions in an explicit language and requiring the model to respond in that language, we can fully capture the model's 's reasoning abilities in that language. After obtain- ing the multilingual reasoning responses, we filter the responses with correct reasoning English, followed by applying Language Imbalance Driven Rewarding.
F.3 EXPERIMENTS ENVIRONMENTS
All experiments were conducted on Ubuntu 22.04 equipped with NVIDIA A100 GPUs. Our code mainly depends Python 3.10 and PyTorch 2.3.0. we fine-tune all models using LLaMA- Factory (Zheng el al., 2024b) framework, and inference models with vLLM (Kwon et al. 2023)
framework. Training for all models was launched with the accelerate (Gugger al. 2022)
Deep-
Speed ZeRO Stage2 (Rajbhandari et al. 2021) and Flash-Attention- 2 (Dao 2023) mechanism.
F.4 HYPERPARAMETERS
All models are optimized using Adam W (Kingma & Ba 2014). with cosine learning rate scheduler that includes a warm-up phase constituting 3% of the total training duration. DPO+NLL runs are trained with Kl.-penalty 0.1. The coefficient is set for all experiments in the paper. The details of hyperparameters are shown in Table T6
Table 16:
The hyperparameters on various experiments. 'LR' refers to the Learning Rate, and 'BS' denotes the Batch Size
Experiments
BS
Epoch
Language Imbalance Driven Rewarding
General Instruction-following Task
5c-7
16
1
Arithmetic reasoning Task
5e-6
64
1
Multilingual Alignment
All Tasks
le-5
128
3
G DETAILED RESULTS AND ANALYSIS
G. MULTHLINGUAL NLP BENCHMARKS
We list the detailed information of the benchmarks as follows:
MMLU (Massive Multitask Language Understanding) (Hendrycks et al. 2020 a bench- mark designed evaluate the knowledge acquired during pre-training. focusing zero- shot and few-shot settings. This makes it more challenging and closer to how humans are
htttes///ugghhhssc////ttt////ss/ mmlu
22

---

Published as a conference paper at ICLR 2025
evaluated. The benchmark spans 57 subjects, including STEM, the humanities, and the social sciences. We test it in a 5-shot setting.
HellaSwag (Zellers et al. 2019) is challenging dataset for evaluating commonsense NLI, which is particularly difficult for state-of-the-art models but trivial for humans. We test it in a zero-shot setting.
The A12 Reasoning Challenge (ARC) dataset (Clark et al. 2018) is a multiple-choice cresston--asseeing dataset based on science exams for grades 3 to 9. It is divided into two partitions: Easy and Challenge, with the latter containing more difficult questions re- quiring reasoning. We test the ARC Challenge in a zero-shot setting.
TruthfulQA (Lin et al. 2021 is a benchmark designed to evaluate whether a language model generates truthful answers. It consists of 817 questions across 38 categories, includ- ing health, law, finance, and politics. Since evaluating generation tasks for truthfulness is challenging, the benchmark provides two multiple-choice formats: MC1 (Single-true) and MC2 (Multi-true), testing the ability to identify true statements, We test it in zero-shot setting.
We report the detailed results in Table 17 of multilingual NLP benchmarks. Although the model contains only 1,000 Alpagasus instructions for each language, find that the model still shows slight improvements on these benchmarks during the iterative process. The results across multiple benchmarks indicate that our method does not introduce alignment tax.
Table 17: The Multilingual NLP Benchmarks.
Training Languages
Unseen
Model
Avg
en
fr
Mulsilnquu MMLU, 5-shor
Mota-Llama---83P 3-B--tt----- (MO)
0.6567 =0000ss
0.5771+0004
0.5335 200044
0,5506
0.5654
0.5162+0004
0.5666:aam4
Iteration (M1)
0.6585gomss
0.5765g00m4
200044
0.5536.0000
0.5678ga094
0.017
0.5687.10m4
Iteration 2 (M2)
0.6590=omms
0.5778
0.5368
:0.044
0.5529
0.5678
0.5178
0.5687=a0:43
Qwen2-7B-Instruee (MS)
0.706220ma17
0.6324-20m:z
0.6005
0.6048
0.6348gonesa
0.6446620m4
0.6387gm0w:
Iteration 1 (MI)
0.707810maa7
0.6333 0.61 18 g0000 0.6068
0.6361 gonese
0.6460ga0ss
0. 0.6402.gm0w
Iteration 2 (M2)
0.70 0.7055+0maa7 0.6342+ammzz 0.61 4 0.6097
0.6346-00042
0.6463+amma
0. 0.6403++am43
Mudrilingual HellaSwwy.
0-shot
Meta Llama 3 8B Instruct (MO)
0.5764_000
0.4877
0.4326_10.051
0.4483
0.4715
0.4181
10001)
0.4724g..001
Iteration 1 (MI)
0.5777=0000
0.4919ga0ms
0.4372
0,4511
0,4782
0,4204
00051
0.4761
Iteration 2 (M2)
0.579120009
0,4921
2:0002
0.4376.2000s
0,4512gacesi
0.4768
2o00s
0,4209
0.4763iamms
Qwen2-7B -e-ns-rtt (MD)
0.6116=-0000
0.5272.40m0s2
0.4730
#0.005
0.4659
0.51 =00002
0.4932+omms2 0.5139+amms
Iteration (M1)
0.6124+00040
+0.052
0.4753 +a.002
0.4654+000022
0.5129
+ow0s2 0.4936+00m5
0.5142++amms
Beration 2 (M2)
0.6110gomm
0.5255gnmms
gn00s 0.4640_noow
0.5114
0.5130gmmmm
Mahillingual ARC challenge, O-shor
1osta-lamaa--SB--nee (MO)
0.531620014
0.4162:zom44 0.3781
=0.a4 0,3978
0.4371
0.376l 0.4228gaam4
Iteration 1 (M1)
0.5324
0.4265
0.3867 0.4140.00044
0.4465-0014
0.3812
0,4312-g0044
Iteration 2 (M2)
0.5358
0.4299
0.3884
4.0183
0.4423-00145
0.3778
0.4321 20.144
Qwen2-7B--nstruct Instruct (MU)
0.510220004
0.4111
20044
0.4098
0,3618
goois
0.4277
2000es
0.4667
0.4312.jни4
Iteration I (MI)
0.508520046
0.4145
0,4072anmas
0,3678_oo0p
0.4226gooses
0.4692
0.4316gnma4
Iteration 2 (M2)
0.5128+00146
0.4205+aaml4
0.4038
+0.014 0,3704=02141
0,431
+00145
0,4641
+on14
0.4 0.4338
Muttifingual TruthfulQ4 MCI, 0-shot
Meta: Llama 3-8B Instruct (MD) 0.361 1oome
0.3333gaomes
0.3541
200070 0,3173
0.3355
0.
0.3417 20000
Iteration 1 (M1)
0.36112-mm
0.3257220mss 20072
0.3135 gootss 0.3532
0.3464_amme
Ieraaiion 2 (M2)
0.3599
0.3604_m007
0.3160_gnom
0.3532gg0mm
0.3617
0.3472.ammes
1een2-7B-Instruue (MO)
0.4064+00112
0.3676+4mm12
0.015
0.3439
+0200
0.3787 +00urs
0.3668
0.3731 +00122
Iteration 1 (M1)
0.4064+0017
0.3688++am12
4..173 0,3414+0210
0.3825+00u75
0.3718+00m22
0,3744+4ao22
leration 2 (M2)
0.4076gomm
0.36631om
0.3731 20007
0.3376_now
0.3863go0174
0.3731
200m
0.3740gmmr7
Multilingaal TrudhfulQA MC2, 0-shor
Mcta-Llama-2-88 Maa-a-------nstrrcct (MO)
0,5171g00112
0.4989.2om57
0.5256gnmm:
0.4890.a00ss
0.5033
0,5119
0.5076gamss
Iteration I (MI)
0.51424-00122
0.5059
0.5328
0.5003
0,5213
0.5250 0.5169
Ieeaatin 2 (M2)
0.5187+00111
0.5061
+00155
0,5331 +0.03)
0,4986
0,5171
#00157
0.5266
0.5165.4a055
Qwen2-75 7B Instruct (MU)
0.5733
0.5231
0.5231gaoma
gomes
0.53192000s
20:00
0,5356gaomol
0,5451
gonr
0.5282gomss
0.5395 1n0e
Iteration 1 (MI)
0.5757
0.5239gm0me
0,5347
20002
0.5325
0,54951o01s
0.5495
gonts
0.5344.0me 0.54181amms
Iteration 2 (M2)
0.5726=00114
0.5235
40mss
0.5346
0.5306
0.5515
=00157
0.5334
somso
0.5410.aams9
23

---

Published as a conference paper at ICLR 2025
H PROMPTS TEMPLATE
H.1 GPT-4 SCORE PROMPT
Prompt in GPT-4 Score
You are helpful assistant tasked with scoring answers for given instruction in [LANG] Please evaluate the following answer based on the provided instruction in [LANG]. good answer should adhere these criteria:
1. It should be [LANG], unless the instruction explicitly requests a different language.
2. It should address the request made in the instruction.
3. It should be factually and semantically coherent.
It should be grammatically correct and fluent.
5. It should he helpful, relevant, detailed, and accurate.
<instruction [INSTRUCTION]
</instructions
<answer>
[OUTPUT]]
</answer>
FIRST, provide a one-sentence explanation of your evaluation, detailing the reasoning behind your
score.
SECOND,
line, only the hetter overall performance. Your response should follow this format:
scale from 0 10, where higher score indicates
explanation ion: <one-sentence expl anat on
Score: <a scale from 0 to 10>
H.2 HRAD-TO-HEAD COMPARISON PROMPT
Prompt in Head-to-head Comparison
Given the question in [LANG] language. You are helpful precise assistant checking quality the answer.
<instruction>
[INSTRUCTION]
</instruction>
<answerl>
[OUTFUT1]
</answer1>
<answer2>
[OUTPUT2]
</answer2>
good answer should follow these rules:
1. It should be in [LANG], unless the instruction explicitly requests a different language.
2. It should be helpful, relevant, detailed and accurate.
3. It should answer the request in the instruction
Please evaluate both answers with your justification. and only provide ranging from 10 after your justifications, the score must an integer. score answer should wrapped by <scorel> and </scorel>, and the score for answer should be wrapped by <score2> and </score2>.
24

---

Published as a conference paper at ICLR 2025
H.3 X-ALPACAEVAL PROMPT
Prompt modified with weighted alpaca eval .pt 4 turbe in AlpacaEval 2.
<I <lim start >ssystmm
You highly efficient assistant, who evaluates and selects the best large language model (LLMs)
based the quality of their responses given instruction. This process will be used to create leaderboard reflecting the most accurate and human- unman-preferred answers.
immeend
immstarrt >user
I require leaderboard for various large language models. I'll provide you with prompts given
these models their corresponding outputs. Your task assess these responses, and select
model that produces the best output from human perspective.
## Instruction
"instruction": """ """iinstruction)""
## Model Outputs
Here are unordered outputs from models. Each output associated with specific model, identified by a unique model identifier.
poneel_identifiere "m", "output"; """ (outpput_]]"
"model mmodel_ident:ife wiiientnffeer" "M", "output"; """ """"/outp"" [output_2 2)*" =*"
## Task
good output should be the same language as the instruction, except when the instruction explicitly requests the output in different language. Evaluate the models based on the quality and relevance of their outputs, and select the model that generated the best output. Answer by providing the model identifier of the best model. We will use your output the name of the best model, make sure your output only contains one of the following model identifiers and nothing else (no quotes, no spaces, no new lines, ...): m or M.
## Best Model Identifier
<| 1limmee immeend
H.4 SELF TRANSLATION PROMPT
Prompt in Self Translation
Please translate the following sentences into [LANGUAGE]. The input sentences are wrapped by <sentence> and < sentence?:
<sentence>
[TEXT]
</sentence>
translated result should wrapped by <translated> </translated>.
25

---

Published as a conference paper at ICLR 2025
H.5 MULTILINGUAL REASONING PROMPT
Prompt in Multilingual Reasoning
Below an instruction that describes a task. Write a response that appropriately completes the request in [LANGTAGE] J. Please answer in [LANGUAGE].
#ill Instruction: [INSTRUCTION [N]]
#### Response:
26


Reviews:
[{"review_id": "C7m707W6u3", "reviewer": "Reviewer_AMKf", "summary": "This paper presents a method for bridging the language abilities for dominant languages and non-dominant languages. The method use the translation between the generation results in the dominant language and non-dominant language to build preference pairs, and conduct an iterative DPO optimization. The experiment shows improvement over the original model.", "questions": "I am not sure why languages such as es and fr only have a reward accuracy of 60%. It seems that they are well represented in the training data. It would be helpful if further explanation or analysis are provided.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The method is effective in improving the performance of general instruction following and mathematical reasoning.\n\nThe paper also presents analysis for the effectiveness and accuracy of the preference pairs, which serves as a nice support for the method.", "weaknesses": "The paper is not the first attempt to improve the multilingual ability of LLMs by cross-lingual optimizations. In one of the cited paper, She et al., ACL 2024, where experiments are conducted in optimizing the preference with DPO and an off-the-shell translator.\n\nI find the proposed method quite similar to the above one, but in this paper there is no clear indication of the potential relations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wjvasgvpRU", "reviewer": "Reviewer_w67r", "summary": "This work proposes to create a preference dataset using a multilingual LLM by exploiting the quality difference in responses in dominant languages (`dl`) and non-dominant languages (`nl`). The paper makes two assumptions that for the same instruction, 1) an `nl` response after being translated to `dl` is still worse than the original `dl` response, and similarly, 2) a `dl` response after being translated to `nl` is still better than the original `nl` response. The authors then provided empirical evidence for these assumptions. Iterative DPO training was done using the synthesized data and the trained models were tested on multilingual AlpacaEval, multilingual MT-bench, and some structured multilingual tasks. Results are shown to be in favour of the proposed method that models can iteratively improve by using data synthesized from itself.", "questions": "1. Table 2 row 2 \"Self Translation\" is confusing me: which language is translated into English (8.03)?\n2. The structure of section 5 looks weird. Why would arithmetic reasoning be a separate section/experiment?\n3. Line 143, Equation (7), the length normalization term $|y_{win}|$ is inside $log$. Is this correct?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- I think the idea of exploiting the gap in an LLM's inherent language capability for self-improvement is interesting and intuitive.\n\n- The paper has extensive experiments across open-ended and close-ended benchmarks. Results are consistent and in favour of the proposed method. It is clear that through iterative data synthesis and training, models can progressively improve in both dominant languages (`dl`) and non-dominant languages (`nl`).", "weaknesses": "1. Several factors in asserting the assumptions could not be carefully controlled. Table 1 line 180: GPT4-as-a-judge is used to confirm the quality of responses in different languages, however, there is no guarantee that the scores for different languages are on the same scale and that GPT4 is able to judge those languages using the same \"standard\". This could apply to Table 2 and Table 3 too.\n\n2. I find the motivation of creating (`nl`, `dl->nl`) preference data reasonable. However, I did not find creating the response pair (`dl`, `nl->dl`) intuitive. Since the model already could generate `dl` well (higher quality), why would having a lower-quality sample benefit alignment, especially as your Finding 2 that \"the dominant language also benefits [...] a phenomenon that has rarely been observed\"? Do you have any explanations for this?\n\n3. I think that all multilingual benchmarks used in this work, both open-ended and structure tests, are machine-translated from English to other languages. Since the preference data is also constructed using self-translation e.g. (`nl`, `dl->nl`), it might be possible that some kind of \"translationese bias\" is exploited as opposed to actually improve multilingual capability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ENBwLHADBD", "reviewer": "Reviewer_n8tX", "summary": "This paper leverage the langauge imbalance of LLMs to bootstrap the multilingual capabilities of LLM in a self-improving manner. The proposed approach involves training through Iterative Direct Preference Optimization (DPO), which leverages preference data generated by the model itself. By translating prompts and responses between dominant and non-dominant languages, the model assigns preferences, with responses in dominant languages generally preferred. The system iteratively fine-tunes the model on these preferences, improving multilingual performance without human-annotated data. Experiments on the Meta-Llama-3-8B-Instruct model show substantial performance improvements across both instruction-following and arithmetic reasoning tasks, with enhancements across all evaluated languages.", "questions": "1. It would be better to use more advanced LLMs like GPT-4o as the judge due to its stonger multilingual capabilities. \n2. Does the approach for weaker models like Llama-2-7B and lower-resource langauges? If it works, this method would be more impactful.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. This work suggests a promising direction for self-improving multilingual LLMs by leveraging intrinsic language imbalance.\n2. Experimental results demonstrate significant improvements over baseline models on multilingual benchmarks.\n3. The method is clearly explained, which is easy to understand.", "weaknesses": "1. The evaluation heavily relies on GPT-4 to assess response quality, but its reliability as a multilingual judge is uncertain, which was discuessed in [1]. The evaluation in Table 1 across various languages may be biased because it's unclear if GPT-4 rates identical responses equally in different languages. A fairer method would be to evaluate responses in the same language, including both directly generated and translated ones, to ensure scores are more comparable. \n2. The method depends on LLMs to self-translate responses, but the quality may be inferior to established systems like Google Translate and NLLB, particularly for low-resource languages, as discuessed in [2]. To enhance the analysis, it would be beneficial to assess the method using low-resource languages such as Swahili, Thai, and Bengali, as well as comparing it with more translation systems like Google Translate.\n\n[1] Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?\n[2] Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)