Language Imbalance Driven Rewarding for Multilingual Self-improving

Authors: Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang
Primary area: foundation or frontier models, including LLMs
Keywords: large language model, self-improving, multilinguality

Abstract:
Large Language Models (LLMs) have achieved state-of-the-art performance across numerous tasks. However, these advancements have predominantly benefited "first-class" languages such as English and Chinese, leaving many other languages underrepresented. This imbalance, while limiting broader applications, generates a natural preference ranking between languages, offering an opportunity to bootstrap the multilingual capabilities of LLM in a self-improving manner. Thus, we propose $\textit{Language Imbalance Driven Rewarding}$, where the inherent imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal. Iterative DPO training demonstrates that this approach not only enhances LLM performance in non-dominant languages but also improves the dominant language's capacity, thereby yielding an iterative reward signal. Fine-tuning Meta-Llama-3-8B-Instruct over two iterations of this approach results in continuous improvements in multilingual performance across instruction-following and arithmetic reasoning tasks, evidenced by an average improvement of 7.46\% win rate on the X-AlpacaEval leaderboard and 13.9\% accuracy on the MGSM benchmark. This work serves as an initial exploration, paving the way for multilingual self-improvement of LLMs.

Decision: accept (poster)
Reviewer scores: 3, 6, 6

Reviews and discussion:
<thread>
creation date:
1730103149661

thread_type:
official_review

<thread_object>
creator: reviewer_n8tx

<content>
summary:
This paper leverage the langauge imbalance of LLMs to bootstrap the multilingual capabilities of LLM in a self-improving manner. The proposed approach involves training through Iterative Direct Preference Optimization (DPO), which leverages preference data generated by the model itself. By translating prompts and responses between dominant and non-dominant languages, the model assigns preferences, with responses in dominant languages generally preferred. The system iteratively fine-tunes the model on these preferences, improving multilingual performance without human-annotated data. Experiments on the Meta-Llama-3-8B-Instruct model show substantial performance improvements across both instruction-following and arithmetic reasoning tasks, with enhancements across all evaluated languages.

strengths:
1. This work suggests a promising direction for self-improving multilingual LLMs by leveraging intrinsic language imbalance.
2. Experimental results demonstrate significant improvements over baseline models on multilingual benchmarks.
3. The method is clearly explained, which is easy to understand.

weaknesses:
1. The evaluation heavily relies on GPT-4 to assess response quality, but its reliability as a multilingual judge is uncertain, which was discuessed in [1]. The evaluation in Table 1 across various languages may be biased because it's unclear if GPT-4 rates identical responses equally in different languages. A fairer method would be to evaluate responses in the same language, including both directly generated and translated ones, to ensure scores are more comparable. 
2. The method depends on LLMs to self-translate responses, but the quality may be inferior to established systems like Google Translate and NLLB, particularly for low-resource languages, as discuessed in [2]. To enhance the analysis, it would be beneficial to assess the method using low-resource languages such as Swahili, Thai, and Bengali, as well as comparing it with more translation systems like Google Translate.

[1] Are Large Language Model-based Evaluators the Solution to Scaling Up Multilingual Evaluation?
[2] Is Translation All You Need? A Study on Solving Multilingual Tasks with Large Language Models.

questions:
1. It would be better to use more advanced LLMs like GPT-4o as the judge due to its stonger multilingual capabilities. 
2. Does the approach for weaker models like Llama-2-7B and lower-resource langauges? If it works, this method would be more impactful.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730591366664

thread_type:
official_review

<thread_object>
creator: reviewer_w67r

<content>
summary:
This work proposes to create a preference dataset using a multilingual LLM by exploiting the quality difference in responses in dominant languages (`dl`) and non-dominant languages (`nl`). The paper makes two assumptions that for the same instruction, 1) an `nl` response after being translated to `dl` is still worse than the original `dl` response, and similarly, 2) a `dl` response after being translated to `nl` is still better than the original `nl` response. The authors then provided empirical evidence for these assumptions. Iterative DPO training was done using the synthesized data and the trained models were tested on multilingual AlpacaEval, multilingual MT-bench, and some structured multilingual tasks. Results are shown to be in favour of the proposed method that models can iteratively improve by using data synthesized from itself.

strengths:
- I think the idea of exploiting the gap in an LLM's inherent language capability for self-improvement is interesting and intuitive.

- The paper has extensive experiments across open-ended and close-ended benchmarks. Results are consistent and in favour of the proposed method. It is clear that through iterative data synthesis and training, models can progressively improve in both dominant languages (`dl`) and non-dominant languages (`nl`).

weaknesses:
1. Several factors in asserting the assumptions could not be carefully controlled. Table 1 line 180: GPT4-as-a-judge is used to confirm the quality of responses in different languages, however, there is no guarantee that the scores for different languages are on the same scale and that GPT4 is able to judge those languages using the same "standard". This could apply to Table 2 and Table 3 too.

2. I find the motivation of creating (`nl`, `dl->nl`) preference data reasonable. However, I did not find creating the response pair (`dl`, `nl->dl`) intuitive. Since the model already could generate `dl` well (higher quality), why would having a lower-quality sample benefit alignment, especially as your Finding 2 that "the dominant language also benefits [...] a phenomenon that has rarely been observed"? Do you have any explanations for this?

3. I think that all multilingual benchmarks used in this work, both open-ended and structure tests, are machine-translated from English to other languages. Since the preference data is also constructed using self-translation e.g. (`nl`, `dl->nl`), it might be possible that some kind of "translationese bias" is exploited as opposed to actually improve multilingual capability.

questions:
1. Table 2 row 2 "Self Translation" is confusing me: which language is translated into English (8.03)?
2. The structure of section 5 looks weird. Why would arithmetic reasoning be a separate section/experiment?
3. Line 143, Equation (7), the length normalization term $|y_{win}|$ is inside $log$. Is this correct?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730800866415

thread_type:
official_review

<thread_object>
creator: reviewer_amkf

<content>
summary:
This paper presents a method for bridging the language abilities for dominant languages and non-dominant languages. The method use the translation between the generation results in the dominant language and non-dominant language to build preference pairs, and conduct an iterative DPO optimization. The experiment shows improvement over the original model.

strengths:
The method is effective in improving the performance of general instruction following and mathematical reasoning.

The paper also presents analysis for the effectiveness and accuracy of the preference pairs, which serves as a nice support for the method.

weaknesses:
The paper is not the first attempt to improve the multilingual ability of LLMs by cross-lingual optimizations. In one of the cited paper, She et al., ACL 2024, where experiments are conducted in optimizing the preference with DPO and an off-the-shell translator.

I find the proposed method quite similar to the above one, but in this paper there is no clear indication of the potential relations.

questions:
I am not sure why languages such as es and fr only have a reward accuracy of 60%. It seems that they are well represented in the training data. It would be helpful if further explanation or analysis are provided.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732079133716

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response about GPT-4 on Cross-Language Evaluation Bias

comment:
Thank reviewer `w67r` and `n8tX` for raising the issue about GPT-4 on cross-language evaluation bias. We greatly appreciate your attention to this matter, and below, we address this issue from three perspectives to ensure the reliability of our findings:

> Addressing Cross-Language Evaluation Bias (`Table 1 of the manuscript` and `Table G1`)

We acknowledge the limitation that GPT-4's scores across languages may not be entirely consistent, as its evaluation capabilities can vary depending on the language. To investigate this potential bias, we employed the more advanced GPT-4o model (**gpt-4o-2024-08-06**) for evaluation. Specifically, we used the same responses as those in `Table 1 of the manuscript` and evaluated their quality using GPT-4o. The results in `Table G1` highlight an inherent imbalance in the model's multilingual capabilities, which aligns with the findings from GPT-4 (**gpt-4-1106-preview**) evaluations in `Table 1 of the manuscript`. The consistent results from GPT-4 (`Table 1`) and GPT-4o (`Table G1`) across different evaluation models indicate the robustness of our findings, even in the presence of potential cross-linguistic evaluation biases.

Table G1:  The average quality of responses evaluated by `the latest GPT-4o`.
| Model               | en   | de   | es   | fr   | it   | ja   | ru   |
|---------------------|------|------|------|------|------|------|------|
| Llama-3-8B-Instruct | **9.68** | 8.99 | 7.64 | 7.65 | 6.40 | 2.97 | 4.51 |

> Ensuring Fairness in Same-Language Evaluations (`Table 2 and Table 3 of the manuscript`)

In `Table 2 of the manuscript`, both "Self Generation" and "Self Translation" involve responses in the same language, while in `Table 3 of the manuscript`, preference pairs are similarly constructed within the same language. These evaluations inherently avoid the cross-lingual evaluation issue, providing a relatively fair comparison and ensuring that the evaluation remains consistent and unbiased within the same language.
 
> Validation on the X-Alpacaeval (`Table 4 of the manuscript` and `Table G2`)

To ensure a fair comparison, we evaluated win rates on the X-AlpacaEval benchmark using GPT-4o (gpt-4o-2024-08-06). However, the existing AlpacaEval repository does not provide a GPT-4o evaluator configuration with human-alignment calibration. As a result, we had to use GPT-4o with a configuration calibrated for GPT-4 Turbo, which may introduce some bias.

The results remain consistent with `Table 4 of the manuscript`, evaluated by GPT-4 Turbo. These findings validate the robustness of our approach and demonstrate consistent multilingual performance improvements, despite potential minor evaluation bias.

Table G2: The X-AlpacaEval Leaderboard evaluated by `the latest GPT-4o`.
| Model                         | en     | es     | ru     | de     | fr     | Avg   |
|-------------------------------|--------|--------|--------|--------|--------|-------|
| Meta-Llama-3-8B-Instruct(M0) | 35.12% | 26.60% | 8.35%  | 9.93%  | 19.21% | 19.84% |
| Iteration1(M1)               | 38.44% | 34.28% | 25.79% | 26.34% | 30.13% | 31.00% |
| Iteration2(M2)               | 39.78% | 33.68% | 28.37% | 26.67% | 31.49% | 32.00% |


By adopting these measures, we believe we have adequately addressed the uncertainty surrounding GPT-4's reliability as a multilingual evaluator, ensuring that the assessments are as fair and consistent as possible. We appreciate the reviewer's suggestion and have incorporated it into our analysis.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper leverages DPO style optimization to improve multilingual instruction following and numerical reasoning in LLMs by leveraging the imbalance between dominant and non-dominant languages within LLMs is leveraged as a reward signal.

Strengths:
- A new method that shows gains in X-AlpacaEval leaderboard on the MGSM benchmark by training Meta-Llama-3-8B-Instruct.

Weaknesses:
- A reviewer mentions that this paper has high similarity with She et al. (2024), but I am not very concerned about this given the innovations that the authors mention clearly in one of their responses.
- Reviewers have concerns about using GPT as judge for evaluation, but I believe this was also addressed by the authors in their responses.