RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Introduction
Recent advances in reinforcement learning with human feedback (RLHF) as a large language model (LLM) post-training technique (Christiano et al., 2017 ; Ziegler et al., 2020 ) highlight the importance of having high-quality data (Wang et al., 2024f ; Dubey et al., 2024 ) and reward model (RM) (Ethayarajh et al., 2022 ; Gao et al., 2023 ; Ji et al., 2023 ; Wang et al., 2024a , e ) . Leveraging synthetic data has contributed to building stronger English RMs due to their efficiency and scalability (Cui et al., 2024 ; Wang et al., 2024b ; Zhu et al., 2024 ) .

Nevertheless, adopting RMs for non-English languages is heavily understudied. While LLM-as-a-Judge can be used as a generative reward model for multilingual RLHF settings (Son et al., 2024 ) , generative RMs have been shown to underperform traditional RMs (Lambert et al., 2024 ; Wang et al., 2024b ) . Meanwhile, Wu et al. ( 2024 ) empirically demonstrates the possibilities of cross-lingual transfer in RMs, but the findings were limited to simple tasks and encoder-decoder models.

In this paper, we show that RMs trained on English-only datasets ( i.e., English RMs) display strong cross-lingual transfer when built on top of multilingual pre-trained language models (MLMs). We first demonstrate the cross-lingual transfer of English RMs by consistently outperforming target language RMs in Multilingual RewardBench . Then, we explain it with two reasons: 1) English preserves representations of the initial MLMs (Section 3.1 ), and 2) representations of MLMs inherently have a strong understanding of languages (Section 3.2 ), concluding that RMs should preserve representations of MLMs for generalizability. Additional analysis of off-the-shelf RMs supports our findings by both classifier and generative RMs based on MLMs having strong cross-lingual transfer. Finally, multilingual alignment experiments exhibit the propagation of strong cross-lingual transfer in English RMs to downstream usage, having an average win rate increase of 9.5% across four non-English languages.
English as a Lingua Franca in RMs
We empirically verify the cross-lingual transfer in reward models (RMs) trained with different languages, thereby showing that the English preference data is a lingua franca in reward modeling.
Cross-lingual transfer
Training multilingual language models (MLMs) at scale has shown to incur cross-lingual transfer in both encoder-only (Devlin et al., 2019 ; Conneau et al., 2020 ; Chi et al., 2022 ) and encoder-decoder (Xue et al., 2021 ) transformer architectures. Recently, studies revealed the implications of cross-lingual transfer in decoder-only models as well (Üstün et al., 2024 ; Wang et al., 2024c ) ; however, they were limited to generative tasks (Zhang et al., 2024 ) or downstream alignment-tuning only (Dang et al., 2024 ) .
Reward modeling
Reward models are trained as a classifier (Christiano et al., 2017 ) to return a scalar value r_{\theta}(\cdot) with the objective with the Bradley-Terry model (Bradley and Terry, 1952 ) :

\mathcal{L}_{RM}=\sigma\left(r_{\theta}(x,y_{w})-r_{\theta}(x,y_{l})\right),

with the prompt x and corresponding preferred and dispreferred responses y_{w} and y_{l} . While crucial in alignment-tuning (Rafailov et al., 2024 ; Hong et al., 2024 ; Meng et al., 2024 ) , reward modeling schemes for multilingual usage are still understudied.
Motivated by this research opportunity, we study the cross-lingual transfer of English-focused RMs with recent autoregressive models and how it propagates to downstream multilingual alignment.
Dataset
We curate a synthetic preference dataset of 86k instances 3 3 3 Refer to Appendix A for detailed process. from five representative English preference datasets: SafeRLHF (Dai et al., 2024 ) , WildGuard (Han et al., 2024 ) , HelpSteer2 (Wang et al., 2024e ) , Offsetbias (Park et al., 2024 ) , and Magpie (Xu et al., 2024b ) . Using English data, we create four parallel machine-translated versions 4 4 4 Spanish (Sp), Italian (It), Korean (Ko), and Chinese (Ch) , utilizing X-ALMA (Xu et al., 2024a ) .
Models
Two state-of-the-art 3B multilingual pre-trained language models are fine-tuned 5 5 5 Refer to Appendix B for detailed hyperparameters. as reward models: Llama-3.2-3B-Instruct (Dubey et al., 2024 ) and Qwen2.5-3B-Instruct (Yang et al., 2024 ) .
Evaluation
We prepare four non-English Multilingual RewardBench by translating RewardBench (Lambert et al., 2024 ) to assess the cross-lingual transfer in RMs.
English RMs show strongest cross-lingual transfer
Average reward model accuracy ("Avg") in Table 1 shows that English RMs surpass target language RMs in general. Specifically, Llama-3.2-3B gained at least 4.3%, where the cross-lingual generalizability of English RMs is more highlighted than Qwen2.5-3B, which gained at most 4.3%. However, considering that all Qwen-based target language RMs outperform the Llama-based target language RMs, Qwen2.5-3B is shown to be a better model choice for training a language-specific RM.
Reasoning tasks significantly benefit from cross-lingual transfer
Generalizability of English RMs is best highlighted in the reasoning tasks ("Reason") in Table 1 , especially in non-Latin languages. Non-Latin languages, Korean and Chinese, improved significantly in English RMs compared to target language RMs, exceeding 12% and 27% in Chinese, for instance.
Analysis on Lingual Tranfer of MLM
This section provides empirical and theoretical insights on why English is lingua franca in reward modeling, given a multilingual language model (MLM) using two arguments: 1) English acts as a lingua franca in reward modeling because it best preserves the representations of the base model, and 2) representations in MLMs should be preserved since they are inherently effective in language-aware encoding.
Non-English reward modeling is detrimental to generalizability
In general, the generalizability of the downstream model is closely connected to how much the representations are preserved during the fine-tuning (Aghajanyan et al., 2021 ; Razdaibiedina et al., 2023 ) . We demonstrate this in RMs by ablating over different languages and tasks.
We assess the general representation preservation of RMs used in Section 2 by comparing their hidden states against the initial model. To do so, we measure how much the distinct representations are collapsed into similar spaces in Figure 1 . In specific, we construct a matrix of the last hidden states \mathcal{H}_{\theta}(x)\in\mathbb{R}^{5\times d_{\text{model}}} across five languages using multilingual dataset BeleBele (Bandarkar et al., 2024 ) :

\mathcal{H}_{\theta}(x)=concat\left[\left\{H_{\theta}^{l}(x_{l})\right\}_{l\in L}\right]\in\mathbb{R}^{|L|\times d_{model}},

where H_{\theta}^{l}(x)\in\mathbb{R}^{d_{\text{model}}} refers to the last hidden state of the model \theta for sequence x_{l} in the language l , but with fixed context. Then, we measure the proportion of the largest singular value in \mathcal{H}_{\theta}(x) :

f_{\theta}(x)=\frac{\sigma_{1}}{\sum_{i=1}^{|L|}\sigma_{i}},S=diag\left(\sigma_{1},\ldots,\sigma_{|L|}\right),

with S as a singular values of \mathcal{H}_{\theta}(x) . Intuitively, having f_{\theta}(x) close to 1 indicates the hidden states in different languages are homogeneous: i.e., representations are embedded into similar space.

In Figure 1 , we plot f_{\theta}(x) with different RMs. English RMs best preserve the representations by staying close to the base instruct model ("Inst"). On the other hand, Korean RMs ("Ko") tend to deviate the most from the base model, thereby homogenizing the multilingual representations the most. Both observations were more extreme in Llama-3.2-3B.
General representation preservation is crucial for cross-lingual/task transfer
Notably, the proclivity in general representation preservation in Figure 1 aligns with the accuracy in Table 1 . Non-English RMs with Llama-3.2-3B tend to introduce stronger representation collapse than Qwen2.5-3B in Figure 1 . This aligns with Section 2.3 as Llama-3.2-3B gets more severe degradation using target language RMs, implying the significance of representation preservation in cross-lingual transfer.

Furthermore, the same tendency holds for cross-task analysis. RewardBench has especially fine-grained divisions under the reasoning category ( e.g., Java, Python, Rust, math) compared to other categories. Thus, strong generalization abilities are crucial to achieving decent scores in the reasoning category. Interestingly, English RMs dominate other languages in reasoning despite the fixed data across the languages in Table 1 , which strongly supports the significance of representation preservation in cross-task generalization.
MLM representations are language-aware
In autoregressive language models (Radford et al., 2019 ) with tied embeddings (Jiang et al., 2023 ; Team, 2024a ) , the logits for next token is:

h_{t}\cdot E=\Bigl{[}\|h_{t}\|\cdot\|e_{i}\|\cdot\cos\left(\theta_{i}\right)\Bigr{]}_{i=1}^{|V|},

where \theta_{i} is the angle between h_{t} and e_{i} . Therefore, the capability of language models in generative tasks is closely related to having good representations (Edunov et al., 2019 ) that could accurately align with the ideal next token.

Token embeddings are a good proxy to understand the effectiveness of representations as they imply the imbalance in pre-training corpora (Chung et al., 2024 ) , especially by linguality in this study (Wen-Yi and Mimno, 2023 ) .
Thus, we can infer that language models with similar embedding norm distribution across the language will have decoder layers that can return language-aware fine-grained hidden states, which deserve to be preserved for their generalizability.
MLMs have similar token embedding norm distributions across the language
We validate this point by comparing the two models in Section 2 with two monolingual pre-trained language models: OLMo-1B (Groeneveld et al., 2024 ) and SmolLM-1.7B (Allal et al., 2024 ) . We clarify the lingualities in each model’s pre-training in Appendix C .

We collect the disjoint language-specific token embedding norms for each model:

\mathbf{e}_{l}=\left\{\|e_{j}\|\right\}_{j\in A_{l}},A_{l}\subset V,\bigcap_{l\in L}A_{l}=\varnothing

where A_{l} is the token indices of language l in V . We compare \mathbf{e}_{L} distribution over five languages.

In Figure 2 , the distribution for English in SmolLM-1.7B and OLMo-1B are distinct from four languages, especially Korean and Chinese, which are non-Latin languages that do not share similar alphabets. However, Qwen2.5-3B and Llama-3.2-3B have similar ranges and distributions across the languages, even in non-Latin languages.

Thus, we can infer that Qwen2.5-3B and Llama-3.2-3B, as MLMs, are sufficiently trained on the multilingual corpus to encode information with diverse linguality by having similar embedding norm distributions across the languages (Dagan et al., 2024 ; Chung et al., 2024 ) . This supports why representation preservation is a crucial condition for generalizable RMs with MLMs, as discussed in Section 3.1 .
Multilingual Alignment using RM
In this section, we perform experiments to outline the effects of using the reward models (RMs) from Section 2 and how their cross-lingual transfer can propagate to the actual alignment process.
Experimental Details
We sample 10k prompts from the cleaned UltraFeedback dataset (Bartolome et al., 2023 ; Cui et al., 2024 ) and translate prompts across target languages. Then, we sample four responses per prompt with Qwen2.5-7B-Instruct (Team, 2024b ) and label them with desired RMs. By selecting the responses with the highest and lowest rewards, we prepare pairwise preference data. We train Qwen2.5-7B-Instruct on each language from the newly curated datasets with Direct Preference Optimization (Rafailov et al., 2024 , DPO) . Refer to Appendix B for the detailed setup.
Evaluation
We evaluate the trained model’s language-specific instruction-following capability with Multilingual AlpacaEval , adopted from the instances and evaluation pipeline of AlpacaEval (Li et al., 2023 ) . We report the detailed process and configurations in Appendix D .
English RM largely improves base models in every language
As shown in Figure 3 , models aligned with English RM show a notable leap compared to Qwen2.5-7B-Instruct ("Base"), by increasing up to 11.8% point. As the win rate was measured against GPT-4-Turbo, a strong proprietary language model, such enhancements strongly support the validity of using English RMs for multilingual alignment in desired languages.
Exploiting English RMs is a desirable choice in multilingual alignment
We emphasize that using high-quality English preference data of better accessibility is a decent choice, considering the efficiency and efficacy in real-world cases. In Figure 3 , models aligned with English RM were comparable to or exceeded ones with target language RMs, having at most 1.5% disparity in the worst scenario. Thus, adopting an English RM for multilingual alignment is a cost-efficient yet performant alternative, discarding the need for scaled translations.
Cross-lingual Transfer of External RMs
Along with the controlled comparisons in Section 2 , we analyze the cross-lingual transfer in off-the-shelf models on the original RewardBench through Multilingual RewardBench . To ensure diversity in reward modeling schemes, we selected two classifier reward models (RMs), ArmoRM-8B (Wang et al., 2024b ) and OffsetBias-8B (Park et al., 2024 ) , alongside two generative RMs, GPT-4o 6 6 6 https://platform.openai.com/docs/models/gpt-4o and Self-Taught-Llama-70B (Wang et al., 2024d ) .
Classifier RMs
Two classifier RMs are both trained on top of Llama3-8B-Instruct (Dubey et al., 2024 ) , which are based on multilingual pre-trained language models (MLMs) as discussed in Appendix C . As in Table 1 , these RMs also demonstrate strong cross-lingual transfer in four languages, mostly exceeding 70% accuracy across the board in Table 2 .
Generative reward models
Interestingly, we can observe strong cross-lingual transfer in the generative RMs in Table 2 , as in the classifier RMs. As discussed in Section 3.2 , fine-grained representation learning is a crucial component for having strong downstream generative abilities. While the extent of multilingual pre-training in GPT-4o is not verifiable, GPT-4o has the least decrement in non-English settings. Meantime, Self-Taught-Llama-70B with extensive multilingual pre-training demonstrates the strongest cross-lingual transfer, achieving the best accuracies in all four non-English Multilingual RewardBench .
Conclusion
We empirically demonstrate English as a lingua franca in reward modeling, given recent multilingual pre-trained language models (MLMs). We explain this with two consecutive arguments. First, English reward models (RMs) best preserve the representations of initial MLMs, while other languages induce representation collapse. Second, MLM representations inherently have a rich understanding of languages and tasks, making them valuable to preserve in downstream tasks. By extending our analysis to the off-the-shelf reward models, we show that using MLMs for reward modeling is crucial for eliciting strong cross-lingual transfer. Through strong cross-lingual transfer in English RMs, we establish a concrete foundation for exploiting English RMs for multilingual alignment.
Limitations
To extend to more languages and evaluation benchmarks, we have mainly utilized a 3B LLM to train the reward model (RM) with only 86k instances. However, as outlined in Appendix E , the 3B RMs are on par with a state-of-the-art RM, ArmoRM, which was trained with over 550k instances. Future works on the effects of data size and mixture will provide an enhanced understanding of our work.

Also, in Section 4 , we use the AlpacaEval evaluation setup, which utilizes LLM-generated reference responses and LLM-as-a-Judge to select a winning response. Therefore, while we show a vast increase in post-training alignment, the process relies on the multilinguality of OpenAI models and the evaluation biases of the LLM-based evaluations outlined in Zheng et al., 2023 .
B.1 Reward Modeling
We used a maximum learning rate of 1e-5 and 10% of warm-up followed by cosine decay. The projection head for the reward model was initialized with \mathcal{N}\left(0,1/\sqrt{d_{\text{model}}+1}\right) (Stiennon et al., 2020 ; Huang et al., 2024 ) . The global batch was set to 128.
B.2 On-Policy Preference Optimization
We fine-tune Qwen2.5-7B-Instruct (Team, 2024b ) with DPO using Liger-kernel (Hsu et al., 2024 ) . We use a cosine decaying learning rate scheduler for single epoch training.
DPO configurations
We apply \beta=0.1 with the learning rate of 5\text{e}-7 . The global batch size was set to 32 using gradient accumulation steps of 8 with a per-device batch size of 1, which was the maximum number for NVIDIA A100 80GiB.
Data curation
To construct the preference pairs for preference optimization, we sample 4 responses from Qwen-2.5-7B-Instruct. Then, we compute the rewards through the reward models and select the response with the highest and lowest reward values as the preference pairs for training the checkpoints through DPO.
(a) Qwen2.5-3B; (b) Llama-3.2-3B; Proportion of the largest singular value in the concatenated hidden states for fixed context translated in five languages with RMs trained in each language. While English ("En") best preserves the representation diversity of the base model ("Inst’), Korean ("Ko") leads to the most homogeneous representations.
(a) OLMo-1B; (b) SmolLM-1.7B; (c) Qwen2.5-3B; (d) Llama-3.2-3B; Embedding norm distribution comparison between English and four other languages (2 non-Latin (top), 2 Latin (bottom)) across four language models: OLMo-1B and SmolLM-1.7B (monolingual pre-training) and Qwen2.5-3B and Llama-3.2-3B (multilingual pre-training). While English and non-English token embedding norm distributions of OLMo-1B and SmolLM-1.7B are distinct, they are similar in Qwen2.5-3B and Llama-3.2-3B.
Figure 3: Multilingual AlpacaEval results of Qwen2.5-7B-Instruct models fine-tuned with DPO on on-policy generations for four non-English languages. The alignment data were labeled with either English RM or target language RM.; Multilingual AlpacaEval results of Qwen2.5-7B-Instruct models fine-tuned with DPO on on-policy generations for four non-English languages. The alignment data were labeled with either English RM or target language RM.
Multilingual RewardBench evaluation results on the target language ("Target") and English ("English") RMs. " \Delta " denotes the accuracy gain of English RMs compared to the target language RMs. English RMs show higher average scores in the lingual axis than target language RMs. Also, English RMs excel target language RMs in reasoning ("Reason") tasks with diverse evaluation sub-categories.
Averaged Multilingual RewardBench results in two classifier RMs (top) and two generative RMs (bottom). Off-the-shelf RMs based on MLMs show strong cross-lingual transfer as in Table 1 .
RewardBench results for reward model comparison across four different categories. (* denotes off-the-shelf models)
Spanish RewardBench results for reward model comparison across four different categories. (* denotes off-the-shelf models)
Italian RewardBench results for reward model comparison across four different categories. (* denotes off-the-shelf models)
Korean RewardBench results for reward model comparison across four different categories. (* denotes off-the-shelf models)
Chinese RewardBench results for reward model comparison across four different categories. (* denotes off-the-shelf models)