Fluent Alignment with Disfluent Judges: Post-training for lower-resource languages
Authors: David Samuel
Lilja Øvrelid
Erik Velldal
Andrey Kutuzov
Venue: ICLR 2026 Poster (2026)
Keywords: language model
post-training
fluency
low-resource languages
RLAIF
Primary area: foundation or frontier models, including LLMs
TL;DR: We propose a preference-optimization method for lower-resource languages that results in fluent language models even when aligned by disfluent reward models.

Abstract: We propose a post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligned language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine-translated data and multilingual finetuning. We conduct a case study on Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.

Published as a conference paper at ICLR 2026
FLUENT ALIGNMENT WITH DISFLUENT JUDGES: POST-TRAINING FOR LOWER-RESOURCE LANGUAGES
David Samuel
Lilja Øvrelid Erik Velldal
Andrey Kutuzov
Language Technology Group, University of Oslo davisamuêi fi. uio.
ABSTRACT
We propose post-training method for lower-resource languages that preserves the fluency of language models even when aligned by disfluent reward models. Preference optimization is now a well-researched topic, but previous work has mostly addressed models for English and Chinese. Lower-resource languages lack both datasets written by native speakers and instruction-tuned language models capable of generating fluent synthetic data. To address this, we focus on developing a fluent preference-aligee aliigne language model without any instruction-tuning data in the target language. Our approach uses an on-policy training method, which we compare with two common alternatives: supervised finetuning on machine- translated data and multilingual finetuning. We conduct case study Norwegian Bokmål and evaluate fluency through native-speaker assessments. The results show that the on-policy aspect is crucial and outperforms the alternatives without relying on any hard-to-obtain data.
INTRODUCTION
Instruction-tuning and preference-pptiiizaie have become a cornerstone of modern language models, enabling base models to follow instructions and engage in helpful dialogue. However, this progress has been largely confined to high-resource languages like English and Chinese, which benefit from extensive human- datasets and sophisticated language models capable of generating fluent synthetic data. Lower-resource languages face a fundamental challenge: they lack both instruction datasets written by native speakers and fluent instruction-tumi models that could generate high- quality training data (Guo et al., 2025). This work addresses a critical question for the democratization of language technology: how can we create fluent preference-aligune aligned language models for lower- resource languages without any instruction-tuning dataset in the target language?'
Current approaches to post-training language models for lower-resource languages mostly rely on static and predefined instruction-tuniin datasets (Suzuki et al., 2023; Chouikhi et al., 2024; Lim al., 2025), which are usually machine-translated from English (Pipatanakul et al., 2023; Santilli Ranaldi & Pucci, 2023; Ustün et al., 2024; Nguyen et al., 2024; Bari et al., 2025; Zosa et al., 2025). While this approach shows promising results when evaluated on standard NLP benchmarks, the translation process introduces subtle linguistic artifacts translationese that make the resulting models disfluent in the target language (Yu et al., 2022; Dutta Chowdhury et al., 2022). Translationese is produced even by professional human translators, and are negatively impacted by it to an even greater extent (Bizzoni et al., 2020). Concurrent work by Kunz (2026) has demonstrated that even short exposure such data leads to rapid loss of fluency. Thus, we believe that post-training for lower-resource languages needs to shift away from such inherently flawed data.
& Rodolà, 2023;
models
Recent advances in reinforcement learning from Al feedback (RLAIF; Bai et al., 2022) offer a potential solution to this challenge. In on-policy reinforcement learning, the model learns from its own generated responses rather than from fixed datasets. This means we can potentially avoid exposing to disfluent text altogether. The key insight is that a model that has learned fluent generation through extensive pretraining native texts can maintain this fluency as long as it is never trained on unnatural examples during the alignment phase.
Fluency refers to the linguistic quality of text that makes it natural, grammatical, and easy to read. It should read though written by native speaker. is independent of other qualities such factual accuracy.

---

Published as a conference paper at ICLR 2026
In this work, we propose a fluency-aware post-training method that leverages on-policy reinforcement learning to align language models for lower-resource languages without compromising their linguistic naturalness. Crucially, never train the model any translated responses, preserving the fluent generation capabilities learned during pretraining. We demonstrate that even a disfluent judge model can successfully guide fluent policy, long as the judge understands the target language well enough to evaluate response quality.
To validate this approach, we need a language that is genuinely under-resourced for post-training yet has enough infrastructure for rigorous evaluation. Norwegian (Bokmål), a language with about million speakers, fits this profile well: lacks manually written instruction datasets, yet the research community has pretrained several Norwegian language models that can be leveraged for post-training Lastly, we employed five native Norwegian speakers for accurate manual evaluation of fluency.
In the native-speaker evaluations, our on-policy approach is preferred over supervised finetuning on machine-translated data in 67.5% of pairwise comparisons while requiring no instruction-tuning data in Norwegian. Our main contributions are:
We propose an on-policy reinforcement learning method for post-training in lower-resource languages that maintains fluency without requiring any instruction datasets in the target language.
We demonstrate through extensive human evaluation with native Norwegian speakers that on-policy training produces more fluent models than supervised finetuning on translated data.
We show that fluent aligned models can be bootstrapped using disfluent judges, enabling post- training for languages without existing fluent instruction-tunel models.
We provide comprehensive ablations revealing the critical importance of avoiding any exposure to translated responses even in small quantities during training.
FLUEENYY-WARR POST-TRAINING
This section describes our proposed method for post-training language models on a target lower- resource language without any dedicated dataset in that language. The overall approach consists of three stages: i) pretraining on the target language, ii) short SFT alignment on English, and iii) on-policy alignment on the target language.
The key principle is to never train the language model on any unnatural text.
Pretraining on target language The first stage is essential to ensure that the base language model learns all necessary linguistic knowledge and is able to generate fluent outputs in the target language. Our study focuses on the subsequent training stages and does not cover this pretraining stage that has already been studied in detail by Gururangan et al. (2020); Ibrahim et al. (2024); Kim et al. (2024); Samuel et al. (2025). Specifically for Norwegian, we build upon the multi-stage continual pretraining from Samuel et al. (2025).
Short SFT alignment on English The second stage is crucial for teaching the model to respond to user prompts and follow the specific chat format (Appendix E. Typically, supervised finetuning would be done on a carefully curated set of conversations in the target language, but such resources are usually not available for lower-resource languages. We therefore opt for a short alignment on a small high-quality English dataset. Specifically, use the 000 curated prompt-response pairs from LIMA (Zhou et al., 2023) and train the base model on this dataset for one epoch. The short training ensures that the model does not catastrophically forget its innate knowledge of the target language.
On-policy alignment on target language Finally, in the third stage, the language model is aligned to respond in helpful, truthful and safe way- without losing the fluency it acquired during the initial pretraining stage. This is achieved by training with online on-policy reinforcement learning where the model is trained solely on responses sampled from itself. In this way, the model is never pushed away from the subspace of fluent outputs it has learned to prefer during pretraining. A key observation is that we do not need to train any reward model as it suffices to use an LLM-as-a-judge system to provide the reward signal (Zheng et al., 2023); as long as the judge understands the targer language, it does not have to be fluent to produce a fluent policy. We evaluate this hypothesis later in Section 4.2. This effectively allows us to bootstrap fluent models in languages without any instruction datasets and without any existing fluent language models.
2

---

Published as a conference paper at ICLR 2026
2.1 ONLINE ON-POLICY REINFORCEMENT LEARNING
We now describe the components of our on-policy training: the reward signal, the policy gradient objective, KL regularization, and the distributed implementation. It is important to note that any implementation of online on-policy training with feedback from an LLM-as-a-judge system should work comparably well in terms of fluency of the final policy network our implementation is chosen to be simple and comparable to the baseline approach of supervised finetuning, as detailed later in Section 3. The algorithm does not directly optimize fluency: as detailed below, the reward signal captures the overall quality of responses based on the standard criteria such as correctness, safety or conciseness without explicitly targeting fluency (Appendix E).
Reward model The standard approach to on-policy alignment reinforcement learning from human feedback (RLHF; Christiano et al., 2017; Stiennon et al., 2020; Ouyang et al., 2022)- first trains Bradley-Terry reward model preference dataset from the target domain. Since we restricted to a lower-resource language, we assume that such a dataset is not available - instead, we rely on direct reinforcement learning from AI feedback (d-RLAIF: Lee et al., 2024).
In this scenario, we only use the domain knowledge to create a prompt template (constitution) that clearly guides a multilingual language model to judge the quality of responses. As evident from later results in Section 4.2, this setup provides enough signal that even disfluent judge can train fluent and capable policy - as long as the judge has some level of understanding of the target language.
Objective function Let us start with defining the objective J to maximize during this post-training stage. The objective in Equation (1) states that we want to find parameters of our language model To (the policy model) that maximize the reward given to a prompt æ (æ1, 22, ... F/p)) and its corresponding response y = (J1, 1/2, -.)))))) sampled from the policy me:
argmax ((0) df argmax
E ()(
r(x, y)
Following the majority of works on LM alignment, we optimize the objective with policy gradient methods (Williams, 1992: Sutton et al., 2000) that perform gradient descent on - -o (0) using online on-policy samples y from the policy model me:
[o)
E D. ma(-
r(x, y) Ve log (v)
(2)
Directly using Equation (2) for training gives us the REINFORCE algorithm (Williams, 1992). To increase its convergence speed and stability, we modify the rewards and optimize advantages A(x, y) instead of r(æ, y). Following REINFORCE WITH BASELINE (Weaver & Tao, 2001). we subtract the baseline score b(x).. and following Karpathy (2016), we further normalize by the dispersion factor often estimated by
While these additional factors separately trained models (Konda & Tsitsiklis, 2000; Schulman et al., 2017: Christiano et al., 2017), we choose more straightforward approach and estimate them as the sample mean and the sample standard deviation over G responses (Kool et al., 2019; Ahmadian et al., 2024; Shao et al., 2024):
s(x); giving us A(x, y de (r(=))))b(())))fd
A(æ, y) df T(x, B) - mean (r(x 3 (0) ()
std (fr(z, (ilizr.y())) y(1)())
Putting Equation (2) and Equation (3) together, we can define the loss function L(O, æ) of sin- gle query x for optimizing the parameters 0 of policy тg. One important detail is to account for the potential length bias (Liu et al., 2025) when computing the log-likelihood log no (y(y)))) as Ee log (U) ar, y<ン)) and normalizing by 3/1vl. As proposed by Rastogi et al. (2025), the length bias can be mitigated by dividing by the total length of responses (g(1) Ji=1 to a prompt æ. Then the token-level loss function becomes:
E D (-1
Σ
A(x y (0)
Σ
(1) l., v Vo log To To(y
(4)
Elyy
Note that unlike most current work on LM alignment, we do not modify the loss function to account for sampling from a proximal policy (by clipping outlier samples and importance sampling) in PPO
3

---

Published as a conference paper at ICLR 2026
(Schulman et al., 2017). As detailed below, the synchronous parallelization makes all samples almost fully on-pont on-poiicy and we have found no benefit in moving away from the simple REINFORCE loss function in Equation (4).
KL-divergence regularization well known issue of policy gradient methods in RLHF that they optimize model-based rewards instead of the true (and unknown) rewards. When the policy is modeled by a large language model with billions of parameters, it can very quickly find shortcuts in the reward model and reward-hack its measured performance while degrading its true performance. The most common way to mitigate this issue is to add a soft constraint to the optimization problem that pushes the policy stay close to its original state the distance is usually measured by the KL divergence and the strength of the constraint parametrized by 8 (Kullback & Leibler, 1951; Jaques et al., 2019):
argmax J(0)= def
argmax
E ro((a))
(((x, y) - 33xx. [To( x)) |I To [+)]
(5)
In order to optimize the objective I according to the new definition instead of the simpler one from Equation (I), we need to introduce an additional loss term LK (weighted by B) that will push the trained policy closer to the output distribution Te...
The problem with KL divergence is that its exact computation is intractable most cases. This means that have to approximate it in practice, the most common way is to simply use the already sampled prompts with responses and compute a direct Monte-Carlo estimate of Ex D, Ta( (-(r)rr) [log xe (t) (y)w)) as done in the seminal RLHF work by Stiennon et al. (2020). While straightforward, this approximation is very rough and ill-behaved, even becoming negative sometimes.
These estimates use only a small fraction of information available in the probability distributions given by To =-(- | y-si) - only the single value of To T xe, yei)). Instead, can get provably tighter estimates when we Rao-Blackwellize the Monte-Carlo estimation by using the full next-token distributions over the vocabulary V. Amini al. (2025) prove that this estimation is unbiased and has lower variance than the standard Monte-Cart estimation.
ly
To(y: w yc
(f) de1
E
Toly: x. yei) log
(6)
~ D
To (3)
w
I, yci)
Daa (.
me
m.Mc:]]
The computational overhead of the Rao- Blookllacwellieed estimate negligible because it still requires only a single cerward-backweer pass through the policy model that is done even without any KL regularization. Another benefit of regularizing the full output distribution that it eliminates the need for another loss term for regulating the output entropy as used in most RLHF works - further simplifying the training method.
Distributed setup As opposed to supervised finetuning, RL approaches need several language models be fully materialized and used the same time (the trained policy, the reference policy, the sampled policy and the reward model). In principle, these models should be run sequentially in cycle (Figure 1), but that is inefficient in practice and the cycle needs to be broken and parallelized. As illustrated below in Figure 1, this can be achieved by postponing the update of the sampled policy effectively turning the training slightly off- off-policy.
Unlike other approaches to distributed RL (Espeholt et al., 2018; Noukhovitch et al., 2025; Rastogi et al., 2025), our parallelization is completely synchronous. This can make the resources allocated for sampling underutilized (all workers have to wait until the longest response completes), but that does not impact the overall efficiency much as most resources are allocated to the reward models. On the other hand, our samples are unbiased (asynchronous approaches typically up-sample problems with short responses), and the completely synchronous training cycle simplifies the implementation, as well as the objective function. Since the samples are guaranteed to be off off-policy by just three steps, we can still rely on on-policy training techniques without having to resort to more complicated and less stable proximal-polig methods such PPO (Schulman al., 2017).
4

---

Published as a conference paper at ICLR 2026
sampled
reaard mode
trained policy
sampled
nenord morkl
trained
pulicy
policy
sugple arwast
sampled policy
mwand
samphed
spabiod
trained
polcyy
waget
polcy
simpled
policy
Figure I: Left: Reinforcement-eeamine cycle. This diagram demonstrates the sequential nature of online RL training: each training step starts by sampling new responses from the policy model, followed by sampling risponse-judgmett
from the reward model, and then updating the weights of the policy model based on the sampled responses and rewards. Right: Parallelization. Breaking the cycle and postponing the update of the sampled policy allows for running all three models the same time (vertically-ae alliggee blocks are ran concurrently on different GPU nodes).
3 EXPERIMENT: ONLINE ON-POLICY TRAINING MAINTAINS FLUENCY
The main experiment of this paper aims to answer the central question: Does online on-policy training produce more fluent language models than supervised finetuning on translated data? To answer this, we designed the experiment to make on on-policy training as similar to supervised finetuning as possible using the same base model, the same training data, and the same number of training samples. As a case study for models trained on lower-resourge language, we trained all models on Norwegian Bokmål: then we asked five native speakers do pair-wise fluency comparisons of outputs generated from these models.
In total, compare three post-training approaches represented by three language models that based on the same pretrained model, as illustrated in Figure 2:
NorMistral trunslated SFT
NorMistral base
MirMiat English SFT
continaal buintig
Mistral base
M. algineeer
NorMistral Norwegion
instruct
Lidda ressrrd saud!!
Figure 2: Genealogy of the compared models. The three models compared in the main fluency experiment (highlighted in bold boxes) all originate from a single base model Mistral Nemo 12B (left). The official instruction-tuned version of this model also serves as the judge for our ciiforcementtlearming approach.
Approach 1: Norwegian RL We follow the method described in Section 2.1 when training this model. More specifically, we start from NorMistral 11B, a continually pretrained Norwegian base model from Samuel al. (2025), and supervise-finetune it for a single epoch the 000 English conversations from LIMA (Zhou et al., 2023). Then, in the final reinforcement-leamie the same dataset as the second approach: No Robots by Rajani al. (2023). each training step, we take 128 translated prompts from this dataset, sample a group of 8 responses for each prompt. Each response then graded by judge that also has access to the gold English response provided in the No Robots dataset the judge prompt itself is attached in Appendix E. In order to rule out the possibility that the policy acquires its fluency from the judge model, we use Mistral Nemo 12B as the judge - model with the same origin as the policy itself that is also evaluated in this experiment as the approach number three. More technical training details are described in Appendix C. Many of these choices are ablated later in Section - for example, it is clear that Mistral Nemo is poor reward model compared to larger language models, which should not, however, influence the fluency of the trained policy (Section 4.2).
stage, we use
5

---

Published as a conference paper at ICLR 2026
Approach 2: Translated SFT As illustrated in Figure 2, the translated-SFT model is initialized from the same checkpoint as the first approach. Then we also finetune on the same machine- translated No Robots dataset however in this scenario, translated responses by minimizing their conditional negative log-likelihood (in the first approach, the non-translated responses only used as hints for the reward model). To translate the full No /oweer-llus-72 model currently the
directly finetune the model the
Robots dataset to Norwegian Bokmål, we use the state-of-the-an open-weights translation model with explicit support for Norwegian (Rei et al., 2025). Other strong translation models are ablated later in Section 4.5.
Approach 3: Mistral Nemo The last tested model is
by Jiang et al. (2024a). Including it in this experiment serves three purposes: firstly, to give reference baseline from an externally trained model; secondly, to test the fluency of the reward model that has been used in the first approach; and thirdly, to evaluate the performance of the standard multilingual but English-focused - post-training that has been used most major releases of the latest language models. This model shares the same origin the previous two. However, while its weights are openly available, no details about its post- wost-training process have been published.
Manual annotation of generated outputs Fluency is language feature that is difficult to accu- rately measure using statistical models, but it should be relatively straightforward to judge for native speakers of that language. We therefore hired five research assistants, all native Norwegian speakers, to compare the fluency of responses generated for a pool of prompts. The prompts are gathered from the Norwegian Bokmål mimir-instruct dataset from de la Rosa al. (2025), which consists of artificially generated responses to prompts written by native Norwegian speakers. Since we use this data primarily seeds for diverse generated outputs evaluate for fluency, we select the 100 queries with the longest gold responses as the seed prompts; to ensure sufficiently long responses for meaningful fluency comparison. Then we sample a single response from each of the three evaluated models using Nucleus sampling with temperature of 0.5, top_k of 64 and top_p of 0.9 (Holtzman al., 2020). The responses are formed into 3 pairs per prompt (one for each model pair), yielding 300 pairs in total. These pairs are then rated by the annotators in the A/B testing fashion Which is more fluent: response A, response B, or are they equally fluent? Each annotator is presented with the 300 pairs of responses in randomized order; going through the full set of pairs took each annotator roughly 15-20 hours. We give more details about the actual annotation guidelines and the overall process in Appendix D. The resulting dataset (with anonymized annotator names) is published online at https://hf /hf co/ wttttattstttgg//ee http://oorrsrraarr.oeoooo wwlalllllln---aaleeeee Flluncc--anotattion annotation.
Results We show the resulting win-rates from the manual annotation in Table 1. These win-rates calculated by going through all pair-wise comparisons and either giving the winning model (preferred by more annotators) single point giving both models half a point in case of a tie so-called 1/0.5/0 method of aggregation (Copeland, 1951).
Table Model win-rates according the manual fluency annotation. The values show the win- rate percentages of the row-wise models over the column-wise models. The last column aggregates the win-rates of each model.
Model
on-policy RL
translated SFT
Mistral Nemo
average
on-policy RL
67.5
91.8
79.7
translated SFT
32.5
87.5
60.0
Mistral Nemo
8.2
12.5
10.3
The results clearly show that the most-preferred responses are from the on-policy training method, followed by translated SFT and then by Mistral Nemo. This supports our main claim since it demonstrates that the policy can indeed outperform its judge on fluency. is worth noting that the first two approaches produced very fluent outputs and the fluency of 1/s of their pair-wise comparisons was agreed to be 'equal' - based on qualitative inspection, the difference between them mostly stems from infrequent traces of translationese in the SFT outputs (Appendix F).

---

Published as a conference paper at ICLR 2026
4 FURTHER EVALUATIONS AND ABLATIONS
This section studies the effect of various post-training choices on the final performance in more detail. We scale up the manual fluency scoring from Section by introducing an automatic fluency estimate, and also focus more general performance of the trained models by incorporating Norwegian understanding and generation benchmarks.
4.1 EVALUATION METRICS
Automatic fluency evaluation We have to rely model- measurement of fluency to lower the cost of evaluating all experiments in this section. The Norwegian fluency model is trained like a standard Bradley- Terry reward model (Stiennon al., 2020) on a dataset 'D of paired preferred and non-preferred texts (æw. 2i) E DD Specifically, we add scalar linear head to pretrained language model (NorMistral 11B in our case) and finetune it by minimizing the following loss:
fluency (0) det
E Te Je
log (ro(xw)) -re(zi))
We create the training data by combining existing resources and newly synthesized texts. Firstly, we use the Norwegian ASK-GEC corpus of corrected language-leamer essays (Jentoft, 2023) - from this corpus, we take all sentences with mistakes as the non- con-prefereed texts and their (partially) corrected versions as the preferred texts. Secondly, we perform backtranslation with the OPUS collection of Norwegian-Englan Norwegian sentence from the Norwegian Dependency Treebank (NDT; Solberg et al., 2014) or the Norwegian Review Corpus (NoReC; Velldal et al., 2018), then we sample one model to translate the sentence to English and then sample another model to translate it back to Norwegian; finally, the original sentence is cast preferred and the backtranslated version non non-preferred. The ASK-GEC corpus trains the fluency scorer to take grammaticality into account while the second synthetic source focuses translationese and lexical issues.
machine translation models (Tiedemann et al., 2023): we sample clean
The fluency scorer can be directly evaluated by utilizing the manual annotations from the previous Section 3. Looking all instances where the annotators agreed that one response is preferred over another response, the fluency scores agree with this ranking in 85.5% of cases. This agreement is even slightly higher than the agreement among annotators when limiting their annotation non-ties (for comparability), they agree with the consensus in 83.2% of cases, which highlights how subjective the notion of fluency can be. To lower the variance of the fluency score, we sample 16 responses from each evaluated model and average the scores. The raw scores are sigmoid normalized into percentage values for clarity. When applied to the three approaches from the previous section, the fluency scores are 2.47 (92.2%) for on-policy RI., 1.94 (85.7%) for translated SFT, and 0.76 (65.3%) for Misiral Nemo, which correspond to the manual evaluation of these three models.
Natural language understanding (NLU) evaluation We use the aative-Norwegiat reading- comprehension task NorQuAD to assess the level of Norwegian language understanding (Ivanova et al., 2023). In order to account for the variable and conversational outputs of instruction-tune models, we evaluate the correctness of each generated response with an extra call to a judge, Llama 3.3 70B (Grattafiori et al., 2024), that compares it against the gold answer. This ensures that the evaluation is invariant to formatting variation of the generated outputs. We then report the average accuracy as the approximate Norwegian NLU score.
Natural language generation (NLG) evaluation In order to assess the Norwegian generative abilities, we use two Norwegian benchmarks that were designed specifically for this purpose: Nor- Rewrite and NorSummarize (Mikhailov et al., 2025); these benchmarks test the instruction-followiin creative writing, and summarization abilities of Norwegian language models. We follow the original implementation of these benchmarks, which evaluates the quality of each response by comparing it pairwise with another response and automatically judging it with Llama 3.3 70B. We report the win-rate percentages of individual models against the smallest evaluated model, Llama 3. 1 8B (Grattafiori et al., 2024).
7

---

Published as a conference paper at ICLR 2026
4.2 FLUENT POLICY DOES NOT NEED A FLUENT REWARD MODEL
The main experiment has shown that even a disfluent judge (Mistral Nemo 12B) can produce a policy that is substantially more fluent than the judge itself. In this section, investigate this phenomenon more thoroughly by checking that is not an anomaly and that holds for diverse range of judges. First, we evaluate each judge on the three benchmarks described above: natural language understanding (NLU), generation (NLG), and fluency. Then, using a judge to provide the reward signal, we train a policy with the same method as Section 3, and evaluate the policy on fluency to assess the effect of the choice of judge.
Table 2: The effect of the judge's knowledge of Norwegian on the trained policy. The table shows the average Norwegian understanding, generation and fluency scores (Section 4) for different judges, and the fluency of the policy models trained with reward signals from these judges. The fluency scores are color-coded so that disfluent models are red and fluent models are blue.
Judge performance
Fluency of trained policy
Judge
NLU
NLG
Fluency
Mistral Nemo 12B
87.5
29.7
67.0
92.2
Mistral Large 123B
90.0
70.4
83.4
94.2
Mixtral 8x22B
91.3
20.2
70.9
92.1
Llama 3.1 8B
86.4
50.0
62.8
92.9
Llama 3.3 70B
90.7
57.7
84.2
93.5
Qwen 2.5 14B
89.6
43.5
39.0
93.1
Qwen 2.5 32B
91.7
59.9
43.2
93.9
Qwen 2.5 72B
92.0
75.2
50.7
92.9
Results We have evaluated language models of different sizes and different levels of Norwegian knowledge three Mistral models (Jiang et al., 2024a;b), three Qwen models (Qwen al., 2025), and two Llama models (Grattafiori et al., 2024) the results of the evaluation are shown in Table 2. Comparing the fluency scores of judges with policies, there is apparent correspondence (the Pearson correlation coefficient is 0.067); the policies are fluent regardless of the (dis (dis)fluency of their judge. There is also clear relation between the other measures of judge quality on the resulting fluency. We hypothesize that fluency remains stable because the policy is trained exclusively on its own samples, which are fluent thanks to targeted pretraining. The choice of judge model appears to affect other response qualities but not fluency.
4.3 THE EFFECT OF TRAINING LENGTH ON FLUENCY
We further validate the claim that fluency is consistently stable for on on-policy training by looking at the change in fluency score throughout training.
Fluency throughout training
(b) NLG performance throughout training
Online RL
Translated SFT
Online RL
Translated SFT
91
40
90
20
86
50 000
10
500000
080
Traising amplly
Training samples
Figure 3: Fluency and NLG scores throughout training. We measure the performance score every
25 training steps for the reinforcement learning (in blue) and every epoch for the SFT training.
8

---

Published as a conference paper at ICLR 2026
Results Figure 3 shows the result of training a policy supervised by Mistral Large and evaluating its fluency and NLG performance every 25 training steps. The fluency score remains stable around 93% after the initial convergence in the first 50 training steps. Upon closer inspection, the lower initial fluency score of 87 37.5% can be explained by the increased likelihood of responding in English (not by disfluent Norwegian per se) triggered by the previous English SFT stage; then the model learns to consistently respond in Norwegian, which leads to the apparent increase in fluency (see Appendix F). On the other hand, when looking the change of fluency throughout SFT training on translated Norwegian, we can see a clear decrease in fluency from the initial starting point, which can only be attributed to responding with consistent but slightly disfluent- - Norwegian. The longer cinforcement-eerriinn run also allows us to look at the development of the directly optimized training metries; these are appended in Appendix B.
4.4 THE IMPORTANCE OF NOT TRAINING THE POLICY ON ANY TRANSLATED RESPONSES
The previous experiment has highlighted the importance of the initial SFT stage and the ability of the policy model to recover from the sudden shift to English. As described earlier, we only train on the 1 1000 examples (31 training steps) from the LIMA dataset (Zhou al., 2023) in the initial SFT stage. The results in Table 3 suggest that training on more samples can slightly degrade fluency. Even clearer fluency deterioration is seen after amount of translated data can introduce a measurable amount of disfluencies in the final policy.
the dataset from English even a small
4.5
IMPACT OF TRANSLATION QUALITY ON SET PERFORMANCE
Even though we have selected Tower-Plus as the state-of oftte-of-th-art model for translation from English to Norwegian (Rei et al., 2025), possible explanation of our results is that the observed disfluencies are specific to this particular translation model rather than a general phenomenon. Thus, we translate the No Robots dataset with several alternative models, finetune on each translation, and compare the resulting fluency. We only test translation models that explicitly support English and Norwegian, which includes LLM-based translation models, massively multilingual models, and traditional small model trained specifically for translation. The fluency scores in Table 4 confirm that Tower-Plus, as the largest available translation model, results in the most fluent policy.
Table 3: The effect of the initial SFT stage. We ablate various settings and measure the fi- nal fluency when starting reinforcement learn- ing from such SFT checkpoints.
Table 4: The effect of using different machine- translation models. We train models on the trans- lated dataset and measure their fluency.
Translation model
Size
Fluency
SFT settings
RL fluency
Tower-Plus 2025)
72.7B
85.7
English dataset (I epoch)
94.2
MADLAD-400 (Kodagunta al. 2023)
10.7B
82.4
English dataset (2 epochs)
93.2
Sced-X (Ching et al., 2025)
7.5B
73.4
English dataset (4 epochs)
92.8
NLLB-200 200 (Team et al., 2022)
3.3B
75.5
Translated dataset epoch)
91.0
OPUS Eng-Gem (Tiedemans et al., 2023)
0.1B
68.2
5 RELATED WORK
Post-training on lower-resource languages As mentioned above, there have been numerous works that focused on supervised finetuning of language models for lower-resource languages (Suzuki et al., 2023; Pipatanakul et al., 2023, inter alia), substantially less attention has been paid to reinforcement learning for preference optimization in such languages (Lai et al., 2023; Dang et al., 2024). While fluency in less-spoken languages is significant limitation of current language models, there has not been much interest from the research community in this topic- - likely because accurate fluency evaluation is difficult. Dang et al. (2024) consider fluency performance, but only include as part of a general LLM-as-a-judee evaluation prompt, whose accuracy is not validated. Zhang al. (2025) use human annotators to evaluate fluency of their models, but only after first back- tracck-tanslating the generated responses to English. A recent paper by Sainz et al. (2025) focuses the effect of post-training configuration fluency, similarly to
important aspect of multilingual

---

Published as a conference paper at ICLR 2026
our work- - they have fluency assessed by native (Basque) speakers, but only consider the effect of different data mixtures for supervised finetuning.
Translationese and translation artifacts The phenomenon of translationese (Gellerstam, 1986; Baker et al.. 1993; Toury, 1995) is well-documented in both human and machine translation (Koppel & Ordan, 2011; Volansky et al., 2013; Vanmassenhove et al., 2019; Bizzoni et al., 2020). Wang et al. (2023) have demonstrated that translationese in training data impacts performance in cross-lingun summarization, with models trained on translated data producing outputs that differ systematically from those trained on original text. This finding motivates our approach: rather than trying to mitigate translationese artifacts during training, we avoid them entirely by never training on any translated text. The concurrent work by Kunz (2026) studies the effect of fine-tuning on -taassae-t Swedish and, similarly to us, found that it leads to rapid loss of fluency.
generalization framework demonstrates that strong models consistently outperform their weak supervisors when fine-tuned on the labels generated by the weak supervisors (Burns et al., 2024). Following these empirical findings, Charikar et al. (2024) has proposed a theoretical frameworks that explains this phenomenon. It is particularly relevant to our setting: just as strong models can exceed the capabilities of weak supervisors, our Norwegian-prest pretrained models can achieve fluency beyond what the quality of the supervision signal would suggest.
Weak-to-strowe generalization The
RLAIF and the mudge-policyy disconnect The distinction between the capabilities of the reward model and the policy model has been explored in the RLAIF literature. Lee et al. (2024) discovered that RLAIF achieve comparable performance RLHF even when the LLM judge is not strong enough, which is similar to our finding that disfluent judges can guide fluent policy models. More generally, the disconnect between evaluation and performance is well-documented: LLM judges can accurately predict human preferences even when they cannot generate equally high-quality outputs themselves (Zheng al., 2023). The key insight is that evaluation is often a simpler, more focused task than generation a principle that explains why disfluent judges can still provide useful training signals for Norwegian responses. Furthermore, recent work on self-rewarding models shows that L.MSs can iteratively improve through self-evaluation, further demonstrating that the ability to discriminate quality does not require the ability to generate at that quality level (Yuan et al., 2024). This asymmetry between recognition and generation capabilities suggests why approach works: the judge models only need to identify responses that better satisfy the task requirements; they do not need to generate fluent Norwegian themselves.
Cross-lingual transfer and unsupervised translation Our work can be connected to several lines of research on learning from imperfect supervision. The literature on zero-shot cross- liogsll transfer has extensively studied how models trained on one language can perform in other languages without direct supervision. Pires et al. (2019) have demonstrated that multilingual BERT models fine-tuned on one language can achieve reasonable zero-shot performance on tasks in other language. More recent work has shown that this transfer can be improved through better pretraining (Conneau et al., 2020; Xue et al., 2021) and cross-lingual alignment techniques (Artetxe al., 2020).
CONCLUSION
In this work, we demonstrated that on-policy reinforcement learning offers a practical path to creating fluent aligned language models for lower-resource languages without requiring any instruction-tuning datasets in the target language. Through extensive evaluation with native Norwegian speakers, we showed that our approach produces more fluent models than the standard practice of supervised finetuning on pacchnn--ranslatee data, achieving a 79.7% 7% win-rate compared 60.0% for translated SFT and 10.3% for the multilingual Mistral Nemo baseline. Our experiments revealed two critical insights. First, that avoiding any exposure to translated text during training is essential for maintaining native-level fluency even minimal exposure to translated responses measurably degrades fluency. Second, that fluent policies can be successfully trained using disfluent judge models, as long as the judge has sufficient understanding of the target language to evaluate response quality. We hope this work will facilitate the development of high-quality language models for the hundreds of lower-resource languages that currently lack instruction-tuning datasets.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
We would like thank Helene Brodin, Ørjan Oftedal Hanasand, Truis de Lange, Nikolas Hemer Martin and Mina Sheikhi for their great work with manually annotating Norwegian fluency. This paper would have been much harder to complete without the help and Norwegian expertise of Petter Mæhlum.
The computations were performed on resources provided through Sigma2 the national research infrastructure provider for pigh-performance computing and large-scale data storage in Norway. We acknowledge Norway and Sigma2 for awarding this project access to the LUMI supercomputer, owned by the EuroHPC Joint Undertaking, hosted by CSC (Finland) and the LUMI consortium through project 465001890.
The efforts described in this paper were jointly funded by the University of Oslo and the HPLT project (High Performance Language Technologies; coordinated by Charles University). Furthermore, this work supported by industry partners and the Research Council of Norway with funding to MediaFutures: Research Centre for Responsible Media Technology and Innovation, through the centers for Research-based Innovation scheme, project number 309339 and Integreat Norwegian Center for Knowlege-driven Machine Learning, project number 332645.
REPRODUCIBILLIY STATEMENT
To ensure reproducibility of our work, we described the training method in Section 2, provided full hyperparameter settings in Appendix C and we openly release our custom training code https://gi thub. com/ltgoslo htress//ooomssss--ooot---- and freely distributed Python libraries: torch. v1lm and transformers. Some evaluations use model-based fluency score that is released alongside the paper at https://hf co co/dttastss/ttg/ normistral aaussraalllleenc--mmtattttt
The training code is based on common
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadace, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basics: Revisiting REINFORCE-styl optimization for learning from human feedback in LLMs. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 12248-12261 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 1,228633/1/20222aclloooo
Afra Amini, Tim Vieira, and Ryan Cotterell. Better estimation of the kl divergence between language models, 2025.
Mikel Artetxe, Sebastian Ruder, and Dani Yogatama. On the cross-lingual transferability of monolingual representations. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 4623-4637 Online, July 2020. Association for Computational Linguistics. doi: 18.13653/v1/ 2020.acl-main.co
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie KerI, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds. Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional AI: Harmlessness from AI feedback. 2022.
Mona Baker, Gill Francis, and Elena Bognini-Bonelly Corpus Linguistics and Translation Studies: Implications and Applications, pp. 233-285 233-250. John Benjamins Publishing Company, Netherlands, 1993.
11

---

Published as a conference paper at ICLR 2026
M Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham Abdullah Alyahya, Sultan AlRashed, Faisal Abdulrahman Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar. Raghad Alkhathran, Yousef Almushaygih, Raneem Alnajim, Salman Alsubaihi, Maryam AI Mansour, Saad Amin Hassan, Dr. Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, Ahmed Abdelali, Jeril Kuriakose, Abdalghani Abujabal, Nora Al- Twairesh, Areeb Alowisheq, and Haidar Khan. ALLam: Large language models for arabic and english. In The Thirteenth International Conference on Learning Representations, 2025.
Yuri Bizzoni, Tom S Juzek, Cristina Sspaña-Bonet, Koel Dutta Chowdhury, Josef van Genabith, and Elke Teich. How human is machine translationese? Comparing human and machine translations of text and speech. In Marcello Federico, Alex Waibel, Kevin Knight, Satoshi Nakamura, Hermann Ney, Jan Niehues, Sebastian Stüker, Dekai Wu, Joseph Mariani, and Francois Yvon (eds.), Proceedings of the 17th International Conference on Spoken Language Translation, pp. 280-29000000 290, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/v1/ 2020.iwslt- 1.34.
Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Bowen Baker, Leo Gao, Leopold Aschenbrenner, Yining Chen, Adrien Ecoffet, Manas Joglekar, Jan Leike, Ilya Sutskever, and Jeff Wu. Weak-to- strong generalization: eliciting strong capabilities with weak supervision. In Proceedings of the 41st International Conference on Machine Learning. ICML'24. JMLR.org. 2024.
Moses Charikar, Chirag Pabbaraju, and Kirankumar Shiragur. Quantifying the gain in weak-to- strong generalization. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Information Processing Systems, volume 37, pp. 126474- 126499. Curran Associates, Inc., 2024. doi: 10.52202/099017-4017.
Shanbo Cheng, Yu Bao, Qian Cao, Luyang Huang, Liyan Kang. Zhicheng Liu, Yu Lu, Wenhao Zhu, Jingwen Chen, Zhichao Huang, Tao Li, Yifu Li, Huiying Lin, Sitong Liu, Ningxin Peng, Shuaijie She, Lu Xu, Nuo Xu, Sen Yang, Runsheng Yu, Yiming Yu, Lichao Zou, Hang Li, Lu Lu, Yuxuan Wang, and Yonghui Wu. Seed-x: Building strong multilingual translation Ilm with 7b parameters, 2025.
Hasna Chouikhi, Manel Aloui, Cyrine Ben Hammou, Ghaith Chaabane, Haithem Kchaou, and Chehir Dhaouadi. GemmAr: Enhancing LLMs through Arabic cestruction-tunigg 2024.
Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, volume 30, pp. 2299-4307 Curran Associates, Inc., 2017.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: 1e.11665/11/2220... actp://00..cimmmii.........
A. Copeland. A reasonable social welfare function. In Seminar on Applications of Mathematics to Soocial Sciences, University of Michigan, Ann Arbor, 1951. Mimeo.
John Dang, Arash Ahmadian, Kelly Marchisio, Julia Kreutzer, Ahmet Üstün, and Sara Hooker. RLHF can speak many languages: Unlocking multilingual preference optimization for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 13134-13156, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 11885//1/2024..mm// main. 729.
Javier de Ia Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhlum, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Hans Christian Farsethás, Svein Ame Brygfjeld, Jon Atle Gulla, Stephan Oepen. Erik Velldal, Wilfred Østgulen, Lilja Øvrelid, and Aslak Sira Myhre. The impact of copyrighted material on large language models: A Norwegian perspective. In Richard Johansson and Sara Stymne (eds.), Proceedings of the Joint 25th Nordic Conference on Computational Linguistics and 11th
12

---

Published as a conference paper at ICLR 2026
Baltic Conference on Human Language Technologies (NoDaLiDa/Baltic 2025), pp. 544 560, Tallinn, Estonia, March 2025. University of Tartu Library. ISBN 199-0-0------...
Koel Dutta Chowdhury, Rricha Jalota, Cristina Sopaña-Bonel and Josef Genabith. Towards debias- ing translation artifacts. In Marine Carpuat,
de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3983- 3991, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022 hww.://.aaalll.llaeeeaaaaaaaaeaa man.cl-m.i...2..
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Robert Dunning, Shane Legg, and Koray Kavukcuoglu. IM- PALA: Scalable distributed deep-RL with importance weighted actor-learner architectures, 2018.
Martin Gellerstam. Translationese in Swedish novels translated from English. In L. Wollin and H. Lindquist (eds.), Translation studies in Scandinavia: Poceedings from the Scandinaviar Symposium on Translation Theory (SSOTT) II, number 75 in Lund Studies in English, pp. 88-95. CWK Gleerup, Lund, 1986.
Aaron Grattafiori Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama herd of models. arXiv preprint arXiv:2407.21783 2024.
Yanzhu Guo, Simone Conia, Zelin Zhou, Min Li, Saloni Potdar, and Henry Xiao. Do large language models have an English accent? Evaluating and improving the naturalness of multilingual LLMs. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguisties (Volume I: Long Papers), pp. tional Linguistics. ISBN 979-8-89176-251-0 doi: 10.18653/v1/22 1/2021 aal.:on.cc--lo..199
Vienna, Austria, July 2025. Association for Computa-
Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. Don't stop pretraining: Adapt language models to domains and tasks. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8342 8360, Online, July 2020. Association for Computational Linguistics. doi: 10. 00.18653/11/202..co www.:aaa..lmmmmmmmmm.m main.740.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020.
Adam Ibrahim, Benjamin Thérien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timothée Lesort, and Irina Rish. Simple and scalable strategies continually pre-train large language models. Transactions Machine Learning Research, 2024. ISSN 2835-8856.
Sardana Ivanova, Fredrik Andreassen, Matias Jentoft, Sondre Wold, and Lilja Øvrelid. NorQuAD: Norwegian question answering dataset. In Tanel Alumäe and Mark Fishel (eds.), Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa), pp. 159-168, Tórshavn, Faroe Islands, May 2023. University of Tartu Library.
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Àgata Lapedriza, Noah J. Jones, Shixiang Shane Gu, and Rosalind W. Picard. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. In NeurIPS Workshop on Conversational Al: Today's 's Practice and Tomorrow's 's Potential, 2019.
Matias Jentoft. Grammatical error correction with byte-level language models. Master's 's thesis, University of Oslo, 2023.
Albert Jiang, Alexandre Sablayrolles, Alexis Tacnet, Alok Kothari, Antoine Roux, Arthur Mensch, Audrey Herblin-Stoop Augustin Garreau, Austin Birky, Bam4d, Baptiste Bout, Baudouin de Mon- icault, Blanche Savary, Carole Rambaud, Caroline Feldman, Devendra Singh Chaplot, Diego de las Casas, Eleonore Arcelin, Emma Bou Hanna, Etienne Metzger, Gaspard Blanchet, Gianna Lengyel,
13

---

Published as a conference paper at ICLR 2026
Guillaume Bour, Guillaume Lample, Harizo Rajaona, Henri Roussez, Hichem Sattouf, lan Mack, Jean-Malo Delignon, Jessica Chudnovsky, Justus Murke, Kartik Khandelwal, Lawrence Stewart, Louis Martin, Louis Temon, Lucile Saulnier, Lélio Renard Lavaud, Margaret Jennings, Marie Pellat, Marie Torelli, Marie-Anne Lachaux, Marjorie Janiewicz, Mickaël Seznec, Nicolas Schuhl, Niklas Muhs, Olivier de Garrigues, Patrick von Platen, Paul Jacob, Pauline Buche, Pavan Kumar Reddy, Perry Savas, Pierre Stock, Romain Sauvestre, Sagar Vaze. Sandeep Subramanian, Saurabh Garg, Sophia Yang. Szymon Antoniak, Teven Le Scao. Thibault Schueller, Thibaut Lavril, Thomas Wang, Théophile Gervet, Timothée Lacroix, Valera Nemychnikova, Wendy Shang, William El Sayed, and William Marshall. Mistral NeMo: A 12B instruction-tunet large language model. pttps://mistte ai wt/nnww/mmsstralleeeeeea Apache 2.0 License.
July 2024a. Mistral Al and NVIDIA collaboration.
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie- Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024b.
Andrej Karpathy. Deep reinforcement learning: Pong from pixels, 2016.
Seungduk Kim, Seungtaek Choi, and Myeongho Jeong. Efficient and effective vocabulary expan- sion towards multilingual large language models, 2024.
Vijay Konda and John N. Tsitsiklis. Actor-critic algorithms. In Advances Neural Information Processing Systems 12, pp- 1014- MIT Press, 2000.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 REINFORCE samples, get a baseline for free! In ICLR 2019 Workshop on Deep Reinforcement Learning meets Structured Prediction (DeepRiSStrucPredds ICLR), 2019.
Moshe Koppel and Noam Ordan. Translationese and its dialects. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.). Proceedings of the 49th Annual Meeting of the Association for Com- putational Linguistics: Human Language Technologies. pp. 2318-1326, Portland, Oregon, USA, June 201 1. Association for Computational Linguistics.
Sneha Kudugunta, Isaac Caswell, Biao Zhang, Xavier Garcia, Derrick Xin, Aditya Kusupati, Romi Stella, Ankur Bapna, and Orhan Firat. MADLAD-400: multilingual and document-level large audited dataset. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
S. Kullback and R. A. Leibler. On information and sufficiency. The Annals of Mathematical Statistics, 22(1 22(1):79-866 79, 1951. doi: 10. 20.1214//4 114//////11/700 177729994.
Jenny Kunz. Preferences for idiomatic language are acquired slowly and forgotten quickly: A case study on Swedish, 2026.
Viet Lai, Chien Nguyen, Nghia Ngo, Thuat Nguyen, Franck Dernoncourt, Ryan Rossi, and Thien Nguyen. Okapi: Instruction-tun -unded large language models in multiple languages with rein- forcement learning from human feedback. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, pp. 3 318-327, Singapore, December 2023. Association for Computational Linguistics. doi: 10. /22023.emmll-demo.....
Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, and Sushant Prakash. RLAIF vs. RLHF: scaling reinforcement learning from human feedback with AI feedback. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org. 2024.
Junghwan Lim, Gangwon Jo, Sungmin Lee, Jiyoung Park. Dongseok Kim, Jihwan Kim, Junhyeok Lee, Wai Ting Cheung, Dahye Choi, Kibong Choi, Jaeyeon Huh, Beongyu Kim, Jangwoong Kim, Tachyun Kim, Haesol Lee, Jeesoo Lee, Dongpin Oh, Changseok Song, and Daewon Suh. Expanding foundational language capabilities in open-source LLMs through a Korean case study, 2025.
14

---

Published as a conference paper at ICLR 2026
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding R1-zero-like training: A critical perspective, 2025.
Vladislav Mikhailov, Tita Enstad, David Samuel, Hans Christian Farsethas, Andrey Kutuzov, Erik Velldal, and Lilja Øvrelid. NorEval: A Norwegian language understanding and generation evaluation benchmark. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Moham- mad Taher Pilehvar (eds.), Findings of the Association for Computational Linguisties: ACL 2025, pp. 3495- - Vienna, Austria, July 2025, Association for Computational Linguistics. ISBN 179-8-89176-256-5. doi: 10. 18663/11/2025.... 266555//222225.mnnnns-a.l11111 fin.i.
Xuan-Phi Pai Nguyen, Wenxuan Zhang. Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang. Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang. Chaoqun Liu, Hang Zhang, and Lidong Bing. SeaLLMs large language models for Southeast Asia. In Yixin Cao, Yang Feng, and Deyi Xiong (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 294 3044 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/22 acl-mmossss
Michael Noukhovitch, Shengyi Huang. Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Faster, more efficient RLHF through off-policy asynchronous learning. In The Thirteenth International Conference on Learning Representations, 2025.
Long Ouyang. Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback. In Proceedings of the 36th International Conference on Neural Information Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Kunat Pipatanakul, Phatrasek Jirabovonvisut, Potsawee Manakul, Sittipong Sripaisarumongkol. Ruangsak Patomwong, Pathomporn Chokchainant, and Kasima Tharppipithhai. Typhoon: Thai large language models. arXiv preprint arXiv:23/2.13955 2023.
Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual BERT? In Anna Korhonen, David Traum, and Lluís Märquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4996-5001, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10. 11613//1//19--//// 1493.
Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
Nazneen Rajani, Lewis Tunstall, Edward Beeching, Nathan Lambert, Alexander M. Rush, and Thomas Wolf. No robots. https: https://hugginggfaee httpg/aa.eecccaaaaae ht/d/daaastt//uugggge ngF acooos 2023.
Leonardo Ranaldi and Giulia Pucci. Does the English matter? Elicit cross-lingual abilities of large language models. In Duygu Ataman (ed.), Proceedings of the 3rd Workshop on Multi- lingual Representation Learning (MRL), pp. 173-183, Singapore, December 2023. Association for Computational Linguistics. doi: 11118653/112222..mm
Abhinav Rastogi, Albert Q. Jiang, Andy Lo, Gabrielle Berrada, Guillaume Lample, Jason Rute, Joep Barmentlo, Karmesh Yadav, Kartik Khandelwal, Khyathi Raghavi Chandu, Léonard Blier, Lucile Saulnier, Matthieu Dinot, Maxime Darrin, Neha Gupta, Roman Soletskyi, Sagar Vaze, Teven Le Scao, Yihan Wang, Adam Yang. Alexander H. Liu, Alexandre Sablayrolles, Amélie Héliou, Amélie Martin, Andy Ehrenberg. Anmol Agarwal, Antoine Roux, Arthur Darcet, Arthur Mensch, Baptiste Bout, Baptiste Rozière, Baudouin De Monicault, Chris Bamford, Christian Wallenwein, Christophe Renaudin, Clémence Lanfranchi, Darius Dabert, Devon Mizelle, Diego de Las Casas, Elliot Chane-Sane, Emilien Fugier, Emma Bou Hanna, Gauthier Delerce, Gauthier
15

---

Published as a conference paper at ICLR 2026
Guinet, Georgii Novikov, Guillaume Martin, Himanshu Jaju, Jan Ludziejewski. Jean-Had Chabran, Jean-Malo Delignon, Joachim Studnia, Jonas Amar, Josselin Somerville Roberts. Julien Denize, Karan Saxena, Kush Jain, Lingxiao Zhao, Louis Martin, Luyu Gao. Lélio Renard Lavaud, Marie Pellat, Mathilde Guillaumin, Mathis Felardos, Maximilian Augustin, Mickaël Seznec, Nikhil Raghuraman, Olivier Duchenne, Patricia Wang, Patrick von Platen, Patryk Saffer, Paul Jacob, Paul Wambergue, Paula Kurylowicz, Pavankumar Reddy Muddireddy, Philomène Chagniot, Pierre Stock, Pravesh Agrawal, Romain Sauvestre, Rémi Delacourt, Sanchit Gandhi, Sandeep Subramanian, Shashwat Dalal, Siddharth Gandhi, Soham Ghosh, Srijan Mishra, Sumukh Aithal, Szymon Antoniak, Thibault Schueller, Thibaut Lavril, Thomas Robert, Thomas Wang, Timothée Lacroix, Valeria Nemychnikova, Victor Paltz, Virgile Richard, Wen-Ding Li, William Marshall, Xuanyu Zhang, and Yunhao Tang. Magistral., June 2025.
Ricardo Rei, Nuno M. Guerreiro, José Pombal, João Alves, Pedro Teixeirinha, Amin Farajian, and André F. T. Martins. Tower+: Bridging generality and translation specialization in multilingual LLMs, 2025.
Oscar Sainz, Naiara Perez, Julen Etxaniz, Joseba Fernandez de Landa, Itziar Aldabe, Iker Garefa- Ferrero, Aimar Zabala, Ekhi Azurmendi, German Rigau, Eneko Agirre, Mikel Artetxe, and Aitor Soroa. Instructing large language models for low-resource languages: A systematic study for basque, 2025.
David Samuel, Vladislav Mikhailov, Erik Velldal, Lilja Øvrelid, Lucas Georges Gabriel Charpentier, Andrey Kutuzov, and Stephan Oepen. Small languages, big models: A study of continual training on languages of Norway. In Richard Johansson and Sara Stymne (eds.), Proceedings of the Joint 25th Nordic Conference Computational Linguistics and 11th Baltic Conference on Human Language Technologies (NoDaliiaaBaltic-H11)) 2025), pp. 573-608, Tallinn, Estonia, March 2025. University of Tartu Library. ISBN 778-9908-53-119-4
Andrea Santilli and Emanuele Rodolà. Camoscio: An Italian instruction-tunct LLaMA. In Federico Boschetti, Gianluca E. Lebani, Bernardo Magnini, and Nicole Novielli (eds.), Proceedings of the 9th Italian Conference on Computational Linguistics (CLiC-ir 2023), pp. 385-395, Venice Italy, November 2023. CEUR Workshop Proceedings. ISBN 979-12-550-0084-6
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Wu, and Daya Guo. DeepSeekMath: Pushing the limits of mathematical reasoning in open language models, 2024.
Per Erik Solberg, Amee Skjærholt, Lilja Øvrelid, Kristin Hagen, and Janne Bondi Johannessen. The Norwegian dependency treebank. In Nicoletta Calzolari, Khalid Choukri, Thierry Declerck, Hrafn Loftsson, Bente Maegaard, Joseph Mariani, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Ninth International Conference on Language Resources and Evaluation (LREC'14), pp. 789-795, Reykjavik, Iceland, May 2014. European Language Resources Association (ELRA).
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F. Christiano. Learning to summarize from human feedback. In Advances in Neural Information Processing Systems, volume 33, pp. 3008-3021. Curran Associates. Inc., 2020.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems 12, pp. 1057-1063. MIT Press, 2000.
Masahiro Suzuki, Masanori Hirano, and Hiroki Sakaji. From base to conversational: Japanese instruction dataset and tuning large language models. In 2023 IEEE International Conference on Big Data (BigData), pp. 1684-5693, Los Alamitos, CA, USA, December 2023, IEEE Computer Society. doi: 10. 00.11099//9 2023...340......0.66600
16

---

Published as a conference paper at ICLR 2026
NLLB Team, Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang. Guillaume Wenzek, Al Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthip Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzmán, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safiyyah Saleem, Holger Schwenk, and Jeff Wang. No language Ieft behind: Scaling human-centered machine translation, 2022.
Jörg Tiedemann, Mikko Aulamo, Daria Bakshandaeva, Michele Boggia, Stig-Ame Grönroos, Tommi Nieminen, Alessandro Raganato, Yves Scherrer, Raúl Vázquez, and Sami Virpioja. Democratizing neural machine translation with OPUS-MT. Language Resources and Evaluation, 58(2), 2023. doi: 10.700777/
Gidcon Toury. Descriptive translation studies and beyond. Benjamins translation library 4. J. Benjamins, Amsterdam, 1995. ISBN 9027221456.
Ahmet Ustün, Viraat Aryabumi, Zheng Yong. Wei-Yin Ko, Daniel D' souuza Ghemileke Onilude, Neel Bhandari, Shivalika Singh, Hui-Lee Ooi, Amr Kayid, Freddie Vargus, Phil Blunsom, Shayne Longpre, Niklas Muennighoff, Marzich Fadace, Julia Kreutzer, and Sara Hooker. Aya model: An instruction finetuned open-access multilingual language model. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 15894-159999 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653/v
Eva Vanmassenhove, Dimitar Shterionov, and Andy Way. Lost in translation: Loss and decay of linguistic richness in machine translation. In Mikel Forcada, Andy Way, Barry Haddow, and Rico Sennrich (eds.), Proceedings of Machine Translation Summit XVII: Research Track, pp. 222-232, Dublin, Ireland, August 2019. European Association for Machine Translation.
Erik Velldal, Lilja Øvrelid, Eivind Alexander Bergem, Cathrine Stadsnes, Samia Touileb. and Fredrik Jørgensen, NoReC: The Norwegian review corpus. In Nicoletta Calzolari, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi. Koiti Hasida, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, Stelios Piperidis, and Takenobu Tokunaga (eds.), Proceedings of the Eleventh International Conference Language Resources and Evaluation (LREC 2018), Miyazaki, Japan, May 2018. European Language Resources Association (ELRA).
Vered Volansky, Noam Ordan, and Shuly Wintner. On the features of translationese. Digital Scholarship in the Humanities, 30 07 2013. ISSN 2055-7671. doi: 10. 1013//
Jiaan Wang, Fandong Meng, Yunlong Liang, Tingyi Zhang, Jiarong Xu, Zhixu Li, and Jie Zhou. Understanding translationese in cross-lingual summarization. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3837-3849, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023, hwn.....innningeeen........
Lex Weaver and Nigel Tao. The optimal reward baseline for gradient-based reinforcement learning. In Proceedings of the Seventeenth Conference on Uncertainty in Artificial Intelligence, UAr'1, pp. 538-545, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608001.
Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist re- inforcement learning. Mach. Learn., 8(3-4):229-256, May 1992. ISSN 0885-6125. doi: 10.1007/BF00992666.
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and low-precision training for large-scale vision-language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
17

---

Published as a conference paper at ICLR 2026
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. m A massively multilingual pre-trained text-to-text transformer. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 483-498, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/w112000 ma..00m.....l-
Sicheng Yu, Qianru Sun, Hao Zhang, and Jing Jiang. Translate-train embracing translationese artifacts. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 362-370, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v1/0022.2e. -ahooo22..ssoooot.....
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24 24. JMLR.org, 2024.
Hongbin Zhang, Kehai Chen, Xuefeng Bai, Yang Xiang, and Min Zhang. LinguaL.IFT: An effective two-stage instruction tuning framework for low-resource language reasoning, 2025.
Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin. Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging LLM-as-a-judge with MT-hench and Chatbot Arena. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: less is more for alignment. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2023. Curran Associates Inc.
Elaine Zosa, Ville Komulainen, and Sampo Pyysalo. Got compute, but data: Lessons from post-training a Finnish LLM. In Richard Johansson and Sara Stymne (eds.), Proceedings of the Joint 25th Nordic Conference on Computationnal Linguistics and / 1Ith Baltic Conference Human Language Technologies (NoDaLiDa/Baliic-lll 2025), pp. 826-832, Tallinn, Estonia, March 2025. University of Tartu Library. ISBN 978-9908-53-10090)
18

---

Published as a conference paper at ICLR 2026
A THE USE OF LARGE LANGUAGE MODELS
Large language models have been used to provide feedback, fix grammatical errors and improve the writing in this paper; in particular, used the Claude family of language models from https: //claude. ai. In addition, we used the auto-completion tool from GitHub Copilot when writing the code used in this work.
B TRAINING METRICS
While the paper focuses fluency the primary goal, it is actually not quality that is directly optimized by any the training methods. The reinforcement-leaming approach optimizes the policy model produce responses that get high rewards. These reward values are generated by LLM--s-a- judge models according to the prompt template listed below in Appendix - this evaluation prompt focuses on the overall quality responses based on criteria such as correctness, safety or conciseness. Figure shows the average rewards, together with other main RL statistics, throughout our training runs.
Training reward
7
6
5
4-
3
2
1
0
300
400
Step
Training KL divergence
Training entropy
Average response length
0.6
3.0
0.5
2.5
600
0.4
2.0
0.3
1.5
400
0.2
1.0
200
0.1
0.5
0.0
100
200
300
400
500
0
100
200
300
400
500
100)
200
300
400)
500
Step
Step
Step
Figure 4: Progression of post-training metries. These figures visualize some of the training
information of the long reinforcementtleamming run from Section 4.3. The figure on top shows the average rewards obtained from the LLM-as-a-judge prompt at each training step; the prompt itself is given below Appendix E.2. We also report the average KL divergence each step, the average per-token entropy of the responses, as well as the average length of each response.
C HYPERPARAMETERS
Second stage - English SET We finetune all model parameters on a single epoch the English SFT dataset. We use the StableAdamW optimizer (with 31 0.9, ,2 0.99 and 1 10 5) for increased robustness to gradient spikes (Wortsman et al., 2023) with constant learning rate of - and 10% linear warm-up phase. The batch size is set to 32 sequences that truncated to 4 096
19

---

Published as a conference paper at ICLR 2026
tokens as the maximum sequence length. We slightly regularize the training with weight decay of 0.1. The user-assistant conversations are formatted according to the minimal chat template listed in Appendix E.1. The loss is only computed the assistant responses, the user queries are ignored in the loss calculation.
Third stage translated SFT This uses the same hyperparameters as the second stage, only changing the training dataset and the optimal number of epochs to 3.
Third stage RLAIF The policy is trained similarly to the second SFT stage except for: the batch size is increased to 128 for increased stability and the learning rate is slightly lowered to 1- 10-6 for the same reason. The weight of the additional KL-divergence term is set to 1 10 The responses are randomly sampled from the (delayed) policy without any adjustment the output probability distribution, they are only truncated to the maximum of 2 048 tokens. The reward-judge judge model uses the prompt template listed in Appendix E.2, its judgments are randomly sampled with softmax temperature reduced to 0.2. If the final numerical score cannot be parsed from the generated judgment, we set it to (out of 10), as the error likely caused by malformed policy response.
D MANUAL ANNOTATION PROCEDURE
The annotation performed by hired team of five MSc students in NLP/CS, all native Norwegian speakers. The annotators compensated for their efforts, with hourly pay rate of 236 NOK per hour. All annotators undergo a training stage, whereby they receive detailed annotation guidelines and annotate 10 examples, followed by a joint seminar where any disagreements are discussed, before proceeding to the main phase of the annotation.
The annotation has been done online via simple application (Figure 5).
Norwegian Fluency Annotation
click here soe the annosatin Akiiiis
Progress: 12/118
Prompt
Jyenneege eagerilen.
Resparae
Resporan
Saga Sagactiin erer ittecer som lgerettgnee taee karilder re trant Keen Apennutegn sagestien len
En ragasi har rown tydeliga igennttgg. lor nomn dt virtipte
Bruk clatog. on rags det eepe dising agrept Bikene glenn vonte
Offer
der agei en furte sce kommoneeer hondlingen loral wever hondllinger
Friel og so spreghide Saguttiin in to nite woid tow woing yoce elk Nostreenti
Sovarde troee lilng Lan eagn en oft tydeily soeeeete avell lngg der formii bedirtter Pock
kos ow bredittser elter herrsparrk rots. Folkeiig utryla disicic: mar sager trakes Beliige uttyykk didsk for ssorice
som dijer og hwor sijer. be sccelske couutvetren eneehf kort fonvito.
Uit skitciing indre indrefeldsor feldsor. sago dec liee ski criing lorteeeeeee indra falescon
autantis sneg
shodkt de: fokus pil yorr handlir ger herdeller
Nosulala budskap: olts movalle bu dlaer monatios
goder og
tronalogia tromatiiln rn.
inanc hogish fromas der hand
sllute.
ondelll fenteene.
brukk or akudldeeedd tumsage
konts lond som kormmanteree sundinngen
renteallkke amenter: Sugar inmahable olt lartas fullae alument ter unill kamanakio
kanaitarsm ragas. Povedans er kelickatta kannise.
bruk tterar arrureddeer, sage truker othe velerr metumtt
underdrrvede
Ritualer agonen rewehatderr ttuaerr agjeee,
knystat
ekoumpel kurakder
apenttllg
bedtwente tohppunder iivv viler lowts fonk studler
moill tineer Sageoortelnggr
enkle-n lnvewrs med
tnegrnvtio, surtie alimoke.
Symbolbruk vier symbeler metsatorer for ghhistareen
betydaing
wenn nerign
is more fuent
Equally Alsnnt
is more flunt
Figure 5: Screenshot of the annotation tool. Each annotators is provided with a randomized sequence response pairs in randomized order.
20

---

Published as a conference paper at ICLR 2026
D.1 ANNOTATION GUIDELINES
These are the full annotation guidelines that the hired annotators followed:
OVERVIEW
This document provides guidelines for evaluating the fluency of responses generated by Norwegian language models. Annotators will compare pairs of responses (Response A and Response B) and determine which response demonstrates better fluency, or if they are equally fluent.
The evaluation focuses exclusively on language quality, naturalness, and grammaticality. Do NOT consider features such as factual accuracy and correctness, completeness of information, creativity and originality, or length and conciseness.
DEFINITIONS
What is fluency?
Fluency refers the linguistic quality of text that makes it natural, smooth, and easy to read. It should look like a text written by native speaker. A fluent text should consistently use either Bokmål or Nynorsk (depending on the prompt), and should sound genuinely Norwegian rather than were translated from another language.
Fluency issues to look for
When evaluating fluency, pay attention to:
Grammar errors: agreement errors (e.g. adjective-noun or determiner-noun disagreement),
incorrect verb tense, incorrect word order (violating V2 requirement), wrong word forms
Awkward phrasing: Unnatural word order, stilted expressions, robotic language
Punctuation problems: Missing or incorrect punctuation that affects readability
Word choice issues: Inappropriate vocabulary, incorrect word usage, repetitive language, wrong use idioms or phrases, incorrect spacing formation compound words ("kaffe kopp" vs "kaffekopp"), preposition errors ("på" vs "("")
Flow disruptions: Abrupt transitions, disconnected ideas within sentences
Spelling errors: Typos and misspellings, wrong capitalization, incorrect use diacritics (e.g. "ă" vs "a", "ø" vs "o")
Translationese: A common problem of language models is that they base their output on English the majority language in the language corpus. This can result unnatural language patterns that look like literal translations from English, such as: "stå opp for seg selv", "gjøre en forskjell", "være for salg".
ANNOTATION PROCEDURE
Step-by-Stee process
Read the prompt: Do not analyze the fluency of the prompt, but look at it to understand the context and language style.
Read both responses completely without making immediate judgments
Identify fluency issues in each response using the criteria above, ignore content accuracy and relevance
Compare the severity and frequency of fluency issues between responses
Make your decision based overall fluency
Decision options
You must select one of three options:
A is fluent: Response A has better overall language quality than Response
21

---

Published as a conference paper at ICLR 2026
B is more fluent:
Response B has better overall language quality than Response A
Both responses have similar language quality (minor differences that don't t clearly favor either response)
Equally fluent:
Important guidelines
Minor differences matter: Even small improvements in fluency should influence your decision
Be consistent: Apply the same standards across all evaluations
When in doubt about equality: If you cannot decisively determine which is better after careful analysis, select "Equally fluent"
EXAMPLES
Here are some examples of texts that should not considered as fluent Norwegian:
"Vi kan også prove à finne mäter gjøre oppgavene dine mer overskuelige og gi deg mer tid til gjøre dem på." (word choice)
"skrivemappa din" (agreement) "en elsket medlem av kongefamilien" (agreement)
"jeg vil se deg neste gang" (English-influencee translationese, more fluent would be "sees neste gang")
"banal hjertroman" (compound)
"den første konge" (double definiteness)
EDGE CASES AND SPECIAL CONSIDERATIONS
Other language than Norwegian: If one of the responses is in a different language (e.g. English), even partly, it should be considered less fluent than the Norwegian response, regardless of its quality.
Technical or specialized language: Technical terminology and domain-speciffe language should be considered fluent if used correctly and consistently, even if it might seem less natural general audience.
Formatting issues: Ignore formatting differences (bold, italics, bullet points) unless they directly impact readability sentence structure.
Code or mathematical expressions: If responses contain code snippets or mathematical expressions, evaluate only the fluency of the natural language portions.
E PROMPT TEMPLATES
Prompt templates are a crucial part of modern training pipelines, we list them here for full trans- parency:
E. CHAT TEMPLATE FOR POLICY MODELS
({- token ))
(K- if ressage['role 'user" X)
[{- "cinstruction>" ressage "'ccntennt' '</nssructiooo" }}
(3- el:f ressage['role role'] == "systen"
{(- "csysten_poorpe" sessage["
's/systersprorpp""
(K- ressage["rale"
"assistant"
(5- generation 8)
((- messagel content "crap"
(5- endginrration
(B- endif 8)
(8- endfor 3)
22

---

Published as a conference paper at ICLR 2026
E.2 PROMPT TEMPLATE FOR THE REWARD/JUDGE MODEL
You are an expert evaluator tasked with assessing the quality of Af resporses in Noraegian conversations You will evaluate in English.
Language Requirieent
[MPORRTATTT The Al assistant KUST respond in Norwegian when the user writes in Norwegian. Responding in English Norwegfan query
Failure should baste expectation Norwegian language assistant. Technical torns programeing may resati English wichin korwegian text.
regardless good the
You will receive three 1SDN objects, your goal is to evaluste the "ai_response" value:
"JJson
"comersation_history
("role:: "user", "content": "Norwegian ressage"), ("rale:: "assistant", "content": "Norvegian assistant response"),
("role:: "user", "content"; "Norvegion user reply").
1.
response "al_response"" "Norweglnn an response to evaluate (this LAST assistant response)"
"gold_responee:
"English
anrotator',
Note: The gold response is a reference point but may not perfect or the only good approach. Gold responses are written in English for evalustion purposes, but the All should respond in Noraegian when users write in Norwegian (unless explicitly asked
another language).
Evaluation Process (Nrite English)
Follow these in order:
isIpportnt::ee For multi-tum conversations, evaluste ONLY the last Al response, not any previous assistant responses in the conversation history.
Sunmarize Briefly describe what the user asking for in the conversation.
Request
111 2: Analyze the Gold
Explain gold (human-written) response aporoaches request. Note key strategy.
111 Step 3: Analyze the AI Response
Explain
being evaluated approaches request. elerents and
tM Step 4: Evaluate Across five Criteria
Correctress Instraction-Folloiing
response directly acdress
the information provided accurate and correct?
Are all parts of the request fuirilled?
horwegian
Norwegian?*s (Critical requiriment)
11td 2. Style
*nLanguage Natchee: Does the response eatch the user's languago? (Nonwegian irpput Norsegian output, unless user requests
sAToness; appropriately casual, polite, friendly Norwegian (unless context derands otherwise)?
*nStructurae:; Are ideas well wall-organtzed with proper paragraphh breaks (tuo newlines between distinct thoughts)?
wMarkdowwee: Is narkdown used effectively to ennance readabiisty?
*Roadabilltyye::
easy
understand?
#1dd 3. Safety
Does the response avold harnful content?
apprapriately balanced nosther pronoting hareful behavšor requests?
cauttous/noraliziingg Srnocuous
#3dd 4. Logical Reasoning and Muance three Jnterconnected
**Muancase: corplexity aporopriate
query?
*nFactualitywe; Are all facts and claims correct?
**Reasoning Qualitye:: Ts the Togical structure sound? Ta there clear, non-rspetitive narrative flow?
Approoriate Conciseness
To the resporse Tength aporopriate for the query?
Does it avoid being too brief unecessarily verbose?
*HHavily panalize
excessively for simple
*#Lenghh Guidelines:e*
Simple factual questions (capitals, dates, nases): 1-2 sentences
Faplasatory
(how/whl something
paragraphs
Casplex or tulart questions: Maltiple paragraphs acceptable
#48 Step 5: Sumary and Final Score
Sunsari2e your evaluation
critaria
final Format:
**Scoro:*4
23

---

off Published as a conference paper at ICLR 2026
Where X is an integer from 1-18:
Excellent
Very good
6-7: Good
4-5: Adequate
Poor
Terrible
The score should be placed a new line and it should be the last iter of your response.
Examples
Example 1: Recipe Request (Score: 9/18)
>Input:iA "soon
"comersstionnhistarry". C
"role": "user",
"concent": "Hei! du gi nog en appskrift pá carbonara? leg har ).
"rale": "assistant",
"contant"; "Selvialgalig! carbonara or klassisk stallandk rett. on oppsier Lngskriit::oee Aot.tttt----iiieee Tngr--onenn---eennnnnn porsjoner) wwnn- 400g spadhettiln- 208g guanciale (elier pancetta/bacon)/r 4 eggepiommer helt eggin- 188g revet Pezorino RouandAn- Svart proper'n- SaltvvvnneFrongangaae wan.rrrnnnngasttaeeeee selp). Kok spaghattion saltet vanrin2. Stek spravi3..
bland samennins. Tilsett pastavann for krenete konsistens bjobbe raskt og holde pannen av varnen blander inn oggene!"
og estiss.
).
"rale":
"contant"; "Takk! jeg forstår
eggane for pannen vere varmen? Og lva guanciale?"
1.
respaness:ss"" "Great questions!
esplain voth-woyn creany sauce, scrarbled oggs. the pan hot, eggs heat: aggs In quickly and becore lurpy/scranbled By taking the pan off heat and using just the residual worthh from the pasta and rendered Fat, the eggs gently cost the pasta and create that signature silky sauce, vineewhat is secciaal::eenoaandee ingredient.
should create
is cured pork jool (chaek),
beautiful fol suhatituteln- Regul bacon works ton, though snsktarin- Laok for guanciale at Italian specialty stores or dalistataThhe rendered fat from the guanciale is crucial helps along with the and cheese!".
traditional
and
you
find
it: Pancetta (Ttalian cured hally)
the creasy
_i__ssooss"": "Flatte spar splr
lage kresete saus, lkke eggerara! begge deter: Aoveetvrrorr varn, wil eggana stivne ned gang og klurpete. äta parnen av varsen bruker du bare restvarsen fra pastaen og det vame fettet til hvarre eggene forsiktig. Dette gir den siTkasyke sausen carbonara kjent for AnysTonk på det son Forskjellen mellor Alage aggerera (hey varme, fast konsistens) og
varren:eehnfggere
parmen
flytende konsistens). Anlnseika guanciale sennuuiciea speket svinekjake altss kjettet fra grisen kinn. Det har dyp, intens snak og gir fra seg deilig fett det stakes, Valetvvi du like Firner guanciale: ln- eeRete alternativ:se Pancetta (Stal tanak speket svineside) yor welest bestersx Vanlig bacon (blir litt snnerledes neykstaken)nn sslivor finne det: Pray italienske spesialbutikker katesssebut ikker
eller velassorted til dan kresete sausen!"
gancialen superviktig blander seg egg og og
<rEvalustion:ee
ssUser request:ss The has follow-up questions about the carbonara specifically asking guanciale
wwGold response approach: Explains both concepts clearly, with the heat explanation focusing on avoiding scrambled eggs. and providing substitution options for gusnciale.
*HAI response sporoach:sx Provides detailed esplanations korwegian for both questions, using an analogy for the egz cooking process and listing substitutes for guanciale with hierarchy of alcermatives.
Correctness cetruuttoo-followinns... Respondi in Norvegion as required.
Directly answers specific
accurate inforration.
Style;sx Very good. Clear Norwezian natching the user's language, heloful fomstting and includes relevant esoji. anelogy comparing scrambled eggs pedagogically
vx). Safety:hr ho issues.
Logical reasoning nuance:es Excellent. complexity approoriate sorcone learning to cook, heloful analogies and practical substitution advice.
**5. fooropriate conciseness:oe Good lengch thorough without being ovenwhelaing for specific questions.
SummaryyA
This is an excellent follow-up response that adoresses both questions clearly and helpfully. The explanations are accurate, practical, and sell-suited to suseone leaming to cook cartonara for the first time. Inportantly, it language.
vxScore:kk
3/18
24

---

Published as a conference paper at ICLR 2026
1M Example 2: Simple Math Question (Score: 4/18)
abbreviated for ciarity
MM Example 10: Overly Verbose Response (Score: 3/10)
abbreviated for clarity ...]
Scoring Guidelines
чл10/1044; Exceptional response that exceeds expectations
**8-9/1BA4: Very good response with sinor areas for ixproverent
>45-7/10AA: Good
basics lacks rinesse
**4-5/18<>; Adequate
significant coonings
**2-3/10>a: BAa: Poor response with major prabiors (including language rismatch or extreme verbosity)
>A1/10w:: Complete failure nonsensical response
*CCritical tes
Responcing in English to a Norwegian query: Paximun score 2/10
Extrene verbosity for simple questions: Significant scare reduction
Evaluation Task
avaluate following resporse using process and criteria above. Remember
1. Arite your evaluation in English
2. Fallow all five stapa in order
provide Heavily penalize responses
Engiissh the Norwegian 5. Heavily panal ize responses that are excessively verbase for sirple questiins
6. End with score in the Fornat X/10
**Input evaluate: sson
((input)))
Regin your avaluation:
F EXAMPLE OUTPUTS
As apparent from the fluency scores in Figure 3(a), the on-policy training appears to be less fluent the beginning of the third stage and then converges fluent state. This is slightly misleading. because when looking at responses more closely, the responses are not in disfluent Norwegian, the issue is that they are sometimes completely in English, which lowers the Norwegian fluency score. Thus, as we hypothesized, on- on-policy training preserves fluency, but the model cannot suddenly learn (at least without any fluency reward signal, as in our case). We illustrate this in the following samples from checkpoints of the RL-trained model in Appendix F.1 and of the SFT-trained model in Appendix F22; these checkpoints correspond to the initial points Figure 3. We also provide samples from the multilingual Mistral Nemo model in Appendix F.3.
F.1 ON-POLICY RL OUTPUTS THROUGHOUT TRAINING
Training step This output is from checkpoint that has been trained on the English SFT dataset before the RL training starts. When the prompt is short, such "Beskriv Eurovision" (Describe Eurovision) below, the model is likely to respond English. We randomly picked one such prompt to show this behavior.
Beskrty Eurovision.
Since Sts Inception in 1956, the Furovision Song Contest has become the largest music event in the world. The contast song contest.
yearly
countries
the "Eurovision
Song Contest Traphy"
The contest is hosted by a country that has von the contest the pravious year. The contest la broadcast 1ive on television June.
song 188 countries and watched countries showtase
radio
1ee million viewers. The contest is usually held way and political sigsificance
way countries show support for one another. The contest is way for countries to shox their support for one another. The contest a way for countries to show their
muse culture. support for one another.
Training step 25 200 training samples) The model starts to respond in Norwegian more confidently, even though the content is not always very helpful.
25

---

Published as a conference paper at ICLR 2026
>. Beskriv sirovision.
arrangeres Eurovision Song Contest, konkurranse europeiake sangere gär videre Finaten. Det. Landet aos vinner Finalen, det Landet son arrangere neste åes korkurranse.
velger song, og
(Every year, the Eurovision Song Contest held, a competition between European countries. Each country chooses a song, and the best songs go on to the final. The country that wins the final is the country that gets to host next year's 's competition.)
Training step 50 (6 400 samples) steadily improves.
The responses are now always in Norwegian and their quality
Beskriv Eurovision.
Det årliig nusikkonkurranse
Den europeiske kringkastingsaionn Hälet åkáre den beste Europa, livert sender satt bidrag. son on sang og musikiviseo. Bidragene hvert land, og av TV-seerne. Vimeren er den son får flest stemer.
jury
(It is an annual music competition organized by the European Broadcasting Union. The goal is to select the best song in Europe. Each country submits ils entry, which consists of song and music video. The entries are judged by a jury in each country and by TV viewers. The winner is the one who receives the most votes.)
Training step 75 (9 600 samples) improving.
The fluency scores have converged, the response quality is still
> Beskriv Eurovision.
europeiske nusikkarrangementet Eurovision, også kjent Eurovision Song Contest, årlig musikkonkurrangg son arrangeres Den europeisie sice tringkastinguriog (EBU). Konkurransen har blitc arrangert siden 1956, og er av de lengstlevende rusikkonkurraansem verden.
Eurovission direktesendt sangkerkurranse
deltakende livert Land har en jury son stemer på de andre landenes bidrag, og det londes som får flest stemer, vinner kankurransen.
Freeifares
tilless songkonkuuransem,
opså et stort runds arrangementet. pressekonferame og en Finolefest. Arrangementet er svert papulert t Europa, og tiltrekker seg willioner av seere hvert är.
annet apningsseremoni,
(The European music event Eurovision, also known as the Eurovision Song Contest, is an annual music competition organized by the European Broadcasting Union (EBU). The competition has been held since 1956 [...])
Training step 100 (12 800 samples)
No apparent qualitative changes.
Beskriy Furovision.
Furoviston Song Contest, ogeå kjent Furovision, sentes rus eustkkknkkrrrat ranaon. Korkurransion har blite arrangert siden 1856 og består av to deler: on sexifinala og en Finale. sexifinalen Frenfarer hvvrrt land an sang, de ti beste går videre till Tinalen. (rinaten stermer hivert land på de sanger, og Vinneren av Euroviation köret gjornos on koabinasjon telefonstemer
Furopa Moonkrreeee årlig
Får
seerne stamer profesfonslle Juryar. tillegg til songkaniurranser, er Eurovision ogsá en plattfors for åvise fres vertsllandets kultur, det arrangeres en rokie fester arrangeeenter Forbindelse red korkorriesan.
(In the Eurovision Song Contest, also known Eurovision, countries from all over Europe come together to compete an annual music competition. The contest has been held since 1956 and (...].)
F.2 SFT OUTPUTS THROUGHOUT TRAINING
Training step 0 The SFT training stage starts from the same checkpoint RL, the responses (and their issues) are the same as in the previous section.
Beskriv Eurovision.
Since its inception in 1956, the Eurovision Song Contest has becone the largest music event the world. The contest is yearly event that brings together 48 countries to cospete in song contest. The kinner is awarded the "Eurovision Song Contest Traphy"
contest
country
contest previous year. contest
and racio to over 100 countries and is watched by over 109 sillion viewers. The concest is usually held in Hay or June.
26

---

Published as a conference paper at ICLR 2026
The contest is song contest. countries to showcase their rusic and culture. It is also way for countries to show their support for one another. The contest is way for countries to show their support for one another. The contest a way for countries to show their for another.
is
political. signtficance it.
way
Training epoch 1 010 training samples) Similarly to on-policy RL, the model quickly starts to respond in Norwegian, according to the SFT training data. However, the texts often contain translationese artifacts. In this case, Basert på opprinnelsen (based [its] origin) sounds unnatural, this type of cataphoric thematization is more common in English. Similarly with sin storhet og prestisje (its grandeur and prestige). Nominalizations should be generally used sparingly in favor of adjectives and verbs.
Eurovision.
Besert pl apprinnelsen een er Eurovision Song Contest en árlig musikienkurranse arrangert av Den europei ske kr bringaattingunion (EOU) siden 1996. kjent for storhet prestisje, tiltrekker oppeerisonhet fra hele verden. mest populare begivenhetere Europa, og kjent
Konkurransen show og underboldning.
kosbinasjon susikk,
(Based on its origin, the Eurovision Song Contest an annual music competition organized by the European Broadcasting Union (EBU) since 1956. It is known for its grandeur and prestige, attracting attention from around the world. The contest is one of the most popular events in Europe and is known for its unique combination of music, showmanship, and entertainment)
Training epoch 2 (20 000 training samples) The topicalization in the example below sounds very unnatural in Norwegian, almost ungrammatical. Using a relative clause would be more fluent in this case.
Deskriv Eurovision.
Ferst arrangert 1956, Furevision Song Contest (FSC) årlig musikookurriese arrangert bringkastingsunion (CDL) blant redlermene.
europel
(First held in 1956, the Eurovision Song Contest (ESC) is an annual music competition organized by the European Broadcasting Union (EBU) among its members.)
Training epoch 3 (30 000 training samples)
The response below sounds natural.
> Beskriv Eurovision.
Eurovision en rosiikorkurranse arrangert europeiske kringlestingsunion (EDU)
livert land sender inn en sang, son deretter konkurrerer TV-sendt finale, soe firner sted på forskjellige steder hvert år. Finalen består av to seni-finaler, livor de ti beste fra hver seni-finele ivalifaserer seg til finalen. En Jury publ ikul bestermer vimnrren. Eurovision
de lengstleverde pVprrggaamere tiltreker rundt
(Eurovision an annual music competition organized by the European Broadcasting Union (EBU) with members from over 50 countries. Each country submits a song, which then competes in televised final, held in different locations each year. The final consists of two semi-finals, with the top ten from each semi-final qualifying for the final. A jury and the audience decide the winner. Eurovision is one of the longest-running TV programs and attracts around 200 million viewers each yeur.)
Training epoch 4 (40 000 training samples) In this case, the preposition in på en natt i mai (on one night in May) is ungrammatical and likely result of literal translation from English. Simply saying natt i mai would be correct.
Beskriv Furovi sion.
Eurovision en årlig musiskorkurranee sanger
Den europeiske kringkastinguniin (EBU) der deltakerlander kankurrerer ikum og nillfoner
sai. land sang freefares Live hjeme. valet er Svirne konkurransen ved 3F3 flest stemer fra en jury og/eller seerne.
(Eurovision is an annual music competition organized by the European Broadcasting Union (EBU)
in which participating countries compete with songs song to be performed live in front of an audience and millions of viewers at home. The goal is to win the competition by receiving the most votes from a jury and/or viewers.)
night in May. Each country selects
27

---

Published as a conference paper at ICLR 2026
F.3 OUTPUTS FROM THE MULTILINGUAL MISTRAL NEMO
Eurovision prompt In this case, the multilingual model responded in Danish (which is correct since "Beskriv Eurovision" is ambiguous).
>. Beskriv Furowl sion.
Eurovision Song
vegne tonstationer. Konkurrencen blev farste gong afholot 1956 og dermed aldste Europa.
songkonkurrence,
atholdd Broadcasting
anddl og iest sete ow-undertoldninnggggrrre
Furovission Song Contest konkurrerer
sanger fra Sangen skal vere skrevet specielt til konkurrencen op kan vere op til tre vinutter lang. Der er ingen begrensninger susikgenren sange kan vere på ethvert sprag.
land blive kåret den
Vinderen of konkurrencen bestermes
jury eksperter har en jury, oer rangerer sangene eil og 12 point. Punktesysteres er baseret pa det, der brugt første Eurovision Song Contest 1956.
publikun delenged
Furovission
glanour, underholching international itet Konkurnencen blevet beskrevet sysbol pa free og forstàelse rellen de europaiske lande og en chance for at vise op feire det bedste i europeisk musik.
kendt
(The Eurovision Song Contest (ESC) an annual song contest organized by the European Broadcast- ing Union (EBU) on behulf of the member countries' national television stations [...])
Unambiguous prompt To give better example to the reader, we also show response to a longer prompt that is unambiguously Norwegian, but containing several ungrammatical noun phrases, such as deres intime og personlige tonen.
Rva regnes kjenteste verket lictoratur fra poetisk realisre perioden
og
Det mest kjente verket i norsk litteratur free postisk realisre i perioden mellon 1859 1878 er sannsynligvis -fante-Ureve- Breve- =Fante-Breves
til sin klareste og hustru, Suzannah Thoreson, han Itella 1864. Brevene kjente deres Iosons tarker og relelser under denne perioden. eFante-Breves or agss kjence for deres beserivelser av den italsenske kulturen Landskapet, de har blitt sett på son et vikcig verk innenfor norsk reise litteratur.
Henrik
sanling
personlige tonen,
gir inmblikk
(What is considered the most famous work of Norwegian literature from the poetic realism period between 1850 and 1870? The most famous work of Norwegian literature from the period of poetic realism between 1850 and 1870 is probably "Fante-Breve" by Henrik Ibsen. "Fante-Bree is collection of letters that Ibsen wrote to his girlfriend and [...] I)
28


Reviews:
[{"review_id": "RGNLx7vt0X", "reviewer": "Reviewer_E2yZ", "summary": "This paper investigates on-policy reinforcement learning for improving low-resource language capabilities in multilingual language models. The key insight is both counterintuitive and practical: using a reward model (judge) that is not fully fluent in on-policy RL can outperform supervised fine-tuning (SFT) on machine-translated instruction-following data. Existing multilingual alignment approaches often rely on translating English instruction data, which introduces translationese that harms fluency. In contrast, on-policy reinforcement learning can achieve alignment purely through model-generated rollouts, without relying on synthetic or translated data. However, this requires a robust reward model to guide the RL process. The authors show that such a reward model need not be fluent in the target language. Using Norwegian as a case study, the authors conduct human evaluations by five native speakers, demonstrating that on-policy RL guided by a disfluent judge achieves higher fluency compared to translation-based SFT baselines.", "questions": "1. Where does the boundary of the main claim lie? i.e., What is the minimum level of language proficiency required for a reward model to provide effective reward signals? For instance, could a strong English model with only weak, translation-based understanding of Norwegian still serve as an effective judge?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "1. The exploration of RL for low-resource language alignment is novel and practically meaningful. It avoids annotation costs and translation artifacts, and shows that even a non-fluent reward model can guide the training fluent policy models in the target language. \n2. The use of human evaluation by native speakers strengthens the credibility of the findings. The results in Table 1 clearly demonstrate the superiority of the proposed approach.\n3. Beyond using a non-fluent judge, the paper further explores different model families as reward models and shows that the final alignment quality in RL is independent of the judge model’s fluency.", "weaknesses": "1. Generalization to other languages. All experiments are conducted solely on Norwegian. It remains unclear whether the proposed method generalizes to other low-resource languages or even to medium- and high-resource ones. Would the same conclusions hold across different language families? For higher-resource languages, might the performance gap with SFT narrow? While human evaluation is costly, automated evaluation (similar to the constructed evaluation model in the paper) could help extend this analysis.\n2. Evaluation scope limited to fluency. Fluency is central to the paper’s evaluation, but other aspects such as helpfulness [1] or safety are equally important. There is a risk of reward hacking [2]: the model might prioritize fluency at the cost of informativeness, producing simpler but less useful responses. It remains unclear whether fluency gains come at the expense of other abilities.\n3. Dependence on SFT for initialization. The Norwegian RL pipeline still relies on some SFT data to establish basic instruction-following ability. The claim would be more solid if the authors demonstrated pure on-policy RL starting from a base model, similar to the success of zero RLVR [3], showing that RL alone can enhance the model’s capabilities without prior supervised alignment.\n4. Limited model diversity. The experiments are conducted mainly with the Mistral base model. Extending the study to additional model families (e.g., Qwen, Llama) would strengthen the generality and robustness of the conclusions.\n5. The section on human evaluation lacks important background information about the annotators. To ensure ethical research practices, the authors should specify the annotators' qualifications and confirm that they received fair and reasonable compensation for their work.\n\n---\n\n**References**\n\n[1] Liu, Ryan, Theodore Sumers, Ishita Dasgupta, and Thomas L. Griffiths. (2024) \"How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?.\" In International Conference on Machine Learning, pp. 31844-31865. PMLR.\n\n[2] Liu, Tianqi, et al. (2024) \"RRM: Robust Reward Model Training Mitigates Reward Hacking.\" The Thirteenth International Conference on Learning Representations.\n\n[3] Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., ... & He, Y. (2025). Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cKHq8HooRv", "reviewer": "Reviewer_dvfW", "summary": "This paper proposes an on-policy, fluency-aware post-training pipeline for lower-resource languages. The key idea is: start from a target-language–pretrained model, give it only a short SFT on high-quality English data to learn instruction-following, and then do online, on-policy RL in the target low-resource language — without ever training on machine-translated target-language responses. Even though the reward source (“judge”) can itself be disfluent, the policy can stay fluent because it only ever updates on its own (already fluent) samples.", "questions": "- Language Scope and Generality. Could you provide evidence that the proposed “disfluent-judge → fluent-policy” phenomenon holds beyond Norwegian Bokmål? Even a smaller-scale or synthetic experiment on a typologically distant language (e.g., a morphologically rich or non-Indo-European one) would help clarify whether the observed effect generalizes or depends on Norwegian’s proximity to English.\n- Dependence on Base-Model Quality. Your pipeline assumes a target-language-pretrained base that is already highly fluent. How much of the final fluency retention actually stems from this strong base versus the on-policy RL stage itself? Could you include an ablation starting from a weaker or multilingual base (zero-shot in Norwegian) to show whether the method can recover fluency or merely preserve it?\n- Judge Sensitivity and Bias. This paper states that the judge only needs “sufficient understanding,” yet its potential biases toward translationese or anglicized phrasing are not analyzed. Did you test robustness when the judge systematically prefers slightly translated-style outputs? A stress-test with biased or intentionally noisy judges would clarify how resilient the policy is to imperfect reward signals.\n- Please describe the precise prompts and calibration used for the multilingual judge. Are the reward decisions consistent across different prompt wordings or random seeds? Providing examples of disagreements and how they affect updates would make the pipeline more reproducible and interpretable.\n- If the base model were only semi-fluent, could the on-policy RL process improve fluency over time, or does it merely avoid degradation? An explicit experiment varying the amount of target-language pretraining would clarify the scope of applicability for teams without high-quality monolingual bases.\n- For truly low-resource languages that lack both strong pretraining corpora and fluent judges, what minimal requirements (dataset size, judge quality, reward signal noise tolerance) are necessary for this approach to remain effective? Quantifying those thresholds would make the paper more actionable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This paper proposes a three-stage recipe: (1) target-language pretraining (taken as given), (2) short English SFT (1k LIMA items, one epoch) just to teach turn-taking / format, (3) online on-policy RL in the target language, where the policy samples multiple responses per prompt and a multilingual LLM-as-judge gives rewards, plus a KL term with a variance-reduced (Rao–Blackwellized) estimator.\n- The experimental section is unusually careful for a “low-resource alignment” paper: same base model lineage, three post-training routes, human A/B judgments (5 annotators, 15–20h each), explicit win-rate table, and multiple ablations (SFT length, translation model, judge choice, training length).\n- The pipeline (Fig. 2) and the 3-stage story are easy to follow; section 3 very explicitly says: we tried (1) Norwegian RL, (2) translated SFT, (3) Mistral-Nemo.", "weaknesses": "- Scope proven only on one language / one family. Everything is on Norwegian Bokmål, which is relatively resourced compared to truly low-resource or morphologically extreme languages (e.g., Amharic, many Niger-Congo languages). It is also Germanic and fairly close to English. So it’s unclear whether the “disfluent judge → fluent policy” effect would hold for typologically distant languages, or for languages whose MT systems produce much noisier translationese. A small second-language experiment (even synthetic / smaller scale) would strengthen generality.\n- The recipe begins with “pretraining on the target language” and even borrows a continuous Norwegian pretraining from Samuel et al. (2025), but the paper explicitly says “we do not cover this pretraining stage.” In practice, the main result (“policy stays fluent”) might be mostly due to starting from a very good Norwegian base — i.e. the method is only as good as your base. The paper could do a clearer disentanglement: what if the base itself is only semi-fluent? The reader can’t currently tell how much credit to give to the RL stage vs the initial base.\n- This paper says the judge only needs “sufficient understanding,” not fluency, and Table 2 reports low correlation — but the judge prompt is not analyzed: how sensitive is the whole pipeline to mis-judgments on subtle register/style errors (exactly the kind of thing that makes translationese detectable)? If the judge systematically likes translationese, the policy may reintroduce it despite being on-policy. A robustness test with deliberately biased judges would make the claim stronger.\n- Base-model ablation. What happens if the target-language base is weaker (fewer continual-pretraining steps, or a multilingual model zero-shot on Norwegian)? Does the proposed RL still “pull” it to a fluent regime, or is strong pretraining actually the dominating factor?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yqlx5XlAfY", "reviewer": "Reviewer_9f26", "summary": "The paper aims to assess whether post-training models in low ressource languages for which no natural SFT instruction set exist is better done through (a)  synthetically translating an english SFT set into the target language using a translation model (of varying target language fluency) or (b) using RL to let the target model generate it's own rollouts in the target language and using a **disfluent** reward model to optimize the model towards generating **fluent** text in the target language. In both cases, models are continually pretrained on unsupervised norwegian and lightly SFTed in english beforehand.\n\nInterestingly, this paper shows (b) leads to stronger results, and the target model can become more fluent than the bigger judge model - achieving weak to strong generalization through the combined use of unsupervised CPT of the target model, and the fact that judging fluency for the reward models is an easier task than being fluent in generation themselves.", "questions": "1. This setup seems very close to rejection sampling SFT: use the english SFTed model, generate many rollouts in norwegian and train only on the top samples as determined by a LLM as a judge. I'm guessing this might not be possible because post english SFT, the model is not able to produce coherent norwegian so several iterations have to be run? What if this is bootstrapped with few shot ICL in norwegian ? Is RL really key here ?\n\n2. Could the use of techniques such as low-rank adapters, replay data, long learning rate re-warmup, etc. help alleviate the norwegian language collapse when doing english SFT ?\n\n> \"The key principle is to never train the language model on any unnatural text.\"\n\n3. Wouldn't the on-policy sampled text be unnatural at least in the first iterations, even after LLM as a judge filtering ? Could you provide some examples (at least for the norwegian speakers that will understand them, and some qualitative insights into how fluency evolves throughout training, what patterns are learned first, etc) I saw quantitative examples were provided in Figure 3.\n\n\nMy main concern here (summing up most of the weaknesses and question I stated) is disambiguating RL/SFT training dynamics versus the question of training/not training on unnatural samples. Irregardless of if RL is the only way to achieve this, the paper clearly shows judges do not need to be fluent themselves to steer fluent generations which I believe is a nice contribution in this domain.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "The paper is clearly written. The goals are clearly motivated making reading easy.  The potential impact is clear - enabling simpler training of instruction models on languages where supervised data is scarce.\nAblations are strong and well designed, answering many of the questions I had on the impact of the quality of the translation models for SFT, or judge reward models.", "weaknesses": "It would be interesting to have compute intensity comparisons between RL and SFT methods. My intuition is that much more data samples could be seen by the model in the SFT setup under the same compute conditions, perhaps somewhat compensating the performance gap with the RL based method. \n\nOne of the big claims of this paper, is that SFT does not work because even with good translation models, the translations may remain unnatural in some low ressource languages. Since the SFT set used is very small, could it have been possible to manually filter the SFT set by norwegian speakers and have a \"perfectly fluent\" SFT set - thus isolating the RL/SFT debate from translation quality ? More cheaply, a way to compensate the unnaturalness of translated text could also have been to filter translations using the same reward models (maybe generating various rollouts using the translation models and keeping only the most natural as determined by the judges for SFT).  As is, on-policy RL might naturally help the model converge to generations that better appeal to human or model judges irrespective of fluent language. \n\nI would appreciate it if the related work could be slightly extended to include works on translation without/with little parallel data, or works on zero-shot transfer of language abilities (train on an anchor language, evaluate on others). More generally, this problem also maps to works on weak-to-strong generalization, or RLAIF with weak judges and previous papers have already explored this notion of disconnect between judging/doing capacity. I believe discussing this would better contextualize this work and give intuitions on why this is possible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)