BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Authors: Yapei Chang
Yekyung Kim
Michael Krumdick
Amir Zadeh
Chuan Li
Chris Tanner
Mohit Iyyer
Venue: NeurIPS 2025 poster ()
Keywords: alignment
instruction-following
LLM
RLVR
GRPO
Primary area: deep_learning

Abstract: Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM backbones. Meanwhile, the increasing availability of high-quality synthetic instruction-following datasets raises the question: can simpler, reference-based metrics serve as viable alternatives to reward models during RL-based alignment? In this paper, we show first that BLEU, a basic string-matching metric, surprisingly matches strong reward models in agreement with human preferences on general instruction-following datasets. Based on this insight, we develop BLEUBERI, a method that first identifies challenging instructions and then applies Group Relative Policy Optimization (GRPO) using BLEU directly as the reward function. We demonstrate that BLEUBERI-trained models are competitive with models trained via reward model-guided RL across four challenging instruction-following benchmarks and three different base language models. A human evaluation further supports that the quality of BLEUBERI model outputs is on par with those from reward model-aligned models. Moreover, BLEUBERI models generate outputs that are more factually grounded than competing methods. Overall, we show that given access to high-quality reference outputs (easily obtained via existing instruction-following datasets or synthetic data generation), string matching-based metrics are cheap yet effective proxies for reward models during alignment. We release our code and data at https://github.com/lilakk/BLEUBERI.

BLEUBERI:
BLEU is a surprisingly effective reward for instruction following
Yapei Chang
Yekyung Kim
University of Maryland, College Park
University of Maryland, College Park
Michael Krumdick Kensho
Amir Zadeh Lambda AI
Chuan Li Lambda AI
Chris Tanner Kensho
Mohit Iyyer University of Maryland, College Park
Abstract
Reward models central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM backbones. Meanwhile, the increasing availability of high-quality synthetic instruction followwig datasets raises the question: can simpler, reference- based metrics serve as viable alternatives alignment? In this paper, we show first that BLEU, basic string matching metric, surprisingly matches strong reward models agreement with human preferences general cnstruction-folloviii datasets. Based on this insight, we develop BLEU- BERI a method that first identifies challenging instructions and then applies Group Relative Policy Optimization (GRPO) using BLEU directly the reward function. We demonstrate that BLEUBERI-rraine models are competitive with models trained via reward model-guided RL across four challenging instruction-folloiin benchmarks and three different base language models. A human evaluation further supports that the quality of BLEUBERI model outputs is par with those from reward model-aligned models. Moreover, BLEUBERI models generate outputs that are more factually grounded than competing methods. Overall, we show that given access to high-quality reference outputs (easily obtained via existing instruction-followee following datasets or synthetic data generation), string matching- based metrics are cheap yet effective proxies for reward models during alignment. We release our code and data httpt:tt: //gi thwb. http://tthh......l..iotaeoteat wttp.////a//////eeeee/ EUBBRII
reward models during RL-based
1 Introduction
Modern LLM alignment often relies on reinforcement learning with reward model that guides the
LLM follow instructions according to human preferences [40]. Reward models are expensive
to train, requiring large-scale human preference data and powerful backbone models [39 55| 60).
Meanwhile, the emergence of high-quality instruction-followingg followwng datasets (e.g., OpenHermes, Magpie, LIMA) has enabled cheaper reference-bate alignment via supervised fine-tuning (SFT) [7 73 80
BBLEBBERI stands for "BLEU-base reward for instruction following."
"Conresonnencee to yapeic@und.com edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

BLEU is surprisingly effective at modeling human preferences: Despite these limitations, we find that BLEU [41]. a simple string-maten matching metric long deemed inadequate for open- [5] rivals large reward models in model- This contrast raises natural question: can we align language models with simple reference-based metrics in place of learned reward models?
Challenges with reference-based rewards: reference-based metrics seems problematic. Obtaining high-quality references for complex, open- ended tasks can be even more expensive than collecting preference judgments, and many instructions (e.g., creative writing) lack single ground-truth answer. Furthermore, aligning an LLM requires balancing multiple criteria (c.g.. helpfulness, harmlessness, factuality) 4 that a single reference may not fully capture. Finally, automatic metrics that score LLM responses against references rely on unreliable and gameable methods like n-gram matching or embedding similarity, which has historically discouraged their use in LLM alignment.
On the surface, replacing reward models with
ended language generation [70, 10 58
44
12
33
ing human preferences. In experiments on general instruction-following tasks in the LMSYS chatbot_ seabbbt_aree__coom aetooaaareaacooersatttiiiio
conversations dataset, BLEU with five synthetic references achieves almost the same agreement (74.2%) with human preferences as a powerful 27B-paramete reward model (75.6%). ). We observe that reference quality critical: references generated by powerful LLMs (e.g., Sonde--.7-Sonnet, GPT-4o) yield significantly higher agreement than those from weaker models. Our analysis reveals that BLEU's strong alignment signal comes from rewarding properties like factuality and proper formatting that are critical to instruction following tasks.
directly using BLEU as reward for RL-based alignment. Motivated by the unexpectedly high agreement between BLEU and human preferences, we propose BLEUBERI, which uses RL to optimize BLEU on general cestructiio-followiii
BLEUBERI:
data. Previous efforts that used n-gram metrics like BLEU as rewards [43] (68) I6) faced obstacles such as unstable training [ 45) and degraded output quality 30 19 2n. We revisit this line of work using modern LLMs within the paradigm of reinforcement learning with verifiable rewards (RLVR) 224 (2), which shows the effectiveness of simple, transparent rewards. BLEUBERI treats BLEU verifiable reward for general instruction-follkee following tasks, using group relative policy optimization (GRPO) (50) to optimize
pretrained base LLM. We apply GRPO with BLEU rewards subset of challenging instructions for which the base model's outputs initially have low BLEU.
Strong instruction-following formance without a reward model: Across three different base models and four diverse instruction-follovingg bench- marks, including ArenaHard[226 and WildBench(28) BLEUBERI matches (and sometimes exceeds) the performance of reward model-guided RL and SFT according to both automatic and human evaluations. This is a striking result, given the simplicity and cost-effectiveness of BLEUBERI compared to training and deploying large reward models for RLHE. Human evaluators find that BLEUBERI- trained models just good those from models aligned with reward models. Furthermore, BLEUBERI models produce more dictually-grounder responses than those aligned with either reward models SFT. Taken as a whole, ours is the first work show that optimizing BLEU-far from overfitting to superficial n-gram matches-actually promotes helpful, factual, and well-formatted responses on general-domain instruction-following tasks. Given access to high-quality
per-
Metric agreement with human preferences
BLEU-Skywark.c www.workk..M22.0. 27B
77.6
BLEU Skywork RM-98 9B
76.7
Skywork
Skywork RM 88
74.5
Metric/ Configgraaai Configuration ation
(5 refi
BLEU (4 net)
73.5
BLEU (3 rof)
72.8
BLEU (2 rofi
72.4
(1 refi
72.3
BERTScore
72.7
BLEU-ROUGE-L mean
ROUGE-L (with stemming)
70.3
I000 X5.15
60.0 62.5 65.0 67.5
70.0 72.5 75.0 77.5
00.0
% Agreement with human preferences
Figure 1: Human agreement rates for BLEU (with vary- ing numbers of references), two reward models, and other reference-based metrics (with single Claude ref- erence). BLEU becomes more competitive with reward models as more references are provided, and combining BLEU with reward model outperforms either alone.

---

synthetic references (readily available via existing datasets or generated from powerful LLMs), our BLEUBERI method presents novel alternative to alignment that entirely avoids training large, complex reward models.
2 How well do simple reference-based metrics capture human preferences?
In this section, we first investigate how well reference-based string matching metrics correlate with with human preference judgments on publicly available instruction-followw we generate synthetic references for a subset of examples, and we find that BLEU is surprisingly competitive with ofate-of-theaar reward models in terms of agreement with human preferences. Moreover, BLEU's agreement with human preferences improves with more synthetic references (especially those from more powerful LLMs).
datasets. To do so,
2.1 BLEU: an 7-gram matching metric
widely-used metric for machine translation BLEU (Bilingual Evaluation Understudy) 41
evaluation. It measures the overlap between a predicted translation and one or more reference translations using modified n-gram precision (n E 1,2, 3, 4), combined with a brevity penalty (BP) to penalize overly short outputs:
N Σ
ife>r r BP = cxp(1 - E) i
BLEU BP exp
Wh log Pn
Here, Pn is the modified n--ramm precision, W, their weights, and BP adjusts for length differences between the prediction (c) and the closest reference (r) When multiple references are available, BLEU computes n-gram matches and takes the maximum count across all references for each n-gram.
2.2 How well does BLEU align with human preferences in single- and multi-reference setups?
To explore this question, we perform an analysis on the LMSYS wwwtttt_tree__oooe conversations dataset (79), which contains conversations evaluated by real users Chatbot Arena. Each instance includes an instruction, two model-ge dederreeee outputs (Ox and Oy) and a human preference label. We randomly select a subset of 900 instances from this dataset (more details in A. then evaluate the following metrics on each pair model outputs (Ox, Oy):
Lenghh baseline: Prior work has found that humans tend to bias towards longer responses 6 46). and that RLHF post-training may implicitly optimize for length over quality [51]. To quantify the impact of output length, we implement a simple baseline that always prefers the longer output.
Reward models: We use strong reward models trained on well-curated preference data: (RM-27B) and 8B). These models assign scalar scores to responses based on given instruction, without needing reference answers. They rank 4th and 11th, respectively, on RewardBench 225
(RM-
Since the Chatbot Arena dataset lacks ground- -ound-truth responses, we construct a set of synthetic reference responses from diverse set of LLMs For each instruction, compute BLEU for Ox and Oy using one or more references for this instruction. The response with the higher BLEU score is considered the winner. We evaluate both single- and multi-reference setups.
BLEU:
'In our experiments, we the huggingface implementation with tokenizer 11a, we apply smoothing to prevent zero scores for higher-order n- n-gram precisions when no matches are found. However, the BLc implementation is much faster and is therefore recommended for future use.
*We do not evaluate the full 33K dataset reduce costs associated with collecting synthetic reference outputs from variety L.Mss. addition, later in our training experiments, we need to collect such synthetic references at much larger scale, so we choose to minimize costs for this analysis.
*These reference models include: Gemini-2.5-Pit
B.] Claude-3.7 3.7dee-t (2) o4-mini 38 Deepseek-
V3 [9) Qwen-Mat 42). GPT-4o [37] Llama-3 Moama---8B-Instreee 8n-Insttuctt M OLMo2-7B 7B-Instruet [36] and Qwen2.5-0.55
Instruct [42] During the reference collection process, some soused models like Gemini would refuse to respond to certain prompts due to built-in safety filters. For our analysis. we only include prompts that received valid responses from all reference models. This results a final set of 889 prompts.

---

Other reference-based metrics: We also evaluate ROUGE 29 and BERTScore 78 two other popular reference-based metries, later in this section.
BLEU agreement increases with more references. As established in $2.1 BLEU is a multi- reference metric. Figure I shows that increasing the number of references used by BLEU (up to five) improves its human agreement, reaching 74.2%. For comparison, the length baseline achieves 65.1% % agreement, whereas RM-8B and RM-27B respectively reach 76.7% and 77.6%. To better understand BLEU's s behavior, we show in |A.3 both n-gram precision and the brevity penalty necessary to achieve high human agreement.
Reference quality matters. Using single reference and varying the reference model, we find that stronger models like Claude-3.7-Sonnet and GPT-4o yield over 72% agreement, while weaker models like Swvn2..-0.5BB. 5B-Innt perform worse (60.9%). even falling below the length baseline. More verbose models tend to yield lower agreement e.g., Gemini-2.5 2.5-Pro scores only 69.5% likely due to generating references 4.5x longer than outputs (Ox/Oy), ), compared to Claude's 's 1.6x. We observe a strong negative Pearson correlation (-0.78) between the absolute difference in length (between the reference and the outputs Ox/Oy) ) and BLEU's agreement with human preferences. We hypothesize that this is because the number of unmatched n-grams increases as the responses get longer, washing out any distinguishing signal. See $A.4 for more analysis on length effects.
Other reference-based metrics also exhibit high agreement. We also evaluate ROUGE [29]. which measures n-gram recall, and BERTS 778. which computes contextual embedding similarity. We also evaluate BLEU+RM, combined metric integrating BLEU with reward models via z-score normalization and averaging As shown in Figure 1 ROUGE-L (with stemming) BERTScore, and BLEU-ROUGE harmonic mean all exhibit human agreement comparable to BLEU. Notably, BLEU+RM (with single Claude reference) reaches higher agreement than either BLEU or the re-
Prompt
Vor Id papulation ton 2000 Give 2 answers. One Ukr Aerainiat and the other in Fiigt Fish Languages.
Nor Population
English
2630. the world population approxinately 6.1
billion people (6,100, .000000.0000 .00).. .00.))
#s Ukrainfan
y 2993 pout насеnенer cetty стзновито прибпиано 6,1
wimssuue rogee (6 160 960 000).
ward model alone, suggesting that they each focus on different aspects of the response. While all of these other rewards are promising to explore, we focus on BLEU for the remainder of this paper due to its simplicity and our limited computational resources.
Respanae A (BLEU:
Respouse (BLEU:
Rасеnеннe catry
uniter Nat According vari the
2008 pout nano 5.1 Minasped vonoeing K.
Fial pos0 on in 2006 was
oillion: Accorsing the
tng Lish: world poputass on 2680 4a5 approximatel
world poputation 2688 billion.
people.
Types of n-grams that contribute to BLEU's agree- ment: When BLEU makes a pairwise judgment that aligns with human preferences, which n-grams con- tribute to that decision? We show one such example in IFigure 2] Here, the presence of Ukrainian and English headers, along with the value 6. 1, contribute to its higher BLEU score, showing that BLEU can effectively capture both format and factuality (additional examples in AA.5).
The highlighted matching grams in this example show that BLEU can capture correct instruction foollwinn behavior as well as the factuality of the response.
Figure 2:
2.3 Domain-specific agreement analysis
To assess BLEU'S effectiveness across domains, we analyze 18,219 filtered examples from the Chatbot Arena dataset using Slaude-3.7-Somme as the reference model. Examples are grouped into six domains: QA, Code, Writing. Math/Reasooing, Multilinguualtty and Planning. Figuree 7 shows BLEU aligns best with reward models in Writing and Code, but performs poorly in Math/Keasoning. likely due to surface-level variation in mathematically equivalent expressions. Limitations may also stem from tokenizer zer_13a's
13a's 's handling of equations and non-whitespace-ddlliiie languages. Full
figure and additional details on this analysis are in $A.6
'We apply z-score standardization to both scores across all examples, then average them to compute a combined score per example. Without stemming, observe drop of 1.8 in agreement with human preferences.

---

3 BLEUBERI: aligning language models to follow instructions with BLEU
Having established that BLEU agrees with human judgments at a similar rate powerful reward models, we turn to actually using BLEU as reward during LLM alignment. Below, we describe our experiment setups and results. In total, all our training experiments (including main and exploratory runs) cost 1357 Nvidia GH200 hours. Across four benchmarks, models trained with BLEUBERI perform comparably to those trained with SFT or GRPO-RM. To our knowledge, this is the first work to demonstrate that reinforcement learning using BLEU directly as the reward can be surprisingly effective for general instruction following.
Training methods for alignment: Besides standard SFT, we also train models using Group Relative Policy Optimization (GRPO) [50), which fine-tunes a language model by sampling K candidate responses yk for a prompt scoring them with R(y,; æ), and computing the group-normalized advantage. We instantiate the reward function R(y;, 3r) ) in two ways:
GRPO-RM:
R(yk: 2) = RRM z),, where the reward is provided by RM-8B/ While reward
models are often used in methods like PPO [47]. they can also be directly used in GRPO to score and rank outputs within group [50, 74]
2. BLEUBERI: R(yk: 2) = BLEU yk. Ref(x))), where the reward is the BLEU score computed against one or more reference responses for x:
We use GRPO with a reward model to enable a controllable comparison to BLEUBERI, which uses GRPO with BLEU reward. GRPO has been shown to be just as effective as other algorithms like PPO [47] when used with a reward model [17 63]. Scoring outputs with BLEU can be up to 48 times faster than using RM-8B (more in $B.I. Although GRPO has been widely applied to reasoning-intensive tasks [8 71], our experiments enforcing reasoning behavior did not yield strong results (more in B.12)
3.1 Training data
In preliminary experiments, we find that while BLEUBERT can work with randomly-selected data, performs best when trained on data with low initial rewards. Specifically, we first construct data pool of prompts, run the base model on each, and compute the BLEU score of its outputs against the reference responses. Prompts with the lowest BLEU scores are treated as "hard" examples. Full experimental details on the data difficulty ablation are provided in $B.2 While the selection hard data requires collecting references for larger pool, obtaining them (via existing datasets LLMs) is still generally cheaper than collecting human preference data and training large reward models.
Creating a data pool: We draw from the Tulu3 SFT mixture (24], which contains 939K examples across 18 data sources covering diverse tasks. Motivated by the strong human agreement of BLEU on writing tasks from 2.. we filter and sample 50K prompts to form our final data pool, where the majority of examples are related to writing. See more details in BB.4
Training hard examples: To ensure consistency in main experiments, train all methods on the 5,000 hardest examples as ranked by BLEU. While this strategy might seem biased against GRPO-RM, we also train GRPO-RM on hard negatives selected by RM-8B in {B.3 show similar performance, indicating that hard negatives do not matter as much when using a reward model. Concurrent work also highlights the benefits of training on difficult data: the GRPO pipeline in the recently released Qwen3 report trains exclusively on 4,000 samples selected to be as challenging as possible [56].
Collecting references for BLEUBERI: Unlike the Chatbot Arena dataset used in $2 Tulu3 includes ground-truth responses for each instruction, which we refer to Tulu3 references. Within our 50K data pool, 45.2% % of the Tulu3 references are human-written responses sourced from previous datasets, while the remaining half are synthetic outputs from LLMs like ChatGPT. In addition, to evaluate the effectiveness of using completely synthetic references from different LLMs, we collect
'We do not train with RM-27B due to computational constraints. As shown in both RM-27B and RM-8B achieve comparable agreement with human preferences.
5

---

outputs from Claude-3.7- 3.nnnnnt,, Gemini-2.5-Pro 04-mini, Deepseek- V3, and Llama-3.1-4 3.1--B-Instruct (more details in $B.1Op. and refer to them as additional synthetic references. In our main experiments (Table T. we use Tudu3 references for hard data selection and training, and discuss results using additional synthetic references later $3.
3.2 Training and evaluation setup
Base models and hyperparameters: We conduct our experiments using four base models: Qwen2.5-14B, Qwen2.5-7B. Qwen2.5-3B. and Llama-3.1-8B) Compared to Qwen, Llama-3.1 8B is significantly weaker base model, having likely seen minimal or no instruction-following data during pretraining. To address this limitation, we train Llama-3. 1-BB on the 50K data pool for one epoch to equip with basic ability to handle chat-style prompts. We refer to this SFT-initialized version of Llama as the "Llama base model" in the remainder of this work. This setup aligns with prior observations that effective GRPO training requires a sufficiently capable base model [1.. For details about GRPO on Qwen after SFT initialization, see [B.5 For all methods, we train for one full epoch on the BLEU-selected 5K data I0 Note that GRPO and SFT use fundamentally different optimization strategies, so it is hard to ensure completely fair comparison. More discussion on this and results for multi-epoch SFT training are in BB.7
Benchmarks: We evaluate our models using four benchmarks: MT-Bench 79 set of 80 manually curated, high-quality multi-turn questions; ArenaHard v1 and v2 (26), two distinct sets of 500 challenging prompts drawn from real-world user queries. ArenaHard v2, released in April 2025, contains an updated collection with more difficult prompts than v1; and WildBench (28). comprising 1,024 complex real-world queries. Inference on these benchmarks are run using vL.LM 23 with greedy coding. We also report 95% confidence intervals computed from bootstrapping. More details on running each benchmark are $B.8 All benchmarks are evaluated using the jujuee framework. To balance cost and performance, we select gpt-4. 1 1-mini as the judge for all benchmarks.
3.3 Experimental results
BLEUBERI performs on par with GRPO-RM and SFT across all benchmarks. Across four base models and four benchmarks, BLEUBERI achieves performance comparable to GRPO-RM (Table 1), as indicated by their closely matched scores and frequently overlapping 95% confidence intervals. This aligns with the human agreement analysis presented in the previous section.
BLEUBERI does not compromise creativity. One potential concern with BLEUBERI's single- reference optimization is that it could lower performance on open-ended creative tasks where a single instruction has many valid responses. To evaluate this, we examine performance on the WildBench Creative Tasks split. The results show that Qwen2.5-7B models trained with BLEUBERI (66.7), GRPO-RM (67.2), and SFT (60.4) perform similarly, indicating that BLEU-based optimization does not hinder creative capabilities. See detailed results in B.9
BLEUBERI with fully synthetic references also shows strong performance. In 2.2 we analyze how well BLEU aligns with human judgments in a single-reference setting, varying the reference model. To investigate whether stronger BLEU-human agreement translates into better training outcomes, we train Qwen2.5-7B using the additional synthetic references collected earlier. We observe Pearson correlation of 0.34 between each reference model's human agreement and the resulting trained model's 's performance, indicating a moderately strong relationship. Among these, Claude and o4-mini produce the best results, matching the performance of GRPO-RM. We also experiment with a 5-reference training setup and find that the resulting model outperforms most of
"We not use the instruct (officially post-trained) versions these models, as they have already undergone extensive post-training optimization for instruction following.
FFor SFT, use learning input and output) to 1024. For GRPO, we set the learning rate to le-6, group size max prompt length and max generation length to 512 tokens, and maintain the same global batch size of 32- meaning each batch consists of 8 generations for each of 4 unique prompts, All training runs are performed on single GH200 GPUs using TRL with DeepSpeed-Zedd ZeRO3 (https: htttps://www..o http::/wwwddeeeeeeeeee//////////
of 5e-6 with global batch size of 32 and max tokens (covering both
httl).

---

Table 1: Results on four cestructioo-folloiiin benchmarks with 95% confidence intervals. For each model, the "Base" row represents its pretrained checkpoint (for Llama-3. 10ma--311-B. this is our SFT-initialized model described $3.2). while the Instruct row the official post-trained checkpoint. Despite the limitations of n-gram matching, BLEUBERI remains competitive with both SFT and GRPO-RM across all models and benchmarks, as indicated by the closely aligned scores and frequently overlapping confidence intervals.
Model
Variant
MT-Bench
ArenaHard v1
ArenaHard v2
WildBench
Average
Base
66.3 #3.8
20,9 #2.1
8.0 +1.0
57.2 #1.4
38.1
SFT
70.7 #3.9
29.0 #2.5
13.6 =1.3
63.5 +1.
44.2
Qwen2.5-14B
GRPO-RM
78.6 +4.0
36.0 #2.5
16.3 #1.4
68.4 +7.2
50.5
BLEUBERI
74.2 23.8
32.8 #2.2
16.5 =7.4
66.7 #7.2
47.6
Instruct
84.6 +3.0
58.5 #2.7
29.7 #
75.7 20.9
62.1
Base
63.5
16.2 11.6
5.6
51.8 17.6
34.3
SFT
67.5
22.1 #2.1
9.9
60.5 #1.3
40.0
Qwen2.5-7B
GRPO-RM
76.9 433.5
29.8
12.2 #1.2
64.8 +1.1
OLEEUBERI
70.8 #37.7
29.3 #2.2
12.9 =1.3
65.4 #7.3
44.6
Instruct
78.8 133.4
37.9 +2.7
16.8 #1.4
71.0 #1.1
51.1
Base
61.1 833.8
7.0 #100
3.1 #0.6
49,0 +1.3
30.1
SFT
59.6 14.2
9.8 =1.1
4.0 #0.7
55.6 #1.4
32.2
Qwen2.5-3B
GRPO-RM
67.8 14.1
12.8 17.4
59.2 +7.2
36.2
BLEEBEER
64.6
11.0 +7.3
3.7
56.1 #7.2
33.8
Instrct
70.0 #33.9
16.8
6.7
63.0 +7.2
39.7
Base
52.9 44.0
6.8 #1.1
2.1 +0.4
55.0 +1.3
29.2
SIT
56.7 =4.5
12.6 #1.6
3,0 #0.6
60.3 +1.3
33.2
Llama-3.1-88 3.1-BB
GRPO-RM
57.2 1:1.3
9.7 11.4
1.6 #0.5
57.8 17.4
31.6
(SFT init.)
BLEUBBERI
56.9 24.3
10.1 #1.5
2,3 #0.5
59.5 #7.2
32.2
Instruct
65.8 #3.88
24.9 #2.4
5.8 +0.9
64.2 +1.2
40.2
the models trained using only a single reference from the same set. Further experimental details and discussion provided in BB.10
Due to computational constraints, our experi- ments primarily focus BLEU the reward funetion. Nevertheless, find that GRPO training with alternative rewards- such as BERT'Score and BLEU-ROUGE harmonic mean-con can achieve performance comparable BLEU. We provide details rewards in $B.11 and hope our findings can encourage more comprehensive explorations other reward functions future work.
Training with reward functions beyond BLEU:
experiments with these alternative
Random rewards do not yield meaningful gains on general instruction following. Shao et al. [49] show that random rewards improve math reasoning in Qwen2.5- models [74), but not in Llama [ or OLMo (36], by eliciting latent reasoning strategies. We test whether this signal also aids general instruction following by training Qwen2.5-7B and Qwen2.5-3B with GRPO using random reward defined as R(ye, x) Uniform(0, 1), and report results in $B.6 Unlike Shao et al. 49 who find large gains in math reasoning, observe improvement
instructioo-follooiin benchmarks as we see no meaningful change for Qwen2.5-77 and degraded performance for Qwen2.5-3B. This suggests that BLEUBERI's benefits are domain-specific and extending RLVR beyond math remains an open challenge.
Analysis and human evaluation of model outputs
In this section, we extend beyond benchmark numbers and look into various qualitative properties of -onee-generated outputs. Our qualitative observation that the GRPO-traine trained model notably follows instructions. whereas the base model often fails to do so. We also find that SFT-trained models tend to generate more verbose and repetitive text compared to BLEUBERI, while GRPO-trained models more frequently use markdown formatting (see I for detailed example and results). Outputs generated by BLEUBERI are also more factually accurate than GRPO-RM and SFT, while a human evaluation shows that BLEUBERI's outputs are rated similarly to those of GRPO-RM.
7

---

4.1 Surface-level qualitative characteristics
In Table 14 we show that SFT models tend to produce more verbose and repetitive outputs. GRPO- RM models exhibit slightly higher refusal rate (i.e., where the model refuses to follow the user instruction). Both BLEUBERI and GRPO-RM models use markdown formatting more frequently than SFT models, with GRPO-RM showing the highest usage. In C we explain how these statistics are computed and provide example outputs.
Qwen models trained with GRPO frequently use affirmative openers. BLEUBERI-trined Qwen models frequently begin responses with Certainly!" (51.6% for Qwen2.5-3B, 27.1% for Qwen2.5-7B), while GRPO-RM models often use Sure!" " (70.6% and 35.2%). These phrases are rare in Qwen base models and all Llama variants 11| The
09
87.0
GRPO AM
DL (LEUBER)
Score Score (%)
frequent use of "Certainly!" is unlikely reward hacking, as the phrase appears in under 1% of the Tulu3 references used our main training experiments. A more plausible explanation is that GRPO amplifies subtle biases latent Qwen's s pretraining, even if absent in base model outputs. This tendency has been observed previously: for instance, the system prompt for previous version of Claude explicitly states: Specifically, Claude avoids starting responses with the word "Certainly" in any way. [2] See more details on this analysis in CC2
be
60
60.7
60.5
51.4 50.5
50
.5.0
40
30
FreshQA (4)
Askhistorian (21) Long
Figure 3: Factuality results for trained Qwen2.5-7B models across three QA datasets evaluated using VERISCORE [53]. The K values (in parentheses on the x-axis) used for each dataset follow the original paper.
4.2 Factuality of model outputs
While BLEU has often been criticized for its weak correlation with factual accuracy due to its lack of sensitivity to semantic content [18 72
recent studies have explored more effective ways to Building on this line of work, we evaluate the factual accuracy of variants of Qwen2.5-7B BB models, using VERISCORE [53] an automatic metric that evaluates factuality in long-form generation tasks [3
65
48
leverage BLEU for factual text generation 59
We use three datasets covering different domains and report Fl @ K as defined in VERISCORE where recall is relative to median number of claims K for a given dataset. FFiuure 3 shows that BLEUBERI consistently outperforms GRPO-RM and SFT on LONGFACT and ASKHISTORIANS and performs
par with GRPO-RM on FRESHQA. These results suggest that optimizing for BLEU using RL can guide models toward more factually grounded generations. See example outputs in Figure 4 and C.3 for more evaluation details.
4.3 Human evaluation of BLEUBERI and GRPO-RM model outputs
Although BLEUBERI models performs competitively with GRPO-RM models on benchmarks, valid concern is that its reliance on n-gram matching could lead to reward hacking, as such matches are relatively easy exploit. To assess whether BLEUBERI produces outputs that align with human preferences, we conduct human evaluation comparing outputs from Qwen2.5-77 models trained with GRPO-RM and BLEUBERI. We sample 120 examples (30 from each benchmark) and asked two annotators to compare the outputs from the BLEU-trand -urinned and RM-trained models, denoted O and On respectively
To prevent implicit bias, we remove affirmative openers for each model
SFT models often begin with To... (e.g.. To address your query...) in ~200225% 25% of responses, similar to base models, suggesting this is not learned during SFT.
"https://doct aecdooss.nttiii. anthropic. wwt.////.eeeeeeeceeeeee htw.///ereeeeeeeeeeeeeeeeeeeeeee woceee-nottsssssseee sttteess/ssteeeppteee prompts specifically the system prompt
for 3laude-3.5-Sonnee July 12th, 2024.
"VVerSSOOrE measures the factuality within the response itself claim level; It does necessarily indicate whether the response correctly answers the question.
"These annotators are co-authors of this paper, but they did not participate in setting up the annotation task and had no prior exposure any model outputs.
8

---

Q: How many Chinese paddlefish left in the wild?
The Chinese peddlefish (Pesohunue gladiae) is ortiically endingered speciss of fish that was once abundent the Yangize River Ohina. It that
paddleffih caught 2005, the species in the wild. The Chinese padclefish now cons dered be functionaaly extinet mearing that it no longer viable in the wid and is uniially to recover without significatt conservat on efforts.
wild
then. nave confirmed sightings capturee
once abundant tall destruction, and the construction of damss, the popuation Chinese packleefiih has dredically declined As 2020, there have been no confirmed sightings of the Chinese paddlefah in the wild, and it is beleved that the speniia is functionally extinct. The last confirmed record a wild Chineee paddlefish was 2003, when one was caught in the Yangtze Fllver Since surveys
The Chinese paddlefish (Pesshuuue gladiae) orttically endangered species However, duue to overfistingg
Yangloe River system China.
GRPO-RM
toring efforts have Natura (UCN)) officially declared the Chinsse paddleliih extinct, stating wild. Conservation efforts have focused on captive broading programs, but these have not been succeschi establishing viable popuration
to any inchviduals,
the international Union Conservation of no massonable expectation the spaciss" survival
The Chinase paddletshh, also crtiicaly endangered spacles and declared extinct 2000. last contrrmed sighting of live Chineso padcleftsh was 2003, and extendive efforts to find and protect the species were unsucessful.hee decine of the Chinese paddlefish was primanly due to human activites sush as overfishing, habitat destruction and the construction of dome along the Yangtze Fiver, which disrupted the fish's migration petterns and breeding grounds summery, there Chinese peddlefish the wild.
Psaphunus gladius, extlinc spacies
that native Yangize Fiver China.
BLEUBERI
An example instruction from FreshQA, where red highlights indicate factually incorrect claims. For this instruction, BLEUBERI produces more factually precise output than GRPO-RM.
Figure 4:
they exist in the outputs (see $ For each pair, we anonymize the model identity and randomize their order. In addition to direct preferences, we also allow a "tic" option.
Humans judge BLEUBERI outputs to be equally as good as those from GRPO-RM. We compute soft preference rate, defined as the proportion of times annotator judges Og to be at least as good as OR While this metric provides a coarse measure of overall favorability toward Og, it should not be interpreted as equivalent to strict preference rate, since ties do not necessarily indicate endorsement. The results are visualized in Figure IT We observe a soft preference for OB in 67.5% of Annotator I's judgments and 52.5% of Annotator 2's. Out of the 63 cases where both annotators express a clear preference (i.e., not a tie), the Cohen's 's Kappa agreement is 0,34, indicating fair agreement 16 These results suggest that human evaluators generally view BLEUBERI outputs as roughly comparable to those of GRPO-RM. More details in $C44
5 Related work
Prior approaches to training with BLEU optimization: A substantial body of prior work has explored sequence-level reinforcement learning for tasks such machine translation, using BLEU as a reward signal. One line of research centers on Minimum Risk Training [35 52 76 27, 20). which directly minimizes expected task-specific loss, enabling optimization
bin-differentiable metrics like BLEU Another line, grounded in policy gradient methods such as REINFORCE [67]. faces well-known challenges including high-variance gradient estimates 3 45 and degraded output quality
30 19 21 To mitigate exposure bias, Ranzato et al. 43 proposed the MIXER algorithm, which also directly optimizes BLEU through a mixed objective. Despite their prevalence, BLEU and other n n-gram-based metrics have been shown to correlate poorly with human judgments across multiple domains, including machine translation
33 [66) summarization [22,, code generation [I], and question answering [69 61. In this context, our work is the first to investigate the use of BLEU as a training signal for general instruction following with modern L.Mss
RLVR with simple metries in other domains: Recent work has found that in mathematical
reasoning tasks, reinforcement learning with simple rule-based rewards can be surprisingly effective
even without the use of reward models [8] 62 75 711. Beyond math, similar efforts have extended to
other domains such as story generation 14), visual perception (32), and medical reasoning [77). Of particular relevance to our work that of Lambert et al. (24. which introduces the term "RLVR" and investigates verifiable rewards for -ooosticallyy tasks; we build on this work by using BLEU as a form of verifiable reward for general instruction following.
Soft preference rate = Og wins ties. "'Our annotation task required subjective judgments on highly similar, high-qualit, responses to complex prompts from four benchmarks MMTcBench, ArenaHard v1/v2, WildBench). These often involved technical or Comain-specific content, making clear preferences difficult. The frequent use of the "tie" option by one annotator further suggests the closeness in output quality.

---

6
Conclusion
In this paper, we revisit reference-based metrics for aligning LLMs and demonstrate that BLEU-a simple n-gram overlap metric- surprisingly well with human preference judgments on general iestruction-oolloinng tasks. While earlier attempts to directly optimize BLEU with RL ran into challenges with unstable training and generated artifacts, our results show that BLEU is indeed a practical reward signal for modern alignment. Our BLEUBERI approach, which optimizes BLEU on open- been-weight language models using the recentl-deevloped GRPO algorithm, achieves performance on par with reward model-guided RI across diverse cotructiio-followiini benchmarks and model scales. Additionally, human evaluations and factuality analyses confirm that BLEUBERT's outputs are as helpful and often more factually grounded than those from comparable L.IMs aligned with reward models. BLEUBERI's success is enabled by stronger base language models, high-quality synthetic reference outputs, and better training algorithms, which collectively address the pitfalls that undermined earlier BLEU-based RL efforts. Overall, BLEUBERI is lightweight, cost-effective alternative to reward model- -uided aligmment, and we hope it facilitates future work in reward design and alignment strategies without expensive human preference supervision.
7 Limitations
While BLEUBERI demonstrates promising results as a lightweight alternative to reward model- based alignment, our study has several limitations. First, experiments limited in scope, covering only two model scales, a moderate-scale data pool (50K examples), and one string-overlap metric (BLEU). We do not fully explore the effects of scaling model size, data volume, training time, or training with other alternative metrics. Second, we do not perform extensive hyperparameter tuning for each model and setup due to computational constraints. Third, BLEU's reliance on surface-form n n-gram overlap makes it sensitive to reference quality and vulnerable in domains with high lexical variation, such as mathematical reasoning and multilingual tasks.
8 Acknowledgements
We thank the University of Maryland Computational Linguistics and Information Processing (CLIP) Lab for their feedback and support. We are grateful to Lambda AI for providing GPU resources for the training experiments. This project was partially supported by awards IIS2046248 and IIS-2312949 from the National Science Foundation (NSF) well as an award from Open Philanthropy.
References
[1] A Meta. Llama 3 model card. 2024. URL http://pp//gggttoococ////aa////p
blob/main/M00Et CARD. md.
[2] Anthropic Claude 3.7 sonnet and claude code.
http::/www..artrqppcc..c/////
2025. Accessed: 2025-02-24.
[3] Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. An actor-critic algorithm for sequence prediction. In International Conference on Learning Representations, 2017. URL https::/ net/forum?id=S
[4] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage. Zac Hatfield-Dodde Danny Hernandez, Tristan Hume. Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training helpful and harmless assistant with reinforcement learning from human feedback, 2022. http:httts://arre http://aaxxi...o//bbbbbb// 2204.22222222 05862
[5] Chris Callison-Burch, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in machine translation research. In Diana McCarthy and Shuly Wintner, editors, 1tth Conference of the European Chapter of the Association for Computational Linguistics, pages 249-250 226,, Trento,
10

---

Italy, April 2006. Association for Computational Linguistics. URL https://aclantthooogg Frg/E06-1032
[6] Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. Humans or llms the judge? study judgement bias. In Conference Empirical Methods in Natural Language Processing, 2024. URL httph://pp::p htttpp://ap..smmttticchooooooooo semanticscholar, wwweeroog//oooo wttp:oorps//t/..
267740522
[7] Hardy Chen, Haoqin Tu, Fali Wang. Hui Liu, Xianfeng Tang, Xinya Du, Yuyin Zhou, and Cihang Xie. Sft or rl? an early investigation into training r1 r1-like reasoning large vision-language models, 2025. URL
11468
[8] DeepSeek-AL. Deepseek-rl: Incentivizing reasoning capability in ilms via reinforcement learning, 2025. URL /RLp::tt:///rxxxioorr http://ww..rrg////5550/0// 12948
[9] DeepSeek-Al. Deepseek-v3 technical report, 2025. URL https///arxiv.orr////////
19437
[10] Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: How should we assess quality of the code generation models? Journal of Systems and Software, 203: 1741, September 2023, ISSN 0164-1212. doi: 10.1016/jss.2023311744
URL h/tp: www..//o.....//..oooo 10160///....111.111..... 20t.://....ss2.....1..1. 111741
[11) Mikhail Evtikhiev, Egor Bogomolov, Yaroslav Sokolov, and Timofey Bryksin. Out of the bleu: How should we assess quality of the code generation models? J. Syst. Softw, 203(C), September 2023. ISSN 0164-1212. doi: 10. jo..016.. URL httphtts//oii.rr///// 1016/ jss. 2023. 111741
[12] Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George Foster, Alon Lavie, and André F. T. Martins. Results of WMT22 metrics shared task: Stop using BLEU neural metrics are better and more robust. In Philipp Koehn, Loic Barrault, Ondřej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussă, Christian Federmann, Mark Fishel, Alexander Fraser. Markus Freitag, Yvette Graham. Roman Grund- kiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri, editors, Proceedings of the Seventh Conference on Machine Translation (WMT), pages 46 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022, Association for Computational Linguistics. URL https://aclaathhne
w0tg. wmt-1.21
[13] Gemini-Team. Gemini 2.5: Our most intelligent ai model. https: /bbos://blog. goog google/ http://gggleeeeeeinn//aaeeeeeeee Accessed: 2025-03-25.
-mdel- http:///eennnnnnoeeoeeeiiiinnnna upac--s-aatttsrchhh--eo
2025.
[14] Alexander Gurung and Mirella Lapata. Learning to reason for long-form story generation, 2025.
URL https://:///r wrg. wwt.:ogg//////... httg/bbs/2503.
22828
[15] Rajarshi Haldar and Julia Hockenmaier. Analyzing the performance of large language models on code summarization. In Nicoletta Calzolari, Min- Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue, editors, 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, Conference Proceedings, 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation, LREC-COLING 2024 Main Conference Proceedings, pages 995-1008. European Language Resources Association (ELRA), 2024. This work sup- ported by Agriculture and Food Research Initiative (AFRI) grant no. accession no. 1024178 from the USDA National Institute of Food and Agriculture.: Joint 30th International Conference on Computational Linguistics and 14th International Conference on Language Resources and Evaluation, LREC-COLING 2024 ; Conference date: 20-05-2024 Through 25-05-2024.
2024 Main

---

[16] Julia Ive, Zixu Wang. Marina Fomicheva, and Lucia Specia. Exploring supervised and un- supervised rewards machine translation. In Paola Merlo, Jorg Tiedemann, and Reut Tsar- faty, editors, Proceedings of the 16th Conference of the European Chapter of the Associa- tion for Computational Linguistics: Main Volume, pages 1908-1920, Online, April 2021. URL
Association for Computational Linguistics. doi: 10. wttp:////222....mmmaa...ammmm https://aclantiiolog..oom - eacl-main. 164/
[17] Fangkai Jiao, Chengwei Qin, Zhengyuan Liu, Nancy F. Chen, and Shafiq Joty. Learning planning-based reasoning by trajectories collection and process reward synthesizing. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors, Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 334-350, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18553/11//2224 emnlp-main.20 URL https::/accaanthology wwwpollooy..og/e/eee www.org//////// wtw.:/2.......mmpp-1mmm emnnp-mminn... 20/
[18] Jenna Kanerva, Samuel Rönnqvist, Rúna Kekki, Tapio Salakoski, and Filip Ginter. Template- free data-to-text generation of Finnish sports news. In Mareike Hartmann and Barbara Plank, editors, Proceedings of the 22nd Nordic Conference Computational Linguistics, pages 242- 252, Turku, Finland, September- October 2019. Linköping University Electronic Press. URL https::/aclanthollee www.:orgg///// W196
[19] Muhammad Khalifa, Hady Elsahar, and Mare Dymetman. distributional approach to con- trolled text generation. In International Conference Learning Representations, 2021. URL https::/openreet
net/forum?id=jjWwwe 9AbL
[20]
Samuel Kiegeland and Julia Kreutzer. Revisiting the weaknesses of reinforcement learning for neural machine translation. In Kristina Toutanova, Anna Rumshisky. Luke Zettlemoyer, Dilek Hakkani- Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou, editors, Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1673- 1681, Online, June 2021. Association for Computational Linguistics. doi: 10.1866.. 18653/v 1/2021 naacI-main. 133. URL htt.hhtts//aaaaathll.ogoo org/
wrg. 2021 nwa.l 133/
[21] Tomasz Korbak, Hady Elsahar, Germán Kruszewski, and Mare Dymetman. On reinforce- ment learning and distribution matching for fine-tuning language models with no catas- trophic forgetting. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. URL https:
/ //openreview. hwtp:.neffooeo
[22]
Wojeiech Kryscinski, Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. Neural text summarization: A critical evaluation. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- ural Language Processing (EMNLP-IJCNLP), pages 540-551, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10. 11518/611DD111 1051. URL httpss:/aclaanthlogg. wwwp:lloyyyyoooogggeo//// 00g
[23] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large lan- guage model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
[24] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu. Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tülu 3: Pushing frontiers in open language model post-training. 2024.
[25] Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khy- athi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardhench: Evaluating reward models for language modeling. httpp::/huggingge htsps////eecc.sssssssssaaaaa///a 2024.
12

---

[26] Tianle Li, Wei-Lin Chiang, Evan Frick, Lisa Dunlap, Tianhao Wu, Banghua Zhu, Joseph E Gonzalez, and Ion Stoica. From crowdsoureed data to high-quality benchmarks: Arena-hard and benchbuilder pipeline. arXiv preprint arXiv:2406. / 1939, 2024.
[27] Zhifei Li and Jason Eisner. First- and second-order expectation semirings with applications to minimum-riss training on translation forests. In Philipp Koehn and Rada Mihalcea, editors, Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing, pages 40-51, Singapore, August 2009. Association for Computational Linguistics. URL
https://aclanthollg..oorr. http://hhoooooooooo/0//---0000
[28] Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking LLMs with challeng- ing tasks from real users in the wild. In The Thirteenth International Conference on Learning Representations, 2025. URL httpp:://oppereeeeeeeee http:///pppneeeei..eeeee/e/rnnnn
[29] Chin- Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL
[30] Chia-Wei Liu, Ryan Lowe, Iulian Serban, Mike Noseworthy, Laurent Charlin, and Joelle Pineau. How NOT to evaluate your dialogue system: An empirical study of unsupervised evaluation metries for dialogue response generation. In Jian Su, Kevin Duh, and Xavier Carreras, editors, Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2122-2123311 2132, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/v1/D16- 1230.
org/ 1230/
[31] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du Wee Sun Lee, and Min Lin. Understanding rl -l-zero-like training: A critical perspective, 2025. URL https
//arxiv. //axxiv.rrg/ab//2222 wrgg/brg////....b./ 20783
[32] Ziyu Liu, Zeyi Sun, Yuhang Zang, Xiaoyi Dong, Yuhang Cao. Haodong Duan. Dahua Lin, and Jiaqi Wang. Visual-rft: Visual reinforcement fine-tuning, 2025. URL https::/arxiv..or Abs/2503.0172
[33] Nitika Mathur, Timothy Baldwin, and Trevor Cohn. Tangled up in BLEU: Reevaluating the evaluation of automatic machine translation evaluation metrics. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault, editors, Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984-4997, Online, July 2020. Association for Computational Linguistics. doi: (0.18653/v1/00000 www.amail-mm444 URL https:://aclanthologg/ 2oggg//0022 acl-main. 448/
[34] Nasrin Mostafazadeh, Adity Kalyanpur, Lori Moon, David Buchanan, Lauren Berkowitz. Or Biran, and Jennifer Chu-Carroll. GLUCOSE: GeneraLized and COntextualized story explanations. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu, editors, Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4569 4586, Online, November 2020. Association for Computational Linguistics. doi: 10. 1 12111663///2222.mmm.pp main..70. httpstts: 3707.
main org/
[35] Franz Josef Och. Minimum error rate training in statistical machine translation. In Proceedings of the 41st Annual Meeting of the Association for Computational Linguistics, pages 160-167, Sapporo, Japan, July 2003. Association for Computational Linguistics. doi: 10.3115/1075066. 1075117. URL htt.ttss//ac.aatholl.g////
1021/
[36] Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, Michal Guerquin, Hamish Ivison, Pang Wei Koh, Jiacheng Liu, Saumya Ma- lik, William Merrill, Lester James V. Miranda, Jacob Morrison, Tyler Murray, Crystal Nam, Valentina Pyatkin, Aman Rangapur, Michael Schmitz, Sam Skjonsberg, David Wadden, Christo- pher Wilhelm, Michael Wilson, Luke Zettlemoyer, Ali Farhadi, Noah Smith, and Hannaneh Hajishirzi. 2 olmo 2 furious. 2024. URL htttps://aaxxv..tttt abs 2201 .06656
13

---

[37] OpenAl. Gpt-4o system card, 2024. URL
21276
[38] OpenAl. Openai o3 and 04-mini system card, 2025. URL URLhhtps:://ccmm hpwp:://///dddoooiii..io conema.comm// com/ com/pdf/ 0221c875-02dc-777 htt.1//0000000043333222220200000 cessed: 2025-04-16.
Ac-
[39] Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll I.. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. Training language models to follow instructions with human feedback. ArXiv, abs/2203.02155, 2022. URL https://appi semanticscholar.oom
CorpusID: 246426909
[40] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744 2022.
[41] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation machine translation. In Proceedings the 40th Anmual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA, 2002. Association for Computational Linguistics. doi: 11115///07338331071333. URL https:///6ll.orr/10..1111//////1
1073135
[42]
Qwen-Team. Qwen2.5 technical report, 2025. URL URL [KLLtttps//aarki.oo/////22222211
[43] Mare' Aurelio Ranzato, Sumit Chopra, Michael Auli, and Zaremba Sequeee level training with recurrent neural networks. 2016. UK//itttps///rrrvv.rr//////11111
[44] Ehud Reiter. A structured review of the validity of BLEU. Computationaal Linguistics, 44(3): September 2018. doi: 10.1162/coli a a_00222. URL/hhtps://aclaattooooyy... org/ J18-3002/
[45] Steven J. Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. Self- critical sequence training for image captioning. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 11 1179-1195, 2017. doi: 10.1 10.1100//VPR...11.111
[46] Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. Verbosity bias in preference labeling by large language models, 2023. httpss//////ariioorrrr hts//bbb/2331 10076
[47] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms, 2017.
06347]
[48] Krithi Shailya, Shreya Rajpal, Gokul S Krishnan, and Balaraman Ravindran. Lext: Towards evaluating trustworthiness of natural language explanations, 2025. /tllhttts://arxiv.com
abs/2504.00 06227
[49] Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang. Sewoong Oh, Simon Shaolei Du, Nathan Lambert, Sewon Min, Ranjay Krishna, Yulia Tsvetkov, Hannanch Hajishirzi, Pang Wei Koh, and Luke Zettlemoyer. Spurious re- wards: Rethinking training signals in rlvr. https::/retthinn-rlvrnoo notion. site/
2025.
Notion Blog.
[50] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024. HRLLittts://aaxvvoorrg/// 03300
[51] Prasann Singhal, Tanya Goyal, Jiacheng Xu, and Greg Durrett. long way to go: Investigating length correlations in rlhf, 2024. URL httttt:////ttiiii
http://xxxi..oggab//////aa org..bs//////..//// 03716
14

---

[52] David A. Smith and Jason Eisner. Minimum risk annealing for training log-linear models. In Proceedings of the COLING/ACL 2006 Main Conference Poster Sessions, pages 787-794. Sydney, Australia, July 2006. Association for Computational Linguistics. URL https://
Sclaathologgy.or//P6--2111
[53] Yixiao Song, Yekyung Kim, and Mohit lyyer. VeriScore: Evaluating the factuality of verifiable claims in long-form text generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen, editors. Findings of the Association for Computational Linguisties: EMNLP 2024, pages 9447-9474, Miami, Florida, USA, November 2024. Association for Computational hat55//11224...niiiiii emnlp.552.
Linguisties, doi: 2024. findingsnnnnnnnnn. 552/
org
[54] Zayne Rea Sprague, Fangcong Yin, Juan Diego Rodriguez, Dongwei Jiang, Manya Wadhwa, Prasann Singhal, Xinyu Zhao, Xi Ye, Kyle Mahowald, and Greg Durrett. To cot or not to cot? Mhain-of-thoughe helps mainly on math and symbolic reasoning. In The Thirteenth International Conference on Learning Representations, 2025. URL http:h:://ppeeeeieeeeeeee// www.eiwwwwwe.oomiiiiii W6nlcS8Kkn
[55] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. http::tts://arii
org/ hwwgg//bb......//////// 31325
[56] Qwen Team. Qwen3 technical report. 2025.
Qwen3_ Teeh33_Thhcccca_epoooo
[57] Teknium. Openhermes 2.5 mistral http:s///wwgginggfeeeeer//////rr
5 5-Mistral-78) 2024.
[58] Katherine Thai, Marzena Karpinska, Kalpesh Krishna, Bill Ray, Moira Inghilleri, John Wieting, and Mohit lyyer. Exploring document-level literary machine translation with parallel paragraphs from world literature. Conference on Empirical Methods in Natural Language Processing. 2022.
[59] David Thulke, Nico Daheim, Christian Dugast, and Hermann Ney. Task-oriented document- grounded dialog systems by hitpr @ rwth for dste9 and dste 10. IEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:733-741, April 2023. ISSN 2329-9290. doi: 10.1 10.109//AAL122223.22738 URL httpp::///ooiii.o http:///ooo.org///// /0/ TT09/TASLP... 2023. 3267832
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris tian Cantón Ferrer, Moya Chen, Guillem Cueurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie- Anne Lachaux. Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams. Jian Xiang Kuan, Puxin Xu, Zhengxu Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melissa Hall Melanie Kambadur, Sharan Narang, eur'elien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.09228, 2023. URL https://api.. wetemannisccholl...
[61] Alex Wang, Kyunghyun Cho, and Mike Lewis. Asking and answering questions to evaluate the factual consistency of summaries, 2020. URL ftttp:://arxiv.org/abte httgg////////..///222222 04228
[62] Yiping Wang, Qing Yang, Zhiyuan Zeng, Liliang Ren, Lucas Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training htwpg//bb//550.... 20571
example, 2025. URL BRLLhhtts:///are
15

---

[63] Zihan Wang, Kangrui Wang. Qineng Wang, Pingyue Zhang, Linjie Li, Zhengyuan Yang, Kefan Yu, Minh Nhat Nguyen, Licheng Liu, Eli Gottlieb, Monica Lam, Yiping Lu, Kyunghyun Cho, Jiajun Wu, Li Fei-Fei, Lijuan Wang, Yejin Choi, and Manling Li. Ragen: Understanding self-evolution in Ilm agents via multi-turn reinforcement learning, 2025. URL https://axxi org/ 20000//2554.200 20073
[64] Ishaan Watts, Varun Gumma, Aditya Yadavalli, Vivek Seshadri, Manohar Swaminathan, and Sunayana Sitaram. Pariksha: A large-scale investigation of human-Ilm evaluator agreement on multilingual and multi-cultural data. In Conference on Empirical Methods in Natural Language Processing, 2024. URL https: www.sssescnnnnssosaooomee htt.://.......rrroooooossi/2223
[65] Johnny Tian-Zheng Wei. On conducting better validatioo studies of automatic metrics in natural language generation evaluation, 2019. httppts/////aa 1331/ 13362
[66] John Wieting, Taylor KirggKirrpattrick, Kevin Gimpel, and Graham Neubig. Beyond BLEU: Training neural machine translation with semantic similarity. In Anna Korhonen, David Traum, and Lluís Märquez, editors, Proceedings of the 57th Annual Meeting of the Association for Computational Linguisties, pages 4344 4355, Florence, Italy, July 2019, Association for Computational Linguistics. doi: 10. 1118655//V1/11--4277 URL httpst httpss///claaahooooooooooo org/
14277
[67] Ronald J. Williams. Simple statistical gradient-following algorithms for connectionist re- inforcement learning. Mach. Learn., ((3-4):229-2535 May 1992. ISSN 0885-6125. doi: 10. 10.1007/BF00099666. URL http:htss:
[68] Lijun Wu, Fei Tian, Tao Qin, Jianhuang Lai, and Tie- Yan Liu. A study of reinforcement learning for neural machine translation. In Conference Empirical Methods in Natural Language Processing, 2018. [tthsttss://aat wwwanticscholaattteeeeeee or wrgg/CrppsIID:ao.ooooooo 5210061
[69] Fangyuan Xu, Yixiao Song, Mohit lyyer, and Eunsol Choi. A critical evaluation of evalua- tions for long-form question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages 3225-3245, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v aclooo..c--ooo.1111. URL
www.ooy..og.o..ooeeooo long. 1817
[70] Fangyuan Xu, Yixiao Song. Mohit lyyer, and Eunsol Choi. A critical evaluation of evaluations for form question answering. Association for Computational Linguisties, 2023.
[71] Haoran Xu, Baolin Peng, Hany Awadalla, Dongdong Chen, Yen-Chun Chen, Mei Gao, Young Jin Kim, Yunsheng Li, Liliang Ren, Yelong Shen, Shuohang Wang, Weijian Xu, Jianfeng Gao, and Weizhu Chen. -hi-4--ini--raaooning: Exploring the limits small reasoning language models in math, 2025. URL https::///// wttp://wwww....///////.....4 21233
[72] Jingjing Xu, Yuechen Wang. Duyu Tang, Nan Duan, Pengcheng Yang, Qi Zeng, Ming Zhou, and Xu Sun. Asking clarification questions in knowledge-based question answering. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan, editors, Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLA) pages 1618-1629, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10.18653/11/D19-11777 1172. URL https://accan http://lllaahhoolgoooooo www.g..orgg//o orw.eyoog/01111112
[73] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Aligmment data synthesis from scratch by prompting aligned LLMs with nothing. In The Thirteenth International Conference on Learning Representations, 2025. net/
URL http:htttt///tm///e http:ettoooorrm//onnnn//oooaaaa
[74] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self self-immpovementt, 2024. URL https::/aara wrw. 12122
16

---

[75] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng. Yuxuan Tong. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Weinan Dai, Yuxuan Song. Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An opun-source Ilm reinforcement learning system at scale, 2025. wrw.
URL
14476
[76] Richard Zens, Saša Hasan, and Hermann Ney. A systematic comparison of training criteria for statistical machine translation. In Jason Eisner, editor, Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning SSMNLP-CoNLL pages 524-532, Prague, Czech Republic, June 2007. Association for Computational Linguistics. URL https://aclanthol..og http///aaanattolooorrrr///////// org/007-1055/
[77] Sheng Zhang, Qianchu Liu, Guanghui Qin, Tristan Naumann, and Hoifung Poon. Med-rlvr: Emerging medical reasoning from a 3b base model via reinforcement learning, 2025. URL
https://arxiv /tg//s////aaaaaooggog.ooggeeaa// 2502. 19655
[78] Tianyi Zhang* Varsha Kishore*, Felix Wu* Kilian Q. Weinherger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. /ttthttss///openeeeeee
http:////.eppeeeeeeeeeeeeeeennne nett/forummid-skehtccDDD
[79] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li. Dacheng Li. Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
[80] Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. LIMA: Less is more for aligmment. In seveth- Conference on Neural Information Processing Systems, 2023. URL hatt::::/opperrreeeee.e////ooooo
17

---

A Chatbot Arena human preference analysis
A.1 Obtaining the 900 subset
To obtain the 900 subset we use for our human preference analysis, we first filter out all examples where human preference label is "tie", and/or the instruction or either of Ox and Oy are shorter than 10 words or longer than 512 tokens. Then, we randomly sample 1,000 prompts and classify them using the approach described in A.6 then filter out 100 examples that receive "N/A" label to reduce noise.
A.2 Multi-reference configurations
Below are the configurations we use for BLEU's multi-reference setups:
1. 2 ref: Gemini-2.5-Pro, Deepseek- V3
2. 3 ref: Gemini-2.5-Pro, Deepseek- V3, o4-mini
3. 4 ref: Gemini-2.5-Proo, Deepseek- V3, o4-mini,
4. 5 ref: Gemini-2.5-Pry Deepseek-V3 V3, o4-mini, Gaude-3.7-Somne. Qwen-Max
5. 6 ref: Gemini-2.5-Pro,, Deepseek-V3, o4-mini, Claude-3.7 3.7de-,,--Sonee, Qwen-Max, GPT-4o
A.3 Ablation on components of BLEU
We separately show the effect of n-gram precision and brevity penalty in Figure 7 alone achieves high agreement with human labels.
Neither component
A.4 Impact of reference length on human agreement
In Figure 5
show the average number of tokens in the reference outputs generated by different LLMs, compared to the average tokens in Ox and Oy.
Average tokon counss reference
Figure 5: Token counts for difference reference model outputs, prompts, and the two model outputs to be scored.
A.5 Additional example for BLEU's n-gram attribution
Please see Figure for another example. Here, BLEU captures both formatting (e.g., JSON structure) and factual accuracy (i.e., providing the correct answer based on reasoning). This suggests that the BLEU score serve useful signal for learning factual accuracy and format consistency.
A.6 Human agreement rates across different domains
We classify the 18K filtered examples into six broad domains using 1-ama-3.1-8B-Insttee QA (information seeking, advice seeking, document QA, data analysis), Writing (writing, role-playing
18

---

Prompt
Assune you are rinancial adviser who flags fraudulent advices. Your task is to review the advice. delinited ted by o, given another financial advisor to their client. Question: Is the your output
traudulent? given by the tinancia: reasoning for the question above. 2. "Final answer" final following keys, answer whether the advice is fraudulen:. Just reply "Yes" if the advice is fraudulent, "No" fraudulent. Advice:
"Reasoning"
Reference
'ssoon
"Reasoning": 'The advice given the financial acvisor is problema:ic for several reasons: (one year), is an unreal guarantee in legitinate investments. \ntn2. No legitimste financial adviser can guarantee such extrenely high returns. as all investments carry risk and tuture performance diversification recomendations, ar qualifications about the patential for loss, Anin4. The prorise of 260 returns in simplistic and lacks the context, analysts. ar r1sk assessment that would "yptcally legitinate investment recomendations VninThis kind of pronise of specific, extrenely high returns 'Final answer"; "Yes"
peomises specific return (208%)
predicted
certainty.
The
any
disclosure.
is characteristic of investment fraud and Ponzi schemes, promises extraorcinary returns.Ankvn
advice
flag investment fraud.".
Response (BLEU: 8.7)
Response 2.77
"eesoning" : 'The statement that an investme сomрany will
Yes. The advice is fraudulent.. The promise of year unrealistic ood to be true. 1s Likely scan to lure inv estors in.
280% rn in one year 15 highly unrealistic and 11k ely fraudulent" "Final answer":"Yes"
88%
Figure 6: Additional example for BLEU's n n-gram attribution on cases where it agrees with human preferences.
Table 2: Hierarchy table of domains and their subcategories, grouped by characteristics. Each subcategory is sourced from (28).
Domain
Subcategory
# of examples
Information seeking
6516
Advice secking
824
QA
Document QA
411
Data analysis
189
Writing Roleplaying
2565
1112
Writing
Summarization
390
Editing
132
Math Reasoning
1302
Math and Reasoning
1064
Classification
662
Code
Coding and Debugging
2095
Multilingual
Multilingual
278
Planning
257
Planning
brainstorming
399
summarization, editing), Math/Reasoning (math, reasoning, classification), Code (coding/debuggingg) Mulrilinguality, and Planning (planning, brainstorming). See Table for the full taxonomy and the number examples in each subcategory.
These domains are derived by first assigning fine-grained labels based on the WildBench taxon- omy [28 and then consolidating them into higher-level groups for more interpretable analysis. Figure shows the human agreement rates with different metrics across all defined domains.
19

---

Metrics
% Agreement rate with human preferences
Length baseline
BLEU precision only
80
RM-88
RM--778
70-
314
60
50
Writing (4199)
Code (2095)
Planning (656)
QA (7940)
Math/Reasoning (3028)
Multilinguality (278)
Figure 7: Agreement rates of each individual metric Length, BLEU precision, Brevity Penalty (BP), BLEU, RM-8B, and RM--7B--v -ithhh human judgment across domains in our 18K filtered Chatbot Arena dataset.
Training
B.1 Runtime comparison between BLEU and reward models
See Table for details on the runtimes two different BLEU implementations and the two reward models. These values were obtained by rumning each method on single example, averaged over 100 runs. While our training experiments use the HuggingFace implementation. SacreBLEU can allow for much bigger speedup, offering considerable benefits reward computation using reward models.
Table 3: Average inference time of BLEU and reward models 100 runs example.
Metric
Time (s)
HF BLEU
0.0048
SacreBLEU
0.0008
RM-8B
0.0393
RM-27B
0.0732
B.2
Data difficulty ablation
In Table 4 present results training BLEUBERI data with varying initial rewards. These experiments were run on the 1K hardest examples selected by BLEU. While BLEUBERI does effectively improve upon the base model in all setups, its improvements most pronounced hard data.
B.3 Training GRPO-RM on RM-selected hardest data
In Table 5 we report results running GRPO-RM on the 5K data where the base models score the lowest according to RM-8B. We find these be similar in values reported in Table suggesting that GRPO-RM is not as sensitive data difficulty.
20

---

Table Difficulty ablation results on 1K data. BLEUBERI benefits the most from hard data.
Base model
Variant
MT-Bench
ArenaHard v1
ArenaHard v2
WildBench
Average
BLEUBERR-random
68.6
22.5
10.4
61.2
40.7
BLEUBERI-ent
62.4
12.9
4.8
53.7
33,4
Qwen2.5-7B
BLEUBERRI-mcdig
67.4
21.1
7.8
58.8
38.8
BLEUBERI-haR
73.0
30.9
13.3
64.0
45.3
Table 5: GRPO-RM model performance when trained the 5K hardest data selected by RM-8B.
Base model
Variant
MT-Bench
ArenaHard v1 ArenaHard v2
WildBench
Average
Qwen2.5-77 5-7B
GRPO-RM
76.3
30.1
14.0
62.9
45.8
Qwen2.5-3B BBBB
GRPO-RM
11.6
5.3
35.4
Llama-3.1 (SFT imit.
GRPO-RM
1.7
27.8
B.4 Processing Tulu3
Obtaining the 50K pool Among all 18 data sources in Tulu3, we focus on the five instruction- following data sources within this mixture: FLAN v2, No Robots, OpenAssistant Guanaco, Tulu Persona IF, and WildChat GPT-4. A full list of all 18 sources is available at ht http::://huggnngf... co/datasets/all wwwpet/ttaalaaaaaaaaeaeeaaaaaeae htt/:/utuuuu-1ee 3------fftt-n mefftmmixx mixture) From these, we filter out examples with instructions responses shorter than 10 tokens or greater than 512 tokens (based on Qwen2.5-7B tokenizer), and those that are not in English (labaled by langdetect). Then, we sample 50K examples, trying to balance between the 5 instruction-follee following data sources. In Table 6 provide detailed counts for examples from each source this 50K pool.
Table 6: 50K data pool source distribution.
Dataset
Num. examples
FLAN v2
13,706
Robots
7,403
OpenA ssistant Guanaco
1,479
Tulu 3 Persona IF
13,706
WildChat GPT-4
13,706
Tulu3 50K data pool task type distribution Please see Figure 8 for the distribution of task types in the Tulu3 50K data pool, labeled by 3Jama-3.1-8B-Insttn
B.5 Initializing Qwen base models with SFT before GRPO training
While Llama-3. 3.18BB benefits from SFT initialization, we find that this does not hold universally across all model families: applying SFT to the Qwen models before GRPO actually results in worse performance than applying GRPO directly. Similar findings have been reported recent work [
B.6 Training with random rewards
Please refer to Table 7 for training results with random rewards.
B.7 Multi-epoch SFT training
In our main experiments, we train both GRPO and SFT on our 5K hardest examples for one epoch. While this setup allows for controlled comparison, one might argue it is unfair to SFT, as the two methods involve different numbers of training steps. Specifically, GRPO uses group size of 8 and global batch size of 32, resulting in 1250 training steps per epoch. To match this step count for SFT, we would need to train for epochs. We address this Table where we extend our main results table to include three rows for SFT models trained for 1250 steps. These models perform significantly better than their --eooch counterparts. Nevertheless, BLEUBERI remains largely competitive with them. That said, it is important to note that there is strictly fair setup for comparing GRPO and
21

---

Table 7: Results on training with random rewards.
Model
Variant
MTT-Bench
ArenaHard vl
ArenaHard v2
WildBench
Average
Qwen2.5-7B
GRPO-random
65.0
17.5
5.6
53.2
35.3
Qwen2.5-3B
RaPOOrradoom
52.7
5.8
3.1
41.7
25.8
Table 8: Results four instruction-followee benchmarks, extended to include SFT models trained for the same number of steps (rather than epoch) as the GRPO models.
Model
Variant
MT-Bench
Arenallard v1
ArenaHard v2
WildBench
Average
Base
63.5
16.2
5.6
51.8
34.3
SFT epoch)
67.3
22.1
9.9
60.5
40.0
SFT (1250 steps)
72.1
31.4
13.0
66.7
45.8
Qwen2.5-7B
GRPO-RM
76.8
29.8
12.2
64.9
45.9
BLEUBERT
70.8
29.3
12.8
65.4
44.6
Base
61.0
7.0
3.2
49.1
30.1
SFT (1 epoch)
59.6
9.8
4.0
55.5
32.2
Qwen2.5-3B
SFT (1250 steps)
66.9
15.6
5.7
62.7
37.7
GRPO-RM
67.8
12.8
5.1
59.2
36.2
BLEUBERI
64.5
11.0
3.8
56.1
33.8
Base (SFT init.)
52.8
6.8
2.0
54.9
29.1
+ SFT (1 epoch)
56.8
12.6
3.1
60.3
33.2
4 SFT (1250 steps)
51.9
12.9
3.0
60.4
32.1
Llama-3.1-8B
GRPO-RM
57.1
9.7
1.6
31.6
BLEUBERI
56.8
2.4
32.2
SFT, as the two rely on fundamentally different optimization procedures. Any direct comparison
necessarily involves trade-offs in faimeess and equivalence.
B.8 Benchmark evaluation setup
For ArenaHard vl, we use gpt-4-031 as the baseline model, which the default for that benchmark. For ArenaHard v2, we instead use mini) is too strong, making it difficult to meaningfully distinguish the performance of our models. For WildBench, report the WB score.macro using their v2.0625 setup. As the original score ranges from -10 to 100, we rescale it to a 0-100 range in all our tables for consistency.
the baseline, since the default (o3-
For WildBench and MT-Benee we run full--ize bootstrap per model over item- level scores (n = valid prompts) with 1000 replicates. For Arena and v2, we report the CI computed with their official code base.
Bootstrapping:
B.9
Performance creative tasks
Table we show detailed results for different models on creative tasks.
B.10 Training BLEUBERI using different synthetic references
Obtaining the additional synthetic references: See Table 11 for detailed breakdown of costs
associated with collecting additional synthetic references for our 50K data pool. In total, collecting them cost around $352.48 USD.
Training results: 2.2 we measure BLEU's agreement with human judgments in single- reference setup, varying the choice of reference model. This raises natural question: we use outputs from these reference models for BLEUBERI training, does higher agreement with human preferences predict better training outcomes? To explore this, we train Qwen2.5-7B separately in a single- reference setup using synthetic references from each five models: Claude-3.7-Sonnee Gemini-2.5. Pro, 04-mini, Deepseek-V33 V3, and Llama-3. 1.1aa...--BB.. 1-8BB-snssttte We observe Pearson correlation of 0.34 between each reference model's human agreement score and the performance of the resulting trained model, suggesting moderately strong positive relationship. Among these, Claude and o4-mini
22

---

Table 9: WB creativity scores across base models and variants.
Base model
Variant
WB creativity
Base
51.3
SFT
60.4
Qwen2.5-7B
GRPO-RM
67.2
GRPO-BLEU
66.7
Instruct
74.4
Base
47.2
SFT
54.1
Qwen2,5-3B
GRPO-RM
61.5
GRPO-BLEU
58.4
Instruct
66.5
Base
61.9
SFT
66.8
Llama-3,1 3.1-8B (SFT init.)
GRPO-RI RM
63.3
GRPO-BLEU
66.4
Instruct
76.1
Table 10: Reference ablation on Qwen2.5-7B. The 5-reference setup uses Tulu3, o4-mini, Claude,
Deepseek, and Gemini references.
Base model
Variant GRPO-RM
MT--eech
ArenaHard
ArenaHard v2
WildBench
Average
Qwen2.5-7B
72.6
28.5
11.8
63.7
44.1
Qwen2.5-7B
BLEUBERI
73.0
30,9
13,3
64.0
45.3
Qwen2.5-7B
BLEUBERI
71.9
28.2
11.9
64.9
44.2
Qwen2.5-7B
BLEUBERI-o-min
72.5
25.1
11.6
62.6
42.9
Qwen2,5-7B
LLEUBERI-claude
70.4
24.8
10.7
63.8
42.4
Qwen2.5-7B
BLEUBERR-Ilan
68.7
23.9
11.9
63.2
41.9
Qwen2.5-7B
BLEUBERI-dcepsce decpssck
19.1
64.7
41.2
Qwen2.5-7B
OLEUBERI-geriin
60.3
61.8
references yield the two best-performing models that are on par with GRPO-RM. In 10 we show detailed results on Qwen2.5-7B, trained on 1K hardest data selected by BLEU. The 5-reference setup uses references from Tulu3, o4-mini, Claude, Deepseek, and Gemini. Interestingly, the model trained on Tulu3 references performs the best-even surpassing the model trained with 5-reference setup. As noted in $3.2 half of the Tulu3 references are generated by powerful LLMs like ChatGPT, and half are drawn from existing datasets with suman-writted responses. While the presence of human annotations likely contributes to its strong performance, the success of models trained purely on synthetic data (e.g.. using Claude or o4-mini) indicates that synthetic references alone can also be highly effective.
Table 11: Comparison model configurations, costs, and estimated runtimes over 50k prompts.
Llama-3. 3.1-8B--BBB.... 8--8--nnstud is not included here because it is not an API model.
Model Name
Arena Rank (SC)
Arg. Input Tokens
Avg. Output Tokens
Cost (8)
Time Estimate
gomini-2.5-pp Prp-----------2222
80
670
0.00
45h
claude-3-souncee 19
11
300
237.00
15h
despsesk-chot-v-.ok
o4-mini-2025-04-1
N/A
400
92.40
10b
B.11 Training with other types of rewards
Table 12 we show training results on Qwen2.5-7B using rewards other than BLBU (on hardest data). "BRF1" refer to the BLEU-ROUGE-L harmonic mean, while "BLEU+RM" refers to the combined metric of BLEU and RM-8B, both evaluated in 2.2] All these metrics demonstrate similar performance the LLEUBERR--raing model.
23

---

Table 12: Results on models trained with different types of rewards.
Base model
Variant
MT-Beechh
ArenaHard
ArenaHard
WildBench
Average
Qwen2.5-77
BLEUBERI-Tubd
73.0
30.9
13.3
64.0
45,3
Qwen2.5-7B
GRPO-BLED B0---BLEU+MMMM Tulu3
74.0
26.4
11.3
63.9
43.9
Qwen2.5-7B
SERO-BERTSCORE
73.1
29.6
10.0
64.1
44.2
Qwen2.5-7B
GRPO-BRFI-Sre
68.1
24.0
11.7
62.5
41.6
Table 13: Impact of training with reasoning (trained IK hardest data). We observe a drop in performance.
Base model
Variant
MT-Bench
ArenaHard v1
ArenaHard v2
WildBench
Average
Owen2.5-7B Qwen2.5-7B
BLEUBERI
73.0
30.9
13.3
64.0
BLEUBERI (reason)
68.0
24.0
2.7
59.6
B.12 Training BLEUBERI with reasoning
On aassning-intennivv tasks like math, prior work has found that directly running GRPO to induce reasoning can be very effective, even without any supervision on the reasoning chain produced. Will enforcing reasoning help in setting of general instruction following? To explore this, modify the training setup to encourage thain-of-thouur
(CoT) reasoning. Specifically, we introduce a system prompt and format reward that enforce the of <think> and <answer> tokens. We also increase max generation length from 512 to 1024 21 and compute BLEU scores on the final answers during training. Under this configuration, enforced reasoning. See (Table 13) This is consistent with Sprague et al. (54), who find that while CoT reasoning improves performance on math and symbolic tasks, has limited or even detrimental effects on others.
observe performance decline compared training without
C Qualitative analysis
C.1 Qualitative statistics and example
To investigate the characteristics of model responses, we analyze qualitative statistics across all four benchmarks, as shown in Table 14 To compute the repetition rate, we split each response into lowercase words and calculate how often 1- to 4-word phrases (n-grams) are repeated. We then compute the repetition rate for each n-gram size and take their average to quantify redundancy the To prevent out-of ofmooy errors, we also reduce batch size from 32 to 16.
NiA Bodice sooking brrinstoringg classfiicaiion codinggsoouuaan cuts @docammnt.co Butting infirmmaiaanooiin muth other planning B reasoring @ roteslaing Suummr traralation @witing
007R
5350
8893
Figure 8: Distribution of task types in the Tulu3 50K data pool, labeled by Llama-3.1-88 8---IBIItuce

---

Table 14: Qualitative statisties for model outputs across all four benchmarks.
Base model
Variant
Avg. tokens
Repetition (%)
Refusal (%)
Markdown usage (%)
SFT
947.3
20.1
3.4
48.3
Qwen2.5-7B
GRPO-RM
554.1
13.2
4.3
90.8
BLEUBERI
686.1
15.5
2.3
71.2
SIF
1259.8
22.6
1.5
57.8
Qwen2.5-3B
GRPO-RM
701.4
15.9
3.7
84.9
BLEUBERI
922.4
19.5
0.6
70.9
SFT
675,4
17.4
3.3
42.4
Llama-3.1-8B
GRPO-RM
763.8
2.4
90.9
BLEUBERI
577.5
2.5
62.1
text. To compute the refusal rate, we check whether the response contains common refusal phrases such "I'm sorry, but" or "As AL..". For Markdown usage, use regular expressions to detect the presence of Markdown syntax elements such as bold, italics, headers, or code blocks. The usage rate is defined the proportion of responses that include at least one such element.
Example responses: Figure 9 shows example responses from three models: Qwen2.5-7B base, GRPO-RM, and BLEUBERI. While the base model generates an irrelevant response the prompt-inctuding a code snippet mixed with bottan-both GRPO-RM and BLEUBERI/ follow the instruction appropriately.
Q: Help me construct a catchy, yet scientifically accurate, headline for an article on the latest discovery in renewable blo-energy. while carefully handling the ethical dilemmas surrounding bio-energy
Propose options.
Base
URPO-RM
BLEUBERI
You have data on tiple types fruit and their coarts two beokeets recorded a pandes Pottarrme:
Surel Here are four hoadline optinn that balance calchiness, scientfic acsunicy
Suro, can help you with that Hore are four potendal headline aim calshy surounding
consklanslans
addnowedging
**G Revoluton: Ethical BieEnne Broaldhrough Future Ststarstiilt""
bib-anarg::
"pytoon
pandas
This headline emphosibss progress, susta natillty, and ethical comaidaratons ena catchy way.
"Blo Energy Breakthrouhh Ethical Consiteaatins
Mout Scarthii Advancaments Sustanable Fuet "New Down Herreanee Blo- Boo-Enery Navigating Greener
data pd DataFramee
bbaset: ('A.. BB, W. B W, Bi; W. 18r),
DDsoding
Energy:A Bio Energy Source a
Ethical
thal:: (apple, "onnnee" lapriw, lonnnee.
"bonanal; orange:
happeet
high the renowable fature benefts, while being mindful of nature blo-energy sources.
scientiic
Dipcoveries llummiate Future Green Power 4. "The Etical Quest for Renewazle Bio Energy.A Scientiic Discovery Could Revdutionie Our Appreach"
16,
""Ehically Harverted The New Bio Energy That's Good the
People
You're noked to ge nerate a now DetaFrame that shows the total count each you soph
sourcing bio-energy thes voet
These headiiee
filld Intersection
baskets
environment human
amphasieng the balence between asheensing tech ogy and addressing the ethical concems assccialed with use.
4. "" innovvaaivv Bio-Energy Sclentifi Advancement Meets Etrical
20)
Standards de for Cleaner Worde
This headline combines ecientif advence ment with ehicca
You can och ooh this using the pivot table" funstion par surstee, speciying apgregation 'sum (omitted below)
stancards, ds, promoting a cleaner, more
Each headiines aims informaive,
respectul to the otical consideration sumounninn Mo-energy.
Figure 9: Outputs from Qwen base, GRPO-RM, and BLEUBERI. While GRPO-RM and BLEU- BERIgenerate responses that follow the instruction well, Qwen produces an irrelevant response
Please see Table 15 for qualitative statistics for model outputs.
C.2 Emergence of affirmative openers in GRPO-trained Qwen models
We visualize the usage of affirmative openers in trained models in Figure 10
C.3 Evaluation on factuality
We evaluate the factuality of model responses using the VERISCORE framework (33. which consists of two steps: (1) extracting verifiable claims from generated responses, and (2) verifying each claim
25

---

Table 15: Qualitative statisties for model outputs by benchmark. Each cell shows "Avg. Tokens Refusal Rate (%) / Markdown Usage (%) / N-gram Repetition Rate (%))"
Model
Variant
MTT-Bench
ArenalHadd
ArenaHard
WildBench
237.6/13/338//11
747.0/ 1.277//
1242.0/29/4555 622.9744/92.7 14.3
883.1/5.1 5/ 83.66 12.8 703.1 65.20//65.//////
Qwen7B
RM
336.4/0.0/ 92.5
492.7/0.8/ 002//08/986 98.2 12.9
BLEU
319.9/2.5/51.3111.
619.6/ 1.0/93/////// 14.9
746.4/2.3 06../223/667//// 15.8
SET
347,5/0.0/435 16.2
1002.1 /0.8 85.4/ 19.9
1587.3/13//33 53.1 /4.66
1217:2/2.1/42 48.8/1//////
Qwen3B
86.3
/9.69//.///.6//////
771./4.1185.9/
706.2/5.1 /7.9/ 16.7
BLEU
288.1 60.0//0.0//// 12.1
746.0/0./ 0..00/08/9///// 17.6
1176.6/04/67.// /7.1
871.90.7/64./ /0.0
/..3/13/////
5798/226634
785.6/1.1141.7/
673.1/4.9/342
Llama88
RM
283.7 81.3 11.4
14.3
988.8/2.0/ 88.5/ 18.4
15.4
494.3/14/
637./00//573
Average starting phrase usage by model
Model: Uama-3.1-88
Model: Qwen2.5-38
Model: Qwen2.5-78
Training variant
70
Bese
Average percentage
GRPO-HH RM
40
26.6
19.5
26.7
70.6
51.6
32.8
26.8
10
1.8
1.6
1.5
'Certainly!"
"Sure!"
"Certainly!" "Sure!"
"Certainly!"
"Sure!"
TTo
Starting phrase
Starting phrase
Starting phrase
Figure 10: Frequency of starting phrases for each model and training method. For Qwen, BLEUBERI models begin with "Certainly!" lot, and GRPO-RM models begin with "Sure!" very often. Numbers are averaged over responses for the four benchmarks.
using retrieved evidence from the web via the Google Search via the Serper API we use VeriScore's 's fine-tuned claim extracto and verifie
For this purpose,
The original VERISCORE paper evaluates on 1l datasets, but to reduce costs incurred by using the Serper API for Google Search, we focus three datasets. We apply VeriScore on the full 200 FRESHQA and sampled 100 examples from each of ASKHISTORIANS and LONGFACT which require long-form answers.
We set the claim extractor's s maximum input/output length to 1024/1000 tokens, and the verifier's to 2048/500 tokens. Each claim is verified using the top 10 retrieved search results.
To quantify factuality, we adopt the supported claims) and recall (whether sufficient number of supported claims are included). The recall cutoff K defined as the median number of extracted claims across responses each dataset, consistent with the original VeriScore setup. We three labels: supported, contradicted, and inconclusive. Claims that are too general or lack verifiable connections are labeled as inconclusive. We follow the labeling guidelines of VERISCORE, which treat both contradicted and inconclusive claims as unsupported. Let M be a language model and a set of prompts. For each x E X, let Mr be the model's response and C' the set of extracted claims. Let support(c, Ez) {0. indicate whether claim is supported by retrieved evidence E,. Define:
metric, which balances precision (the proportion of
S(r) support(c, Ec), P(r) S(r) |C|
S(r) K
R(r) min
SEC
htttp://serper..r dev/
httpp:: https://huggingfacee https////a.///////ssaaaaaaaaaaa
26

---

Table 16:
Factual consistency results for trained Qwen2.5-7B models across three QA datasets
evaluated using VERISCORE [53]
The K values used for each dataset follow the original paper.
Dataset (K)
Variant Avg. tokens
Total claims
F1@ K
Precision
Recall
SFT
87.3
836
45.0
57.2
44.3
FRESHQA (4)
GRPO-RM
234.2
1838
60.7
56.0
73.6
BLEUBERI
124.8
1330
60.5
63.9
64.0
SFT
505.1
4240
81.6
82.6
85.0
LONGFACT (32)
GRPO-RM
478.0
4364
87.0
81.6
93.8
BLEUBERI
583.1
4964
89.0
84.9
94.2
SFT
441.2
2882
51.4
48.4
ASKHISTORIANS (21)
GRPO-RM
437.4
2962
50.5
45.0
61.2
BLEUBERI
454.1
3014
54.2
49.4
64.8
2P(r)R())
if if >0
FI@K(r)
+(r)+M() 0
otherwise
VERISCORE(M)
1
FI@K(Mæ) aEX
Σ
XX|
Table 16 presents detailed VERISCORE results, and Table 17 illustrates examples of extracted verifiable claims and their verification using VeriScore.
Table Example of extracted claims using VeriScore
Question
How old is the world's oldest verified living person?
SFT
Response
The world's oldest verified living person Kane Tanaka, who was born on March 5, 1903. As of my last update in 2023, she is 120 years old, Kane Tanaka Japanese woman who has been living the city Fukuoka, Japan. She is known for her longevity and has been subject of interest for researchers studying the factors that contribute to long life.
1. Kane Tanaka is the world's oldest verified living person. /
2. Kane Tanaka was born on March 5. 1903. X
3. Kane Tanaka was 120 years old in 2023. X
Claim List
4. Kane Tanaka is Japanese woman. /
5. Kane Tanaka lives in Fukuoka, Japan. /
Kane Tanaka is known for her longevity.
7. Kane Tanaka has been studied by longevity researchers. X
GRPO-RM
27

---

Response
To find the age of the world's oldest verified living person, we need to consider the most recent data available. As my last update in October 2023, the world's s oldest verified living person is Kane Tanaka from Japan. She was born on March 21, 1903, and as October 2023, she was 120 years old. Here's a step-by-step breakdown of how arrived at this answer:
1. **Identify the Oldest Person** The Guinness World Records and other reliable sources maintain records of the oldest living people. As of my last update, Kane Tanaka is recognized as the oldest verified living person.
2. **Determine Birth Date** Kane Tanaka was born March 21, 1903.
3. Calcullte Age*: To find her age in October 2023, we subtract her birth year from the current year (2023) and adjust for her birthday not having occurred yet in the current year.
Age = 2023 - 1903 = 120 years
4. **Verify Status **: It's important to note that this information based on the most recent data available up to October 2023. If you are looking for the most current information, would be best to check the latest Guinness World Records or official health records for updates.
Therefore, as of October 2023, the world's oldest verified living person, Kane Tanaka, is 120 years old.
Claim List
As of October 2023, Kane Tanaka is the world's oldest verified living person. X
2. Kane Tanaka is from Japan.
3. Kane Tanaka was born on March 21, 1903. X
4. As of October 2023, Kane Tanaka was 120 years old. X
5. The Guinness World Records maintains records of the oldest living people.
/
6. Other reliable sources maintain records of the oldest living people. /
7. As of October 2023, Kane Tanaka recognized as the oldest verified living
person. /
8. To find the age of the world's oldest verified living person in October 2023, subtracts their birth year from the current year. X
9. Kane Tanaka was born in 1903. X
10. The information about the world's oldest verified living person is based the most recent data available up to October 2023. X
11. Kane Tanaka was the world's oldest verified living person as of October
2023. /
12. Kane Tanaka was 120 years old as of October 2023. X
BLEUBERI
Response
The world's oldest verified living person is Kane Tanaka, a Japanese woman who was born on July 2, 1903, As of my last update in October 2023, she is still alive and has been recognized by Guinness World Records as the oldest living person. However, it's important to note that the status of the world's oldest person can change as individuals pass away and new centenarians reach the age of 110.
Kane Tanaka has lived through significant historical events, including World War I, World War II, and the Great Depression. She has also seen numerous technological advancements and social changes throughout her long life. It's worth mentioning that there are other individuals who have claimed to the world's oldest person, but their claims have not been verified by Guinness World Records or other official organizations. The age of the world's oldest person is subject of ongoing interest and debate, and it's possible that new records will be set in the future.
Claim List
1. Kane Tanaka is the world's ' oldest verified living person. /
2. Kane Tanaka was born on July 2, 1903. X
3. Kane Tanaka is a Japanese woman. /
28

---

Preference type
GRRO RM preterred GIUO
70
67.5
60
Percentage preferences
52.5
50
47.5
20
10
Annotator
Annotstor
Annotator
Figure 11: Human preference results. For each annotator, the bar on the right represents the soft preference rate for BLEUBERT. Based on these evaluations, BLEUBERI outputs are often on par with GRPO-RM outputs.
4. Kane Tanaka was recognized by Guinness World Records as the oldest living person as of October 2023. /
5.
Kane Tanaka was still alive as of October 2023. X
6. The status of the world's oldest person can change as individuals pass away.
/
7. The status of the world's oldest person can change as new centenarians
reach the age of 110. /
8. Kane Tanaka lived through World War /
9. Kane Tanaka lived through World War II. /
10. Kane Tanaka lived through the Great Depression. /
11. Kane Tanaka has lived through significant historical events. /
12. Kane Tanaka has lived through World War
13. Kane Tanaka has lived through World War II. /
14. Kane Tanaka has lived through the Great Depression.
15. Kane Tanaka has numerous technological advancements throughout
her life.
16. Kane Tanaka has social changes throughout her life. /
17. There have been individuals who have claimed to be the world's oldest person. /
18. The claims of individuals to be the world's oldest person have not been verified by Guinness World Records. /
19. The claims of individuals to be the world's oldest person have not been verified by other official organizations. /
C.4 Human evaluation details
We show detailed distribution of each annotator's labels in Figure 11
29

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See the abstract and $ We clearly outline our contributions without exagger- ating their significance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [No]
30

---

Justification:
We do not have theoretical proofs.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See 33.1 and 3.2
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
31

---

Answer: [Yes]
Justification:
We submit our data and code to facilitate reproduction of our results.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https::/nips.co
stw.oodeeeeuumiiiiioooooooiy..e for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https / //nips. chips.cccpubb..... ceppubllc/guu wwtpbliicggii/eeeeeeeeeee dee//CoeeSubii ssionoolicyl for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: See 3
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We do not evaluations multiple times due to resource constraints, as these are LLM--bsee evaluations.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
32

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: Yes, comply with these requirements.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
This work does not have such societal impact.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
33

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [No]
Justification:
This paper has no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Yes, provide proper citations.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, |ppper swi thcode. com/ cona/datsets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
34

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed documentation on these assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details of the dataset/codeemmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This work does not involve any hired human annotators. Annotations were done by co-authors.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [Yes]
Justification: The project was reviewed by an IRB board and deemed exempt from federal regulations that govern human subject research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
35

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [No]
Justification: This work does not use LLMs in the described ways.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer our LLM policy httpss///eeerrrssscc//oforeeeees for what should or should not be described.
36


Reviews:
[{"review_id": "M47BKtmhmC", "reviewer": "Reviewer_R3fD", "summary": "This paper introduces BLEUBERI, an approach for aligning Large Language Models (LLMs) with human preferences by directly using BLEU, a simple string-matching metric, as a reward function in reinforcement learning (RL). Traditionally, LLM alignment relies on costly reward models that require extensive human-labelled preference data. The authors surprisingly demonstrate that BLEU effectively models human preferences for general instruction-following tasks, even rivalling large reward models in agreement. Specifically, BLEU with five synthetic references achieved a 74.2% agreement with human preferences, comparable to a 27B-parameter reward model's 75.6%.", "questions": "Please refer to weaknesses above. \n\n1. How would this work be adapted to dealing with mathematical reasoning tasks? Any initial experiments or thoughts?\n2. How does variability in reference quality impact the effectiveness of the training process\n3. Can BLEU be prone to more subtle reward hacking?\n4. Can this metric work well in multi-step reasoning tasks? Why was there a performance decline with enforced CoT reasoning? Could you elaborate please?", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. BLEU surprisingly matches strong reward models in modelling human preferences for general instruction following tasks\n2. Cost effectiveness\n3. Experimental evaluation demonstrating enhanced factuality and positive human evaluation\n\nWeaknesses\n\n1. Reliance on reference quality is the biggest concern\n2. Absence of statistical significance results \n3. Limited experiments on exploring effects of scaling model size, data volume or extensive training time\n4. Known issues in mathematical reasoning tasks", "was_revised": true, "final_justification": "The \"reference\" is the part that I am not convinced about even after the final response by the authors. If you use existing LLMs or existing datasets, they have already put in a ton of effort/cost to generate that reference. I had requested for evaluation with not so great references to see if the performance can be attributed to the contribution of the paper. However, authors never answered or did experiments with regards to that.  So, I am sticking to my earlier scores.", "rebuttal": "We appreciate the reviewer’s recognition of our work, including BLEUBERI’s competitive performance with GRPO-RM, BLEUBERI’s cost efficiency, and our strong results in both factuality and human evaluations. Below, we respond to each of the weaknesses and questions in detail.\n\n## Weaknesses\n\n***`1. Reliance on reference quality is the biggest concern`***  \nWe acknowledge this limitation, but **obtaining high-quality references is straightforward**. Such data can come from two sources:\n\n* **Synthetic outputs from frontier LLMs**: Generating synthetic data has become common in LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These data can be highly effective in practice.  \n* **Ground-truth responses from existing datasets**: Many publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023).\n\nIn contrast, **training reward models demands extensive human-labeled preference data, which is costly and time-consuming**. Lightweight reference-based metrics like BLEUBERI offer a more accessible and efficient path to alignment.\n\nReferences:\n\n* Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024\\)  \n* Qwen2.5 Technical Report (Qwen Team, 2024\\)  \n* Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023\\)  \n* Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023\\)\n\n***`2. Absence of statistical significance results`***  \nWe thank the reviewer for raising this point. In response, **we reran benchmark evaluations with bootstrapping and report results below**. Overall, we find that:\n\n* Both **BLEUBERI and GRPO-RM show consistent improvements over the base model across all three model types**, as suggested by minimal or no overlap between their confidence intervals and those of the base model.  \n* Between BLEUBERI and GRPO-RM, the confidence intervals almost always overlap, suggesting that **BLEUBERI performs on par with GRPO-RM overall**.\n\nWe will add these results into the next version of the manuscript.\n\n**Benchmark results with bootstrapping:**  \n| Base model       | Variant   | MT-Bench | 95% CI         | ArenaHard v1 | 95% CI         | ArenaHard v2 | 95% CI         | WildBench | 95% CI         |  \n|------------------|-----------|----------|----------------|--------------|----------------|--------------|----------------|-----------|----------------|  \n| Qwen2.5-7B       | Base      | 63.5     | (-4.0, \\+4.1)   | 16.2         | (-1.6, 1.6)     | 5.6          | (-0.8, \\+0.7)    | 51.8      | (-1.6, \\+1.6)   |  \n|                  | SFT       | 67.5     | (-4.3, \\+3.7)   | 22.1         | (-2.1, 2.2)     | 9.9          | (-1.0, \\+1.1)    | 60.5      | (-1.3, \\+1.3)   |  \n|                  | GRPO-RM   | 76.9     | (-3.8, \\+3.2)   | 29.8         | (-2.2, 2.2)     | 12.2         | (-1.2, \\+1.3)    | 64.8      | (-1.1, \\+1.3)   |  \n|                  | BLEUBERI  | 70.8     | (-3.8, \\+3.6)   | 29.3         | (-1.8, 2.7)     | 12.9         | (-1.4, \\+1.2)    | 65.4      | (-1.3, \\+1.3)   |  \n|                  | Instruct  | 78.8     | (-3.5, \\+3.3)   | 37.9         | (-2.8, 2.5)     | 16.8         | (-1.1, \\+1.5)    | 71.0      | (-1.2, \\+1.1)   |  \n| Qwen2.5-3B       | Base      | 61.1     | (-3.9, \\+3.7)   | 7.0          | (-1.0, 1.0)     | 3.1          | (-0.6, \\+1.0)    | 49.0      | (-1.3, \\+1.5)   |  \n|                  | SFT       | 59.6     | (-4.2, \\+4.2)   | 9.8          | (-1.1, 1.6)     | 4.0          | (-0.6, \\+0.8)    | 55.6      | (-1.4, \\+1.3)   |  \n|                  | GRPO-RM   | 67.8     | (-4.2, \\+4.0)   | 12.8         | (-1.3, 1.6)     | 5.1          | (-0.6, \\+0.6)    | 59.2      | (-1.2, \\+1.3)   |  \n|                  | BLEUBERI  | 64.6     | (-4.1, \\+3.9)   | 11.0         | (-1.3, 1.3)     | 3.7          | (-0.5, \\+0.6)    | 56.1      | (-1.2, \\+1.3)   |  \n|                  | Instruct  | 70.0     | (-4.2, \\+3.8)   | 18.9         | (-2.0, 1.9)     | 6.7          | (-0.8, \\+0.9)    | 63.0      | (-1.2, \\+1.3)   |  \n| Llama-3.1-8B     | Base      | 52.9     | (-4.3, \\+4.2)   | 6.8          | (-1.0, 1.2)     | 2.1          | (-0.5, \\+0.6)    | 55.0      | (-1.3, \\+1.3)   |  \n| (SFT init.)      | SFT       | 56.7     | (-4.5, \\+4.5)   | 12.6         | (-1.6, 1.7)     | 3.0          | (-0.6, \\+1.0)    | 60.3      | (-1.1, \\+1.3)   |  \n|                  | GRPO-RM   | 57.2     | (-4.4, \\+4.3)   | 9.7          | (-1.2, 1.4)     | 1.6          | (-0.4, \\+0.6)    | 57.8      | (-1.2, \\+1.4)   |  \n|                  | BLEUBERI  | 56.9     | (-4.4, \\+4.3)   | 10.1         | (-1.4, 1.5)     | 2.3          | (-0.4, \\+0.6)    | 59.5      | (-1.2, \\+1.1)   |  \n|                  | Instruct  | 65.8     | (-3.8, \\+3.6)   | 24.9         | (-2.4, 2.8)     | 5.8          | (-0.8, \\+1.1)    | 64.2      | (-1.2, \\+1.1)   |\n\nArena-Hard v1/v2 already include bootstrapping in their official code: they draw 100 resamples and re-fit a Bradley-Terry model for each, which is computationally intensive, hence the cap at 100\\. We reran the scripts and report the median point estimates and 95% confidence intervals (CIs). For MT-Bench and WildBench, we implemented a similar procedure with 1000 full-size prompt resamples per model, reporting the median score and two-sided 95% CI.\n\n***`3. Limited experiments on exploring effects of scaling model size, data volume or extensive training time`***  \nIn our preliminary experiments, we trained for multiple epochs but observed no significant performance gains for either BLEUBERI or GRPO-RM. One possible explanation is that larger models might benefit more from longer training. Regarding data volume, we note that our setup aligns with that of the Qwen3 pipeline (Qwen Team, 2025), which trains exclusively on 4,000 challenging samples, as noted on Line 204\\.\n\nOverall, we agree that more scaling experiments can provide helpful insights. However, due to limited compute resources, we leave such explorations to future work. We would like to highlight again that all training experiments (exploratory and final) cost about 1357 GPU hours (around **$4750 USD**), which represents a substantial computational cost for an academic lab.\n\nReferences:\n\n* Qwen3 Technical Report (Qwen Team, 2025\\)\n\n***`4. Known issues in mathematical reasoning tasks`***  \nWe assume the reviewer is referring to the results shown in **Figure 7**, which shows that on the Math/Reasoning domain, the gap between BLEU’s and the two reward models’ human agreement rates is large. We would like to reiterate that **the primary focus of this work is on general-domain instruction following**. For structured domains like math, it is straightforward to develop reward functions that checks the correctness of the final answer. If model outputs consisted solely of final answers, BLEU would essentially function as a correctness-based reward. However, many Math/Reasoning examples in the Chatbot Arena dataset include intermediate reasoning steps, which reduce BLEU’s effectiveness as a proxy for correctness. In our experiments, we chose not to optimize for any specific domain.\n\n## Questions\n\n***`1. How would this work be adapted to dealing with mathematical reasoning tasks? Any initial experiments or thoughts?`***  \nBuilding on the discussion of Weakness 4, one potential way to improve both general-domain and math performance is to combine BLEU and a correctness reward into a single objective function, where BLEU could be used to assess the quality of reasoning traces. We plan to explore this approach in a future version of the manuscript.\n\n***`2. How does variability in reference quality impact the effectiveness of the training process`***  \nWe provide training results with different references in **Table 9** in the Appendix. These results generally align with our findings from Section 2.2 (Line 124), suggesting that stronger reference models will lead to better results, which is intuitive. As discussed under Weakness 1, obtaining high-quality references is straightforward.\n\n***`3. Can BLEU be prone to more subtle reward hacking?`***  \nWe interpret the reviewer’s comment as referring to subtle, hard-to-detect artifacts. As noted in **Section 4.1 (Lines 280–294)**, GRPO-RM models often begin with “Sure\\!”, while BLEUBERI models favor “Certainly\\!”—though this trait is not unique to BLEUBERI. Importantly, human evaluations (Section 4.3) rate BLEUBERI and GRPO-RM responses as equally preferable, and benchmark results (Table 1\\) and factuality assessments (Section 4.2) show that **any such artifacts do not meaningfully impact perceived quality**.\n\n***`4. Can this metric work well in multi-step reasoning tasks? Why was there a performance decline with enforced CoT reasoning? Could you elaborate please?`***  \n**Prior work shows that explicit reasoning does not always help**, especially when instruction adherence is the main goal. In our experiments, enforcing CoT reasoning reduced performance, consistent with Li et al. (2025), who found that CoT can distract from constraint-relevant tokens and introduce errors. Sprague et al. (2024) similarly observed that **CoT primarily benefits math and symbolic tasks, with little or no gain and sometimes harm on general instruction-following**. Our BLEUBERI results align with these findings, supporting the idea that reasoning should be applied selectively.\n\nIt is possible that larger models or longer generation lengths may allow more effective use of reasoning in general instruction-following tasks. We plan to explore this in a future version of the manuscript.\n\nReferences:\n\n* When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs (Li et al., 2025\\)  \n* To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoning (Sprague et al., 2024\\)"}, {"review_id": "XPGsS3ECHE", "reviewer": "Reviewer_dqb5", "summary": "This paper investigates the feasibility of using the BLEU metric as a reward for the GRPO algorithm in instruction-following alignment. As a pivotal study on metric effectiveness, the authors show that BLEU’s agreement with human judgment improves as the number of references increases or when it is combined with other metrics. In the main experiments, BLEUBERI performs comparably to GRPO-RM across most settings. Notably, BLEUBERI demonstrates potential advantages in improving factuality while maintaining creativity.", "questions": "I will be insterested to see the results on larger base model sizes (32B) with the same reward models comparison (RM-8B vs BLEUberi). If the results are positive for BLEUberi, this would be a strong demonstration for light weight reward function.\n\nA more direct quantification for GRPO algorithm speed-up between RM-8B and BLEU might be presented.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n1. The paper revealed that Reward Models are costly for alignment GRPO and discovered that BLEU is surprisingly effective reward for instruction following. This topic has rarely discussed before and provides interesting findings to the community.\n\n2. The paper adopted synthethic data from tulu3, which might potentially usefule to researchers. Also, the paper has done an comparison of metric's effectivity, which produces insight to the community.\n\nWeaknesses\n\n\n1. Limited Analysis of Reward Hacking Risks. While the factual presentation is appreciated, the discussion of potential reward hacking remains limited. Upon reviewing the Appendix, I observed that BLEU used as a reward exposes models to two notable risks:\n\nVerbosity Bias: As shown in Table 13, BLEUBERI causes a marked increase in generated output length, indicating a verbosity issue.\n\nRepetition Bias: Table 13 also illustrates that BLEUBERI leads to higher repetition in generated text, a common failure mode for text generation models.\n\nThese factors raise significant concerns about the effectiveness of BLEU as a reward metric for instruction following. A more nuanced analysis—including mechanisms to detect and mitigate these biases—would improve the discussion.\n\n2. Insufficient Discussion of Alternative Rule-Based/Verifiable Metrics. More thorough analysis on rule-based/verifiable metrics. as presented in section2, Appendix B.10, there should be chances to in-depth discussion. It is hard for me to recognize why ROUGE-L, BERTScore, BLEUrt are not competitive to BLEU. Without such, it is hard to assess BLEU’s relative contribution as a low-cost reward. Also, the emsemble of rewards should also be considered. \n\n3. Concerns with Human Evaluation Reporting (line 320): The soft preference rate is misleading. One cannot calculate the tie rate into the 'for' rate. The same problem exists in Appendix Figure 11. Besides, Cohen Kappa of 0.36 for triple selection indicates the observed two annotators agreement is around 45%, which does not demonstrate strong reliability for the human evaluation results. Emphasizing these limitations and contextualizing the implications for the overall conclusions would be prudent.", "was_revised": true, "final_justification": "The authors have partially addressed my concerns, which has led me to revise my rating from 'Borderline reject' to 'Borderline accept'", "rebuttal": "We appreciate the reviewer’s recognition of our work, including BLEUBERI as a cost-effective alternative to GRPO-RM for alignment, the novelty and interestingness of our findings, the value of our collected synthetic data, and the informative comparison of different metrics. Below, we respond to each of the weaknesses and questions in detail.\n\n## Weaknesses\n| **Base Model** | **Variant** | **Avg. Tokens** | **Repetition (%)** |\n| -------------- | ----------- | --------------- | ------------------ |\n| Qwen2.5-7B     | SFT         | 947.3           | 20.1               |\n|                | GRPO-RM     | 554.1           | 13.2               |\n|                | BLEUBERI    | 686.1           | 15.5               |\n| Qwen2.5-3B     | SFT         | 1259.8          | 22.6               |\n|                | GRPO-RM     | 701.4           | 15.9               |\n|                | BLEUBERI    | 922.4           | 19.5               |\n| LLaMA-3.1-8B   | SFT         | 675.4           | 17.4               |\n|                | GRPO-RM     | 763.8           | 16.0               |\n|                | BLEUBERI    | 577.5           | 18.9               |\n\nThese numbers suggest that **BLEUBERI is not consistently more verbose than GRPO-RM, and overall differences between the two are relatively modest**.\n\nOn length bias:\n\n* We note that our benchmarks try to mitigate length bias by design. Both ArenaHard v1 and v2 apply style control, which control for factors like length and markdown usage. WildBench similarly has a length penalty mechanism, which converts small wins/losses into ties if one response is much longer than the other.\n\nOn repetition bias:\n\n* As shown in Table 13, BLEUBERI has lower repetition rates than SFT and rates that are generally close to—or only slightly higher than—those of GRPO-RM. Despite this, our human evaluation (Section 4.3) finds **BLEUBERI model outputs to be just as preferable as GRPO-RM’s**. This indicates that the minor verbosity and repetition seen in Table 13 do not substantially affect the perceived quality of responses.\n\n***`2. Insufficient Discussion of Alternative Rule-Based/Verifiable Metrics`***  \nWe currently discuss alternative metrics in Section 2.2 and Appendix B.10. We agree that further exploration of such metrics could provide valuable insights. **Our primary goal, however, was to demonstrate that even simple metrics like BLEU can be effective.** Given our resource constraints, we focused on BLEU in the training experiments due to its simplicity and surprisingly strong performance. To the best of our knowledge, **our work is the first to apply simple reference-based metrics to enhance general-domain instruction following**. We hope this serves as a foundation for future research on rule-based or verifiable reward functions.\n\n*`2.1. “It is hard for me to recognize why ROUGE-L, BERTScore, BLEUrt are not competitive to BLEU. Also, the ensemble of rewards should also be considered.”`*  \nROUGE-L and BLEU \\+ ROUGE-L ensemble **(new experiments)**\n\n* We ran additional experiments on the Chatbot Arena dataset with ROUGE-L. With stemming enabled and multiple reference responses, **ROUGE-L can achieve human agreement levels comparable to BLEU**. Moreover, **combining BLEU and ROUGE-L (with stemming) as an ensemble reward function by averaging also yields strong agreement results**. These findings will be included and discussed in the next version of the manuscript.\n\n| \\# references | ROUGE-L (no stemming) | ROUGE-L (with stemming) | BLEU & ROUGE-L mean |  \n|--------------|------------------------|--------------------------|----------------------|  \n| 1            | 68.5%                  | 70.3%                    | 73.0%                |  \n| 2            | 72.3%                  | 73.9%                    | 74.4%                |  \n| 3            | 72.7%                  | 74.1%                    | 75.5%                |  \n| 4            | 71.7%                  | 72.2%                    | 75.5%                |  \n| 5            | 72.2%                  | 72.9%                    | 75.0%                |\n\nBERTScore\n\n* To clarify, results in the current version of the manuscript indicate that **BERTScore *is* competitive with BLEU** in the Chatbot Arena analysis (Figure 1\\) and in training experiments (Table 11, Appendix B.10). However, due to computational constraints, we did not pursue extensive training with BERTScore.\n\nBLEURT\n\n* This current manuscript does not mention BLEURT. We leave the exploration of more reference-based metrics like this to future work.\n\n***`3. Concerns with Human Evaluation Reporting (line 320)`***  \nThank you for raising these concerns. Our reported soft preference rate was not intended to imply BLEUBERI’s superiority, but rather to reflect how often its outputs were **at least as good as** GRPO-RM’s (i.e., including ties and wins). This metric indicates general parity, not dominance. We acknowledge it could be misleading without context and will revise the manuscript to clearly distinguish it from strict preference counts.\n\nRegarding the Cohen’s Kappa of 0.34, we note that **our annotation task required subjective judgments on highly similar, high-quality responses to complex prompts from four benchmarks** (MT-Bench, ArenaHard v1/v2, WildBench). These often involved technical or domain-specific content, making clear preferences difficult. The frequent use of the “tie” option by one annotator further suggests the closeness in output quality. Prior studies have also observed that subjective evaluation tasks often yield lower inter-annotator agreement (Wang et al., 2024; Plank, 2022).\n\nOur human evaluation supports the key finding that **BLEUBERI-trained models can rival reward-model-trained counterparts in output quality.** We will revise the evaluation section to better highlight its limitations, clarify metric interpretation, and contextualize their relevance to our conclusions.\n\nReferences:\n\n* The “Problem” of Human Label Variation: On Ground Truth in Data, Modeling and Evaluation (Plank et al., 2022\\)  \n* HelpSteer2: Open-source dataset for training (Wang et al., 2024\\)\n\n## Questions\n\n***`1. Interest in seeing the results on larger base model sizes (32B)`***  \nWe understand that results on a much larger base model would offer valuable insights into the effectiveness of simple reference-based reward functions like BLEU. However, due to limited compute resources, we leave such explorations to future work. We would like to highlight again that all training experiments (exploratory and final) cost about 1357 GPU hours (around **$4750 USD**), which represents a substantial computational cost for an academic lab.\n\n***`2. More direct quantification of GRPO algorithm speedup between RM-8B and BLEU`***  \nFor two runs trained with BLEU and RM-8B as the reward, the average reward computation times over 250 steps (profiled by TRL) were:\n\n* RM-8B: \\~1.15s  \n* BLEU: \\~0.03s\n\nDespite BLEU being much faster, both runs took roughly the same wall-clock time (\\~2h45m). **In our setup, the main bottleneck was output generation per group**, which was slow due to using standard Hugging Face Transformers without fast inference backends like vLLM. As a result, BLEU’s speed advantage didn’t significantly impact end-to-end training time.\n\nHowever, in more optimized setups with faster generation or larger group sizes, BLEU’s efficiency could offer greater benefits. Not having to load a reward model should free up memory for larger batch sizes. In our experiments, we use identical hyperparameters for BLEUBERI and GRPO-RM for fairness. But in practice, **using lightweight metrics like BLEU could enable better scaling**."}, {"review_id": "6DGwn01ldU", "reviewer": "Reviewer_dhgB", "summary": "This paper presents BLEUBERI, which uses BLEU as a reward signal during GRPO instead of expensive reward models for alignment for instruction following. Despite that BLEU is just a simple n-gram matching metric, this paper demonstrates that it can be surprisingly effective in matching the performance of sophisticated reward models for instruction-following tasks.", "questions": "1. What would happen if experimented with ROUGE (more recall focused) instead of BLEU (more precision focused), or use both?\n\n2. how sensitive is it to use BLEU with different hyper-parameters, e.g., different n-gram length?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- BLEU is efficient to compute compared to calling expensive reward models, thus the empirical findings of this paper is valuable to the broader empirical community.\n- Despite its simplicity, BLEUBERI-trained models perform comparably to more expensive approaches relying on reword models across four challenging benchmarks (MT-Bench, ArenaHard v1/v2, WildBench) and three different base models (Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-3B).\n- The authors also demonstrate that the closer guidance based on BLEU signals leads to models that are more factually grounded than those trained with traditional reward models or supervised fine-tuning, as measured by VeriScore factuality evaluation.\n\nWeaknesses:\n- This approach is applicable only when high-quality references are available to compute BLEU against. \n- The strengths still outweigh the above weakness, given that collecting reference data can be often more cost-effective than calling the reward models, and one can also leverage already existing datasets with references.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer’s recognition of our work, including BLEU’s computational efficiency compared to reward models, the strong performance of BLEUBERI-trained models across four benchmarks and multiple base models, and the improved factual grounding achieved through BLEUBERI training. Below, we respond to each of the weaknesses and questions in detail.\n\n## Weaknesses\n\n***`1. This approach is applicable only when high-quality references are available to compute BLEU against.`***  \n***`2. The strengths still outweigh the above weakness, given that collecting reference data can be often more cost-effective than calling the reward models, and one can also leverage already existing datasets with references.`***  \nWe acknowledge this limitation, but as the reviewer also noted, **obtaining high-quality references is relatively straightforward**. Such data can come from two sources:\n\n* **Synthetic outputs from frontier LLMs**: Generating synthetic data has become a standard component of modern LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These synthetic outputs have proven to be highly effective in practice.  \n* **Ground-truth responses from existing datasets**: Numerous publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023), and Alpaca (Wang et al., 2022).\n\nIn contrast, **training reward models demands extensive human-labeled preference data, which is costly and time-consuming**. Reference-based approaches like BLEUBERI offer a more accessible and efficient path to alignment, demonstrating that even simple metrics like BLEU can serve as practical alternatives to reward models.\n\nReferences:\n\n* Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024\\)  \n* Qwen2.5 Technical Report (Qwen Team, 2024\\)  \n* Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023\\)  \n* Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023\\)  \n* Self-Instruct: Aligning LM with Self Generated Instructions (Wang et al., 2022\\)\n\n## Questions\n\n***`1. What would happen if experimented with ROUGE (more recall focused) instead of BLEU (more precision focused), or use both?`***  \nWe currently discuss alternative metrics like ROUGE-L in Section 2.2 and Appendix B.10. Below, we provide some **new experiment results** with ROUGE-L and a BLEU \\+ ROUGE-L ensemble metric on the Chatbot Arena analysis:\n\n| \\# references | ROUGE-L (no stemming) | ROUGE-L (with stemming) | BLEU & ROUGE-L mean |  \n|--------------|------------------------|--------------------------|----------------------|  \n| 1            | 68.5%                  | 70.3%                    | 73.0%                |  \n| 2            | 72.3%                  | 73.9%                    | 74.4%                |  \n| 3            | 72.7%                  | 74.1%                    | 75.5%                |  \n| 4            | 71.7%                  | 72.2%                    | 75.5%                |  \n| 5            | 72.2%                  | 72.9%                    | 75.0%                |\n\nFrom these new results, we find that **with stemming enabled** and **more references available**, ROUGE-L can achieve human agreement comparable with BLEU. Combining BLEU and ROUGE-L also yields strong agreement results. We will add and discuss these results in the next version.\n\n***`2. How sensitive is it to use BLEU with different hyper-parameters, e.g., different n-gram length?`***  \nWe present **new experiment results** below, which shows that BLEU performance is relatively stable across different maximum n-gram orders, as long as the order is greater than 1\\.\n\n| max ngram order | alignment |  \n|-----------------|-----------|  \n| 1               | 69.3%     |  \n| 2               | 71.5%     |  \n| 3               | 72.9%     |  \n| 4               | 72.4%     |  \n| 5               | 71.2%     |  \n| 6               | 71.5%     |\n\nFor the standard setup with max ngram order set to 4, we also experimented with both smoothed and unsmoothed BLEU variants and found that **using smoothing is crucial** for achieving high human agreement. With smoothing, the agreement is 72.4%. Without smoothing, the agreement drops to 54.7%. Smoothing prevents BLEU scores from collapsing to zero when higher-order n-gram matches are sparse, even in outputs that are meaningfully aligned. In all experiments in the paper, we apply smoothing (as noted in Footnote 2 in the paper). We will add and discuss these results in the next version."}, {"review_id": "UPtPjxSJWm", "reviewer": "Reviewer_wK8y", "summary": "This work presents a new post-training algorithm that uses the BLEU score on references as the reward. The BLEUBERI reward can be used in any RL optimization algorithms, including GRPO. The proposed method is evaluated on multiple tasks with two models (Qwen 2.5 and Llama 3.1). Models trained with BLEUBERI show similar or better performance compared to SFT or GRPO with normal RM rewards. This work also contains interesting analyses showing BLEUBERI leads to more grounded responses, and the method is also compatible with synthetic data or other reference-based metrics.", "questions": "1. BLEUBERI with synthetic data feels like some sort of distillation on a high level. Have you compared it with distillation methods empirically?\n2. What will happen if you do not filter out easy data when training with BLEUBERI? Are there some common reward hacking patterns?\n3. In the agreement study in Sec. 2, the best method is to use BLEU + RM. Have you tried similar approaches during training? Will it also lead to the best performance?\n4. How much markdown formatting do you have in your training data? Does that explain the trends in Sec. 4.1?\n5. I don't get the \"amplifies subtle biases\" part in line 291. Can you elaborate?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. Using BLEU as the reward during post-training is an original but natural idea. It's interesting to see that its performance is good and even surpasses SFT or RLHF with RM in some cases.\n2. This paper is well-written, and the analyses in this work are thorough. I like how this work starts with an agreement study and also complements the experiments with a number of interesting analyses (e.g., experiments with synthetic data, other reference-based metrics, and qualitative analysis on behavior difference).\n\n## Weaknesses\n1. This work is tested on two different models: Qwen 2.5 and Llama. The performance on Qwen is really good. However, the trend on Llama is quite different. The different trends across different models are a bit concerning, as one may wonder if the advantages of using BLEUBERI are an artifact of the Qwen model. This work currently contains a few experiments, only focusing on Qwen models. Will the findings be the same if they are conducted on Llama models?\n2. This is not really a weakness of the current work, but I hope this work can provide more insights into how different post-training methods lead to different behaviors. Using reference-based metrics as a reward feels like a middle ground between SFT (which enforces copying every token in the reference) and RL with RM (which does not tie closely to the reference at all). I'm super curious about how these supervisions with different \"strengths\" lead to different behavior. While the current work already provides some interesting case studies, there are no conclusive or systematic findings yet.", "was_revised": true, "final_justification": "The main idea of this paper is enough interesting so I'd inclined to accept this paper despite that there are real room for improvements for this paper (as pointed out by me and other reviewers). The authors provide a good response to my review, but it does not fully address my comment in weakness 1 (Qwen results are still much better than Llama results), so I'll keep my score unchanged.", "rebuttal": "We appreciate the reviewer’s recognition of our work, including the originality of using BLEU as a reward signal for alignment, the clarity of our writing, and the thoroughness of our analyses, including the agreement study, synthetic data experiments, alternative metrics, and qualitative behavior analysis. Below, we respond to each of the weaknesses and questions in detail.\n\n## Weaknesses\n\n***`1. One may wonder if the advantages of using BLEUBERI are an artifact of the Qwen model.`***  \nWe acknowledge that Qwen is a much stronger base model than Llama, and thus it is easier to get RL to work on the Qwen models compared to Llama. Prior work has observed similar findings in RL experiments in other domains (Shao et al., 2025). In our experiments, we apply a brief stage of SFT to Llama before running RL to provide the model with a basic understanding of instruction-style data. A more detailed discussion can be found in **Section 3.2, Line 218-223**. Importantly, our results in **Table 1** show that on Llama-3.1-8B (with SFT initialization), BLEUBERI *can* achieve competitive performance with GRPO-RM and SFT, suggesting that **the effectiveness of BLEUBERI is not merely an artifact of the Qwen base model**.\n\nReferences:\n\n* Spurious Rewards: Rethinking Training Signals in RLVR (Shao et al., 2025\\)\n\n***`2. More insights into how different post-training methods lead to different behaviors.`***  \nWe agree that using reference-based metrics as a reward is like a middle ground between SFT and RL with reward models. To summarize, our current findings suggest that:\n\n* SFT produces the most verbose and repetitive outputs, followed by BLEUBERI, then GRPO-RM (**Table 13**).  \n* BLEUBERI-trained models produce more factually grounded outputs than SFT-trained and GRPO-RM-trained models (**Section 4.2**).\n\nTo the best of our knowledge, our work is the first to explore applying simple reference-based metrics to improve general-domain instruction following. We hope these findings **lay the groundwork for more systematic explorations** into how different forms of supervision shape model behavior in instruction following tasks.\n\n## Questions\n\n***`1. BLEUBERI with synthetic data feels like some sort of distillation on a high level. Have you compared it with distillation methods empirically?`***  \nWe agree that BLEUBERI shares a high-level resemblance to knowledge distillation. In our current experiments, SFT acts as the closest analog to distillation, though it does not incorporate teacher logits in the way classical distillation methods do. As shown in Table 1, BLEUBERI consistently outperforms SFT across benchmarks. This is likely because **SFT tends to induce broad behavioral shifts, whereas GRPO facilitates more targeted, fine-grained adjustments** (Rajani et al., 2025).\n\nIn the next revision of the manuscript, we plan to include a standard distillation baseline for comparison.\n\nReferences:\n\n* Scalpel vs. Hammer: GRPO Amplifies Existing Capabilities, SFT Replaces Them (Rajani et al., 2025\\)\n\n***`2. What will happen if you do not filter out easy data when training with BLEUBERI? Are there some common reward hacking patterns?`***\n\nWe report results from training with BLEU on easy, medium, and randomly sampled data in Table 4 (Appendix B.2). While random sampling improves over the base model, training on the hardest examples yields significantly better performance. This is in line with findings from prior work (Qwen Team, 2025).\n\nRegarding common reward hacking patterns, we do observe that models trained on easy data tend to produce more repetitive outputs compared to those trained on harder examples. We will try to quantify this better in the next version.\n\nReferences:\n\n* Qwen3 Technical Report (Qwen Team, 2025\\)\n\n***`3. In the agreement study in Sec. 2, the best method is to use BLEU + RM. Have you tried similar approaches during training? Will it also lead to the best performance?`***  \nYes, we report training results using the combined BLEU+RM reward in **Table 11 (Appendix B.10)**. Interestingly, while this reward showed the highest agreement with human preferences in the evaluation study (Section 2), the BLEUBERI model slightly outperforms the BLEU+RM-trained model in terms of benchmark performance. \n\nFor the BLEUBERI and BLEU+RM models in Table 11 (both trained on 1,000 examples), we performed bootstrapping to estimate confidence intervals (please refer to our response to Reviewer R3fD for details). The performance difference between the two models appears modest, as their confidence intervals frequently overlap. Furthermore, we did not tune the reward weights in the BLEU+RM setup, which could leave room for further optimization.\n\n| Base model   | Variant         | MT-Bench | 95% CI         | ArenaHard v1 | 95% CI         | ArenaHard v2 | 95% CI         | WildBench | 95% CI         | Average |  \n|--------------|----------------|----------|----------------|--------------|----------------|--------------|----------------|-----------|----------------|---------|  \n| Qwen2.5-7B   | BLEUBERI        | 73.1     | (-3.7, \\+3.7)   | 30.9         | (-1.7, 2.3)     | 13.3         | (-1.2, \\+1.2)   | 64.0      | (-1.2, \\+1.3)   | 45.3    |  \n| Qwen2.5-7B   | GRPO-BLEU+RM    | 74.1     | (-3.4, \\+3.4)   | 26.4         | (-1.8, 2.7)     | 11.4         | (-1.2 / \\+1.4)  | 63.9      | (-1.3, \\+1.2)   | 43.9    |\n\n***`4. How much markdown formatting do you have in your training data? Does that explain the trends in Sec. 4.1?`***  \nAmong the 5,000 training examples used for each model in Table 1, here is the percentage of data where the reference response contains markdown formatting:\n\n* Qwen2.5-7B: 21.8%  \n* Qwen2.5-3B: 24.0%  \n* Llama-3.1-8B (SFT init.): 13.8%\n\nWhile markdown is not overwhelmingly dominant in the training data, its presence likely contributes to the trends in Section 4.1 (Table 13). GRPO-based methods may amplify features that correlate with higher rewards (e.g., structured formatting), so even modest amounts of markdown in the training data could be reinforced. In contrast, SFT matches references token-by-token, which may underemphasize sparse formatting cues, leading to lower markdown usage in outputs.\n\n***`5. I don’t get the “amplifies subtle biases” part in line 291. Can you elaborate?`***  \nIn line 291, “amplifies subtle biases” refers to the phenomenon observed in Zhao et al. (2025), where RL post-training tends to magnify preferences or stylistic patterns already weakly present in the pretrained model, even if they are not strongly expressed in the base model outputs. We will revise this sentence in the next version and add the appropriate citation for clarity.\n\nReferences:\n\n* Echo Chamber: RL Post-training Amplifies Behaviors Learned in Pretraining (Zhao et al., 2025\\)"}, {"review_id": "ixY46fC4Mm", "reviewer": "Reviewer_G3EU", "summary": "This paper describes experiments using BLEU as the reward function for training models with GRPO. Experiments have been conducted on 4 instruction-following benchmarks and they find that effectiveness of using BLEU is contingent on being able to create high-quality references during training. Specifically, the authors use data from the Tulu v3 SFT mixture, and use 4 powerful models to create reference answers. Along with the use of human answers from the SFT dataset 4 LLMs are trained using BLEU as the reward function. Experiments conducted on MT Bench, Arena Hard v1 and v2, Wildbench suggest using BLEU as a reward function outperforms SFT and is marginally below using an existing LLama based reward model. Human evaluation studies on a small set of $120$ responses reveal that model outputs are comparable to RM-trained models.", "questions": "Q1. A surface-level metric like BLEU likely just steers the model towards responding \"like\" strong models which are already human-aligned. Is this really about BLEU -- isn't the use of n-gram overlap just saying, the model should prefer responses to be similar to the way the \"frontier\" models respond (if one uses the LLM references) or asking the model to \"prefer the response that's  {\\em more} `similar' to the reference answer? I note that the results in Table 1 do not use the additional synthetic references\n\nQ2. Related to my first question, a baseline that feels missing is one that's based on an LLM as a judge -- tasked with the same/similar task as BLEU. Lets say we ask an LLM something like \"which of these responses is closer/equivalent to the ground-truth reference\"? Would we expect this to do even better than BLEU considering it might do a more meaningful comparison with a reference answer? And if it would, aren't we just saying GRPO can use a reward function that compares the relative goodness/closeness/alignment to a high quality reference as the training criterion?", "limitations": "Discussing aspects related to the questions in the review may be a useful discussion.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths\n  - Interesting idea to use a metric such as BLEU\n  - Extensive experiments (including some interesting ones in the appendix)\n- Weakness\n   - See questions", "was_revised": true, "final_justification": "The paper will benefit from a round of revision especially including baselines discussed during the rebuttal period. I would like to retain my score.", "rebuttal": "We appreciate the reviewer’s recognition of our work, including the interestingness of using BLEU as a reward signal for alignment, and our extensive experimental scope. Below, we respond to each of the weaknesses and questions in detail.\n\n## Weaknesses (same as questions)\n\n***`1. BLEU’s effectiveness is dependent on reference quality`***  \nWe acknowledge this limitation, but we also want to highlight that **obtaining high-quality references is relatively straightforward**. Such data can come from two sources:\n\n* **Synthetic outputs from frontier LLMs**: Generating synthetic data has become a standard component of modern LLM training pipelines (Lambert et al., 2024; Qwen Team, 2024). These synthetic outputs have proven to be highly effective in practice.  \n* **Ground-truth responses from existing datasets**: Numerous publicly available datasets already provide high-quality reference answers. Prominent examples include Tulu 3 (Lambert et al., 2024), OpenOrca (Mukherjee et al., 2023), UltraChat (Ding et al., 2023), and Alpaca (Wang et al., 2022).\n\nIn contrast, **training reward models demands extensive human-labeled preference data, which is costly and time-consuming**. Reference-based approaches like BLEUBERI offer a more accessible and efficient path to alignment, demonstrating that even simple metrics like BLEU can serve as practical alternatives to reward models.\n\nReferences:\n\n* Tulu 3: Pushing Frontiers in Open Language Model Post-Training (Lambert et al., 2024\\)  \n* Qwen2.5 Technical Report (Qwen Team, 2024\\)  \n* Enhancing Chat Language Models by Scaling High-quality Instructional Conversations (Ding et al., 2023\\)  \n* Orca: Progressive Learning from Complex Explanation Traces of GPT-4 (Mukherjee et al., 2023\\)  \n* Self-Instruct: Aligning LM with Self Generated Instructions (Wang et al., 2022\\)\n\n***`2. A baseline that feels missing is one that’s based on an LLM as a judge -- tasked with the same/similar task as BLEU`***  \nWe agree that using an LLM-as-a-judge to compare responses against a reference could, in principle, provide a more semantically nuanced reward than BLEU. However, incorporating it during training introduces potential drawbacks that may undermine the primary appeal of reference-based metrics (simplicity and efficiency):\n\n* LLMs are much more expensive to run than BLEU and similar metrics like ROUGE. Additionally, if the LLM judge generates long-form justifications, it will be much slower than reward models.  \n* Reference-based metrics like BLEU provide a fully transparent and deterministic reward signal, whereas LLM-based judgments are inherently opaque.\n\nTo clarify, our focus on BLEU throughout the paper stems from its unique position among reference-based metrics: **it has a particularly simple form, yet it can already be surprisingly effective**. Our intent is not to argue that BLEU is the optimal reference-based reward function. Rather, we demonstrate that **even such a basic metric can rival the performance of learned reward models (which are essentially finetuned LLM judges)**. We hope this insight can encourage further exploration of other reference-based metrics."}]

Author rebuttal:

Meta-review: This paper introduces BLEUBERI, a method that leverages BLEU as a reward signal in GRPO for instruction-following alignment, replacing the need for costly reward models. Despite BLEU being a relatively simple n-gram matching metric, the paper shows that it can be surprisingly effective, achieving performance comparable to GRPO-RM across most settings.

The reviewers agreed that the core idea is both practical and novel. They highlighted several strengths: BLEU is efficient to compute compared to expensive reward models, and BLEUBERI achieves competitive or even superior results across four challenging benchmarks and three different base models.  The paper also shows that BLEUBERI-trained models tend to be more factually grounded than those trained with reward models or SFT, and the authors provide thoughtful analyses (agreement study, synthetic data experiments, metric comparisons, and qualitative analyses) that strengthen the work. While the evaluation could be further expanded, the results presented, along with the additional rebuttal experiments, provide convincing evidence of the method’s potential. In the discussion, two reviewers expressed strong enthusiasm for the paper, and those who participated were overall positive, including one of the weak rejects. The other weak reject did not take part in the discussion.

Overall, the paper makes a useful and timely contribution. This is a simple but elegant idea, executed well, and the results are both interesting and insightful. The work provides a practical and cost-effective alternative to reward-model-based alignment that will be useful for the community.  I recommend acceptance, and I encourage the authors to incorporate the reviewers’ feedback and the new rebuttal experiments into the final version to maximize the paper’s impact.
Decision: Accept (poster)