Title: BLEUBERI: BLEU is a surprisingly effective reward for instruction following
Venue: NeurIPS 2025
Accepted: yes
Keywords: alignment

instruction-following

LLM

RLVR

GRPO
Abstract: Reward models are central to aligning LLMs with human preferences, but they are costly to train, requiring large-scale human-labeled preference data and powerful pretrained LLM backbones. Meanwhile, the increasing availability of high-quality synthetic instruction-following datasets raises the question: can simpler, reference-based metrics serve as viable alternatives to reward models during RL-based alignment? In this paper, we show first that BLEU, a basic string-matching metric, surprisingly matches strong reward models in agreement with human preferences on general instruction-following datasets. Based on this insight, we develop BLEUBERI, a method that first identifies challenging instructions and then applies Group Relative Policy Optimization (GRPO) using BLEU directly as the reward function. We demonstrate that BLEUBERI-trained models are competitive with models trained via reward model-guided RL across four challenging instruction-following benchmarks and three different base language models. A human evaluation further supports that the quality of BLEUBERI model outputs is on par with those from reward model-aligned models. Moreover, BLEUBERI models generate outputs that are more factually grounded than competing methods. Overall, we show that given access to high-quality reference outputs (easily obtained via existing instruction-following datasets or synthetic data generation), string matching-based metrics are cheap yet effective proxies for reward models during alignment. We release our code and data at https://github.com/lilakk/BLEUBERI.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper describes experiments using BLEU as the reward function for training models with GRPO. Experiments have been conducted on 4 instruction-following benchmarks and they find that effectiveness of using BLEU is contingent on being able to create high-quality references during training. Specifically, the authors use data from the Tulu v3 SFT mixture, and use 4 powerful models to create reference answers. Along with the use of human answers from the SFT dataset 4 LLMs are trained using BLEU as the reward function. Experiments conducted on MT Bench, Arena Hard v1 and v2, Wildbench suggest using BLEU as a reward function outperforms SFT and is marginally below using an existing LLama based reward model. Human evaluation studies on a small set of $120$ responses reveal that model outputs are comparable to RM-trained models.\n\nSTRENGTHS AND WEAKNESSES:\n- Strengths\n  - Interesting idea to use a metric such as BLEU\n  - Extensive experiments (including some interesting ones in the appendix)\n- Weakness\n   - See questions\n\nQUESTIONS:\nQ1. A surface-level metric like BLEU likely just steers the model towards responding \"like\" strong models which are already human-aligned. Is this really about BLEU -- isn't the use of n-gram overlap just saying, the model should prefer responses to be similar to the way the \"frontier\" models respond (if one uses the LLM references) or asking the model to \"prefer the response that's  {\\em more} `similar' to the reference answer? I note that the results in Table 1 do not use the additional synthetic references\n\nQ2. Related to my first question, a baseline that feels missing is one that's based on an LLM as a judge -- tasked with the same/similar task as BLEU. Lets say we ask an LLM something like \"which of these responses is closer/equivalent to the ground-truth reference\"? Would we expect this to do even better than BLEU considering it might do a more meaningful comparison with a reference answer? And if it would, aren't we just saying GRPO can use a reward function that compares the relative goodness/closeness/alignment to a high quality reference as the training criterion?\n\nLIMITATIONS:\nDiscussing aspects related to the questions in the review may be a useful discussion.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work presents a new post-training algorithm that uses the BLEU score on references as the reward. The BLEUBERI reward can be used in any RL optimization algorithms, including GRPO. The proposed method is evaluated on multiple tasks with two models (Qwen 2.5 and Llama 3.1). Models trained with BLEUBERI show similar or better performance compared to SFT or GRPO with normal RM rewards. This work also contains interesting analyses showing BLEUBERI leads to more grounded responses, and the method is also compatible with synthetic data or other reference-based metrics.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. Using BLEU as the reward during post-training is an original but natural idea. It's interesting to see that its performance is good and even surpasses SFT or RLHF with RM in some cases.\n2. This paper is well-written, and the analyses in this work are thorough. I like how this work starts with an agreement study and also complements the experiments with a number of interesting analyses (e.g., experiments with synthetic data, other reference-based metrics, and qualitative analysis on behavior difference).\n\n## Weaknesses\n1. This work is tested on two different models: Qwen 2.5 and Llama. The performance on Qwen is really good. However, the trend on Llama is quite different. The different trends across different models are a bit concerning, as one may wonder if the advantages of using BLEUBERI are an artifact of the Qwen model. This work currently contains a few experiments, only focusing on Qwen models. Will the findings be the same if they are conducted on Llama models?\n2. This is not really a weakness of the current work, but I hope this work can provide more insights into how different post-training methods lead to different behaviors. Using reference-based metrics as a reward feels like a middle ground between SFT (which enforces copying every token in the reference) and RL with RM (which does not tie closely to the reference at all). I'm super curious about how these supervisions with different \"strengths\" lead to different behavior. While the current work already provides some interesting case studies, there are no conclusive or systematic findings yet.\n\nQUESTIONS:\n1. BLEUBERI with synthetic data feels like some sort of distillation on a high level. Have you compared it with distillation methods empirically?\n2. What will happen if you do not filter out easy data when training with BLEUBERI? Are there some common reward hacking patterns?\n3. In the agreement study in Sec. 2, the best method is to use BLEU + RM. Have you tried similar approaches during training? Will it also lead to the best performance?\n4. How much markdown formatting do you have in your training data? Does that explain the trends in Sec. 4.1?\n5. I don't get the \"amplifies subtle biases\" part in line 291. Can you elaborate?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

comments: SUMMARY:
This paper presents BLEUBERI, which uses BLEU as a reward signal during GRPO instead of expensive reward models for alignment for instruction following. Despite that BLEU is just a simple n-gram matching metric, this paper demonstrates that it can be surprisingly effective in matching the performance of sophisticated reward models for instruction-following tasks.

STRENGTHS AND WEAKNESSES:
Strengths:
- BLEU is efficient to compute compared to calling expensive reward models, thus the empirical findings of this paper is valuable to the broader empirical community.
- Despite its simplicity, BLEUBERI-trained models perform comparably to more expensive approaches relying on reword models across four challenging benchmarks (MT-Bench, ArenaHard v1/v2, WildBench) and three different base models (Llama-3.1-8B, Qwen2.5-7B, Qwen2.5-3B).
- The authors also demonstrate that the closer guidance based on BLEU signals leads to models that are more factually grounded than those trained with traditional reward models or supervised fine-tuning, as measured by VeriScore factuality evaluation.

Weaknesses:
- This approach is applicable only when high-quality references are available to compute BLEU against. 
- The strengths still outweigh the above weakness, given that collecting reference data can be often more cost-effective than calling the reward models, and one can also leverage already existing datasets with references.

QUESTIONS:
1. What would happen if experimented with ROUGE (more recall focused) instead of BLEU (more precision focused), or use both?

2. how sensitive is it to use BLEU with different hyper-parameters, e.g., different n-gram length?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the feasibility of using the BLEU metric as a reward for the GRPO algorithm in instruction-following alignment. As a pivotal study on metric effectiveness, the authors show that BLEU’s agreement with human judgment improves as the number of references increases or when it is combined with other metrics. In the main experiments, BLEUBERI performs comparably to GRPO-RM across most settings. Notably, BLEUBERI demonstrates potential advantages in improving factuality while maintaining creativity.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. The paper revealed that Reward Models are costly for alignment GRPO and discovered that BLEU is surprisingly effective reward for instruction following. This topic has rarely discussed before and provides interesting findings to the community.\n\n2. The paper adopted synthethic data from tulu3, which might potentially usefule to researchers. Also, the paper has done an comparison of metric's effectivity, which produces insight to the community.\n\nWeaknesses\n\n\n1. Limited Analysis of Reward Hacking Risks. While the factual presentation is appreciated, the discussion of potential reward hacking remains limited. Upon reviewing the Appendix, I observed that BLEU used as a reward exposes models to two notable risks:\n\nVerbosity Bias: As shown in Table 13, BLEUBERI causes a marked increase in generated output length, indicating a verbosity issue.\n\nRepetition Bias: Table 13 also illustrates that BLEUBERI leads to higher repetition in generated text, a common failure mode for text generation models.\n\nThese factors raise significant concerns about the effectiveness of BLEU as a reward metric for instruction following. A more nuanced analysis—including mechanisms to detect and mitigate these biases—would improve the discussion.\n\n2. Insufficient Discussion of Alternative Rule-Based/Verifiable Metrics. More thorough analysis on rule-based/verifiable metrics. as presented in section2, Appendix B.10, there should be chances to in-depth discussion. It is hard for me to recognize why ROUGE-L, BERTScore, BLEUrt are not competitive to BLEU. Without such, it is hard to assess BLEU’s relative contribution as a low-cost reward. Also, the emsemble of rewards should also be considered. \n\n3. Concerns with Human Evaluation Reporting (line 320): The soft preference rate is misleading. One cannot calculate the tie rate into the 'for' rate. The same problem exists in Appendix Figure 11. Besides, Cohen Kappa of 0.36 for triple selection indicates the observed two annotators agreement is around 45%, which does not demonstrate strong reliability for the human evaluation results. Emphasizing these limitations and contextualizing the implications for the overall conclusions would be prudent.\n\nQUESTIONS:\nI will be insterested to see the results on larger base model sizes (32B) with the same reward models comparison (RM-8B vs BLEUberi). If the results are positive for BLEUberi, this would be a strong demonstration for light weight reward function.\n\nA more direct quantification for GRPO algorithm speed-up between RM-8B and BLEU might be presented.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper introduces BLEUBERI, an approach for aligning Large Language Models (LLMs) with human preferences by directly using BLEU, a simple string-matching metric, as a reward function in reinforcement learning (RL). Traditionally, LLM alignment relies on costly reward models that require extensive human-labelled preference data. The authors surprisingly demonstrate that BLEU effectively models human preferences for general instruction-following tasks, even rivalling large reward models in agreement. Specifically, BLEU with five synthetic references achieved a 74.2% agreement with human preferences, comparable to a 27B-parameter reward model's 75.6%.

STRENGTHS AND WEAKNESSES:
Strengths

1. BLEU surprisingly matches strong reward models in modelling human preferences for general instruction following tasks
2. Cost effectiveness
3. Experimental evaluation demonstrating enhanced factuality and positive human evaluation

Weaknesses

1. Reliance on reference quality is the biggest concern
2. Absence of statistical significance results 
3. Limited experiments on exploring effects of scaling model size, data volume or extensive training time
4. Known issues in mathematical reasoning tasks

QUESTIONS:
Please refer to weaknesses above. 

1. How would this work be adapted to dealing with mathematical reasoning tasks? Any initial experiments or thoughts?
2. How does variability in reference quality impact the effectiveness of the training process
3. Can BLEU be prone to more subtle reward hacking?
4. Can this metric work well in multi-step reasoning tasks? Why was there a performance decline with enforced CoT reasoning? Could you elaborate please?

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-04