Better Instruction-Following Through Minimum Bayes Risk
Authors: Ian Wu
Patrick Fernandes
Amanda Bertsch
Seungone Kim
Sina Khoshfetrat Pakazad
Graham Neubig
Venue: ICLR 2025 Spotlight (2025)
Keywords: LLM
instruction-following
test time compute
decoding
MBR
minimal bayes risk
LLM judges
self-improvement
Primary area: foundation or frontier models, including LLMs
TL;DR: We investigate Minimum Bayes Risk decoding with LLM judges as utility metrics to improve instruction-following LLMs.

Abstract: General-purpose LLM judges capable of human-level evaluation provide not only a scalable and accurate way of evaluating instruction-following LLMs but also new avenues for supervising and improving their performance. One promising way of leveraging LLM judges for supervision is through Minimum Bayes Risk (MBR) decoding, which uses a reference-based evaluator to select a high-quality output from amongst a set of candidate outputs. In the first part of this work, we explore using MBR decoding as a method for improving the test-time performance of instruction-following LLMs. We find that MBR decoding with reference-based LLM judges substantially improves over greedy decoding, best-of-N decoding with reference-free judges and MBR decoding with lexical and embedding-based metrics on AlpacaEval and MT-Bench. These gains are consistent across LLMs with up to 70B parameters, demonstrating that smaller LLM judges can be used to supervise much larger LLMs. Then, seeking to retain the improvements from MBR decoding while mitigating additional test-time costs, we explore iterative self-training on MBR-decoded outputs. We find that self-training using Direct Preference Optimisation leads to significant performance gains, such that the self-trained models with greedy decoding generally match and sometimes exceed the performance of their base models with MBR decoding.

Published as a conference paper at ICLR 2025
BETTER INSTRUCTION-FOLLOWING THROUGH MINI- MUM BAYES RISK
Ian Wu' Patrick Fernandes Amanda Bertsch² Graham Neubig"
Seungone Kim?
Sina Pakazad
C3 AI Carnegie Mellon University Instituto de Telecomunicações "Instituto Superior Técnico, University of Lisbon
(ian. wu, sina.pakazad) @c3. ai (pfernand, gneubig (eteiiggssss âcs. cmu, edu (abertsch, seungonk ) @andrew. cmu.edu
ABSTRACT
General-purpose LLM judges capable of human-level evaluation provide not only a scalable and accurate way of evaluating eetrucctionfollooiing LLMs but also new avenues for supervising and improving their performance. One promising way of leveraging LLM judges for supervision is through Minimum Bayes Risk (MBR) decoding, which uses reference-basee evaluator to select high-quality output from amongst a set of candidate outputs. In the first part of this work, we explore using MBR decoding a method for improving the test-time performance instructiion-follooinng LLMs.. We find that MBR decoding with reference reference-based greedy decoding, best-of-N decoding
LLM judges substantially improves with reference-free judges and MBR decoding with lexical and embedding baaeedddgg-baedd metries on Alpacalvval and MT-Bench. These gains are consistent across LLMs with up to 70B parameters. demonstrating that smaller LLM judges can be used supervise much larger LLMs. Then, seeking to retain the improvements from MBR decoding while mitigating additional test-time costs, we explore iterative self-training on MBR-ddccded outputs. We find that self-training using Direct Preference Optimisation leads to significant performance gains, such that the self- trained models with greedy decoding generally match and sometimes exceed the performance of their base models with MBR decoding.
INTRODUCTION
Instruction dearuction-followingg large language models (LLMs) (Chung et al. 2022 Wei et al., 2022 have shown remarkable potential as generalist problem-solvers prompting extensive efforts improve their performance. One task that has seen tremendous progress due to LLMS is the evaluation of text generation itself. Recent works find that "LL.M-as-a-Judee
frameworks (Zheng et al.) 2023
Li et al.) 2023) Dubois et al.) 2024a) demonstrate strong correlation with human evaluations and significantly outperform lexical Lin 2004b; Papineni et al. 2002) and embedding-baaee based methods
Zhang et al.) Yuan et al., 2021 Qin ct al. 2023) across a wide range of instruction- instruction-folloinng tasks.
While the use of LLM judges has largely focused on the evaluation of outputs, LLM judges can also provide a way to supervise the generations of other LLMs. This generally involves using the judge as a reference-free evaluator score candidate outputs produced by the LLM and then selecting the highest-scoring candidate as the final output in what is known as best-of-N (BoN) decoding (Song) jet al., 2024). However, prior works find that L.M judges, including powerful proprietary L.LMs such as GPT-4, significantly underperform when no human-curated reference answer is available (Ye et al. | 2024 Zheng et al. 2023). contrast, reference- based evaluation, where a human- curated reference answer is available, shows significantly higher correlation with human evaluations of out- puts. This poses a chicken-and-ege problem: how can we leverage reference- based LLM judges for test time generation if no human references are available?
Minimum Bayes Risk (MBR) decoding (Bickel & Doksum, 1977) provides a way of overcoming this problem. In place of the inaccessible human references, MBR decoding leverages other candidate

---

Published as a conference paper at ICLR 2025
Section 3: HBR Inference
Paeude-Raternnce
Biinddee in
Region of ultr dress spazatime.
Darkinstein space.
Dialuator
Eighole in apece.
1,0
0.38
0.77
0.72
Decoded Outpet
black if your okl.
Moa sood
Region of ufm
derse spacetime.
0.43
1.0
0.35
0.57
Dava holo in
space.
Prompt
LLM
hala
0.03
0.29
1,0
0.73
Gendidatoos
Margnalto
Section & MBR Dies Blation
Figure 1: Illustration of MBR decoding. Multiple candidates are first sampled from an LLM. Then, each candidate is evaluated against all other candidates (pseedd--rfferences) using a reference-basee evaluator. The pscudo-references are marginaliscd to produce final scores, and the candidate with the highest score is selected.
outputs as preuuo-references, and uses the evaluator, also known as the utility metric, to conduct reference-based evaluation of all candidate outputs against all pseudo-reffrences. The final output is then chosen as the candidate output with the highest average score: see Figure !
In this work, we explore whether MBR decoding using LLM judges as utility metrics can be used to enhance instruction-followinn LLMs. We divide our work into two main parts. First, inspired by the effectiveness of scaling inference time compute (Welleck et al. 2024 Snell al., 2024). we investigate whether MBR decoding with LLM judges can improve the performance of instruction- following LLMs during test time (denoted as "MBR inference") (Section D} Second, following recent works demonstrating that iterative self-training can improve LLM performance (Xu et al., 2023 Chen al.) 2024] Yuan et al., 2024a Wu al.) (024)), we examine whether MBR decoding with LLM judges be used select high-quality model outputs for use in subsequent iterations of self-training (denoted as "MBR distillation") (Section D This both provides a way of training models without access to external labels and also allows associated with MBR inference.
mitigate the inference-time costs
From MBR inference experiments, we find that MBR decoding with LLM judge Prometheus- 2-7B (Kim et al., 2024) improves performance by +3.6% on AlpacaEval and +0.28 on MT-Bench on average across five L.LMs relative greedy decoding. Notably, Llamma--77b with Prometheus MBR decoding outperforms Llama-2-13b with greedy decoding on MT-Bench, while Prometheus MBR decoding with Llama-2- outperforms Llama-2-70b with greedy decoding on AlpacaEval 2.0. Gains persist even for large 70B models, demonstrating that small LLM judges can supervise larger LLMs through MBR decoding. We also compare MBR decoding against other methods that use LLM judges for supervision. We show that Prometheus MBR decoding is far more effective than metrics
MBR decoding with word match-based metrics (e.g. ROUGE) or semantic similarity-base (e.g. BEERTScore). Comparing MBR to BoN decoding, we find that MBR decoding consistently outperforms BoN decoding across multiple LLMs and LLM judges, and that the gains from MBR decoding increase the supervising LLM judge increases in size and ability.
From our MBR distillation experiments, we find that self-training with Direct Preference Optimisa- tion (DPO) (Rafailov et al. 2024) preference pairs selected using MBR decoding (Yang et al.) 2024 with Prometheus-22 2-7B substantially improves greedy decoding performance. For instance, MBR self-trained Llama-2- 13b improves by +7.1% on AlpacaEval 2.0 and +0.90 on MT-Bench rel- ative to its baseline SFT counterpart when evaluated using only greedy decoding, far surpassing the corresponding gains from BoN self-training. We also find that MBR self-trained models evaluated with greedy decoding generally match and sometimes exceed the performance of their base models evaluated with MBR decoding, thereby demonstrating that MBR distillation an effective way of mitigating the inference-ting costs of MBR decoding while retaining improved performance.
2 BACKGROUND
Language models are autoregressive probabilistic models; i.e., the probability of a token y: depends on prompt and all previous tokens in the sequence:
p(yl)) = plulu ............... 2).
(1)
2

---

Published as a conference paper at ICLR 2025
During inference, outputs typically obtained either using maximum a-posteriori-based decoding methods that attempt to maximise probability, such as greedy decoding (y/ = arg m p(uily=a. x)) and beam search (Graves 2012), by tokenwise sampling from the distri- bution (y/= (MMl)ea.2)) z)). Both rely the model's ss distribution indicative of output quality.
Alternatively, we can first obtain a hypothesis set Hhyp comprising Neaud candidate outputs from the model (for example, by sampling multiple times), and then select the final output from Hayp based on some external criteria. For example, given some reference-fre evaluator u (e.g. an LLM judge), best-of-N (BoN) decoding selects the output ý Hhyp such that
y arg max u(y)-
yEHhyp
As reference-free estimation of output quality be difficult problem, MBR decoding replaces the reference-free evaluator with a reference-based evaluator v(y, y (e.g. a reference-hased LLM judge) that evaluates candidate y relative to a reference y In the MBR literature, this evaluator is known as a utility metric (Freitag et al. 20222 Fernandes et al. [2022] Finkelstein et al., (022)). MBR decoding selects the final output y that maximises expected utility under the model distribution:
ÿ = arg max yE Hayp
Ey Eye~plvin Bue-ppva)alu(y.3
Neand Neand u(y, y())
where the expectation is approximated as a Monte Carlo sum using model samples y(a), y()),...,y((aann))) ~ p(y]z). In practice, this amounts to computing the utility of each candidate in Hhyp using all other candidates as the highest average utility as the final o see Appendix [. for an algorithmic description of MBR decoding. The MBR-decoded output can therefore be interpreted as being the candidate with the highest "consensus" utility measured by the utility metric, as it achieves the highest aver- age utility when evaluated against all other candidate outputs. It is therefore crucial choose a utility function, this
and then selecting the candidate with
reference-basee metric that is good proxy for human preferences ensures that a "high- congesconsessss output corresponds to a "high-quality" output.
3 MBR INFERENCE
In this experiment, we investigate using MBR decoding with LLM judge utility metrics improve instruction-folloiing LLMs at test time.
3.1
EXPERIMENTAL SETUP
3.1.1 MODELS AND GENERATION PARAMETERS
We use the chat and instruct variants of the Llama-2 Touvron et al. 20233) and Llama-3 (Dubey) al., 2024) models in this experiment. All models have undergone prior SFT and demonstrate strong instruction-fir -orruuttioooooooiinnn and conversation abilities. We generate Neand = 30 candidates using temperature sampling with 0.3 for all MBR decoding experiments unless otherwise specified.
3.1.2 MBR UTILITY METRICS
A.Mijudge We choose Prometheus-2-7B (Kim et al. 2024 representative reference- based LLM judge. Prometheus is a specialist judge model finetuned from Mistral-7b (Jiang et al.] 2023) that correlates strongly with human judges and GPT-4. It takes inputs task prompt, scoring rubric (see Appendix (). a candidate and a reference, and outputs an explanation its judgement followed by score from 5, which we interpret as utility score. Crucially, Prometheus can also act as reference-free judge by simply omitting the reference from its input. This allows us to directly compare MBR with BoN decoding using the same LLM judge utility metric.
!Certain evaluators (c.g. L.M judges) "task-aware" and take prompt an input when performing evaluation. Such utility metries can then be written as u(y; 2) and u(y, v)
"The expectation can also be computed over separate set of model outputs known as the evidence set Fikema & Aziz 2020 Bertsch al. 2023.. We do not explore this setting work.
3

---

Published as a conference paper at ICLR 2025
We compare using LLM judges for MBR decoding with three other classes of utility metries:
ROUGE ROUGE (Lin 2004a) is a word-level F-measure designed for measuring summarisation and machine translation performance (Lin 2004a). We use ROUGE-1 in main experiments but include results for other ROUGE variants in Appendix A.3
BERTScore BERTScore is neural evaluation metric that computes the token-level contextual similarity between a candidate and a reference. Like ROUGE, BERTScore is not task-aware. As model outputs may be longer than 512 tokens, we use Longformer-large (Beltagy et al. 2020) as our BERTScore model, which supports inputs up to 4094 tokens long.
Dense embedders Dense embedders generate contextual embeddings of text passages for use in downstream tasks. One such task is measuring the level of semantic similarity between two text passages (Agirre et al. 2012). This task is directly relevant MBR decoding, we can treat pairs of candidates as text passages and their similarity score as the utility, To the best of our knowledge, using dense embedders as a utility metric for MBR decoding has never been explored before. In our work, we use the cnttuuttionfollowiiit embedder 2FR-Embedder-22 R (Meng et al. 2024) as our representative dense embedder. We include results for two other dense embedders Appendix A.3
3.1.3 BASELINES
In addition greedy decoding and beam search (BS) with k 10 beams, we also experiment with LONGEST decoding, where we select the longest candidate from the hypothesis set (as measured in characters) as the final output, and best-of-N (BoN) decoding. We generate Neand 30 candidates using temperature sampling with = 0.3 for both longest and BoN decoding. See Appendices A.2 and A.3 for comparisons with additional baselines.
3.1.4 EVALUATION
AlpacaEval 2.0 AlpacaEval (Li et al. 2023) is LLM-based evaluation metric. It consists of an 805-sample, highly diverse single-turn instruction-following conversational dataset and an as- sociated evaluation framework. In AlpacaEval 2.0 (Dubois al.! 2024b), evaluation is conducted by performing head-to-head comparison of candidate answers against GPT-4-generated answers fa- cilitated by a judge LLM. The judge model is prompted to output single token representing its choice of winner, with the rog-pobbaiilites of the token used to compute a weighted win rate. In (LC) win rates, which
addition to standard win rates, AlpacaEval 2.0 also provides length-controlee are debiased versions of the standard win rates that control for the length of the outputs. Both the AlpacaEval standard and LC evaluation demonstrate strong correlation with human judgements.
MT-Bench MT-Bench (Zheng et al. 2023 is an 80-sample, two-turn eestruction-follovvii con- versational dataset. can be evaluated using either head-to-head comparison or direct assessment with an LLM judge. In the direct assessment setting, the judge LLM is prompted to generate an explanation followed by a score between and 10, with no reference answer used. MT-Bench with GPT-4-judge matches crowdsourced human preferences well, achieving over 80% agreement, which is the same level of agreement between human evaluators (Zheng et al. 2023).
We use GPT-4o (OpenAl al. 2024) the LLM judge for both AlpacaEval 2.0 and MT-Bench. For AlpacaEval, we report L.C win rates unless otherwise stated. For MTT-Bench, we use direct assessment for all experiments. See Appendix B for further details on our evaluation strategy, and Appendix H for human study findings verifying the alignment of our automatic LLM evaluation results with human judgements.
3.2 EXPERIMENTAL RESULTS
Our main experimental results are documented in Tables I and
Prometheus MBR decoding provides significant and consistent gains The gains associated with Prometheus MBR decoding are significantly larger than those associated with other utility metrics, yielding an average improvement of +3.6% on AlpacaEval 2.0 and +0.28 on MT-Bench. For comparison, the performance gap between Llama-2-7b and Llama-2- 13b with greedy decod- ing is +4.6% on AlpacaEval 2.0 and +0.18 on MT-Bench, while the corresponding gap between Llama-2-13h and Llama-2-70b is +3.8% and +0.60. Notably, Llama-2-7h with Prometheus MBR
4

---

Published as a conference paper at ICLR 2025
2-7B
2-13B
2-70B
3-8B
3-70B
Avg. A
Greedy
14.4
19.0
22.8
34.4
42.7
0
BS
14.8
18.2
21.5
33.9
42,4
-0.50
Longest
10.5
15.2
19.8
29.8
40.4
-3.51
Promethcus BoN
16.4
20.8
25.0
35.5
44.3
1.74
ROUGE MBR
16.2
20.0
24.7
35.4
43.7
1.33
BERTSccor MBR
16.2
20.5
24.4
35.7
44.0
1.50
SFR-Embedde MBR
12.1
16.6
22.2
32.5
42.8
-1.42
Prometheus MBR
17.7
23.4
26.2
37.9
46.0
3.62
Table 1: AlpacaEval 2.0 win rates (%) for various models and decoding strategies, along with the average win rate differences compared to greedy decoding across all models (denoted as Avg. A). MBR decoding with Prometheus consistently outperforms all baseline methods and other MBR decoding methods.
2-7B
2-13B
2-70B
3-8B
3-70B
Avg. A
Greedy
5.72
5.90
6.50
7.54
8.29
0
BS
5.58
6.49
7.30
8.20
-0.09
Longest
5.67
6.03
6.59
7.22
8.22
-0.04
Prometheus BoN
5.77
6.08
6.65
7.66
8.42
0.13
ROUGE MBR
5.78
6.11
6.68
7.63
8.31
0.11
BERTScore MBR
5.68
6.72
7.52
0.08
SFR FER-Embedder MBR
5.73
6.04
6.54
7.45
8.33
0.03
Prometheus MBR
6.10
6.26
6.79
7.69
8.50
0.28
Table MT-Bench scores for various models and decoding strategies, along with the average score differences
compared to greedy decoding across all models (denoted as Avg, A). MBR decoding with Prometheus consis- tently outperforms all baseline methods and other MBR decoding methods.
decoding outperforms Llama-2-133 2-13b with greedy decoding on MT-Benchh while Prometheus MBR decoding with Llama-2-130 outperforms Llama-2-70b model over five times bigger with greedy decoding on AlpacaFval 2.0. We also find that Prometheus MBR decoding yields larger gains than Prometheus BoN decoding; we explore this further in Section 3.3.
We also highlight that the performance gains associated with Prometheus MBR decoding are sig- nificant across models all sizes, even for much larger models such as Llama-3-70b. This scaling property suggests that small judge models can still be used to supervise much larger models.
ROUGE and BERTScore MBR decoding provide small but consistent gains ROUGE and BERTScore MBR decoding improve average performance relative greedy decoding by +1.3% and +1.5% on AlpacaEval 2.0 and by +0.11 and +0.08 on MT-Bench respectively. This benefit is present for all models. This improvement suggests that selecting outputs without awareness of the task and using only word- or token-level measures of consistency can still yield meaningful improvements in the intructiionfollowiini setting.
SFR-Embedder MBR decoding fails to yield consistent gains SFR-Embedder MBR decoding reduces performance relative greedy decoding by -1.4% AlpacaEval 2.0 while improving per- formance by +0.03 on MT-Bench on average. We hypothesise that embedder models, which are trained distinguish between semantically-siminn
high level between text passages, cannot detect nuanced differences outputs. We also note that embedder MBR decoding generally se- lects for longer outputs, which may explain the discrepancy between its performance on AlpacaEval 2.0 (which is length-com congh--eotrolled) and MT-Bench See Appendix A.4 for amalysis on the generation lengths of various decoding strategies.
Beam search and LONGEST decoding degrade performance Beam search and LONGEST de- coding reduce performance relative to greedy decoding by -0.5% and -3.5% on AlpacaEval 2.0 and -0.09 and -0.04 on MT-Bench respectively. The poor performance of beam search further under- scores the idea that optimising for output probability alone is not enough to improve output quality.
3.3 ANALYSIS OF PROMETHEUS MBR DECODING
Given the promise shown by Prometheus MBR decoding, we conduct additional experiments to better understand its properties.
5

---

Published as a conference paper at ICLR 2025
26
27.5
Win Rate GPP144
25.0
25
22.5
24
Win Rate
20.0
23
Grondy
BoN
15.0
BoN
10
15
20
25
30
0.0
0.2
0.4
0.6
0.8
1.0
1.2
144
Moand
Figure 2: AlpacaEval 2.0 win rates (%) for Llama-2-700 with varying hypothesis set size Neand (left) and gen- cration temperature (right) values for Prometheus MBR and BoN decoding. Performance for both methods initially increases with Neand and platcaus at around Neand = 20. Performance also initially increases with but drops rapidly after t = 1.0.
Prometheus MBR decoding performance and Neand We plot AlpacaEval 2.0 win rates a function of Neand and t in Figure 2 for Llama-2-70b with Prometheus MBR and BoN decoding. We find that performance initially increases with Neand but plateaus around Neand = 20, sug- gesting that expanding the size of the hypothesis set beyond this yields little benefit, Performance also initially increases with highlighting the benefits of increased candidate diversity, although rapidly degrades at high temperatures as the individual candidate outputs decline in quality.
Win GPT-4
1
Technical witting
Recommendations
Creative wiiting
Coding
Mathematical reasaning
Howr-to
Factual
Argumentation
Puzztes lngical reasoning
Other
Figure 3: Difference in AlpacaEval 2.0 win rates (%) between Prometheus MBR decoding and greedy decoding averaged five L.MSs and broken down by question category. positive value indicates that MBR decoding outperforms greedy decoding on the given category. Orange bars represent standard error. find that Prometheus MBR decoding improves performance across a wide range of question categories.
Prometheus MBR decoding performance by question category We classified questions from the AlpacaEval dataset into one of ten categories using GPT-4o (see Appendix for details), and then computed the differences in win rates between Prometheus MBR decoding and greedy de- coding by question category, averaged over all five LLMs. We find that MBR decoding improves output quality across most question categories. These include reasoning- coding and mathematical reasoning, although the largest improvements are seen across writing- based categories such as technical, recommendations and creative writing. We hypothesise that this discrepancy arises due to (I) the higher bar for correctness associated with reasoning tasks which limits the number of good answers that can be found amongst candidate outputs; and (2) limitations of existing utility functions, which may struggle to handle difficult reasoning tasks.
categories such as
3.3. FURTHER COMPARISONS WITH BEST-OF-N DECODING
As BoN decoding can also leverage LLM judges as a utility metric, we conduct additional exper- iments to compare its performance against MBR decoding. We compare BoN and MBR decod- ing for five different LLM judges on MT-Bench and report the results Table S In addition to Prometheus-2444 and JudgeLL--B3 (Zhu et al. 2023c), which are two judge models finetuned from LLaMA models Touvron al. 2023a). We also assess -lama-3-8h-Instruu and Jlmma-3770b- 70b---rruut zero-
also evaluate its larger sibling Prometheus-2-8xxBB
well JudgeLMM-7h

---

Published as a conference paper at ICLR 2025
2-7B
2-13B
2-70B
3-8B
3-70B
Avg. A
Greedy
5.72
5.90
6.50
7.54
8.29
0
-romethhuss22-7BBBae
5.77
6,08
6.65
7.66
8.42
0.13
Prometheus-227B-MMBR
6.10
6.26
6.79
7.69
8.50
0.28
Promethees-2-8xx7BBBB
6.01
6.17
6.80
7.75
8.41
0.24
Prometheus-2-xx7BB--o
6.26
6.32
6.87
7.79
8.64
0.39
JudgeLM-7b-BoN
5.63
5.95
6.69
7.37
8.26
-0.01
JudgeLM-7b--ke
6.00
6.11
6.79
7.69
8.44
0.22
JudgeLMM333--Bo
5.68
6.03
6.58
7.37
8.35
0.01
JudgeLLM-33b-MBBR
5.94
6.27
6.88
7.92
8.50
0.31
Llama-3-8b--nsstrre---oo
5.83
6.05
6.61
7.60
8.38
0.10
Bauma-3----IIntrut--MBBB
5.96
6.28
6.84
7.80
8.47
0.28
Saama-3-70b-Instuuce
5.77
6.16
6.57
7.39
8.35
0.06
6.22
6.43
6.94
7.87
8.52
0.41
Table 3: MT-Bench scores for BoN and MBR decoding with various judge LLMs as utility metrics, along with the average score differences compared to greedy decoding across all models (denoted Avg. A). MBR decoding consistently outperforms BoN decoding across all comparablc utility metrics.
shot judges for MBR decoding (see Appendix D for our prompts). All chosen judges can act as both reference-free and reference-basee based judges, allowing us to compare MBR and BoN decoding fairly]
We find that MBR decoding consistently outperforms BoN decoding across all selected judge mod- els. This difference is especially large for the JudgeLM models and for Llama-3-706 hlama-3-7bbInstruee Instruutt where BoN fails to significantly improve on greedy decoding. One explanation for this discrepancy is that our LLM judges are insufficiently good at reference- free evaluation for BoN decoding to effec- tive. This idea is supported by prior studies comparing reference-free free and reference- based evalua- tion, which consistently show that reference- free methods tend to underperform, even when using strong judge models like GPT-4 (Ye et al., 20244] Kim et al., [024]] ZZheng et al.) 2023). Another explanation is that MBR decoding provides a smoothing effect that arises from our use of expected utility in place of utility point estimates for output selection, tying back to our hypothesis that select- ing "high-consensus" outputs yields significant benefit. This averaging process reduces the impact of individual mistakes made by the imperfect LLM judge, thereby providing for more stable and reliable measure of quality. We leave further exploration of these ideas to future work.
Notably, in Table 3 MBR performance improves by scaling the size of the LLM judge, with Prometheus-2-8x77 outperforning Prometheus-2-7B, JudgeI_M-33b outperforming Jud7eI.M-7b, and Llama-3-70b-Insse Instructt outperforming Llama-3-8b-Instrune This suggests that improving the LLM judge utility metric directly improves MBR decoding performance and that MBR decoding will benefit as newer and better LLM judges are developed.
MBR DISTILLATION
Our results so far demonstrate the potential of MBR decoding to significantly improve the test-time performance of instruction- -oollwiin-folooiingg models, but this comes at the cost of substantial inference-time compute costs due the linear cost for generating Neand candidate outputs and the quadratic cost for computing utility across these candidates. To mitigate this, we explore distilling MBR-decoded outputs back into the model itself and aim obtain MBR decoding-level (or better) performance without needing to perform MBR decoding at test time.
4.1 EXPERIMENTAL SETUP
4.1.1 TRAINING AN SFT MODEL
We start by performing SFT on the base Llama-2-7h and Llama-2-13h models. This is necessary to instil instruction-follooiing behaviour in the models so that they can be used to generate instruction- following self-training data. We choose not to use the official chat variants of these models as we
Because sequence classifier reward models (Stiennon et al. 2022) do not support reference- based evalua- tion, it is not possible to fairly compare BoN decoding with these methods to MBR We therefore do not discuss this in the main text and report our findings in Appendix (aneendixFFinteeee. Finstead.
7

---

Published as a conference paper at ICLR 2025
wish retain control over the training procedure and avoid inheriting any biases introduced through prior finetuning and alignment. We use 3000 random samples from UltraChat (Ding et al. (023) dataset created using GPT-3.5- Turbo. Each sample consists of multi-turn prompts and responses, although we only take the first turn of each sample in order to simplify experimentation. We designate our SFT models as sft.
for SFT. UltraChat is diverse conversational instruction-followii
4.1.2 ITERATIVE DPO ON
OUTPUTS
Having obtained SFT models, we now conduct DPO to improve the models on their own MBR- decoded outputs, an approach first proposed by Yang et al. (202) for improving machine transla- tion. We start by randomly drawing a further 3000 prompts from UltraChat (excluding the samples that have already been selected for SFT). Next, we generate Neand 12 candidate outputs from our sft models using these prompts. We use a smaller Neand than for MBR inference to balance per- formance and compute cost, as we know from Figure 2 that using an Neand value above 10 already yields significant gains. Following Yang et al. (2024), we then score the candidate outputs using utility metric and form preference pairs from the highest-scoring and lowest-scoring outputs. This preference pair dataset is then used for DPO on the sft models, yielding dpo-MBR-1. We extend upon Yang al. (2024) by iteratively repeating this process twice more, each time using the latest dpo models as the base model paired with a fresh set of 3000 prompts, yielding the models dpo- MBR-2 and dpo-MBR-3. See Appendix K for a summary of our SFT and DPO hyperparameters, Appendix G.5 for experimental results from using another preference pair selection strategy, and Appendix 1.2 for mathematical and algorithmic overviews of MBR distillation.
upon|Yang
4.1.3 UTILITY METRICS AND EVALUATION
We use Prometheus-2-77 2-BBB as our utility metric, although we also try MBR self-training with ROUGE and 1lama-3-8h-Instruc as the utility metrics (Appendix and We compare our dpo models with greedy decoding against the sft models with greedy decoding, beam search and MBR decoding. For MBR decoding, we use Neand = 30 and t 0.3 with Prometheus-2-1 as the utility metric. We also baseline against models trained with SFT on 12000 UltraChat samples (sft-full). Finally, experiment with BoN self-training, again using Prometheus-22 the same procedure for MBR self- seff-training, which yields the models dpo-BoN-J dpo-BoN-2 and dpo-BoN-3.
as the utility metric and following
We evaluate our trained models using greedy decoding AlpacaEval 2.0, once again reporting length-controlled win rates vs. GPT-4, and MT-Bench. As we only train our models to engage in single-turn conversations evaluation results in the Appendix, including head-to-head results between various self-trained mod- els and sft with greedy decoding (Appendix G.I). evaluation results on a selection of popular NLP benchmarks (Appendix G.2.. and human study results (Appendix H
evaluate only on the first turn of MT-Bench. We report additional
4.2 RESULTS
A AlpacaEval 2.0
MTT-Bench
AlpacaEval 2.0
MTT-Bench
sft- -MBR
5.52
5.48
7B
13B
7B
13B
2ft-2
6.75
5.43
sft Greedy
5.18
8.24
5.43
5.85
3ft-3MMBR
6.48
5.51
sft MBR
9.99
13.6
5.78
6.31
sft-full
6.35
9.40
5.55
6.26
Table (Left) AlpacaFval 2.0 win rates (%) and
dpo-I B-BoN
5.78
10.3
5.78
6,08
MT-Bench scores for models self-trained using DPO. After three rounds of training, the self-trained mod- els consistently outperform their BoN counterparts and SFT baselines. (Top) AlpacaEval 2.0 win rates (%) and MT Bench scores for models self-trained us- ing SFT. Self-training with SFT yields substantially worse results than self-training with DPO.
dpo-2-BoN
6.22
11.2
5.91
6.41
dpo-3-BoN
6.40
12.8
5.88
6.56
dpo-1-No 1-MBR
5.68
10.8
5.78
6.48
2-o-2-MBR
7.22
13.9
6.11
6.73
dpo-3-MBR
8.86
15.3
6.14
6.75
DPO self-training significantly improves model performance We report the results of our self- training experiment in the left subtable of Table 4 We find that three rounds of MBR self-training with DPO significantly improves model performance, with the 7B dpo-3-MBR model outperforming
8

---

Published as a conference paper at ICLR 2025
Win GPT-4
138
Argumentation
Recommendations
Creative writing
Technical eniting
Cading
Puzties logical reasoning
Factual
Mathematical reasaning
Hoerto
Other
Figure 4: Differences in AlpacaEval 2.0 win rates (%) between dpo-3 3- MBR models and their respective sft with greedy decoding baselines on different question categories. The largest improvements are seen in open-ended reasoning-focussee focussed tasks (c.g. mathematical reasoning and coding).
writing tasks, with less improvement on reasoning-
17.5
Arg. Genseation Time
bale vn Greeny
Avg. Decoding Time
57.5
BeN
HBR
52.5
<p0 MBR Groedy
5.0
2.5
0.0
40
100
120
Groady
BoN
MBR
Figure 5: (Left) AlpacaBval 2.0 win rate (%) vs. sft with greedy decoding against generation throughput. dpo-3-MBB with greedy decoding matches the performance of sft with MBR decoding, but with significantly higher throughput. (Right) Average generation and decoding times the AlpacaFval dataset. The decoding step in MBR decoding takes disproportionately long. This problem can mitigated through MBR self-training.
the 13B sft model with greedy decoding on both AlpacaEval 2.0 and MT-Bench. The improvements saturate by the third round of self- eeff-training as measured on MT-Bench (6.11 vs. 6.14 (7B) and 6.73 vs. 6.75 (13B) Table although there appears to room for further improvement on AlpacaEval 2.0 (7.22 vs. 8.86 (7B) and 13.9 vs. 15.3 (13B) in Table 4). Both dpo-3-MBR models outperform their sfi-full counterparts, which suggests that training on MBR-decoded outputs more beneficial than SFT larger split of UltraChat. The dpo-3- 3-MBR models also generally outperform sft with MBR decoding, and this is especially prominent for MT-Bench, which suggests that DPO on MBR- decoded outputs enables models to recover and then exceed their MBR-decoding performances. We find that DPO on BoN-decoded outputs also improves model performance, although less so than DPO with MBR-decoded outputs. We attribute this the relative strength of MBR decoding.
SFT self-training yields smaller gains than DPO self-training We experiment with iterative SFT self-training, using the 7B sft model. We document our results in the right subtable of Table We use the same sets of prompts as for DPO and select as our SFT labels the highest-scoring sample as determined by MBR, following Finkelstein et (2024). As before, we conduct three rounds of iterative training, yielding IMBR, sff-2-MBA and 3ft-3-MBR. We find that SFT training yields significantly less improvement than DPO. This indicates that MBR self-training benefits most from preference learning, where the model learns to contrast its highest- and lowest-quality outputs.
DPO self-trained model performance by question category We repeat the analysis on perfor- mance by question category for dpo-3-MBR in Figure 4 Self-training improves performance on almost all question categories, with generally larger improvement on writing- based categories and smaller improvement on reasoning- based categories. We attribute this difference to the writing- skewed distribution of question categories our UltraChat training data (see Appendix (..6)..
Analysis of compute costs We illustrate the savings on compute introduced by self-training in Figure We perform inference with various 7B models and decoding strategies on AlpacaEval 2.0, generation
using 2xA100 GPUs and vLLM (Kwon et al.] (203) batch size of 1, a LLM judge utility calculation batch size of 32, and Naand = 12. We find that MBR decoding imposes significant overhead, largely due the quadratic O(22 (aad)) cost incurred during
inference engine. We

---

Published as a conference paper at ICLR 2025
the utiliity calculation step. This overhead is removed through MBR self-training which nonetheless
retains performance gains. Note that dpo-3-MBR generates longer outputs than sff,, which explains why its average generation time seen in the right-hand plot of Figure ils higher.
RELATED WORK
MBR decoding MBR decoding has been explored in the context of machine translation using
variety of translation metries such as COMET (Rei et al. 2020) and BLEURT (Sellam et al. 20200) with promising results (Freitag et al. 2022 2023] Farinhas et al.) 2023 Stannjevvi & Sima an
2014). Prior works (Bertsch et al. 2023 Jinnai et al. 2023) also study MBR decoding for sum- marisation, using ROUGE and BERTScore as metries. Suzgun al. (2022) apply MBR decoding to several tasks, including summarisation, machine translation and three different JIG-Bench tasks (Srivastava et al. 2023). None of these works explore the use of MBR decoding in the more open- domain, nor do they consider using LLM judges as utility metrics.
ended custruction-followee
LLM judges Based on the strong cntructiionfollowingg capabilities of LLMs, recent works ex- plore prompting LLMs to judge responses from other LLMs (Li et al.) 2023) Zheng et al.) 2023). Follow-up works suggest that training on the evaluation traces of strong models may equip smaller models with strong evaluation capabilities (Kim al. 2023 2024 Zhu et al. 2023b Vu et al.) 2024
Wang et al., 2024b). These works focus on training LLMs to produce scoring decisions matching those of humans. In our work, instead of viewing evaluation evaluation capabilities of LLM judges as supervision to improve cnaructtio-followinn LLMs.
end goal, we explore utilising the
Inference-time algorithms Many inference-time algorithms generate candidate outputs and lect a final output based on external criteria. In addition to MBR and BoN decoding, examples in- clude Self-Consistence (Wang al.) 2023). which selects the most self-consistent answers through marginalisation of th-ougt reasoning paths and Universal Self- coff-sCnnstteccy (USC) (Chen) al. 2023), where the LLM used self-select consistent chain-of-thought reasoning paths from amongst many reasoning paths. Kuhn et al.] (2023) propose an MBR-esque algorithm that uses dense embedders and clustering to measure semantic uncertainty. Other inference- algorithms prompt the LLM to perform additional inference steps in structured manner. Examples include Tree-of-Thouuht Yaoo et al. 2023) and Graph-of-Thoughee (Besta et al. 2024), as well as recursive improvement strategies such as Self-Refine (Madaan et al. (023) and Retiexion (Shinn et al.. 20233.
Self-training Self-training a promising avenue for model improvement as it enables training with- out labelled data. Gulcehre et al. (2023) introduce algorithm that generates samples from a policy and then updates the policy using offline RL. Yuan et al. (2024b) train models to score their own outputs, and then use these scores to create preference datasets which for distillation. [Huang et al. (2022) train models on their own coabbsst-oonidenne outputs as determined by majority voting. Self- training on MBR-dccoded outputs has also been explored for machine translation. Fiikelsteinn et al. 2024) train models with SFT on their own MBR and quality estimation outputs for machine trans- lation and demonstrate that this yields improvements over baseline models. WWan al. (2024a) use MBR to generate targets for sequence-level distillation, again for machine translation. Yang et al. 2024 are the first to use DPO upweight the model's own MBR generations, allowing them to recover much of their original MBR performances on translation using only greedy decoding.
6 CONCLUSION
In this work, we investigate using LLM judges to supervise other LLMs on instruction-follloing tasks through MBR decoding, and find that this yields significant and consistent improvements to model performance relative to greedy decoding, beam search and BoN decoding. These benefits persist across wide range of question categories and are also consistent across models of various sizes, demonstrating that small L.LM judges can be used to improve much larger L.MMs at inference time. To mitigate the significant inference-time costs associated with MBR decoding, we also ex- plore iterative self-training MBR- outputs. We find that MBR self-training using DPO, but not SFT, enables models to recover and even exceed their base MBR decoding performance us- ing only greedy decoding. We hope our work further highlights the potential of using LLM judges for supervision and inspires future research into MBR decoding beyond its traditional domains and applications, particularly through the development of new utility metrics.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
In an effort to make our work reproducible, we document all prompts (Appendices and B as well as training and inference hyperparameters (Appendix K) used throughout our experiments. We also include version information for all API-based LLMs (Appendix [B), and choose to use open-source models (the Llama-2, Llama-3, Prometheus-2 and JudgeLM families) where possible.
ACKNOWLEDGEMENTS
AB was supported by a grant from the National Science Foundation Graduate Research Fellow- ship Program under Grant No. DGE2140739. PF was supported by the Portuguese Recovery and Resilience Plan through project C64500882200000055 (Center for Responsible Al) and by FCT/MECI through national funds and when applicable co-funded EU funds under UID/50008: Instituto de Telecomunicações. Any opinions, findings, and conclusions or recommendations ex- pressed this material are those of the author(s) and do not necessarily reflect the views of the sponsors.
REFERENCES
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agiree SemEval-2012 task pilot on semantic textual similarity. In Eneko Agirre, Johan Bos, Mona Diab, Suresh Manandhar, Yuval Marton, and Deniz Yuret (eds.). *SEM 2012: The First Joint Conference on Lexical and Com- putational Semantics Volume 1: Proceedings of the main conference and the shared task, and Volume Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012), pp. 385-393, Montréal, Canada, 7-8 June 2012. Association for Computational Linguis- tics. URL https::pp//..mmm hatttps:/aclaathhl..g... /ww.aaaaaaatooooooe.ooaaaa orq/ S12-1051
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The long- don-mdoumeett transformer, 2020. htllattps://aaxtt. wttp://....r//ba//....000000
Amanda Bertsch, Alex Xie, Graham Neubig, and Matthew R. Gormley. It's mbr all the way down: Modern generation techniques through the lens of minimum bayes risk, 2023.
Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gi- aninazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of thoughts: Solving elaborate problems with large language models. Pro- ceedings of the AAAI Conference on Artificial Intelligence, 38( doi: 10. 10.1609/aaii ww8.1....881....... URL httth:/t://add doi../x...o.i...o http://xxxxcci..r/1///// 1609/
March 2024.
ISSN 2159-5399.
aaai. v38 29720
P.J. Bickel and K.A. Doksum, Mathematical Statistics: Basic Ideas and Selected Topics. Prentice Hall, 1977. ISBN 9780135641477..
gowgle. com cocogleee.cm.ss/bbooo..........
UCMEAΩAAIAAJ
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation, 2023.
Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, and Quanquan Gu. Self-play fine-tuning converts weak language models to strong language models, 2024. URL https://arxix.
org/abs/2400 /401, 01335
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pel- lat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. Scaling instruction finetuned language models, 2022. URL ptt.tta://ariiiee www.xxi..oor.aaoo abs 0. 11416

---

Published as a conference paper at ICLR 2025
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge, 2018. URLlttts:: http::::///axxv..ooo wwg. hatp//aaa///1..55553
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations, 2023.
wrwiv. org/ abs/2305. 14233
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang. Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Roziere, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits. David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor I Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee,, Georgia Lewis Ander- Son, Graeme Nail. Gregoire Mialon, Guan Pang. Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, liyan Zarov, Imanol Arrieta Ibarra, Isabel Kloumann, Ishan Mísra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu. Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, Khalid El-Arini, Krithika Iyer, Kshitiz Malik, Kuenley Chiu, Kunal Bhalla, Lauren Rantala- Laurens van der Maaten, Lawrence Chen, Liang Tan, Liz Jenkins, Louis Martin, Lovish Madaan, Lubo Malo, Lukas Blecher, Lukas Landzaat, Luke de Oliveira, Madeline Muzzi, Mahesh Pasupuleti, Man- nat Singh, Manohar Paluri, Marcin Kardas, Mathew Oldham, Mathieu Rita, Maya Pavlova, Melanie Kambadur, Mike Lewis, Min Si, Mitesh Kumar Singh, Mona Hassan, Naman Goyal, Narjes Torabi, Nikolay Bashlykov, Nikolay Bogoychev, Niladri Chatterji, Olivier Duchenne, Onur Çelebi, Patrick Alrassy, Pengchuan Zhang, Pengwei Li, Petar Vasic, Peter Weng, Prajjwal Bhar- gava, Pratik Dubal, Praveen Krishnan, Punit Singh Koura, Puxin Xu, Qing He, Qingxiao Dong, Ragavan Srinivasan, Raj Ganapathy. Ramon Calderer, Ricardo Silveira Cabral, Robert Stojnic, Roberta Raileanu, Rohit Girdhar, Rohit Patel, Romain Sauvestre, Ronnie Polidoro, Roshan Sum- baly, Ross Taylor, Ruan Silva, Rui Hou, Rui Wang. Saghar Hosseini, Sahana Chennabasappa, Sanjay Singh, Sean Bell, Seohyun Sonia Kim, Sergey Edunov, Shaoliang Nie, Sharan Narang, Sharath Raparthy, Sheng Shen, Shengye Wan, Shruti Bhosale, Shun Zhang, Simon Vandenhende, Soumya Batra, Spencer Whitman, Sten Sootla, Stephane Collot, Suchin Gururangan, Sydney Borodinsky, Tamar Herman, Tara Fowler, Tarek Sheasha, Thomas Georgiou, Thomas Scialom, Tobias Speckbacher, Todor Mihaylov, Tong Xiao, Ujjwal Kam, Vedanuj Goswami. Vibhor Gupta, Vignesh Ramanathan, Viktor Kerkez, Vincent Gonguet, Virginie Do. Vish Vogeti, Vladan Petro- vic, Weiwei Chu, Wenhan Xiong, Wenyin Fu, Whitney Meers, Xavier Martinet, Xiaodong Wang, Xiaoqing Ellen Tan, Xinfeng Xie, Xuchao Jia, Xuewei Wang, Yaclle Goldschlag. Yashesh Gaur, Yasmine Babaei, Yi Wen, Yiwen Song, Yuchen Zhang, Yue Li, Yuning Mao, Zacharie Delpierre Coudert, Zheng Yan, Zhengxing Chen, Zoe Papakipos, Aaditya Singh, Aaron Grattafiori, Abha Jain, Adam Kelsey, Adam Shajnfeld, Adithya Gangidi, Adolfo Victoria, Ahuva Goldstand, Ajay Menon, Ajay Sharma, Alex Boesenberg, Alex Vaughan, Alexei Baevski, Allie Feinstein, Amanda Kallet, Amit Sangani, Anam Yunus, Andrei Lupu, Andres Alvarado, Andrew Caples, Andrew Gu, Andrew Ho, Andrew Poulton, Andrew Ryan, Ankit Ramchandani, Annie Franco, Aparajita Saraf. Arkabandhu Chowdhury, Ashley Gabriel, Ashwin Bharambe, Assaf Eisenman, Azadeh Yazdan, Beau James, Ben Maurer, Benjamin Leonbardi, Bernie Huang, Beth Loyd, Beto De Paola, Bhargavi Paranjape, Bing Liu, Bo Wu, Boyu Ni, Braden Hancock, Bram Wasti, Bran- don Spence, Brani Stojkovic, Brian Gamido, Britt Montalvo, Carl Parker, Carly Burton, Catalina Mejia, Changhan Wang, Changkyu Kim, Chao Zhou, Chester Hu, Ching-Hsiang Chu, Chris Cai, Chris Tindal, Christoph Feichtenhofer, Damon Civin, Dana Beaty, Daniel Kreymer, Daniel Li, Danny Wyatt, David Adkins, David Xu, Davide Testuggine, Delia David, Devi Parikh, Diana Liskovich, Didem Foss, Dingkang Wang, Due Le, Dustin Holland, Edward Dowling, Eissa Jamil, Elaine Montgomery, Eleonora Presani. Emily Hahn, Emily Wood, Erik Brinkman, Esteban Ar- caute, Evan Dunbar, Evan Smothers, Fei Sun, Felix Kreuk, Feng Tian, Firat Ozgenel, Francesco
12

---

Published as a conference paper at ICLR 2025
Caggioni, Francisco Guzmán, Frank Kanayet, Frank Seide, Gabriela Medina Florez, Gabriella Schwarz, Gada Badeer, Georgia Swee, Gil Halpern, Govind Thattai, Grant Herman, Grigory Sizov, Guangyi, Zhang, Guna Lakshminarayanan, Hamid Shojanazeri, Han Zou, Hannah Wang, Hanwen Zha, Haroun Habeeb, Harrison Rudolph, Helen Suk, Henry Aspegren, Hunter Gold- man, Ibrahim Damlaj, Igor Molybog, Igor Tufanov, Irina-Flena Veliche, Itai Gat, Jake Weissman, James Geboski, James Kohli, Japhet Asher, Jean-Baptiste Gaya, Jeff Marcus, Jeff Tang, Jennifer Chan, Jenny Zhen, Jeremy Reizenstein, Jeremy Teboul, Jessica Zhong, Jian Jin, Jingyi Yang. Joe Cummings, Jon Carvill, Jon Shepard, Jonathan McPhie, Jonathan Torres. Josh Ginsburg, Junjie Wang, Kai Wu, Kam Hou U. Karan Saxena, Karthik Prasad, Kartikay Khandelwal, Katayoun Zand, Kathy Matosich, Kaushik Veeraraghavan, Kelly Michelena, Keqian Li, Kun Huang, Kunal Chawla, Kushal Lakhotia, Kyle Huang, Lailin Chen, Lakshya Garg. Lavender A, Leandro Silva, Lee Bell, Lei Zhang, Liangpeng Guo, Licheng Yu, Liron Moshkovich, Luca Wehrstedt, Madian Khabsa, Manav Avalani, Manish Bhatt, Maria Tsimpoukelll, Martynas Mankus, Matan Hasson, Matthew Lennie, Matthias Reso, Maxim Groshev, Maxim Naumov, Maya Lathi, Meghan Ke- neally, Michael L. Seltzer, Michal Valko, Michelle Restrepo. Mihir Patel, Mik Vyatskov, Mikayel Samvelyan, Mike Clark, Mike Macey, Mike Wang, Miquel Jubert Hermoso, Mo Metanat, Mo- hammad Rastegari. Munish Bansal, Nandhini Santhanam, Natascha Parks, Natasha White, Navy- ata Bawa, Nayan Singhal, Nick Egebo, Nicolas Usunier, Nikolay Pavlovich Laptev, Ning Dong. Ning Zhang, Norman Cheng, Oleg Chernoguz, Olivia Hart, Omkar Salpekar, Ozlem Kalinli, Parkin Kent, Parth Parekh. Paul Suab, Pavan Balaji, Pedro Rittner, Philip Bontrager, Pierre Roux, Piotr Dollar, Polina Zvyagina, Prashant Ratanchandani. Pritish Yuvraj. Qian Liang, Rachad Alao, Rachel Rodriguez, Rañ Ayub, Raghotham Murthy, Raghu Nayani, Rahul Mitra, Raymond Li, Rebekkah Hogan, Robin Battey, Rocky Wang, Rohan Maheswari, Russ Howes, Ruty Rinott, Sai Jayesh Bondu, Samyak Datta, Sara Chugh, Sara Hunt, Sargun Dhillon, Sasha Sidorov, Sa- tadru Pan, Saurabh Verma, Seiji Yamamoto, Sharadh Ramaswamy, Shaun Lindsay, Shaun Lind- say, Sheng Feng, Shenghao Lin, Shengxin Cindy Zha, Shiva Shankar, Shuqiang Zhang, Shuqiang Zhang, Sinong Wang, Sneha Agarwal, Soji Sajuyigbe, Soumith Chintala, Stephanie Max. Stephen Chen, Steve Kehoe, Steve Satterfield, Sudarshan Govindaprasad, Sumit Gupta, Sungmin Cho, Sunny Virk, Suraj Subramanian, Sy Choudhury, Sydney Goldman, Tal Remez, Tamar Glaser, Tamara Best, Thilo Kohler, Thomas Robinson, Tianhe L.i, Tianjun Zhang, Tim Matthews, Tim- othy Chou, Tzook Shaked, Varun Vontimitta, Victoria Ajayi, Victoria Montanez, Vijai Mohan, Vinay Satish Kumar, Vishal Mangla, Vitor Albiero, Vlad Ionescu, Vlad Poenaru, Vlad Tiberiu Mihailescu, Vladimir Ivanov, Wei Li, Wenchen Wang. Wenwen Jiang, Wes Bouaziz, Will Con- stable, Xiaocheng Tang, Xiaofang Wang, Xiaojian Wu, Xiaolan Wang, Xide Xia, Xilun Wu, Xinbo Gao, Yanjun Chen, Ye Hu, Ye Jia, Ye Qi, Yenda Li, Yilin Zhang, Ying Zhang, Yossi Adi, Youngjin Nam, Yu, Wang, Yuchen Hao, Yundi Qian, Yuzi He, Zach Rait, Zachary DeVito, Zef Rosnbrick, Zhaoduo Wen, Zhenyu Yang, and Zhiwei Zhao. The llama 3 herd of models, 2024.
URL
org/ abs /2407 21783
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Longth-comtrolled al- pacaeval: A simple way to debias automatic evaluators. arXiv preprint arXiv:2404.04445, 2024a.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B. Hashimoto. Length-controlled alpacaeval: A simple way to debias automatic evaluators, 2024b. URL https://arxiv.
hogg/ab/2400 04475
Bryan Eikema and Wilker Aziz. Is map decoding all you need? the inadequacy of the mode in neural machine translation, 2020.
Angela Fan, Mike Lewis, and Yann Dauphin. Hierarchical neural story generation, 2018 URL https:://arxis org/ abs 2os5/805.. 04833
António Farinhas, José de Souza, and Andre Martins. An empirical study of translation hypoth- esis ensembling with large language models. Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 11956-11970, Singapore, December 2023, Association for Computational Linguis- tics. doi: 10. 1/216653//1222..mmml. wtt.:023..mmlpmmam.... emmnppmma in. 733
URL htthttt:: seattpss//aclaanheloyy org/2023.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Ma-
13

---

Published as a conference paper at ICLR 2025
rine Carpuat, Marie-Catherine Catherine de Mamnefe, and Ivan Vladimir Meza Ruiz (eds.), Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 1396-1412 Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18.33v naa02nnaacclmaan 100. URI.
https::/acclattho wtt.aaatoooooooooooo222222.2
wrg/ naac ma 100
Mara Finkelstein, Subhajit Naskar, Mehdi Mirzazadeh, Apurva Shah, and Markus Freitag. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods, 2024.
URL http:::://aaxxv.ort htsp/bbb/////// 2001.
Rudolf Flesch. A new readability yardstick. Journal of Applied Psychology, 32(3):221-255 2333, 1948. doi: 10.1037/h0055532..
Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics,, 10:811-825, 2022. doi: 10.1162/tacl a 00491. URL https:
/ //aclarnchlllaa..org//2222 ttt.
Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. Epsilon sampling rocks: Investigating sampling strategies for minimum bayes risk decoding for machine translation, 2023.
Alex Graves. Sequence transduction with recurrent neural networks, 2012.
URL
https:
3711
Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, and Nando de Freitas. Reinforced self-training (rest) for language modeling, 2023. https:/ts:///att /tttps://arxivi http://rwwwv..ogabb/ba/aaaaaaaa/
abs 008.23088888888888
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Ja- cob Steinhardt. Measuring massive multitask language understanding, 2021. URL https:
//arxiv. wttpv///rrgga/////00////// 03300
Ari Holtzman, Jan Buys. Li Du, Maxwell Forbes, and Yejin Choi. The curious case neural text degeneration. 2020. URL https:::: http://axvvvvo/////////... org/ abs/1904.09755
Jiaxin Huang, Shixiang Shane Gu, Le Hou, Yuexin Wu, Xuezhi Wang, Hongkun Yu, and Jiawei Han. Large language models can self-improve, 2022. URL attps://arxiis org/ /
2210.11610
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William Sayed. Mistral 7b, 2023. URL hctps:
Yuu Jinnai, Tetsuro Morimura, Ukyo Honda, Kaito Ariu, and Kenshi Abe. Model-based minimum bayes risk decoding. 2023.
Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thome, et al. Prometheus: Inducing fine-grained evalua- tion capability in language models. The Twelfth International Conference on Learning Repre- sentations, 2023.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An open source language model specialized in evaluating other language models, 2024.
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. arXiv preprint arXiv:2302.09666 09664, 2023.
14

---

Published as a conference paper at ICLR 2025
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. Rewardbench: Evaluating reward models for language modeling, 2024. URL
Chankyu Lee, Rajarshi Roy, Mengyao Xu, Jonathan Raiman, Mohammad Shoeybi. Bryan Catan- zaro, and Wei Ping. Nv-embed: Improved techniques for training Ilms as generalist embedding models, 2024. htt.hhttss//aaatt org. http:///b//2400//0 2405. 17428
Yaniv Leviathan, Matan Kalman, and Yossi Matias. inference from transformers via speculative decoding, 2023.
http:///aaxvv..oo/aa////2211/ 17199
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. Alpacaeval: An automatic evaluator of instruction-followinn models. htttps:/ http://///.tuuuc.ooooaaaaaaaaaaa
aaw.aa/appaaaaaaaaaa _vall 5 2023.
Shin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004a. Association for Computational Linguis- tics. URL htttts::/aclaaatolloogooooo////e
crg. 001//04-10000
Chin-Yews Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches oul, pp. 74-81, 2004b.
Stephanie Lin, Jacob Hilton, and Owain Evans, Truthfulqa: Measuring how models mimic human falsehoods, 2022. /tts:ttt:://// /atp:i///axvvv//a wrt. /tt.b////199 07958
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback, 2023. URL https: /ettps//arxiv..oo httpaaaiv.v.gaaaaaaaa/.. abs/
2303 17651
Rui Meng, Ye Liu, Shafiq Rayhan Joty, Caiming Xiong, Yingbo Zhou, and Semih Yavuz. segembbbddiing-missaaeeenaa text retrieval with transfer learning. Salesforce AI Research Blog, 2024.
URL https:: sa waalsforrceall renceeccrreeerchcc com/
lfg-enbeedee--nisttaa..
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng- -in Yong. Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafcai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. 2023. 01786)
URL
Zach Nussbaum, John X. Morris, Brandon Duderstadt, and Andriy Mulyar. Nomic embed: Training a reproducible long context text embedder, 2024. URL htt.tp:://aaxxi..og//bbbaa
01613
OpenAl, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman. Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Moham- mad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapirg Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brock- man, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila
15

---

Published as a conference paper at ICLR 2025
Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford. Leo Gao, Elie Georges, Christian Gib- son, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-opess Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hal- lacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin. Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Ka- mali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, tukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade I Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Matcusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Míra Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Ra- jeev Nayak, Arvind Neelakantan, Richard Ngo. Hyeonwoo Noh, Long Ouyang, Cullen O' Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Sel- sam, Kyla Sheppard, Toki Sherbakov, Jessica Shich. Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddic Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Pre- ston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vi- jayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Work- man, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL
https: //axivv www.org///.....
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. T5score: Discriminative fine-tuning of generative evaluation metrics. In Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 15185-15202, 2023.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model secretly a reward model, 2024. URL
/205 18290
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Comet: A neural framework for mt evaluation, 2020.
Brian Richards. Type/token ratios: What do they really tell us? Journal of Child Language, 14(2): 201-209, 1987. doi: 10. 10.0017/530050090012855.
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. Bleurt: Learning robust metrics for text generation, 2020. /
04696
16

---

Published as a conference paper at ICLR 2025
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL
https:
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling 1lm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXivv2408..03314 2024.
Yifan Song, Guoyin Wang, Sujian Li, and Bill Yuchen Lin. The good, the bad, and the greedy: Evaluation of Ilms should not ignore non-determinism, 2024. URL httts:://axxi...
abs/2a0 2407. 10457
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowyez, Akshat Agarwal, Alethea Power, Alex Ray. Alex Warstadt, Alexander W. Kocurek, Ali Safaya, Ali Tazarv, Alice Xiang, Alicia Parrish, Allen Nie, Aman Hussain, Amanda Askell, Amanda Dsouza, Ambrose Slone, Ameet Rahane, Anantharaman S. Iyer, An- ders Andreassen, Andrea Madotto, Andrea Santilli, Andreas Stuhimüller, Andrew Dai, An- drew La, Andrew Lampinen, Andy Zou, Angela Jiang, Angelica Chen, Anh Vuong, Animesh Gupta, Anna Gottardi, Antonio Norelli, Anu Venkatesh, Arash Gholamidavoodi, Arfa Tabas- sum, Arul Menezes, Arun Kirubarajan, Asher Mullokandov, Ashish Sabharwal, Austin Her- rick, Avia Efrat, Aykut Erdem, Ayla Karakaş, B. Ryan Roberts, Bao Sheng Loe, Barret Zoph, Barthomiej Bojanowski, Batuhan Özyurt, Behnam Hedayatnia, Behnam Neyshabur, Benjamin Inden, Benno Stein, Berk Ekmekci, Bill Yuchen Lin, Blake Howald, Bryan Orinion, Cameron Diao, Cameron Dour, Catherine Stinson, Cedrick Argueta, César Ferri Ramírez, Chandan Singh, Charles Rathkopf, Chenlin Meng, Chitta Baral, Chiyu Wu, Chris Callison-Burch, Chris Waites, Christian Voigt, Christopher D. Manning, Christopher Potts, Cindy Ramirez, Clara E. Rivera, Clemencia Siro. Colin Raffel, Courtney Ashcraft, Cristina Garbacea, Damien Sileo. Dan Gar- rette, Dan Hendrycks, Dan Kilman, Dan Roth. Daniel Freeman, Daniel Khashabi, Daniel Levy, Daniel Moseguí González, Danielle Perszyk, Danny Hernandez. Danqi Chen, Daphne Ippolito, Dar Gilboa, David Dohan, David Drakard, David Jurgens, Debajyoti Datta, Deep Ganguli, De- nis Emelin, Denis Kleyko, Deniz Yuret, Derek Chen, Derek Tam, Dieuwke Hupkes, Diganta Misra, Dilyar Buzan, Dimitri Coelho Mollo. Diyi Yang, Dong-Ho Lee, Dylan Schrader, Eka- terina Shutova, Ekin Dogus Cubuk, Elad Segal. Eleanor Hagerman, Elizabeth Barnes, Eliza- beth Donoway, Ellie Pavlick, Emanuele Rodola, Emma Lam, Eric Chu, Eric Tang, Erkut Erdem, Ernie Chang, Ethan A. Chi, Ethan Dyer, Ethan Jerzak, Ethan Kim, Funice Engefu Manyasi, Ev- genii Zheltonozhskii, Fanyue Xia, Fatemeh Siar, Fernando Martinez-Plumed, Francesca Happé, Francois Chollet, Frieda Rong, Gaurav Mishra, Genta Indra Winata, Gerard de Melo, Germán Kruszewski, Giambattista Parascandolo, Giorgio Mariani, Gloria Wang, Gonzalo Jaimovitch- López. Gregor Betz, Guy Gur-Ari, Hana Galijasevic, Hannah Kim, Hannah Rashkin, Hannaneh Hajishirzi, Harsh Mehta, Hayden Bogar, Henry Shevlin, Hinrich Schütze, Hiromu Yakura, Hong- ming Zhang. Hugh Mee Wong, Ian Ng. Isaac Noble, Jaap Jumelet, Jack Geissinger, Jackson Kernion, Jacob Hilton, Jaehoon Lee, Jaime Fernández Fisac, James B. Simon, James Koppel, James Zheng, James Zou, Jan Kocoń, Jana Thompson, Janelle Wingfield, Jared Kaplan, Jarema Radom, Jascha Sohl-Dickstein, Jason Phang. Jason Wei, Jason Yosinski, Jekaterina Novikova, Jelle Bosscher, Jennifer Marsh, Jeremy Kim, Jeroen Taal, Jesse Engel, Jesujoba Alabi, Ji- acheng Xu, Jiaming Song, Jillian Tang, Joan Waweru, John Burden, John Miller, John U. Balis, Jonathan Batchelder, Jonathan Berant, Jörg Frohberg, Jos Rozen, Jose Mrnnandez-Orallloo Joseph Boudeman, Joseph Guerr, Joseph Jones, Joshua B. Tenenbaum, Joshua S. Rule, Joyce Chua, Kamil Kanclerz, Karen Livescu, Karl Krauth, Karthik Gopalakrishnan, Katerina Ignatyeva, Katja Markert, Kaustubh D. Dhole, Kevin Gimpel, Kevin Omondi, Kory Mathewson, Kristen Chia- fullo, Ksenia Shkaruta, Kumar Shridhar, Kyle McDonell, Kyle Richardson. Laria Reynolds, Leo Gao, Li Zhang, Liam Dugan, Lianhui Qin. Lidia Contreras-Ochandee Luca Moschella, Lucas Lam, Lucy Noble, Ludwig Schmidt, Luheng He, Luis Oliveros Colón, Luke Metz, Lütfi Kerem Şenel, Maarten Bosma, Maarten Sap, Maartje ter Hoeve, Maheen Fa- rooqi, Manaal Faruqui, Mantas Mazeika, Marco Baturan, Marco Marelli, Marco Maru, Maria Jose Ramírez Quintana, Marie Tolkiehn, Mario Giulianelli, Martha Lewis, Martin Potthast, Matthew L. Leavitt, Matthias Hagen, Mátyás Schubert, Medina Orduma Baitemirova, Melody Arnaud, Melvin McElrath, Michael A. Yee, Michael Cohen, Michael Gu, Michael Ivanitskiy, Michael Starritt, Michael Strube, Michal Swędrowski, Michele Bevilacqua, Michihiro Yasunaga Mihir Kale, Mike Cain, Mimee Xu, Mirac Suzgun, Mitch Walker, Mo Tiwari, Mohit Bansal,
Louis-Philippe Morency,
17

---

Published as a conference paper at ICLR 2025
Moin Aminnaseri, Mor Geva, Mozhdeh Gheini, Mukund Varma T, Nanyun Peng, Nathan A. Chi, Nayeon Lee, Neta Gur-Ari Krakover, Nicholas Cameron, Nicholas Roberts. Nick Doiron, Nicole Martinez, Nikita Nangia, Niklas Deckers, Niklas Muennighoff. Nitish Shirish Keskar, Niveditha S. lyer, Noah Constant, Noah Fiedel, Nuan Wen, Oliver Zhang, Omar Agha, Omar El- baghdadi, Omer Levy, Owain Evans, Pablo Antonio Moreno Casares, Parth Doshi, Pascale Fung. Paul Pu Liang, Paul Vicol, Pegah Alipoormolabashi, Peiyuan Liao, Percy Liang, Peter Chang, Pe- ter Eckersley, Phu Mon Htut, Pinyu Hwang, Piotr Milkowski, Piyush Patil, Pouya Pezeshkpour, Priti Oli, Qiaozhu Mei, Qing Lyu, Qinlang Chen, Rabin Banjade, Rachel Etta Rudolph, Raefer Gabriel, Rahel Habacker, Ramon Risco, Raphaël Millière, Rhythm Garg, Richard Barnes, Rif A. Saurous, Riku Arakawa, Robbe Raymaekers, Robert Frank, Rohan Sikand, Roman Novak, Ro- man Sitelew, Ronan LeBras, Rosanne Liu, Rowan Jacobs, Rui Zhang. Ruslan Salakhutdinov, Ryan Chi, Ryan Lee, Ryan Stovall, Ryan Teehan, Rylan Yang. Sahib Singh, Saif M. Moham- mad, Sajant Anand, Sam Dillavou, Sam Shleifer, Sam Wiseman, Samuel Gruetter, Samuel R. Bowman, Samuel S. Schoenholz, Sanghyun Han, Sanjeev Kwatra, Sarah A. Rous, Sarik Ghaz- arian, Sayan Ghosh, Sean Casey, Sebastian Bischoff, Sebastian Gehrmann, Sebastian Schus- ter, Sepideh Sadeghi, Shadi Hamdan, Sharon Zhou, Shashank Srivastava, Sherry Shi, Shikhar Singh, Shima Asaadi, Shixiang Shane Gu, Shubh Pachchigar, Shubham Toshniwal, Shyam Upad- hyay, Shyamolima, Debnath, Siamak Shakeri, Simon Thormeyer, Simone Melzi, Siva Reddy, Sneha Priscilla Makini, Soo-Hwan Lee, Spencer Torene, Sriharsha Hatwar, Stanislas Dehaene, Stefan Divic, Stefano Ermon, Stella Biderman. Stephanie Lin, Stephen Prasad, Steven T. Pianta- dosi, Stuart M. Shieber. Summer Misherghi, Svetlana Kiritchenko. Swaroop Mishra, Tal Linzen, Tal Schuster, Tao Li, Tao Yu, Tariq Ali, Tatsu Hashimoto, Te-Lin Wu, Théo Deshordes, Theodore Rothschild, Thomas Phan, Tianle Wang, Tiberius Nkinyili, Timo Schick, Timofei Kornev, Ti- tus Tunduny, Tobias Gerstenberg, Trenton Chang, Trishala Neeraj. Tushar Khot, Tyler Shultz, Uri Shaham, Vedant Misra, Vera Demberg, Victoria Nyamai, Vikas Raunak, Vinay Ramasesh, Vinay Uday Prabhu, Vishakh Padmakumar, Vivek Srikumar, William Fedus, William Saun- ders, William Zhang, Wout Vossen, Xiang Ren, Xiaoyu Tong. Xinran Zhao, Xinyi Wu, Xudong Shen, Yadollah Yaghoobzadeh, Yair Lakretz, Yangqiu Song, Yasaman Bahri, Yejin Choi, Yichi Yang. Yiding Hao, Yifu Chen, Yonatan Belinkov, Yu Hou, Yufang Hou, Yuntao Bai, Zachary Seid, Zhuoye Zhao, Zijian Wang, Zijie J. Wang, Zirui Wang, and Ziyi Wu. Beyond the im- itation game: Quantifying and extrapolating the capabilities of language models, 2023. URL hittps://arxx wwggabs.////ooooooooot....t..a.r abs 6206. 04615
Miloš Stanojević and Khalil Sima'an an. Fitting sentence level translation evaluation with many dense features. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 202- 206. Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v1/ 1025
Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. Learning to summarize from human feedback, 2022. URL
https: wttp///aaivvvvv/am///a/ org. wwtpaass...... 01325
Mirac Suzgun, Luke Melas Myriazi,rizzi, and Dan Jurafsky. Follow the wisdom of the crowd: Effective text generation via minimum bayes risk decoding, 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models, 2023a. URL httpt::/t////..t.. htt.///axiv../ wttpg//////.......e///// 13077
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao. Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nic, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh
18

---

Published as a conference paper at ICLR 2025
Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine-tuned chat models, 2023b. URL
2230/ 09288
Tu Vu, Kalpesh Krishna, Salaheddin Alzubi, Chris Tar, Manaal Faruqui, and Yun-Hsuan Sung- Foundational autoraters: Taming large language models for better automatic evaluation. arXiv preprint arXiv:2407. 10817, 2024.
Jun Wang. Eleftheria Briakou, Hamid Dadkhahi, Rishabh Agarwal, Colin Cherry, and Trevor Cohn. Don't throw away data: Better sequence knowledge distillation, 2024a. URL https:
10456
Peifeng Wang, Austin Xu, Yilun Zhou, Caiming Xiong, and Shafiq Joty. Direct judgement prefer- ence optimization. arXiv preprint (rXXiv2409.14666 2024b.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, Sharan Narang, Aakanksha Chowdh- ery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models, 2023.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2022. URL
https://arxizz http:///aaxxv.oo/a/////////
01652
Sean Welleck, Amanda Bertsch, Matthew Finlayson, Hailey Schoelkopf, Alex Xie, Graham Neubig, Ilia Kulikov, and Zaid Harchaoui. From decoding to meta-generaaiion: Inference-time algorithms for large language models, 2024. URL http::///wwwivv...aaaaa//aa/////
16838
Tianhao Wu, Weizhe Yuan, Olga Golovneva, Jing Xu, Yuandong Tian, Jiantao Jiao, Jason Weston, and Sainbayar Sukhbaatar. Meta-iewarding language models: Self-improving alignment with ilm-as-a-mes a-taa------aajuggee arXiv preprint arXiv:2407. 29..24997..9944 2024.
Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, and Jason Weston. Some things are more cringe than others: Preference optimization with the pairwise cringe loss. arXiv preprint arXiv:2312.16682. 2023.
Guangyu Yang. Jinghong Chen, Weizhe Lin, and Bill Byrne. Direct preference optimization for neural machine translation with minimum bayes risk decoding, 2024. URL httpp:///aa:
org/abs/ 2211. 08380
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023.
URLttsts:://axt http://xxii..orraa//333330 wttgabs///.....a// 10601
Seonghyeon Ye, Doyoung Kim, Sungdong Kim, Hyeonbin Hwang, Seungone Kim, Yongrae Jo, James Thorne, Juho Kim, and Minjoon Seo. Flask: Fine-grained language model evaluation based on alignment skill sets, 2024. URL
2307. 10928
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text as text gener- ation. Advances in Neural Information Processing Systems, 34:27263-27277, 2021.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024a.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models, 2024b. URL
abs/2401 10020
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence?, 2019. URL UNL/tttp:://arrllooo////bb wttpo////////.00.0//3333
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. Bertscore: Evaluat- ing text generation with bert. In International Conference Learning Representations.
19

---

Published as a conference paper at ICLR 2025
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing. Hao Zhang. Joseph Gonzalez, and lon Stoica. Judging -m-as---judge with mt-bench and chatbot arena, 2023.
Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao. Starling-7b: Improving Ilm helpfulness & harmlessness with rlaif, November 2023a.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: Fine-tuned large language models are scalable judges. 2023b.
Lianghui Zhu, Xinggang Wang, and Xinlong Wang. Judgelm: fune-tuned large language models scalable judges, 2023c. htLhttps: http::/t: //tp:///w...r///////////// 17631
20

---

Published as a conference paper at ICLR 2025
A
MBR DECODING: ADDITIONAL RESULTS
This section presents additional results from experiments exploring the use of MBR decoding to improve test-time performance.
A.1 ADDITIONAL MBR UTILITY METRICS
Llama-2-7B
Llama-2-70B
Avg. A
Greedy
14.4
22.8
0
ROUGE-1 MBR
16.2
24.7
1.85
SFR-Embedder MBR
12.1
22.2
-1.45
Prometheus MBR
17.7
26.2
3.35
ROUGE-2 MBR
16.6
24.6
2.00
ROUGE-L MBR
15.5
24.7
1.50
NVEmbed-Embedlee MBR
14.1
22.1
-0.50
Nomic-Ember MBR
16.3
1.60
Table 5: AlpacaEval 2.0 win rates (%) for additional MBR decoding experiments, along with the average win differences compared greedy decoding across models (denoted Avg. A).
We experiment with two additional ROUGE variants and two additional dense embedders utility metries. The two ROUGE variants are ROUGE-2 and ROUGE-L, which detect bigram overlap and longest co-oceurring n-gram overlap respectively. The two dense embedders NVEmbed- v2 (Lee et al.] (2024) and Nomic-Text-v1.5 (Nussbaum et al., (024), which, like SFR Embedder, strong, long-context dense embedders that rank highly the MTEB leaderboard (Muennighofi et al. 2023).
We find that MBR decoding with ROUGE-2 performs slightly better than MBR decoding with ROUGE-1, while MBR decoding with ROUGE-L performs slightly worse. MBR decoding with NVEmbed and Nomic both perform better than MBR decoding with SFR Embedder, with Nomic showing some improvement relative to greedy decoding. The latter result suggests that dense embed- ders could potentially be used for MBR decoding, although further work is required to understand what properties make for good MBR embedders. Overall, none of our additional utility metrics provide comparable improvements to MBR with Prometheus.
A.2 COMPARISON WITH UNIVERSAL SELF- CONSISTENYY
We compare MBR decoding Universal Self-Consistency (USC) (Chen al. 2023 In USC, Neand outputs are sampled from the LLM and passed directly to the LLM for consistency detection. This entails prompting the LLM choose the most consistent output. In Chen al. (2023). the authors demonstrate that USC improves over greedy decoding for mathematical reasoning, code generation, summarisation and question-ant -ansternn..
The limited context lengths of LLMs poses significant challenge when using USC, as it requires fitting all Neand 30 samples into single prompt. In contrast, MBR decoding only requires fitting two outputs into a single prompt as utility is computed pairwise. As our existing choice of models have limited context lengths (4096 tokens for Llama-2, 8192 tokens for Llama-3) and our outputs can be long (up 1024 tokens), are unable assess USC on equal footing with MBR decoding using these models without significantly reducing Neand- In order to facilitate fair comparison, we therefore use the Llama-3.1 models (Dubey al. | 2024) in place of the Llama-2 2 and Llama-3 models for this experiment. The 1 models possess context lengths 128k, thereby allowing us to fit all Neand samples into single to prompt required. Our USC prompt is follows:
21

---

Published as a conference paper at ICLR 2025
You are given collection of 30 responses to prompt. Select the most consistent response based on majority consensus. The most concistent response should be the most representative of all the responses provided. You should consider a variety of factors when evaluating consistency, including content, arguments and examples employed, style, structure and final answer, if not pass judgement on the quality or correctness of the response. Consider only consistency.
relevant.
Provide a short explanation of your choice, followed by your choice. Your choice should follow this format: "Most Consistent
Response: [[Response ID]]", for example: "Most Consistent
Response: [[15]]" if response 15 is the most consistent amongst all responses.
Responses:
(responses)
Llama-3.1-8B
Llama-3.1-70B
Avg.
Greedy USC
34.2
42.1
0
37.3
43.7
2.35
MBR Prometheus
40.2
45,8
4.85
Table 6: AlpacaEval 2.0 win rates (%) for Llama-3.1 models with greedy decoding, USC and MBR decoding with Prometheus, along with the average win rate differences compared to greedy decoding across all models (denoted Avg. A).
We document our findings in Table 6 greedy decoding, this improvement is smaller than the improvement provided by MBR decoding with Prometheus.
We find while that USC provides some improvements over
A.3 SAMPLING BASELINES
We conduct additional baseline experiments with top-p (Holtzman et al., 2020) and top-k (Fan al., 2018) sampling. We use these sampling methods to directly obtain a final output, and do not explore replacing standard temperature sampling with these sampling strategies for generating the hypothe- sis set we leave this to future work.
Llama-2-70
-lama-2-70B
Avg, A
Greedy
14.4
22.8
0
ROUGE-1 MBR
16.2
1.85
Prometheus MBR
17.7
26.2
3,35
top-p 0.9, 0.3)
14.3
23.7
0.40
top-p (p 0.9, 0.7)
14.7
23.9
0.70
top-p (p 0.5, 0.3)
15.3
1.50
top-p (p 0.5. 0.7)
15.0
24.9
1.35
top- (k 50, =0.3)
14.7
23.6
0.55
top-k (k 50, 0..)
15.0
23.5
0.65
top-k (k 20, 0.3)
15.0
1.25
top-k (k 20, 0.7)
14.7
24.0
0.75
Table 7: AlpacaEval 2.0 win rates for top-p and top-k decoding, along with the average differences compared to greedy decoding across all models (denoted Avg. A). Top-p and top-k sampling improve over greedy decoding. but do not match the performance improvements Prometheus MBR decoding.
We find that top-p and top-k sampling improves over greedy decoding, and can achieve performance close to that MBR decoding with ROUGE. The improvements are nonetheless much smaller than MBR decoding with Prometheus. Nonetheless, these results demonstrate that top-p and top- sampling could be used to produce hypothesis sets containing higher quality candidates, which could in turn improve downstream MBR decoding performance.
22

---

Published as a conference paper at ICLR 2025
A.4
GENERATION LENGTHS
Avg. Generation Length, WWrds
400
380
360
340
320
300
280
Greedy
85
Langest
BoN Prometheus
MBR ROUGE
MBR BERTScore
MBR Embredder
MBR Promethous
Figure 6: Average generation lengths (in words) for various decoding strategies, averaged over all five generator LLMs. MBR with Prometheus produces slightly longer outputs than most baseline methods, although these outputs are still far shorter than those produced by MBR with SFR-embedder and by the longest decoding baseline.
420
Llama2-7b
Avg. Generation Length Words
Llama2-13b
400
Llama2-70b
Llama3-8b
380
Llama3-70b
360
340
320
300
Greedy
BS
Longest
BoN Prometheus
MBR ROUGE
MBR BERT
MBR Embedder
MBR Prometheus
Figure 7: Average generation lengths (in words) for various models and decoding strategies AlpacaEval. MBR with Prometheus produces slightly longer outputs than most bascline methods, although these outputs are still shorter than those produced by MBR with SFR-embedder by the longest decoding baseline.
We compute the average generation lengths (in words) of our five generator LLMs on AlpacaEval 2.0 with different decoding strategies and plot the results in Figures 6 and 7 We find that MBR decoding with Prometheus yields outputs that are on average longer than those yielded by greedy decoding, beam search or Prometheus BoN decoding. They are nonetheless much shorter than the outputs yielded by MBR decoding with STR-Embedder and decoding by selecting the longest output. We hypothesise that MBR decoding with Prometheus encourages selection of more detailed responses which improves model performance, although note that verbosity alone is not enough to improve performance on our benchmarks as the LONGEST baseline fails to yield any gains, and length-controlled metrics for evaluation.
23

---

Published as a conference paper at ICLR 2025
Bullet List
Numbered List
Both
Neither
Greedy
14.2
29.1
26.4
30.2
Prometheus BoN
13.6
29.8
26.0
30,8
Prometheus MBR
13.9
30.1
28.1
27.8
Table 8: Percentage of responses to AlpacaEval containing bullet lists, numbered lists, both bullet and numbered lists, and neither bullet nor numbered lists. These responses were generated by both Llama-3-8h -------stttttt and 1aima-3-00b-Inttuuee using various decoding strategies. We find that there is little difference in the formatting of MBR-generated, BoN-generated and greedy-decodes outputs.
TTR
FK
Greedy
0.514
12.13
Prometheus BoN
0.520
12.40
Prometheus MBR
0.521
12.24
Table 9:
Averaged type token ratios (TTR) and Flesch Kincaid readability scores (FK) across outputs generated using various decoding strategies. We find that there
by both Llama-3-8b- ----a----b-Instruct and Llama-3-700 little difference in lexical diversity (as measured by TTR) and readability measured by FK scores) between MBR- generaede,, BoN NoN-generated and greedy-decoder outputs.
A.5 LINGUISTIC FEATURES
We attempt to identify linguistic differences (beyond generation length) between MBR- -gRe-genrateed responses and responses produced using greedy decoding and BoN decoding. We first consider looking for distinctive features in the formatting of MBRRgenneattd responses by identifying sponses to AlpacaEval that make use of bullet and numbered lists and then computing the percent- age of outputs that fall into each category. We use Llama-3-8b-Instruee and Llama-3-70b-Instnen as the generator models and Prometheus-2-76 as the MBR judge, and document the results in Table
We find that there is little difference in the formatting of MBR- MBBRggenerated, BoN- -aN-generaeed and greedy-decoded outputs.
Next, we analyse the lexical diversity and readability of MBR-ggenerater outputs, using the same dataset, generator and judge model as for the formatting experiments. For lexical diversity, we measure the type token ratio (Richards) 1987) of outputs, and for readability we compute Flesch Kincaid readability scores (Flesch) 1948). We document our findings in Table 9 and leave further analysis of linguistic differences between various decoding strategies to future work.
A.6 SPECULATIVE DECODING
Avg time per generation (s)
Ayg tokens /s
GPT-4o score
Vanilla MBR
59.3
4.48
6,83
MBR
Spec. Decoding
$2.9
5.21
6.90
Table 10: Results demonstrating that MBR decoding can be combined with speculative decoding to increase autoregressive decoding speed. We measure the average time per generation and the average tokens per second associated MBR decoding with and without speculative decoding, and show that speculative decoding improves decoding speed without sacrificing output quality.
We can combine MBR decoding with speculative decoding (Leviathan et al. 2023 to speed up the MBR decoding step. We demonstrate this on 100 samples uniformly drawn from AlpacaEval, using Llama-3-7000 3-70--Instruet as both the generator and the MBR judge (see hugg ng face. co/ bm fms amaaamma---m -0-00----------- acceterator for our chosen draft model), and report our findings in Table 10
A.7 RANK CORRELATION WITH GPT-40 SCORES
We used GPT-4o as reference- free judge to score the outputs of (lama-3-8b-Insttue (Neaud = 30, t 0.7) the first turn of MT-Bench. Outputs were scored scale of to 10, using the standard MT-bench evaluation approach. We used GPT-4o judge temperature of 0.5 and generated three scores per generation, taking as final scores the average of the three sampled scores. Then, for
24

---

Published as a conference paper at ICLR 2025
Avg. A over Greedy
Avg. Rank Corr
Prometheus-
0.28
0.119
Prometheus-8yy 8x7b
0.39
0.136
Judg
0.22
0,053
JudgeL.Mr .M-3hh
0.31
0.113
-lama-3-8b- 3-b--nnstrttt 30bt--t--
0.28
0.116
0.41
0.144
Table 11: Spearman's rank correlation between the GPT-4o scores and MBR scores generated using various MBR judge models, along with average A over greedy decoding, computed on Llama-3-8b- --maa---b-Instruet outputs on turn of MT-Bench. We find that stronger judges (as measured by the avg. A over greedy) are associated with higher average rank correlations with GPT-4o scores.
every sample, we computed the Spearman's rank correlation between the GPT-4o scores and the MBR scores.
We find that stronger judges (higher avg. A over greedy) are generally associated with better cor- relation with GPT-4o scores, although the absolute value of this correlation is not very high. This suggests that our judges are unlikely to overly biased towards GPT-4o, although stronger judges will generally agree with GPT-4o more.
B GPT-4O-JUDGE DETAILS
We use GPT-4o as judge for both AlpacaEval 2.0 and MT-Bench. More specifically, we use gpt-40-2024-055 accessed through an Azure endpoint.
For AlpacaEval 2.0, we use the official AlpacaEval implementation to conduct evaluation. We use the httpghhtte_pppcaaa_aeea eval__papt. 4 turo evaluator and bascline results against the default gpt-4-1106-previee generations unless other specified.
For MT-Bench, we use the single- and multi-tury judge prompts provided by LLMSYS FastCha as our judge prompts. Due to stochasticity in the outputs of the judge models, even with temperature set to zero, we generate three judgements per sample and take as the final score median of three judgement scores.
C PROMETHEUS SCORING RUBRICS
Prometheus takes as input scoring rubric that defines scoring criteria to be used during evaluation. We single, generic scoring rubric for all our experiments:
[Consider wide range of factors such as the heipfulness, relevance, accuracy, depth, creativity, and level of detail of the response.]
Score 1: The answer is completely unhelpful and incorrect. Nothing useful can be learned from it.
Score 2: The answer contains some helpful and useful information, but major flaws, in terms of facuality, accuracy and relevance, are alse present.
The answer is mostly helpful and relevant, although minor flaws exist.
Score 3:
Score 4: The answer is accurate, relevant and helpful, although there are some clear improvements that can be made with respect to depth, creativity and detail.
Score The answer is excellent. It is completely accurate and relevant, and ramonstrates high degree of depth and creativity.
"gihab..om//sssu-bbappaaa..uu githuh.com/lm-ssyyfaaahhhh
25

---

Published as a conference paper at ICLR 2025
We hypothesise that the performance of MBR and BoN decoding with Prometheus could be im- proved through further optimisation of the scoring rubric, particularly with ments, where unique rubrics tailored to each question. We leave this to future work.
adjust-
D ALPACAEVAL CATEGORIES
We classify questions from AlpacaEval and MTT-Bench and then evaluate performance by category.
For AlpacaEval, we perform manual inspection on the dataset and identify ten common question categories. We then use GPT-4o to classify questions based on these categories, using the following prompt:
Categorise an instruction based on the following list of categories. Only choose category, and only return the category, nothing else.
Creative writing
Business, technical and scientific writing Arqumentation, debate and persuasion
Mathematical reasoning
Puzzles and logical reasoning
Coding
How-to and other guides
Recommendations and advice
Factual question-answeringg
Other
Example Instruction: Bob has 5 sisters and 1 brother. How many siblings does of Bob's sisters have?
Category: Puzzles and logical reasoning
Example Instruction: Write me a resignation email for my job as an accountant, explaining that I am leaving to pursue dream of becoming lion tamer.
Category: Business, technical and scientific writing
Example Instruction: What factors gave rise to the English Civil War. Category: Factual question-answering
Example Instruction: I am visiting Kyoto next April. Recommend me 10 things to do!
Category: Recommendations and advice Example Instruction: Pretend to be Donald Trump write speech announcing that you are becoming a Democrat. Category: Creative Writing Instruction: (instruction) Category:
The percentage of questions assigned them are listed in Table 12
The results of our performance by category analysis for MBR decoding with Prometheus on Al- pacaEvvll are illustrated in the main text, in Figure
LLAMA-3 AS AN MBR AND BON UTILITY METRIC
We experiment with using Llama-3-8b Instrruct and 1aima----70-Innttete MBR and BoN utility metrics in Section 3.3.I This entails prompting the models to act as reference- based or reference- free evaluators. Our prompt for single-turn reference-hased evaluation is as follows:
26

---

Published as a conference paper at ICLR 2025
Category
Percentage of Questions
Factual question-ans answernngg
24.3
How-to and other guides
20.6
Recommendations and advice
12.1
Mathematical reasoning
3.2
Other
2.2
Crcative writing
11.8
Business, technical and scientific writing
12.7
Puzzles and logical reasoning
3.5
Coding Argumentation, debate and persuasion
6.7
2.7
Table 12: AlpacaEval question categories identified by GPT-4o.
[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by AI assistant to the user question displayed below. In addition to the user question, you are also given reference answer. This is the best possible answer provided by human expert. You should evaluate the assistant's response based on this. good assistant's s answer should share the content and style of the reference answer. Begin your evaluation by providing short explanation. Be as objective as possible. After providing your explanation, you must rate the response on a scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
[Question] (question]
[Reference Answer] (reference)
[The Start Assistant's s Answer]
(answer)
[The End Assistant's s Answer]
Our prompt for single-tur reference-free evaluation is as follows:
[Instruction]
Please act as an impartial judge and evaluate the quality of the response provided by an AI assistant the user question displayed below. Begin your evaluation by providing short explanation. Be as objective as possible. After providing your explanation, you must rate the response by strictly following this format: "[[rating]]" for example: "Rating: [[5]]".
scale of to 10
[Question] (question)
[The Start of Assistant's s Answer]
(answer)
[The End of Assistant's s Answer]
For multi-ture turn evaluation, we use a system prompt to specify the rules. For reference- based evalua- tion:
27

---

Published as a conference paper at ICLR 2025
Please act impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should focus on the assistant's answer to the second user question. In addition to the user question and conversation history, you answer. This is the best possible answer to the second user question provided by a human expert. You should evaluate the assistant's response based on this. A good assistant's s answer should share the content and style of the reference answer. Begin your evaluation by providing as possible. After providing your explanation, you must rate the response scale of 1 to 10 by strictly following this format: "[[rating]]", for example: "Rating: [[5]]".
also given reference
short explanation. Be as objective
For tur, reference-free evaluation:
Please act an impartial judge and evaluate the quality of the response provided by an AI assistant to the user question displayed below. Your evaluation should focus on the assistant's answer to the second user question. Begin your evaluation by providing short explanation. Be as objective as possible. After providing your explanation, you must rate the response scale of to 10 by strictly following this format: "[[rating]]" for example: "Rating: [[5]]".
The prompt template for both reference-based and reference-free multi-turn evaluation is:
<| The Start Assistant A Conversation with User >>
#*# User: {question_1)
#* Assistant A: (answer_1)
### User: {question_2)
#+ Assistant A: [answer_2]
The End of Assistant A's Conversation with User |>
F
REWARD MODEL AS A BON UTILITY METRIC
2-7B
2-13B
2-70B
3-8B
3-70B
Avg. A
Greedy
5.72
5.90
6.50
7.54
8,29
0
BS
5.58
5.95
6.49
7.30
8.20
-0.09
Prometheus BoN
5.77
6.08
6.65
7.66
8.42
0.13
Starling-RM-77
5.99
6.85
7.88
8.46
0.34
Prometheus MBR
6.10
6.26
6.79
7.69
8,50
0.28
Table 13: MT-Bench scores for various models and decoding strategies, along with average score differ- ences compared to greedy decoding across all models (denoted Avg. A). BoN decoding with reward model StarlingRM-7b marginally outperforms MBR decoding with Prometheus.
28

---

Published as a conference paper at ICLR 2025
We evaluate BoN decoding using Starling-RM-- Baa---M---B--pppa (Zhu al. 2023a) the utility metric. Starling-RMM is strong 7B sequence classifier reward model trained to facilitate RLHF (Stiennon et al. 2022) that achieves similar overall score to Prometheus-2-1
RewardBench (Lambert 2024). It takes as inputs prompt and a single candidate and outputs a scalar reward. As with et al. reward models in general, Starling-RM does support reference-based evaluation.
We find that BoN decoding with Starling-RN, as utility metric outperforms MBR decoding with Prometheus by small margin. We have possible explanations for this discrepancy. Firstly, Starling-RM achieves a much higher score than Prometheus-2-777 on RewardBench's Chat task (likely due the distribution of training data), suggesting that may simply be more suited to our particular benchmarks. Secondly, by providing continuous scalar rewards instead of discrete integer scores, Starling-RM enables more fine-grained evaluation of candidate outputs, allowing to distinguish between outputs that Prometheus might have rated equally. We believe that using reference- based reward model as the metric for MBR decoding could combine the advantages of fine-grained scoring with the increased reliability of consensus-based output selection. We leave the development of such models to future work.
G SELF-TRAINING: ADDITIONAL RESULTS
G.1 HEAD- RESULTS
65.3
at Grocds
62.5
60.0-
$7.5
55.0
$2.5
50.0
47.3
Groedy
Spo-Manu
dpo-Mon-2
dpp
doo BoN-
BoNi3
Figure 8: AlpacaFval 2.0 win rates (%) for self-trained models and various SFT baselines against sft with greedy decoding. Generation for all dpo models is done with greedy decoding. We find that MBR self-training with DPO allows models to match their MBR- -ecodicg performance.
We conduct head-to-head evaluation of our DPO self-trained models and various SFT baselines against sft with greedy decoding using the AlpacaEval 2.0 and illustrate our findings in Figure g Our head-to-head results show that our MBR self-trained models outperform our BoN self-trained models, the sft with beam search baseline, and the full sft with greedy decoding baseline. Our MBR self-trained models match the performance of the sft with MBR decoding baseline.
G.2
RESULTS ON NLP BENCHMARKS
Model sff
MMLU (f)
ARC challenge (f)
HellaSwag (t)
TruthfulQA (1)
47.2
57.3
80.6
51.6
7B
dpo-3-BoN
47.2
57.5
80.6
53.5
dpo-3-MBR
47.3
57.1
80.7
52.6
56.1
62.3
83.5
48.4
13B
dpo-3-Bol
56.3
62.5
83.5
48.2
dpo-3-MBR
56.1
62.6
83.5
47.4
Table 14:
Evaluation results of Prometheus trained models on four different NLP benchmarks. We find that MBR and BoN self self-training maintains performance on across all four datasets compared with the sft models.
We assess our self-traine trained models on four different NLP benchmarks: MMLU (Hendrycks et al,, 2021) ARC challenge (Clark al.) 2018). HellaSwag (Zellers et al.) 2019) and TruthfulQA (Lin et al. 2022), and report our results in Table [4] We find that telf-nraining maintains performance all four benchmarks despite using training dataset that is irrelevant for these tasks. This
29

---

Published as a conference paper at ICLR 2025
shows that MBR self- training can be used to improve the essructiio-following abilities of models without jeopardising other skills.
G.3 MBR TRA-NINGNNNG WITH ROUGE
AlpacaEval 2.0
MT-Bench
7B
13B
7B
13B
sft
5.18
8.24
5.43
5.85
dpo- Apo-1-MMn 1-MB1-MBPPPPP Peoomeetteuus
5.68
10.8
5.78
6.48
dpo-2-MBB M-R--Pometthuus
7.22
13.9
6.11
6.73
8.86
15.3
6.14
6.75
dpo- 1-MB-MM----- BEB-RROUEE
4.66
5.61
7.65
5.98
5.83
5.78
9.01
6.06
dpo-3-MBR-ROUGE 5.42
5.67
8.31
5.91
Table 15: AlpacaEval 2.0 win rates (%) and MT-Bench scores for models self-trained using DPO with Prometheus and with ROUGE-1 1 as utility metries. MBR self-training with ROUGE fails to yield substan- tial gains.
We explore using ROUGE-1 as the utility metric for MBR self-training. We find that MBR self- training with ROUGE fails to yield substantial gains. Improvements also saturate quickly, with model performance decreasing after the third training iteration. These results highlight the impor- tance choosing the correct MBR utility metric for self-training.
G.4 MBR SELF-TRAINING WITH LLAMA-3 3-8B-INSTRUCT
We also consider using Llama-3-4 3-8b- 8--8-IInsee Io--ruttutt in place of Prometheus the utility metric to trlf-train Llama-3-8b. We compare this approach to using Prometheus to train Llama-3-8b, and find that both approaches lead significant gains. The fact that Hama-3-8b-IIntruun improve Llama-3-8b suggests that self-improvement using MBR decoding is possible. We leave this future work.
the judge can be used to
Prometheus
Llama-3-8h-Inse Instrrr
s/1
6.70
6.70
dpo-1 1-MBR
6.94
6.99
dpo-2-MBB
7.45
7.51
dpo-3-MBR
7.55
7.52
Table 16: MT-Bench scores for Llama-3-8b self-trained using DPO with either Prometheus Llama-3-8b- Instruct as the MBR judge. Generation done using greedy decoding for models. We find that both approaches lead to significant gains.
G.5 ALTERNATIVE PAIR SELECTION STRATEGIES
AlpacaEval 2.0
MT-Bench
sft
5.18
5.43
1ho-1-MBR
5.68
5.78
dpo-2-MBR
7.22
6.11
dpo-3-MBR
8.86
6.14
dpo-1 1MBR BAW
4.95
5.78
dpo-2-MBR
8.06
5.96
dpo-3-MBRR
8.88
5.97
Table 17: AlpacaEval 2.0 win rates (%) and MT-Bench scores for models self-trained using DPO with BW and BMW preference pair selection strategies with fh as the base model. Generation is done using greedy decoding for all models. We find that BMW models perform similarly the original BW models on AlpacaEval and slightly worse on MT-Bench.
We investigate using alternative MBR preference pair selection strategy. Following Yang et al. (2024), we create two preference pairs from each sample, one formed from the highest (best) and median-scoring (mid) outputs, and another formed from the median-scoring and lowest-scoring
30

---

Published as a conference paper at ICLR 2025
(worst) outputs. We follow the exact same DPO training procedure as before, but replace original BW training set with this new BMW training set.
We document our results in Table 17 We find that BMW models perform similarly to the original BW models on AlpacaEval 2.0 and slightly worse on MT-Bench. We do not pursue this line of work further and leave additional investigations to future work.
G.6 PERFORMANCE BY CATEGORY
Category
Percentage of Questions
Factual question-answering
26.4
How-to and other guides
21.1
Recommendations and advice
4.4
Mathematical reasoning
0.1
Other
1.3
Creative writing
18.6
Business, technical and scientific writing Puzzles and logical reasoning
19.0
0.1
Coding Argumentation, debate and persuasion
6.3
2.7
Table 18: UltraChat-200k question categories identified GPT-4o. We perform this analysis subsample 1000 prompts sampled randomly from our 12000 training prompts.
We replicate the question category analysis described in Section 3 and Appendix D for our Prometheus MBR self-trained models and report results in Figure We find that performance rela- tive to the sft models improves across almost all question categories, with performance writing- based categories improving more than on reasoning- reasoning remains unchanged for the 7B model and decreases for the 13B model.
categories. Performance on mathematical
To better understand this discrepancy, we sample 1000 prompts from our 12000 UltraChat training prompts and categorise them using GPT-4o, following the same procedure described in Section 3.3 and Appendix [ We document our results Table I8] We find that reasoning-baa based questions (coding, puzzles and logical reasoning, mathematical reasoning) with mathematical reasoning and puzzles and logical reasoning especially underrepresented. We attribute our models' inconsistent improvements in these to this lack of data.
underrepresented in this dataset,
G.7 GENERATION LENGTIIS
ojpo-BoN
dpc
Average Caneration Longth, WWrdds
240
SBR-MBR
220
200
180
5i
Iter
Iter 2
Iter 3
Figure 9: Average generation lengths (in words) after iterative Prometheus MBR and BoN self-training on AlpacaEval. MBR self-training with DPO teaches the model generate more detailed responses.
We measure the generation lengths of our Prometheus MBR and BoN self-trained models on Al- pacaEval. We find that MBR self-training with DPO teaches the model to generate longer and more detailed responses at every iteration. In contrast, BoN self teff-training and MBR self ooff-training with SET only results in small increases in generation lengths.
31

---

Published as a conference paper at ICLR 2025
H
HUMAN STUDY
Win
Draw
Loss
Prometheus MBR vs. Greedy
30.0
53.3
16.7
Prometheus BoN vs. Greedy
21.7
60.0
18.3
Table 19: Head-to-head evaluation of Prometheus MBR and Prometheus BoN greedy decoding for Llama- 2-70b conducted on the AlpacaEval dataset by human evaluators.
Win
Draw
Loss
dpo-3-MBR sft dpo-3-BoN vs. sft 33.3
46.7
43.3
10.0
55.0
11.7
Table 20: Head-to-head evaluation of 13B dpo-3-MBR and dpo-3-BoN vs. sft conducted on the AlpacaEval dataset by human evaluators. Greedy decoding used for all models.
We conduct a human study for key MBR inference (Section 3) and MBR distillation (Section experiments. The objective of this study is to verify that our I.I.M evaluation results (AlpacaFval 2.0 and MT-Bench) align with real human judgements.
We recruited 4 volunteers, each of whom were given 60 samples in total to evaluate. Each sample consisted randomly-sampled AlpacaEval prompt and two corresponding generations displayed in random order. Volunteers were asked to select their preferred generation and, if neither genera- tion was preferred, to then rate the generations as equal.
We conducted four head-to-heat head experiments. The first two experiments, corresponding to exper- iments in Section g were between Prometheus MBR vs. greedy decoding and Prometheus BoN greedy decoding with Llama-2-706 The next experiments, corresponding experiments in Section were between 13B dpo-3-MBR vs. sft and dpo-3-BoN vs. .fft. We used 60 samples for each experiment. Volunteers were given an distribution of samples from each experiment.
Our results show good alignment with our LLM evaluation results. We find that Prometheus MBR decoding performs well against greedy decoding, with its win rate higher than the corresponding win rate for Prometheus BoN decoding vs. greedy decoding. We also find that dpo-3-MBR significantly outperforms sft, and its margin of victory is greater than the margin of victory of dpo-3-BoN vs. sft. Furthermore, the margin of improvement associated with MBR distillation is larger than the corresponding margin for MBR inference. These findings align with our findings from our automatic LLM evaluation experiments.
I
ALGGORITHMS
I.1 MBR INFERENCE
We provide the algorithm for MBR inference in Algorithm complementing the mathematical overview provided Section
I.2 MBR DISTILLATION
In MBR distillation, we first gather a preference dataset using MBR decoding over set input prompts. Given input prompts X and base model To.-1- we first sample Neand outputs per prompt
( ne (-12r), where x Xx. forming hypothesis set Hhyp (1), ( ,..., (Neand)).
We then compute expected utility for elements in Hhyp
Neand
(1)
Neand
32

---

Published as a conference paper at ICLR 2025
Algorithm MBR Inference
Inputs: Prompt generator model p, reference- basedd utility metric number of candidates N Neand, sampling temperature
Output: MBR output y
Initialise Hhyp Ø i ,,, do
Sample y(") ~ p(-(z) with temperature t
Add y()) to Hayp end for for ( E "hhpp do Compute (() Neand end for Select ÿ = arg max yEXwer ū(y)
Form hypothesis set
1 Ex=rm u(y(), y00)
y(())
// Compute expected utility
return y
and form preference pairs using the output that maximises expected utility ÿ+ and the output that minimises it
g" arg max i(y)
ye Hhyp
9 arg min ü(y)
We collect these preference pairs and their prompt (I, 9 . in dataset we denote Vk-
then use these preference pairs for DPO (Rafailov et al. the following policy objective:
2024) training. In DPO, we minimise
To(i+(a) [ror((+12)
3 log To ūū (x)) Tre (ú [x)
LDO
Ea & ~D)) log
[Blgg
(5)
where my is the policy and Tref the reference model. We repeat this process iteratively with k 1,2....,K. K. The initial model Te, is the base sft model. We choose = 3 in our experiments.
We also provide the algorithm for MBR distillation with DPO in Algorithm 2
J LIMITATIONS
While our work demonstrates the significant potential of MBR decoding, there are limitations that should be addressed in future research. Firstly, although we demonstrate using existing judge LLMs utility metries that MBR decoding consistently outperforms BoN decoding, this does not preclude the existence of reference- free metrics that are powerful enough to match or surpass the performance of their direct reference-basee based counterparts. This relates possible broader limitation the benefits using consensus quality for output selection, as the consensus solution may not always be the optimal one. We encourage future work train better utility metrics in order better understand these limitations. A second limitation of our work is that we do not study the biases introduced by the utility metric. One particularly pernicious form of bias is "reward-hacking" behavior, where the utility metric (likely as a result of its own training) selects outputs that evaluate well on our benchmarks but that are actually worse in quality. While we preclude this from being the case in our experiments via our human study (Appendix H), this does not mean that such pemniciuus behavior cannot arise in other settings. Finally, we do not study limitations on scalability. Although we show that small judge L.Mss (7B) can serve utility metrics for much larger models (70B), it is likely that weaker utility metrics cease being useful for very strong LLMs and on very complex tasks. Further research is needed to determine when this breakdown occurs.
33

---

Published as a conference paper at ICLR 2025
Algorithm 2 MBR Distillation with DPO
Inputs: Prompt sets X1, ..,..., XK. sft model The reference-basee utility metric number of candidates Neand, sampling temperature number of self-training iterations K. Output: Self-trained model Tox
for k E 21, 3 do Initialise 3/ - Ø
for E X do Initialise Hhyp + Ø for (1, Neand) do
Sample y6t) no (112) with temperature
Add (6) to Hhyp end for
Form hypothesis set
for yl) Hhyp do Compute ((y)(0)= Neand 21-1 u(y y00)) end for
Noaud
// Compute expected utility
Select ÿ arg ma ū(y) Select O = arg mEn ((y)
Select highest scoring output |/ Select lowest scoring output
Add (Øt ( to Vk
// Form preference pairs
end for
Update Th.
DP (T
// DPO training preference pairs
end for
return nex
K TRAINING AND INFERENCE HYPERPARAMETERS
Hyperparameter
Value
Learning Rate
5e-6
Num Epochs
Batch Size
Optimiser 81
AdamW W
0.9
Ba
0.95
1e-8
Weight Decay
0.1
Scheduler
Cosine
Table 21: Hyperparameters for SFT and MBR self-training with SFT.
Hyperparameter
Value Learning Rate
5e-7
Num Epochs Batch Size
8
Optimiser
RMSProp
0.99
@DPO Scheduler
Constant with warmup
Warmup Steps
150
Table 22:
Hyperparameters for MBR self-training with DPO.
Our SFT and DPO hyperparameters for self-training experiments in Section 4.2 are provided in Tables 2T and 22 We use bf1 6 mixed precision training with 8xA 100 GPUs for all experiments.
For inference,
4xA 100 GPUs with bf 6 quantisation for all LLMs and judge LLMs, other than for the Analysis of compute costs experiments in Section where we use 2xA100 GPUs. We vLLM (Kwon al. 2023) as the inference engine for all experiments.
34


Reviews:
[{"review_id": "nDuN4zaxOj", "reviewer": "Reviewer_Nmeb", "summary": "This paper proposes an approach for selecting one of N generation hypotheses using a Minimum Bayes Risk (MBR) method. MBR decoding alone results in improved performance. However, as MBR decoding is resource-intensive and may not be practical for real-life applications, an alternative use case—self-training with DPO on preference pairs selected via MBR—also demonstrates performance gains.", "questions": "Are there any distinctive linguistic features in MBR-decoded outputs? How does it affect diversity, style, or tone?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper explores the use of previously developed approaches—specifically, Minimum Bayes Risk (MBR)—for selecting a generation hypothesis at the decoding stage in LLMs. The MBR-selected hypothesis is treated as the one with the highest average utility according to a specified utility metric. The choice of this approach is well-reasoned, with a clear motivation behind it.\n\n* The experimental setup is solid. The paper first demonstrates improvements using MBR decoding. Given its high computational cost, the paper then explores self-training with MBR-decoded outputs, which also leads to improvements. The evaluation is conducted on two standard benchmarks (Alpaca-Eval 2.0 and MT Bench), with comparisons across different decoding approaches and a variety of utility metrics used in MBR. The discussion of results is well-organized, providing clear and comprehensive takeaways.", "weaknesses": "The paper is well-written and easy to follow. The reviewer does not identify any major weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pkSu7K3zJ7", "reviewer": "Reviewer_hFLa", "summary": "The paper presents a novel approach to improving the test-time performance of instruction-following LLMs through the application of Minimum Bayes Risk (MBR) decoding. The authors leverage LLM judges as reference-based evaluators to select high-quality outputs from a set of candidate outputs. They demonstrate that MBR decoding with LLM judges significantly outperforms greedy decoding and other decoding methods without references on benchmarks like AlpacaEval and MT-Bench. Furthermore, the paper explores iterative self-training on MBR-decoded outputs to retain performance improvements while mitigating additional test-time costs. The authors find that self-training using Direct Preference Optimisation leads to significant performance gains, matching or exceeding the performance of base models with MBR decoding.", "questions": "- **Question 1:** How does the performance of MBR decoding with LLM judges compare to other state-of-the-art decoding methods beyond those presented in the paper?\n- **Question 2:** Can the authors elaborate on any potential negative impacts of using LLM judges, such as the risk of overfitting to the judge's biases?\n- **Question 3:** The hyperparameter N~cond~ is set to 12 for generating candidates for DPO but increased to 30 during decoding. How did the authors determine these values, and how do they align with the experiment illustrated in Figure 2?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "1. The application of MBR decoding with LLM judges is a creative approach that combines recent advances in LLM evaluation with decoding techniques.\n2. The experiments are thorough, and the benchmarks used are relevant and well-established in the field.\n3.  This paper also explores the guiding role of MBR decoding in LLM’s DPO training, which is inspiring in subsequent model training.", "weaknesses": "Main weakness:\n- The presentation lacks clarity and reads more like an experimental report than an academic paper. The methods section is merged with the experiments and results, lacking any formal formulations. For instance, in Section 4.1.2, Iterative DPO on MBR-Decoded Outputs, adding mathematical formulations would improve both understanding and reproducibility of the approach.\n\nOthers:\n- This paper only compares some relatively simple decoding methods. If some better decoding methods such as speculative decoding and medusa can be added, the method will be more credible.\n- The paper could also benefit from a discussion on the computational costs associated with MBR decoding and self-training, especially when scaling to larger models or datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eMWHN8Inrq", "reviewer": "Reviewer_Z8nY", "summary": "This paper explores MBR decoding with reference-based LLM judges for selecting one of n outputs for instruction finetuned LLMs at inference time. \nThe paper uses Prometheus2, LLama3, and JudgeLM models as a judge for Llama 2 & 3 models and evaluates on AlpacaEval and MT-Bench, comparing MBR decoding with best-of-n inference, and other one-best decoding strategies, and explore non-LLM utility functions for MBR as well.\nThey find gains across the bench, with smaller judges also being successful as utility function in MBR decoding for guiding larger LLMs. MBR decoding is furthermore combined with self-training (DPO) to yield further gains and overcome the added decoding costs.", "questions": "1. Could the LLM model itself be used as a judge? In the spirit of self-improvement.\nJinnai et al. evaluated their utility metrics on benchmarks without LLM judge. How would LLM judges perform there? (e.g. machine translation, summarization)\n2. Could there be some kind of overfitting/bias towards the GPT-4o judge that is dominating the win-rates in the comparison (the “closer” the judge to GPT-4o, the better the MBR)? In the extreme case, what if GPT-4o was used as a judge - I’m sure that would beat even Prometheus. Perhaps one could compare the outputs selected by the evaluated judges to GPT-4o decisions to find out if the ranking of win-rates corresponds to agreeing with GPT-4o.\n3. I would love to see some qualitative analysis on how BoN and MBR generation selection differ with the same judge. The paper explains that the smoothing effect of MBR might be helpful to find the best generation, but I would like this to be made a little more built out and supported by examples.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The experiments are thorough (and well documented), including multiple sizes of learner and judge models, and multiple utility functions for comparison.\n2. The analyses are interesting and well presented, explained and visualized. Important ablations, such as hypotheses size and temperature, are included.\n3. The distillation solution is very attractive for boosting performance in practice without sacrificing inference time, which is contributing to a timely trend. There's a good chance this approach will be added to everyone's box of inference tricks.\n4. Compute costs are well discussed and addressed with the distillation solution.\nSolid and interesting review of past works, especially about Machine Translation where MBR decoding has already become popular.", "weaknesses": "1. The method is not particularly novel, as MBR decoding with LLMs has been studied in the past (as discussed in Related Work), but not with an LLM as judge.\n2. Given the novelty being the use of a judge to find the consensus output, I would love to see more analysis on the dependence of the LLM judge’s quality. There is a notable gap with other utility functions, so it would be helpful to understand where they diverge. See questions below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Spotlight)