Uncertainty-Aware Decoding with Minimum Bayes Risk
Authors: Nico Daheim
Clara Meister
Thomas Möllenhoff
Iryna Gurevych
Venue: ICLR 2025 Poster (2025)
Keywords: mbr
uncertainty
llms
decoding
machine translation
language generation
variational learning
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We generalize MBR into an uncertainty-aware decoding method.

Abstract: Despite their outstanding performance in the majority of scenarios, contemporary language models still occasionally generate undesirable outputs, for example, hallucinated text. While such behaviors have previously been linked to uncertainty, there is a notable lack of methods that actively consider uncertainty during text generation. In this work, we show how Minimum Bayes Risk (MBR) decoding, which selects model generations according to an expected risk, can be generalized into a principled uncertainty-aware decoding method. In short, we account for model uncertainty during decoding by incorporating a posterior over model parameters into MBR’s computation of expected risk. We show that this modified expected risk is useful for both choosing outputs and deciding when to abstain from generation and can provide improvements without incurring overhead. We benchmark different methods for learning posteriors and show that performance improves with prediction diversity. We release our code publicly.

Published as a conference paper at ICLR 2025
UNCERTAINTY-AWAT DECODING WITH MINIMUM BAYES RISK
Nico Daheim', Clara Meister?, Thomas Möllenhoff Iryna Gurevych'
UUiquitous Knowledge Processing Lab (UKP Lab)
Department of Computer Science and Hessian Center for AI (hessian.Al)
Technical University of Darmstadt "ETH Zurich
3 RIKEN Center for Advanced Intelligence Project, Tokyo, Japan
ukp. uu-darmsstad,, de
ABSTRACT
Despite their outstanding performance in the majority of scenarios, contemporary language models still occasionally generate undesirable outputs, for example, hal- lucinated text. While such behaviors have previously been linked to uncertainty, there notable lack of methods that actively consider uncertainty during text generation. In this work, we show how Minimum Bayes Risk (MBR) decoding, which selects model generations according to an expected risk, can be general- ized into a principled uncertainty-aware decoding method. In short, we account for model uncertainty during decoding by incorporating a posterior over model parameters into MBR's computation of expected risk. We show that this modified expected risk is useful for both choosing outputs and deciding when to abstain from generation and can provide improvements without incurring overhead. We benchmark different methods for learning posteriors and show that performance improves with prediction diversity. We release our code publicly[]
INTRODUCTION
Today's 's language models can generate fluent and coherent text. While they perform well in many scenarios, there still instances where they fail and, for example, hallucinate factually incorrect
outputs or generate harmful language (Ye et al. [023 Bhandari & Brennan 2023 Li et al. 2024). Previous works have shown that these behaviors are often related
-uito--distribution inputs (Ren
et al. 2023) and (cpistemic) uncertainty (Xiao & Wang, 2021; van der Poel et al. 2022 Fadeeva
al.
2024) which are both connected uncertainty about the parameters of the model. Yet there
still lack of methods that adjust for this type of uncertainty during decoding in language generation.
Minimum Bayes Risk (MBR) decoding was originally proposed for statistical machine transla- tion (Kumar & Byrne 2002). motivated by similar model shortcomings. The idea of MBR is to make use of the entire distribution when choosing an output, because, while the model distribution might be good overall representation of the target distribution (Smith) 2011) 1). individual samples might not be adequate. More recent works have shown that such problems persist with modern models (Stahlberg & Byrne| 2019 Cohen & Beck 2019 Eikema & Aziz 2020), precipitating the resurgence of MBR (Freitag et al. 2022). In this work, we show how a small adjustment to MBR decoding can enhance it beyond this scope and turn it into uncertainty aware decoding method.
In short, we modify MBR's 's definition of expected risk by incorporating an additional expectation over posterior distribution over model parameters. This adjustment enables to account for uncertainty in parameter estimates when judging the quality of different hypotheses from a model. We present different estimators for this expected risk which use multiple models from the (approximate) posterior to generate outputs Two of these estimators combine outputs at the sequence-level, i.e. full strings generated by model, which useful for combining the outputs of black-box LLMs for which one does not have access to output probabilities. Another estimator combines token- distributions.
https:: / thub wttp///KLaaaaaccr/22555--- unnuuertttt
This approach has previously been shown to improve downstream performance in classification tasks Blun dell al. 2015: Lakshminarayaman al. 2017 Maddox al. 2019. Shen al. 2024

---

Published as a conference paper at ICLR 2025
Overall, we find strong evidence that accounting for weight uncertainty can improve decoding and reduce hallucinations when finetuning and pretraining from scratch, even without computational overhead. We find that improvements trend with the expressiveness of the posterior. Likely related to this, the performance of uncertainty-awar MBR is highly correlated with the prediction diversity across the combined models. We also find that weight uncertainty provides useful signal for selective prediction, where we observe that the uncertainty-awan expected risk can be used to decide when to predict abstain from generation. Furthermore, we show that performance scales: improves with more models and larger hypothesis set sizes. Finally, we show the effectiveness of this framework when used to ensemble outputs from black box LLMs.
BACKGROUND
2.1 PROBABILISTIC LANGUAGE GENERATION
Modern models for language generation are predominantly locally-normalized, autoregressive models of a conditional distribution over next tokens. The probability of a sequence of tokens forming a string can be determined by the product of all next-token probabilities in the sequence. Formally, given input x and model pa the probability of an output sequence y = (y1, ......) is computed as
ly
P x) = Po(ve x)..)).
II
Here, each y/ is a token from some predetermined vocabulary V and 0 E Id are the parameters of the model which are also often called weights. The input could be text but, for example, also images.
Learning Po. The parameters of the model Pe are generally learned given paired examples D (x0), (1) Ji=1-2 loss function and an optimization procedure. The loss function then indicates how well the model Pe captures the data- getta-gnnrating distribution p(- x) from which we assume ' is sampled. In most cases, language generation models are learned by minimizing an empirical risk over data examples in terms of one parameter set 0 E R", for example, using AdamW (Loshchilov & Hutter 2019). However, such approaches can not directly model weight uncertainty. In this work, we instead use Bayesian methods to model weight uncertainty. We describe them in 1 $4.2
Decoding from pa- At inference time, our goal is to generate a string from po x). The set of decision rules used in this process is often referred to as the decoding strategy. One such strategy is simply to sample tokens autoregressively until a stopping criterion, usually a fixed maximum length or special end-of-sequence token, is met. Another strategy is to (approximately) search for the maximum probability string according to pa x). Both of these approaches have proved problematic empirically (Fan al.) 2018 Holtzman al. 2020 Eikema & Aziz 2020)] Hewiit et al., 2022). prompting the exploration of alternative strategies. The shortcomings of all of these strategies have been (at least partially) attributed to the fact that they do not consider a string's utility, which
may not perfectly align with its probability. Minimum Bayes Risk decoding aims to solve this issue.
2.2 MINIMUM BAYES RISK DECODING
Minimum Bayes Risk decoding is derived from Bayesian Decision Theory, which states that optimal decisions are those that minimize an expected risk or, equivalently, maximize expected utility (see DeGrooi 2005) inter alia). Given a utility function u : V* x V + Rzo which assigns to each pair of strings nng-tiettive utility, MBR aims to find the string that maximizes expected utility with respect to the target distribution. This principle is especially appealing when working with possibly imperfect model of the target distribution, such as Pe, because it allows using the full model distribution instead of relying on the adequacy of individual samples, which is argued to be the downfall of other decoding strategies (Eikema & Aziz 2020). We thus choose the hypothesis:
y arg max y'EV' y~pa(11x)
[u(y.y')
arg max Σ Y'EV' YEV-
Poly (x)u(y,'').
(3)
2

---

Published as a conference paper at ICLR 2025
There are several obstacles to computing Eq. (3). Both summing over all possible strings in V* to compute the expectation and searching over them to find the expectation-maxiiiiing hypothesis are computationally infeasible Thus, approximations to Eq. (3) are used in practice.
The common approach to circumvent these obstacles is to employ an (often Monte Carlo) estimator of the expected utility and limit the search space subset of V*. Since the estimator requires sample of strings from the distribution of interest, the same strings are often used in both the utility estimation and approximate search [ We refer this collection as the hypothesis set and denote the samples used in our estimator as H = [y( li=1. In the case of a Monte Carlo estimator, where all
y(*) Po. we denote this collection Ho. This leads to the following approximation to Eq. 3
Σ u(y.y'), yEHo
arg max
y'EHo
which has proven to be a useful criterion for selecting examples for knowledge distillation (Finkel-
stein & Freitag 2024 Yang et al. 2024b). Most prior work on MBR has focused on making the approximation in Eq. more efficient (Eikema & Aziz 2022; Fernandes et al. 2022 Cheng & Vlachos 2023 Vamvas & Sennrich 2024) or better choices for utility functions (Freitag et al.
2022) but few have considered an important underlying assumption: that Pv is a good substitute for If uncertainty over the suitable model parameters 0 (i.e. weight uncertainty) is high, e.g., when training data is limited, using a single Pa may not provide a good substitute. Bayesian modeling already provides tools to account for such uncertainty by marginalizing a distribution over possible parameters. We use this approach next to establish incertainty-awan decoding schemes that account for weight uncertainty.
3 MINIMUM BAYES RISK DECODING WITH WEIGHT-UNCERTAN
In this section, we show how simple change can turn MBR into uncertainty-awant decoding method. We first introduce weight uncertainty. Then, we use it to establish an uncertainty-awarl variant of MBR before presenting three practical decoding methods based on
3.1 GENERALIZING MBR WITH WEIGHT UNCERTAINTY
Often, distribution over possible model parameters is used to model weight uncertainty based on Bayesian principles (Maddox al., 2019 Osawa al. 2019 Möllenhoff Khan 2023). According
to Bayes' theorem, the probability of parameterization is p(O D) x P(D Ø) p(() where p(0) a prior and 'D is our data. Because calculating an exact distribution p(0 D) over model parameters is intractable, an approximate distribution g(.) is usually used. There are numerous methods one can use for obtaining g-)) for example, Laplace (Mackay/ [1992] Daxberger et al., 202I] and variational learning Graves 2011 Blundell et al. 2015). We use variational learning, which we describe in $4.2
Access to a posterior q()) allows prediction by combining the outputs of multiple Pa, weighted by the probability (@) of each parameterization 0. The resulting distribution is often referred to as the predictive posterior distribution, which we denote as Pe. Empirically, this has been shown to improve calibration (Yang et al.) 2024a) and uncertainty estimation (Shen et al. 2024). However, in modern language generation, it is not immediately clear how model predictions should be combined in practice. Combining predictions in probability space is difficult, because many modern LLMMs only accessible via APIs that do not return probabilities at either the token- or sequence-level. Standard Monte-Carlo-basse methods avoid this issue, but they may be problematic: even for larger sample sizes, a given string would likely only be sampled once. While generations might be approximately similar, e-g., differing only in punctuation, this approach treats them as completely disparate. We now show how natural extension of MBR provides principled framework for combining model predictions that circumvents these issues.
'The latter problem is not unique to MBR, and faced by all maximization- decoding strategics for autoregressive language generators. Hence, approximation algorithms are also used for these strategies.
*Some works have explored using different subsets for these two steps (Fikema & Aziz 2022 Fernandes)
ct al.
2022;; we leave the exploration of the interaction of this design choice with our methods to future work. We drop the normalizing term for succinctness does not affect the arg max operation.
3

---

Published as a conference paper at ICLR 2025
We propose to generalize MBR by replacing the definition of in Eg. with the predictive posterior Pg to account for weight uncertainty. Then, the search problem is
arg max
Σ (y x))(y.y)) Y'EV" YEV-
(5)
We recover standard MBR when using the delta method to approximate pe. i.c. approximating the predictive posterior using one model parameterized by the mean of q (Khan Rue, 2023 App. C).
Conte-Carloobaseed approximations of MBR do not require knowledge of string probabilities but only the ability sample from the model. Further, the utility function can be chosen as a soft matching between strings to account for similarities between samples instead of treating them as completely distinct, which addresses the aforementioned issues. We next discuss three decoding algorithms for
approximately solving Eq. 5 using sequence- (.3.2 and token-level posteriors ($3. 3..
3.2 JEQUENCE-LEVEL POSTERIORS FOR ENCERTAINTY-AWAR DECODING
While autoregressive language models are trained to model a distribution over tokens, the quantity of interest often the probability of an entire sequence y E V Therefore, it is natural to model predictive posterior on a sequence-level by using an expectation over sequence probabilities:
ly II
(ve4) Pe
y
x)
[Poly x)] = e
EE 8-wy
Po year x)
Using a sequence-level posterior to replace the model distribution in Eq. (3) admits two practical methods for (soft) model averaging, because when is bounded or non-negative the order of the two expectations in Eq. (5) can be switched due to Fubini's theorem (DeGroot 2005 Sec. 8.9; Robert 2007] Sec. 2.3). Note that the second expectation (over models) implicit the definition of pe This suggests using the equivalence:
arg max Y'EV-
Σ YEV.
E trg
(y x)] u(y.y'
arg E y'ev- 0~g
Poly xx)((r,'').
(8)
YEv'
Monte-Carke estimators of Eq. 7 and Eq. then correspond to either sampling a collection of generations from a set of models M (0 (( sampled i.i.d from q and using Eq. (4) or using Eq. independently for each model before summing the utilities of each output across models.
Formally, let He denote the hypothesis set for each model 0 and H. WeeM the collection of all generated outputs across them. Note that both H.M indicated by H, and Hg preserve sample counts to ensure unbiased estimator. Then, the approximate solutions become
g arg max
Σ u(y,y')
9
8 arg max y'EHM BEMyEHo
EE u(y,y'.
(10)
y'EHM YEHM
This is convenient because it allows us to ensemble any set of LLMs given just the ability to sample from them and can casily be parallelized. No access to probabilities is required. For Eq. (IO). even utility computation can be parallelized across models. There are trade-offs between both estimators, especially in terms of computational complexity, which discuss next.
Computational Costs. Eq. (9) requires (IM) - (Hol)² evaluations of u. This might be impractical for large sizes of Ho but, intuitively, the larger amount of comparisons might be helpful for MBR. Eq. (10) requires only [MM |Ho utility computations. This is faster and, especially for more costly utility functions that e.g. use LLMs as judges (Wu et al. 2025), can enable larger hypothesis set sizes.
Here, θ denotes all possible parameterizations 0 of the model and is used to indicate a predictive posterior. We refer to Malinin & Gales (201 Sec. App.A) for further discussion of token- and sequence-level posteriors for uncertainty estimation for autoregressive models.
"Many commonly used utility functions for MBR are bounded and non-negative. For example, BL.EU (Pap ineni et al. 2002 and BERTScore (Zhang et al. 2020 return scores from 0 to 100 or to 1, respectively.
4

---

Published as a conference paper at ICLR 2025
Discussion. When preserving sample counts, Eq. 9 provides unbiased estimate of Eq. Intuitively, this is advantageous because highly probable sequences can contribute more to the decision. This differentiates a set union. Recent work (Parinhas et al. 2023) also uses Eq.. (9) but does not explore the connection weight uncertainty. Our methods draw parallels between MBR, which aims to minimize expected risk, and PAC-Bayes bounds (Alquier) 2024), which study the expected risk of predictive posteriors. Finally, it also helps understand early system aggregation methods that use similar decision rules as shown here, e.g., by optimizing scalar model weights (González- Sonzález-Rubio et al. 2011 Eq. 8).
from prior work, such as Kobayashi (2018] Alg. 1), who rather use
3.3 TOKEN-LEVEL POSTERIORS FOR UNCERTAINTY-AWAR DECODING
While our definition of uncertainty-aware MBR uses sequence-level Bayesian modeling, language models generally define distributions over tokens. It is thus natural to consider predictive posterior defined over token-level distributions, i.e., by averaging token-level probabilities at each time-step:
(tok) Pe
(y x) = II Bmg
yeee; xce:
(11)
Note, though, that for a given sequence, this approach will in general not assign the same probabilities as sequence-level ensembling el Consequently, the decisions when using MBR might also be different. Since the expectation over models is intractable in Eq. (I), we Monte Carlo estimator. The estimator averages the token-level probabilities given by the models M during generation:
(lok) Pe (
X)..))
Σ Polue y<,,x).
(12)
[M
BEM
When sampling the hypotheses set He from this distribution, i.e., sampling each token according to (tok) MBR estimator like the one in Eq. (5) can be used to incorporate weight-uncertainty 5
Pe
ÿ = arg max
E u(y.y').
(13)
y'EHe yEHo
There are several intuitive reasons why this could improve decoding. Perhaps the foremost is that probabilities obtained from model averaging often beller-calibrated than those of a single model (Yang et al. 22024a Shen et al. 2024), inter alia). By averaging predictions, modeling weight uncertainty (over 0) enable better estimates of predictive uncertainty (over y). Since predictive uncertainty has been shown to correlate with hallucinations (Xiao & Wang 202T). one hope would be that better estimates of it would downweigh e.g. hallucinated outputs.
Computational Costs. Token-level posteriors only require |76|2-many MBR comparisons when the hypothesis set size is equal to [7]. Sequence-level combination requires |M -M comparisons for Eq. (TO) even (IM) /mll) comparisons for Eq. (9) if all hypothesis sets have the same size. However, fitting all models for token-level combination on one GPU can be hard and communication overhead is high when distributing them across GPUs. Further, token-level posteriors can not be used with black-box APIs that do not provide token-level probabilities.
3.4 SELECTIVE PREDICTION WITH BAYES RISK
For some inputs, for example, grammatically-inoorti good predictions. Then, it can be wise to abstain from answering and, e.g., defer to human expert instead. Selective prediction tackles this by abstaining for inputs (or outputs) that score highly in some criterion V - R that assigns a score for a given input x. (Geifman & El- Yaniv 2017 Ren al., 2023) Kuhn et al. 2023). In practice, given > and test dataset Dues. we only evaluate the model's 8 answers for the top-[ae [Dresil] examples according should improve as a decreases and we evaluate a smaller and smaller subset of outputs.
strings, even good model may not provide
If is reliable, performance
Thii is because sequence-level modeling uses a an expectation of products approach while token-level modeling uses a product of expectations approach. Since expectation and product operations do not neces- sarily commute, these two ensemble definitions will, in general, assign different probabilities to the same sequence (Malinin & Gales 202T
5

---

Published as a conference paper at ICLR 2025
Expected utility promises to be good criterion: if we expect low utility, should abstain from answering; if we expect high utility, we can place more trust in the model's answer. We compare different methods for using expected utility the selective prediction criterion. We first consider the maximum-utiliity output in He or H. for Eq. (3) and Eq. (10). i.e
Stok (x)
max y'HHe
Σ y')
Sseg (x)
max Σ u(y.y').
(14)
Y'ERM BEMYEHO
YEHs
Note that we can easily define a similar risk for Eq. (9) by replacing HH with HM in the definition of sus (x). Another strategy is to use the expected utility across outputs for the given input. We can do this by averaging the utility of all outputs in the hypothesis set He H.M [
Stok (x)
Σ Σ u(y.y')
Sszg (x)
Σ E DEM Y'E yEHo
E u(y,y').
(15)
Y'EHa YEH
EXPERIMENTS & RESULTS
Here, we demonstrate empirically that incorporating weight uncertainty improve decoding. First. we provide brief experimental details and discuss how we learn weight uncertainty in $4.1 and $4.2
More details about our experiments are found in App. A Then, we show results using prompted, fine- tuned and from-scratch-trained models in $4.3 where we explore different posteriors and model com- bination methods. $4.4 looks into the trade-off between performance and ensemble diversity and $4.5
Bayes risk for selective prediction. Finally, we show the scaling behavior of various methods in $4.6
4.1 EXPERIMENTAL DETAILS
Datasets. We use 4MT14 (Bojar et al. 2014), IWSLT14 (Cettolo et al. 2014). afroMT (Reid et al.
2021), IWSLT17 (Cettolo et al. 2017). WMT18 (Bojar et al. 2018). and WMT19 (Barrault et al.
2019) for machine translation, XSUM (Naaaan et al. 2018 and SAMSum (Gliwa et al. 2019) for summarization, E2E-NLG (Nvikova al. 2017) for tata-to-text generation, and STS-B Cer et al. 2017) for scoring. For the latter, the model outputs string representation of its numerical prediction and MBR corresponds to an empirical mean of the numerical predictions (Lukasik al. 2024).
Models. We zero-shot prompt Llama-3 8B (Dubey et al. 2024), Mistral 7B (Jiang et al. 2023),
Gemma-2 9B (Gemma Team 2024a), and Qwen-2 7B (Yang et al.
2024). We finetune Gemma-2B: 2BB
it (Gemma Team (2024b using LoRA (Hu et al. 2022) with ca. 0.9M trainable parameters. For training from scratch, we use the Transformerui architecture with ca. 261M parameters for WMT14
and Transformerpue with 86M-126X parameters otherwise, following Vaswani et al. (2017).
Metrics. For machine translation, we use the SacreBLEU implementation Post 2018) of BLEU (Pa- pineni et al. (2002). chrF (Popović) 2015), the quality estimator COMET Rei et al. 2022), and LaBSE (Feng et al. 2022) to evaluate hallucinations which has shown strong correlation with human judgements (Dale et al. [023) Himmi al. 2024). For Summarization and data-to-texx generation we use ROUGE (Lin 2004) and regression is evaluated using root mean-sgae srcannsquared error (RMSE). We use FactCC for hallucination evaluation on XSUM (Kryscinski et al., 2020). For the utility function u we use BERTScore (Zhang et al. 2020), except for IWSLT14 and afroMT where we use BLEU.
4.2 LEARNING WEIGHT UNCERTAINTY.
We use the variational learning algorithm IVON (Shen et al. 2024) estimate posterior distribution over model weights and model weight uncertainty. We choose it, because each training run with IVON has only negligible overhead compared to AdamW (Loshchilov & Hutter 2019) and gives comparable performance, also shown in Tab. [ It is also possible to use other Bayesian Deep Learning methods, such as, Laplace (Daxberger et al. 202I) or SWAG (Maddox et al. 2019) but we leave their exploration for future work. IVON learns unimodal Gaussian posterior q ()) N/O m, E) with mean m and (diagonal) covariance matrix Σ. Setting model parameters equal to the mean of this distribution (m) is similar to standard neural network training but Σ also provides an estimate of its stability. To be precise, for each parameter m, the variance Σn; indicates how much this parameter can be changed without significant performance degradation which be seen

---

Published as a conference paper at ICLR 2025
IWSLT17 En-De
WMT18 Ti-En
XSUM
SAMSum
E2E
NLG
STS-B
Method
BLEU
COMET
LaBSE
BLEU
COMET
LaBSE
R-1
R-L
FacCCC
R-1
R-L
R-1
R-L
RMSE
MBR (AdamW)
19,93
76.62
83.47
14.75
78.20
76.02
33.63
25.67
27.50
46.47
36.21
67.88
44.41
0.330
MBR Mean
19.73
76.60
83.51
15.27
78.44
77.12
33.04
25.19
23.56
46.17
35.98
68.74
45.16
0.284
Sequence-level- Eg. (9)
Unimodal
20.39
77.42
84.01
15.66
79.01
77.79
33.39
25.73
26.07
46.40
36.51
69.36
45.57
0.271
Deep Ensemble
21.24
77.94
84.20
15.63
79.01
77.60
33.37
25.68
27. 40
46.71
36.87
69.56
45.77
0.269
Sequence-level- Eq. (10)
Unimodal
21.08
77.63
83.96
15,46
78.84
77.35
33.05
25.46 25.46
27.50
46.21
36,44
69.13
45.38
0.271
Deep Ensemble
21.20
77.91
84.04
15.69
79.10
77.56
33,10
25.50
32.86
46.14
36.48
69.19
45.31
0.269
Table 1: Sequence-level model combination to account for weight-uncertainty can improve the performance of finetuned Gemma-2B model various language generation and scoring tasks. Even simple posteriors that do not incur overhead during finetuning can give "for-freee" improvements (unimodal). The number of total MBR comparisons is the same for all methods and each dataset. MBR @ mean denotes decoding with a single model that is the mean of a variational distribution.
WMT14 En-De
IWSLT14
Dn-En
Sampling
Beam Search
Sampling
Beam Search
MBR comparisons
Effectivv beam size
Method
BLEU
COMET
BLEU
COMET
BLEU
COMET
BLEU
COMET
MBK
23.37
71.04
27.56
75.23
33.69
74.71
35.90
76.65
400
20
24.30
72.15
27.53
75.18
34.53
75.18
36.07
76.76
1600
40
Sequence-level
Eq.
9
Unimodal
24.31
72.09
27.52
75.16
34.59
75.15
35.78
76.55
1600
40
Deep Ensemble
24.70
72.39
28.99
76.02
36.03
75.79
38.30
78.01
1600
40
Sequence-level Eq.
10
Unimodal
24.21
72.15
27.56
75.21
34.65
75.20
35.99
76.67
1600
80
Deep Ensemble
24.67
72.58
28.29
75.70
35.42
75.84
37.42
77.69
1600
80
Token-level
Unimodal 23.44
71.36
27.75
75.19
33.62
74.68
35.94
76.66
400
80
Deep Ensemble
23.95
71.58
28.98
76.08
34.61
75.06
38.56
78.31
400
80
Table 2: Weight uncertainty improves decoding when training from scratch and using ancestral sampling and beam search. More complex posteriors (Deep Ensemble) provide better improvements. We use Transformerhig on WMT14 14 and Transformenhas on IWSLT17. Effective beam size = number of beams per model times number of models (we use four).
measure of uncertainty. We also use multiple models obtained from independent IVON training runs to form Deep Ensemble (Lakshminarayanan et al. (017) in order to study multimodal token- or sequence-level posteriors. This can be seen as constructing a mixture-of-GGausee equal mixture component weights but ineurs training overhead, since training time increases linearly with the number of mixture components. Unless otherwise stated, we use four models in total for MBR, i.e. [M] 4. For deep ensembles, we use the mean of each training run and for the unimodal method using IVON we use four samples from the posterior. For smaller models we train all parameters but for larger models we only train newly-inserted LoRA parameters 0' 12 following IVON-LoRA Cong et al. (2024). IVON-LoRA then learns a distribution g(0') = =\((0' m Σ while the original pretrained model parameters remain fixed.
posterior with
4.3 WEIGHT UNCERTAINTY & DECODING
Weight uncertainty improves decoding. Tab. D and Tab. 2 show results using finetuned Gemma- 2B and Transformer models that were pretrained from scratch, respectively, on various language generation and scoring benchmarks. Results on two low-resourg tasks from afroMT are found App. B.I For fair comparison, we match the number of MBR comparisons, i.e. evaluations of the utility function for the estimator, with the single-model MBR baseline, as described in A.4
We find in Tab. D and Tab. 2 that weight uncertainty improves performance across all benchmarks, even with matched compute budgets. That is, using just one model (and thereby neglecting weight uncertainty) performs worse than using multiple models sampled from the posterior and averaging their predictions. Improvements also tend to hold when compared to training with Adam W (Tab. []. In particular, when using Eq. ( with unimodal posteriors both training time and time needed for decoding are the same as for the single-mode MBR baseline. We ensure that the time needed for decoding is the same by i) using only as many MBR comparisons as MBR mean for our methods and ii) always using the same smaller effective beam size, which is measured by the number of
7

---

Published as a conference paper at ICLR 2025
IWSLT17 De-En
WMT19 Cs-En
XSUM
MBR Compurisons
Effective Beam size
2 Models
Models
2 Models
Models
Method
BLEU
COMET
BLEU
COMET
BLEU
COMET
R-1
R-L
Single Model
80.24
82.95
26,99
19.05
Sequence-leve -eeeelleeeeee---eeeeeeee Eq.
26.66
81.60
29.12
83.06
30.60
84,12
28.27
20.22
400/900 200/300
20/30
Sequence level Eq.
T0
26.02
81.47
26.50
81.86
30.25
83.99
27.43
19.33
20/30
Table Sequence-level model combination is also useful for ensembling zero-shot prompted LLMs. Eq.
(9) performs better but requires more computation.
78
38
Deep Ens.
COMET
BLEU
Uninodal
Deep Ens.
Unimodal
77.75
Snapshot
36
Temperatur
0
77.5
10
20
30
40
50
60
10
20
30
40
Diversity
Diversity
Figure 1: Our methods more successful when the ensembled models diverse. We compare unimodal to mixture-based posteriors using Snapshot Ensembles and Deep Ensembles. Sampling from unimodal posterior with higher temperature can increase diversity and improve performance (in blue). Left: token-level combination on IWSLT14 using beam search and Transformerpse. Right: sequence-level combination (Eq. (I0) on IWSLT17 using ancestral sampling and Gemma-2B.
beams per model multiplied the number of models. We validate this empirically in App. only do results improve when using word-overlap metries like BLEU, but also when using quality estimation (COMET) and hallucination metrics (LaBSE). Notably, IWSLT17 all improvements vs. standard MBR indicate there is an
B.3 Not
observed COMET score when using uncertainty-awer estimated > >55% chance that humans would distinguish the former system better- per Koemi jet al. 2024). Improvements also hold for the STS-B sentence similarity scoring task. The estimators of Eq. (9) and Eq. (TO) perform similarly even though Eq. ( uses smaller hypothesis size than Eq. 10
Comparison uni- and multimodal posteriors. Next, we compare unimodal posteriors that can be learned without overhead during training to multimodal posteriors based on Deep Ensembles. Such posteriors incur significant overhead during training, because one separate training run with different initialization and data order is required per ensemble member, but can incorporate knowledge from different loss basins-a characteristic that has proven to be beneficial (Lion et al. 2023).
When training from scratch (Tab. 2) unimodal posteriors do not consistently outperform the single model baseline when compute budgets are equivalent. In contrast, multimodal Deep Ensemble posteriors can deliver significant improvements. On the other hand, when finetuning (Tab. [ unimodal posteriors can provide strong improvements, performing on par with Deep Ensembles. We hypothesize that this difference a smaller subspace of potential posterior parameters and may therefore pose a comparably easier learning problem than estimating the variance of posterior over all parameters. Further, finetuning may not work that well for Deep Ensembles due to the models still landing in the same basin (Frankle We connect our findings to prediction diversity in $4.4
be attributed the of LoRA for finetuning-whit explores
al. 2020).
Comparison of sequence- and token-level posteriors. Here, compare the use of sequence- and token-level posteriors (Eqs. (9), (00) and (13)) in MBR. Tab. shows that improvements over the baseline with token-level combination are much stronger when using beam search instead of ancestral sampling to create hypothesis set{ When using a mixture-based posterior, performance improved in both settings. Sequence-level combination, the other hand, provides similar improvements for
Beam search provides biased estimate and is similar to sampling from low-temperature distribution.
8

---

Published as a conference paper at ICLR 2025
40
60
60
Heattteee Sumple USil
Milkree
MRRC
Segehed (hop)
Dapp Eaundll
40
Seacch
35
30
0.4
0.6
0.2
0.4
01.6
0.8
30
0.2
0.4
01.6
0.5
30
0.2
0.4
01.5
0.5
(a) Total sample utility (b) Unimodal, Beam Search
(c) Beam Search
Sampling
Figure 2: Total risk and best-output-risk are useful for selective prediction. (a) Creating hypothesis sets with sampling performs better than beam search. (b) Increasing temperature when sampling from unimodal posteriors improves selective prediction. (c) When using beam search more Deep Ensembles work best. (d) For sampling, methods work well. Results on IWSLT14 14 with TransformeThue-
both settings, with Eq. (9 providing similar results to token-level aggregation. Hence, the preferred method may also depend on the decoding algorithm used to create hypothesis set.
shows results obtained when ensembling the outputs of various zero-shot prompted LLMs on IWSLT17 De-En with a hypothesis set size 10. We compare the estimator using additive union of hypothesis sets (Eq. (9) to using soft model average (Eq. (10) and the average single model performance. Both estimators are effective for ensembling but Eq. performs best, albeit with the highest computational complexity. Details are in App. A.3
Ensembling zero-shot models. Tab. 3
4.4 CORRELATION OF QUALITY AND DIVERSITY
Next, we show that the performance of MBR with weight-uncertaint is correlated with the prediction diversity of ensembled models, potentially, due to incorporating knowledge from multiple loss basins. This is in line with prior works ensembling which have found that diversity is important for good performance (Fort et al. performance (Abe et al. 2022 Wood al. 2023
2019: Masegosa 2020) but can form a trade-off with individual model
We empirically validate this in Fig. 1 where we plot BLEU and COMET on IWSLT14 and IWSLT17 against the prediction diversity. We measure diversity 100 minus average self-BLEU; self-BLEU scores are measured on the set of greedy decoding outputs of each ensemble member, similar to Shen al. (219)) For finetuning, the models from the unimodal posterior are more diverse than when pretraining. The plot shows clear correlation between both metrics. We ask two questions: 1) can diversity promoted in unimodal pretrained posteriors to improve performance and can we find method with the same pretraining overhead as unimodal posterior but more expressiveness?
For the first, note that the variance of the IVON posterior is o² 1/A(h+0)), where h is the expected Hessian of the loss, 8 is weight-decay and À the effective sample size which can be seen as an (inverse) temperature parameter. We decrease À gradually, which samples models from the posterior with higher temperature. This improves diversity and can improve performance. For the latter, we use mixture-of-GGassiaa consisting checkpoints from one training run, denoted by "snapshot" (Huang al. 2017). This comes at training time increase but improve performance by incorporating knowledge from different regions along the optimization trajectory, as shown in Fig.
4.5 SELECTIVE PREDICTION WITH BAYES RISK
Here, we explore the use of expected Bayes risk for selective prediction on IWSLT14, We observe that both the maximum output utility and the expected output utility (i.e., average expected utility across outputs) can be used effectively for selective prediction. Our results are summarized in Fig. 2
First, find in Fig. (a) that using the total expected utility for selective prediction performs slightly better than just using the expected utility of the chosen output. This is especially true when creating hypothesis sets with beam search, which performs much worse than ancestral sampling. Next,

---

Published as a conference paper at ICLR 2025
00
BLEU
Esacabll size
Esscuble size
Rloam cioo
Biorn cloo
Ancestral Sampling
(b) Beam Search
Ancestral Sampling
(d) Beam Search
Figure 3: Scaling behavior on IWSLT14 with TransformeT in terms of ensemble (a, b) and hypothesis set size (c, d). (a, b) For unimodal posterior (D), larger ensembles improve token-level combination using sampling but not beam search. For Deep Ensemble posteriors (o), larger ensembles generally improve performance. (c, d) Sequence-leve combination (Eq. (TO) performs better for smaller beam sizes but is outperformed by token-level combination at larger ones. Scaling the hypothesis set produces stronger improvements for ancestral sampling than beam search.
again sample from the unimodal posterior with different temperatures (via decreasing A). We find that this improves selective prediction with MBR when using beam search (Fig. 2 (b)).
Finally, we evaluate the influence of the posterior approximation. First, we find that hypothesis set built with ancestral sampling reliable independent of the used posterior. Even the single model baseline works well but is outperformed by using ensemble and more expressive posteriors give bigger improvements. For beam search, the baseline completely fails and token-level combination can be unreliable. Sequence-level combination (Eq. (10)) performs much better, especially with more expressive multimodal posteriors. These results are shown Fig. 2 (c. d).
4.6 SCALING BEHAVIOR
Lastly, we examine the scaling behavior of token- and sequence-level combination (Eq. (10) with different posteriors. Results are summarized in Fig. 3 First, we show scaling the ensemble size in Fig. (a) for ancestral sampling and beam search (b). Using beam search, both token- (in blue) and sequence-level (in black) combination using unimodal posteriors provide no improvements. For ancestral sampling, we find improvements with a unimodal posterior, especially at larger ensemble sizes of 32 models, but sequence-level combination of a unimodal posterior only improves until 4 models. In all other settings, scaling the ensemble size usually beneficial.
When scaling hypothesis sets with beam search, the improvements small, likely because the hypothesis sets lack diversity. Ancestral sampling shows a different picture and we obtain strong improvements when scaling hypothesis sets. For small hypothesis sets it is better to use sequence-level ensembling but for larger sizes level- combination can be better.
CONCLUSION
In this work, we explore using Minimum Bayes Risk approach to account for weight uncertainty in language model decoding. We investigate different methods which combine predictions from multiple models either during generation or afterwards. Importantly, the latter can be used to ensemble any set LLMs. We benchmark the methods on different language generation and scoring tasks for prompted and finetuned models, as well as models trained from scratch. We show that weight uncertainty can effectively improve decoding. We evaluate the effects of using different posterior distributions. More complex distributions can sometimes provide stronger performance improvements but also simple methods can improve performance. Crucially, the improvements with simpler methods not incur training test-time overhead. We also connect our findings to prediction diversity, which is important for both standard MBR and when using its expected utility for selective prediction, and show that improvements scale with model and sample sizes. Overall, find that the uncertainty- variant of MBR proposed this paper leads to better and more robust language generation. Altogether, our method provides principled approach for scaling test-time compute.
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our work uses probabilistic language models to generate language. Even when used with care, such models can produce outputs that are, among others, harmful, toxic, and hallucinated and our methods can not guarantee that such outputs are not generated. However, we aim to improve the robustness of language generation methods and, therefore, aim alleviate these issues. Therefore, we believe there to be no direct ethical concern in our work.
ACKNOWLEDGEMENTS
This project has received funding by the German Federal Ministry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the National Research Center for Applied Cybersecurity ATHENE. Clara Meister was supported by Google PhD Fellowship. This work is supported by JST CREST Grant Number JP-MJCR21
We thank Seyed Arshan Dalili for help with running the LLM experiments in Tab.
REFERENCES
Taiga Abe, E. Kelly Buchanan, Geoff Pleiss, and John Patrick Cunningham. The best deep ensembles sacrifice predictive diversity. In Can't Believe s Not Better Workshop: Understanding Deep
Learning Through Empirical Falsification, 2022. URL httt/////n/////eeeeeeeeeeee/eeer hwtp://wwwweeeeooeeeeeieeee 6sBiAIpkUi0
Pierre Alquier. User-friendly introduction to pac-bayes bounds. Foundations and Trends@ in Machine Learning. 17(2): -03 2024. ISSN 1935-8237. doi: 10.1561/2200000000,, URL
http://dx doi wtt/10..///// 1561/22000001688
Loic Barrault, Ondřej Bojar, Marta R. Costa-jusså, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Koehn, Shervin Malmasi, Christof Monz, Mathias Müller, Santanu Pal, Matt Post, and Marcos Zampieri. Findings of the 2019 conference machine translation (WMT19). In Ondfej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, André Martins. Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Fourth Conference on Machine Translation (Volume Shared Task Papers, Day 1), pp. 1-61, Florence, Italy, August 2019. Association for Computational Linguistics. doi: 10.18653/v1/1195555
URL
Prabin Bhandari and Hannah Brennan. Trustworthiness of children stories generated by large language models. In C. Maria Keet, Hung-Yi Yi Lee, and Sina Zarrieß (eds.), Proceedings of the 16th International Natural Language Generation Conference, pp. 352-361, Prague, Czechia, September 2023, Association for Computational Linguistics. doi: 10.18653/v1/2022 10t.53//11222..nn/// 24. URL www.aaaannnoooooooooo////o inlg-main.2
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in
neural network. International conference on machine learning, pp. 1613-1622. PMLR, 2015.
URL httpps://proceeeinggss mlr, http://eeddnnnnsmmrmrrrrss////// vtress/v37/lludde1113 html
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ale Tamchyna. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 12-58, Baltimore, Maryland, USA. June 2014. Association for Computational Linguisties. URL http://www aclweb. org. wwwpp/rrgg/ahhhhoooeo
3302
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. Findings of the 2017 conference on machine translation (WMT17). In Ondfej Bojar, Christian Buck, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Mathiis Huck, Antonio Jimeno Yepes, Philipp Koehn,
11

---

Published as a conference paper at ICLR 2025
and Julia Kreutzer (eds.), Proceedings of the Second Conference Machine Translation, pp- (69,214, Copenhagen, Denmark. September 2017. Association for Computational Linguistics. doi: NE111655/11/W11-40
URL http:tss//ac.aattoooeyyr.r/////2
Ondrej Bojar, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Philipp Kochn, and Christof Monz. Findings of the 2018 conference machine translation (WMT18). In Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Shared Task Papers, pp. 272-303, Belgium, Brussels, October 2018. Association for Computational Linguistics, doi: 10.1865//1/W18-64000 URL / http:///d..aattoooooo...////11/1 0rg//18-6661111
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2011 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Steven Bethard, Marine Carpuat, Marianna Apidianaki, Saif M. Mohammad, Daniel Cer, and David Jurgens (eds.), Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017) pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10.18653/v1/S17-2000
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th IWSLT evaluation campaign. In Marcello Federico, Sebastian Stüker, and François Yvon (eds.), Proceedings of the 11th International Workshop on Spoken Language Translation: Evaluation Campaign, pp. 17, Lake Tahoe, California, December 4-5 2014. URL. https hraphoooooooooog////// o 2014. messstt-vallaiion.
Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Nichues, Sebastian Stüker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation campaign. In Sakriani Sakti and Masao Utiyama (eds.), Proceedings of the 14th International Conference on Spoken Language Translation, pp. 2-14, Tokyo, Japan, December 14-15 2017. International Workshop on Spoken Language Translation. URLhhtpp::/aallaatholoyyy org/ iwslt-1
Julius Cheng and Andreas Vlachos. Faster minimum Bayes risk decoding with confidence- pruning. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Natural Language Processing, pp. 12473-12480, Singapore, December URL
2023. Association for Computational Linguistics. doi: 10.18653/ 1/26:///112223..mmmmmmmmaaa https::/aclanthology..oom wwwpetttwooooooooo///22/2ee www.or////......//mmeeee.. meallp-mmin....
Eldan Cohen and Christopher Beck. Empirical analysis of beam search performance degradation in neural sequence models. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 1290-1299. PMLR, 09-15 Jun 2019. URL httttttss////oceeeiiee html
mlr press/v97/coher 9a
Bai Cong, Nico Daheim, Yuesong Shen, Daniel Cremers, Rio Yokota, Mohammad Emtiyaz Khan, and Thomas Möllenhoff. Variational low-rank adaptation using IVON. In NeurIPS 2024 Workshop on Fine-Tuning in Modern Machine Learning: Principles and Scalability, 2024. URL httpss
David Dale, Elena Voita, Loie Barrault, and Marta R. Costa-jusså. Detecting and mitigating hal- lucinations in machine translation: Model internal workings alone do well, sentence similarity Even better. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 36-50, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: w1w.//////223..aa///aaa
URL httptht:///aaaaatoollla/// www.ooggoooooooooooooooooo
2org/2022 2023. acl-long.2
Erik Daxberger, Agustinus Kristiadi, Alexander Immer, Runa Eschenhagen, Matthias Bauer, and Philipp Hennig. Laplace effureeffortless NeurIPS, 2021. URL htt.uttttp//pprrrennnn..eeer cc/paper ftlpsssceccpppeeeee//ee///a/aa Ttlles/paper/222//// 77c5557aadd7544443000eeeeeaaaaa3
Bayesian deep learning. In
12

---

Published as a conference paper at ICLR 2025
Morris H DeGroot. Optimal statistical decisions. John Wiley & Sons, 2005. URL https:// onlinelibrary. wwlewy.ll...............e....w wttpey.coommooooooooo/eoe 10. 108.2007172779000
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.2 2024. URL BRtphsts:///axxii org/ abs/ 2407 21783
Bryan Eikema and Wilker Aziz. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.), Proceedings of the 28th International Conference on Computational Linguistics, pp. 4506-4520, Barcelona, Spain (Online), December 2020. International Committee Computational Linguistics. doi: 10.18653/ v1/2020.coling-mainn maingg...inn. URL htt.t::://cclaatooooo..oooro wrw.l/gggoorg//20 wtw.:o///222..ooonnnmemeeeee coling-main. 398
Bryan Eikema and Wilker Aziz. Sampling- approximations to minimum Bayes risk decoding for neural machine translation. In Yoav Goldberg, Zormitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp. 10978-10993, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18663/1/22222.emms mainnllppmi..... main. 754. URL https: https::/aalanntoloogy 002/2022 emn main..mmainn22 754
Ekaterina Fadeeva, Aleksandr Rubashevskii, Artem Shelmanov, Sergey Petrakov, Haonan Li, Hamdy Mubarak, Evgenii Tsymbalov, Gleb Kuzmin, Alexander Panchenko, Timothy Baldwin, Preslav Nakov, and Maxim Panov. Fact-checking the output of large language models via token-level uncertainty quantification. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics ACI. 2024, pp. 9367-9385, Bangkok, Thailand and virtual meeting, August 2024. Association for Computational Linguistics. URL https //aclanthology.org/22 wtt.e//.///2222...iiio..e..e findings-acl. 558
Angela Fan, Mike Lewis, and Yann Dauphin. Hicrarchical neural story generation. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 889-898, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10. 1188633////P1// 1082. URL|https //acl //ac.lantolloy.orgg//eo 1ww.://oy..or//1-1118988
António Farinhas, José de Souza, and Andre Martins. An empirical study of translation hypothesis ensembling with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 11956-11970, Singapore, December 2023. Association for Computational Linguisties. doi: 10. http3//w1/223.mmmlpp main. 7333. URL htttts://cclanthoo...g
org/ 0023. emnlp- emnnp-mmin... 733
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. Language-agnostic BERT sentence embedding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
I: Long Papers). pp. 878-891, Dublin, Ireland, May 2022. Association for Computational Linguis- tics. doi: 10.18653/v1 1/022//./22222c//c// acl-long.62. URL http:::://aclaanhooooooo//////// acl-loo 62
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. Quality-aware decoding for neural machine translation. In Ma- rine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 1412, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 10.18653/w1/0222.maaa naacl main. URL https://aclannthooogs hwwpthhhoogg.ooooo//..ooo org.//g///2222 maacclmainng 100
Mara Finkelstein and Markus Freitag. MBR and QE finetuning: Training-time distillation of the best and most expensive decoding methods. In The Twelfth International Conference on Learning Representations, 2024. URL
Stanislav Fort, Huiyi Hu, and Balaji Lakshminarayanan. Deep Deep ensembles: A loss landscape perspec- tive. arXiv preprint 19Xiv 2019. http:htttss///ddv.o////bbb/////p 02757
13

---

Published as a conference paper at ICLR 2025
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings of Machine Learning Research, pp. $259-32699 PMLR, 13-18 Jul 2020. URL htt.ttts///prrceeennggs mlr
press/v119/frankle220 html
Markus Freitag. David Grangier, Qijun Tan, and Bowen Liang. High quality rather than high model probability: Minimum Bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics, 10:811-825, 2022. doi: 10. 10.1162/tac1. a ..049. URL
httpp:://ac.aantooooo..oor//////
tac1-1.47
Yonatan Geifman and Ran EI- Yaniv. Selective classificatton for deep neural networks. Advances in neural information processing systeems, 30, 2017. URL /tthp//utttttsss//aaaes nips. hatppppppfflless.
Gemma Team. Gemma 2: Improving open language models at a practical size. arXiv preprint
arXiv:2408.00118, 2024a. URL httl:/ts///aaxxv.or///////////// 00144088811108
Gemma Team. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08225, 2024b. URLLhttss///axxi..orr/a/////// 08295
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Lu Wang. Jackie Chi Kit Cheung. Giuseppe Carenini, and Fei Liu (eds.), Proceedings of the 2nd Workshop on New Frontiers in Summarization, pp. 70-79, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 1 1//33/11/D19-54/ URL httpttts////caaanntoolg.. orgg
Jesús Gonrález-Rubio, Alfons Juan, and Francisco Casacuberta. Minimum Bayes-risk system combination. In Dekang Lin, Yuji Matsumoto, and Rada Mihalcea (eds.), Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. pp. 1268-12277 Portland, Oregon, USA, June 2011. Association for Computational Linguistics. URL hattps://aclaanthooogg http:///_aantoooooo..r///1111111 org/P11-1127
Alex Graves. Practical variational inference for neural networks. Advances in neural information pro- cessing systems, 24, 2011. URL httpphh nips. htt////pprrllllle/pppr//////////
John Hewitt, Christopher Manning, and Percy Liang. Truncation sampling as language model desmoothing. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 3414-3427, Abu Dhabi, United Arab Emirates, December 2022, Association for Computational Linguistics. doi: 18.186/3/v1/ findings-emnll.255 URL https:ttp///o httpps://aclaanhhooogg.. .rg/ 2022. findings-emnlp. 249
Anas Himmi, Guillaume Staerman, Marine Picot, Pierre Colombo, and Nuno M Guerreiro. Enhanced hallucination detection in neural machine translation through simple detector aggregation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 18573-188 -1553,, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 10..18553vvll 1665/////222ccmmmppmm 1033. URL /tttpss://ach /ttt/ anthlllgy /rg/ 2024. emmlp-main. 1033/
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2020. URL https:
Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022. URL https: http:///peeeevieeeeeeeeee nett/orrmmiddnneWWeerr
Gao Huang, Yixuan Li, Geoff Pleiss, Zhuang Liu, John E. Hoperoft, and Kilian Q. Weinberger. Snap- shot ensembles: Train 1, get m for free. In International Conference on Learning Representations, 2017. URIIhttpss: http:p/://ogeerreeeeee/////////s www.nt//foruum////JJJWW]]]]]
14

---

Published as a conference paper at ICLR 2025
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Ame Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. /tt.https://aaxiiv.. / abs/2310. 06825
Mohammad Emtiyaz Khan and Håvard Rue. The baycsian learning rule. Journal of Machine Learning Research, 24(281):1 (4281))1-466, 2023. URL http:::tp///mr..oo pttppps//2//2222222222 /ttt::///.....rrr///o/s///////21 html
Hayato Kobayashi. Frustratingly easy model ensemble for abstractive summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun ichh Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 4165-4176, Brussels, Belgium, October-Novemble 2018. Association for Computational Linguisties. doi: 18653/v1//1S 1449. URL httppsss/aaaaantoooogg..ooo htwp://g....///---//////
Tom Koemi, Vilém Zouhar, Christian Federmann, and Matt Post. Navigating the metries maze: Reconciling score magnitudes and accuracies. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1999-2014, Bangkok, Thailand, August 2024. Association for Computational Linguistics. htt/httss////caanttt
canggy.oor. .rgg/ 2e1///2224.000 acl 110
Wojeiech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1332-9346, Online, November 2020. Association for Computational Linguistics. doi: 10.186533/ 1/203/1//222./2// cemllp main. 750. URL http:tttc/cclaatholllg//oo orgo////2/222 emmlppmmiin. 750
Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. Semantic uncertainty: Linguistic invariances for uncertainty estimation natural language generation. In The Eleventh International Conference on Learming Representations, 2023. URL https:///pppeereeeeew wttp://wwwntfformteeeeemm net/ffrrum?id-VD-AYYEee
Shankar Kumar and William Byrne. Minimum Bayes-risk word alignments of bilingual texts. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing (EMNLP 2002), pp. 140-147. Association for Computational Linguistics, July 2002. doi: 10.3115/ 1118693. 18712. URL httpps//aaa.aattooooooor///////p
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable pre- dictive uncertainty estimation using deep ensembles. Advances in neural information pro- cessing systems, 30, 2017. URL httpttttt///pprrsss mimpperrrsnnss...mmmm. wwwpccpppppp..eeee httpp//c//pppeefllle//////////// 9ef2ed4b7fd2c8 htw.:////2222244444aaaaaa3333333 a8araaa8bbbce888 aaaar7/ff...aacceeeabbbtrrr..... html
Junyi Li, Jie Chen, Ruiyang Ren, Xiaoxue Cheng, Xin Zhao, Jian- Nie, and Ji-Rong Wen. The dawn after the dark: An empirical study on factuality hallucination in large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp. 10879 10899, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL acl long. 18ng.586 586
httpp::/aac..nthoooo...or//////
Chin-Yew Lin. ROUGE: package for automatic evaluation of summaries. In Text Summarization Branches Out, 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics. URL htt.tttsss/allaattoo..g.. hwwpoolloyyyooogggggoooo
Kai Lion, Gregor Bachmann, Lorenzo Noci, and Thomas Hofmann. How good is single basin? In UniReps: the First Workshop on Unifying Representations in Neural Models, 2023. URL
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019. URL httpps://peerevview. wttp:/wwwwwweeewtntiieeei
Bkg6RiCqy7
15

---

Published as a conference paper at ICLR 2025
Michal Lukasik, Harikrishna Narasimhan, Aditya Krishna Menon, Felix Yu, and Sanjiv Kumar. Regression aware inference with LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguisties: EMNLP 2024, pp. 13667-13678, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 18.18653/ v1/2024.fndins-eemnlp
/0gp:///oogorrg/2222..o/ URL htt.ittss///cllanhoooooo..oorrr org wwn..........nn..... findings-ennlia 799/
David John Cameron Mackay. Bayesian methods for adaptive models. California Institute of Technology, 1992. http::htts///heeeee cibracccatcccccccccccccc cetttteeecccc.ceec/////
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gordon Wilson. A simple baseline for bayesian uncertainty in deep learning, Advances in neural information pro- cessing systems, 32, 2019. URL httpttt:::::ppeeeeeeeeeppepp nips. caaapperfiles/pappppa (13922eebb233322266567777766 habpsaiict.cc-at.tttttttt..t..
9/hash/
Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction. In International Conference on Learning Representations, 2021. URL htttsss///ooeeeeeiiieee net/
forum?1d==jSy-225500
Andres Masegosa. Learning under model misspecification: Applications varia- tional and ensemble methods. tems, 33:5479-5491, 2020. URL https:// neurips. httppppppccpppp/ppppppeppppeeppp heaaper/2220/hhh/ hash/
Advances in Neural Information Processing Syy-
Thomas Möllenhoff and Mohammad Emtiyaz Khan. SAM as an optimal relaxation of bayes. In The Eleventh International Conference on Learning Representations, 2023. URL https
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Ellen Riloff, David Chiang, Julia Hockenmaier, and annichhi Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Longuage Processing, pp. 1797-1807, Brussels, Belgium, October- November 2018. Association for Computational Linguistics. doi:
1206. URL
https://aclantholog..rr/////////
Jekaterina Novikova, Ondrej Dušek, and Verena Rieser. The E2E dataset: New challenges for end-to-end generation. In Proceedings of the 18th Annual Meeting of the Special Interest Group on Discourse and Dialogue, Saarbrückenn Germany, 2017. URL http://////xxw.or////ooo
009254]
Kazuki Osawa, Siddharth Swaroop, Mohammad Emtiyaz. E Khan, Anirudh Jain, Runa Eschenhagen, Richard E Turner, and Rio Yokota. Practical deep learning with bayesian principles. Advances in neural information processing systems, 32, 2019, URL https: httls:://pppeeee httppppppss..///////// nips. cc/paper files/ Abstract.html
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. fairseq: fast, extensible toolkit for sequence modeling. In Waleed Ammar, Annie Louis, and Nasrin Mostafazadeh (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)) pp. 48-53, Minneapolis, Minnesota, June 2019, Association for Computational Linguistics. doi: 10.18653/v1/ N19-4009. URL/https///acaaahhhllgg anhooooy org orgo (11e/N19-440 9-00
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Pierre Isabelle, Eugene Charniak, and Dekang Lin (eds.), Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 00.31115/077003.10333333 URL /t//////////////aaaaaaaoooorrr/r 1040
Maja Popović. chrlP: character n-gram F-score for automatic MT evaluation. In Ondřej Bojar, Rajan Chatterjee. Christian Federmann, Barry Haddow, Chris Hokamp, Matthias Huck, Varvara Logacheva, and Pavel Pecina (eds.), Proceedings of the Tenth Workshop Statistical Machine Translation, pp. 392-395, Lisbon. Portugal, September 2015. Association for Computational Utk.hhttps///aclaattoooogg /ttp////oooooo///////333333
Linguistics. doi: 10.
16

---

Published as a conference paper at ICLR 2025
Matt Post. A call for clarity in reporting BLEU scores. In Ondiej Bojar, Rajen Chatterjee, Christian Federmann, Mark Fishel. Yvette Graham, Barry Haddow. Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Matt Post, Lucia Specia, Marco Turchi, and Karin Verspoor (eds.), Proceedings of the Third Conference on Machine Translation: Research Papers, pp. 186-191, Brussels, Belgium, Octoher 2018. Association for Computational Linguistics. doi: 10.18653/11/118-666
URL https:://alaanthll..o/ org/
W18-6319
Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. COMET-22: Unbabel-IST 2022 submission for the metries shared task. In Philipp Koehn, Loic Barrault, Ondiej Bojar, Fethi Bougares, Rajen Chatterjee, Marta R. Costa-jussä, Christian Federmann, Mark Fishel, Alexander Fraser, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Paco Guzman, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Tom Koemi, André Martins, Makoto Morishita, Christof Monz, Masaaki Nagata, Toshiaki Nakazawa, Matteo Negri. Aurélie Névéol, Mariana Neves, Martin Popel, Marco Turchi, and Marcos Zampieri (eds.), Proceedings of the Seventh Conference on Machine Translation ( (WM7), pp. 578-585, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguisties. URL pettps:://alaanthollog .rg. /2t.//2222.. wmi 52
Machel Reid, Junjie Hu, Graham Neubig, and Yutaka Matsuo. AfroMT: Pretraining strategies and reproducible benchmarks for translation of 8 African languages. In
Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pp- 1306- 1320, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/ v1/2021 wwwnle-mainnn..... URL htttss:/aclanttolll../ org 2221. wtt.:r/rg22.....mmmmam...... 99
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Kentaro Inui, Jing Jiang, Vincent Ng. and Xiaojun Wan (eds.), Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982 9992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.11655//1/D11-14410
URL https://aclanttho..gr.
Jie Ren, Jiaming Luo, Yao Zhao, Kundan Krishna, Mohammad Saleh, Balaji Lakshminarayanan. and Peter Liu. Out-of-distribution detection and selective generation for conditional language models. In International Conference Learning Representations, 2023. URL https:// het//forummid=kkUSSSDDDoPPP
Christian P. Robert. The Bayesian choice: from tecision-theoretic foundationn to computational implementation, volume 2. Springer, 2007. UtL/httpss://ooi.orrg
1807/00-877-71599
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 1715-1725, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10. 11885///1/P16--. 1162..
URL https:
org/p16-1162
Tianxiao Shen, Myle Ott, Michael Auli, and Marc' Aurelio Ranzato. Mixture models for diverse machine translation: Tricks of the trade. In International conference on machine learning, 2019.
URL U.R.httpss//proceeedngg.uu /ttttp///orrceeddos.s..///////// wwtps//97//sheeeeeeeee 9c. html
Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Clement Bazan, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas Moellenhoff. Variational learning is effective for large deep networks. ICML, 2024. URL http::tt/////eeeiieeee netz/forum2id<cNBVV076kk
Noah A. Smith. Linguistic Structure Prediction. Synthesis Lectures on Human Language Technolo- gies. Morgan and Claypool, May 2011. URL https:://inkk.st imppr.ing.....
htwps//ennnnn.coooooooooe// wwt.///o/ww..o..//// 1007/
978-3-031-02143-5
Felix Stahlberg and Bill Byme. On NMT search errors and model errors: Cat got your tongue?
In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceedings of the 2019
17

---

Published as a conference paper at ICLR 2025
Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3356-3362, Hong Kong, China, November 2019. Association for Computational Linguisties. doi: 10. 10.18653/11/D19-1333 URL www.httss//accanntol...g... htgp:/y-y...///1111//0/
Mirac Suzgun, Luke Veras-Kyriazzi, and Dan Jurafsky. Follow the wisdom of the crowd: Effective text generation via minimum Bayes risk decoding. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACI. 2023, pp. 4265-4293, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 16653/v1/20233 www.:/2....nninnaaaa..22.. URL
orgggy...rrg/oo wtgg//23///. findings-acl. 262
Jannis Vamvas and Rico Sennrich. Linear-time minimum Bayes risk decoding with reference aggregation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics ( (Volume 2: Short Papers), pp. 790-801, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: URL
10.18663/v1/2024.acl--e
wwwpy///g222442/ acl-short.7l 71
Liam der Poel, Ryan Cotterell, and Clara Meister. Mutual information alleviates hallucinations in abstractive summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5956-5965, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: mmnlp-main... 399
URL https://aalaantoologgy orgp////rrr//22
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30. Curran Asso- ciates, Inc., 2017. URL /utp//ttss//prrcceeiigg neuriss. ceeerrip..ccpppperrr httppppprr.illlle/pp//////////// 3f5ee243547dee911 hodoaeeeeeaetd000111ccccc444aaa3
Florian Wenzel, Kevin Roth, Bastiaan Veeling, Jakub Swiatkowski, Linh Tran, Stephan Mandt, Jasper Snoek, Tim Salimans, Rodolphe Jenatton, and Sebastian Nowozin. How good is the Bayes posterior in deep neural networks really? In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pp. 10248-10259. PMLR, 13-18 Jul 2020. URL httpss://rrcceeeeee 20a. 20tm
mlr
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the thrt--rrt natural language processing. In Qun Liu and David Schlangen (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38-45, Online, October 2020. Association for Computational Linguistics. doi: 10. 10.1655//112222eemlpppmme 6. URL https::/aclanthhnn www.llooy.ooogoogooooooo 2020 www.meemllppeeem demos.
Danny Wood, Tingting Mu, Andrew M Webb, Henry WJ Reeve, Mikel Lujan, and Gavin Brown. A unified theory of diversity in ensemble learning. Journal of Machine Learning Research, 24(359):
1-49, 2023. JRL/https:///mlr
org/ http:///...pppppp///eeee/eeeee/e pwtpppa/e//////////aep 0041. html
Ian Wu, Patrick Fernandes, Amanda Bertsch, Seungone Kim, Sina Khoshfetrat Pakazad, and Graham Neubig. Better inslruction-folloiingg through minimum bayes risk. In The Thirteenth International Conference Learning Representations, 2025. URL https://oppenreiiw.eee/eeeee 7XCSK9BLPy
Yijun Xiao and William Yang Wang. On hallucination and predictive uncertainty in conditional language generation. In Paola Merlo, Jorg Tiedemann, and Reut Tsarfaty (eds.). Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 2734-2744, Online, April 2021. Association for Computational Linguistics. doi: 10.18653/v1/202 eacl- main.234 236. URL httpph/s/aaacaanoooooooooo//////
facl-main. 236
18

---

Published as a conference paper at ICLR 2025
Adam X. Yang, Maxime Robeyns, Xi Wang. and Laurence Aitchison. Bayesian low-rank adaptation for large language models. In The Twelfth International Conference on Learning Representations, 2024a. URL /ttpss:s:///ppeereiiiie wwtp:/eeeoooommmdfFiiii...
1m
Guangyu Yang. Jinghong Chen, Weizhe Lin, and Bill Byrne. Direct preference optimization for neural machine translation with minimum Bayes risk decoding. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 2: Short Papers), pp. 391-398, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 12.18663w1/222244aaall--tooo naacl-short.34
URL https://aclaanthologgy org /2224
An Yang et al. Qwen2 technical report, 2024. nttn
10671
Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. Cognitive mirage: review of hallucinations in large language models. arXiv preprint arXiv:230906699,, 2023. URL
https://arrxw.orr//bb///2/ 06794
Tianyi Zhang. Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating
text generation with bert. In International Conference on Learning Representations, 2020. URL
https://openrevieew nett/forum?id--ketuccee
A EXPERIMENTAL DETAILS
A.1 TRAINING FROM SCRATCH
Datasets Our usage of the WMT14 English-to-German translation tasks (Bojar et al. 2014) follows the set-up from (Vaswani et al.
2017) but augments the training data by the news-commenttry--/12 2017). In total, we train on ca. 3.9M paired examples. We also use a validation set during traiming order to pick checkpoints which consists of ca 39.4K examples. We use the original newstest2014 data which consists of 3,003 examples for evaluation.
data from WMT17 (Bojar et al.
We also the IWSLT14 German-to-Han Englishh translation task (Cettolo et al.. 2014) which consists of ca 160K training examples. The validation set consists of ca. 7.3K examples. The test set consists of 6,750K examples.
Furthermore, we use two language pairs from AfroMT (Reid et al. 2021), namely En-Bem (English- Bemba) which consists of 275K training, 3K validation, and 3K test examples. We do not use any monolingual data but only train from scratch on the parallel data. We use En-Run (English-Rung Rundi) in the same way, which consists of 253K training. 3k validation, and 3k test examples.
All data usages can be reproduced by following the instructions from the Fairseq repository under https://gii thubbbb...tt.... com/f ccm/facceie aaaaoooreeeerrcc/aaaaaoo and will be published along our code.
Models All models follow the Transformer architecture from Vaswani et al, (2017) which consists of encoder-decoder Transformer with 6 encoder and 6 decoder layers. We use the Transformen architecture for IWSLT2014 4 and afroMT and Transformenhi for WMT14 which has larger embedding and feed forward dimensions. The models use a vocabulary of Satte-Par-EEncoding tokens (Sennrich et al. 2016). The input and output embedding parameters of the decoder are shared. The IWSLT model has an input vocabulary size of 8848 and an output vocabulary size of 6632 for in total 39, 469, 056 parameters. The en-run and en- bem models both have an input and output vocabulary size of 80000 each and a total of 126, 058, 496 parameters. The WMT model has an input vocabulary size of 40480 and an output vocabulary size of 42720 for total of 261, 431, 296 parameters.
Training& & Decoding We train all models from scratch using the fairseq library (Ot et al. 2019) which we extend for variational learning and a Bayesian interpretation of neural networks. Fairseq is licensed under MIT license [I] which permits our form of usage. We will release our code publicly
19

---

Published as a conference paper at ICLR 2025
Dataset
Instruction
IWSLT17 En-De
Translate from English to German:
WMT18 Tr-Fi
Translate from Turkish to Finglish:
XSUM
Summarize:
SamSum
Summarize:
E2E-NLG
Convert set of two-to-nine key-value attribute pairs in the restaurant domain simple text: How similar are these sentences from 1??
STSB
Table 4: Simple instructions used when finetuning Gemma-2B-it.
in the future for further research in software repository under Apache License 2. We train all models with the IVON optimizer (Shen et al. 2024) and place diagonal Gaussian posterior over neural networks. We use IVON with a isotropic Gaussian prior and initialize all entries of the Hessian with 0. We use an effective sample size of 10-8, a small weight-decay of 0.0001, and learning rate of 0.1. We set 31 0.9 and 32 0.9999, All models are trained with batch size of 32 or up to 1024 tokens and we use MC samples from the posterior during training for afroMT and IWSLT2014 14. For WMT14 we just use MC sample due to the heavier compute requirements. We clip gradients elementwise at 0.001 and use a dropout rate of 0.2. We train the models until performance in terms of BLEU has not improved for least epochs and then stop with the exception for WMT14, where we train only up to 20 epochs. The results for the single model baseline and unimodal posterior are averaged over four runs.
For the snapshot-like like approach, we add randomly-sample distributions that were trained with least 10 epochs to the best-performing one. For Deep Ensembles we always use four runs with different random seeds unless stated otherwise and for unimodal posteriors we sample four models from each posterior. In all experiments we sample from the posterior "as-is" and only vary the temperature by reducing the effective sample size when explicitly mentioned.
All models are trained on a single GPU which is an NVIDIA GPU with either 80GB, 40GB, 32GB or 24GB GPU memory. Training takes around 1-3 hours for the IWSLT14 and afroMT models and ;2 days for the WMT models.
Following prior work, we use a length-penalty of 0.6 for decoding Vaasaani et al. 2017
A.2 FINETUNING
Datasets For all datasets we use the versions from the huggingface hub (https://huggiingface co/
co/). We use the En-De split of the IWSLT17 evaluation campaign (https://hugg
datasets
http://TTTTWSS//000 Catasets/IwSLT/iss.ee 12017 (Cettolo et al. 2017) with 206,122 training and 8079 test examples and the WMT18 Tr-En split (https://hugginff htt.://oaaadaa//eeeeeeeee wmttt 8) (Bojar et al. (018) with 205,756 training and 3,000 test examples for machine translation. For summarization periments, we use XSUM
http:///wggnnnfccccc.co////tssss al. 2018) and SAMSum (httoss:/huugginggaae. ct /tt/aatasett///assunggg/oo/ samsum) (Gliwa et al.
(Narayan)
2019). XSUM has 204,045 training examples-we train only on the first 50% to reduce computational load and 11,334 test examples. SAMSum much smaller and consists only of 14,732 train and 819 test examples. Finally, we use E2E-NLG (https://huggill http:://hgggnngagooe
wtt.at/atat.///ot/eoo/oeoo htttaatttttettttttttteeeeeeeeeee tusthchkk/ (ee nlg) (Novikova et al. (2017) with 33,524 train and 1,846 test examples for data-to tata-to-text generation. ace. wtt.aaaee..oooaaeeeeooooooe to/datasets// ttttttssssssetteeeeeeeee trans jet al. 2017) with 5,749 train and 1,379 test examples for sentence similarity scoring. Note that we the version provided with the sentence transformers library (Reimers & Gurevych] 2019) which uses ratings from to
as well as STS-B
(Cer)
Models For finetuning results, we use the Gemma-2B-i (Gemma Team 2024b) checkpoint, which can be found under httpp:://uugginggfaae www.ac..co..oooooooeeeeeeeeooooe wowg//ogll/gggo in total 2.5IB parameters.
www.oooggggggaaaaaooooog gew.a//mmaaaaa 2b-it on the huggingface hub, with
Training & Decoding We finetune the model using LoRA Huu et al. 2022) with a rank r = 32 and a dropout rate of 0.1. In total, this introduces 921, 600 new parameters that are learned with IVON and, correspondingly, the diagonal variance consists of 921, 600 further parameters that
hhttps://www.apctttt: www.aappch..rr/lienee/////Lsssse
20

---

Published as a conference paper at ICLR 2025
Dutaset
Instruction
IWSLT17 Da-En
Translute the following English text to German. Make sure to only generate the translation without extra text:
WMT19 Cs-En
Translate the following Czech to English. Make to only generate the translation without extra text: Given BBC article, write short summury of the article in one sentence.
XSUM
Table 5: Prompts used for zero-shot experiments.
are learned. We use the chat template provided with huggingface (Woli et al. 2020), which we adapt to organize our experiments in line with the Apache 2.0 license it is distributed under, to organize training and decoding. As we use an instruction-tun dataset which outlined in Tab. 4 We train the model both the prompt and the output labels and do not only calculate gradients for the latter.
model, we use simple instructions for each
We again use IVON to learn unimodal diagonal Gaussian posterior. We use four separate runs with different random seeds for the Deep Ensembles (which entails different data order and initialization of new parameters) and sample four models for the unimodal posterior. Results for the unimodal posterior and single model baseline are averaged over four seeds. For all experiments we use the same hyperparameter setting. We use an initial learning rate of 0.03 which we anneal cosine decay. We set 3 = 0.9, 32 0.99999, and use a small weight decay of 10-6. We again clip gradients to unit norm and element- -ise with maximum value of 0.001, All hessian values are initialized at 0.0003. We set the effective sample size (or inverse temperature) to 10? for training but 10 for decoding, because we have found this to perform better empirically, potentially due to the cold posterior effect (Wenzel et al. 2020).
with
For training with AdamW, we set (81, (2) = (0.9, 0.999) and perform sweep over learning rates {1 10-5, 1 10-4, 5 10-4 4). We again anneal the learning rates to O, set a small weight decay of 10 6 and rescale gradients to unit norm but do not clip them element-wise.
We train for 1 epoch for IWSLT17 and XSUM, epochs for E2ENLG, 2 epochs for WMT18, and for 4 epochs on SamSUM. We always take the final checkpoints after training has ended.
A.3 ZERO-SHOT RESULTS
In addition trained models, also evaluate zero-shot prompted models. While we do not have an explicit posterior in this setting, ensembling such models can be understood as a crude approximation to sampling from the unknown Bayes posterior.
Datasets In addition to IWSLT17 De-En and XSUM, which are described in App. A.2 the Cs-En partition of WMT19 (https: / htwggggggggggggeeeeeeeeeeegg ace 2019). On XSUM we only evaluate on the first 1000 examples of the test set due to computational load.
we use
Barrault et
Models We use different models for our experiments. In particular, we use Gemma-2 9B httt:s//wugggnnggeeee..oooo////m
(Gemma Team 2024a). Llama-3 8B
7B httpps///hhgggnnnnoecccc/eeeeeen -nn (Dubey et al. 2024). Mis-
3) (Jiang et al. 2023).
and Qwen-2 7B (https://hugiingfaae.co wwtp:///wweee//eee2----------nne use the Instruttion-unnc version of each model. We select the models used for each dataset based on manual inspection of their performance returned czech text when asked to translate from ezech to english and was therefore not included in the experiment, and Mistral tended to produce too long summaries for XSUM when compared to other models. We use the following models for each dataset: Gemma-2, Llama-3, and Mistral for IWSLT17, Gemma-2, Qwen-2, Llama-3 for XSUM, and Llama-3 and Mistral for WMT19. The prompts are shown in Tab. Our prompt for XSUM is taken from (Suzgun et al. 2023).
(Yang et al. 2024). We
each dataset. For example. Gemma sometimes
Decoding
We ancestral sampling with temperature of 1.0 for all experiments.
21

---

Published as a conference paper at ICLR 2025
AfroMT En-Bem
AfroMT En-Run
MBR
Sumpling
Beam Search
Sampling
Beam Search
MBR
Effective
BLEU
chrF
BLEU
chrF
BLEU
chrF
BLEU
chri
compurisoms
beam size
MBR (Mean)
18.26
47.47
19.70
24.97
53.29
26.67
54.79
400
18.63
47.89
19.70
49.02
25.58
53.76
26.67
54.80
1600
Sequence-level (Eq.
)
Unimodal
18.58
47.84
19.46
48.88
25.80
53.86
26.38
54.65
1600
Deep Ensemble
19.71
48.77
21.28
50.35
26.52
54.56
28.19
56.02
1600
Sequence-level (Eq.
(10)
Unimodal
18.43
47.75
19.62
48.95
25.34
53.66
26.58
54.77
1600
80
Deep Ensemble
19.48
48.49
20.69
49.88
25.86
54.22
27.40
55.42
1600
80
Token-level
Unimodal
17.90
47.29
19.60
48.94
24.86
53.29
26.57
54.79
400
80
Deep Ensemble
19.32
48.49
21.51
50.54
25.46
53.71
28.44
56.28
400
80
Table Results on afroMT with Transformer trained from scratch.
A.4 HYPOTHESIS SET SIZES
For the finetuning experiments, we use 40 candidate hypotheses for the single model baseline and token-level combination, and 20 per model for Eq. (TOP and per model for Eq. (9) except for XSUM, where we use 20, 10, and candidate hypotheses, respectively.
A.5 SELECTIVE PREDICTION
For selective prediction we reuse the models and set-up from App. A.1 which were used for Tab. In particular,
the sequence-level model combination of Eq. (10) and token-level combination with both ancestral sampling and beam search. The beam size is always 40 for MBR @ mean, 20 for each model used in sequence-level combination and 10 for each model used in token-level combination. All training details are the same in App. A.1
A.6 SCALING EXPERIMENT
Again, we use the set-up from App. A. with Transformenhase trained from scratch on IWSLT14. scale all methods according to the same training recipe as described there but with different random seeds to train the different models.
B ADDITIONAL RESULTS
B. RESULTS ON AFROMT
Tab. 6 shows results the En-Run and En-Bem partitions of afroMT. We find similar patterns results presented in Tab. 2 Seap-EEsembleBee based weight uncertainty always improves performance, even with matched compute budgets, while unimodal posteriors perform similarly single model baseline.
B.2 RESULTS WITH LABSE FOR TROM-SCRATCH-TRATTI MODELS
Tab. and Tab. show LaBSE scores for hallucination evaluation for the same evaluation setting as in Tab. Again, we find hallucinations to be reduced when weight uncertainty is accounted for.
B.3 INFERENCE- MEASUREMENTS
Tab. 9 shows the time needed for decoding in seconds as well as the obtained results for the E2ENLG experiment from Tab. [ All results were obtained on NVIDIA GeForce RTX 3090 GPUs with 24GB memory.
22

---

Published as a conference paper at ICLR 2025
Sampling COMET
Beam Search
Method
BLEU
LaBSE
BLEU
COMET
LaBSE
MBR Mean
33.69
74.71
85.33
33.90
76.63
86.44
Sequence-level
Eq. (9)
Unimodal
34.53
75.15
85.65
35.78
76.55
86.42
Deep Ensemble
36.03
75.79
85.98
38.30
78.01
87.16
Sequencc-level
Eq. (10)
Unimodal
34.65
75,20
85.68
35.99
76.67
86.45
Mixture
35.42
75.84
86.07
37.42
77.69
86.97
Token-level
Unimodal
33.62
74.68
85.39
35,94
76.66
86.45
Mixture
34.61
75.06
85.88
38.56
78.31
87.34
Table Measuring hallucinations with LaBSE (higher is better) on IWSLT14 with Transformerbue shows similar trends as quality estimation metrics: incorporating weight-uncertainty can reduce hallucinations, especially when a complex posterior is used. Here, we use a hypothesis set size of 20 for all methods but Eq. (9) which uses a size of 10.
Sampling COMET
Beam Search
Method
BLEU
LaBSE 86.97
BLEU 27.56
COMET
LaBSE
MBR @ Mcan
23.37
71.04
75.23
88.46
Sequence-level
Eq. (9)
Unimodal
24.3T
72.09
87.36
27.52
75.16
88.42
Deep Ensemble
24.70
72.39
87.61
28.99
76.02
88.68
Sequence-level
Eq. (10
Unimodal
24.2T
72.15
87.32
27.56
75.21
88.44
Deep Fnsemble
24.67
72.58
87.56
28.29
75.70
88.75
Token-level
Unimodal
23.44
71.36
86.84
27.75
75.19
88.35
Deep Ensemble
23.95
71.58
87.16
28.98
76.08
88.75
Table Measuring hallucinations with LaBSE (higher is better) WMT14 with TransformeThage shows similar trends as quality estimation metrics: incorporating weight-uncertainty can reduce hallucinations, especially when complex posterior is used. Here, we use hypothesis size of 20 for all methods but Eq. (9) which uses a size of 10.
Method
Creation of H (s)
Utility Calculation (s)
Total (s)
R-1
R-L
MBR @ meann Sequence-lovel Eq. Sequence-level Eq.
5824
402
6226
68.74
45.16
5472
408
5880
69.36
45.57
5881
418
6299
69.13
45.38
Table 9: Time (in seconds) taken for decoding for the results on E2ENLG from Tab.
23


Reviews:
[{"review_id": "rrVIY3WLqd", "reviewer": "Reviewer_Xat4", "summary": "This paper examines Minimum Bayes Risk decoding for generated text while also ensembling together multiple models (or parameter settings).", "questions": "None", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper has a fairly extensive number of experiments.", "weaknesses": "- I'm quite familiar with MBR, ensembling, and other related methods, and to be honest the papers seemed like it was made overly complex/mathy while still omitting details about important parts. For instance, explaining the fact that you're sampling from multiple models and doing MBR over the resulting samples (equation 10) is not a difficult concept to explain, but it was couched in lots and lots of verbose equations. On the other hand, important parts that are non-trivial, such as the IVON method introduced in section 4.1, were not explained sufficiently. I feel like the paper could benefit significantly by trying to explain things in simpler/more straightforward terms.\n- There is no discussion of time or memory complexity of the methods. I expect that these methods require significant computation, so it would have been better if that could be discussed.\n- It's not very clear whether the methods proposed here significantly moved forward the state-of-the-art. As far as I know, the BLEU scores on WMT14 are quite low compared to the state-of-the-art for single models, which is 35 or so.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jTmfa8Z1lj", "reviewer": "Reviewer_TRc3", "summary": "This work proposes minimum Bayes' risk (MBR) decoding considering the uncertainty of model parameters. The MBR is a decision rule for decoding which maximizes the expected utility, not a maximum a posteriori estimate, to avoid the model errors. This work takes into account the uncertainty in model parameters so that the decision rule can reflect the uncertainty in the model parameter estimation. This work also proposes a token-level sampling but approximated by Monte Carlo in order to avoid complex computation by taking average in each step. Experiments are carried out on several natural language tasks, e.g., machine translation and summarization, and show that the proposed method achieves gains when compared with a conventional MBR method.", "questions": "Please check my comments regarding weaknesses.\n\n- Thank you for the clarification, and I feel this work has an interesting contribution in the field.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work demonstrate the use of MBR considering the uncertainty in model parameters. The idea is straightforward and the proposed approach of two decision rules is a sound method in performing either summation across samples drawn from different parameters (Eq. 9) and summing within samples from the same parameters (Eq. 10).\n\n- The proposed approach and the experimental design is coupled with IVON optimization which take into account the variance of parameters in training. Since the variance is also estimated during training, it seems to be easy to draw samples from sampled parameters.", "weaknesses": "The detail of the experimental design is unclear.\n\n- The number of samples is unclear. The line 352 stats that is is using ensemble and samples for 4, but it probably implies the number of samples from a single model parameter, and this work has no report on how many samples were drawn for model parameters, that is the size of $\\mathcal{M}$. Does it mean that 1 sample was drawn for the proposed method for each parameter sample set drawn from IVON so that the proposed approach is comparable to the baseline MBR with 4 samples?\n\n- The relation of Deep Ensemble and the proposed method is unclear. The lines 311-319 state that Deep Ensemble is performed for each training run, so that multiple IVON parameters are learned. In this case, only a single parameter set is drawn for each learned IVON model, and performed MBR decoding using the proposed method, but is it correct? Note that Deep Ensemble is performing ensemble of decoding from multiple model parameters, and thus, it is basically the same as the token-level inference in Eq. 11. When mentioning Deep Ensemble, is it performing the token-level average for each step? Or, does this work leverage Deep Ensemble to learn several parameters to compare with IVON, which employs only a single training run?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xnEDUhMGBw", "reviewer": "Reviewer_xUiD", "summary": "Minimum Bayes Risk Decoding (MBR) has a long history in machine translation and speech recognition research. As a variation of the Maximum A Posteriori decoding, it introduces a risk at the sentence level. The goal is to improve the output sentence globally. In this paper, The risk becomes utility and the goal is to find the hypothesis that maximizes the expected utility.  This paper proposes to extend this framework to include uncertainty on the parameters of the model. Therefore it requires a \"double\" expectation (or an expectation wrt the joint distribution of the sequence and the parameters).  Experiments are carried out on different tasks of conditional text generation: machine translation, summarization and data-to-text.", "questions": "- It could help to better describe how the q distribution on large pre-trained model is estimated.\n- It looks like uncertainty improves over MBR, but can you clearly state in the table what are  the baseline systems (w/o MBR) and what are your improved ones ? \n- The empirical results and the associated discussion could be improved. When you say \"We find that weight uncertainty improves performance across all benchmarks, even with matched\ncompute budgets.\", it could help to ground this claim in your tables.  \n- The paragraph on the complexity is a bit short. MBR decoding is costly. Maybe you solution does not increase the complexity but could you provide inference time ? \n- Do you think the improvement in performance is worth the  overall computational cost ? This question stands for MBR and you method. \n- When you say \"Connected to this, since predictive uncertainty has been shown to correlate with hallucinations (Xiao & Wang, 2021), ...\", this not the same uncertainty ! In this work they use the entropy of the word distributions not on parameters. Could you comment on that ?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "The motivation of Bayes inference is to marginalize what is uncertain. This paper proposes to take into account the uncertainty on the parameters on the top of MBR. This idea is very interesting and can be applied to both large pre-trained models, and smaller models trained from scratch only for one task.", "weaknesses": "The clarity is clearly the main issue with this paper; It is true that Bayesian methods are always difficult to describe, but in this paper it is sometime difficult to follow. One reason maybe that some important information are scattered in the paper. \n\nFor instance, the capital theta is not clearly defined. We can deduce it from the context but it does not help to understand. More importantly, there are many equations in the paper (this is not an issue of course) and we can get lost among their variants: the theoretical description vs  the approximated version, ... \nMaybe it could help to first describe the theoretical part of the method, with the probability distributions, and then in a following part how it is proposed to approximate the intractable summations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HB5NxGNDHI", "reviewer": "Reviewer_nT47", "summary": "This paper extends Minimum Bayes Risk (MBR) decoding to incorporate model uncertainty. During the computation of expected risk, standard MBR decoding assumes a model with fixed parameters that approximates the target distribution. This paper argues that the uncertainty in model parameters should also be considered, and proposes to compute the expected risk based on predictive posterior distribution. Specifically, the paper introduces two types of posterior distributions for uncertainty-aware MBR decoding: sequence-level and token-level posteriors. Empirical evaluation across various text generation tasks, including machine translation and text summarization, shows that the proposed method achieves consistent and strong improvements over unimodal MBR decoding.", "questions": "Could you elaborate the difference between the two estimators in Eq.(9) and Eq.(10), and why Eq. (10) has lower complexity?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written, clearly motivated, and theoretically grounded.\n2. The proposed method shows consistent and strong improvements to unimodal MBR decoding.", "weaknesses": "1. MBR is not a very efficient decoding algorithm. I am afraid that efforts on MBR decoding may have little chance to have practical applications.\n2. The concept of token-level posterior is not well-established from a theoretical perspective. As these models represent sequence-level probability distributions, their ensemble, the predictive posterior should only be an expectation over sequence probabilities.\n3. The baseline of standard MBR decoding is missing. Although there is a baseline of MBR@mean, it is not equivalent to the standard MBR decoding, where the model should be trained without weight uncertainty.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)