Venue: NAACL 2025

# A Bayesian Optimization Approach to Machine Translation Reranking

Julius Cheng\({}^{1}\) Maike Zufle\({}^{2}\) Vilem Zouhar\({}^{3}\) Andreas Vlachos\({}^{1}\)

\({}^{1}\)University of Cambridge \(~{}~{}^{2}\)Karlsruhe Institute of Technology \(~{}~{}^{3}\)ETH Zurich

{jncc3,av308}@cam.ac.uk maike.zuefle@kit.edu vzouhar@ethz.ch

###### Abstract

_Reranking_, or scoring a list of prediction candidates from a machine translation system with an external scoring model and returning the highest-scoring candidate, remains a simple and effective method for improving prediction quality. However, reranking with high quality scoring models can add substantial computational cost to the translation pipeline, which we address in this work by framing list reranking as a Bayesian optimization (BayesOpt) problem over the candidate list, where unknown scores are modeled with a Gaussian process. This algorithm scores candidates iteratively, choosing next candidates by balancing between exploration, choosing to score those that differ from candidates already scored, and exploitation, choosing to score those that resemble high-scoring candidates. This procedure finds high-scoring candidates while scoring only a fraction of the candidates list; given candidate lists of 200 random samples (before dedupletion), our method achieves the same CometKiwi score using only 70 scoring evaluations on average compared to scoring a random subset of 180 candidates. We also propose multi-fidelity BayesOpt for list reranking, where scores obtained from a noisier but cheaper _proxy_ scoring model are incorporated into the search process. We show that well-trained distilled proxy scorers can further improve the performance of BayesOpt.

+
Footnote †: Paper code: github.com/juliusc/bayesopt_reranking

## 1 Introduction

_Reranking_ is a framework for prediction where probabilistic _generator_ model produces a list of candidates, and a separate _evaluator_ or _scoring model_ produces scores for each of the candidates which are use to determine the final prediction. Reranking has a long history in natural language processing for sequential prediction problems such as dependency parsing Collins and Koo (2005); Charniak and Johnson (2005) and language modeling problems such as summarization Ravaut et al. (2022) and machine translation MT; Fernandes et al. (2022).

The quality of models for automatic MT evaluation has surged in recent years due to innovations in neural network architecture Rei et al. (2020); Juraska et al. (2023); Sellam et al. (2020) as well as the abundance of training data Freitag et al. (2023b); Kocmi et al. (2024a). These evaluation models are often repurposed for reranking to further improve the performance of an MT system. For instance, in the WMT 2024 shared task Kocmi et al. (2024a), 5 out of 19 systems, including the overall best submission Rei et al. (2024) use reranking with Comet models Rei et al. (2020) and/or minimum Bayes risk decoding MBR; Eikema and Aziz (2020), which can be interpreted as a form of reranking. Prior to the application of automatic evaluation metrics to reranking, other scoring methods have been proposed, including discriminatively

Figure 1: A machine translation system generates candidates Aa, Bb, Cc, Dd, and Ee. The goal of BayesOpt is to find the highest scoring candidate with fewer scoring calls. An acquisition function selects the next candidate to score repeatedly until budget is reached, and the candidate with the highest score so far is returned.



trained classifiers (Lee et al., 2021; Bhattacharyya et al., 2021) and noisy channel decoding (Yee et al., 2019).

So, while LMs for MT generation for greatly improved in recent years, scoring models have seen a commensurate increase in quality (Zerva et al., 2022), and thus reranking remains relevant method for improving translation quality. However, the scoring models have also grown dramatically in size, increasing the computational requirements for reranking.

In this work, we address the computational cost of reranking by framing it as a search problem over the list of candidates. The goal of search in this setting is to find high-scoring candidates in a small number of steps, thereby avoiding the cost of scoring the full list. Our proposed algorithm uses Gaussian processes to model uncertainty about unseen scores and Bayesian optimization (BayesOpt; Shahriari et al., 2016) to choose which candidates to score next.

GPs are flexible priors over functions which are able to model the complex and nonlinear relationship between each candidate and its score. GPs make very few assumptions about the distribution and base their predictions are mostly on observed points, which enables them to easily adapt to different candidate lists across translation instances. BayesOpt is a sequential black-box optimization method that uses the posterior mean and variance of unobserved data points to decide which points to evaluate next.

We apply BayesOpt and GPs (BayesOpt+GP) to MT list reranking in a straightforward manner and show that it obtains close to the maximum achievable score with only a fraction of score evaluations. For example, the maximal obtainable score across 200 randomly sampled candidates on our test set is 0.8216 CometKiwi; our method achieves 0.8210 with 70 score evaluations on average, while scoring 70 random candidates attains 0.8149, a difference of 0.0061 which is likely to be human-detectable according to (Kocmi et al., 2024b). We also propose a number of search-based baselines which outperform random selection, all of which are outperformed by BayesOpt+GP.

Then, building upon previous works that use a faster but noisier proxy scoring function to prune the candidate list Fernandes et al. (2022); Eikema and Aziz (2022), we propose a _multi-fidelity_ extension to BayesOpt which incorporates proxy scores to improve estimation. This is related in motivation to coarse-to-fine methods (Petrov, 2011) and model cascading (Chen et al., 2023), where the use of a faster proxy model reduces the use of the main model. In our multi-fidelity experiments, we find that smaller proxy scoring models distilled from the main model can assist BayesOpt+GP in finding high-scoring candidates earlier.

## 2 Background

### Translation generation and reranking

In a typical machine translation setting, a conditional language model (LM) is trained to model the probability of the next token \(y_{t}\) given a source sentence \(x\) and previous tokens: \(p(y_{t}|x,y_{1},...,y_{t-1})\). These probabilities can be autoregressively combined to model a sequence probability \(p(y|x)\). Usually, beam search is used to search for a \(y\) which maximizes log probability combined with a length normalization objective Wu et al. (2016).

In a basic list reranking setting, given \(x\), the LM is used to generate a candidate list \(\mathcal{C}_{x}=[y_{1},...,y_{n}]\) with a _decoding algorithm_ such as beam search or ancestral sampling. A scoring function \(s(x,y_{i})\) is then applied to each \(y_{i}\in\mathcal{C}_{x}\), and the best scoring sequence \(\arg\max_{y_{i}\in\mathcal{C}_{x}}s(x,y_{i})\) is returned. A common choice of scoring function is a quality estimation (QE) model which directly predicts a scalar value representing the quality.

Reranking with high-quality evaluation metrics has been shown to be highly effective at improving translation output (Freitag et al., 2022), though it can skew results when the same metric is also used for evaluation (Kocmi et al., 2024a). Reranking performance improves as the number of candidates increases (Vernikos and Popescu-Belis, 2024) and when multiple scoring metrics are combined to form a stronger prediction (Fernandes et al., 2022).

Reranking adds significant computational costs to prediction and may be prohibitive to use at test time, but it can be used to benefit LM training instead of test time prediction; high-quality predictions obtained from reranking can be used for knowledge distillation (Wang et al., 2024) and self-training (Finkelstein et al., 2024). Such methods can improve the performance of an MT system without additional costs during test time.

Previous work on efficient reranking for MT is relatively limited. Fernandes et al. (2022) and eikema and aziz (2022) perform a two-stage reranking by first pruning with a faster and noisier 

scoring function to a fixed size before evaluating the target score. There has been recent interest in efficient approximations for MBR (Cheng and Vlachos, 2023; Deguchi et al., 2024; Trabelsi et al., 2024; Vamvas and Sennrich, 2024), but these methods are not applicable to general scoring functions. (Singhal et al., 2023) propose to represent the candidate space compactly in a lattice over which a token-level reranker can efficiently score many candidates. In this work, we attempt to address a more general setting: the reranking of candidate lists with arbitrary black-box scoring functions.

### Bayesian optimization with Gaussian process prior

Bayesian optimization is a sequential algorithm for optimizing a black-box function \(f\). \(f\) is assumed to be drawn from a prior distribution over functions. The main loop of BayesOpt is as follows: given a set of (possibly noisy) observations of \(f(a_{1}),...,f(a_{i})\), the prior distribution over \(f\) is updated to a posterior distribution with Bayes theorem. An _acquisition function_ determines a _query point_\(a_{i+1}\) at which to evaluate \(f\) next. \(f(a_{i+1})\) is evaluated and added to the set of observations. This repeats until a stopping criteria is reached. The principal design choices in BayesOpt are the prior distribution of \(f\) and the acquisition function.

A common choice of prior is the Gaussian process, which assumes that any subset of points \(f(a_{1}),...,f(a_{i})\) are drawn jointly from a multivariate Gaussian distribution \(\mathcal{N}(\mu,\mathcal{K})\), where \(\mathcal{K}\) is the covariance matrix defined by a kernel function such as the radial basis function kernel (RBF). RBFs define the covariance of two points \(a\) and \(a^{\prime}\) as:

\[\mathcal{K}_{\mathrm{RBF}}(a,a^{\prime})=\exp\left(-\frac{||a-a^{\prime}||^{2 }}{2w^{2}}\right), \tag{1}\]

where \(w\) is the _bandwidth_ hyperparameter which determines scaling. The choice of kernel dictates prior assumptions about the shape of \(f\); with RBF, points that are closer in Euclidean space have larger covariance. RBFs are a popular choice of kernel due their ability to adapt to complex nonlinear functions.

The assumption that \(f(a_{1}),...,f(a_{i})\) are jointly Gaussian gives rise to a convenient posterior distribution. Given a vector of observed data points **a** and their observed values \(f(\textbf{a})\), the posterior mean \(\mu_{a}\) and variance \(\sigma_{a}\) of a point \(a\) are given by the conditional multivariate Gaussian distribution:

\[\begin{split}\mu_{a}&=\mu+\mathcal{K}(a,\mathbf{a })(\mathcal{K}(\mathbf{a},\mathbf{a})+\sigma^{2}I)^{-1}f(\mathbf{a})\\ \sigma_{a}&=\mathcal{K}(a,a)+\sigma^{2}-\\ &\qquad\quad\mathcal{K}(a,\mathbf{a})(\mathcal{K}(\mathbf{a}, \mathbf{a})+\sigma^{2}I)^{-1}\mathcal{K}(\mathbf{a},a)\end{split} \tag{2}\]

where \(\mu\) is the unconditional mean of the distribution, \(\sigma^{2}\) is a constant Gaussian noise on observations, \(I\) is the identity matrix, and \(\mathcal{K}\) here returns elementwise kernel values when given vector arguments.

The acquisition function in BayesOpt is the strategy for selecting the next point to evaluate in the optimization process. Acquisition functions can seek the highest expected improvement (EI; Mockus, 1974), an upper confidence bound if the scores are noisy (Srinivas et al., 2009), or information gain (Hennig and Schuler, 2011). We use EI, defined as:

\[\alpha(a)=\mathbb{E}[\max(f(a)-f(a^{+}),0)], \tag{4}\]

where \(a^{+}\) is the location of the current best observation. When \(f\) is Gaussian and there is no observation noise, this has the following closed-form solution (Jones, 2001):

\[\alpha(a)=\sigma_{a}(z\cdot\operatorname{cdf}(z)+\operatorname{pdf}(z)), \tag{5}\]

where \(z=\frac{f(a^{+})-\mu_{a}}{\sigma_{a}}\), and \(\operatorname{cdf},\operatorname{pdf}\) are the Gaussian cumulative distribution function and probability density function, respectively. EI encourages both exploration of uncertain points and exploitation of high-scoring points; the quantity in Equation 5 can be increased by increasing \(\mu_{a}\) or \(\sigma_{a}\).

The generality of BayesOpt and modeling freedom enjoyed by GPs make them suitable for a great variety of tasks, including spatial monitoring (Krause et al., 2008) and hyperparameter optimisation (Bergstra et al., 2011). GPs have been applied to text regression tasks (Beck et al., 2013, 2014; Beck and Cohn, 2017), but they are not as well-studied in NLP compared to many other domains.

## 3 Methods

### MT reranking with Bayesian optimization

Our main algorithm is an adaptation of BayesOpt with GPs as described in Section 2.2 to the reranking setting. Each source sentence \(x\) and its associated candidate list is treated as a standalone BayesOpt problem, meaning that no observations are shared across different \(x\). Thus for brevity, we 

omit \(x\) from notation when discussing BayesOpt for a particular instance.

Let \(s\) be the scoring function, an MT quality estimator. Let \({\cal C}\) be a set of candidates, \({\cal C}_{\mathrm{obs}}\subseteq{\cal C}\) the subset of candidates for which we have observed \(s(y)\), and \(\bar{\mathcal{C}}_{\mathrm{obs}}\) be all other \(y\) (\(\bar{\mathcal{C}}_{\mathrm{obs}}={\cal C}\setminus{\mathcal{C}}_{\mathrm{obs}}\)). To perform reranking for an instance, we first generate candidates \({\cal C}\) and initialize the algorithm by scoring a random \(\alpha\)-sized subset of the list with \(s\). In one iteration in the algorithm loop, we normalize the observed scores to mean 0 and 1 variance at every step and assume a 0 unconditional mean. Then we compute the GP posterior of all \(y\in\bar{\mathcal{C}}_{\mathrm{obs}}\) with Equation 2 and 3 given the scores of \({\cal C}_{\mathrm{obs}}\), which is then used to compute EI with Equation 5, assuming no observation noise. We score the \(k\) candidates in \(\bar{\mathcal{C}}_{\mathrm{obs}}\) with the highest EI, adding them to \({\cal C}_{\mathrm{obs}}\) (as well as removing them from \(\bar{\mathcal{C}}_{\mathrm{obs}}\)), and repeat the loop, terminating when a predefined budget of \(n\) calls to \(s\) is reached (or when all candidates have been evaluated, in the case that \(|{\cal C}|\leq n\).). Finally, we choose \(\arg\max_{y\in{\cal C}_{\mathrm{obs}}}s(y)\) as the prediction.

We now describe our choice of GP kernel. \(y\in{\cal C}\) are strings, and we seek a representation that is fast to compute and to compare, since \(|{\cal C}|\) representations are generated, and the computing the GP covariance matrix requires \(|{\cal C}|^{2}\) comparisons. Our kernel is \({\cal K}_{\mathrm{MT}}(y_{i},y_{j})={\cal K}_{\mathrm{RBF}}(\operatorname{emb }(y_{i}),\operatorname{emb}(y_{j}))\), where \(\operatorname{emb}\) returns the mean-pooled token-level outputs of the final decoder layer when generating \(y\), normalized to the unit norm after pooling. \(\operatorname{emb}\) uses meaning representations produced automatically during candidate list generation, so the additional cost to compute it is negligible. Also, the covariance matrix is fast to compute given the candidate list sizes and embedding dimensionality used in our experiments.

### Multi-fidelity BayesOpt

We also propose an extension to BayesOpt+GP for the setting where observations are available from a different but related _proxy score_ function \(s^{\prime}\). We refer to this as BayesOpt+GP+P. \(s^{\prime}\) is assumed to have non-trivial covariance with the scoring model \(s\) and to be cheaper to evaluate. This is known as _multi-fidelity_BayesOpt in the literature, but while the multi-fidelity settings of Kandasamy et al. (2016); Wu et al. (2020) use acquisition functions that may choose to evaluate lower-fidelity scores, we study a simpler setting: \(\beta\) observations of \(s^{\prime}\) are obtained at the start where \(\beta>\alpha\), and only \(s\) may be evaluated during the BayesOpt loop. In the multi-fidelity setting, observations are made on \(\langle y_{i},s_{i}\rangle\), a combination of a data point and scoring function, instead of the data point alone.

Our kernel for BayesOpt+GP+P is the product 

of the RBF kernel from Section 3.1 and a kernel over score functions \(f\):

\[{\cal K}_{\rm mult}\big{(}\langle y_{i},s_{k}\rangle,\langle y_{j}, s_{l}\rangle\big{)}=\\ {\cal K}_{\rm MT}(y_{i},y_{j}){\cal K}_{\rm score}(s_{k},s_{l}). \tag{6}\]

\({\cal K}_{\rm mult}\) is a valid kernel because a product of two kernels defined on different spaces is also a kernel Rasmussen and Williams (2005). With \({\cal K}_{\rm mult}\), the covariance between two observations depends on both the difference between scoring functions and the distance between data points. This way, an observation influences the posterior for all other data points at all choices of scoring function, as long as the scoring functions are correlated. This formulation enables the use of any number of scoring functions, but in this work, we consider at most two: the main scorer \(s\) and a proxy scorer \(s^{\prime}\).

We set \({\cal K}_{\rm score}(s_{k},s_{l})\) to be the empirical covariance between \(s_{k}\) and \(s_{l}\) measured over a validation set, where all scores are normalized per-instance so that in each instance, the scores of all candidates for a particular scorer have 0 mean and 1 variance. Then for each scoring function, concatenate all candidate scores across instances, and compare the resulting lists to obtain the covariance. Covariance is a valid kernel because the covariance calculation can be expressed as a dot product, and dot products are valid kernels.

Proxy scores are incorporated into posterior estimation given by Equations 2 and 3 by redefining \(a\) to be a tuple of \(\langle\text{data point},\text{scoring function}\rangle\) and **a** to be a vector of such tuples. The kernel \({\cal K}\) is set to \({\cal K}_{\rm mult}\) which takes as input two tuples of data point and scoring function. The full BayesOpt+GP+P algorithm is in Algorithm 1.

### Proxy scores

We train smaller scoring models to have high covariance with \(s\) for use in BayesOpt+GP+P. In this work, our scoring functions are based on the Comet referenceless quality estimation architecture Rei et al. (2020), also known as CometKiwi. These models encode the source and hypothesis jointly with a bidirectional transformer. Activations from all transformer layers are pooled to form a fixed-size representation, which is passed to a feed-forward regression head. The vast majority of computation in this models is spent in the encoder. Thus, faster Comet models can be obtained by reducing the size of the encoder.

We train Comet models using two differently sized pretrained multilingual encoder models in two ways: (1) training on the same training set as CometKii and (2) distillation. Among distillation methods, we attempt in preliminary experiments (1) training on the same training set as CometKiwi with ground truth scores replaced with CometKiw scores and (2) training on a synthetic dataset comprising of LM samples along with their associated CometKiw scores. The latter achieves higher correlation with CometKiwi on sampled candidates, which is to be expected since the training distribution is more suitable for the reranking use case. We therefore use this latter distillation method for all subsequent experiments. A similar procedure has been described in Rei et al. (2022).

### Candidate list generation

In preliminary experiments, we consider generating the candidate list using beam search with 128 outputs versus sampling 200 candidates using \(\epsilon\)-sampling Hewitt et al. (2022) with \(\epsilon=0.02\), a setting which effectively balances quality and diversity for MBR Freitag et al. (2023). Under beam search, the candidates exhibit high lexical overlap, and while the mean score of candidates is higher, the average maximum score is lower. The effectiveness of truncated sampling over beam search in larger conditional language model has also been observed by Fernandes et al. (2022).

Furthermore, beam search suffers from out-of-memory errors on long translations, whereas with sampling, we simply reduce the batch size when out of memory. While it is possible to implement beam search in a batched manner, this does not exist in any popular conditional language model libraries, to the best of our knowledge.

For these reasons, we generate 200 candidates per instance with \(\epsilon\)-sampling, \(\epsilon=0.02\) in all experiments. The sampled candidate list is then deduplicated, resulting in \(\sim\)178 candidates on average per instance.

## 4 Experiments

We now discuss the details and findings of our Bayesian optimization experiments, followed by analysis of our trained proxy scoring models, concluding with runtime measurements. All run time values are measured on a A100-SXM4-40GB GPU. For exact values for figures in this section, see Table 3 in the Appendix. Appendix A contains 

extensive statistical significance tests.

For BayesOpt experiments, we grid search for the optimal value of RBF bandwidth parameter \(w\) on the entire validation set, setting scoring budget \(n=100\) and batch size \(k=1\). While it is possible to optimize it for every unique combination of language pair, \(n\), \(k\), proxy scoring function, and \(\beta\), we find that the results are not statistically significantly different within a range of settings. For simplicity, and to demonstrate the robustness of our methods, we use the same \(w\) for all experiments.

In all experiments, we use \(\alpha=10\) initial randomly scored candidates. We set \(k=1\) in Sections 4.2 and 4.3 to demonstrate the effectiveness of BayesOpt+GP under ideal conditions, but since \(k\) can have a large impact on speed, we experiment with varying it in Section 4.5.

### Models and datasets

For candidate generation, we use the 600M-parameter distilled NLLB model Team et al. (2022) in all experiments. For the main scoring model, we use CometKiwi-22 Rei et al. (2022b).

As a dataset used for proxy model training, we use data from the WMT Metrics Shared Task up to 2022 Freitag et al. (2023b), which contains tuples of \(\langle\text{source}\), hypothesis, human score\(\rangle\). The human scores were largely collected with the DA+SQM annotation protocol Kocmi et al. (2022).

For BayesOpt experiments, we select the first 1000 and 500 source sentences per language pair from the WMT23 Metrics Shared Task dataset as the validation and test set, respectively, for 7 language pairs: English-Czech, English-German, English-Japanese, English-Chinese, and the reverse directions of the latter 3 pairs.

CometKwi is based on the encoder of XLM-Robertalarge Conneau et al. (2019) (2.2GB memory). For proxy scorers we train smaller models based on XLM-Robertabase (1.1GB), and Multilingual-MinLM-L12-H384 Wang et al. (2020) (469MB).

### BayesOpt+GP

The goal of reranking BayesOpt+GP is to improve the speed by only evaluating a subset of available candidates. We evaluate this through quality-cost tradeoff curves, where quality is determined by final selected candidate's CometKwi score, and cost is determined by the number of calls to the scoring function. As another measure of approximation quality, we also show the percentage of instances in which the actual best scoring candidate is returned. We devise several baselines with which to compare BayesOpt+GP. Each is a strategy for selecting a subset of candidates to score from which the best scoring candidate is returned. The baselines are:

* **UniqRandom**: Shuffle the candidate list _before_ de-duplication, then de-duplicate while preserving the order of the first appearance of each candidate. Select the first \(\min(n,|\mathcal{C}|)\) candidates in the resulting list.
* **LogprobdAvgSum**: Sort \(\mathcal{C}\) in order of negative sequence log probability (either average or sum), and then select the first \(\min(n,|\mathcal{C}|)\).
* **HillClimbing**: Let \(y^{+}\) be the highest scoring observation point at any time step. Iteratively select \(\arg\min_{y\in\tilde{\mathcal{C}}_{\mathrm{obs}}}||\mathrm{emb}(y)-\mathrm{ emb}(y^{+})||\) as the next

Figure 2: Left: Performance of reranking methods measured as the average CometKiwi score of the selected candidate. Beam search with beam size 5 achieves a score of 0.754 and is too low to be pictured here. Right: percentage of instances where the selected candidate had the highest score (right). The x-axis is the scoring budget. Legends show the normalized area under the curve of CometKwi score of each method in brackets.

 

observation point until \(\min(n,|{\cal C}|)\) candidates are scored.

UniqRandom simulates the effect of iteratively sampling candidates until \(n\) unique candidates are obtained. LogprobFirst{Avg,Sum} are included to verify whether more advanced methods indeed outperform simple subset selection using statistics obtained for free. HillClimbing is a heuristic iterative selection strategy which, like BayesOpt, is black-box and derivative-free Conn et al. (2009).

In Figure 2, BayesOpt+GP outperforms all baselines, and HillClimbing is the best among the baselines, with LogprobAvg following behind. LogprobSum severely underperforms UniqRandom in score, confirming findings on the inadequacy of very high probability translations Eikema and Aziz (2020). Informally speaking, UniqRandom is a simple "exploration" strategy that ignores existing observations, while HillClimbing is a simple "exploitation" strategy, only searching over neighbors nearest the best observation while ignoring the full search space. These results confirm that balancing these respective deciderata helps to find the optimal candidate more efficiently.

### BayesOpt+GP+P

#### 4.3.1 Proxy score evaluation

We first evaluate trained proxy scorers independently of their use in BayesOpt according to (1) actual runtime, (2) correlation with human ratings in the WMT23 dataset, (3) correlation with CometKiwi on source-hypothesis pairs in WMT23, and (4) correlation with CometKwi on a synthetic candidates for an instance, averaged over instances. For correlations we use Kendall's \(\tau_{c}\), which is commonly used in MT metric evaluation Freitag et al. (2023b).

Table 1 shows the results for the proxy models. The model size corresponds closely to inference time. As desired, training proxies using distillation results in much higher correlation with CometKiwi, although it loses some correlation with human judgments. In subsequent experiments, we consider Distilled-{S,M} only. While LogprobAvg has comparatively much lower correlation, we nevertheless consider it as a proxy score since it is obtained for free during candidate generation.

Figure 4: Average CometKiwi score of the selected candidate (y-axis) for BayesOpt+GP+P with different choices of proxy score.

Figure 3: Average CometKiwi score of the selected top candidate (y-axis) for BayesOpt+GP+P with Distilled-S (left) and Distilled-M (right) compared to the ProxyFirst baseline. This figure disregards the additional compute costs for these proxy metrics in order to show the marginal score increase from proxy observations.

 



#### 4.3.2 Reranking results

When \(s^{\prime}\) is sufficiently fast and correlated with \(s\), it can further improve the quality-cost tradeoff in BayesOpt+GP. Recall that BayesOpt+GP+P initializes with \(\beta\) evaluations of \(s^{\prime}\). Figure 4 shows the quality-cost curve when all proxy scores are known, or \(\beta=200\). The relative performance when including proxy scores correspond to their correlation with CometKiwi as shown in Table 1; DistilledM outperforms DistilledS, and both outperform LogprobAvg. This demonstrates the importance of ensuring high correlation in the proxy score. The addition of LogprobAvg to BayesOpt+GP has little effect, showing that poorly correlated proxies are too noisy to help and may even hinder performance. Beyond \(n=70\), all methods achieve close to the maximum attainable score.

We also examine the effect of initializing with a fraction of proxy observations rather than all of them. For some choice of \(\beta\), an appropriate baseline is to rank the top-\(n\) candidates among the \(\beta\) observed proxy scores. We call this ProxyFirst. The results when using DistilledM and DistilledS as proxies are shown in Figure 3. In both cases, the difference between BayesOpt+GP+P and ProxyFirst is smaller when \(\beta=200\) than when \(\beta=50\), and this gap is smaller for DistilledM. This is to be expected because as the covariance of \(s\) and \(s^{\prime}\) increases, using ProxyFirst with \(\beta=200\) approaches standard full-list reranking. The marginal benefit of BayesOpt+GP+P is more clear when \(\beta=50\), where proxy scores help to find promising candidates earlier.

Overall, proxy observations can indeed improve quality for a particular \(n\). However, for sufficiently large \(n\), BayesOpt+GP converges, so proxy observations are unnecessary. Proxy evaluations add to the runtime cost which we discuss in Section 4.4. Therefore, while we show that the multi-fidelity kernel is capable of leveraging proxy scores to improve search, in practice, the overall computational budget should be considered along with the quality and cost of the proxy scoring function to ensure that using the method is worthwhile.

### Runtime

Our reranking algorithm significantly reduces actual runtime compared to scoring all candidates for a source sentence. We profile the full pipeline, from generating candidates to making a final selection, on three settings: (1) BayesOpt+GP with \(n=90\), and (2) multi-fidelity BayesOpt+GP with 50 Distilled-S scores and \(n=70\), and 3) the baseline of evaluating CometKiwi on all candidates. \(n,\beta\) are selected to balance the final scores of the two algorithms (\(0.8213\) and \(0.8211\) respectively, as shown in Table 3).

For the runtime calculations, we select 50 source sentences from each language pair and generate 200 candidates for each. For the baseline, we compute scores for all candidates with a batch size of 200. For BayesOpt+GP methods, we profile the additional steps required: computing the kernel, computing the posteriors at each step, and evaluating proxy scores. BayesOpt+GP(+S) uses batch size \(k=10\), which does not affect scores compared to using \(k=1\) (see Section 4.5). Memory bandwidth can be a major overhead in large neural networks, making it inefficient to run small batches. Since BayesOpt+GP obtains \(k\) candidates per step, in order to use large batches, we process candidates for multiple instances in parallel.

Results are shown in Table 2. In all cases, candidate generation and CometKiwi calculations dominate the overall runtime. The extra cost from BayesOpt-related computations is compensated by the savings from reducing CometKiwi evaluations, despite similarity matrix computation being \(\mathcal{O}(|\mathcal{C}|^{2})\), and matrix inversion for posterior calculation at each iteration being \(\mathcal{O}(|\mathcal{C}|^{3})\)

BayesOpt+GP+P with Distilled-S reduces the runtime by further reducing the number of CometKiwi calculations to 70, with the cost of loading and running the Distilled-S proxy metric introducing minimal overhead.

### Batch size \(k\) in BayesOpt+GP

We examine the effect of batch size \(k\) in BayesOpt+GP for \(k=1,2,5,10\). Figure 5 shows that as expected, larger \(k\) diminishes performance, although the differences nearly vanish at \(n{>}70\).

\(k\) impacts how often the BayesOpt loop is run and thus has a large effect on speed. Fortunately, we observe for sufficiently large \(n\), \(k\) can be increased without sacrificing quality.

## 5 Conclusion

In this work, we formalize MT reranking as a Bayesian optimization problem, leveraging the basic observation that similar translations are more likely to have similar quality scores. We also extend the framework to accept observations from proxy scoring functions, which is applicable when the target score is very costly: large QE models, MBR, or human evaluation. In realistic experiments, we show that our methods improve reranking efficiency over strong baselines. We also propose several design choices that make the methods useful in practice; a GP kernel that requires minimal overhead, and effective proxy model training via distillation.

We consider our work a first step in applying BayesOpt to MT reranking. Future directions include integrating BayesOpt with candidate generation, alternative acquisition functions, and further exploration of GP kernels for MT.

## 6 Limitations

The optimization problem considered in this work is to maximize score from a scoring model. We show that BayesOpt is an effective optimizer, but we do not explore to what extent the optimization problem is flawed due to flaws in the scoring model. We refer to Kocmi et al. (2024b) to understand what magnitude of score difference between systems is significant. However, the existence of "metric overfitting" when directly optimizing an evaluation metric is debated and may affect the interpretation of score differences Fernandes et al. (2022); Wang et al. (2024).

BayesOpt+GP requires matrix inversion, a \(\mathcal{O}(|\mathcal{C}|^{3})\) operation that is performed once per iteration. While it is inexpensive for the \(|\mathcal{C}|\) we consider, this limits the number of observations that can be used for posterior computation without resorting to approximations Noack et al. (2023).

As an iterative algorithm, BayesOpt can score no more than \(k\) candidates in a batch for a single instance. Small batch sizes introduce a significant bottleneck for large neural networks, so in order to maintain large batch sizes, we propose processing multiple instances in parallel. However, this requires additional engineering.

#### Acknowledgements

Julius Cheng is supported by a scholarship from Huawei. Part of this work received support from the European Union's Horizon research and innovation programme under grant agreement No 101135798, project Meetween (My Personal AI

Figure 5: Difference between BayesOpt+GP with batch size of 1 (top line in red in Figure 2) and BayesOpt+GP with higher batch sizes. Negative values mean that higher batch size performed worse than BayesOpt+GP with batch size of 1.

 

Mediator for Virtual MEETtings BetWEEN People). We thank the organizers of MT Marathon 2024, where the authors met and this work was conceived. We also thank Beni Egressy for useful discussions and Will Tebbutt for lending expertise on GPs.

## References

* D. Beck and T. Cohn (2017-05)Learning kernels over strings using Gaussian processes. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pp. 67-73. Asian Federation of Natural Language Processing. Cited by: SS1.
* D. Beck, T. Cohn, and L. Specia (2014)Joint emotion analysis via multi-task gaussian processes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 1798-1803. External Links: Link, Document Cited by: SS1.
* D. Beck, K. Shah, T. Cohn, and L. Specia (2013)SHEF-Lite: when less is more for translation quality estimation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pp. 337-342. External Links: Link, Document Cited by: SS1.
* J. Bergstra, R. Bardenet, Y. Bengio, and B. Kegl (2011-07)Algorithms for hyper-parameter optimization. In Advances in Neural Information Processing Systems, C. A. Associates, Inc. Cited by: SS1.
* S. Bhattacharyya, A. Rooshenas, S. Naskar, S. Sun, M. Iyyer, and A. McCallum (2021-07)Energy-based reranking: improving neural machine translation using energy-based models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 4528-4537. External Links: Link, Document Cited by: SS1.
* E. Charniak and M. Johnson (2005)Coarse-to-fine n-best parsing and maxEnt discriminative reranking. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pp. 173-180. External Links: Link, Document Cited by: SS1.
* L. Chen, M. Zaharia, and J. Zou (2023)Frugalgpt: how to use large language models while reducing cost and improving performance. Cited by: SS1.
* J. Cheng and A. Vlachos (2023)Faster minimum bayes risk decoding with confidence-based pruning. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 12473-12480. External Links: Link, Document Cited by: SS1.
* M. Collins and T. Koo (2005)Discriminative reranking for natural language parsing. Computational Linguistics31 (1), pp. 25-70. External Links: Link, ISSN 0021-909X, Link, [https://doi.org/10.1111/j.1077.113.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.021.0021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.0210.021.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.021021.0210.0210.0210.021021.0210.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.021021.02102.021021](https://doi.org/10.1111/j.1077.113.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.0021.021.0021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.021.0210.021.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.0210.021021.0210.0210.0210.021021.0210.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.0210.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.02102.021021.021021.02102.021021) 

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023b. Results of WMT23 metrics shared task: Metrics might be guilty but references are not innocent. In _Proceedings of the Eighth Conference on Machine Translation_, 578-628. Association for Computational Linguistics.
* Hennig and Schuler (2011) Philipp Hennig and Christian J. Schuler. 2011. Entropy search for information-efficient global optimization. _ArXiv_, abs/1112.1217.
* Hewitt et al. (2022) John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In _Findings of the Association for Computational Linguistics: EMNLP 2022_, 3414-3427. Association for Computational Linguistics.
* Jones (2001) Donald R. Jones. 2001. A taxonomy of global optimization methods based on response surfaces. _Journal of Global Optimization_, 21:345-383.
* Juraska et al. (2023) Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023metrics shared task. In _Proceedings of the Eighth Conference on Machine Translation_, 756-767. Association for Computational Linguistics.
* Kandasamy et al. (2016) Kirthevasan Kandasamy, Gautam Dasarathy, Junier B Oliva, Jeff Schneider, and Barnabas Poczos. 2016. Gaussian process bandit optimisation with multi-fidelity evaluations. In _Advances in Neural Information Processing Systems_, volume 29. Curran Associates, Inc.
* Kecmi et al. (2024a) Tom Kocmi, Eleftherios Avramidis, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Markus Freitag, Thammen Gowda, Roman Grundkiewicz, Barry Haddow, Marzena Karpinska, Philipp Kobenh, Benjamin Marie, Christof Monz, Kenton Murray, Masasaki Nagata, Martin Popel, Maja Popovic, Mariya Shmatova, Steinthor Steingrimsson, and Vilem Zouhar. 2024a. Findings of the WMT24 general machine translation shared task: The LLM era is here but MT is not solved yet. In _Proceedings of the Ninth Conference on Machine Translation_, 1-46, Miami, Florida, USA. Association for Computational Linguistics.
* Kocmi et al. (2022) Tom Kocmi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Pebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novak, Martin Popel, and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, 1-45. Association for Computational Linguistics.
* Kocmi et al. (2024b) Tom Kocmi, Villem Zouhar, Christian Federmann, and Matt Post. 2024b. Navigating the metrics maze: Reconciling score magnitudes and accuracies. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 1999-2014, Bangkok, Thailand. Association for Computational Linguistics.
* Krause et al. (2008) Andreas Krause, Ajit Singh, and Carlos Guestrin. 2008. Near-optimal sensor placements in gaussian processes: Theory, efficient algorithms and empirical studies. _J. Mach. Learn. Res._, 9:235-284.
* Lee et al. (2021) Ann Lee, Michael Auli, and Marc'Aurelio Ranzato. 2021. Discriminative reranking for neural machine translation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, 7250-7264. Association for Computational Linguistics.
* Mockus (1974) Jonas Mockus. 1974. On bayesian methods for seeking the extremum. In _Proceedings of the IFIP Technical Conference_, 400-404.
* Noack et al. (2023) Marcus M. Noack, Harinarayan Krishnan, Mark D. Risser, and Kristofer G. Reyes. 2023. Exact gaussian processes for massive datasets via non-stationary sparsity-discovering kernels. _Scientific Reports_, 13(1).
* Petrov (2011) Slav Petrov. 2011. _Coarse-to-Fine Natural Language Processing (Theory and Applications of Natural Language Processing)_. Springer Publishing Company, Incorporated.
* Rasmussen and Williams (2005) C.E. Rasmussen and C.K.I. Williams. 2005. _Gaussian Processes for Machine Learning_. Adaptive Computation and Machine Learning series. MIT Press.
* Ravaut et al. (2022) Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022. SummaReranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 4504-4524. Association for Computational Linguistics.
* Rei et al. (2022a) Ricardo Rei, Ana C Farinha, Jose G.C. de Souza, Pedro G. Ramos, Andre F.T. Martins, Luisa Coheur, and Alon Lavie. 2022a. Searching for COMETINHO: The little metric that could. In _Proceedings of the 23rd Annual Conference of the European Association for Machine Translation_, 61-70. European Association for Machine Translation.
* Rei et al. (2024) Ricardo Rei, Jose Pombal, Nuno M. Guerreiro, Joao Alves, Pedro Henrique Martins, Patrick Fernandes, Helena Wu, Tania Vaz, Duarte Alves, Amin Farajian, Sweta Agrawal, Antonio Farinhas, Jose G. C. De Souza, and Andre Martins. 2024. Tower v2: Unbabel-IST 2024 submission for the general MT shared task. In _Proceedings of the Ninth Conference on Machine Translation_, 185-204, Miami, Florida, USA. Association for Computational Linguistics 

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, 2685-2702. Association for Computational Linguistics.
* Rei et al. [2022b] Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chryssoula Zerva, Ana C Farinha, Christine Maroti, Jose G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and Andre F. T. Martins. 2022b. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, 634-645. Association for Computational Linguistics.
* Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, 7881-7892. Association for Computational Linguistics.
* Shahriari et al. [2016] Bobak Shahriari, Kevin Swersky, Ziyun Wang, Ryan P. Adams, and Nando de Freitas. 2016. Taking the human out of the loop: A review of bayesian optimization. _Proceedings of the IEEE_, 104:148-175.
* Singhal et al. [2023] Prasann Singhal, Jiacheng Xu, Xi Ye, and Greg Durrett. 2023. EEL: Efficiently encoding lattices for reranking. In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 9299-9316. Association for Computational Linguistics.
* Srinivas et al. [2009] Niranjan Srinivas, Andreas Krause, Sham M. Kakade, and Matthias W. Seeger. 2009. Information-theoretic regret bounds for gaussian process optimization in the bandit setting. _IEEE Transactions on Information Theory_, 58:3250-3265.
* Team et al. [2022] NLLB Team, Marta R. Costa-jussa, James Cross, Onur Celebi, Maha Elbayad, Kenneth Heafield, Kevin Hefernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, AI Youngblood, Bapi Akula, Loic Barrault, Gabriel Mejia Gonzalez, Prangthy Hansanti, John Hoffman, Semarley Jarrett, Kaushik Ram Sadagopan, Dirk Rowe, Shannon Spruit, Chau Tran, Pierre Andrews, Necip Fazil Ayan, Shruti Bhosale, Sergey Edunov, Angela Fan, Cynthia Gao, Vedanuj Goswami, Francisco Guzman, Philipp Koehn, Alexandre Mourachko, Christophe Ropers, Safhyyah Saleem, Holger Schwenk, and Jeff Wang. 2022. No language left behind: Scaling human-centered machine translation.
* Trabseli et al. [2024] Firas Trabseli, David Vilar, Maria Finkelstein, and Markus Freitag. 2024. Efficient minimum bayes risk decoding using low-rank matrix completion algorithms.
* Vanvas and Sennrich [2024] Jannis Vanvas and Rico Sennrich. 2024. Linear-time minimum Bayes risk decoding with reference aggregation. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, 790-801, Bangok, Thailand. Association for Computational Linguistics.
* Vernikos and Popescu-Belis [2024] Giorgos Vernikos and Andrei Popescu-Belis. 2024. Don't rank, combine! combining machine translation hypotheses using quality estimation. In _Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, 12087-12105, Bangok, Thailand. Association for Computational Linguistics.
* Wang et al. [2024] Jun Wang, Eleftheria Briakou, Hamid Daddihai, Rishab Agarwal, Colin Cherry, and Trevor Cohn. 2024. Don't throw away data: Better sequence knowledge distillation.
* Wang et al. [2020] Wenhui Wang, Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. 2020. MiniLM: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
* Wu et al. [2020] Jian Wu, Saul Toscano-Palmerin, Peter I. Frazier, and Andrew Gordon Wilson. 2020. Practical multi-fidelity bayesian optimization for hyperparameter tuning. In _Proceedings of The 35th Uncertainty in Artificial Intelligence Conference_, volume 115 of _Proceedings of Machine Learning Research_, 788-798. PMLR.
* Wu et al. [2016] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, Jeff Klingner, Apurva Shah, Melvin Johnson, Xiaobing Liu, Lukasz Kaiser, Stephan Gouws, Yoshikiyo Kato, Taku Kudo, Hideto Kazawa, Keith Stevens, George Kurian, Nishant Patil, Wei Wang, Cliff Young, Jason Smith, Jason Riesa, Alex Rudnick, Oriol Vinyals, Greg Corrado, Macduff Hughes, and Jeffrey Dean. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation.
* Yee et al. [2019] Kyra Yee, Yann Dauphin, and Michael Auli. 2019. Simple and effective noisy channel modeling for neural machine translation. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, 5696-5701. Association for Computational Linguistics.
* Zerva et al. [2022] Chrysoula Zerva, Frederic Blain, Ricardo Rei, Piyawat Lervittyayakumjorn, Jose G. C. de Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orasant, Marina Fomicheva, Andre F. T. Martins, and Lucia Specia. 2022. Findings of the WMT 2022 shared task on quality estimation. In _Proceedings of the Seventh Conference on Machine Translation (WMT)_, 69-99. Association for Computational Lin 



\begin{table}
\begin{tabular}{l c r r r r r r r r r r}  & \multicolumn{10}{c}{**ContextKwi runs**} \\

- \\ ProxyFirst 50 Distilled-M & 3 & 0.8073 & 0.8107 & 0.8119 & 0.8122 & 0.8124 &
- \\ BayesOpt+GP & 2,4,3 & 0.7917 & 0.8121 & 0.8167 & 0.8190 & 0.8201 & 0.8206 & 0.8210 & 0.8213 & 0.8214 \\ BayesOpt+GP+PH with LogprohAvg & 4 & 0.7956 & 0.8123 & 0.8166 & 0.8187 & 0.8198 & 0.8205 & 0.8208 & 0.8210 & 0.8213 & 0.8214 \\ BayesOpt+P with 200 Distilled-S & 4,3 & 0.8081 & 0.8165 & 0.8190 & 0.8200 & 0.8207 & 0.8210 & 0.8212 & 0.8213 & 0.8214 & 0.8215 \\ BayesOpt+GP+PH with 200 Distilled-M & 4,3 & 0.8119 & 0.8182 & 0.8199 & 0.8205 & 0.8209 & 0.8211 & 0.8213 & 0.8214 & 0.8215 \\ BayesOpt+GP+PH with 50 Distilled-S & 4,3 & 0.8054 & 0.8153 & 0.8184 & 0.8196 & 0.8204 & 0.8208 & 0.8210 & 0.8213 & 0.8214 & 0.8214 \\ BayesOpt+GP+PH with 50 Distilled-M & 4,3 & 0.8073 & 0.8164 & 0.8187 & 0.8200 & 0.8207 & 0.8209 & 0.8211 & 0.8213 & 0.8214 & 0.8215 \\ \end{tabular}
\end{table}
Table 3: Exact values (selected candidate score) for Figures 2 to 4.


Statistical Significance

We measure statistical significance between two methods based on the final candidate CometKiwi scores with either budget 30, 60, 90, or across the budget range from 10 to 190 in Table 4. To determine whether one method is better than another one, we use one-sided paired Student's t-test with p-value threshold \(0.01\) which is run across the individual samples.

[TABLE:A2.T4][E

\begin{table}
\begin{tabular}{r r c