Title: Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms
Venue: NeurIPS 2024
Accepted: yes
Keywords: Machine Translation

Minimum Bayes Risk

Natural Language Processing

Low-Rank Matrix Completion
Abstract: Minimum Bayes Risk (MBR) decoding is a powerful decoding strategy widely used for text generation tasks but its quadratic computational complexity limits its practical application. This paper presents a novel approach for approximating MBR decoding using matrix completion techniques, focusing on a machine translation task. We formulate MBR decoding as a matrix completion problem, where the utility metric scores between candidate hypotheses and reference translations form a low-rank matrix. First we empirically show that the scores matrices indeed have a low-rank structure. Then we exploit this by only computing a random subset of the scores and efficiently recover the missing entries in the matrix by applying the Alternating Least Squares (ALS) algorithm, thereby enabling fast approximation of the MBR decoding process. Our experimental results on machine translation tasks demonstrate that the proposed method requires 1/16 utility metric computations compared to the vanilla MBR decoding while achieving equal translation quality measured by COMET on the WMT22 dataset (en<>de, en<>ru). We also benchmark our method against other approximation methods and we show significant gains in quality.

Efficient Minimum Bayes Risk Decoding using Low-Rank Matrix Completion Algorithms

Abstract

Minimum Bayes Risk (MBR) decoding is a powerful decoding strategy widely used for text generation tasks, but its quadratic computational complexity limits its practical application. This paper presents a novel approach for approximating MBR decoding using matrix completion techniques, focusing on the task of machine translation. We formulate MBR decoding as a matrix completion problem, where the utility metric scores between candidate hypotheses and pseudo-reference translations form a low-rank matrix. First, we empirically show that the scores matrices indeed have a low-rank structure. Then, we exploit this by only computing a random subset of the scores and efficiently recover the missing entries in the matrix by applying the Alternating Least Squares (ALS) algorithm, thereby enabling a fast approximation of the MBR decoding process. Our experimental results on machine translation tasks demonstrate that the proposed method requires 1/16 utility metric computations compared to vanilla MBR decoding while achieving equal translation quality measured by COMET22 on the WMT22 dataset (en ↔ de, en ↔ ru). We also benchmark our method against other approximation methods and we show gains in quality when comparing to them.

1 Introduction

The generation process in most conditional natural language processing tasks is usually guided by the maximum-a-posteriori (MAP) rule: given an input x , generate the output ˆ y that maximizes the posterior probability distribution: ˆ y = argmax y p ( y | x ) . It can be shown that MAP decoding is optimal under a 0-1 loss criterion. However for more nuanced tasks, where different outputs can be considered correct, and the quality of the output is not just a binary decision between 'right' and 'wrong', MAP decoding is no longer optimal. Neural Machine Translation (NMT) is a prominent example of these types of tasks. For NMT, a system is trained to generate a sentence in a target language given a source sentence in another language. For a given sentence, there exists a variety of possible translations, each of which has a different quality level. Eikema and Aziz (2020) demonstrated that MAP decoding methods are suboptimal for NMT, and showed that other generation strategies may be preferred. Furthermore, NMT models often assign human translations lower probabilities than their own beam search outputs, due to model calibration issues (Ott et al., 2018; Freitag et al., 2020).

As an alternative, Eikema and Aziz (2020, 2022) applied MBR decoding for NMT models. MBR decoding follows a two-step approach, where a model is used to generate a list of candidate translations and a list of pseudo-references (which may be the same as the list of candidates). The candidates

Pseudo-referencesR

r1: Hi, world!

r2:hi, everyone!

r3:Hi!

r1: Hi, world!

!H:

0.85

0.70

0.82

0.85

0.70

0.79

Hypotheses H

h1:Hello,world!

h1:Hello, world!

ExactValue

0.81

0.80

MC

0.81

0.80

0.65

Avg

h2

0.75

Unknown Value

Approximated

h2: Hey,world!

h2: Hey, world!

Value

0.72

0.90

0.72

0.70

0.90

0.77

Completion

MC:Matrix

h3:Hello!

h3: hello!

are then scored with a performance metric using the pseudo-references as an approximation of the true references, and the candidate with the maximum expected quality (or equivalently minimum risk) is then selected. In contrast to MAP decoding, MBR decoding is not designed to generate the translation with the highest estimated model probability; instead it aims to directly optimize a utility function. Subsequent research conducted by Freitag et al. (2022a) showed that MBR decoding with neural utility metrics leads to significant improvements over beam search decoding. However, MBR is computationally expensive, with a time complexity of O ( N 2 ) for a candidate list containing N samples and N pseudo-references (usually the two lists coincide). According to Freitag et al. (2022a), ideally N ranges between 100 and 1 000 , which involves thousands to millions of utility function computations. Note than when using neural metrics, each of the O ( N 2 ) 'computation steps' is itself expensive, requiring a forward pass through a large neural network.

In this work, we propose to reduce the number of metric computations by scoring only a subset of candidate-pseudo-reference pairs. We then proceed to use a matrix completion algorithm (ALS in our case) to estimate the remaining utility scores. For such completion algorithms to work, the full matrix has to fulfill some conditions, specifically to be a low-rank matrix. We empirically show that this is indeed the case for the utility matrices for MBR decoding. Intuitively, this makes sense: akin to recommendation systems, where similar users are expected to have similar book or movie preferences, similar pseudo-references are expected to have similar 'translation preferences'. Experimental results show the effectiveness of our method, where the performance of the full MBR algorithm can be matched with only a fraction of the computational cost. Compared to other approaches that also seek to reduce the number of computations, our method does not compromise translation quality, as confirmed by human evaluation.

Our scientific contributions are as follows:

We empirically show that the utility matrices for MBR decoding are low-rank.

We apply ALS to a subset of scores to approximate the full MBR matrix.

We show that using our method we can reduce the number of computations by a factor of 16, while maintaining the same translation quality level.

2 Related Work

While MT research has traditionally relied on MAP decoding or generating k -best lists through beam search for MBR decoding, Eikema and Aziz (2020) proposed an approximation of MBR decoding via unbiased sampling. Their method aims to address the limitations of MAP decoding (Eikema and Aziz, 2020; Müller and Sennrich, 2021; Eikema and Aziz, 2022) by demonstrating that samples drawn from the NMT model align more faithfully with training data statistics when compared to beam

search. Freitag et al. (2022a) showed that using neural metrics results in significant improvements in translation quality. Freitag et al. (2023a) reported that the choice of sampling approach is important, and epsilon sampling (Hewitt et al., 2022) is ideal for MBR decoding and reranking.

While the improvements in translation quality afforded by MBR are widely acknowledged, its high computational cost limits its application in practice. Different approaches have been proposed to speed up MBR computation. Eikema and Aziz (2022) propose to decouple the candidate and pseudoreference lists to allow for different sizes, and propose a coarse-to-fine refinement of the hypothesis space. Cheng and Vlachos (2023) speed up MBR decoding by gradually increasing the number of samples used to estimate the utility, while additionally pruning the hypothesis space. Jinnai and Ariu (2024) formulate MBR as a medoid identification problem, and apply approximate algorithms developed on this problem. Vamvas and Sennrich (2024) aggregate the set of pseudo-references, allowing for just one utility computation per candidate. This greatly accelerates the decoding process, but the utility metric needs to fulfill certain conditions to be applicable. Finkelstein et al. (2024) use MBR decoding in a knowledge-distillation framework to simulate MBR decoding with single-pass search. Tomani et al. (2024) train quality-aware translation models in order to reduce the size of the candidate list. Similar in spirit to MBR decoding, QE-rescoring approaches (Fernandes et al., 2022) also directly optimize a utility function, with linear-time cost.

Low-Rank Matrix completion is an active area of research and multiple algorithms have been developed to perform it. Nguyen et al. (2019) is an extensive survey for such methods. Some of the popular algorithms are: Singular Value Thresholding (Cai et al., 2008), Bayesian Probabilistic Matrix Factorization (Akulwar and Pardeshi, 2016), Maximum Margin Matrix Factorization (Srebro et al., 2004) and Alternating Least Squares (Zachariah et al., 2012), which is the one we chose for this work. To the best of our knowledge, this work is the first one to apply matrix completion algorithms for completing a partial MBR score matrix.

3 Preliminaries

We are given an NMT model P Θ ( y | x ) which serves to estimate the probability of a hypothesis segment y , given a source segment x , with Θ being the learned parameters of the neural network. MAP decoding involves searching for the most probable translation under P Θ ( y | x ) . However, determining the hypothesis with the maximum probability is computationally intractable due to the expansive and combinatorially complex search space. Consequently, approximations like beam search (Graves, 2012; Sutskever et al., 2014) are often employed.

If we want to generate diverse hypotheses, e.g. in generative tasks where creativity is desired instead of selecting the candidate with the highest probability (or an approximation thereof), we sample the output sentence following the probability distribution defined by the model. For NMT, this approach is used for generating a list of candidate translations. Specifically, epsilon sampling, as outlined by Hewitt et al. (2022), has emerged as the leading sampling technique for MBR. It was shown by Freitag et al. (2023a) to outperform other methods such as ancestral, topk or nucleus sampling (Holtzman et al., 2020). Epsilon sampling prunes away any token with a probability lower than a threshold ε , thereby guaranteeing that each token within a sample is allocated a fair probability mass.

3.1 Minimum Bayes Risk Decoding

In MBR decoding (Bickel and Doksum, 1977; Berger, 1985), given a set of candidate hypotheses H , the goal is to select the optimal hypothesis based on its expected utility, measured by a function u , with respect to the distribution over human references within the space of all references Y .

Since the true distribution remains unknown, we resort to sampling from the model instead, which relies on the assumption that the model provides a reliable approximation for the true underlying distribution over human translations. Furthermore, the integration over the vast space of all possible references Y is computationally intractable. Therefore, MBR adopts a finite sample estimate by sampling a set of pseudo-references R from P θ ( ·| x ) . This approximation can be expressed as:

Usual practice is to set H = R , i.e. the same set of model hypotheses serves both as the candidate list H as well as the pseudo-reference list R . The computational time complexity of MBR decoding is O ( N 2 ) with N the size of the candidate list.

Note that this quadratic expression refers to each sentence to translate, i.e. for a corpus of size S , the total cost will be O ( S · N 2 ) . Also there is a hidden (multiplicative) constant, namely the cost of the computation of the utility function. For surface level metrics (e.g. BLEU, ChrF), this cost is negligible, but for neural metrics it involves computing the forward pass of a large neural network. Therefore, any reduction in the number of metric computations has an important effect on the total running cost.

3.2 Low-Rank Matrix Completion

Low-Rank Matrix Completion is a fundamental problem in machine learning and data analysis with popular application such as Collaborative Filtering (Rennie and Srebro, 2005) and Image Denoising (Candes and Recht, 2008). The goal of matrix completion is to estimate the missing entries of a partially observed matrix, under the assumption that the underlying matrix is low-rank. This assumption implies that the matrix can be well-approximated by a product of two smaller matrices, capturing the latent factors that explain the observed data. Candes and Recht (2008) proved that perfect approximations can be achieved if the number of observed entries is larger than CN 1 . 2 r log( N ) for some positive numerical constant C , for most N × N matrices of rank r with very high probability.

One simple and efficient algorithm is Alternating Least Squares (ALS) (Zachariah et al., 2012). To recover any matrix M , the algorithm approximates it by two smaller matrices M ≈ X T Y and then minimizes the following equation given the observed entries.

The algorithm achieves this by alternatively solving for X and Y as shown in Algorithm 1 The algorithm has three hyperparameters: λ a regularization term, r the second dimension of the smaller matrices and n the number of alternating steps performed. The main motivation for picking this algorithm in our approach is its simple implementation.

4 MBRMatrix

4.1 Definition of MBR matrix

Given a source sentence, we use an NMT model to generate a set H of hypotheses such that |H| = N . As explained in the preliminaries section, the MBR method uses two different sets of hypotheses and pseudo-references, but in practice we use the same set of samples for both H and R . The pairwise scores for all hypotheses in H gives an N × N matrix M such that M [ i, j ] = U ( h i , h j ) for all ( h i , h j ) ∈ H × H and a utility metric U that computes some similarity between two hypotheses.

Metricx

chrF

102

102

101

101

Singular Value

Singular Value

100

100

10-1

10-2

10-1

0

20

40

60

80

100

120

0

20

40

60

80

100

120

Order

Order

With this matrix formulation, MBR decoding reduces to picking the row with the highest average (since each row maps to one sample in the hypotheses list).

4.2 MBRmatrices are low rank

Intuitively, the values within the MBR matrix are highly correlated, since by definition each value M [ i, j ] is computing a similarity score between two hypotheses given a utility metric. This is a key assumption that our work is built on since low-rank matrices have theoretical bounds on the number of entries required to recover the full matrix (Candes and Recht, 2008).

We verify this assumption empirically. We generated 1024 samples for each example in the WMT 2022 en ↔ de and en ↔ ru datasets. We then generated the N × N matrices for different values of N ∈ { 64 , 128 } by only considering a random subset of the samples using two different utility metrics: MetricX (Freitag et al., 2022b) and ChrF (Popovi´ c, 2015). We then perform singular value decomposition and look at the distribution of the singular values, shown in Figure 2 and Table 1. We observe that across utility metrics and matrix dimensions, σ 1 glyph[greatermuch] σ 2 . On average across datasets we have σ 2 /σ 1 < 0 . 05 , this means that most of the information within the matrix can be captured by a single dominant direction or component and thus can be approximated by a rank-1 matrix.

5 The PMBR Method

We propose an approximation method for MBR decoding that leverages the low-rank structure of the MBR matrix. The procedure is shown in Algorithm 2. Given an NMT model, we start by generating a set of hypotheses H similar to the vanilla MBR method. Then, instead of computing all the pairwise scores in the utility matrix, we only compute a random subset of the scores that we denote with Ω . The size of Ω depends on the computation budget available. We define the budget as the ratio of

computations performed with respect to the total amount of computations to compute the full matrix. Thus for any given budget 1 /r , we end up with N 2 /r entries observed in the matrix. The next step is to apply ALS on M [Ω] as described in Algorithm 1, where M [Ω] denotes the matrix of size N × N where only the entries of Ω are non-null. Finally, with all the pairwise scores recovered, we perform vanilla MBR decoding. We call this procedure PMBR for Probabilistic MBR decoding.

Time Complexity The time complexity of this algorithm is dominated by the utility metrics computations. The utility metrics are deep neural networks that require a number of floating-point operations in the order of millions while ALS requires only a few hundred operations. For reference, 30 steps of the ALS algorithm with r = 10 running on a CPU takes on average 0.2 seconds to run while the MetricX inference takes 3.4 seconds on a TPUv4 platform. Thus, the savings in run time achieved by our approximation is close to proportional to the savings in number of utility computations. Note that this analysis focuses only on the second stage of MBR decoding, i.e. we do not take the cost of generating the hypotheses into account.

6 Experimental Setup

6.1 Metrics

We use MetricX (Juraska et al., 2023) as the utility function for all variants of MBR decoding as it has been shown that neural fine-tuned metrics outperform word-overlap metrics like BLEU (Papineni et al., 2002) and ChrF (Popovi´ c, 2015) for MBR decoding (Freitag et al., 2022a). MetricX is an extension of BLEURT (Sellam et al., 2020), showing higher correlation with human judgment (Freitag et al., 2023b) and has been designed to also work on multi-sentence segments (Deutsch et al., 2023) and not only sentences in isolation. In addition, we report COMET22 (Rei et al., 2020, 2022) scores as there is a risk of overfitting (Amrhein and Sennrich, 2022) on MetricX. In addition, for one selected experiment we conducted expert-based human evaluations using MQM (Freitag et al., 2021), a human evaluation scheme centered on marking errors present in the translations. We report results by varying the budget available to the MBR methods. For each budget, we randomly sample from the full MBR matrix, and report the average results of 1000 trials.

6.2 Datasets and Model

We run experiments using the WMT 2022 test sets for English ↔ German (en ↔ de) and English ↔ Russian (en ↔ ru). The official WMT test sets (Kocmi et al., 2022) are split into sentences but come with document information. We constructed multi-sentence (paragraph) level test sets with the following method: for each document, we concatenate sentences together as long as we do not exceed 500 sentence piece model (SPM) tokens (given the MetricX SPM model). We respect sentence boundaries and do not truncate sentences. In WMT22, there are four different domains. Some domains lack document context, so segments remain as single sentences, even within multi-sentence test sets. Test data statistics can be seen in Appendix 5. We use PaLM8B (Chowdhery et al., 2022) as translation model and sample 1024 examples for each sentence using epsilon sampling with glyph[epsilon1] = 0 . 02 (Freitag et al., 2023a) and using 3-shot prompting with examples taken from the FLORES corpus (Guzmán et al., 2019)

budget

1

/

32

1

/

16

1

/

8

1

/

4

1

/

2

1

/

1

# utility calls

512

1024

2048

4096

8192

16384

Matrix

size

|

Ω

|

)

512

1 024

2 048

4 096

8 192

16 384

N

×

K

128

×

4

128

×

8

128

×

16

128

×

32

128

×

64

128

×

128

S

×

S

22

×

22

32

×

32

45

×

45

64

×

64

90

×

90

128

×

128

73.6 -

80.1

73.4

80.0

MetricX

COMET

79.9

73.2

79.8

73.0

full_mbr

full_mbr

approx(PMBR)

79.7

approx(PMBR)

72.8

approx(Sxs)

approx(Sxs)

approx(NxK)

approx(NxK)

512

1024

2048

4096

8192

16384

512

1024

2048

4096

8192

16384

#UtilityCalls

#UtilityCalls

6.3 Decoding Methods

We compare our approximation PMBR against three other decoding methods. To enable a fair comparison, we adapt each method so that the number of utility function computations is the same for each method. (Recall that for a given budget 1 /r , we only observe N 2 /r entries in the matrix when performing PMBR.) We compare PMBR with the following methods:

FMBR: This is the full MBR method. This is the only method that is not affected by the budget i.e the full matrix is observed.

N × K : This method was proposed by Eikema and Aziz (2022) and works by shrinking the pseudo-references list size. For a budget 1 /r the MBR matrix gets reduced to an N × K matrix with K = N/r . The K pseudo-references are randomly sampled.

S × S : This method corresponds to FMBR, but reduces the total size of the utility matrix to a size of S · S , where the total number of entries corresponds to the available budget, i.e. S = √ N 2 /r . The S examples are randomly sampled.

6.4 Hyperparameter Tuning

The ALS algorithm has three hyperparameters λ , n and r as described in Algorithm 1. We perform a grid search to optimize these hyperparameters, setting our loss function to be the accuracy with respect to the vanilla MBR method. Concretely, for each example sentence we rank all samples by running the vanilla MBR. Let us denote with h MBR the hypothesis selected by the full MBR method, and let pos PMBR ( h MBR ) the rank the position of h MBR after ordering the hypotheses according to the scores predicted by PMBR. The loss function is then just the sum of pos PMBR ( h MBR ) for all the hypotheses in a subset of the data. We minimize this loss per language pair on 10 examples that we hold from the data generated with the WMT 2022 datasets with this search space { λ ∈ { 0 . 1 , 0 . 15 , 0 . 2 }} × { r ∈ { 5 , 6 , . . . 14 , 15 }} × { n ∈ { 10 , 11 , . . . , 29 , 30 }}

7 Results

The main experimental results are summarized in Table 2 and Table 3. In Table 2, we fix N = 128 and we study the behaviour of each approximation method by limiting their budget to a fraction of the

full computational cost on each language pair. The top row comprises the results obtained with the full MBR method ( FMBR ) running on the complete list of N = 128 candidates, and can be considered as an upper bound for the performance of each approximation method. The number of utility calls for FMBR is 128 2 = 16384 . In Figure 3, we plot the data for the for de → en from Table 2. In Table 3, we fix the language pair to (de ↔ en) and we set N to different values. This simulates the behavior of approximation methods as the candidate list grows. Similar results for (en ↔ de) are shown in Appendix 6. MQM human evaluation results are summarized in Table 4.

As measuring performance with the same metric we are optimizing for has the risk of overfitting, we mainly focus on COMET22 to assess translation quality. These are the main findings:

(1) PMBR outperforms all other tested approximation methods PMBR outperforms both the N x K and S x S approximation methods across language pairs, sample sizes and budgets. The gap between the approximation methods closes as the budget increases. Moreover the results in Table ?? show that the same pattern holds when the size of the hypotheses list changes.

(2) PMBR is competitive to FMBR We can reduce the computational cost by up to r = 1 / 32 with PMBR without any loss in translation quality as measured by COMET22. Interestingly, we observe that MetricX scores slightly drop when reducing the budget. As this does not affect the final translation quality as measured by COMET22, we argue that this is a good sign and PMBR acts as some kind of regularization.

(3) Human Evaluation confirms (1) and (2) To verify our findings based on COMET22, we do run a MQM human evaluation with professional translators. Results are summarized in Table 4. The results confirm our previous findings: (1) PMBR is the best approximation method when compared to N x K and S x S , and (2) PMBR is getting close to the performance of FMBR.

8 Conclusions

In this paper we have shown the inherent low-rank structure of Minimum Bayes Risk (MBR) score matrices which we leveraged to develop an approximation method for MBR decoding that achieves competitive performance while significantly reducing computational complexity. Our empirical results demonstrate the efficacy of this approach across diverse language pairs and evaluation metrics, suggesting its potential for wider application in machine translation and other natural language generation tasks.

Future research could explore the efficacy of alternative matrix completion algorithms to further enhance the low-rank approximation. In addition, the observed low-rank property could be exploited to inform sampling strategies, potentially leading to more efficient and informative data collection for MBR decoding. Another promising avenue is to investigate the applicability of this work to domains beyond natural language generation tasks.

9 Limitations

While we have verified that the MBR matrices are low-rank, we did not conduct an empirical analysis on their coherence. A low-rank matrix is easier to complete if its energy spreads evenly across different coordinates. This property is captured by the notion of coherence (Candes and Recht, 2008).

In this paper, we only run experiments with MetricX as utility function. The computational costs for computing all pairwise utility scores is expensive. However, we showed that the low-rank matrix structure holds for both MetricX and chrF which gives us confidence that PMBR will generalize regardless of the utility function.

Our human evaluation is limited in size because it is costly. With automatic metrics, we can simulate multiple runs of scoring the datasets but this is not feasible with human evaluations. Thus, we put less statistical significance on our human evaluation.

References

Pooja Akulwar and Sujata Pardeshi. 2016. Bayesian probabilistic matrix factorization - a dive towards recommendation. In 2016 International Conference on Inventive Computation Technologies (ICICT) , volume 3, pages 1-5.

Chantal Amrhein and Rico Sennrich. 2022. Identifying weaknesses in machine translation metrics through minimum Bayes risk decoding: A case study for COMET. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 1125-1141, Online only. Association for Computational Linguistics.

James O Berger. 1985. Statistical decision theory and Bayesian analysis; 2nd ed. Springer series in statistics. Springer, New York.

Peter J Bickel and Kjell A Doksum. 1977. Mathematical statistics: Basic ideas and selected topics. Holder-Day Series in Probability and Statistics, Holder-Day, San Francisco .

Jian-Feng Cai, Emmanuel J. Candes, and Zuowei Shen. 2008. A singular value thresholding algorithm for matrix completion.

Emmanuel J. Candes and Benjamin Recht. 2008. Exact matrix completion via convex optimization.

Julius Cheng and Andreas Vlachos. 2023. Faster minimum bayes risk decoding with confidence-based pruning.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language modeling with pathways.

Daniel Deutsch, Juraj Juraska, Mara Finkelstein, and Markus Freitag. 2023. Training and metaevaluating machine translation evaluation metrics at the paragraph level. In Proceedings of the Eighth Conference on Machine Translation , pages 996-1013, Singapore. Association for Computational Linguistics.

Bryan Eikema and Wilker Aziz. 2020. Is MAP decoding all you need? the inadequacy of the mode in neural machine translation. In Proceedings of the 28th International Conference on Computational Linguistics , pages 4506-4520, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Bryan Eikema and Wilker Aziz. 2022. Sampling-based approximations to minimum Bayes risk decoding for neural machine translation. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing , pages 10978-10993, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages 1396-1412, Seattle, United States. Association for Computational Linguistics.

Mara Finkelstein, Subhajit Naskar, Mehdi Mirzazadeh, Apurva Shah, and Markus Freitag. 2024. Mbr and qe finetuning: Training-time distillation of the best and most expensive decoding methods.

Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 2021. Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics , 9:1460-1474.

Markus Freitag, Behrooz Ghorbani, and Patrick Fernandes. 2023a. Epsilon sampling rocks: Investigating sampling strategies for minimum Bayes risk decoding for machine translation. In Findings of the Association for Computational Linguistics: EMNLP 2023 , pages 9198-9209, Singapore. Association for Computational Linguistics.

Markus Freitag, David Grangier, and Isaac Caswell. 2020. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 61-71, Online. Association for Computational Linguistics.

Markus Freitag, David Grangier, Qijun Tan, and Bowen Liang. 2022a. High quality rather than high model probability: Minimum bayes risk decoding with neural metrics. Transactions of the Association for Computational Linguistics , 10:811-825.

Markus Freitag, Nitika Mathur, Chi-kiu Lo, Eleftherios Avramidis, Ricardo Rei, Brian Thompson, Tom Kocmi, Frederic Blain, Daniel Deutsch, Craig Stewart, Chrysoula Zerva, Sheila Castilho, Alon Lavie, and George Foster. 2023b. Results of wmt23 metrics shared task: Metrics might be guilty but references are not innocent. In Proceedings of the Eighth Conference on Machine Translation , pages 578-628, Singapore. Association for Computational Linguistics.

Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. T. Martins. 2022b. Results of WMT22 metrics shared task: Stop using BLEU - neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 46-68, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Alex Graves. 2012. Sequence transduction with recurrent neural networks. arXiv preprint arXiv:1211.3711 .

Francisco Guzmán, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc'Aurelio Ranzato. 2019. The FLORES evaluation datasets for low-resource machine translation: Nepali-English and Sinhala-English. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 6098-6111, Hong Kong, China. Association for Computational Linguistics.

John Hewitt, Christopher Manning, and Percy Liang. 2022. Truncation sampling as language model desmoothing. In Findings of the Association for Computational Linguistics: EMNLP 2022 , pages 3414-3427, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Yuu Jinnai and Kaito Ariu. 2024. Hyperparameter-free approach for faster minimum bayes risk decoding.

Juraj Juraska, Mara Finkelstein, Daniel Deutsch, Aditya Siddhant, Mehdi Mirzazadeh, and Markus Freitag. 2023. MetricX-23: The Google submission to the WMT 2023 metrics shared task. In Proceedings of the Eighth Conference on Machine Translation , pages 756-767, Singapore. Association for Computational Linguistics.

Tom Kocmi, Rachel Bawden, Ondˇ rej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel, and Maja Popovi´ c. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 1-45, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Mathias Müller and Rico Sennrich. 2021. Understanding the properties of minimum Bayes risk decoding in neural machine translation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages 259-272, Online. Association for Computational Linguistics.

Luong Trung Nguyen, Junhan Kim, and Byonghyo Shim. 2019. Low-rank matrix completion: A contemporary survey. IEEE Access , 7:94215-94237.

Myle Ott, Michael Auli, David Grangier, and Marc'Aurelio Ranzato. 2018. Analyzing uncertainty in neural machine translation. In International Conference on Machine Learning , pages 3956-3965. PMLR.

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.

Maja Popovi´ c. 2015. chrF: character n-gram F-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation , pages 392-395, Lisbon, Portugal. Association for Computational Linguistics.

Ricardo Rei, José G. C. de Souza, Duarte Alves, Chrysoula Zerva, Ana C Farinha, Taisiya Glushkova, Alon Lavie, Luisa Coheur, and André F. T. Martins. 2022. COMET-22: Unbabel-IST 2022 submission for the metrics shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT) , pages 578-585, Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.

Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MTevaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 2685-2702, Online. Association for Computational Linguistics.

Jasson D. M. Rennie and Nathan Srebro. 2005. Fast maximum margin matrix factorization for collaborative prediction. In Proceedings of the 22nd International Conference on Machine Learning , ICML '05, page 713-719, New York, NY, USA. Association for Computing Machinery.

Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 7881-7892, Online. Association for Computational Linguistics.

Nathan Srebro, Jason Rennie, and Tommi Jaakkola. 2004. Maximum-margin matrix factorization. In Advances in Neural Information Processing Systems , volume 17. MIT Press.

Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to sequence learning with neural networks. Advances in neural information processing systems , 27.

Christian Tomani, David Vilar, Markus Freitag, Colin Cherry, Subhajit Naskar, Mara Finkelstein, Xavier Garcia, and Daniel Cremers. 2024. Quality-aware translation models: Efficient generation and quality estimation in a single model.

Jannis Vamvas and Rico Sennrich. 2024. Linear-time minimum bayes risk decoding with reference aggregation.

Dave Zachariah, Martin Sundin, M. Jansson, and Saikat Chatterjee. 2012. Alternating least-squares for low-rank matrix reconstruction. IEEE Signal Processing Letters , 19:231-234.

A Appendix / supplemental material

A.1 DeEn graphs for all hypothesis size lists

Figures 7, 5 and 6 show the performance of PMBR for different hypothesis list sizes. The number are represented in the main results section in Table 3. PMBR outperforms all the other methods measured by both COMET22 and MetricX.

A.2 WMT2022 paragraph level data statistics

We provide the statistics of the dataset used after combining the sentences from the same document in Table 5. The procedure to create paragraph level data is described in details in the data section.

A.3 Scores for EnDE while varying the hypotheses list size

In Table 6, we summarize all COMET22 and MetricX scores after varying the hypotheses list size. We see similar results as shown in the main results sections. PMBR outperforms all the other approximation methods in most cases.

A.4 Standard deviations for DeEn and significance test values

In Table 7, we present the standard deviation for values presented in Table 3. We also run a p-value significance test to verify that the gap in performance between PMBR and the other systems is significant.

A.5 Other matrix completion algorithms

We experimented with running Singular Value Thresholding (SVT) instead of Alternating Least Square (ALS) algorithm to perform the matrix completion. SVT under performed compared to ALS as shown in Figure 8. This behavior might be caused by a mistuning of the SVT hyperparameters, but it highlights the importance of the matrix completion algorithm in the PMBR procedure.

A.6 Different samples for the hypotheses and pseudo-references lists

In all our experiments, we have used the same set of samples for both hypotheses and pseudoreferences which is a common practice for MBR decoding in the NMT use case. In Figure 9, we verify that the matrices are still low rank even when the two set of samples are different. We also run an experiment to benchmark the performance of PMBR in this case. The results are shown in Figure 10

73.6

80.1

73.4

80.0

73.2

79.9

73.0

79.8

MetricX

COMET

72.8

79.7

72.6

79.6

72.4

79.5

qii

full_mbr

72.2

approx(PMBR)

79.4

approx(PMBR)

approx(SxS)

approx(SxS)

72.0

approx(NxK)

79.3

approx(NxK)

64

128

256

512

1024

64

128

256

512

1024

#UtilityCalls

#UtilityCalls

WMT22 paragraph-level DeEn scored 1000 times with N=64

73.50

80.0

73.25

73.00

79.8

MetricX

COMET

72.75

79.6

72.50

72.25

qin

79.4

full_mbr

approx(PMBR)

approx(PMBR)

approx(SxS)

approx(SxS)

72.00

approx(NxK)

approx(NxK)

128

256

512

1024

2048

4096

128

256

512

1024

2048

4096

#Utility Calls

#Utility Calls

WMT22paragraph-levelDeEnscored1000 times withN=256

80.1

73.6

80.0

73.4

MetricX

COMET

79.9

73.2

79.8

73.0

ful_mbr

79.7

full_mbr

approx(PMBR)

approx(PMBR)

72.8

approx(SxS)

approx(SxS)

approx(NxK)

79.6

approx(NxK)

1024

2048

4096

8192

1638432768

65536

1024

2048

4096

8192

1638432768

65536

#Utility Calls

#UtilityCalls

A.7 Compute Resources

We give a high level estimate of the resources to run the experiments:

Samples Generation: We used around 500 TPUv5 for 5 hours per language pair to generate the samples.

MetricX pairwise computations: We used around 2000 TPUv4 for 24 hours per language pair to compute all the scores.

Scoring simulations: These were run on CPUs in parallel on a cluster of 1000 machines. Each setting (budget, hypothesis length) takes around 15 minutes to run.

p-Value significance test for DeEn N=128 for PMBR vs all other systems

MetricX

COMET

mbr

SXS

nxs

mbr

SXS

nxs

1

1

32

0.00

0.00

0.00

32

0.00

0.00

0.00

16

0.00

0.00

0.00

Significance

16

0.00

0.00

0.00

Significance

8

0.03

0.00

0.00

8

0.00

0.00

0.00

4

0.42

0.00

0.00

4

0.00

0.00

0.00

2

一

0.00

0.00

0.00

2

0.00

0.00

0.00

0

0

WMT22 paragraph-level DeEn scored 1000 times with N=128

80.1

73.6

80.0

73.4

MetricX

COMET

79.9

73.2

79.8

73.0

full_mbr

=-full_mbr

approx(PMBR_SVT)

approx(PMBR_SVT)

approx(PMBR_ALS)

79.7

approx(PMBR_ALS)

72.8

approx(Sxs)

approx(SxS)

approx(NxK)

approx(NxK)

79.6

512

1024

2048

4096

8192

16384

512

1024

2048

4096

8192

16384

# Utility Calls

# Utility Calls

MetricX-Candidate == Ref

MetricX -Candidate != Ref

102

102

101

101

Singular Value

Value

100

100

10-1

gular

10-1

Sing

10-2

10-3

10-2

10-4

10-3

10-5

0

20

40

60

80

100

120

0

20

40

60

80

100

120

Order

Order

WMT22 paragraph-level DeEn scored 1000 times with N=128, with [H|!= |R|

80.1

73.6

80.0

73.4

MetricX

COMET

73.2

79.9

73.0

full_mbr

79.8

ful_mbr

approx(PMBR)

approx(PMBR)

approx(SxS)

approx(SxS)

72.8

approx(NxK)

approx(NxK)

79.7

512

1024

2048

4096

8192

16384

512

1024

2048

4096

8192

16384

#UtilityCalls

#UtilityCalls

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes] .

Justification: Our experimental results show the claims made in the abstract.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The paper includes a Limitations section that discusses this.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA] .

Justification: The answer NA means that the paper does not include theoretical results.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The PaLM64B is avaialble through the cloud API. MetricX and COMENT are publicaly avaialble. The PMBR algorithm code is submitted as supplemental.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We share the code of all the approximation methods used including our new algorithm. We also use an open dataset WMT22. We are working on clearing internal Google policies to release the generated data and scores.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: We share our hypeparameter search space and optimizer. We also share our data splits for eval and train.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report 1-sigma error bars in our graphs.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: We provide an estimate of the resources in the appendix.

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The paper adheres to the NeurIPS Code of Ethics.

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA] .

Justification: The answer NA means that there is no societal impact of the work performed.

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA] .

Justification: The answer NA means that the paper poses no such risks.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes] .

Justification: The PaLM model is cited. The MetricX and Comet metrics models are cited. The WMT22 dataset is cited.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA] .

Justification: The answer NA means that the paper does not release new assets.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [Yes] .

Justification: We used humans to collect MQM annotations as described in the original paper.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA] .

Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMinimum Bayes risk (MBR) decoding is a widely used technique for machine translation (MT) that involves generating $N$ candidate translations that are then scored according to a utility function, typically an automatic reference-based MT metric. In practice, this is done by comparing each candidate in the set to all the other candidates (that are used as pseudo-references), which is computationally very expensive, with a time complexity of O(N^2). This paper focuses on improving the efficiency of MBR decoding by reducing the number of metric computations. Based on the assumption that the MBR utility matrix is low-rank, they score only a subset of the candidate-pseudo-reference pairs and use the ALS algorithm to estimate the missing values.  Their experiments show that their method does not compromise translation quality.\n\nSTRENGTHS:\n- This paper offers a simple solution to a problem that remains unsolved so far (making MBR faster/cheaper) without compromising the final translation quality.\n- I believe that framing this as matrix completion problem is a novel approach.\n- The experimental results are good and confirmed with human evaluation.\n\nWEAKNESSES:\n- The experiments are only performed on two language pairs (EN-DE and EN-RU). Can we expect the same findings for other languages, including lower resource ones? The WMT test sets include other languages, so it shouldn’t be hard to validate this. Also, only PaLM8B is used. It would be good to try other models such as Tower (Alves et al., 2024) with a varying number of samples.\n- There is a lot of discussion about related work on making MBR decoding more efficient in Section 2 but the comparison to some of these efficient MBR approaches is missing in the experimental part. Also, it would be good to compare with linear-time cost approaches such as QE-reranking (Fernandes et al., 2022) since this is a clear baseline.\n- See my questions below for other things.\n\nTower: An Open Multilingual Large Language Model for Translation-Related Tasks (Alves et al., 2024)\n\nQuality-Aware Decoding for Neural Machine Translation (Fernandes et al., 2022)\n\nQUESTIONS:\nComments and questions:\n- Although the definition of MBR matrix is presented in Section 4, it would be beneficial to briefly explain this a bit better earlier in the paper (e.g., in the paragraph in L44-54). It may not be straightforward to all readers to understand where this matrix comes from and why it can be assumed to be low-rank. \n- Where is Figure A.6 and Table 4.2, mentioned in L155? This is pointing to the sections… After a closer look, Table 1 and Figure 5 seem to be the ones that you are referring to.\n- What happens if you fill in the missing values in the MBR matrix with random numbers instead of using ALS? I believe this is an important baseline.\n- The results in the tables are averaged over 1000 trials. Can you please provide standard deviations too?\n\nMinor comments:\n- Fix citation style in L84 (no parenthesis) and L86 (missing whitespace).\n- Missing punctuation in Eq. 2.\n- Missing whitespace in L202.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-09"}

comments: SUMMARY:
- this paper uses alternating least square method for matrix completion when only a small part of the utility matrix in MBR decoding is calculated to decrease the computational cost for full MBR decoding and at the same time guarantee the translation quality 
- the authors first justify the use of their method by showing that the utility matrix in MBR decoding is indeed low rank
- by conducting experiments on WMT dataset and four translation directions, they show that PMBR achieved the best COMET22 score and MetricX on most of the settings

STRENGTHS:
- the method they propose is a simple yet effective one to shrink the computational cost for MBR decoding in NMT task 
- Across all the experimental settings the metric score is indeed better than other baseline methods while keeping the cost lower

WEAKNESSES:
- ALS is one method for matrix completion, ablations should be done using other matrix completion methods to see if the cost can be further reduced by using less examples 
- using matrix completion method to lower the cost for computation intensive tasks itself is not a novel approach 
- PMBR is compared against other MBR decoding related methods but not compared against e.g. naive beam search

QUESTIONS:
- there are other matrix completion methods that also focus on picking the most representative matrix entries for completion. Why did you not compare against those methods?

LIMITATIONS:
- yes the authors addressed the limitations
- and there is no potential societal impact
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes an approximate method for minimum Bayes risk (MBR) decoding by employing matrix completion. Basic idea is to fill-in only a fraction of the score matrix to compute MBR scores, and to leverage alternating least squares (ALS) algorithm to estimate the empty slots assuming that the utility score matrix is low rank. This work empirically show that the utility score matrix is actually low rank for MetricX and chrF under some language directions. Empirical results also show that the proposed approximate MBR decoding is a better approximation method when compared with other approximation method, e.g., shrinking the dimension of pseudo-reference list size and sampling-based method.\n\nSTRENGTHS:\n- The proposed method is interesting in that it leverages the low-rank matrix completion to estimate the full utility score matrix for MBR decoding assuming that utility score matrices are low rank. The singular values of MT metrics empirically show that the low-rank property holds for some language directions. I think the finding somewhat agrees withe the matrix completion for human assessment [1].\n- Empirical results show that the proposed method is better than other approximation methods even with only a small number of samples, and achieving comparable performance to the non-approximation method even the budget for 1/4. Human judgement also shows that the proposed method is closer to the non-approximation method.\n\n[1] https://aclanthology.org/P13-2025/\n\nWEAKNESSES:\n- This work needs further analysis on the utility score matrices after convergence since there exist some gaps when compared with the non-approximation method under the budget of 1/32. In particular, the true scores filled by the metric might be changed after completion, and I'm curious how that will impact MBR decoding. Also, it would be good to analyze the deltas of the completed scores to see how that will impact the end accuracies. \n- In addition to random sapling to determine the position to fill-in by the true score, I'd like to see a more controlled setting in which the positions are skewed, in which some columns or rows are completely empty and thus complemented by ALS, and how that will impact the estimated scores and the final MBR decoding.\n\nQUESTIONS:\nQuestions:\n- It is not clear how many iterations are needed for the convergence in general. Also, it would be good to show the average differences and variances of the score matrices by the approximated methods, i.e., prediction performance for the missing values, to quantify the impact of the approximation.\n\nSuggestions:\n- I think Figure 1 is not mentioned anywhere in the paper, and please describe what is indicated by the plots. Also, do not use a bitmap format but use a vector format for clarity. \n- line 155: Figure A.6 and Table 4.2 (?)  -> please check the references.\n- Better to run statistical significance tests on the numbers in Table 2 and 3 to make sure the differences are statistically significant or not.\n\nLIMITATIONS:\nThis paper has discussion on the limitations, e.g., measuring only MetricX, but this work is actually running experiments for COMET as a utility function.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes yet another method for speeding up the utility computation of MBR decoding. The algorithm exploits the structure of the problem that the utility matrix tends to be lower rank as the utility is kind of a similarity metric. The experiments are compared against the standard MBR and show that it achieves a good speedup over the standard MBR.\n\nSTRENGTHS:\n- The proposed algorithm is interesting. Prior works do not consider the structure of the utility matrix and rather seek to optimize the computation via hyperparameter tuning or to optimize the worst-case complexity.\n- The observation that the utility matrix is low rank is interesting and is a contribution to the community.\n- Human evaluation is nice to have.\n\nWEAKNESSES:\nThe paper brings an interesting new idea to MBR decoding. However, their contribution is claimed ambiguously. \n\n- The experiment does not compare any of the recent algorithms on efficient MBR decoding: Confidence-based pruning (Cheng and Vlachos 2023), Adaptive MBR (Jinnai and Ariu 2024), Linear time MBR (Vamvas and Sennrich 2024), and Centroid-based MBR (Deguchi et al. 2024). Thus, the paper does not show evidence that their method is state-of-the-art with respect to the empirical performance, which will not be a reason for rejection.\n- Empirical evaluation of concurrent work is too much to ask, but I would say that they should be explained fairly. Vamvas and Sennrich (2024) are mentioned that “the utility metric needs to fulfill certain conditions to be applicable.” (Line 77). This is true but I would say that “certain conditions” should be explained clearly in this paper, because the proposed method is only effective when these “certain conditions” are not satisfied, if I understand it correctly. Also note that the proposed method also exploits the structure of the utility metric that results in a low-rank matrix, which may not be universally true. The proposed method also requires that certain conditions are satisfied. I believe this will not be a reason to reject, only if this limitation is clarified in the paper.\n\nQUESTIONS:\n- Figure 5 shows that the utility matrices are likely to be rank-1. My understanding is that in theory it requires a single comparison per each candidate is enough to estimate the quality of the candidate, assuming that we can ignore the noise. I was thinking if this observation would rather motivate the Linear time MBR (Vamvas and Sennrich 2024) as they use a single comparison against the mean of the references, which we expect to have the least noise.\n- Although the rank of the matrix is estimated to be 1, the ranks of the matrices are set to 5 to 15 (Line 224) in the experiment. Why is it better to set it larger than 1? I’m assuming r in Line 224 refers to the rank of the matrix. r is used as the rank and also the reduction ratio in Algorithm 1.\n- What is the definition of O(millions) and O(hundreds)?  (Line 171)\n- Does the method require that candidates and references are the same set of samples? I see several papers in MBR using different sets of samples (Cheng and Vlachos 2023; Ohashi+2024) so it would be better if it works with both cases. I don’t see it as a reason to reject it, but I think it needs to be clearly mentioned that the proposed method can’t use a different set of samples, if so.\n- It would be nice to have a comparison of the overall walltime, including the time for sample generation to show in the end how much it improves the speed of the NMT system.\n\nTypos\n- Line 51: effectiveness of out method → effectiveness of our method\n- Line 91: a NMT model → an NMT model\n- Line 155: Figure A.6 and Table 4.2 → I guess Figure 5 and Table 1\n- Line 196: SPM tokens → no definition of SPM\n- Line 197: Note:\n- Line 414 Algorithm 2: There are N and n that I believe they both refer to the same parameter.\n- Line 431: The sentence is incomplete.\n- Line 513, 522: Doesn’t MQM involve human subjects?\n\nLIMITATIONS:\nI think the paper does not explicitly explain the critical limitations of the method.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}