Unveiling the Power of Source: Source-based Minimum Bayes Risk Decoding for Neural Machine Translation
Authors: Boxuan Lyu, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura


Reviewer paper summaries: The paper describes a new approach for decoding (sMBR) that bridges two well studied methods: MBR and QE reranking

The novelty of the method comes from the fact that instead of scoring each candidate against each other using a reference-based metric, they propose to score each candidate against a set of alternative source sentences using a reference-free metric. The motivation for their method is mostly because of the inference cost of MBR.
Summary of strengths: The method is novel and shows promising results. The paper is also well-written and easy to follow.
Summary of weaknesses: The main weakness of the paper is that it uses a setup that is a bit outdated. The authors experimented with WMT19 Fairseq NMT model which is already 5 year old translation model. The field has been moving tower LLM based translations and there are already plenty of open LLMs available for translation (e.g ALMA and Tower). These models, because they are multilingual, they also allow for a more comprehensive study of their method across different languages (from high resource ones to low resource ones). I have a similar comment to the paraphrasing model: is FLAN-T5-large the best paraphraser out there? what if we use a stronger model? could we increase further the number of source paraphrases and get higher gains?
Comments, suggestions and typos: Since the MBR and sMBR use COMET22 and CometKiwi22 I suggest using XCOMET-XXL and MetricX-XXL as evaluation metrics. These two metrics might be "too expensive" for MBR and sMBR yet, since they are trained on MQM data, rather than Direct Assessments, they should correlate less with COMET22 and CometKiwi22. 

BLEURT20, COMET22, CometKiwi22 are all trained on very similar data and thus they might express the same biases. Adding XCOMET and MetricX should mitigate these biases and the results would be more sound. Another alternative would be human evaluation.

For sMBR-BT what happens if instead of Beam-search we use another sampling technique? for example top-k (k=10) or epsilon-sampling?  I am afraid that the high lexical overlap between the different sources comes from using Beam-search and maybe sMBR-BT can be competitive to sMBR-PP when used with a decoding method that creates more diverse sentences.

There has been a lot of work in improving MBR efficiency. Example of such works are: 
Faster Minimum Bayes Risk Decoding with Confidence-based Pruning (EMNLP 2023 Best Paper Award)
Centroid-Based Efficient Minimum Bayes Risk Decoding

This line of work should be mentioned in the paper and maybe even compared to.
Overall assessment scores: 3