MT-Ranker: Reference-free machine translation evaluation by inter-system ranking
Authors: Ibraheem Muhammad Moosa
Rui Zhang
Wenpeng Yin
Venue: ICLR 2024 spotlight (2024)
Keywords: Machine Translation Evaluation
Primary area: general machine learning (i.e., none of the above)
TL;DR: We apply pairwise ranking approach to machine translation evaluation in reference-free scenario.

Abstract: Traditionally, Machine Translation (MT) Evaluation has been treated as a regression problem -- producing an absolute translation-quality score. This approach has two limitations: i) the scores lack interpretability, and human annotators struggle with giving consistent scores; ii) most scoring methods are based on (reference, translation) pairs, limiting their applicability in real-world scenarios where references are absent. In practice, we often care about whether a new MT system is better or worse than some competitors. In addition, reference-free MT evaluation is increasingly practical and necessary. Unfortunately, these two practical considerations have yet to be jointly explored. In this work, we formulate the reference-free MT evaluation into a pairwise ranking problem. Given the source sentence and a pair of translations, our system predicts which translation is better. In addition to proposing this new formulation, we further show that this new paradigm can demonstrate superior correlation with human judgments by merely using indirect supervision from natural language inference and weak supervision from our synthetic data. In the context of reference-free evaluation, MT-Ranker, trained without any human annotations, achieves state-of-the-art results on the WMT Shared Metrics Task benchmarks DARR20, MQM20, and MQM21. On a more challenging benchmark, ACES, which contains fine-grained evaluation criteria such as addition, omission, and mistranslation errors, MT-Ranker marks state-of-the-art against reference-free as well as reference-based baselines.

Published as a conference paper at ICLR 2024
MT MA-RANKER: REFERENCE-FREE MACHINE TRANSLA- TION EVALUATION BY INTER-SYSTEM RANKING
Ibraheem Muhammad Moosa, Rui Zhang & Wenpeng Yin Department of Computer Science and Engineering Pennsylvania State University (ibraheem, moosa, rmz5227, wenpeng Saapnngggpsuueeee
ABSTRACT
sion Traditionally, Machine Translation (MT) Evaluation has been treated as regres- an absolute tramslation-quality score. This approach has two limitations: i) the scores lack interpretabilityy and human annotators struggle with giving consistent scores; ii) most scoring methods are based on (reference, translation) pairs, limiting their applicability in real-world scenarios where ref- erences are absent. In practice, we often care about whether a new MT system is better or worse than some competitors. In addition, reference-free MT evalu- ation is increasingly practical and necessary. Unfortunately, these two practical considerations have yet be jointly explored. In this work, we formulate the reference- free MT evaluation into pairwise ranking problem. Given the source sentence and pair of translations, our system predicts which translation bet- ter. In addition to proposing this new formulation, we further show that this new paradigm can demonstrate superior correlation with human judgments by merely using indirect supervision from natural language inference and weak supervision from our synthetic data. In the context of reference-free evaluation, MT- Ranker, trained without any human annotations, achieves bate-of-the-as art results on the WMT Shared Metrics Task benchmarks DARR20, MQM20, and MQM21. On a challenging benchmark, ACES, which contains fine-grained evaluation cri- teria such as addition, omission, and mistranslation errors, MT MT-Ranke marks wate-of-thee-ne against reference-free as well as reference-based baselines []]
INTRODUCTION
Automatic MT evaluation is crucial to measure the progress of MT systems. Compared to human evaluation, automatic evaluation is much cheaper and less subjective. Thus, progress in MT has been synonymous with achieving a higher BLEU score (Papineni et al. 2002), the most popular automatic MT evaluation metric. BLEU measures the similarity of machine translation with reference translation using n-gram precision. Even though it is still the most widely used metric for MT' evaluation, it has low correlation with human judgment (Freitag et al., 2022). This has prompted researchers to design better automatic evaluation metrics for MT (Bojar et al. 2016 2017 Ma al. 2018 2019 Mathur et 2020 Freitag et 2021b 2022).
In recent years, the design of automatic MT evaluation metrics has been dominated by the use of large language models (LLMs) trained on synthetic and aunan-annotated data (Rei al., 2020b] 2021 Zhang et 2020). These metrics can be broadly categorized into reference- bbaeneeeebaedddde approaches that compare the machine translation with reference translations and reference-free approaches that score the machine translation with only the source sentence. The latter approach is arguably more useful in practice since reference translations may be unavailable real-world scenarios. Inter- estingly, reference-free evaluation has recently become competitive with reference-based evalua-
tion (Rei et al. 2021).
Irrespective of whether MT evaluation system uses reference translation, these systems usually produce a quality score for the machine translation. An alternative approach to machine translation

---

Published as a conference paper at ICLR 2024
is pairwise ranking, where instead of predicting the quality score for single translation, pair of translations are compared, and a better-worse judgment is given. This approach was initially proposed by (Ye et al. 2007) and later used by (Duh 2008 Guzmán al. [014 2015; Mouratidis & Kermanidis 2019). The pairwise ranking approach is sufficient for the most important use case of automatic evaluation metrics: comparing machine translation systems.
However, the pairwise ranking approach remains underexplored. Specifically, has only been ap- plied in the reference-based evaluation scenario. In this work, we focus on the reference-free sce- nario. Given the source sentence and a pair of translations, our system will predict which translation better. This formulation has three benefits compared to the existing MT evaluation approaches. (i) Simplification of the target task: The pairwise ranking task is more straightforward than a regression- based task. (ii) Reference-free evaluation: In practice, references are often unavailable. Additionally, the reference-basee evaluation introduces a reference bias (Freitag et al., 2020). A reference-free free evaluation system bypasses these issues since it measures the translation quality di- rectly with the source sentence. (īii) Less reliance on high quality manual annotations: Collect- ing consistent quality scores for machine translations is difficult. For reproducible results, requires collecting 15 direct assessment annotations per translation (Graham et al., 2015). In contrast, rela- tive ranking annotation from direct assessment with a large enough threshold has been used with as few one annotation (Ma al., 2018). Generating synthetic data for the pairwise ranking problem is also much easier. Using synthetic data for pretraining is very effective for training evaluation metric systems (Sellam et al. 2020, Wan et al. 202T). For regression- based approaches, this re- quires generating a synthetic quality score. In contrast, the pairwise ranking approach only requires a better-worse judgment, which can naturally arise from the synthetic data generation technique.
These benefits motivate us to explore the reference-free pairwise ranking approach to machine trans- lation evaluation. We train our system with supervision only from multilingual natural language in- ference and some synthetic data. Experiments on DARR20 (Mathur et al. 2020p. MQM20 (Freitag ct al. 2021a), MQM21 (Freitag et al., 2021b), MQM22 (Freitag et al. al.] 2022), and ACES (Amrhein et al. 2022) demonstrate the state-of-the thete-r performance of our system without touching any task-
specific human supervision. Our contributions can be summarized as follows:
We are the first to model reference-free MT evaluation as pairwise ranking problem.
MT-Ranker demonstrates state-of-the-art correlation with human judgments across di- verse benchmarks without relying on
data.
By eliminating the dependency on human-provided reference translations and comparison data, our system exhibits enhanced practical utility.
2 RELATED WORK
Regresssoo--asel MT evaluation. Since the introduction of BLEU (Papineni et al. 2002). it has been the dominant machine translation evaluation metric. Despite its popularity, BLEU has signifi- cant shortcomings, such as low correlation with human judgment (Callison-Burc) Burch et al. 2006) and reliance on local lexical features. Researchers have developed evaluation metrics that correlate more with human judgment to address these issues. BERTScore (Zhang al. 2020) used contextual em- bedding from BERT (Devlin et al., 2019) to produce a similarity score between the reference and candidate translation, addressing two issues of BLEU: it works even if the candidate translation is a semantically equivalent paraphrase of the reference and considers long-distance dependency. In contrast to BLEU and BERTScore, which are unsupervised, other reference-basee evaluation met- rics such as RUSE (Shimanaka et al., 201 8), BLEURT (Sellam et al.) 2020). COMET (Rei et al., 2020a b 202T) were trained on human annotations of translation quality scores. BLEURT intro- duced the use of large-scale synthetic data. The COMET family of models used a dual-encoder architecture that separately embeds source, reference, and candidate translation using transformer encoder models, generating features from combining the embeddings. Trained on human annota- tions, these metrics achieved higher correlations with human judgments. Interestingly, the reference- free COMET-QE (Rei et al.) 2021) model achieved comparable performance with reference-basee models. UniTE (Wan et al. (022) combined reference-free
sxterence-basedd and source- referecccee combined evaluation into a single model by training the three settings simultaneously using multitask learning. PRISM (Thompson & Post 2020) and BARTScore (Yuan et al.) (02T) formu- lated evaluation as text generation and use the perplexity of the candidate translation the quality
2

---

Published as a conference paper at ICLR 2024
Traditional Reference-Based Systems
Traditional Reffreneereee
Systems
Reference
Evaluation System
Quality Score
Source
Evaluation System
Quality Score
Translation
Translation
Our System
Binary Decision
Source
Evaluation System
Translation Pair
Figure 1: Our system receives a pair translations and makes a binary decision on which translation has better quality. In contrast, traditional reference-free evaluation systems generate quality score for a single translation. The main difference between our approach and previous approaches is highlighted in red.
score. These were unsupervised models and performed better than previous supervised models.
T5Score (Qin al. 2022 extended upon BARTScore by discriminative fine-tuning on human
notations. Some recent works (Fernandes et al. 2023] Koemi & Federmann, [023 Rei et al. al.] (023)
have explored using GPT-4 (OpenAl (022) and PALM-2 (Anil et al. 2023). for translation evalua-
tion, concluding that these models still lag behind on segment-level translation evaluation.
Pairwise Evaluation. Reference-basee pairwise evaluation of machine translations was first pro- posed by (Ye et al.) 2007). They argued that the ranking annotations have higher inter-annotator agreement than translation quality score annotations. They designed independent features for the reference and each translation. (Duh, 2008) compared ranking with the scoring approach by con- sidering new features constructed from the translation pair under evaluation. (Guzmán et al. 2014) extended upon (Duh) 2008) by considering tree-based features constructed from both the reference and the translation pairs simultaneously. (Guzmán et al., 2015) extended upon (Guzmán et al, 2014) using word embeddings train neural network models the pairwise evaluation task. Task- specific pairwise ranking annotations was leveraged for training evaluation metrics (Song & Cohn 2011 Zhang & Genabith, 2020). While the models were trained the comparison data, they ultimately worked in the regression scheme, predicting the quality score for a single translation.
In contrast to all previous approaches, explore reference-free pairwise evaluation and achieve state-of-the-art correlation with human judgments without using supervised annotation.
3 METHODOLOGY
In this section, we first discuss the architecture of MT-Ranker and the input formulation (Sec- tion 3.I). Then we discuss training with indirect supervision from cross-lingual NLI and weak supervision from synthetically generated data (Section 3.2
3.1 MODEL AND INPUT FORMULATION
We formulate the input to our system single text. Given the source sentence S and two transla- tions, To and Ti, we formulate the input to the model as follows.
Source: Translation 0: To Translation
We show example input to system in Figure Here, two English translations of French sentence are being compared. The first translation mistakenly translates the French word tapis to the English word bed, while the correct translation should be carpet. Thus, given this input would like our system to predict that the second translation better.
We use the encoder of multilingual T5 (Raffel et al., 2020) the backbone of our models. The encoder model attends to all the tokens of the source and the two translations simultancously. We put mean pooling and logistic regression layer on top of the encoder model.

---

Published as a conference paper at ICLR 2024
Binary Decision
NT-Rankez
Logistic Regression Layer
Pooling Embedding Layer
Pretrained Bidirectional LLM
Source: Le chat est sur la tapis. Translation The cat is on the bed. Translation 1: The cat is on the carpet.
Figure 2:
Illustration the input format and the architecture of MT-MM--RTE Ranker. The source sentence and the translation pairs are formatted single text input to the system. The bidirectional LLM attends over all the tokens of the source and the translations simultaneously.
3.2 MODEL TRAINING
We train WT-Ranker on synthetically generated translation pairs where one the translations can be considered better than the other. Each training sample can be formally represented follows:
(S, (To: Tı), y)
where S is the source sentence, (To, Ti) is the translation pair and
0 Ty is better than Ti
(3)
otherwise
The training performed in three stages: pretraining with indirect supervision from cross-lingual NLI, fine-tuning on the task of discriminating between human translation and machine translation, and further fine-tuning on weakly supervised synthetic data.
Stage I: Pretraining with indirect supervision from cross-lingual NLL. We first pretrain model on the XNLI (Conneau et al., 2018) dataset. XNLI is a NLI dataset with examples of "En- tailed", "Noonentailled, and "Neutral" hypotheses for each premise in fourteen languages. We take the premise in one language and a pair of entailed and non-entailed hypotheses in another language. We consider the entailed hypothesis be beller Iranslation of the premise than the non-entailed one. Although the hypothesis in this dataset does not correspond to actual translations of the premise, this stage serves as an indirect supervision for our model to prefer translations that do not contradict the source sentence. The training samples for this stage can be formally represented as follows:
(S, (To, T1).
y SO if is entailed by S
otherwise
Stage I:: Discriminating hetween human translation and machine translation. At this stage, we construct training pairs based on the assumption that a human-written reference translation is generally better than machine translations. The reference translation R is paired with machine translation, and the task is to predict which one is the reference translation. The training samples for this stage be formally represented follows:
(S, (To. T1).y)
either To = R or Ti = R
(6)
y= 0 if =
otherwise
We fine-tune our model from Stage on this task and collect examples of source, reference and machine translations from the Direct Assessment datasets published from 2017 2020 (Bojar al. 2017 Ma et al.) 2018) 2019) Mathur et al.) 2020). These datasets contain machine translations generated by different machine translation systems for given source sentence.

---

Published as a conference paper at ICLR 2024
Stage III: Weakly Supervised Training. One issue with the training pairs used in Stage II is that one of the pairs the reference translation, which can be of much higher quality than the machine translation. The model never sees translation pairs covering the full spectrum of translation quality. To mitigate this issue, we further fine-tune our model on synthetic data generated by two approaches.
Synthetic data from Direct Assessment datasets. We take samples of machine translation pairs from the Direct Assessment datasets and generate labels for the translation pairs using an unsuper- vised machine translation evaluation metric M. The metric M generates quality score for each of the translations. which can be used to provide a better-worse judgment for the translation pair. The translation pairs and the target for this stage can be formally represented follows:
(S. (Tu: T)).
(8)
So (M, > M(S,R,T) otherwise
Even though the evaluation metric M has access to the reference Rt our model does not have access to the It during training or testing. Further, we use an unsupervised metric, BERTSCORE, this work, so that the model does not have indirect access to human supervision.
Synthetic data from machine translation datasets. Given a translation, examples of worse translations be generated by perturbing the translation using simple heuristics (Sellam et al, 2020). In general, the perturbed translation has worse quality than the reference translation. Thus, the reference translation and the perturbed translation form training pair for which can naturally provide better- better-worse judgment label. This provides a very simple way to generate a lot of synthetic data for the pairwise evaluation approach. We can formally represent these samples follows. Given a source sentence S, the corresponding translation T,, and perturbation function P, we can generate the following samples
(8, ((T)),
(10)
(S, (P(T),
(11)
We generate these types of synthetic data from a subset of the MT-PRISM (Thompoon & Post, 2020) dataset corresponding to the language pairs represented the Direct Assessment datasets. MT-PRISM is machine translation dataset collected from diverse sources across 39 languages. We apply the following perturbations to the reference translations to generate examples of worse translations.
Word Drop: We randomly drop 15% of words from the translation. This approach generates examples of worse translations that are not fluent.
Word Replacement using Masked Language Model: We randomly replaced 15% of words in the reference translation using the pretrained XLMRoberta (Conneau et al. (0200) model. The worse translation generated with this approach may be fluent but have a different mean- ing.
Backtranslation: We translate the sentence French and back to the original language using the M2M100 (Fan et al. 2021) translation model. The worse translations generated by this approach may preserve the same meaning.
Word Replacement after Backtranslation: We randomly replaced 15% of words from the backtranslated sentence. With this approach, we get examples of worse translations that may not preserve the meaning as well being worded differently than the original trans- lation.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets. Our benchmark datasets are the WMT20 Shared Metries Task dataset (DA20) (Mathur et al.] 2020). the MQM20 (Freitag et al. (021a), the MQM21 (Freitag et al. (021b), the MQM22 (Freitag al. 2022) and the ACES (Amrhein al. 2022 datasets.
5

---

Published as a conference paper at ICLR 2024
DA20 (Mathur et al., 2020). This dataset contains machine translations from 208 MT systems covering 18 language pairs. Each machine translation is annotated with a Direct Assessment quality score by human annotators. Better-worse judgments of translation pairs of the same source constructed from these DA scores when the scores differ by at least 25 points. The dataset has about 250k such examples of botter-worse translation pairs. The large size and the coverage of many language pairs make this dataset highly suitable for evaluating MT evaluation systems.
MQM20-22 (Freitag et al. 2021a by 2022). The MQM datasets are constructed by using the Multidimensional Quality Metric (MQM) (Burchardi 2013) approach by professional translators. These datasets only cover a few language pairs, but the quality score annotationsare of high quality. The MQM20 dataset was constructed by re-annotating a subset of the DA20 dataset. The MQM21 and MQM22 datasets were constructed for the WMT21 and WMT22 Shared Metrics tasks. The MQM20, MQM21, and MQM22 datasets contain 95k, 74k, and 215k examples of better-worse translation pairs, respectively. Even though these datasets cover few language pairs, due to the higher quality of annotations, these datasets are important for benchmarking MT evaluation systems.
ACES (Amrhein et al., 2022). The ACES dataset (Amrhein et al., 2022) challenge dataset covering 68 phenomena of translation errors in 10 broad categories: Addition (A), Omission (0), Mistranslation (M), Untranslated (U), Do Not Translate (DNT), Overtranslation (Ov), Undertrans- lation (Un). Real World Knowledge (RWK), Wrong Language (WK) and Punctuation (P). The dataset contains about 36k samples covering 146 language pairs. We use this dataset for fine-grained evaluation of our systems different challenge scenarios.
Evaluation Metric. Since WMT14 (Macháček Bojar 2014), the Kendall-like Tau correlation with human judgments has been used to evaluate machine translation evaluation systems. We use this metric to evaluate our system. The formal definition of the metric is given in Appendix A.I
Baseline Systems. We compare our system against reference-free evaluation metrics. For eval- uation on the WMT20 Shared Metrics Task dataset, our baselines are the two berf-peeforming reference-free free systems submitted to WMT20 Metrics Shared Task: COMET-QU (Rei et al. 2020b) and OPENKKWI-XILMO
(Kepler et al. 2019) and the reference-free version of T5-SCORF (Qin et al. 2022) which is a recent state-of-the-ar the- system. For the MQM datasets, our baselines are UNITE (Wan et al. 2022), COMET-QE and COMETKIWI (Rei et al. 2022). COMET-QU and COMETKIWI are the state-of-the-ee reference-free evaluation metrics the MQM21 and MQM22 benchmarks, respectively. We additionally consider KG-BERTScore (Wu et al. 2023 as a baseline
for the ACES benchmark since it is the state-of-art on that benchmark.
Implementation Details. We consider three variants of the multilingual model with increasing parameter count: Base (290M), Large (600M), and XXL (5.5B). We use the implementation of these models available the Huggingface library (Wolf et al. 2019). Additional details, including hy- perparameters, development set construction for fine-tuning hyperparameters, and training hardware setup, are given in Appendix A.3
4.2 RESULTS
4.2.1 RESULTS ON WMT20 SHARED METRICS TASK DATASET
We evaluate our systems on seven X-to-Engiie Eto-EEnlissh language pairs and seven English-to-X language pairs from the WMT20 Shared Metrics Task dataset. In addition to the baselines mentioned in section we also show results for SENTBLEU (Post. 2018) and BERTSCORF (Zhang et al. 2020). which are popular reference-based unsupervised machine translation evaluation metrics.
Table D shows the segment level Kendall's Tau correlation for all the language pairs. Results for the best-performing model are shown in bold. All our models outperform the baselines. On the X- to to-English language pairs, our perforenn model, SrTRannee-XXL. outperforms the nearest supervised baseline BIENKKII--LLMMM by 4.4 points average. On the English-to- language pairs, our best-performing model, MT-Ranker XXLL, outperforms the nearest supervised baseline T5SCORE- XL by 3.8 points average.

---

Published as a conference paper at ICLR 2024
Table Segment-level Kendall's Tau correlations on language pairs from the WMT20 Shared Met- rics Task dataset. Avg. denotes the average score across all language pairs.
cS-en
de-en
ja-en
pl-en
ru-en
ta-en
zh-en
Avg
UNSUPERVISED BASELINES
SENTBLEU
6.8
41.1
18.8
-2.5
-0.5
16.3
9.3
12.8
BERTSCORE
11.7
45.2
24.3
4.7
6.0
21.9
13,4
18.2
TSSCORE- X
3.9
26.6
19.7
2,4
6.6
16.1
5.1
11.5
X-to-En
SUPERVISED BASELINES
COMET- QE
9.2
40.8
15.3
4.6
10.1
16.7
9.2
15.1
OPENKIWI-XLME
46.3
5.9
9.2
18.8
17.6
TSSCORE-XLL XLsup
7.1
41.7
22.1
3.0
7.3
24.7
7.4
16.2
OUR SYSTEMS
MT TTT-RaneerBBase
10.7
46.8
23.2
6.8
16.9
20.9
14.8
20.0
MT-Ranker-Large
11.8
48.6
26.2
8.8
16.4
24.2
16.3
21.8
MT-Ranker-oo XXL
13.1
48.5
25.6
9.2
16.3
24.9
16.2
22.0
en-Os
en-de
en-ja
en-pl
en-ru
en-la
en-zh
Avg
UNSUPERVISED BASELINES
SENTBLEU
43.2
30.2
47.9
5.1
39.5
39.7
31.6
BERTSCORE
51.1
39.5
53.8
28.5
20.5
60.4
41.1
42.1
TSSCORE-XL XLLN
19.0
20.7
47.7
11.1
12.5
39.9
18.2
24.2
EN-to-X
SUPERVISED BASELINES
COMET-QE
34.6
51.2
39.8
42.3
OPENKIWIXXLMR 60.7
36.9
55.3
34.7
27.9
60.4
37.7
44.8
T5SCORE-XLE XLsur
62.7
40.0
58.7
37.6
28.2
66.2
45.6
48.4
OUR SYSTEMS
UT--Rnker-T Base
38.0
56.9
23.3
66.6
36.7
45.7
UT--Rnker-Lang
69.5
46.6
60.6
42.9
27.7
69.7
41.1
51.1
MT MT-Ranker-XXL
69.1
47.5
63.2
44.1
30.0
70.4
41.0
52.2
4.2.2 RESULTS ON MQM
We report the Kendall's Tau of baselines and our systems the MQM benchmark datasets. How- ever, the implementation differs from the one used in (Freitag et al. 2021b 2022). (Freitag et al.) 2021b) flattened the segment-system score matrix before calculating the correlation. Since sys- tem does not predict a quality score for translation, this approach to Kendall's 's Tau is unsuitable. Thus, keep Kendall's 's tau implementation the same the used for evaluation the DA20 dataset. We note that this implementation is the same as the segment averaging approach discussed in (Freitag al. 2022).
We report the MQM20, MQM21, and MQM22 results in Table We only show results on MQM22 for COMETKIWI since it uses MQM20 and MQM21 training data. Our hest-performing model, MI ErT-Ranker-XXi outperforms the nearest baselines on MQM20, MQM21, and MQM22 by 1.3, 0.4 and 2.8 points respectively.
4.2.3 RESULTS ON ACES
In T we show the performance of our system on the ten categories of the ACES dataset. The final column on the table shows the ACES-Score, a weighted combination performance the ten categories. Our system achieves state-of-the-a art results in terms of ACES- against the previous state-of-the-ar system, KG- -G-BEETSccore (Wu et al. 2023). Besides the ayong-languee category of errors that unsolvable for reference-free systems (Amrhein al., 2022), system performs well in all categories. We see especially strong performances by our system in detecting omission, mistranslation, do-not-translate, and punctuation emror..

---

Published as a conference paper at ICLR 2024
Table 2: Segment-level Kendall's s Tau correlation on the MQM datasets. Avg denotes the average correlation achieved by a system across all language pairs on each dataset.
MQM20
MQM21
MQM22
en-de
Avg
en-de
zh-en
Avg
en-de
zh-en
en-ru
Avg
UNITE COMET-QE
11.1
12.2
11.6
11.4
7.3
9.3
16.3
22.4
24.8
21.2
18.6
15.2
12.0
21.6
27.2
24.7
COMETKIWI
20.8
25.3
31.7
25.9
MT OT-Ranker-Base
19.4
13.7
13.1
10.0
15.4
18.5
17.4
MT Ra-Ranker-Large
11.6
17.9
14.8
11.0
10.6
10.8
19.8
21.9
27.2
23.0
MT Mr-Ranker-XXI
25.5
19.1
22.3
14.7
10.1
12.4
24.9
26.6
31.5
27.7
Table 3: Kendall's Tau correlation on the ACES dataset.
A
O
M
DNT
Ov
Un
RWK
WL
P
ACES-
Score
UNITE
0.29
0.93
0.60
-0.62
0.86
0.70
0.54
0.54
-0.42
0.73
15.70
COMET-QR QE
-0.54
0.40
0.38
0.14
0.12
0.62
0.44
0.32
-0.51
0.25
6.61
COMETKIWI
0.36
0.83
0.63
0.23
0.78
0.74
0.57
0.58
-0.36
0.49
16.95
KG-BERTSCORE
0.79
0,49
-0.46
0.76
0.65
0.53
0.49
0.31
0.26
17.49
WT-Ranker-XXe
0.65
0.97
0.63
0.25
0.84
0.63
0.54
0.66
-0.53
0.97
18.46
5 ANALYSIS
We analyze our systems to answer four questions: (Q1) How much does each stage of training con- tribute to the final system? (Q2) How much fur-
40
Average Kendall's Tau
ther can the performance improved given access to human annotated training data? (Qx) Does system generalize to unseen language pairs? (Q1)
35-
there any specific scenario where systems per- form poorly?
30.
Ablation Study, To answer 21, study whether the three training stages are necessary for our model's good performance. In Figure 3 we show model
25
Stages
w/o
w/o
w/o
the performance of our MT-Ranker-lan on the DA20 dataset after removing different stages from the training process. We plot the average Kendall's Tau across all language pairs. see that performance drops after removing each of the training stages, indicating the necessity of each training stage. The highest performance drop hap- pens from removing stage 3, showing the effectiveness of our synthetic data generation approach.
Figure 3: Impact of removing training stages on the performance of MT MT-Ranker-Large on the DA20 dataset.
Using Human Supervision. To answer 22, we study whether the performance of our system can be further improved by fine-tuning on human annotations. We construct better-word woote translation pairs following the Direct Assessment Relative Ranking approach (Ma et al. 2019) from the Di- rect Assessment datasets from WMT17, WMT18 and WMT19 Shared Metrics Tasks. We show the results in Figure Across all benchmark datasets, the supervised systems show improved perfor- mance over the unsupervised systems.
Zero-shot performance on unseen language pairs. Except for the XNLI pretraining stage, systems are trained on language pairs from the WMT Direct Assessment datasets. To answer Qs, investigate whether our system generalizes well to language pairs unseen during training. In Table 4 we show Kendall's Tau correlation for WMT and non- WMT language pairs and their cor- relation difference three phenomena from the ACES dataset. The three phenomena and the
8

---

Published as a conference paper at ICLR 2024
Table Correlation difference of our system between WMT and non-WMT language pairs on three phenomena from the ACES dataset.
antonym-
real-world-
nonsense
replacement
knowlcdge-
commonsense
WMT
Non-
6
WMT
Non-
8
WMT
Non-
8
Avg-
WMT
WMT
WMT
MT-Ranker- R-se
0.360
0.326
0.034
0.321
0.262
0.059
0.832
0,470
0.362
0.152
MT-Ranker-Large 0.728
0.621
0.107
0.490
0.490
0.000
0.804
0.531
0.273
0.127
MT-Ranker-XXi 0.776
0.735
0.041
0.650
0.591
0.060
0.930
0.711
0.219
0.107
Table 5: Kendall's Tau correlation on the untranslated phenomena of the ACES dataset.
copy-source
untraanslated-vv--re--woddd
untranslatcd-vv--ynooyyy
MT WT-Ranker-Base 0.91
-0.29
-0.47
MI-Ranker-Large 0.93
-0.16
-0.28
MI-Ranker-XXI 0.41
-0.25
-0,30
language pairs were chosen following (Amrhein et al. 2022). We can make the two observations from the results. First, the correlation difference between and unseen language pairs is small for the antonyn-replen replaceeeet and real-world weeldd-nowlege--oomeeee conro phenomena. Second, the corre- lation difference falls with increasing model size, indicating higher generalization we scale up our models.
Performance on the Untranslated Phenomena. To answer Q4. we focus on the three phenomena of errors the ACES dataset covered under cat- egory untranslated: copy-source, untranslated-vs- ref-word and umtransiated/vs--yyooo whole machine translation remains untranslated in these types of errors. The untranslated portion can make the translation more similar to the source tence, and thus, these translation errors can be partic- ularly challenging for reference-free evaluation sys- tems. Table 5 shows the the performance of our sys- tems on these three types of errors. On the copy- source errors our systems perform well. On the other two phenomena, our systems perform poorly. This shows our systems' limitations, indicating room for further improvement.
Supervised Baseline Unsupervised MT-Ranker Supervised MT-Ranker
35-
Kendall's Tau
30
Part or the
25
20
15
DA20
MOM20
MQM21
MQM22
Figure 4: Performance improves on all
benchmarks after supervised training.
CONCLUSION AND DISCUSSION
Machine translation evaluation has been focused on predicting translation quality scores. The alter- native approach pairwise evaluation has remained underexplored. The pairwise evaluation approach sufficient for the most important application of machine translation evaluation systems, compar- ing machine translation systems. In this paper, explored pairwise evaluation in the reference-free scenario. This approach can deal with the most practical scenario where reference translation is un- available. We show that this approach achieve state-ofad of-the-art correlation with human judgments across five benchmark datasets without requiring any supervision from human-annotated data.
REFERENCES
Chantal Amrhein, Nikita Moghe, and Liane Guillou. ACES: Translation accuracy challenge sets for evaluating machine translation metrics. In Seventh Conference Machine Translation (WM722),

---

Published as a conference paper at ICLR 2024
Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, Eric Chu, Jonathan H. Clark, Laurent EI Shafey, Yanping Huang, Kathy Meier-Hellstern, Gaurav Mishra, Erica Moreira, Mark Omernick, Kevin Robinson, Sebastian Ruder, Tay, Kefan Xiao, Yuanzhong Xu, Yujing Zhang, Gustavo Hemaande Abrego, Junwhan Ahn, Jacob Austin, Paul Barham, Jan Botha, James Brad- bury, Siddhartha Brahma, Kevin Brooks, Michele Catasta, Yong Cheng. Colin Cherry. Christo- pher A. Choquette-Choo, Aakanksha Chowdhery, Clément Crepy, Shachi Dave, Mostafa De- hghani, Sunipa Dev, Jacob Devlin, Mark Díaz, Nan Du, Ethan Dyer, Vlad Feinberg, Fangxiaoyu Feng, Vlad Fienber, Markus Freitag. Xavier Garcia, Sebastian Gehrmann, Lucas Gonzalez, Guy Gur-Ari, Steven Hand, Hadi Hashemi, Le Hou, Joshua Howland, Andrea Hu, Jeffrey Hui, Jeremy Hurwitz, Michael Isard, Abe Ittycheriah, Matthew Jagielski, Wenhao Jia, Kathleen Kenealy, Maxim Krikun, Sneha Kudugunta, Chang Lan, Katherine Lee. Benjamin Lee, Eric Li, Music Li, Wei Li, YaGuang Li, Jian Li, Hyeontaek Lim, Hanzhao Lin, Zhongtao Liu, Frederick Liu, Mar- cello Maggioni, Aroma Mahendru, Joshua Maynez, Vedant Misra, Maysam Moussalem, Zachary Nado, John Nham, Eric Ni, Andrew Nystrom, Alicia Parrish, Marie Pellat, Martin Polacek, Alex Polozov, Reiner Pope, Siyuan Qiao, Emily Reif, Bryan Richter, Parker Riley, Alex Castro Ros, Aurko Roy, Brennan Sacta, Rajkumar Samuel, Renee Shelby. Ambrose Slone, Daniel Smilkov, David R. So, Daniel Sohn, Simon Tokumine, Dasha Valter. Vijay Vasudevan, Kiran Vodrahalli, Xuezhi Wang, Pidong Wang, Zirui Wang, Tao Wang, John Wieting, Yuhuai Wu, Kelvin Xu, Yun- han Xu, Linting Xue, Pengcheng Yin, Jiahui Yu, Qiao Zhang, Steven Zheng, Ce Zheng, Weikang Zhou, Denny Zhou, Slav Petrov, and Yonghui Wu. Palm 2 technical report, 2023.
Ondiej Bojar, Yvette Graham, Amir Kamran, and Miloš Stanojević. Results of the WMT16 metrics shared task. In Proceedings of the First Conference Machine Translation: Volume 2, Shared Task Papers, pp. 199-231, Berlin, Germany, August 2016. Association for Computational Lin- ac wwwaannnthotty.ooooo wwwg:/rrwwww.....o 2302
guistics. doi: 111/666////116---2
Ondřej Bojar, Yvette Graham, and Amir Kamran. Results of the WMT17 metrics shared task. In Proceedings of the Second Conference on Machine Translation, pp. 489-513, Copenhagen, Den- mark, September 2017. Association for Computational Linguistics. doi: 10. D0.18633v1//111-.
URL
Aljoscha Burchardt. Multidimensional quality metrics: a flexible system for assessing translation quality. In Proceedings of Translating and the Computer 35, London, UK, November 28-29 2013. Aslib. baattps://aclantteee wwtptttooooooooooolll...oo
Chris Callison Burchh, Miles Osborne, and Philipp Koehn. Re-evaluating the role of Bleu in ma- chine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, pp. 249-256, Trento, Italy, April 2006. Association for Computa- tional Linguistics. htthttts:// http://llaaattoooooo/o - orq/ E00-11322221
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel Bowman, Holger Schwenk, and Veselin Stoyanov. XNLI: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2475-2485, Brussels, Belgium, October-November 2018. Association for Computational Lin- guisties. doi: 10.18653/v//D18. 1269. URL
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Un- supervised cross-lingual representation learning at scale. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8440-8451, Online, July 2020. Association for Computational Linguistics. doi: //aclanthology. orq/ acw-mmi.....mm.....mm..m.mm..... 747
https::::
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18653/v//N19-1 1423. URL https:
7/acl anthlnnthoooooooooooo hww.hhooooo..oogg//1/1/o N
10

---

Published as a conference paper at ICLR 2024
Kevin Duh. Ranking regression in machine translation evaluation. In Proceedings of the Third Workshop on Statistical Machine Translation, pp. 191-194, Columbus, Ohio, June 2008. Associ- ation for Computational Linguistics. URL Tttt:pss//aclaatcllooy... org. -
Angela Fan, Shruti Bhosale, Holger Schwenk, Zhiyi Ma, Ahmed El-Kishky, Siddharth Goyal, Man- deep Baines, Onur Celebi, Guillaume Wenzek, Vishrav Chaudhary, Naman Goyal, Tom Birch, Vitaliy Liptchinsky. Sergey Edunov, Edouard Grave, Michael Auli, and Armand Joulin. Beyond english-centric multilingual machine translation. J. Mach. Learn. Res., 22(1), jan 2021. ISSN 1532-4435.
Patrick Fernandes, Daniel Deutsch, Mara Finkelstein, Parker Riley, André F. T. Martins, Gra- ham Neubig, Ankush Garg, Clark, Markus Freitag, and Orhan Firat. The devil is in the errors: Leveraging large language models for fine-grained machine translation evalua- URL https: hoss://appl api. www.nticsccoolaa.ccoooo
tion. ArXiv, abs/2308.00 072286, 2023.
Corpus ID ID: 260886680000
Markus Freitag, David Grangier, and Isaac Caswell. BLEU might be guilty but references are not innocent. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 61-71, Online, November 2020. Association for Computational Linguistics. doi: 11/866///w/2222.mmm1pl--oo 2020. eet.mmmmlpmmmin...
URL https://aclantheen
org/
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. Experts, emrrr, and context: A large-scale study of human evaluation for machine translation. Transactions of the Association for Computational Linguistics, 9: 9:1460-1474, 2021a. doi:
wwtp:/11.rgg002000.0 (ttp://2221.aaal11110 tacl-aallll 1.77
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondfej Bojar. Results of the WMT21 metries shared task: Evaluating metrics with expert- TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pp. 733-774, Online, November 2021b. Association for Computational Linguistics. URL https: http:::ps//aaaaathoooooooo
based human evaluations
org/ 2021. wmt 1..
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Kocmi, George Foster, Alon Lavie, and André F. Martins. Results of WMT22 metrics shared task: Stop using BLEU - neural metries are better and more robust. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 46-68, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics. URL https: //aclanthologee 2222 wmt -.2
Yvette Graham, Timothy Baldwin, and Nitika Mathur. Accurate evaluation of segment-level ma- chine translation metrics. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 1183- 1191, Denver, Colorado, May-June 2015. Association for Computational Linguistics. doi: 10.3115/1/NN5--1224
Francisco Guzmán, Shafiq Joty, Lluís Mårquez, Alessandro Moschitti, Preslav Nakov, and Massimo Nicosia. Learning to differentiate better from worse translations. In Proceedings of the 2014 Con- ference on Empirical Methods Natural Language Processing (EMNLP), pp. 214-220, Doha, Qatar, October 2014. Association for Computational Linguistics. doi: 10.3115/v 1/155/111///LL 1027. URL htt.httss///aclaattti http://cccatttooooooooor//////-/
Francisco Guzmán, Shafiq Joty, Lluís Mårquez, and Preslav Nakov. Pairwise neural machine transla- tion evaluation. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing ( (Vol- ume 1: Long Papers). pp. 805-814, Beijing, China, July 2015. Association for Computational Linguistics. doi: 10.3115/v1/P15-1000 1078. URLpsttss://acaa ptttps://aclanthology
org/P15-1078
Fábio Kepler, Jonay Trénous, Marcos Treviso, Miguel Vera, and André F. T. Martins. OpenKiwi: An open source framework for quality estimation. In Proceedings of the 57th Annual Meeting of the Association for Computational Singuistics-Systee Demonstrations, pp. 112.122. Florence, Italy, July 2019. Association for Computational Linguistics. URL https: www. aclweb.
lorg/anth ogy hogy/P194 PPtp//19913000
11

---

Published as a conference paper at ICLR 2024
Tom Koemi and Christian Federmann. Large language models are state-of-the-at evaluators of translation quality. In European Association for Machine Translation Conferences/Worisstpp. 2023. URL
Qingsong Ma. Ondiej Bojar, and Yvette Graham. Results of the WMT18 metrics shared task: Both characters and embeddings achieve good performance. In Proceedings of the Third Con- ference on Machine Translation: Shared Task Papers, pp. 671-688, Belgium, Brussels, Oc- tober 2018. Association for Computational Linguistics. doi: 10.18653/11W1188645555 URL
https: http://aalatttttc Logy. wog..ogg..oooooooooo..oo.......o wrg.or///wwo..... /W18 6450
Qingsong Ma, Johnny Wei, Ondiej Bojar, and Yvette Graham. Results of the WMT19 metrics shared task: gegmenn-level and strong MT systems pose big challenges. In Proceedings of the Fourth Conference on Machine Translation (Volume 2: Shared Task Papers, Day 1), pp. 62-90, Florence, Italy, August 2019, Association for Computational Linguistics. doi: 10. N0.18653/11/119555 URL /attpps:/aclantho...
org. W1 5302
Matouš Machážek and Ondřej Bojar. Results of the WMT14 metrics shared task. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pp. 293-301, Baltimore, Maryland, USA, June 2014. Association for Computational Linguistics. doi: 00.3115/v1/11-33336 URL wwwpooyy..roo111101/0
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. Results of the WMT20 metries shared task. In Proceedings of the Fifth Conference on Machine Transla- tion, pp. 688-725, Online, November 2020. Association for Computational Linguistics. URL httts::/aclaanthit http:///cllathhocc//oooo logy 2ttpg//2.22.m...111.2.. wmt-1.77
Despoina Mouratidis and Katia Lida Kermanidis. Comparing a hand-crafted to an automatically generated feature set for deep learning: Pairwise translation evaluation. In Proceedings of the Sltman-Informedd Translation and Interpreting Technology Workshop (HiT-IT 2019). pp. 66- 74, Varna, Bulgaria, September 2019. Incoma Ltd., Shoumen, Bulgaria. doi: 10.26615/isso 2683-0078. 11188.20199000 URL htt.tttt:://// accl anthology.
OpenAl. Gpt-4 technical report, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu, Bleu: a method for automatic
evaluation of machine translation. In Proceedings of the 40th Anmual Meeting of the Associa- fion for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. doi: 10.3 10.3115/1073083.1073333 URL https:
Matt Post. A call for clarity reporting BLEU scores. Proceedings of the Third Conference
Machine Translation: Research Papers, pp. 186-191, Belgium, Brussels, October 2018. As-
sociation for Computational Linguistics. URL htllhttp::/ /wwww www....leb...................... org/antho ogy/
W18-6319
Yiwei Qin, Weizhe Yuan, Graham Neubig, and Pengfei Liu. T5score: Discriminative fine-
tuning of generative evaluation metrics. ArXiv, abs/2212.05726, 2022. URL /nttps://ppi.
semanti semanticscholar... oorpscoooooaroooooororpsss ID: 2545644688
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http:
7/jmlr. wttp::ppppppppp.....p.. 1/00
20 074 html
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), ), pp. 2685-2702 Online, November 2020a. Association for Computational www.0000mmllmmainn.
Linguistics. doi: 10. 18633vv 00066//////222.... 2020. www.e-nnnmmm.... mnn..aii... 213
URL httpp::/aclanttoll.ss//// org/
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. Unbabel's s participation in the WMT20 metrics shared task. In Proceedings of the Fifth Conference Machine Translation, pp. 911- 920, Online, November 2020b. Association for Computational Linguistics. URL https: //
acl antaahhhhyyyyooo org. wmt 101
12

---

Published as a conference paper at ICLR 2024
Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. Are references really needed? unbabel-IST 2021 submission for the metries shared task. In Proceedings of the Sixth Conference Machine Translation, pp. 1040-1040, Online, November 2021. Association for Computational Linguistics. URL htt.hhttpss... //cc anthoolaattoooooo ogy. 2021. wmt 1.11
Ricardo Rei, Marcos Treviso, Nuno M. Guerreiro, Chrysoula Zerva, Ana C Farinha, Christine Maroti, José G. C. de Souza, Taisiya Glushkova, Duarte Alves, Luisa Coheur, Alon Lavie, and André F. T. Martins. CometKiwi: IST-unbabel 2022 submission for the quality estimation shared task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pp. 634-645, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Lin- guistics. URL ULLLatttps///C..aattooooooooo/re org. wmt-1.60
Ricardo Rei, Nuno M. Guerreiro, José Pombal, Daan van Stigt, Marcos Treviso, Luisa Coheur, José G. C. de Souza, and André T. Martins. Scaling up cometkiwi: Unbabel-ist 2023 submission for the quality estimation shared task, 2023.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computa- lional Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguis- URL https: //al:///ccaaaaa onthologg logy. org/2020.
tics. doi: 10. 2a20:///222..clmmminn 704
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru Komachi. RUSE: Regressor using sentence embeddings for automatic machine translation evaluation. In Proceedings of the Third Con- ference on Machine Translation: Shared Task Papers, pp. 751-758 Belgium, Brussels, Oc- tober 2018. Association for Computational Linguistics. doi: 10.18653/1/W18--6533 URL pettps://aclathhen
org /W1///1112- 6456
Xingyi Song and Trevor Cohn. Regression and ranking based optimisation for sentence level MT evaluation. In Proceedings of the Sixth Workshop Statistical Machine Translation, pp. 123- 129, Edinburgh, Scotland, July 2011. Association for Computational Linguistics. URL htthss:::
//aclanthology. .rg. W11 2113
Brian Thompson and Matt Post. Automatic machine translation evaluation in many languages via zero-shot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods Nat- ural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguisties.
Yu Wan, Dayiheng Liu, Baosong Yang. Tianchi Bi, Haibo Zhang, Boxing Chen, Weihua Luo, Derek F. Wong, and Lidia S. Chao. ROBLEURT submission for WMT2021 metrics task. In Proceedings of the Sixth Conference on Machine Translation, pp. 1053-1058, Online, Novem- ber 2021. Association for Computational Linguistics. URL aatp:ass//aaaaatoooooog... org/
2021 wmt wmw.1.11m. 114
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek Wong, and Lidia Chao. UniTE: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the Associ- ation for Computational Linguistics (Volume Long Papers), pp. 8117-8127, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 1/28655/////22222aacooooo... URL
attp:::/accLanthooooyy.ooo wwtp://ooooo....222....22... acl- 558
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. Huggingface's trans- formers: State-of-the-art natural language processing, ArXiv, abs/1910.03771. 2019.
Zhanglin Wu, Min Zhang, Ming Zhu, Yinglu Li, Ting Zhu, Hao Yang, Song Peng, and Ying Qin. Kg-bertscore: Incorporating knowledge graph into bertscore for reference-free machine translation evaluation. In Proceedings of the 11th International Joint Conference on Knowl- edge Graphs, IJCK '22, pp. 121-125, New York, NY, USA, 2023. Association for Com- doi: 10.11453579051..35999 7/doi. 110.0...115555955551 .05.5799990033.1.3333 3579065
puting Machinery. ISBN 9781450399876.
URL https:
13

---

Published as a conference paper at ICLR 2024
Yang Ye, Ming Zhou, and Chin- Lin. Sentence level machine translation evaluation ranking. Proceedings of the Second Workshop on Statistical Machine Translation, pp. 240- 247, Prague, Czech Republie, June 2007. Association for Computational Linguistics. URL https: a/ccl//cllaaa antho org..rg///.. 00sgyoorrg0000000000 -07-
Weizhe Yuan, Graham Neubig, and Pengfei Liu. Bartscore: Evaluating generated text gen- cration. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 27263-27277. Curran As- sociates, Inc., 2021. /ttp/tttss:://rrcceddnnng neurips....e...sseeeee........ http:a/wpppp.cpppppeeee//// wwtppppppee/////////...
Jingyi Zhang and Josef van Genabith. Translation quality estimation by jointly learning to score and rank. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 2592-2598, Online, November 2020. Association for Computational Linguistics. doi: 2020. www.e-nnnlmmnn 205
main.205. URL ht https: htt.s:://claathhologg/ orq/
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Eval- uating generation with BERT. In 8th International Conference on Learning Representa- fions, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020, OpenReview.net, 2020. URL
nttps://openrevier net. cost/forumidd- wiw.idddsseeEEEE
A APPENDIX
A.1 FORMAL DEFINITION OF KENDALL'S TAU CORRELATION
The Kendall-like Tau correlation with human judgments can be formally defined as,
Concordant) Discordant)
(12)
Concordand) Discordant]
where Concordant is the of all translation pairs where the metric agrees with the human anno- tators on which translation is better and Dicordant is set of all translation pairs where it disagrees.
A.2 SYNTHETIC DATASET CREATION
Here we discuss the details of synthetic dataset creation for our third stage of training.
A. BERTSCORE
We used BERTScore (Zhang et al. 2020 to generate some of the synthetic data. For the To-English language pairs we use microsofirddeertta-larrr ficial repository of BERTScore. For the From-Englin language pairs use xlm-roberta-llage version as it achieved the best performance on our development set.
variant of BERTScore as suggested in the of-
A.2.2 SUBSAMPLING THE MT-PRISM DATASET
We take at most 25000 samples per language pair from the MT-PRISM dataset. We only consider the language pairs covered in WMT17 WMT20 Shared Metries task. total we have about million samples in this dataset.
A.2.3 WORD DROP
We used the XLMRoberta (Conneau al.) 2020) tokenizer to tokenize the translation and randomly drop 15% of the tokens from the translation. Here we give an example of sample generated by this approach.
Source: Doch die Reduktion von CO2 ist eine besonders unwirksame Methode, um den Armen und Hungernden dieser Welt zu helfen.
14

---

Published as a conference paper at ICLR 2024
Original Translation: But cutting back on CO2 is particularty ineffective way help the world's poor and hungry.
Purturbed Translation: But back on CO2 is particularlyive way to help the world's poor and hungry.
A.2.4 WORD REPLACEMENT USING MASKED LANGUAGE MODEL
Similar to Word Drop we use the XLMRoberta (Conneau et al. 2020) tokenizer to tokenize the translation. Then we randomly replace 15% of the tokens using the XLMRoberta model.
Here we give an example of sample generated by this approach.
Source: Er hasst wenn fremde Menschen in seine Welt eintrelen.
Original Translation: He doesn't seem like it when other characters intrude on his territory.
Purturbed Translation: He doesn't It seem to like it when other animals intrude on his territory.
A.2.5 BACK TRANSLATION
We take a small subset of 50000 translations from our subsampled MT-PRISM dataset. The transla- tions are then translated french using the M2M100 (Fan et al. 2021) machine translation model. Then the french translations are back-translated to original language using the same model. We use greedy decoding to generate the translations.
Here we give example of sample generated by this approach.
Source: Und trotzdem hat die Ungleichheit zwischen dem lündlichen und dem urbanen Raum zugenommen.
Original Translation: And yet, inequality between rural and urban areas has increased.
Purturbed Translation: Inequalities between rural and urban areas have increased.
A.2.6 WORD REPLACEMENT AFTER BACK TRANSLATION
We again used the XLMRoberta model to perform word replacement. Word replacement was per- formed on all of 50000 back translated sentences.
Here we give an example of sample generated by this approach.
Source: Deshalb will die Türkei um jeden Preis die Zeitspanne für mögliche diplomatische Lösungen verlängern
Original Translation: As result, Turkey wants to extend, all costs, the time available for diplo- macy.
Purturbed Translation: As a result, Turkey has use, all costs, the time available for diplomacy.
A.3 IMPLEMENTATION DETAILS
A.3.1 DEVELOPMENT SET
To tune hyperparameters we construct a development set from DA17,DA18 and DA19 datasets. We randomly take 50 source sentences per language pair from these datasets. We use the relative ranking samples corresponding to these source sentences as development set. The Kendall's Tau correlation on this development set is used the validation metric.
A.3.2 HYPERPARAMETERS
In Table 6 we show the hyperparameters used for training models. Learning rate and batch size was tuned based on the validation metric. We use early stopping by evaluating the models every 1000 steps of and choose the checkpoint with the highest validation metric.
15

---

Published as a conference paper at ICLR 2024
Table 6: Hyperparameters used for training our models.
Batch Size Learning Rate #Training Steps (Stage 1) # #Training Steps
MT-Ranker-BBl Rasser-BBase
128
5 10
100k
20k
MI LanR
10-
100k
20k
MI MI-Ranker-XXI
32
1 1.10-6
20k
20k
A.3.3 TRAINING SETUP
The models trained A 100 GPUs. Each model was trained single GPU. We used gradient accumulation for training MT MT-Ranker-XXL with a batch size of 32. Training Tr--Ranker-Bar Mr-Ranker-Large and MT-Ranker-XXL takes about 6, 12, and 52 hours respectively.
A.4 BASELINE MODELS FOR THE MQM BENCHMARK
Since our evaluation method differs from the standard for the MQM benchmarks we need to repro- duce the results of baselines for the MQM datasets. We evaluate against publicly available baselines. Here we give the details of the baseline models.
A.4.1 UNITE
We chose the Unite-MUP (Wan et al. 2022) model available from the COMET library (Rei et al.) 2020a).
We use the model in reference- free scenario.
A.4.2 COMET-QE
We chose the Comet- QE-DA-20 (Rei et al. chose
2020b) model for evaluation on MQM20 dataset. We (Rei et al. 2021 for evaluation MQM21 and MQM22 dataset. Both of these models are available from the COMET library (Rei et al. 2020a).
A.4.3 COMETKIWI
We chose the WMT22-CometKiwiil (Wan al. 2022 model available from the COMET li- brary [Rei et al. 2020a).
A.5 SYSTEM LEVEL EVALUATION
Given the source sentence and pair of translations our systems predict which translation better. Thus our systems perform evaluations at the segment level. Here we discuss a straightforward approach perform system level evaluation from the segment level evaluations.
In system level evaluation we are given the machine translations of set of source sentences from multiple systems. Since our model predicts probability of which translation is better we can aggregate these probabilities across all the MT pairs to give probability score for which MT system better for each pair of MT systems. Then for each MT system we can average these probability scores to get a score for the system. These scores indicate how likely a MT system is to beat another MT system.
For example, suppose we have three MT systems A, B and C under consideration. We apply our system each pair and get the scores shown Table 7 Each cell indicates the probability that the system the row beats the system the column. Thus, system A beats system B with probability of 0.7. We take the average across rows to get the score for each system. Based on the scores we conclude the ordering C A> B.
We apply this approach to system-level evaluation on the DA20 dataset and achieve competitive results with baseline models. In the Table g we report the system-level Pearson correlation of the baseline models and our systems. The Pearson correlation is calculated against the system level gold human evaluation scores.
16

---

Published as a conference paper at ICLR 2024
Table System level evaluation for hypothetical scenario.
A
B
C Average
A
0.7
0.3
0.5
0.3
0.4
0.35
0.7
0.6
0.65
Table 8: System-level Pearson correlations on language pairs from the WMT20 Shared Metrics Task dataset. Avg. denotes the average score across all language pairs.
eS-en
de-en
ja-en
pl-en
ru-en
zh-en
Avg
SUPERVISED BASELINES
X-to-En
COMET-QE
75.5
93.9
89.2
44.7
88.3
79.5
84.7
79.4
OPENKIWI-XLMN
76.0
99.5
93.1
44.2
85.9
79.2
90.5
81.2
T5SCORE-XL. XLLP
73.0
99.5
95.9
51.3
94.0
92.0
87.9
84.8
OUR SYSTEMS
MT -a--anner-Base
80.0
100.0
94.0
52.0
91.0
92.0
88.0
85.3
MT UT-Ranker-Larg
80.0
99.0
94.0
60.0
92.0
93.0
89.0
86.7
WT-Ranker-XXI.
80.0
99.0
94.0
58.0
90.0
92.0
88.0
85.9
en-cs
en-de
en-ja
en-pl
en-ru
en-ta
en-zh
Avg
SUPERVISED BASELINES
EN-1o-X
COMET-QE
98.9
90.3
95.3
96.9
80.7
88.7
37.5
84.0
OPENKIWI-XLMP
97.2
96.8
99.2
95.7
87.5
91.0
-1.0
80.9
TSSCORE- XL
98.5
96.3
96.4
96.8
88.7
95.2
96.0
95.4
OUR
MI Sr--Raneer-Bast
99.0
94.0
98.0
97.0
63.0
93.0
47.0
84.4
Mr-Ranker-Laane
99.0
95.0
99.0
98.0
76.0
94.0
55.0
88.0
WT-Ranker-XXI.
99.0
95.0
99.0
98.0
75.0
94.0
57.0
88.1
17

---

Published as a conference paper at ICLR 2024
A.6 INCONSISTENCIES IN PAIRWISE RANKING DECISIONS
We have discussed above how to get system level scores with our systems by averaging across the row of pairwise comparison probabilities. With the system-level scores, there no contradictions. However, there chance of contradiction among the better-wors worse judgments of systems we do not provide this system level score and rely on each pairwise comparison probabilities make decisions. To that end, we have checked if there is any inconsistency among the pairwise system ranking generated by our system.
Table Percentage of triples from DA20 where MT-Ranker-Large makes inconsistent predictions.
zh-en en-de
ta-en
en-2h
pl-en
en-pl
0.54% 0.14% 0.91 % 0.27% 0.91% % 0.41% 0.27%
Among the language pairs on the DA20 datasets we have only found inconsistency on 7 of them. On the following table we show the percentage of triples that show inconsistency. We note the inconsistency level is less than one percent for the language pairs.
A.7 RESULTS FOR FURTHER LANGUAGE PAIRS ON WMT20 SHARED METRICS TASK
On main table for the WMT20 Shared Metries Task did not report results for the language pairs km-en, ps-en, in-en, en-iu for brevity of presentation. In Table I0 we report results for these language pairs.
Table 10: Segment-level Kendall's Tau correlations on further language pairs from the WMT20 Shared Metrics Task dataset.
km-en
ps-en
iu-en
en-iu
SUPERVISED BASELINES
COMET-QE
14.9
9.2
3.2
-6.3
OPENKIWIXXLMP
24.4
10.6
75SCORE-XLag
22.5
10.6
2.6
-2.6
OUR SYSTEMS
MT Rer-Ranker-Baan
30.3
19.0
MT- Tr-RRnkee--aage
33.4
20.1
5.2
14.7
MT-Ranker-XXL 33.1
19.7
6.6
18.0
Our systems strongly beat the baseline systems on these languages even though our systems were not trained on human annotations. For the iu-en and en-iu language pairs we noticed that we filtered these language pairs also from our training. The strong performance on these two pairs also show the generalization of our system to unseen language pairs during training.
18


Reviews:
[{"review_id": "rqVI05WrXW", "reviewer": "Reviewer_DxQx", "summary": "This paper proposes a novel reference-free machine translation evaluation method which directly compares the two hypotheses from two systems by pre-trained language models, e.g., mT5.", "questions": "1. How should the paper address the ranking of two MT systems? Is there a more robust method than simply tallying the number of better translations?\n2. When scaling up the evaluation to multiple systems, how might one resolve apparent ranking contradictions, such as the scenario where System A outperforms System B, System B outperforms System C, but System C outperforms System A?\n\nThese questions are pivotal in addressing the practical implications of the proposed method and its capacity to function in a more complex, real-world evaluation environment. A more thorough exploration of these aspects could substantially strengthen the paper’s contribution to the field.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The manuscript is commendably clear in its presentation, providing a lucid explanation of the method's underpinnings and its design rationale. The method itself is logically structured and appears to be grounded in a sound understanding of the underlying technical principles.", "weaknesses": "(Main) 1. **Experimental Settings**: Upon meticulous examination, I observe that the experimental setup deviates from the conventional practices of evaluating numerous systems simultaneously. The study opts to assess a custom set of 'Better-worse judgments' between pairs of system outputs. This focus narrows the scope of the evaluation and raises concerns about the validity of the correlation results. The paper's method shows a strong correlation with these judgments but fails to conclusively demonstrate the superiority of one system over another. The more critical challenge lies in integrating these pairwise comparisons into a comprehensive evaluation framework that can handle multiple systems. The current approach's limitations in addressing this challenge may undermine its utility and applicability.\n\n2. **Scope and Generalization**: The narrow focus of the study may limit its applicability beyond its stated domain. The paper could benefit from an expanded discussion on how the proposed method might adapt or extend to other evaluation contexts, such as the assessment of large language models (LLMs). While ICLR might be receptive to specialized domain contributions, the paper's current emphasis suggests that it might find a more fitting audience at a dedicated NLP conference.\n\n**Suggestions**\n1. **Contextualizing with Recent Advances**: Recent developments in MT evaluation metrics, such as xCOMET (https://arxiv.org/abs/2310.10482) and SLIDE (https://arxiv.org/pdf/2309.08832.pdf), emphasize the importance of error span evaluations. While it is not mandatory to compare against the latest publications, incorporating insights from these advancements could significantly enhance the robustness and relevance of the proposed method. \n\n2. **Related Work**: I recommend that the authors consider the insights from \"DABERTScore\" (https://aclanthology.org/2021.acl-short.5.pdf). There are conceptual overlaps between this work and the manuscript under review, which merit a thorough comparison and discussion within the paper to enrich the context and underscore the novel contributions of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "46k6erTsS9", "reviewer": "Reviewer_Qwxh", "summary": "In this paper, the authors propose a method to learn pairwise reference-less evaluation of MT. This scenario corresponds to a common, real use where reference translations are mostly unavailable and the interest is mostly in comparing systems, rather than absolute scores. The pairwise ranking is a good framework since it is easier to collect synthetic data with pairwise judgments, as opposed to assigning quality scores to synthetic examples. Pairwise rankings also achieve higher inter-annotator agreement than human judgments. The method proposes a 3-stage pipeline for finetuning with various kinds of synthetic data. \n\nThe results show that the approach can achieve state-of-the-art performance comparable or better than supervised approaches. The analysis also shows that adding supervised data can further improve performance modestly. Interpreted in another way, it also means that the synthetic data generated is sufficient to capture most of the attributes of supervised data for pairwise ranking of systems.", "questions": "* Equation 11 should be y=0?\n* 3 languages from the WMT-20 Metrics tasks have not been included in the evaluation. It would be good to include those results as well for getting a complete view of the WMT-20 Metrics task.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "* The paper is well-written and explains the motivation for the work well. \n* The experiments are extensive and establish that reference-less evaluation is very competitive with reference-based metrics. \n* The use of synthetic data for pairwise ranking is a very clean way of using synthetic data and helps train high-quality reference-less metrics.", "weaknesses": "While pairwise evaluations of systems are useful, a more practical utility would be to rank multiple models. Score-based systems enable that easily. With pairwise ranking-based systems, multiple comparisons have to be run. Every time a new system has to be ranked, it has to be compared with multiple existing systems.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uI28EXQElL", "reviewer": "Reviewer_GQxu", "summary": "This paper proposes a comparative MT evaluation metric: instead of comparing machine translations to references, it compares multiple machine translations. The model is built on a bidirectional LLM, that encodes pairs of translations, and a pooling and logistic regression layer on top. It is trained with data from crosslingual NLI, pairs of human and machine translations, and synthetically rated or corrupted pairs of translations. Evaluation is done on a set on the WMT20 metrics task, several MQM datasets and ACES, a challenge dataset. The proposed model is compared to previous state-of-the-art reference-free metrics and largely outperforms them across languages, as well as supervised baselines.", "questions": "- Can you explain what the sentence “relative ranking annotation from direct assessment with a large enough threshold has been used with as few as one annotation” (Intro) means? Where does the threshold come into play and what does it mean to have one annotation only (I assume one per input, but not only one input)?\n- Figure 1 is not adding much, its content is clear from the text. The space could be used to elaborate the connections to QE (see above) and report more empirical results on newer datasets.\n- Is there any train/test overlap of the training data of mT5 and the benchmarks’ test data? \n- What if references were removed from the fine-tuning sets in Stages 2 and 3? This would be a useful ablation to make the models truly free from references.\n- Which of the evaluation differences are significant? \n- What if Stages 1&2 are dropped and only Stage 3 is performed? This would further illustrate the importance of that stage and make it clear for future use where most time investment should go.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Idea and method are simple and well explained. Given that it requires much less costly data than the competing methods, it poses an attractive solution for MT evaluation.\n- The reported results are strong, given that they do not require references or direct supervision. The proposed model outperforms both supervised and unsupervised baselines.\n- The ablations give an insight into the importance of the different stages and generalization to unseen languages, which allows one to get a more thorough understanding of the method and its benefits.", "weaknesses": "The novelty seems limited / overemphasized. In Quality Estimation (QE) reference-free ranking approaches have been used for MT quality estimates before it was re-invented in the context of MT metrics competitions. For example, in the very first QE task in 2012 (https://www.statmt.org/wmt12/quality-estimation-task.html) ranking based evaluations (without references) were already designed, and as a result, ranking based methods have been developed as well (e.g. Avramidis, Eleftherios. \"Sentence-level ranking with quality estimation.\" Machine translation 27.3-4 (2013): 239-256.; Eleftherios Avramidis. 2012. Comparative Quality Estimation: Automatic Sentence-Level Ranking of Multiple Machine Translation Outputs. In Proceedings of COLING 2012, pages 115–132, Mumbai, India.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xKPhoTUwFs", "reviewer": "Reviewer_2G4a", "summary": "This work proposes Comparator, a reference-free MT evaluator that treats the evaluation as a inter-system comparison problem. The model consists of a pre-trained encoder that accepts the concatenation of the source sequence and a pair of translations to compare and a comparator head that pools the output embeddings and produces a binary decision on which translation is better. The model is trained in three stages: XNLI pre-training (preferring entailment over non-entailment), human/machine translation discriminating (preferring human translation over machine translations) and weakly-supervised tuning (pairs of translations judged by BertScore and synthetic data by perturbation). The proposed model is evaluated on various MT eval datasets and the results show its effectiveness and benefits even over supervised baselines.", "questions": "- What pre-trained model did the baseline systems use? Did they use up to XXL models? (Same or similar base models should be utilized for fair comparisons.)\n- Are there any ways to convert the relative comparisons to absolute scores? Sometimes, we might still need the scores (for example as rewards for RL).\n- In Stage 2 and the first part of Stage 3, references are required for the training purpose. If those two parts are ablated, how would it influence the results?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The proposed method is straight-forward and shows good performance over a range of datasets.\n- Some of the indirect and weakly supervised training method is interesting and might be inspiring for future study of MT evaluation.", "weaknesses": "- I’m wondering if the comparison is fare for other systems since the proposed model is trained with multiple external resources such as XNLI and especially some data with parallel sentences, and is based on large base pre-trained encoders. I think it would be more convincing if there can be ways to directly compare different evaluation systems (ref-free vs ref-included, score-based vs comparison-based) with the same training resources and base models. (But surely, it would be indeed a benefit if the proposed system can better utilize extra resources.)\n- There should be more analysis on the proposed methods (such as those in Section 5). For example, more ablation studies on the training stages and especially the usage of different resources, and more detailed analysis on the metric and perturbation methods in Stage 3. Some of the result and setting sections may be shorten or moved to the appendix.\n# --\n- (Updates): Most of these concerns are addressed by the authors' responses, and the authors should provide those details in later versions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (spotlight)