T5Score: Discriminative Fine-tuning of Generative Evaluation Metrics
Authors: Yiwei Qin
Weizhe Yuan
Graham Neubig
Pengfei Liu
Venue: EMNLP 2023 Findings (2023)
Keywords: text generation
evaluation

Abstract: Modern embedding-based metrics for evaluation of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs are of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the probabilities of a generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguishing between good and bad outputs, while generative metrics can be trained using abundant raw text. In this paper, we present a framework that combines the best of both worlds, using both supervised and unsupervised signals from whatever data we have available. We operationalize this idea by training T5Score, a metric that uses these training signals with mT5 as backbone. We perform an extensive empirical comparison with other existing metrics on 5 datasets, 19 languages and 280 systems, demonstrating the utility of our method. Experimental results show that: T5Score achieves the best performance on all datasets against existing top-scoring metrics at the segment level.

T5SCORE: Discriminative Fine-tuning of Generative Evaluation Metries
Yiwei Qin
Weizhe Yuan
Graham Neubig
Pengfei Liu
Carnegie Mellon University, New York University, O Inspired Cognition
(yiweiq, gneubig, pliu3 Jecs. cmu.edu wy8858nyu. edu
Abstract
gen
Generative Training
Modern embedding- baaaeddd-g- metrics for evalua- tion of generated text generally fall into one of two paradigms: discriminative metrics that are trained to directly predict which outputs of higher quality according to supervised human annotations, and generative metrics that are trained to evaluate text based on the proba- bilities of generative model. Both have their advantages; discriminative metrics are able to directly optimize for the problem of distinguish- ing hetween good and had outputs, while gen- erative metrics can be trained using abundant text. In this paper, we present frame- work that combines the best of both worlds, using both supervised and unsupervised signals from whatever data have available. We op- erationalize this idea by training T5SCORF, metric that uses these training signals with mT5 backbone. We perform an extensive empiri- cal comparison with other existing metrics on datasets, 19 languages and 280 systems, demon- strating the utility of our method.2 Experimen- results show that: T5SCORE achieves best performance all datasets against exist- ing top-scoring metrics at the segment level.
Parallel output
Encoder
Decoder
Parallel input
<
Parallel output
Better hypothesis
Discriminative Training
Decoder
Score Score 2
Encoder
Luss
Source/Referenee
Decoder
1 Worse hypothesis
Our framework supports generative and dis- criminative training. The former uses parallel data and maximizes the probability output data conditioned on the input data. The latter ranks hypotheses by their manual scores and maximizes the probability of the bet- ter hypothesis while minimizing the probability of the worse hypothesis. Lyen and Lais denote generative loss and discriminative loss respectively.
Figure 1:
1 Introduction
These models are generally trained by taking sen- tence embedding model and fine-tuning using human judgments of generated text quality as a learning signal, allowing metrics to directly predict the quality score of a text. Popular examples in- clude COMET (Rei et al., 2020) and BLEURT (Sel- lam et al., 2020). However, the effectiveness of this method comes at the cost of expensive manual annotation of human judgements, and thus these models are less broadly applicable than more com- mon lcxical metrics such as BLEU (Papineni et al., 2002), ROUGE (Lin, 2004).
Automatically evaluating the quality of generated text plays essential role in the development of text generation systems (Lin and Hovy, 2003; Peyrard, 2019; Mathur al., 2020a). A key ele- ment of this evaluation is the design of an auto- mated metric that can recognize high-quality lexts. The current most-popular approach to create such high-quality metrics is the discriminative paradigm.
"Corresponding author
'We mT5 because supports languages pared to other options (e.g.. mBART) and provides different scales of models (e.g.. 3B, 11B).
More recently, there has been promising work on generative metrics. These metrics recognize high- nigh-quality texts by formulating evaluation as a
'Appendix A.9.1
the dataset details.
WWe release our code and models at a/ato///anonyoouus.
4open.

---

generation task and using the generative likelihood an indication of quality, making it possible to train the models without explicit human annotation. Examples of these metrics include BARTScore (Yuan et al., 2021) and PRISM (Thompson and Post, 2020a). However, because such generative models do not utilize human judgements at training time, these models are inherently at a disadvantage compared metrics that can utilize supervision.
chines generate higher-quality texts.
Task Formulation
Text generation evaluation aims to design func- tion auto_ sitio___al-) that takes in source text x, some reference outputs y and a system output y and pre- diets a scalar value that indicates the quality of the system output. The validity of the designed function depends on the degree of correlation be- tween auto_ aatt__eva(-)) and human judgements (which can be denoted as manual maanull_eval()) The better the correlation between the two, the more effective we consider our designed function to be.
In this work, we argue that is crucial to utilize all possible supervision symbols that could indicate the quality of the text. To this end, we propose a framework for learning evaluation metrics based on the assumption that generative and discriminative objectives can work in concert train better evaluator, shown in Fig. 1.
Specifically, in this work, ation function (1) source-based if it takes only and ý and predicts using auto_evall auto_eval(x,)) (2) and call an evaluation function reference-based if takes only y and y and predicts using irto_eval(y,3)) y). or it takes x. y and ý and predicts using pooo_eval(xx,. y..
call an evalu-
We achieve this idea by (1) starting with the pre-trained model mT5 (Xue et al., 2021), (2) train- ing mT5 in a generative fashion by maximizing the probability of existing parallel data, then (3) fine-tuning mT5 discriminatively by minimizing contrastive loss function to teach it to ensure that the generative probability of high-quality texts is higher than that of low-quality texts. At evaluation time, the probability of generating text is used as the quality score, because the model has learned to assign high probability to superior texts. Our frame- work has the flcxibility to choose from a supervised training strategy and an unsupervised training strat- egy depending on if human judgements are avail- able for given language task, while keeping the evaluation process the same.
3 Metric Design
In this section, we describe TSSCORE and explain how to train the metric in both a generative and discriminative fashion.
3.1 Evaluation as Generation
Following Yuan et al. (2021), formulate text generation evaluation as text generation problem. Specifically, the quality of generated text is measured by calculating the per-token conditional probability of one text a given another text b, which we also abbreviate as "b a":
We evaluate the proposed metric (TSSCORE) on 5 datasets covering machine translation (MT) and summarization tasks across 19 languages. Regard- ing reference-based experiments, at the segment level, T5SCORE trained generatively achieves the dataset without human
1
TSSCORE
log (ogp(a|b;0)
(1)
aa
O are the parameters of the sequence-to-sequence model used to calculate these probabilities. De- pending on which strings we use for and b we can evaluate the text from different perspectives, We adopt the definition of Precision, Recall and score based on different generation directions (Yuan et al., 2021):
best performance annotated training examples; T5SCORE trained dis- criminatively achieves the best performance on datasets with human annotated training examples against top-scoring counterparts. At the system level, T5SCORE trained discriminatively achieves the best performance in 5 of 6 test settings (2 cor- relation methods x 3 datasets). Empirical results also show the effectiveness of generative training, especially for tasks without human judgments. Re- garding source-based experiments, we find it bet- ter at evaluating top-scoring systems compared to reference- based evaluation, showing the impor- tance of developing source- based evaluation as ma-
Precision: (x or + 9): Calculate probability from reference (or source) text to generated hy- pothesis p(ý y; 0) (or p(}[x;0)).
Recall (ý + or y): Calculate probability from generated hypothesis to reference (or source) text p(y (ý; 8) (or p(x
score (x or y + ý): The arithmetic average Precision and Recall to consider both directions.

---

According to preliminary experiments. score correlated better with human evaluation scores on the DA20 dataset ($4.1) than Precision and Recall, so we adopt score for default. order to support multilingual evaluation, we choose mT5 (Xue et al., 2021) as our pre-trained model.
the corpus PRR and changing x or y which is orig- inally the model's input the output and changing which originally the model's output to the in- put. Thus, half of the time we calculate f(ý) 1 Σ log p(yely <t: v,0) for reference based methods, and f(}) = / -== log p(x/ &. 0) for source based methods.
3.2 Generative Training for T5SCORE
Experimental Setup
Generative training aims to teach the model to gen- erate target text from the input text with a standard negative log likelihood loss:
4.1 Evaluation Datasets
We evaluate on 5 datasets: the Direct Assess- ment (DA) corpus from WMT20 metrics shared task (DA20; Mathur et al. (2020b)); datasets ob- tained by re-annotating the outputs from WMT20 and WMT21 shared task according to the Multi- dimensional Quality Metrics (MQM) framework (MQM20 & MQM21; Lommel et al. (2014)); the dataset of WMT20 shared task on Quality Estima- tion (QE20; Specia et al. (2021)); I)); and a multilin- gual summarization dataset (MultiSumm; Koto et al. (2021)). Details in Appendix A.9.1.
1 m
Cgen
log 1o) (2)
We use the MT dataset ParaCotta (Aji et al.,
2022) and paraphrasing dataset MT MT-prism (Thomp- and Post, 2020b) as parallel corpora" to train our models generatively.
3.3 Discriminative Training for T5SCORE
We also design discriminative training methods where human judgments for generation quality are available. Suppose we have an annotated training dataset D = {xi, Yi, Vi. Vi, mali = I..... N}, where X; yi. yr. and m, denote the i-th example of the source text, the reference text, the hypothesis text, and the manual score, respectively (f; and m, can be multiple hypotheses with their correspond- ing quality scores). We first generate relative rank dataset PRR = {xi. yi V,T . m my 1,... N} by finding a pair of hypotheses y, with higher manual score mit and ý; with lower man- ual score m, for the same source text x, and ref- erence text y;. Then, to encourage the model to assign higher probabilities to the better hypothesis y!, we adopt contrastive loss function, following Liu et al. (2022); Hopkins and May (2011):
4.2 Correlation Measurements
We consider both system-level and segment-level correlations with human judgments when evaluat- ing automated metries.
System-level evaluation calculates average human scores for each generation system to produce a scalar rating for the system performance. We em- ploy the Pearson correlation (sys-p) and Kendall's Tau correlation (sys-k) the evaluation measure for system-level metrics.
Segment-level correlation measures the correlation over segment-level assessments. We keep the same sctup as in Mathur et al. (2020b) converting Direct Assessment (DA) to DA relative rank (DARR) and adopting a Kendall's s Tau-like (seg-k) formulation as the evaluation measure. We adopt the bootstrap- ping method (p-value 0.05) (Kochn, 2004; Gra- ham al., 2014) for pair-wise significance tests.
Lais = m0, f(5)) - s(S+) + a(m+ m
(3)
where is the weight of the margin term. f is defined as f()) 1 for reference-based methods, and f(}) 1 ET log p(pelr<<. 0) for source-based methods, where m is the number of tokens in ý. Because we adopt F score for evaluation by de- fault, our training process also considers two gener- ation directions: from or y ÿ and from ý to y. We augment the training samples by repeating
4.3 Baseline Metries
in E log p( y.
We consider the following baseline metrics for comparison: BLEU (Papineni et al., 2002) which is the precision of n-grams of the MT output compared to the reference; ROUGE (Lin, 2004) which measures the lexical overlap between the system and reference; COMET (Rei et al., 2020) which is a discriminative metric that uses XLM- RoBERTa to encode source, hypothesis and refer- ence and can be optimised towards different ob-
"Appendix A.9,2 shows the corpus details.

---

jectives; BERTScore (Zhang et al., 2019) which computes the cosine similarity between the refer- ence and hypothesis tokens' embeddings based on BERT (Devlin et al., 2018); BLEURT (Sellam et al., 2020) which is a BERT-based regression model trained on synthetic examples and ratings from WMT; PRISM (Thompson and Post, 2020a) which is generative metric that scores MT system outputs conditioned their respective human ref- erences; BARTSccoe (Yuan al., 2021) which is a generative metric that uses BART (Lewis al.,
Figure System-level Kendall's Tau and Pearson cor- relations for the WMT DA20 corpus. Detailed results can be found at Appendix A.4.
..BBLEU
BERTS
PRISM
BARTSSOM
BLEURT
COMPT
90
Correlation
80
2019) to evaluate the generated text."
70
5 Reference-based Evaluation
60
sys-p
sys-k
We consider two tasks in reference-basee evalu- ation: machine translation (DA20, MQM20 and MQM21) and summarization (MultiSumm).
5.2 Results
For DA20, Tab. shows segment level Kendall's Tau correlation results of diverse met- rics for 10/8 language pairs with English as tar- get/source:" Fig.2 shows system level results on av- erage. For MQM20 and MQM21, Tab.2 shows both segment level and system level results. For Multi- Summ, Fig. illustrates the segment Kendall's Tau correlation."
5.1 Training Details
We consider four different sizes of base models: mTT5B (580M parameters), mT5-L (1.2B parame- ters), mT5-XL (3.7B parameters), and mT55XXL (11B parameters). Both generative and discrimina- tive training are considered, with the former based on ParaCotta corpora and the latter based on WMT DA corpora from 2017 to 2019. Our model imple- mentation is based on Huggingface transformers (Wolf et al., 2020). More details of the hyperpa- rameters, training time, computing resources can be found at Appendix A.2.
In all tables, &denotes correlations not signifi- cantly outperformed by any other metric for the given language pair, while +denotes correlations not significantly outperformed by any other unsu- pervised metric. The highest correlation for each language pair by unsupervised methods is under- lined, and the highest correlation overall is bold.
DA20 Evaluation of the training models is carried out on the DA20 dataset.
From the above tables and figures, we observe:
1) At segment level, our method achieves the best performance average. Supervised T5SCORE-
MQM We consider various discriminative training data, resulting in the following models:
'The specific models used for each bascline Metric found Appendix A.
"More training results could be found at Appendix A.
(a) T is trained on WMT DA corpus from 2017 to 2019.
"We find that one MT system NiuTrans, 1511 of language pair en-zh generates unnecessary spaces between Chinese characters. Most evaluation metrics cannot handle the spaces well, causing an obvious outlier MT system and influencing correlation greatly. especially Pearson's 's correlation which is notably sensitive outliers. To remove the influence, we delete the spaces between Chinese characters of MT system's outputs at evaluation time for every evaluation metric.
(b) T5SCORF_*21 is trained on WMT DA corpus from 2017 to 2020.
(c) T5SCORE-** mqm is (b) further trained for 1 additional epoch on MQM20.
FFor simplicity, report F score of model and other basclines like BERTScore, BARTScore which differenti- ate Precision, Recall
MultiSumm Due to the small size of the Multi-
score.
Because segment scores are reliable only when averaged over sufficient number of judgments (Mathur et al., 2020b), we choose Kendall's 's Tau-like segment score as our correlation metric, different from the original paper (Koto et al., 2021), which uses segment Pearson correlation. We also report Pear- son correlation results in Appendix A.8 for reference.
Summ dataset (135 examples per language pair), we do not undertake the additional training model specific to summarization. Instead, we use models trained on ParaCotta and WMT directly.

---

Table 1: Segment-lovel Kendall's Tau correlations for the WMT DA20 corpus. Avg-en denotes the average correlation achieved by metric across all x-en language pairs. Avg-x denotes the average correlation across all en-x language pairs, and Avg denotes the average correlation across all language pairs. This table shows the results all x-en language pairs and the results of en-x language pairs can be found at Appendix A.3.
es-en
de-en
iu-en
ja-en
km-en
pl-en
ru-en
ta-en
zh-en
Avg-en
Avg-x
Avg
UNSUPERVISED METHODS
scntBLEU
6,8
41.1
18.1
22.6
-2.5
-0.5
16.3
9.3
13.9
30.0
21.1
BERTScore
11.7
45.2
21.6
27.9
4.7
15.9
21.9
13.4
19.3
39.3
28.2
PRISM
13.5$
46.5
25.5
26.3
30.4
6.6
16.5
10.0
23.0
14.5
21.3
41.9
30.5
BARTScore
12.4
48.5f
23.5
26.6
31.8
9.1f
16.0
12.8f
23.8
16.39
22.1
43.4
31.6
T5SCORE-BBu
12.9
48.4
24.3
26.0
30.4
8.3
19.43
11,9
24.0f
15.9
22.2
42.8
31.3
TSSCORE-Lun
13.0f
48.7f
26.6f
27.9f
30,9
8.5
17.7
12.8
24.01
16.5t
22.7
32.7
T5SCORE-XS xl
13.0
48.8f
26.1
27.7f
29.8
9.2f
17.7
13.2f
23.8f
15.9
22.5
46.7
33.3
SUPERVISED METHODS
BLEURT
13.6$
47.6 6
27.1
28.1
31.2
4.7
18.43
10.3
25.3
14.7
22.1
50.0
34.5
COMET
12.9
48.5
28.1
27.4
29.8
9.94
15.8
15.6
24.2
17.13
22.9
51.0
35.4
TSSCORE-B:up
13.94
48.5
29.23
28.1
30.3
9.64
17.4
23.9
15.5
22.9
47.2
33.7
TSSCORE-Lou
14.0:
49.3$
28.53
28.99
30.1
8.3
17.6
15.3$
25.94
16.3
23.4
35.4
TSSCORE-XS PSScORE-XLoum
12.8
49.63
29.23
29.18
31.53
9.3
18.0
15.2$
25.4
15.8
23.6
51.4
36.0
XL surpasses all baselines for DA20, MQM20 and MQM21; unsupervised T5SCORF-L. surpasses all baselines for MultiSumm.
Figure 3: Segment-level Kendall's Tau correlation on MultiSumm corpus. Details Appendix A.8.
ROUGE:
T&Score XLaup
COMET
D
BERTScore
2) At segment level, as language model size in- creases, the metric performance tends to saturate.
BLEURT
PRISM
BARTScoxe
TSScoro-Lan
In Tab.7, from T5SCORE-B to T5SCORF-L and from T5SCORE-L to T5SCORE-XL, the perfor-
Correlation
50
mance of our unsupervised metric improves by 1.4
and 0.6 on average, respectively; while the perfor-
40
mance of our supervised metric improves by 1.7
and 0.6, respectively." 10 However, this is not clear at system level. A possible reason is that the system level, there are usually less than 20 sys- tems to be evaluated, much fewer than the number examples at the segment level, so tiny differ- ences in one MT system can have a large impact on the final results.
30
Focus
Coverage
4) For datasets without human annotated train- ing examples, our unsupervised method achieves the best performance. In Fig. 3, our supervised methods perform worse than unsupervised meth- ods, and other supervised methods do not work well either. The reason could be that these methods are trained on MT data, and their direct use for the summarization task may impair their performance. These results also indicate that our method has the advantage that the unsupervised version still works well in the absence of human annotated data.
3) At system level, method better at Kendall's Tau correlation compared Pearson cor- relation. 22 our method achieves the highest Kendall's Tau correlation compared to other base- lines while performs slightly worse than COMET in terms of Pearson correlation. This can be at- tributed to our training process, which adopts contrastive loss function, making our models bet- ter at predicting the relative rank of examples or systems instead of the absolute score.
Source-Based Evaluation
10 "From T5SCORE-XL to T5SCORE-XXL. the performance of our unsupervised metrics improves by 0.3. Detailed results can be found at Appendix A.5. Due to the limited computa- tional resources and the limited performance improvement of T5SCORE-XXI. we don't train supervised T5SCORE-XXL.
We also support source-based discriminatively trained methods. this section, we show the effec- tiveness the source-baser method. We consider the task of machine translation and inspect the re-

---

Table 2: Segment Kendall's Tau, system Pearson and system Kendall's Tau of different metrics on MQM20 and MQM21 dataset. Avg. denotes the average correlation achieved by metric across language pairs and years. Method COMET uses model wm120-comet-da and wmt2 1mmt21-come 1aollll-omtmmmm for MQM20 and MQM21 respectively. Method T5SCORE-XLug uses model T5SCORE-*22 and T5SCORE-* mqm for MQM20 and MQM21 respectively.
MQM-2020
MQM-2021
avg
en-de
zh-en
en-de
zh-en
sys-p sys-k seg-k
sys-p
sys-k
seg-k
sys-p
sys-k
seg-k
sys-p
sys-k
seg-k
sys-p
sys-k seg-k
UNSUPERVISED METHODS
SentBLEU
82.8
52.4
11.3
43.8
57.1
7.6
88.0
82.1
2.8
35.4
28.2
1.5
62.5
54.9
5.8
BERTScore
79.1
42.9
20.4
51.5
35.7
15.2
88.6
82.1
11.6
48.7
33.3
5.3
67.0
48.5
13.1
PRISM
98.9
81.0
27.8
77.8
64.3
23.3
80.7
56.4
12.7
49.0
30.8
10.0
76.6
58.1
18.5
BARTScore
91.9
90.5
25.0
54.0
50.0
20.8
86.7
79.5
17.9
43.2
30.8
9.5
69.0
62.7
18.3
T5SCORE-Bu.
94.6
71.4
23.4
52.2
42.9
20.2
82.8
64.1
13.3
49.0
35.9
10.4
69.7
53.6
16.8
DSSCoRE-Lun
95.1
81.0
25.6
57.5
42,9
21.8
84,2
64.1
15.1
51.4
35,9
10.4
72.0
56.0
18.2
T5SCORE-XI
93.4
81.0
27.7
66.6
64.3
22.7
86.4
71.8
16.2
51.7
41.0
10.5
74.5
64.5
19.3
SUPERVISED METHODS
BLEURT
95.5
71.4
30.7
91.6
78.6
24.3
79.7
64.1
16,4
47.3
41.0
10.3
78.5
63.8
20.4
COMET
96.5
71.4
27.7
88.9
71.4
22.1
77.1
66.7
19.8
55.9
33.3
14.1
79.6
60.7
20.9
TSSCORE-BE
98.4
71.4
25.5
65.2
42.9
21.8
91.7
79.5
18.0
56.5
46.2
11.5
78.0
60.0
19.2
TSSCORE-Lom
98.5
81.0
29.6
85.5
64.3
25.1 1
88.9
74.4
20.7
58.9
48.7
13.6
83.0
67.1
22.3
T5SCORE-XL.ug
98.9
81.0
31.8
89.0
78.6
25.5
86.0
69.2
19.2
62.9
53.8
12.5
84.2
70.7
22.3
sults on three datasets: DA20, MQM21 and QE20.
Tab.3 illustrates both the segment level and system level results. For DA20, we compare our super- vised model with COMET sre baseline (Rei et al., 2021). 12 Results can be found at Appendix A.6. For QE20, we choose PRISM (Thompson and Post, 2020a) which is reference free version of PRISM the unsupervised baseline, and Tran- sQuest (Ranasinghe et al., 2020b,a) which is the winner of WMT2020 Shared Task Quality Es- timation (Specia et al., 2021) as the supervised baseline." Tab. illustrates the segment Pearson correlation of different evaluation metrics.
6.1 Training details
Hyperparameters are kept the same in Sec. 5.1.
DA20 Generative training performed on MT- prism, while diseriminative training performed on the WMT DA corpus from 2017 to 2019. MQM21 We take the models from DA20 and fur- ther train them for additional epochs on MQM20, QE20 DA20 models are further trained on the QE20 train split. The best checkpoint is picked based its performance QE20 development split and the results on the test split are reported.
We have the following observations:
6.2 Results
1) For MQM21, T5SCORE surpasses the base- line at both segment and system level on average. For DA20, TSSCORE is better than the baselinc at the segment level on average, and better than the baseline for most language pairs at the system level. For QE20, both supervised and unsupervised TSSCORE surpass the corresponding baselines.
Table 3: Segment Kendall's Tau, system Pearson and system Kendall's Tau on the MQM21 dataset for source- based methods. The highest correlation for each lan- guage pair under each correlation method is hold.
en-de
zh-en
2) Overall, source-based models perform worse than the reference-basee models, but their differ- ences are much smaller on MQM21 than DA20. We
sys-p
sys-k seg-k
sys-p
sys-k seg-k
COMET migas
68.5
46.2
16.2
50.5
41.0
10.0
TSSCORE- B.mgu
42.9
23.1
13.0
53.0
41.0
6.2
TSSCORE- Lage
67.6
48.7
16.9
60.8
51.3
8.6
TSSCORE- XL
77.8
56.4
17.9
58.7
48.7
9.3
DA scores from 2017 to 2020 and adapted to MQM by fine- tuning on MQM20. We use the model w -comet-qe-mqqm "CCOMET,. is trained WMT DA scores from 2017 to 2019. We use the model wmm20-commt--e-daa Implementation and the model pogttrannpeess-dd-mulliinngggiii L'The correlation metric follows Specia et al. (2021).
For MQM21, We compare our supervised model with the COMET baseline (Rei al., 2021),
I 1ee use
COMET is source-based metric trained on WMT

---

conjecture that the reason for this may be related the human evaluation process where WMT-DA uses a mixture of source-based and reference-based annotations, while MQM uses the source.
Figure 4: Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations of different mod- els with and without generative training on DA20. T5SCORE-B and TSSCORE-Lw, are models with gen- erative training, while T5SCORF-Bwo. and T5SCORE- I Lwo are models without generative training.
Table 4: Segment level Pearson correlation QE20 corpus for source- based methods. The highest corre- lation by unsupervised method underlined, highest correlation overall is bold.
TSScoresiee
Correlation
80
si-en ne-en et-en ro-en en-de en-zh Avg
UNSUPERVISED METHODS
60
PRISM
2.9
-13.6
69.4
82.9
46.4
30.4
36.4
TSSCORE-B,
53.0
55.7
58.1
77.1
13.5
21.3
46.4
T5SCORE-L.
59.3
60.4
64.1
80.0
20.4
25.9 51.7
40
TSSCORE-XI..
58.8
61.7
64.0
80.7
26.7
28.5
53.4
SUPERVISED METHODS
seg-k
sys-p
sys-k
TO
58.9
75.5
76.6
88.0
42.3
44.1
64.2
T5SCORE-B.
58.1
69.9
71.8
83.4
44.7
41.4
61.6
T5SCORE-L,
58.9
75.5
76.8
86.4
49.8
44.0
65.2
egories the MQM datasets. To evaluate the per- formance of the metrics on given error category, use the score of each example in that error cate- gory as the gold standard to compare with the score given by the automated evaluation metrics. There are six error categories total, including the five error categories described in Sec. 4.1 and an overall category that measures all errors. Fig. 5 shows the Root Mean Square Error (RMSE)16 of diverse eval- uation metrics under different observe that: (1) Our model T5SCORE-XLaap ranks first overall in every error category except accuracy where BLEURT ranks first. (2) Supervised met- rics (BLEURT, COMET, T5SCORE-XL per- form better than other unsupervised metrics. (3) The evaluation perspectives that all metrics excel at are the same. All metrics perform best in terms of accuracy, much better than other error categories.
T5SCORE-XL
60,2
74.7
77.8
87.1
49.8
45.6
65.9
7 Analysis
We design analyses to better understand the mech- anism of TSSCORE and its strenghs over other metrics, specifically asking three questions: Q1: Is generative training necessary before discrim- inative training? Q2: What are the strengths and weaknesses of each evaluation metric? Q3: When will source-based evaluation outperforms reference-based evaluation?
categories. We
Effectiveness Generative Training our ex- periments, discriminative training is based on the model trained generatively. To answer Q1, we com- pare the performance of discriminatively trained models with and without generative training on DA20. The results shown in Fig. 4. We observe that (1) Both T5SCORE-B and T5SCORF- L are enhanced by generative training before dis- criminative training under three correlation met- rics (except that TSSCORE-B using segment level Kendall's Tau correlation has a little performance drop), which means that generative training is nec- essary to improve model performance. (2) Larger model T5SCORE-L benefits more from generative training, compared to T5SCORE-B B, indicating that larger models are better keeping knowledge from generative training.
Top-k Analysis To answer Q3, we conduct exper- iments on MQM21 and evaluate on the subset of the data from the top-k performing MT systems." 17 Re- sults presented in Fig. 6. We find that: (1) The advantage of the source-hased version of T5SCORF over the reference-based version increases we evaluate fewer systems, i.e., only high-quality sys- tems. Although not as pronounced in COMET, which also uses the source for its reference-basee
LoWe use RMSE instead seg-k, because a certain error category only few examples, the number of examples calculate seg-k k for category will too small accurate. Scores are normalized before calculating RMSE. "To get more high quality translations, we also evaluate human translations in this experiment. All other experiments in this paper don't 't include human translations.
Multi- Analysis For Q2, we compare diverse evaluation metrics under different error cat-
15 AAppendix A.7 has detailed results of every language pair.

---

Figure 5: RMSE different metrics in each error category on MQM dataset.
LimBART UL lOmBERT DUPRISM IimTS-XI-u FCCOMET le BLEURT I0mTi-XL-oup
RMSE
Aceuracy
Fluccryy
Locale
Style
Terminology
ovenal
Figure 6: Metrics performance of the top-k MT systems. X-axis is number k. Y-axis is correlation results.
T5-XL-STC T5-XL-ref COMET-sIC COMET-ref
18
80
80
16
60
sys-k
gys-
70
40
14
60
20
12
50
8
9
10
12 13
1415
9 10
110110101
1213
1415
8 9 1011 1311415
approach, it has roughly the same trend. This sug- gests that the source-based approach is more suit- able for evaluating top systems, and that source- based evaluation should be considered as machine systems are improved. (2) T5SCORE outperforms COMET on all top-k systems under all three cor- relation measures, except for the reference- based version under seg-k correlation. The better perfor- mance of COMET's reference-baseee version may be attributed to its combination of reference and source, further indicating the importance of source.
ing the lexical or semantic similarity between refer- and generated texts with surface-based string match (e.g., BLEU (Papineni et al., 2002), ROUGE (Lin, 2004) and CHRF (Popovié. 2015)) or unsuper- vised pre-trained components (c.g., BERTSCORE (Zhang et al., 2019), YISI (Lo, 2019; Lo and Larkin, 2020) and MOVERSCORE (Zhao et al., 2019)).
Recently, works such PRISM (Thompson and Post, 2020a) and BARTScore (Yuan et al., 2021) formulate evaluation as a generation task, which can not only make full use of pre-trained knowledge but also find more training data that can provide useful supervision for metrics to learn.
8 Related Work
In this paper, we propose framework that al- lows different types of signals be incorporated into metrics. Concurrent with our work, Uvilval (Zhong et al., 2022) formulates evaluation as a boolean question answering task and trained on remi-supervise data. By contrast, our method is based on the generation formulation, which enables us to utilize large amount of raw parallel data.
The increasing performance of generation systems equipped with large pre-trained models puts for- ward a higher requirement of the evaluation ability of automated metrics. As such, researchers are ex- ploring different evaluation frameworks by teach- ing metric to learn diverse types of knowledge.
The directest way is to supervise metrics with manually annotated judgments with trainable model, typical works include BEER (Stanojević and Sima' an, 2014), BLEURT (Sellam et al., 2020), ROBLEURT (Wan et al., 2022), COMET (Rei et al., 2020) and C-SPEC (Takahashi et al., 2021). Despite the superior performance of optimizing the correlation with human judgments, this method is expensive in creating human annotations. To by- pass this challenge, researchers attempt to evaluate generated texts in an unsupervised way by calculat-
Conclusions
In this paper, we augment evaluation metrics with the ability use different types of signal from data, which based the assumption that a good evalua- tion metric not only should be informed of how score different qualities of texts but also how high- quality texts are generated. We achieve this goal by proposing a discriminative generation frame-

---

work for the evaluation of generated texts, which outperform existing top-performing metrics on datasets from languages.
Experts, errors, and context: A large-scale study of human evaluation for machine translation. Transac- tions of the Association for Computational Linguis- rics, 9:1460-1474.
Limitations
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondiej Bojar. 2021b. Results of the wmt21 metrics shared task: Evaluating metrics with expert-based human evaluations ted and domain. In Proceed- ings of the Sixth Conference on Machine Translation, pages 734-774.
This study has potential limitations: (1) While dif- ferent perspectives (e.g. informativeness, fluency, factuality) of text could be evaluated, assign one overall quality score, which cannot necessar- ily reflect the quality of a certain aspect. In the future work, more interpretable metrics could be designed specifically for a certain evaluation per- spective. (2) Due to the contrastive training objec- tive of T5SCORE, our metric may not be as good at predicting the absolute score of segments sys- tems compared to predicting their relative ranks. (3) We focus on evaluating automatically gener- ated texts by machine systems. Human generated texts which might have different features from ma- chine generated texts could be addressed in future work. (4) We study reference-based and source- based metrics separately. A combination of both could be studied in the future. (5) The potential risk associated with our model is that the quality anno- tations may have been generated according the preferences of certain demographic group, for which explicit metadata is not available. As our model is trained on these annotations, it may inherit the underlying bias.
Yettte Graham, Nitika Mathur, and Timothy Baldwin. 2014. Randomized significance tests in machine translation. In Proceedings of the Ninth Workshop Statistical Machine Translation, pages 266-274.
Mark Hopkins and Jonathan May. 2011. Tuning ranking. Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 1352-136..
Philipp Koehn. 2004, Statistical significance tests for machine translation evaluation. In Proceedings of the 2004 conference on empirical methods in natural language processing, pages 388-395.
Philipp Koehn. 2005. Europarl: A parallel corpus for statistical machine translation. In Pruceedings of machine translation summit papers, pages 79 -6.
Fajri Koto, Jey Han Lau, and Timothy Baldwin. 2021. Evaluating the efficacy of summarization evaluation across languages. arXiv preprint 1Xxv:2106.01478
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: De- noising sequence-to-sequenc to-squencce pre-training for natural language generation, translation, and comprehension. arXiv preprint 1rXiv:1910.13::
References
Alham Fikri Aji, Tirana Noor Fatyanosa, Radityo Eko Prasojo, Philip Arthur, Suci Fitriany, Salma Qonitah, Nadhifa Zulfa, Tomi Santoso, and Mahendra Data. 2022. Paracotta: Synthetic multilingual paraphrase corpora from the most diverse translation sample pair. arXiv preprint arXiv:2205.044
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. In Text summarization branches out, pages 74-81.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint AXi::1810.04805
Chin-Yes Lin and Eduard Hovy. 2003. Automatic evaluation of summaries using n n-gram co-occurrence statisties. In Proceedings of the 2003 Human Lan- guage Technology Conference of the North American Chapter of the Association for Computational Lin- guistics, pages 150-157.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xi- aodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao- Hsiao-Wuen Hon. 2019. Unified language model pre- pre-training for natural language understanding and generation. Advances in Neural Information Process- ing Systems, 32.
Yang Liu and Mirella Lapata. 2019. Text summa- rization with pretrained encoders. arXiv preprint 1XXXv:1908.0844
Andreas Eisele and Yu Chen. 2010. Multiun: A mul- tilingual corpus from united nation documents. In LRECC
Yinhan Liu, Jiatao Gu. Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020. Multilingual denoising pre- training for neural machine translation. Transac- tions of the Association for Computational Linguis- tics, 8:726-742.
Markus Freitag, George Foster, David Grangier, Viresh Ratnakar, Qijun Tan, and Wolfgang Macherey. 202

---

Yixin Liu, Pengfei Liu, Dragomir Radev, and Graham Neubig. 2022. Brio: Bringing order to abstractive summarization. arXiv preprint arXiv:2203. 16804.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020a. Transquest at wmn2020: Sentence- level direct assessment. In Proceedings of the Fifth Conference on Machine Translation.
Chi-kiu Lo. 2019. YiSi a unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceed- ings of the Fourth Conference on Machine Transla- tion (Volume 2: Shured Task Pupers. Day 1), pages 507-513, Florence, Italy. Association for Computa- tional Linguistics.
Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov. 2020b. Transquest: Translation quality esti- mation with cross-lingual transformers. In Proceed- ings of the 28th International Conference Compu- lational Linguistics.
Ricardo Rei, Ana C Farinha, Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro Ramos, Taisiya Glushkova, André FT Martins, and Alon Lavie, 2021. Are references really needed? unbabel-ist 2021 sub- mission for the metrics shared task. In Proceedings of the Sixth Conference on Machine Translation, pages 1030- 1040.
Chi-kiu Lo and Samuel Larkin. 2020. Machine trans- lation reference-less evaluation using YiSi-2 with bilingual mappings of massive multilingual language model. In Proceedings of the Fifth Conference on Machine Translation, pages 903-910, Online, Asso- ciation for Computational Linguistics.
Arle Lommel, Aljoscha Burchardt, and Hans Uszkor- cit. 2014. Multidimensional quality metrics (mqm): A framework for declaring and describing transla- tion quality metries. Tradamâtica: tecnologies de la traducció, 0:455-463.
Ricardo Rei, Craig Stewart, Ana Farinha, and Alon Lavie. 2020. Comet: A neural framework for mt evaluation. arXiv preprint arXiv:2009.09025.
Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2019. Wiki- matrix: Mining 135m parallel sentences in 1620 language pairs from wikipedia. arXiv preprint arXiv:1907.0579)
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020a. Tangled up in BLEU: Reevaluating the eval- uation of automatic machine translation evaluation metrics. Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics, pages 4984-4997, Online. Association for Computa- tional Linguistics.
Abigail See, Peter Liu, and Christopher D Man- ning. 2017. Get to the point: Summarization with pointer- pointer-generator networks. arXiv preprint arXiv: 04366.4468.
Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondřej Bojar. 2020b. Results of the wmt20 metrics shared task. In Proceedingy of the Fifth Con- ference Machine Translation, pages 688-725.
Thibault Sellam, Dipanjan Das, and Ankur P Parikh. 2020. Bleurt: Learning robust metrics for text gener- ation. arXiv preprint 2r0iv:2004. 04696.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th anmual meeting of the Association for Computa- tional Linguistics, pages 311 1-3118.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost.
Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary. and André FT Martins. 2021. Findings of the wmt 2021 shared task on quality estimation. In Proceed- ings of the Sixth Conference on Machine Translation. pages 684-725.
Maxime Peyrard. 2019, Studying summarization eval- uation metrics in the appropriate scoring range. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 5093- 5100, Florence, Italy. Association for Computational Linguistics.
Miloš Stanojević and Khalil Sima' an. 2014. Beer: Bet- evaluation as ranking. In Proceedings of the Ninth Workshop on Statistical Machine Translation, pages
Maja Popović. 2015. chrf: character n-gram f-score for automatic mt evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pages 392-395.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers. pages 186- 191, Belgium, Brussels. Association for Computa- tional Linguistics.
Kosuke Takahashi, Yoichi Ishibashi, Katsuhito Sudoh, and Satoshi Nakamura. 2021. Multilingual machine translation evaluation metrics fine-tuned pseudo- negative examples for wmt 2021 metrics task. Proceedings of the Sixth Conference on Machine Translation, pages 1049-1052.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach, Learn Res,, 21:140: 40:4400:44000 1-14677
Brian Thompson and Matt Post. 2020a. Auto- matic machine translation evaluation in many lan- guages via zero-shot paraphrasing. arXiv preprint arXiv:2004. 200-004444.4..

---

Brian Thompson and Matt Post. 2020b. Automatic ma- chine translation evaluation in many languages via zero-shol paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 90-121, Online. Association for Computational Linguistics.
is trained on WMT DA corpus from 2017 2020 and further trained for I additional epoch on MQM-20," Tab.5 shows that adding one more year's DARR corpus (WMT-DA-20) as training data, T5SCORE-*2 has better performance than T5SCORE-#2) #21 Besides, adding MQM in training has large performance improvement, although the total number of MQM training samples are much less than samples in DA corpus.
Yu Wan. Dayiheng Liu, Baosong Yang, Tianchi Bi Haibo Zhang, Boxing Chen, Weihua Luo, Derek F Wong, and Lidia S Chao. 2022. Robleurt submis- sion for the wmt2021 metries task. arXiv preprint arXiv:2204. 13352.
Table 5: Segment Kendall's Tau, system Pearson and system Kendall's Tau of different metrics on MQM21 with different training data. highest correlation for each language pair under each correlation method is bold.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistae, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer. Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020. Transform- ers: ofate-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38 45, Online, Association for Computational Linguistics.
en-de
zh-en
sys-p
sys-k seg-k
sys-p
sys-k seg-k
COMET
82.3
64.1
18.4
37.2
41.0
11.9
COMET map
77.1
66.7
19.8
55.9
33.3
14.1
T5SCORE-B
75.8
53.8
14.5
42.5
28.2
10.0
Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami A1-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021. m55: A massively multilingual pre-trained text-to-text transformer. In Proceedings of the 202/ Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies. pages 483-498, On- line. Association for Computational Linguistics.
TSSCORE-B"
82.3
56.4
16.3
41.6
28.2
10.9
T5SCORE-BaL
91.7
79.5
18.0
56.5
46.2
11.5
TSSCORE-1." 20
79.4
56.4
15.8
44.1
38.5
11.3
L 21
83.9
59.0
17.7
44.0
33.3
11.1
T5SCORE-L 21
88.9
74.4
20.7
58.9
48.7
13.6
TSSCORE-XI 21
83.5
61.5
18.0
47.3
35.9
12.4
T5SCORE-XI. 20
78.7
53.8
16.6
53.4
43.6
12.2
TSSCORE-XLL mmqm
86.0
69.2
19.2
62.9
53.8
12.5
Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021. Bartscore: Evaluating generated text text genera- tion.
A.2 Training Details
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Eval- uating text generation with bert. arXiv preprint arXiv:1904.0960 1904.0967.
We adopt the Adafactor (Shazeer and Stern, 2018) optimizer following Raffel et al. (2020). For gen- erative training, our model is trained on ParaCotta, and WMT-19 is used as validation set, while WMT- 20 is used test set. The hyper-parameter tuned is learning rate. For discriminative training, on the base of the generative model, we further train our model on the z-score of WMT DA corpus from 2017 to 2019, using the discriminative loss func- tion in cquation3. The spprmeprraeeters tuned are learning rate, dropout rate and We choose 4 language pairs (ru-en, en-ru, en-pl, en-cs) as the validation set to tune the hyper-parameters and pick the best model checkpoint.
Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Chris- tian M Meyer, and Steffen Eger. 2019. Moverscore: Text generation evaluating with contextualized em- beddings and earth mover distance. arXiv preprint arXiv:1909.0200
Ming Zhong, Yang Liu, Da Yin, Yuning Mao, Yizhu Jiao, Pengfei Liu, Chenguang Zhu, Heng Ji, and Jiawei Han. 2022. Towards unified multi- dimensional evaluator for text generation. arXiv preprint 210.022210.07117.
A Appendix
Tab.6 shows the pyper--arameters, training time and computing resources of T5SCORE. We find as the model size increases, we need less training steps to get the best performance on the valida- tion set, so maximum training steps is decreased
A.1 Training with Different Corpora
We compare our model discriminatively trained on different training corpora: T5SCORE_*3)
T5SCORF_ and T5SCORF. mym with baselines: 20 which is trained on WMT DA cor- pus from 2017 to 2019 and C0MET21 mgm which
COMET22 uses the model wmit2-comet-da and COMET mmm uses the model 1mm21-comet-mqm.

---

for larger models. We use the linear learning rate scheduler and use 10% of the maximum training steps as the wamp up step. In the table, when GPU is larger than 1, we use model parallelism.
A.8 MultiSumm Segment Level Pearson Correlation Result
Tab.20/Tab.21 illustrates the segment Kendall's s Tau/Pearson correlation of diverse evaluation meth- ods for language pairs MultiSumm dataset.
A.3 WMT-DA segment level results
A.9 Dataset
A supplement to the Tab. showing the segment- level results on language pairs with English as source for the WMT DA20 corpus.
A.9.1 Evaluation Dataset
DA20 DA20 is the Direct Assessment (DA) cor- pus from WMT20 metrics shared task (Mathur et al., 2020b) which includes 23,293 segments across 18 language pairs. DA20 covers 18 language pairs and 211 systems:km-en(7), en-cs(12), pl- en( (1)14), ru-en(11), iu-en(11), en-iu(11), ta-en(14), en-pl(14), en-ta(15), zh-en(16), en-zh(12), en(12), de-en 12), ja-ee(1())))) en-ja(1 en-de(14), ps-en(6), en-ru(9).
A.4 WMT-DA system level results
Besides segment level correlation, we also calcu- late system level correlation on WMT DA20 cor- pus, and the results are shown in 1,0.8,9,10011.
A.5 Unsupvised 15SCORE-XXI Results
Tab. 12, we show results on WMT DA20 corpus using generative TSSCORE-XXL.
MQM20 & MQM21 MQM20 (Freitag al., 2021a) and MQM21 (Freitag al., 2021b) are datasets obtained by professional translators who re-annotated the outputs from WMT20 and WMT21 1 shared task according to the Multidimen- sional Quality Metrics (MQM) framework (Lom- mel et al., 2014). The MQM framework tains assessments of five aspects of the text, which are accuracy, fluency, terminology, style, and lo- cal. MQM20 covers language pairs and sys- tems: en-de(10), zh-en(10) and comprises 1,418 and 2,000 segments for language pair en-de, zh- en respectively. In our experiments, we excluded 3 human translation systems for en-de and 2 hu- man translation systems for zh-en. MQM21 covers language pairs and 32 systems: en-de(17), zh- en 15) including 527 and 650 samples for en-de. zh-en respectively.
A.6 WMT-DA Source Based Evaluation Results
We conduct source-based evaluation experiments on WMT DA20 corpus using supervised TSSCORE. We compare our supervised source-based model with COMET baseline (Rei et al., 2021), which is a source-based metric trained to predict WMT DA scores from 2017 to 2019. Tab. 13 illustrates the segment Kendall's Tau correlation of diverse evaluation metrics on WMT-DA. The results show comparable
that at segment level T5SCORE-B. to the baseline and larger models surpass the base- line. Our system level results, shown in Tab. 14,15 are also comparable to or better than the bascline for most language pairs except iu-en and en-iu. In all tables, Avg-en denotes the average correlation across all x-en language pairs; Avg-x denotes the average correlation for all en-x language pairs; Avg denotes the average correlation for all language pairs.
QE20 QE20 (Specia et al., 2021) is the dataset of WMT20 shared task on Quality Estimation (QE). covers 6 language pairs: en-de, en-zh, ro-en. et-en, ne-en, si-en, comprising 7,000/1,000 ,000/1,000 1000 segments for Train/Dev/Test set for each language pair respectively, and each language pair uses one state-of-the-arr machine translation models built using the fairseq toolkit (https: //gi com/ hatbbookresearcc/aairsseee
A.7 Effectiveness of Generative Training
To show the importance of unsupervised gener- ative training, we compare the performance of TSSCORE-B and TSSCORE-L with and without un- supervised generatively training in Tab. 1,, 17,18,,1 19. Avg denotes the average correlation for all lan- guage pairs.
MultiSumm MultiSumm (Koto al., 2021) is multilingual summarization dataset containing texts and their summaries in eight languages (en, id, fr, tr, zh, ru, de, es). The dataset collects 135 documents in each language, as well as summaries
"COMETT> bee uses the model wm220l-cmmet-gedd..

---

Table 6: Hyper-parameelers, training time and computing resources. Max-step means the maximum training steps.
Save-step means save checkpoint every save-step. GPU shows the GPU type and the number of GPUs. A6000 is the NVIDIA RTX A6000. Time is the wall clock training time.
max-step
save-step
hatch-size
warm-up
learning-rate
dropout
GPU
time
T5SCORE-B...
10,000
500
10
1,000
5.00c-5
0.10
A6000*1
0.6h
T5SCORE-Lan
10,000
500
1,000
5.00e-5
0.10
A6000*
Ih
T5SCORE-XL.n
10,000
500
10
1,000
5.00c-5
0.10
A6000*
3h
TSSCORE-FR
300,000
10,000
32
1.00e-5
0.05
A6000*1
60h
TSSCORE-Lasp
200,000
10,000
1.00e-5
0.05
A6000*2
80h
TSSCORE xX-xpp
50,000
5,000
32
1.00e-5
0.05
A6000
40h
Table Segment-level Kendall's Tau correlations on language pairs with English source for WMT DA20 сorpus. Avg denotes the average correlation achieved by metric across all en-x language pairs.
en-es
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-zh
Avg
UNSUPERVISED METHODS
SEn
43.2
30.2
19.1
47.9
15.3
5.1
39.5
39.7
30.0
BERTScore
51.1
39.5
19.5
53.8
28.5
20.5
60.4
41.1
39.3
PRISM
61.2f
43.7f
19.6
57.8
39.9t
26.8t
39.3
47.3f
41.9
BARTScore
55.9
42.0
26.0
57.9
32.1
24.4
62.8
46.2
43.4
T5SCORE-Bum.
52.4
39.4
35.81
55,9
31.9
22.4
62.1
42.8
42.8
T5SCORE-L... -
56.3
41.1
35.9;
57.4
35.3
25.3
65.9
45.4
45.3
T5SCORE-XL...
58.9
42.2
34.0
59.81
38.3
27.4+
66.31
47.11
46.7
SUPERVISED METHODS
bleurt
69.3/
45.9
32.6
60.8
45.5
30.1
65.3
50.3
50.0
comet
66.8
46.8t
32.2
62.4
46.24
34.5
67.1
52.3
$1.0
T5SCORF-BR BBB
61.2
43.1
32.4
60.5
38.9
28.9
64.1
48.5
47.2
T5SCORE-Lan
66.6
45.6
33.4
62.8$
43.9
31.8
66.7
52.24
50.4
T5SCORE-XLE -L
68.0
46.84
34.1
63.0t
45.6
33.8$
68.1
51.74
51.4
Table 8: System level Pearson correlations language pairs with English as target for the WMT DA20 corpus. Avg. denotes the average correlation across all x-en language pairs.
es-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
nu-en
ta-cn
zh-en
Avg
UNSUPERVISED METHODS
ScntBLEEU
84.4
97.8
65.2
97.4
96.9
50.1
88.8
91.7
92.5
94.8
85.9
BERTScore
81.6
99.8
73.0
97. /4
95.0
59.2
93.0
92.7
89.5
96.1
87.7
PRISM
81.8
99.8
83.3
97
95.0
50.2
96.6
90.8
89.8
95.7
88.0
BARTScore
84.4
99.8
79.2
97.7
94.4
52.7
95.1
92.4
91.4
95.8
88.3
T5SCORE-B...
86.1
99.9
80.9
98.1
95.2
$2.9
95.8
93.1
90.7
96.1
88.9
TSSCORE-
84.4
99.9
83.8
98.
95.2
53.6
96.7
92.9
89.9
96.2
89.1
T5SCORE-XL...
83.9
99.9
84.0
97.8
95.0
53.7
96.7
92.5
89.4
96.2
88.9
SUPERVISED METHODS
bleurt comet
78.9
99.7
84.0
96.5
99.4
56.6
95.7
89.8
92.0
94.7
88.8
78.3
99.8
85.2
96.4
97.1
59.1
94.1
92.3
88.0
95.2
88.6
T5SCORE-B
81.9
99.4
78.9
98.2
97.8
57.4
95.1
91.9
88.1
96.4
88.5
TSSCORE-L sup
80.4
99.3
77.8
98.0
98.6
59.4
95.4
92.2
89.6
96.6
88.7
TSSCORE- -
79.3
99.3
81.7
97.4
98.3
58.0
95,9
92.0
88.4
95.5
88.6

---

Table 9: System level Pearson correlations on language pairs with English as source for the WMT DA20 corpus. Avg. denotes the average correlation across en-x language pairs and Avg-all denotes the average correlation across all language pairs.
eT-Cs
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-zh
Avg
Avg-all
UNSUPERVISED
METHODS
sentBLEU
84.1
93.4
13.5
94.6
95.0
98.1
88.2
92.8
82.5
84.4
BERTScore
88.4
95.5
68.4
97.0
91.4
97.2
95.4
92.4
90.7
89.1
PRISM
94.9
95.8
85.9
93.2
95.8
72.4
91.6
96.8
90.8
89.3
BARTScore
90.8
94.5
86.2
94.7
95.5
61.8
95.8
94.7
89.3
88.7
TSSCORE Bai
89.1
95.9
65.8
95.5
96.0
63.8
96.1
96.4
87.3
88.2
T5SCORE-Lu:
91.0
95.9
67.0
95,9
96.1
63.0
96.6
96.6
87.8
88.5
T3SCORE-XS XLm
92.4
95.2
63.2
94.9
96.4
70.6
96.7
97.0
88.3
88.6
SUPERVISED METHODS
bleurt
98.8
95.4
74.8
95.8
98.3
82.6
92.6
98.3
92.1
90.2
97.8
97.2
86.0
97.4
98.1
92.5
94.4
98.2
95.2
91.5
T5SCORE-
93.7
97.2
67.9
98.2
97.8
92.9
93.7
97.6
92.4
90.2
T5SCORE-Logg
96.4
97.3
67.4
97.4
98.0
94.8
95.9
97.8
93.1
90.7
T5SCORE-XI sup
96.8
96.8
61.3
95.9
98.3
95.7
95.0
98.6
92.3
90.2
Table 10: System level Kendall's Tau correlations language pairs with English target for the WMT DA20 corpus. Avg. denotes the average correlation across all x-en language pairs.
cs-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
ta-en
zh-en
Avg
UNSUPERVISED METHODS
ScntBLEU
78.8
75.8
45.5
73.3
61.9
27.5
60.0
60.0
69.2
85.0
63.7
BERTScore
78.8
72.7
63.6
73.3
71.4
47.3
86.7
52.7
67.0
80.0
69.4
PRISM
75.8
72.7
67.3
86.7
71.4
34.1
86.7
56.4
64,8
80.0
69.6
BARTScore
81.8
75.8
67.3
86.7
71.4
36.3
86.7
56.4
64.8
83.3
71.0
TSSCORE-BB
84.8
72.7
70.9
82.2
71.4
36.3
86.7
56.4
67.0
81.7
71.0
T5SCORE-Lon
81.8
75.8
74.5
86.7
71.4
36.3
86.7
56.4
64.8
80.0
71.4
T5SCORE-XL.
78.8
75.8
74.5
82.2
71.4
36.3
86.7
56.4
62.6
80.0
70.5
SUPERVISED METHODS
bleurt comet
75.8
$1.8
63.6
77.8
100.0
42.9
86.7
49.1
60.4
73.3
71.1
72.7
75.8
63.6
77.8
100.0
40.7
86.7
56.4
62.6
73.3
71.0
TSSCORE-B, B
72.7
75.8
56.4
82.2
90.5
42,9
86.7
56.4
60.4
83.3
70.7
T5SCORF-Lam.
75.8
78.8
56.4
77.8
100.0
42,9
86.7
52.7
60.4
71.7
70.3
TSSCORE- -I XI=
78.8
78.8
67.3
82.2
90.5
45.1
86.7
52.7
62.6
76.7
72.1

---

Table 11: System level Kendall's 'Tau correlations language pairs with English source for the WMT DA20
corpus. Avg. denotes the average correlation across all en-x language pairs and Avg-all denotes the average correlation across all language pairs.
en-es
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-2h
Avg
Avg-all all
UNSUPERVISED
METHODS
SentBLEU
51.5
23.6
85.5
60.4
72.7
69.4
BERTScore
51.5
34.5
85.5
56.0
94.4
72.7
PRISM
81.8
45.5
81.8
67.0
61.1
81.8
72.4
70.8
BARTScore
54.5
41.8
78.2
64.8
61.1
75.8
69.7
T5SCORE-Bun
54.5
84.6
34.5
85.5
67.0
61.1
86.7
69.7
68.0
69.7
TSSCORE-L.m
51.5
82.4
34.5
67.0
61.1
84.8
75.8
68.3
70.0
T5SCORE-XI.
51.5
82.4
27.3
85.5
67.0
66.7
84.8
72.7
67.2
69.0
SUPERVISED METHODS
bleurt
90.9
84.6
34.5
81.8
73.6
66.7
73.3
84.8
73.8
72.3
comett
90.9
84.6
38.2
74.5
73.6
72.2
77.1
81.8
74.1
72.4
TSSCORE-
75.8
86.8
34.5
85.5
69.2
72.2
77.1
84.8
71.8
TSSCORE-Lan
30.9
85.5
69.2
72.2
84.8
72.5
T5SCORE-XL.g 90.9
23.6
71.4
72.2
87.9
73.5
Table 12: Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations all language pairs for the WMT DA20 corpus using generative T5SCORE-XXL. Avg-en denotes the average correlation across all x-en language pairs, Avg-x denotes average correlation across en-x language pairs, and Avg denotes the average correlation across all language pairs
cs-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
ru-en
la-en
zh-en
Avg en
seg-k
13.5
26.9
29.9
16.7
13,9
15.8
22.7
sys-p
83.0
83.7
97.6
55.2
92.4
96.2
89.0
sys-k
72.7
74.5
77.8
71.4
36.3
86.7
56.4
80.0
en-cs
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-zh
Avg-x
Avg
seg-k
60.4
33.8
60.6
40.0
26.9
65.3
47.2
47.3
33.6
sys-p sys-k
93.1
64.2
96.4
68.5
96.7
97.2
88.5
88.8
60.6
82.4
27.3
89.1
64.8
66.7
82.9
75.8
68.7
69.2
Table 13: Segment level Kendall's Tau correlations on WMT DA20 corpus using source-based models. The highest correlation for each language pair is bold.
es-en
de-en
iu-en
ja-en
km-en
pl-en
ps-cn
ru-en
ta-en
zh-en
Avg-en
COMET
9.2
40.8
15.3
14,9
9.2
10.1
16.7
9.2
13.3
T5SCORE-B
43.2
16.2
11.9
9,7
7.5
19.2
9.4
12.9
T5SCORE-Lac
19.2
16.7
10.5
8.4
22.5
8.1
13.2
T5SCORE-XLL
22.1
22.5
7.3
7.4
14.9
en-cs
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-2h
Avg-x
Avg
COMET ur
61.3
-6.3
46.7
26.4
39.8
23.5
T5SCORE-Bas
55.3
36.4
51.2
24.7
41.2
23,5
T5SCORE-LE
60.5
-5.2
56.9
27.2
44.3
25.2
T5SCORE-XI..
62.7
40.0
-2.6
58.7
28.2
45.6
27.0

---

Table 14: System level Pearson correlations WMT DA20 corpus using source based models.
CS-cn
de-en
iu-cn
ja-en
km-en
pl-en
ps-en
ru-en
ta-cn
zh-en
Avg-cn
COMET 75.5
93.9
70.6
89.2
89.6
44.8
83.2
88.3
79.5
84.7
79.9
T5SCORE-BEr
82.6
99.7
-7.8
94.1
86.9
47.4
92.2
90.0
81.6
95.1
76.2
TSSCORE Lire
73.9
99.7
-17.3
96.2
92.2
52.6
92.0
91.1
85.5
92.6
75.8
T5SCORE-XLL
73.0
99.5
1.0
95.9
99.1
51.3
94.0
92.0
87.9
91.5
78.5
en-cs
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-zh
Avg-x
Avg
COMET EC
98.9
90.1
86.3
95.2
96.9
80.0
88.8
97.5
91.7
85.2
T5SCORE-Bss
95.8
97.3
31.5
96.8
93.5
79.4
91.7
93.7
85.0
80.1
T5SCORE-Lae
98.2
96.6
27.4
96.8
96.2
84.9
94.9
95.7
86.3
80.5
15SCORE-XLL
98.5
96.3
22.4
96.4
96.8
88.7
95.2
96.0
86.3
82.0
Table 15: System level Kendall's 's Tau correlations on WMT DA20 corpus using source based models.
cs-en
de-en
ju-en
ja-en
km-en
pl-en
ps-en
ru-en
la-en
zh-en
Avg-en
COMET
69.7
78.8
52.7
77.8
90.5
29.7
73.3
45.5
51.6
55.0
62.5
T5SCORE-Be:
66.7
66.7
-9.1
64.4
61.9
31.9
86.7
56.4
60.4
65.0
55.1
T5SCORE-Las
72.7
78.8
-12.7
82.2
81.0
31,9
52.7
56.0
68.3
58.4
T5SCORE-XL..:
72.7
78.8
5.5
86.7
90.5
29.7
73.3
52.7
64.8
68.3
62.3
en-cs
en-de
en-iu
en-ja
en-pl
en-TU
en-ta
en-zh
Avg-x
Avg
COMFT
84.8
80.2
60.0
70.9
80.2
66.7
56.2
84.8
73.0
67.1
T5SCORE-B.r
84.8
78.0
12.7
81.8
71.4
61.1
65.7
81,8
67.2
60.5
T5SCORE-Lan
93.9
78.2
71.4
66.7
77.1
81.8
63.0
TSSCORE XLo.
93.9
78.2
66.7
82.9
78.8
65.1
Table Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations for WMT DA20 corpus without unsupervised training using T5SCORE-B.
CS-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
ru-en
ta-en
zh-en
seg-k
14.0
48.4
27.8
28.8
30.7
18.0
16.4
sys-p
82.1
99.2
79.9
97.8
97.7
56.3
94.7
91.8
96.3
sys-k
72.7
75.8
56.4
82.2
90.5
40.7
86.7
80.0
en-de
en-ja
en-pl
en-ru
Avg
seg-k
60.9
43.4
33.4
59.3
39.6
29.3
63.7
48.6
33,9
sys-p
93.9
96.7
64.0
97.2
97.9
91.5
93.9
89.8
sys-k
69.7
86.8
30.9
85.5
69.2
72.2
Table Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations for WMT DA20 corpus without unsupervised training using model T5SCORE-L.
CS-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
ru-en
ta-en
zh-en
seg-k
48.8
30.5
28.9
30.7
9.6
18.4
16.1
sys-p
81.5
99.3
79.8
97.9
97.9
58.2
95.2
96.5
sys-k
72.7
78.8
56.4
77.8
90.5
40.7
86.7
73.3
en-de
en-iu
en-ja
en-pl
en-zh
Avg
seg-k
65.1
44.5
14.8
63.0
44.6
30.5
67.2
51.2
34.3
sys-p
96.1
96.9
54.4
96.8
98.1
93.1
95.6
97.2
89.8
sys-k
90.9
86.8
20.0
85.5
71.4
72.2
79.0
87.9
71.7

---

Table 18: Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations for WMT DA20 corpus with unsupervised training using model T5SCORE-B.
CS-Cn
de-cn
iu-en
ja-on
km-cn
pl-cn
ps-en
ru-ch
ta-en
zh-en
seg-k
13.9
48.5
29.2
28.1
30.3
9.6
17.4
13.1
23.9
15.5
sys-p
81.9
99.4
78.9
98.2
97.8
57.4
95.1
91.9
88.1
96.4
sys-k
72.7
75.8
56.4
82.2
90.5
42,9
86.7
56.4
60.4
83.3
en-es
en-de
en-iu
en-ja
en-pl
en-ru
en-zh
Avg
seg-k 61.2
43.1
32.4
60.5
38.9
28.9
64.1
sys-p 93.7
97.2
67.9
98.2
97.8
92.9
93.7
sys-k 75.8
86.8
34.5
85.5
69.2
72.2
77.1
Table 19: Segment Kendall's Tau, system Pearson and system Kendall's Tau correlations for WMT DA20 corpus with unsupervised training using TSSCORE-L.
CS-en
de-en
iu-en
ja-en
km-en
pl-en
ps-en
ru-en
ta-en
zh-en
seg-k
14.0
49.3
28.5
28.9
30.1
8.3
17.6
15.3
25.9
16.3
sys-p
80.4
99.3
77.8
98.0
98.6
59.4
95.4
92.2
89.6
96.6
sys-k
75.8
78.8
56.4
77.8
100.0
42.9
86.7
52.7
60.4
71.7
en-es
en-de
en-iu
en-ja
en-pl
en-ru
en-ta
en-zh
Avg
seg-k 66.6
45.6
33.4
62.8
43.9
31.8
66.7
52.2
35.4
sys-p sys-k
96.4
97.3
67.4
97.4
98.0
94.8
95.9
97.8
90.7
90.9
89.0
30.9
85.5
69.2
72.2
79.0
84.8
72.5
Table 20: Segment level Kendall's Tau correlation on MultiSumm corpus. The highest correlation is bold
Focus
Coverage
EN
ID
FR
TR
ZH
RU
DE
ES
AVG
EN
ID
FR
TR
ZH
RU
DE
ES
AVG
ROUGE1
49.0
48.9
48.1
39.6
48.5
7.5
47.1
36,8
40.7
52.9
50.5
48.5
32.0
47.5
9.8
8.9
51.6
37.7
ROUGE2
34.7
51.1
36.5
35.4
50.5
-67.7
35.6
28.4
25.6
52.9
41.9
34.0
30.0
43.4
-66.7
1.3
20.9
19.7
ROUGEL
44.9
46.7
48.1
37.5
48.5
5.4
49.4
38.9
39.9
54.9
46.2
44.3
30.0
49.5
7.8
11.4
42.9
35.9
COMET
36.7
55.6
44.2
33.3
48.5
39.8
60.9
43.2
45.3
41.2
48.4
25.8
32.0
47.5
29.4
31.6
51.6
38.4
BERTScore
34.7 51.1
48.1
50.0
36.6
7.5
63.2
43.2
41.8
49.0
52.7
44.3
38.0
41.4
29.4
31.6
49.5
34.6
BLEURT
40.8 51.1
55.8
43.8
50.5
57.0
47.1
49.5
49.4
45.1
46.2
42.3
34.0
45.5
70.6
21.5
60.4
45.7
PRISM
32.7
48.9
57.7
25.0
46.5
61.3
54.0
43.2
46.2
45.1
59.1
34.0
38.0
59.6
54.9
31.6
53.8
47.0
BARTScore
32.7
55.6
63.5
41.7
54.5
41.9
67.8
49.5
50.9
41.2
44.1
44.3
36.0
49.5
$2.9
34,2
60.4
45.3
TSSCORE- Bus
42.9 51.1
69.2
43.8
48.5
52.7
67.8
43.2
$2.4
51.0
52.7
44.3
30.0
47.5
52.9
26.6
60.4
45.7
TSSCORE-La.
26.5
60.0
67.3
39.6
56.4
61.3
63.2
45.3
52.5
33.3
59.1
52.6
28.0
59.6
64.7
21.5
60.4
47.4
T5SCORE-XL...
24.5
55.6
59.6
39.6
54.5
57.0
65.5
49.5
50.7
33.3
63.4
44.3
26.0
49.5
68.6
19.0
67.0
46.4
TSSCORE-BA Bug
46.9
44.4
38.5
35.4
42.6
-3,2
54.0
30.5
36.1
58.8
41.9
32.0
16.0
39.4
-37.3
16.5
38.5
25.7
T5SCORE-Lom
38.8
60.0
42.3
37.5
42.6
-14.0
42.5
32.6
35.3
54.9
52.7
32.0
26.0
37.4
-1.2
13.9
42.9
27.3
T5SCORE-XI
24.5
60.0
42.3
50.0
48.5
14.0
65.5
41.1
43.2
52.9
61.3
15.5 32.0 37.4
-788
19.0
49.5
32.5
Table 21: Segment level Pearson correlation on MultiSumm corpus.
Focus
Coverage
EN
ID
FR
TR
7H
RU
DE
ES
AVG
EN
ID
FR
TR
7H
RU
DE
ES
AVG
ROUGE1
$9.0
70.2
68.7
81.0
82.6
52.0
87.3
60.1
70.1
62.3
70.5
66.2
75.6
77.9
46.6
88,8
63.7
68.7
ROUGE3
53.6
62.8
68.0
77.8
78.7
51.6
85.8
61.2
68.1
53.5
65.1
67.3
73.6
73.6
47.2
88.2
64.9
66.9
ROUGEL
58.2 69.3
68.8
80.0
81.6
51.3
86.8
61.0
69.7
61.0
70.7
66.3
75.5
78.1
46.2
88.3
64.7
68.6
COMET
50.6 67.6
53.1
72.6
73.1
49.9
84.8
55.6
62.8
50.1
68.7
50.2
76.6
70.8
49.9
80.1
62.1
62.8
BERTScore
58.5 69.8
71.8
83.2
77.7
49.4
89.6
58.0
69.7
61.7
71.6
70.2
80.0
75.9
39.0
89.4
66.4
68.9
BLEURT
52.4 61.1
70.5
79.0
72.6
55.8
87.4
66.0
69.1
59.5
61.7
71.4
79.0
75.3
58.7
87.8
65.2
71.0
PRISM
59.3 59.5
68.1
78.9
70.8
42.5
87.0
53.9
65.8
61.2
60.9
64.4
78.3
69.3
45.5
89.0
60.0
66.8
BARTScore
61.1 69.0
72.0
80.3
75.6
54.6
89.3
62.5
70.8
59.9
67.7
68.9
80.1
70.9
57.7
90.7
66.3
70.7
T5SCORE-B
59.0
64.9
70.8
79.6
75.6
50.0
88.7
58.5
68.9
59.0
63.9
68.2
79.4
71.5
54.9
90.1
65.7
69.8
T5SCORE-Lm
58.5
68.6
71.1
80.6
76.4
49,3
89,1
59.1
69.2
58.2
68.5
70.1
81.4
73.0
55.8
90.4
66.4
70.8
T5SCORE-XI -I
57.8
69.9
70.2
80.2
77.0
50.2
88.9
59.7
69.1
58.0
70.4
70.2
81.3
72.9
54.7
90.0
65.7
70.4
T5SCORE-B Buup
55.7
63.5
58.9
70.6
71.9
26.6
84.4
42.8
58.7
51.1
58.9
49.8
66.3
69.5
23.0
84,6
46.3
55.8
T5SCORE-Lam
56.7
68.8
59.2
71.8
70.0
18.1
83.1
47.1
58.0
50.6
65.3
50.0
66.6
69.4
12.3
82.3
45.7
53.8
T5SCORE-XI.
54.2
66.7
63.0
77.0
73.2
32.9
87.7
50.2
62.6
53.8
67.8
54.1
74.4
70.7
34.4
88.1
55.7
61.6

---

generated by systems: Pointer-Generator (See al., 2017) and BERT (Liu and Lapata, 2019; Dong al., 2019) model.
A.9.2 Parallel Dataset
ParaCotta (Aji et al., 2022) A synthetic parallel paraphrase corpus across 17 languages. We use lex- ical BLEU between paraphrases to filter the dataset and keep the paraphrases with low lexical BLEU, in another word, high lexically diverse." 20
MT-prrssm (Thompson and Post, 2020b) A Ma- chine Translation dataset includes 99.8M training sentences across 39 languages. The data sources are WikiMatrix (Schwenk et al., 2019), Global Voices, 21 EuroParl (Koehn, 2005), SETimes²² United Nations (Eisele and Chen, 2010).
A.10 Models of Baseline Metrics
BLEU We use the implementation of Sacre- BLEU (Post, 2018) which is licensed under the Apache 2.0 License.
ROUGE We use the implementation of py- rouge (https://github. wtw.hubb..com/hheizeengnggg pyrouge) which is licensed under the MIT License.
COMET We use the estimator model wmt20- comet-da trained on WMT DA17 to DA19 un- less otherwise stated in Sec.4. It is licensed under Apache-2.0 license.
BERTScore We use the default multilingual BERT model: bert-bus-mullinngual-ccss It is licensed under the MIT License.
BLEURT We use the model BLEURT-20 trained on human ratings from WMT15 to WMT19. It is licensed under the Apache-2.0 license.
PRISM We use the model trained on machine translation data for 39 language pairs. It is licensed under the MIT License.
The original version only supports English, generalize it to multilingual ver- sion BARTScore by using mBART (Liu et al., 2020) and fine-tune on ParaCotta. is licensed under the Apache-2.0 license.
BARTScore
"We keep the paraphrases with lexical BLEU in the range [0,20]. while the possible range [0,100]. 21 https://cssmaaatt wwtpaa///o/pp////aaaooiiicc..... "hup://ipp.fiz hww.://gh.....ooooooocoosoossene


Reviews:
[{"review_id": "pdzIhSjPUE", "reviewer": "Reviewer_zxyM", "summary": "", "questions": "QA: In Finding (1) of 6.2 --> Top-k Analysis, you state that the advantage of the source-based version of T5Score over the reference-based version increases as you evaluate fewer systems. Looking at Figure 6, I understand \"fewer systems\" to mean small k --> left part of x-axis. It seems from Figure 6 that Finding (1) only holds true for the first (leftmost) subfigure in Figure 6--seg-k?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We appreciate the time and effort you have invested in providing this feedback. We will address your concerns and make the necessary revisions to the paper accordingly.\n\nFor your question about the clarity of Section 6.2's Finding (1) regarding the Top-k Analysis. We appreciate the opportunity to clarify this point.\n\nFirstly, we acknowledge that our observation serves more as a general trend rather than a stringent claim. When comparing T5-XL-src (depicted by the blue line) with T5-XL-ref (depicted by the red line), you are correct that the trend—\"source-based has more advantages over reference-based when evaluating fewer systems\"—is most clearly seen in the leftmost subfigure of Figure 6, focusing on seg-k.\nWhile not as conspicuous, this trend does persist in the middle and rightmost subfigures as well. In the middle subfigure, the gap between T5-XL-src and T5-XL-ref is relatively larger on the left part, which evaluates 8-11 systems, and narrower on the right part, which evaluates 12-15 systems. In the rightmost subfigure, T5-XL-ref gradually surpasses T5-XL-src as more systems are evaluated. We agree that this trend is not universally consistent across all k-values (e.g. the gap of T5-XL-src and T5-XL-ref for the fewest 8 systems is not always the largest among all), but it is present to varying degrees.\n\nAs for why this trend is most pronounced in the first subfigure focusing on segment-level measurement, one plausible explanation is that system-level measurements are generally more robust when considering a larger number of systems. Therefore, reducing the system number could potentially make the measurements less robust compared to segment-level metrics.\n\nRegarding COMET-src and COMET-ref, the trend is indeed less obvious. As we noted in the main text, COMET-ref employs both source and reference information, making it a different case from T5Score. Our primary claim is thus primarily based on observations from the T5Score model.\n\nWe hope this clarifies any confusion, and we appreciate your thorough review which helps us improve the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Motivation is clear (we have discriminative and generative text generation methods; why not take advantage of both approaches?)\n- Proposed solution, the T5Score metric is simple but shown to be effective.\n- Authors released code and models for community use.\n", "reasons_to_reject": "Confusion about Claim (1) in 6.2 -> Top-k analysis (See Question A)", "questions_to_authors": "QA: In Finding (1) of 6.2 --> Top-k Analysis, you state that the advantage of the source-based version of T5Score over the reference-based version increases as you evaluate fewer systems. Looking at Figure 6, I understand \"fewer systems\" to mean small k --> left part of x-axis. It seems from Figure 6 that Finding (1) only holds true for the first (leftmost) subfigure in Figure 6--seg-k?", "paper_topic_and_main_contributions": "The authors propose a novel text generation metric (T5Score) trained on the signals used by both generative and discriminative metrics. They detail the generative and discriminative training procedures for T5Score, and show that it outperforms leading metrics on segment-level outputs.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "outperforms --> outperform in Line 393"}, "extra_text": {}}, {"review_id": "9xcGhu6IlT", "reviewer": "Reviewer_NgGN", "summary": "", "questions": "1) Can the authors please refer to previous section (a) and (b) and address these? I would be willing to update my scores if these concerns are addressed.\n2) What was the checkpoint used for BLEURT? Was it BLEURT-20?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We appreciate the time and effort you have invested in providing this feedback. We will address your concerns and make the necessary revisions to the paper accordingly.\n\nConcerning a) the minor gains in correlations as illustrated in Figure 4:\nWe acknowledge that Figure 4 may not clearly convey the performance improvements due to its presentation of variables (seg-k, sys-p, sys-k) with differing ranges. To provide a clearer understanding, please refer to Appendix A.7, Tables 16-19, which include detailed statistics: For T5Score-Base, the average seg-k across 18 languages changes minimally from 33.9 to 33.7 with unsupervised generative training. However, sys-p improves from 89.8 to 90.2, and sys-k from 71 to 71.8. In contrast, T5Score-L sees more robust improvements across all three metrics. The average seg-k improves from 34.3 to 35.4, sys-p from 89.8 to 90.7, and sys-k from 71.7 to 72.5.\nOne possible explanation is that our generative training process is analogous to pre-training, and larger models like T5Score-L benefit more from such training. These models are better at retaining knowledge gained during generative training, whereas smaller models like T5Score-Base may not realize significant gains due to their lower capacity, potentially leading them to \"forget\" some knowledge.\n\n\nIn response to point b):\nYour question raises a valid concern about the fairness of comparing models with different parameter counts. However, there are several aspects that, when considered, make the comparison between T5Score and the other models fair.\n\n1. Architectural Differences: We utilize T5, an encoder-decoder model, whereas other leading models like COMET and BLEURT utilize encoder-only architectures. According to the T5 paper, an \"encoder-decoder model with L + L layers has roughly the same computational cost as a language model with only L layers.\" Therefore, from a computational cost perspective, the comparison remains fair between an encoder-only model with an encoder-decoder model with 2x parameters despite the parameter difference.\n\n2. Smaller Variant for Comparison: We also present results using a smaller variant, mT5-L (1.2B parameters), which is more comparable to baseline methods with ~500M parameters. mT5-L outperforms these baselines in most experimental settings, strengthening our argument for the effectiveness of our approach.\n\nFor example:\n- In the reference-based MT task using the MQM dataset (Table 2), mT5-L significantly outperforms BLEURT and COMET across multiple correlation metrics.\n- For the reference-based summarization task (Figure 3), mT5-L surpasses all baselines.\n- In the source-based task on the QE20 dataset (Table 4), mT5-L also exceeds the performance of baseline methods.\n\nRegarding Question 2, we used BLEURT-20 for our experiments, which was trained on human ratings spanning from WMT15 to WMT19. This information can be found in Appendix A.10 of our paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper is well written and is extremely structured in its approach and the experimental design. It supports its claims with ample set of experiments and results. The gains from this metric are reflected in all the 5 datasets that cover 19 languages and, in both system, and segment level settings.\n\nExcept in few instances, they explore and demonstrate gains on wide variety of tasks that include gains on MQM and DA settings. It was also interesting to see the unsupervised approach performing better than many of the existing metrics in Table 1. They do demonstrate a promising approach of combining generative and discriminative approaches that could be pursued further. Overall, I feel the paper does a good job and would love to see the paper in conference this year.\n", "reasons_to_reject": "While the approach does seem to be promising, I do have some concerns and queries and it would be helpful if the authors could help resolve them:\na) The gains in the correlations that is demonstrated from generative training in Figure 4 does not seem to be huge. It would be helpful if the authors could help me understand why the gains are minor and even in case of the T5 Score-B using segment level Kendel Tau's correlation shows a negative gain which has not been addressed properly. Can the authors suggest some explanation to this?\n\nb) I also have a question in terms of fair comparison. The model mT5-XL that outperforms the existing metric has 3.7 billion parameters and most of the other metrics have around ~500 million params. Is the comparison between existing metrics and mT5-XL fair for stating that it outperforms the existing metrics on all datasets?\n\nAnother factor that I felt to be missing but maybe out of scope of present paper is that there is not enough clarity on breakdown what accuracy, locale or style actually constitutes in MQM section. It would have been much more interesting to see how this metric performs on the different controlled linguistic perturbations. Authors could have used an existing curated dataset specifically designed from these tasks like BLIMP or DEMETR etc. to understand the granular cases where metrics work good or not. ", "questions_to_authors": "1) Can the authors please refer to previous section (a) and (b) and address these? I would be willing to update my scores if these concerns are addressed.\n2) What was the checkpoint used for BLEURT? Was it BLEURT-20?", "paper_topic_and_main_contributions": "While generative based metrics exist (PRISM), combining it with discriminative approach might help better differentiate between a good and a bad MT translation. The paper's work might lead to more research growth in this direction of combining the generative as well as discriminative approaches in addition to the metrics development. Also, the new proposed metric T5 score could potentially be helpful in propelling field of Machine Translation and developing better MT systems in general. \n\n", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "None "}, "extra_text": {"Missing_References": "I believe authors missed out on one section of work that related to the metrics evaluation in the \"related works\" section and given they have space; I think they should ideally include these works in the discussion and references:\n(1) https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00321/96452/BLiMP-The-Benchmark-of-Linguistic-Minimal-Pairs\n(2) https://arxiv.org/abs/2210.13746\n(3) https://arxiv.org/pdf/2305.11806.pdf\n(4) https://aclanthology.org/2023.acl-long.795/\n\n"}}, {"review_id": "DzSBB3svfU", "reviewer": "Reviewer_SSiu", "summary": "", "questions": "1. Is there one separate model for every task and language? If so, I think the potential of mT5 is not fully utilized and the usage of T5SCORE is limited.\n2. In 3.3, how to find hypothesis with higher and lower ratings? Are there many2one data pairs in the original dataset for all supervised datasets?\n3. Why the author use different model in different settings? In Figure2, T5SCORE-XL is used while in figure 3, T5SCORE-L is used.\n4. Why the author didn't use the newest MQM dataset, like MQM-2022?\n5. The author should also evaluate some diverse generation task performance to validate the performance of the proposed metric instead of only MT and Summarization.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review and valuable suggestions. We appreciate the time and effort you have invested in providing this feedback. We will address your concerns and make the necessary revisions to the paper accordingly.\n\nFor the concerns:\n\n1.We appreciate your observation regarding the lack of technique novelty in the individual components of our method. Our main contribution and novelty is indeed not in proposing the generative or discriminative training approach per se, but in presenting a comprehensive framework that synergistically combines these two approaches. This integration enables us to leverage both supervised and unsupervised signals, including parallel data and human annotations, to better understand the characteristics of high-quality generations and to differentiate between superior and inferior hypotheses. While we acknowledge that the individual components, such as contrastive learning, are not new, our work demonstrates the benefits of utilizing different kinds of signals in a unified framework, which, to the best of our knowledge, has not been studied in previous work. In particular, SESCORE2 emphasizes self-supervised training without using human-annotated ratings, while our main focus is on utilizing as many possible learning signals as possible. We will ensure to cite and compare our work with SESCORE2 in the revised version of our paper. \n\n2.Thank you for pointing out the absence of a comparison with newer methods like SESCORE and SESCORE2, but we also kindly remind that the original version of this draft was released earlier than SESCORE2, making a direct comparison challenging at the time. Nevertheless, we appreciate the importance of benchmarking against recent advancements and will incorporate comparative analyses with these methods in our revised submission.\n\t\t\t\t\t\t\n3.Thank you for your observation regarding the mixed performance of various metrics in our experiments. We appreciate the opportunity to clarify this aspect of our work.\nGiven the comprehensive nature of our experiments—which span five datasets, 19 languages, and employ three different correlation metrics at both the segment and system levels—it is indeed challenging for a single metric to uniformly outperform others across all settings. We believe that examining the average performance across various configurations can offer a more balanced view of each metric's capabilities. For example, the average score across all language pairs in the WMT DA task (as presented in Table 1) and the average score over different years of data (as shown in Table 2) can serve as more stable indicators of system performance.\nIt's important to note that our research does not claim that T5SCORE is universally superior. Rather, we carefully delineate the conditions and settings under which T5SCORE shows advantages. Thus, we concur with your observation that no single metric 'wins' in all circumstances; our paper aims to highlight under what specific conditions T5SCORE has shown to be more effective.\n\n\nFor the Questions:\n\n1.To clarify, while we do employ distinct models for each task, we utilize a single model for all languages within each task. This approach aligns with the domain-specific nature of the tasks under consideration.\n\nWe believe that using different models for each task is justified for several reasons. First, different tasks often require tailored datasets to optimize performance. For instance, our model trained on MQM-21 integrates additional data sets like DA-20 and MQM-20, specifically to enhance its performance on MQM-20.\n\nSecond, our comparison metrics are also in line with this approach. The baseline models we use for comparison purposes, such as COMET-20 and COMET-21-MQM, also come in multiple versions optimized for different tasks. We align our models with the respective best-performing version of each baseline model to ensure a fair and accurate comparison.\n\nLastly, our research setting varies depending on whether we are evaluating reference-based or source-based metrics. Given these varying conditions, it would be inconsistent to use a one-size-fits-all model across all tasks and metrics.\n\n2.We employ two strategies for identifying hypotheses with higher and lower ratings, which depend on the dataset in question.\n\nFor datasets like MQM and WMT-DA, each source input is associated with multiple hypotheses that have different human-assigned ratings. This many-to-one relationship between source inputs and hypotheses makes it straightforward to identify better and worse hypotheses for a given source.\n\nFor datasets like QE, each source input corresponds to exactly one hypothesis with an associated human rating, forming one-to-one pairs like (s1,h1,score1), (s2,h2,score2), etc. In this case, we randomly select two pairs and construct \"better\" and \"worse\" hypotheses based on their scores. It's worth noting that these pairs have different source inputs. Our model structure is designed to be flexible and does not strictly require that the compared hypotheses stem from the same source input.\n\n3.Thank you for pointing out the different models used in figures 2 and 3. We developed three variations of the model: T5Score-B, T5Score-L, and T5Score-XL for our research. For each setting, we tested all three models and included the results of all of them in the appendix. However, for the main body of the paper, we opted to present the results of the best-performing model for each specific setting to make the paper more concise and easier to follow.We encourage readers interested in the performance of all models to refer to the appendix for complete results.\n\n4.The experiments conducted for this paper were primarily carried out in 2022. At that time, the MQM-2022 dataset had not yet been released, making it unavailable for inclusion in our study.\n\n5.We appreciate your suggestion to extend our evaluation to more diverse generation tasks. While we agree that such an extension would provide a more comprehensive validation, we were constrained by space limitations in this paper. Additionally, our work already includes a variety of experimental settings—such as multiple languages, source-based and reference-based evaluations, and both supervised and unsupervised training frameworks. Given these constraints, we chose to focus on Machine Translation and Summarization as they are representative and widely-studied tasks in the NLP community. We will certainly consider expanding the scope of tasks in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "T5SCORE can evaluate both generation task with or without reference. Due to the fact that it's constructed on mT5, it can evaluate generated texts in different languages.", "reasons_to_reject": "1. There is no technique novelty in this paper. For the generative metric, it just learns a distribution like traditional unsupervised task. For discriminative metric, contrastive learning is not new. For example, like SESCORE2: Learning Text Generation Evaluation via Synthesizing\nRealistic Mistakes, which is also a text generation evaluation metric and used contrastive learning. The author should also cite and compare with this work.\n2. The proposed method didn't compare with some new methods like SESCORE, SESCORE2.\n3. The experimental results exhibit mixed performance. For example, in table2, BARTScore, COMET, BLEURT also perform best on some tasks, while T5SCORE with different settings perform best on different tasks, such as T5SCORE-B on en-de (sys-k, sys-p) and T5SCORE-L on en-de (seg-k).", "questions_to_authors": "1. Is there one separate model for every task and language? If so, I think the potential of mT5 is not fully utilized and the usage of T5SCORE is limited.\n2. In 3.3, how to find hypothesis with higher and lower ratings? Are there many2one data pairs in the original dataset for all supervised datasets?\n3. Why the author use different model in different settings? In Figure2, T5SCORE-XL is used while in figure 3, T5SCORE-L is used.\n4. Why the author didn't use the newest MQM dataset, like MQM-2022?\n5. The author should also evaluate some diverse generation task performance to validate the performance of the proposed metric instead of only MT and Summarization.", "paper_topic_and_main_contributions": "This paper proposes T5SCORE, a text generation evaluation metric. T5SCORE ensembles a discriminative one, which is trained on unsupervised data using contrastive learning, and a generative one, which learns a conditional distribution. The proposed metric is evaluated on machine translation task and summarization task.", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "Not All Errors Are Equal: Learning Text Generation Metrics using Stratified Error Synthesis. Xu et al. ACL 2021.\nSESCORE2: Learning Text Generation Evaluation via Synthesizing Realistic Mistakes. Xu et al. ACL 2022."}}]

Author rebuttal:

Meta-review: This paper describes a new text generation metric (T5Score) trained on the signals used by both generative and discriminative metrics. The motivation is clear. The main objective of this framework is to learn evaluation metrics based on the assumption that generative and discriminative objectives can work in concert to train a better evaluator (having the best of the two worlds).
The paper is well written. 
This new framework is evaluated on 5 datasets and tasks and the new metric is compared to a various of existing metrics (BLEU, ROUGE, COMET, BERTScore, BLEURT, PRISM and BARTScore). These experiments show that this new metric outperforms leading metrics on segment-level outputs.
Decision: Accept-Findings