MT Metrics Correlate with Human Ratings of Simultaneous Speech Translation
Year: 2023

Abstract:
There have been several meta-evaluation studies on the correlation between human ratings and offline machine translation (MT) evaluation metrics such as BLEU, chrF2, BERTSCORE and COMET. These metrics have been used to evaluate simultaneous speech translation (SST) but their correlations with human ratings of SST, which has been recently collected as Continuous Ratings (CR), are unclear. In this paper, we leverage the evaluations of candidate systems submitted to the EnglishGerman SST task at IWSLT 2022 and conduct an extensive correlation analysis of CR and the aforementioned metrics. Our study reveals that the offline metrics are well correlated with CR and can be reliably used for evaluating machine translation in simultaneous mode, with some limitations on the test set size. We conclude that given the current quality levels of SST, these metrics can be used as proxies for CR, alleviating the need for large scale human evaluation. Additionally, we observe that correlations of the metrics with translation as a reference is significantly higher than with simultaneous interpreting, and thus we recommend the former for reliable evaluation.

heading: 1 Introduction
text: The current approach to evaluate simultaneous speech translation (SST, Cho and Esipova, 2016; Ma et al., 2019) systems that have text as the output modality is to use automatic metrics which are designed for offline text-to-text machine translation (MT), alongside other measures for latency and stability. Researchers tend to use offline metrics, such as BLEU (Papineni et al., 2002), chrF2 (Popović, 2017), BERTSCORE (Zhang et al., 2020), COMET (Rei et al., 2020) and others (Freitag et al., 2022) in SST despite no explicit evidence that they correlate with human ratings.
However, simultaneous speech-to-text translation has different characteristics compared to offline text-to-text MT. For example, when the users
are following subtitles in real-time, they have limited time for reading and comprehension as they cannot fully control the reading pace by themselves. Therefore, they may be less sensitive to subtle grammar and factual flaws than while reading a text document without any time constraints. The human evaluation of SST should therefore reflect the simultaneity. The users may also prefer brevity and simplicity over verbatim word-for-word translation. Even if the reference is brief and simpler than the original, there may be lots of variants that the BLEU score and other MT metrics may not evaluate as correct.
Furthermore, SST and MT differ in their input modalities. MT sources are assumed to originate as texts, while the SST source is a speech given in a certain situation, accompanied by para-linguistic means and specific context knowledge shared by the speaker and listener. Transcribing speech to text for use in offline evaluation of SST may be limiting.
In this paper, we aim to determine the suitability of automatic metrics for evaluating SST. To this end, we analyze the results of the simultaneous speech translation task from English to German at IWSLT 2022 (Anastasopoulos et al., 2022), where we calculate the correlations between MT metrics and human judgements in simultaneous mode. There are five competing systems and human interpreting that are manually rated by bilingual judges in a simulated real-time event. Our studies show that BLEU does indeed correlate with human judgements of simultaneous translations under the same conditions as in offline text-to-text MT: on a sufficiently large number of sentences. Furthermore, chrF2, BERTSCORE and COMET exhibit similar but significantly larger correlations. To the best of our knowledge, we are the first to explicitly establish the correlation between automatic offline metrics with human SST ratings, indicating that they may be safely used in SST evaluation in
169
the currently achieved translation quality levels. Additionally, we statistically compare the metrics with translation versus interpreting reference, and we recommend the most correlating one: translation reference and COMET metric, with BERTSCORE and chrF2 as fallback options.
We publish the code for analysis and visualisations that we created in this study.1 It is available for further analysis and future work.

heading: 2 Related Work
text: We replicate the approach from text-to-text MT research (e.g. Papineni et al., 2002) that examined the correlation of MT metrics with human judgements. The strong correlation is used as the basis for taking the metrics as reliable. As far as we know, we are the first who apply this approach to SST evaluation in simultaneous mode.
In this paper, we analyze four metrics that represent the currently used or recommended (Freitag et al., 2022) types of MT metrics. BLEU and chrF2 are based on lexical overlap and are available for any language. BERTSCORE (Zhang et al., 2020) is based on embedding similarity of a pre-trained BERT language model. COMET (Rei et al., 2020) is a neural metric trained to estimate the style of human evaluation called Direct Assessment (Graham et al., 2015). COMET requires sentence-tosentence aligned source, translation and reference in the form of texts, which may be unavailable in some SST use-cases; then, other metric types may be useful. Another fact is that BERTSCORE and COMET are available only for a limited set of languages.

heading: 3 Human Ratings in SST
text: As far as we know, the only publicly available collection of simultaneous (not offline) human evaluation of SST originates from IWSLT 2022 (Salesky et al., 2022) English-to-German Simultaneous Translation Task, which is described in “Findings” (Anastasopoulos et al., 2022, see highlights of it we discuss in Appendix A). The task focused on speech-to-text translation and was reduced to translation of individual sentences. The segmentation of the source audio to sentences was provided by organizers, and not by the systems themselves. The source sentence segmentation that was used in human evaluation was gold (oracle). It only approximates a realistic setup where the
1github.com/ufal/MT-metrics-in-SimST
segmentation would be provided by an automatic system, e.g. Tsiamas et al. (2022), and may be partially incorrect and cause more translation errors compared to the gold segmentation.
The simultaneous mode in Simultaneous Translation Task means that the source is provided gradually, one audio chunk at a time. After receiving each chunk, the system decides to either wait for more source context, or produce target tokens. Once the target tokens are generated, they can not be rewritten.
The participating systems are submitted and studied in three latency regimes: low, medium and high. It means that the maximum Average Lagging (Ma et al., 2019) between the source and target on validation set must be 1, 2 or 4 seconds in a “computationally unaware” simulation where the time spent by computation, and not by waiting for context, is not counted. One system in low latency did not pass the latency constraints (see Findings, page 44, numbered 141), but it is manually evaluated regardless.
Computationally unaware latency was one of the main criteria in IWSLT 2022. It means that the participants did not need to focus on a low latency implementation, as it is more of a technical and hardware issue than a research task. However, the subtitle timing in manual evaluation was created in a way such that waiting for the first target token was dropped, and then it continued with computationally aware latency.

heading: 3.1 Continuous Rating (CR)
text: Continuous Rating (CR, Javorský et al., 2022; Macháček and Bojar, 2020) is a method for human assessment of SST quality in a simulated online event. An evaluator with knowledge of the source and target languages watches a video (or listens to an audio) document with subtitles created by the SST system which is being evaluated. The evaluator is asked to continuously rate the quality of the translation by pressing buttons with values 1 (the worst) to 4 (the best). Each evaluator can see every document only once, to ensure one-pass access to the documents, as in a realistic setup.
CR is analogous to Direct Assessment (Graham et al., 2015), which is a method of human text-totext MT evaluation in which a bilingual evaluator expresses the MT quality by a number on a scale. It is natural that individual evaluators have different opinions, and thus it is a common practice to
have multiple evaluators evaluate the same outputs and then report the mean and standard deviation of evaluation scores, or the results of statistical significance tests that compare the pairs of candidate systems and show how confident the results are.
Javorský et al. (2022) showed that CR relates well to comprehension of foreign language documents by SST users. Using CR alleviates the need to evaluate comprehension by factual questionnaires that are difficult to prepare, collect and evaluate. Furthermore, Javorský et al. (2022) show that bilingual evaluators are reliable.
Criteria of CR In IWSLT 2022, the evaluators were instructed that the primary criterion in CR should be meaning preservation (or adequacy), and other aspects such as fluency should be secondary. The instructions do not mention readability due to output segmentation frequency or verbalizing nonlinguistic sounds such as “laughter”, despite the system candidates differ in these aspects.

heading: 3.2 Candidate Systems
text: Automatic SST systems There are 5 evaluated SST systems: FBK (Gaido et al., 2022), NAIST (Fukuda et al., 2022), UPV (Iranzo-Sánchez et al., 2022), HW-TSC (Wang et al., 2022), and CUNIKIT (Polák et al., 2022).
Human Interpreting In order to compare the state-of-the-art SST with human reference, the organizers hired one expert human interpreter to simultaneously interpret all the test documents. Then, they employed annotators to transcribe the voice into texts. The annotators worked in offline mode. The transcripts were then formed as subtitles including the original interpreter’s timing and were used in CR evaluation the same way as SST. However, human interpreters use their own segmentation to translation units so that they often do not translate one source sentence as one target sentence. There is no gold alignment of the translation sentences to interpreting chunks. The alignment has to be resolved before applying metrics to interpreting.

heading: 3.3 Evaluation Data
text: There are two subsets of evaluation data used in IWSLT22 En-De Simultaneous Translation task. The “Common” subset consists of TED talks of the native speakers.See the description in Findings on page 9 (numbered as 106). The “Non-Native” subset consists of mock business presentations of European high school students (Macháček et al.,
2019), and of presentations by representatives of European supreme audit institutions. This subset is described in Findings on page 39 (numbered page 136). The duration statistics of audio documents in both test sets are in Findings in Table 17 on page 48 (numbered 145).

heading: 4 Correlation of CR and MT Metrics
text: In this section, we study the correlation of CR and MT metrics BLEU, chrF2, BERTSCORE and COMET. We measure it on the level of documents, and not on the test set level, increasing the number of observations for significance tests. There are 60 evaluated documents (17 in the Common subset and 43 in Non-Native) and 15 system candidates (5 systems, each in 3 latency regimes), which yields 900 data points.
We discovered that CUNI-KIT system outputs are tokenized, while the others are detokenized. Therefore, we first detokenized CUNI-KIT outputs. Then, we removed the final end of sequence token (
) from the outputs of all systems. Finally, we calculated BLEU and chrF2 using sacreBLEU (Post, 2018), BERTSCORE and COMET. See Appendix B for metric details and signatures.
In total, there are 1584 rating sessions of 900 candidate document translations. Each candidate document translation is rated either twice with different evaluators, once, or not at all. We aggregate the individual rating clicks in each rating session by plain average (CR definition in Appendix C) to get the CR scores. Then, we average the CR of the same documents and candidate translations, and we correlate it with MT metrics.
Correlation Results In Table 1, we report correlation coefficients with and without averaging, together with the number of observations. Figure 1 displays the relation between CR and COMET.
Pearson correlation is considered as strong if the coefficient is larger than 0.6 (Evans, 1996). The results show strong correlation (above 0.65) of CR with BLEU, chrF2, BERTSCORE and COMET at the document level on both test subsets. When we consider only one subset, the correlation is lower, but still strong for chrF2, BERTSCORE and COMET (0.63, 0.68 and 0.76, resp.). It is because the Common subset is generally translated better than Non-Native, so with only one subset, the points span a smaller part of the axes and contain a larger proportion of outliers.
The strong correlation is not the case of BLEU on the Common subset where the Pearson coefficient is 0.42. We assume it is because BLEU is designed for use on a larger test set, but we use it on short single documents. However, BLEU correlates with chrF2 and COMET (0.81 and 0.62 on the Common subset). BLEU also correlates with CR on the level of test sets, as reported in Findings in the caption of Table 18 (page 48, numbered 145).
We conclude that with the current overall levels of speech translation quality, BLEU, chrF2, BERTSCORE and COMET can be used for reliable assessment of human judgement of SST quality at least on the level of test sets. chrF2, BERTSCORE and COMET are reliable also at the document level.
Translation vs Interpreting Reference There is an open question whether SST should rather mimic offline translation, or simultaneous interpreting. As
Macháček et al. (2021) discovered, translation may be more faithful, word-for-word, but also more complex to perceive by target audience. Simultaneous interpreting, on the other hand, tends to be brief and simpler than offline translation. However, it may be less fluent and less accurate. Therefore, we consider human translation (TRANSL) and transcript of simultaneous interpreting (INTP) as two possible references, and also test multi-reference metrics with both.
Since interpreting is not sentence-aligned to SST candidate translations, we consider two alignment methods: single sequence (SINGLESEQ), and mWERSegmenter (Matusov et al., 2005, MWER). SINGLESEQ method means that we concatenate all the sentences in the document to one single sequence, and then apply the metric on it, as if it was one sentence. mWERSegmenter is a tool for aligning translation candidates to reference, if their sentence segmentation differs. It finds the alignment with the minimum WER when comparing tokens in aligned segments. For translation, we also apply the default sentence alignment (SENT).
In Table 2, we report the correlations of metric,
reference and alignment variants and their significance, with more details in Appendix D.

heading: 4.1 Recommendations
text: Taking CR as the golden truth of human quality, we make the following recommendations of the most correlating metric, reference and sentence alignment method for SST evaluation.
Which metric? COMET, because it correlates significantly better with CR than BERTSCORE does. From the fall back options, chrF2 should be slightly preferred over BLEU.
Which reference? The metrics give significantly higher correlations with CR with translations than with interpreting as a reference. Difference between translation reference and two references (TRANSL+INTP) is insignificant. Therefore, we recommend translation as a reference for SST.
Which alignment method? With an unaligned reference, COMET and BERTSCORE correlate significantly more with SINGLESEQ than with MWER, probably because the neural metrics are trained on full, complete sentences, which are often split to multiple segments by mWERSegmenter. chrF2 correlates insignificantly better with MWER than with SINGLESEQ.

heading: 5 Conclusion
text: We found correlation of offline MT metrics to human judgements of simultaneous speech translation. The most correlating and thus preferred metric is COMET, followed by BERTSCORE and chrF2. We recommend text translation reference over interpreting, and single sequence alignment for neural, and mWERSegmenter for n-gram metrics.

heading: 6 Limitations
text: The data that we analyzed are limited to only one English-German language pair, 5 SST systems from IWSLT 2022, and three domains. All the systems were trained in the standard supervised fashion on parallel texts. They do not aim to mimic interpretation with shortening, summarization or redundancy reduction, and they do not use document context. The used MT metrics are good for evaluating individual sentence translations and that is an important, but not the only subtask of SST. We assume that some future systems created with a different approach may show divergence of CR and the offline MT metrics.
Furthermore, we used only one example of human interpreting. A precise in-depth study of human interpretations is needed to re-assess the recommendation of translation or interpreting as reference in SST.

heading: Acknowledgements
text: We are thankful to Dávid Javorský and Peter Polák for their reviews.
This research was partially supported by the grants 19-26934X (NEUREM3) of the Czech Science Foundation, SVV project number 260 698, and 398120 of the Grant Agency of Charles University.

heading: A Highlights of IWSLT22 Findings
text: The Findings of IWSLT22 (Anastasopoulos et al., 2022) are available in PDF. The most up-to-date version (version 2) is 61 pages long.2 We highlight the relevant parts of Findings with page numbers in Table 3 so that we can refer to them easily.
Note that findings are a part of the conference proceedings (Salesky et al., 2022) as a chapter in a book. The order of findings pages in PDF does not match the page numbers at the footers.
Also note that in Section 2.4 on page 4 (in PDF, 101 in Proceedings), there is a description of MLLP-VRAIN which corresponds to the system denoted as UPV in all other tables and figures.

heading: B Metric Signatures
text: BLEU and chrF2 SacreBLEU metric signature is case:mixed|eff:no|tok:13a|smooth:exp|version:2.3.1.
For BERTSCORE, we used F1 with signature bert-base-multilingual-cased_L9_no-idf_version=0.3.12(hug_trans=4.23.1)_fast-tokenizer.
We use COMET model wmt20-comet-da (Rei et al., 2020). For multi-reference COMET, we run the model separately with each reference and average the scores.
The standard way of using mWERSegmenter is to segment candidate translation according to reference. However, COMET requires aligned source as one of the inputs, and mWERSegmenter can not align it because it is in other language. For COMET INTP MWER variant, we therefore aligned interpreting to translation, which is already aligned to source. For the other metrics with INTP MWER, we aligned translation candidate to interpreting, which is the standard way.

heading: C Aggregating Continuous Ratings
text: We revisited the processing of the individual collected clicks on the rating buttons into the aggregate score of Continuous Rating.
2https://aclanthology.org/2022.iwslt-1.10v2. pdf
We found two definitions that can yield different results in certain situations: (1) The rating (as clicked by the evaluator) is valid at the instant time point when the evaluator clicked the rating button. The final score is the average of all clicks, each click has the equal weight. We denote this interpretation as CR.
(2) The rating is assigned to the time interval from the click time to the next click, or between the last click and the end of the document. The length of the interval is considered in averaging. The final score is the average of ratings weighted by interval lengths when the rating is valid. We denote this interpretation as CRi. 3
To express them rigorously, let us have a document of duration T , and n ratings (ri, ti), where i ∈ {1, . . . , n} is an index, ri ∈ {1, . . . , 4} is the rated value and 0 ≤ t1 < · · · < tn ≤ T are times when the ratings were recorded.
Then, the definitions are as follows:
CR = 1
n
n∑
i=1
ri
CRi = 1
T − t1
( n−1∑
i=1
(ti+1 − ti)ri + (T − tn)rn )
If the judges press the rating buttons regularly, with a uniform frequency, then both definitions give equal scores. Otherwise, the CR and CRi may differ and may yield even opposite conclusions. For example, pressing “1” twelve times in one minute, then “4” and then waiting for one minute results in different scores: CR = 1.2, CRi = 2.
To examine the relationship between these definitions, we counted CR and CRi for each annotation of each document in the evaluation campaign. The results are in Figure 2 where we observe correlation between the two definitions. The Pearson correlation coefficient is 0.98, which indicates a very strong correlation.
Summary Based on the correlation score we observed, we conclude that both definitions are interchangeable, and any of them can be used in further analysis.
3Other interpretations are also conceivable, for instance assuming that the rating applies to a certain time before the click and then till the next judgement.
marker PDF page numbered page description Section 2 3-5 100-102 Simultaneous Speech Translation Task Figure 1 6 103 Quality-latency trade-off curves Section 2.6.1 5 102 Description of human evaluation Figure 5 8 105 Manual scores vs BLEU (plot) Two Test Sets (paragraph) 39 136 Non-Native subset Test data (paragraph) 9 106 Common (native) subset of test data Automatic Evaluation Results 44 141 Latency and BLEU results (table) A1.1 (appendix) 38-39 135-136 Details on human evaluation
Table 17 48 145 Test subsets duration
Table 3: Relevant parts of IWSLT22 Findings (https://aclanthology.org/2022.iwslt-1.10v2.pdf) for EnDe Simultaneous Speech Translation task and human evaluation.

heading: D Pairwise Metrics Comparison
text: We test the statistical difference of correlations with Steiger’s method.4 The method takes into account the number of data points and the fact that all three compared variables correlate, which is the case of the MT metrics that are applied on the same texts. We use two-tailed test.
We applied the test on all pairs of metric variants. The results for both subsets are in Figure 3. Figure 4 displays results on the Common subset, and Figure 5 for the Non-Native subset. These results are analogous to those in Table 1 in Section 4. The correlation scores for the two subsets treated
4https://github.com/psinger/CorrelationStats/
separately are lower and the differences along the diagonal are less significant. We explain it by the fact that in smaller data set, there is larger impact of noise.

heading: Future Work
text: We assume that some future systems created with a different approach may show divergence of CR and the offline MT metrics. Furthermore, we used only one example of human interpreting. A precise in-depth study of human interpretations is needed to re-assess the recommendation of translation or interpreting as reference in SST