Exploring Context-Aware Evaluation Metrics for Machine Translation
Authors: Xinyu Hu
Xunjian Yin
Xiaojun Wan
Venue: EMNLP 2023 Findings (2023)
Keywords: Machine Translation
Automatic Evaluation Metric
Context Awareness
TL;DR: We introduce context-aware methods for MT evaluation and propose our improved Cont-COMET metric.

Abstract: Previous studies on machine translation evaluation mostly focused on the quality of individual sentences, while overlooking the important role of contextual information. Although WMT Metrics Shared Tasks have introduced context content into the human annotations of translation evaluation since 2019, the relevant metrics and methods still did not take advantage of the corresponding context. In this paper, we propose a context-aware machine translation evaluation metric called Cont-COMET, built upon the effective COMET framework. Our approach simultaneously considers the preceding and subsequent contexts of the sentence to be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce a content selection method to extract and utilize the most relevant information. The experiments and evaluation of Cont-COMET on the official test framework from WMT show improvements in both system-level and segment-level assessments.

Exploring Context-Aware Evaluation Metrics for Machine Translation
Xinyu Hu and Xunjian Yin and Xiaojun Wan
Wangxuan Institute of Computer Technology, Peking University (huxinyu, xjyin, wanxiaojunJ$pku.u edu. cn
Abstract
However, these studies focused primarily on the evaluation of the single sentence, while content of the document that contains it should also be consid- ered. The sentence itself may be semantically am- biguous and thus require surrounding sentences for more proper understanding and evaluation. In fact, some work has improved document-level machine translation by introducing contextual information. Furthermore, the WMT Shared Task (Kocmi al., 2022), which mainly contributes to the annotation data of MT evaluation, has incorporated contextual information into the process of human annotation since 2019. Unfortunately, most existing methods ignored the effect of context and only trained the metrics at the sentence level. Vernikos et al. (2022) proposed additionally encoding previous sentences during the calculation of evaluation scores. Al- though they achieved improvement, their attempts were not comprehensive enough and performed poorly on segment-level correlation.
Previous studies machine translation evalu- ation mostly focused on the quality of individ- ual sentences, while overlooking the important role of contextual information. Although WMT Metries Shared Tasks have introduced context content into the human annotations of transla- tion evaluation since 2019, the relevant metries and methods still did not take advantage of the corresponding context. In this paper, propose a context-aware machine translation evaluation metric called CoMt-COMET, built upon the effective COMET framework. Our ap- proach simultancously considers the preceding and subsequent contexts of the sentence be evaluated and trains our metric to be aligned with the setting during human annotation. We also introduce content selection method to extract and utilize the most relevant informa- tion. The experiments and evaluation of Cont- COMET on the official test framework from WMT show improvements both system-level and segment-level assessments.
In this paper, we further explore context-aware methods for MT evaluation and propose our trained metric, Cont-COMET. To align with the annota- tion setting of human evaluation, we improve and train our model with additional encoded contex- tual information based on the COMET (Rei et al., 2020) framework, which has shown promising per- formance. In particular, we simultaneously utilize both the preceding and subsequent nearby contexts and attempt different numbers of sentences. Fur- thermore, given that the document is usually too long for the model to encode entirely, we propose a context selection method to extract the most rele- vant portion of the document as additional context. We evaluate our metric on the official Multidimen- sional Quality Metrics (MQM) judgments from WMT21 (Freitag al., 2021) and WMT22 (Freitag et al., 2022). The experimental results demonstrate our Cont-COMET achieves great improvements in segment-level assessment and better system-level correlation compared with previous work.
1 Introduction
Automatic evaluation metrics play an important role in the field of machine translation (MT) and promote the advancement of MT research. Many traditional metrics, such as BLEU (Papineni al., 2002) and METEOR (Banerjee and Lavie, 2005), were carlier proposed based on lexical-level matching between the human reference and MT- generated hypothesis. Despite excellent efficiency and ease of use, some recent work (Koemi et al., 2021; Freitag et al., 2022) has proven that these string- utinngbbsed metrics like BLEU perform poorly in terms of consistency with human evaluation. In- stead, many metric models (Lo, 2019; Mukherjee al., 2020; Sellam et al., 2020; Rei al., 2021; Perrella et al., 2022; Wan al., 2022; Rei al., 2022) using neural networks have emerged recently and achieved great capabilities for MT evaluation with the help of large-scale human annotations.

---

Current Sentence
Evaluation Score
Sentence
Context Concalenation
Sentence
Regression Estimator
Sentence 3
Context Selection
Pooling Layer
Cross-lingual Encoder
Sentence
Document
Similarity Score
Selected Contexi
Source
Reference Hypothesis
Figure 1: The brief framework of our context-aware training and context selection method for MT evaluation.
In general, our primary contributions are as fol- lows: (1) We observe the gap context utilization between current neural metries and human annota- tion from WMT and improve the training approach with additional context. (2) We explore different incorporations of surrounding sentences and pro- pose a context selection method to extract more important content. (3) We collect and collate the annotated data with document information and ac- cordingly train our metric Cont-COMET, which achieves overall improvements.
ditional encoding of context content. In previous work, the representations of the source (S), refer- ence (R), and hypothesis (H) for each sentence were directly encoded and vvraae--pooled. In- stead, we aim to incorporate contextual informa- tion when processing the sentence to be evaluated, so as to align with the human annotation setting. Unlike the work of Vernikos et al. (2022), we uti- lize the content of both the previous and next tences adjacent the current sentence, denoted Spren: Rprees H Hyrew and Swent Rueats Hneat-
We then conduct context-aware encoding, as shown Figure 1. As for reference, the current Reurr is concatenated with surrounding content in the original order, resulting in an expanded se- quence R = [Rprev' Reurri Rncat]. The positions of the tokens in the current sentence as Pourr. Then, we encode R and apply average pooling to the embeddings at the position Peurr obtain the context-aware representation:
2
Our Methods
We first introduce the COMET framework, which was proposed for MT evaluation and achieved su- perior performance in the annual WMT Metrics Shared Task. The basic architecture consists of three main components: the cross-lingual encoder, layer pooling, and estimator module. To leverage the knowledge of pretrained language models, the XLM-RRBERTaa-arre (Conneau et al., 2020) served as the cross-lingual encoder. It generates the representation = (co: c1. ... ( of the input se- quence r = (zo. ... 2n)) for each layer l. Then these representations are fused into single embed- ding sequence through trained layer-wise atten- tion mechanism. And the average pooling is em- ployed derive the final sentence embedding for each input. We s, r, and h refer to the final em- beddings of source, reference, and MT-generated hypothesis, respectively. A feed-foree forwwrd regression estimator is trained according to the combined vec- tor e r; & & rr; / s|; |/-//) rr], where @ represents the element- wise product. More details about COMET can be found in Rei et al. (2020).
masked
Fc Lrtrnge_pooling(nnceee (eandderrRR,,a Pourr)
And the corresponding combined vector for the regression estimator becomes Ve [hej ho Sei h @ rei hh sel: -re]--
The expanded content of the source and ref- erence consists of their respective context sen- tences, while the hypothesis is concatenated with the conext of reference, namely [Rerev; Heur: Rmest]] can avoid error propa- gation in the hypothesis and was also employed by Vernikos et al. (2022). And our following ex- periments prove that this setting is also superior for trained metric. In addition, we conduct ablation studies to explore the effects of different usages of context sentences and training approaches.
2.1 Context-aware Training
In summary, we improve the original training ap- proach based on the COMET framework with ad-
2.2 Context Selection
Despite the intuition that using as much context as possible should beneficial to the evaluation per- formance, the maximum input length of the XLM-
Our context-aware dataset and Cont- COMET metric are available at

---

RoBERTa-large used in COMET is limited 512 tokens. In many cases, the entire document actu- ally exceeds the length, and in the meantime, the effect of encoding may drop as the processed input becomes longer. Therefore, instead of directly con- catenating adjacent context sentences, we propose a context selection method to extract more relevant context for the evaluation. We aim to take better ad- vantage of the capabilities of the pretrained models and reduce the noise caused by irrelevant content. Specifically, for the document D containing sen- tences whose references are (R1, R2, : Rma). we first obtain their corresponding context-free rep- resentations (r1, 72, ('m) through the encoder. We suppose the more relevant sentence in the con- text should have higher similarity with the current sentence. And the similarity score between R₃ and R, is calculated as follows:
Context Setting
21 Sys 21 Seg
Sys
22 Seg
COMET-21
0.657
0.282
0.910
0.349
+ 2 prev sentences
0,656
0.240
0,940
0.283
+ 1 prev & next sentences
0.655
0.258
0.944
0.322
+ 4 prev sentences
0.637
0.232
0.944
0.291
+ 2 prev & next sentences
0.656
0.237
0.950
0.301
+ prev sentences
0,656
0.222
0.946
0.283
+ prev & next sentences
0.659
0.230
0.952
0.292
+ prey sentenees
0.654
0.224
0.947
0.283
+ prev & sentences
0.658
0.229
0.953
0.292
Table I:
The results of COMET-21 under different set- tings of context usage during testing. Sys, Seg, 21, and 22 refer to the system-leve and segment-level correla- tions, WMT21 and WMT22 test sets, respectively.
WMT 2022 were not available. Therefore, we col- leet the publicly released DA scores from WMT 2020, matching the document information, as our context-aware training set. Moreover, use the Multidimensional Quality Metries (MQM) data from WMT 2020 our validation set, and MQM data from WMT 2021 and WMT 2022 for evaluat- ing our trained metric, the same as previous work. More details are stated in Appendix B.
Sim R4, R2)= = Cosinc_similarity ry)ss Py)
where a is a decay factor for the distance between two sentences. For the current sentence, most rel- evant sentences are selected in order according to the similarity above, until the number of sentences meets the requirement the length is about to exceed 512. The selected contexts and the current sentence are then concatenated in the original order, followed by similar training, as shown in Figure
3.2 Evaluation Methods
To evaluate the performance of our metric, we mea- sured system-level and segment-level consistency with the gold standard of human judgment. We use Kendall's Tau (Freitag et al., 2022) for the segment- level evaluation and Pearson correlation coefficient for the system-level evaluation, which align with the assessments in WMT Metrics Shared Tasks. The system-leve pairwise accuracy is also addi- tionally considered, and all the definitions are same as Freitag et al. (2021).
3 Experiments
The following experiments and evaluations based the annotated dataset provided by WMT Metrics Shared Tasks and MTME toolkit 3. Our Cont-COMET metric has been trained upon the COMET-21 model proposed by Unbabel 4. The detailed papae-pprraetters and training settings are described in Appendix A.
4 Results and Discussions
We argue the conclusions of Vernikos et al. (2022) that using two preceding sentences during infer- ence yields the best performance are not adequately substantiated. Their work was based on the findings of previous studies (Kim et al., 2019; Castilho et al., 2020) and they did not attempt more sentences or other context settings. Therefore, we conducted similar but more in-depth experiments, taking into account the role of both preceding and subsequent context, as well different numbers of sentences.
3.1 Datasets
WMT Metrics Shared Tasks have significant im- pacts on MT evaluation and release large-scale annotated data each year, promoting ttate-ofthhe- art research. Although the integration of contex- tual information into the human annotation has been applied since 2019, the official dataset from WMT 2019 misses much meta information, and Direct Assessment (DA) scores in WMT 2021 and
The results of the original COMET-21 model using additional context sentences during inference without extra training are presented in Table 1. For
htttps//giiuub..oo/gooo wwwp//ooggl.--.sssrrttttmmmmmme hhtps///unbablll.g www.hhahww..gggg..... Sithuub..oCoMMET//////nneee html.

---

WMT21.TED
WMT2L.NWws Seg.Cor
WMT22
Model
Sys.Cor
Seg.
Sys.Acc
Sys.Cor
Sys.Acc
Sys.Cor
Seg.Cor
Sys.Acc
COMET-21
0.653
0.254
76.92
0.661
0.311
70.85
0.910
0.349
83,94
3 prev & next sentences
0.672
0.211
77.73
0.645
0.249
69.64
0.952
0.292
85.77
Our Cont-COMET
context sentences selection
0.665
0.265
78.54
0.664
0.306
71.66
0.938
0.353
84.31
context sentences selection
0.669
0.265
78.54
0.663
0.306
70.45
0.941
0.353
84.67
context sentences selection
0.678
0.265
78.95
0.665
0.303
71.26
0,943
0.354
84.31
context sentences selection
0.677
0.263
78.95
0.664
0.303
70.85
0.943
0.354
84.67
Table The results of our Cont-COMER metric with context selection of different numbers of sentences. Sys.Cor, Seg. Cor and Acc.Acc refer to the system-level correlation, segment-level correlation, and system-level pairwise accuracy according to human judgments, respectively.
brevity, the results of two domains, TED and News in WMT 2021 averaged, which is the same for the following Table 3. Moreover, each test set actually involves three language pairs: en-de, en-ru, and zh-en. Due to space limitations, we average their results to represent the performance of the corresponding test set in our experiments. More details and complete results are shown in Appendix C with other baseline metrics.
Different Training
21 Sys
21 Seg
22 Sys
22 Seg
Our COME-COMET
0.672
0.284
0,943
0.354
wlo context selection
0.669
0.284
0.942
0.353
w/o context sentences
0.665
0.230
0.953
0.296
w/ previous sentences
0.665
0,280
0.936
0.351
w/ hypothesis context
0.643
0.284
0.923
0.358
Table 3: The results of different training approaches for the ablation study, involving context with six sentences.
The results demonstrate that the combined ef- fect of preceding and subsequent context yields better performance when just using the same num- ber of previous sentences. Furthermore, although this method without additional training improves system-level performance compared to the original COMET-21, as more contexts are used, it exhibits significant declines in correlations at the segment level. Table 2 shows the detailed results of our trained Cont-COMET metric. We also evaluate the different numbers of context sentences during the context selection method. Similar to the results in Table 1, Cont-COMET achieves the best per- formance when using context containing approx- imately six sentences. It may be attributed to the degraded performance of the encoder in processing longer content. Moreover, compared previous settings without training in Table 1, ConI-COMET significantly improves the segment-level correla- tion. In terms of system-level performance, except for a slight drop on WMT22, there are also im- provements on the other two test sets. Overall, our demonstrates superior performance the original COMET-21 in all aspects of evalua- tion.
proaches, with corresponding results shown in Ta- ble 3. Our context selection method obtains slight improvements at both system level and segment level. Moreover, similarly train and evaluate the metric only using the previous sentences. To prove the necessity of introducing context during training, we additionally train the contrast metric without context but use the same context as Cont-COMET in testing. We also attempt the concatenation of the hypothesis with its own context rather than that of the corresponding reference, which has been mentioned in Section 2. The results of the latter two approaches demonstrate significant drops in the segment-level and system-level correlations, re- spectively. When only utilizing previous sentences. is inferior in all aspects of assessment, consistent with the results in Table 1. Overall, our proposed context-aware training has been proven effective and essential for the MT evaluation.
Conclusions
In this work, we explore context-aware training for MT automatic evaluation by incorporating con- text content into the encoding of sentences to be evaluated. We believe that method can better align with the annotation settings of the existing large-scale annotated dataset, which was ignored in previous studies. Additionally, we introduce a con- text selection method to extract the most relevant
5 Ablation Study
Furthermore, we conduct the ablation study to verify the effectiveness of different training ap-

---

content from the document enhance the con- textual information, and train our Cont-COMET metric accordingly. The experiments and ablation studies prove the effectiveness of context utiliza- tion and the overall improvement of our metric. In the future, we hope our work will encourage the community to take notice of context-aware eval- uation and conduct more in-depth research, and our proposed metric will be applied to document machine translation studies.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke 7ettle- moyer, and Veselin Stoyanov. 2020. Unsupervised cross- minsullnguual representation learning at scale. In Pro- ceedings of the 58th Annual Meeting of the Associa- rion for Computational Linguistics, ACI. 2020, On- line, July 5-10, 2020, pages 1440-8451, Association for Computational Linguistics.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, Eleftherios Avramidis, Tom Koemi, George F. Foster, Alon Lavie, and André F. T. Mar- tins. 2022. Results of WMT22 metries shared task: Stop using BLEU neural metrics are better and more robust. In Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhubi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 46-68. 68. Association for Computational Linguistics.
7 Limitations
The available annotated data for MT' automatic eval- uation that involves context information is insuffi- cient for the time being. So our proposed metric may not be trained adequately, and efforts at related data augmentation can be attempted. Moreover, current neural metrics cannot really encode the en- tire document content, due to the length limitations of commonly used pretrained language models. In future research, we will explore more specific and novel methods to further take advantage of the con- text for MT evaluation.
Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George E. Foster, Alon Lavie, and Ondrej Bojar. 2021. Results of the WMT21 metries shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Pro- ceedings of the Sixth Conference on Machine Transla- tion, WMT@ EMNLP 2021, Online Event, November 10-11, 2021, pages 733-774. Association for Com- putational Linguistics
8 Acknowledgements
Yunsu Kim, Due Thanh Tran, and Hermann Ney. 2019. When and why is document-level context useful in neural machine translation? In Proceedings of the Fourth Workshop on Discourse in Machine Transla- fion, DiscoMT@ EMNLP 2019, Hong Kong, China, November 3, 2019, pages 24-34. Association for Computational Linguistics.
This work was supported by National Key R&D Program of China (2021 1 YFF0901502), National Science Foundation of China (No. 62161160339), State Key Laboratory of Media Convergence Pro- duction Technology and Systems and Key Lab- oratory of Science, Technology and Standard in Press Industry (Key Laboratory of Intelligent Press Media Technology). We appreciate the anonymous reviewers for their helpful comments, and everyone who has provided assistance in this work. Xiaojun Wan is the corresponding author.
Tom Kocmi, Rachel Bawden, Ondrej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel, Thamme Gowda. Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz, Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák, Martin Popel. and Maja Popovic. 2022. Findings of the 2022 conference on machine translation (WMT22). In Proceedings of the Seventh Conference chine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 1-45, Association for Computational Linguisties.
Ma-
References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
an automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the Workshop on Intrinsic and Extrinsic Evalua- tion Measures for Machine Translation and/or Sum- marization @ ACL 2005, Ann Arbor, Michigan, USA, June 29, 2005. pages 65-72. Association for Compu- tational Linguistics.
Tom Koemi, Christian Federmann, Roman Grund- kiewicz, Marcin munezys-Dowmund Hitokazu Mat- sushita, and Arul Menezes. 2021. To ship or not to ship: An extensive evaluation of automatic metrics for machine translation. In Proceedings of the Sixth Conference on Machine Translation, WMT@ EMNLP 2021, Online Event, November 10-11, 2021, pages 478 494, Association for Computational Linguistics.
Sheila Castilho, Maja Popovic, and Andy Way. 2020. On context span needed for machine translation eval- uation. In Proceedings of The 12th Language Re- sources and Evaluation Conference, LREC 2020, Marseille, France, May 11-16, 2020, pages 3735- 3742, European Language Resources Association.
Chi-kiu Lo. 2019. Yisi unified semantic MT quality evaluation and estimation metric for languages with different levels of available resources. In Proceed- ings of the Fourth Conference on Machine Transla- tion, WMT 2019, Florence, Iialy, Augast 1-2, 2019

---

Volume 2: Shared Task Papers, Day I. pages 507- 513. Association for Computational Linguistics.
ACL Online, July 5-10, 2020, pages 7881-7892
Association for Computational Linguistics.
Ananya Mukherjee, Hema Ala, Manish Shrivastava, and Dipti Misra Sharma. 2020. MEE : An automatic metrie for evaluation using embeddings for machine translation. In 7th IEEE International Conference on Data Science and Advanced Analytics, DSAA 2020, Sydney, Australia, October 6-9, 2020, pages 292-299. IEEE
Giorgos Vernikos, Brian Thompson, Prashant Mathur, and Marcello Federico. 2022. Embarrassingly casy document-level MT metrics: How to convert any pretrained metrie into document-level metric. Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 118- 128. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, July 6-12, 2002, Philadelphia. PA, USA, pages 311-318 ACL.
Yu Wan, Dayiheng Liu, Baosong Yang, Haibo Zhang, Boxing Chen, Derek F. Wong, and Lidia S. Chao. 2022. Unite: Unified translation evaluation. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 8117-8127. Association for Computa- tional Linguisties.
Stefano Perrella, Lorenzo Proietti, Alessandro Sciré, Niccolò Campolungo, and Roberto Navigli. 2022. Matese: Machine translation evaluation as sequence tagging problem. In Proceedings of the Seventh Con- ference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 569-577. Association for Computa- tional Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinherger, and Yoav Artzi. 2020. Bertscore: Evalu- ating text generation with BERT. In 8th International Conference on Learning Representations. ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenRe- view.net.
Maja Popovic. 2015. chrf: character n-gram f-score for automatic MT evaluation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, WMT@ EMNLP 2015, 17-18 September 2015, Lis- bon, Portugal, pages 392-395. The Association for Computer Linguisties.
A Training Settings
Since our context-aware training data only involves DA scores from WMT 2020, which is far less than those of the existing neural metrics, we conduct continuing training on the COMET-21 model (Rei et al., 2021). It has been trained on WMT15-20 DA scores, and we only fine-tune the parameters of the regression estimator to prevent overfitting. The encoder and embedding layer are frozen dur- ing training. The detailed hyper-parametee of our trained metric are shown in Table 4. During the concatenation of the context and current sentence, the separate tokens <SEP> are added. Morcover, when using the context selection method, we at- tempted different sentence representations to cal- culate similarity scores. In the original work, the final representations were obtained by combining the representations from different layers through laier attention. In our experiments, we em- ploy the average representations from the first and last layers, which yields better performance.
Ricardo Rei. José G. C. de Souza, Duarte M. Alves, Chrysoula Zerva, Ana C. Farinha, Taisiya Glushkova, Alon Lavie, Luísa Coheur, and André F. T. Martins. 2022. COMET-22: unbabel-ist 2022 submission for the metries shared task. In Proceedings of the Seventh Conference on Machine Translation, WMT 2022, Abu Dhabi, United Arab Emirates (Hybrid), December 7-8, 2022, pages 578-585. Association for Computational Linguistics.
Ricardo Rei, Ana C. Farinha. Chrysoula Zerva, Daan van Stigt, Craig Stewart, Pedro G. Ramos, Taisiya Glushkova, André F. T. Martins, and Alon Lavie. 2021. Are references really needed? unbabel-ist 2021 submission for the metrics shared task. In Pro- ceedings the Sixth Conference on Machine Transla- tion, WMT@ EMNLP 2021, Online Event, November 10-11, 2021, pages 1030-1000 1040. Association for Com- putational Linguistics.
Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2020, Online, November 16-20, 2020, pages 2685-2702. Association for Computational Linguistics.
B Dataset Statistics
In fact, the officially released datasets from WMT Metries Shared Tasks have multiple sources that are inconsistent with each other and miss some meta information. After the comparison of data quality, we finally selected data from the MTME toolkit, fil- tered out the samples without annotated scores, and
Thibault Sellam, Dipanjan Das, and Ankur P. Parikh. 2020. BLEURT: learning robust metries for text generation. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics,

---

Hyper-Parameters
ferent training approaches, which are included in the part of our Con--COMET in Table They cor- respond to the experiments Table and involve different numbers of context sentences. Moreover, the results with different language pairs compared with other baseline metrics are displayed in Table 7 for a more comprehensive comparison.
optimizer
A 5e-6
learning rale
dropout
0.15
batch size epoch
layer
mix avg True
pool
_rozee keep_enne ccop_embeddding..o frozen
Table 4: The parammprrssserss during training.
Language Pair
#Assessment
Czech- English
7968
Ferman-Fnglisne
10205
Inuktitut-Eann Engglihh
35652
Japanese- Engglish
9930
KhmeerEnnglish
18560
Polish-English
14014
Pashto-English
19033
Russian-English
11892
Tamil-Fnglish
13958
ChineseEEnglsh
34000
English-Crech
18434
English- German
English-German
24106
English-Inuktitut
35652
English-Japanes
12000
English-Polish
15000
English-Russian -usiisnn
20020
English-Tari
16000
English-Chinen
19852
Table 5: The detailed statistics of our collected context- aware training dataset.
matched the corresponding contextual information. general, the context-aware dataset we collected includes 18 language pairs, and the statistics are shown in Table 5. We will release the dataset for future in-depth research. On the other hand, the MQM data used for evaluating our trained metric also comes from MTME and involves three lan- guage pairs: en-de, en-ru, and zh-en. These data annotated by professional evaluators with context content, and also serve as the official test datasets for the corresponding WMT Metrics Shared Task. And we follow the official evaluation framework with MTME.
C Additional Results
We add more detailed experimental results in Ta- ble 6, including the additional system-level pair- wise accuracy. As For the COMET-21 and the dif- ferent integration of context in the first seven lines of Table 6, they have not received additional train- ing and have just been tested with the described content, On the other hand, we attempt several dif-

---

WMT2LTED Sys.Cor
WMT21 Seg.Cor
WMT22
Model
Seg.Cor
Sys.Acc
Sys.Cor
Sys.Acc
Sys.Cor Seg.Cor
Sys.Acc
COMET-21
0.254
76.92
0.661
0.311
70.85
0.349
prev next sentences
0.656
0.246
76.52
0.270
70.85
0,944
0.322
85.40
+ prev sentences
0.661
0.232
78.14
0.651
0.248
70.45
0,940
0,300
86.13
prev 2 next sentences
0.665
0.231
77.73
0.648
0.242
68.83
0.950
0.301
85.77
prev sentences
0.207
77.73
0.645
0.257
70.04
0,944
0.291
86.13
prev 3 next sentences
0.672
0.211
77.73
0.645
0.249
69.64
0,952
0.292
85.77
prev sentences
0.669
0.201
77.73
0.643
0.243
70.85
0.946
0.283
86.13
Our Cont-COMET
context sentences selection
0.665
0.265
78.54
0.664
0.306
71.66
0,938
0.353
84.31
prev next sentences
0.264
77.73
0.661
0.306
71.66
0.353
84.67
prev sentences
0.664
0.264
77.73
0.664
0.302
72.47
0,933
0.351
85.40
w/o context training (2 sentences)
0.247
77.73
0.665
0.271
70.85
0,945
0.324
84.67
w/ hypothesis context (2 sentences)
0.653
0,262
76.52
0.644
0,301
70.45
0.925
0,352
83.21
context sentences
selection
0.669
0.265
78.54
0.663
0.306
70.45
0,941
0.353
84.67
prev &
0.668
0.263
77.73
0.662
0.304
71.26
0,942
0.353
84.31
prev sentences
0.674
0,264
77.73
0.660
0.302
71.26
0.351
85.04
w/o context training sentences)
0.667
0.234
78.
0.660
0.245
70.04
0,951
0.305
85.04
w/ hypothesis context (4 sentences)
0.656
0.266
77.73
0.637
0.299
70.85
0.923
0.357
83.94
context sentences selection
0.678
0.265
78.95
0.665
0.303
71.26
0.943
0.354
84.31
prev & sentences
0.675
0.263
78.54
0.662
0.306
71.26
0.942
0.353
84.31
prev sentences
0.672
0.261
78.54
0.658
0.298
70.45
0.936
0.351
85.04
w/o context training sentences)
0.675
0.214
78.14
0.656
0.246
69.64 64.64
0.953
0.296
85.40
w/ hypothesis context sentences)
0.655
0.268
77.73
0.631
0.301
70.85
0.923
0.358
84.31
Table 6:
The detailed results of our Cont-COMET metric with different training approaches, together with the context-aware testing of the COMET-21 model. Sys.Cor, Seg. Cor and Sys.Acc refer the system-lewe correlation, segment-level correlation, and pystem-leved pairwise accuracy according to human judgments, respectively.
System-level Correlation
Segment-level Correlation En-De
Model
System-level Accuracy
En-De En-Ru
Zh-En
En-Ru
Zh-En
WMT21.TED
COMET-21
0.781
0.872
0.306
0.268
0.274
0,220
76.92
prev & next sentences
0.783
0.897
0.336
0.202
0.268
0.163
77.73
Our Cont-COMET
0.795
0.889
0.350
0.263
0.289
0.243
78.95
BLEU (Papineni et al., 2002)
0,620
0.828
0.324
0.113
0.112
0.092
74.1
chrl (Popovic. 2015)
0.471
0.825
0.363
0.146
0.189
0.124
71.3
YiSi-1 (Lo, 2019)
0..114
0.9055
0.310
0.212
0.204
0.195
75.7
BERTScore (Zhang al., 2020)
0.506
0.831
0.306
0.189
0.185
0.199
72.1
BL.URT-220 (Sellam et al., 2020)
0.739
0.868
0.239
0.283
0.255
0.224
74.9
WMT21 News
COMET-21
0.812
0.654
0.517
0.253
0.307
0.372
70.85
+ 3 prev & next sentences
0.832
0.636
0.467
0.197
0.276
0.273
69.64
Our COn-COMET
0.823
0.676
0.495
0.247
0,305
0.356
71.26
BLEU (Papineni et al., 2002)
0.937
0.507
0.310
0.083
0.120
0.176
74.1
chrl (Popovic. 2015)
0.846
0.783
0,302
0.114
0.193
0.201
74.5
Y (Lo, 2019)
0.789
0.761
0,515
0.172
0.233
0.302
73.7
BERTScore (Zhang et al., 2020)
0.930
0.629
0,542
0.169
0.185
0,296
74.5
BLEURT-20 (Sellam et al., 2020)
0.802
0.768
0.563
0.264
0,286
0.354
75.3
WMT22
COMET-21
0.856
0.917
0.958
0.333
0.362
0.351
83.94
prev &
0.929
0.956
0.972
0.280
0.326
0.271
85.77
Our Cont-COMET
0.915
0.943
0.971
0.341
0.372
0.348
84.31
BLEU (Papineni 2002)
0.179
0.724
0.594
0.169
0.140
0.145
70.8
chrF (Popovic, 2015)
0.346
0.815
0.630
0.214
0.168
0.147
73.4
YiSi-1 (Lo. 2019)
0.626
0.935
0.235
0.227
0.296
79.2
BERTScore (Zhang et al., 2020)
0.428
0.924
0.232
0.192
0.316
BLEURT-20 (Sellam et al., 2020)
0.719
0.959
0.938
0.344
0.359
0.361
84.7
Table 7: The additional results with different language pairs and other baseline metrics. And our Cont-COMET model shown here utilizes six context sentences selection.


Reviews:
[{"review_id": "CCzN2Esj0k", "reviewer": "Reviewer_C6zw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and suggestions, and we will answer your questions as follows:\n\n> 1. The method proposed in the paper is slightly lacking in innovation, and perhaps further exploration of more efficient context utilization and modeling methods can be made.\n\nThe main contribution of our work lies in observing the context issues in existing translation evaluation metrics research and proposing corresponding improved metrics and datasets. We hope we can help the community promote the relevant research more effectively. And our method may slightly lack innovation, so we submitted a short paper, leaving further exploration of more efficient context utilization and modeling methods for future research.\n\n> 2. Based on the results of the ablation experiments in the paper, the effect change brought by context selection is small, which is not consistent with the hypothesis in the previous text. The authors may add some additional explanations.\n\nIn fact, as the context length used by the model increases, the performance gains from context-aware methods do diminish. We will supplement more relevant experimental results to demonstrate the effectiveness of our context selection method.\n\n> 3. According to the results in Table 2 and Table 3, the improvements brought by the proposed approach compared to the COMET-21 metric are mostly due to the introduction of contextual information in training. The overall level of innovation is slightly insufficient.\n\nThe main contribution of our work lies in observing the context issues in existing translation evaluation metrics research and proposing corresponding improved metrics and datasets. We hope we can help the community promote the relevant research more effectively. The integration of contextual content into the automatic translation evaluation metric plays the most important role, and the context selection method is supplementary.\n\n> 4. The paper could consider adding some other automated evaluation metric baselines to highlight the improvement brought by the current approach.\n\nWe agree that adding more automatic evaluation metric baselines can demonstrate the improvements of our method. We will collect the relevant results and add a table in the appendix to present them.\n\n> 5. It would be preferable to provide a brief introduction to the COMET framework in the paper.\n\nGiven the limited space of the short paper, we only provided a very brief description of the COMET framework in the final submitted paper. And we will provide more detailed information in the appendix.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The authors analyzed the impact of contextual information on machine translation evaluation and verified the hypotheses through reasonable experiments, providing a better automated evaluation metric.\n\n2. The authors added the context selection module based on the COMET framework, which enables more comprehensive utilization of information within the allowed length of pretrained language model encoders.\n\n3. The authors collected and collated a series of publicly available corpora, resulting in a high-quality corpus that can be used for automatic metric training.", "reasons_to_reject": "1. The method proposed in the paper is slightly lacking in innovation, and perhaps further exploration of more efficient context utilization and modeling methods can be made.\n\n2. Based on the results of the ablation experiments in the paper, the effect change brought by context selection is small, which is not consistent with the hypothesis in the previous text. The authors may add some additional explanations.\n\n3. According to the results in Table 2 and Table 3, the improvements brought by the proposed approach compared to the COMET-21 metric are mostly due to the introduction of contextual information in training. The overall level of innovation is slightly insufficient.\n\n4. The paper could consider adding some other automated evaluation metric baselines to highlight the improvement brought by the current approach.\n\n5. It would be preferable to provide a brief introduction to the COMET framework in the paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper focuses on automatic evaluation metrics for machine translation, and proposes a context-aware automatic evaluation metric called Cont-COMET. This metric builds on the COMET framework and introduces additional contextual information, enhancing the utilization of information through a context selection mechanism. Finally, the paper validates the effectiveness of the metric through various experiments and analyzes the impact of different settings through ablation experiments.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g1rzzRh5Fz", "reviewer": "Reviewer_844M", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and suggestions, and we will answer your questions as follows:\n\n> 1. The results are marginal, test of statistical significance could confirm whether the improvements are significant.\n\nThe results reported in our paper were obtained by averaging results from multiple experiments with different random seeds, ensuring reliability. Therefore, the improvements from our approach can be identified as significant.\n\n> 2. The sentence similarity is based on the reference. What happen if the selection is based on the hypothesis (or even the source)?\n\nIndeed, we did conduct these experiments previously, where we used sources or hypotheses as the objects for calculating similarity. However, the experiment's results were not satisfactory. We suppose that it might be attributed to our context extension mainly relying on references. Hence, maintaining the same context setting could yield the best improvement.\n\n> 3. Inclusion of the context hypothesis sentences basically means that the quality of their translation has some impact on the score of the hypothesis itself. This could (possibly) help system level correlation but can hurt the segment level one.\n\nWe have already experimented with the context of hypothesis sentences in the ablation studies in Table 3 and displayed the corresponding results. The system-level correlation exhibited a noticeable decline, whereas the segment-level correlation did not. This point is interesting, and we intend to further explore it in the future.\n\n> 4. The number of context sentences is fixed. Another approach would be to use a similarity threshold and variable number of sentences.\n\nWe agree with your opinion, which provides another approach worth exploring. We will supplement the results of relevant experiments for better comparison.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The context sentence selection method is an interesting novel idea that can motivate other researchers to further extend it.\nBlindly including previous and next sentences improves system level correlation but hurts segment level one. But the proposed method improves both. ", "reasons_to_reject": "The results are marginal, test of statistical significance could confirm whether the improvements are significant.\n\nThere are a few unexplored points of the presented method:\n1. The sentence similarity is based on the reference. What happen if the selection is based on the hypothesis (or even the source)?\n2. Inclusion of the  context hypothesis sentences basically means that the quality of their translation has some impact on the score of the hypothesis itself. This could (possibly) help system level correlation but can hurt the segment level one.\n3. The number of context sentences is fixed. Another approach would be to use a similarity threshold and variable number of sentences. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes a modification of the COMET MT evaluation metrics which adds context of the scored sentence into consideration. The paper improves the previous methods going in the similar direction by selection of the context sentences based not on sentence order but rather on sentence similarity. This is the main contribution of the paper.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ny5Iaa6WJz", "reviewer": "Reviewer_KJ38", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and suggestions, and we will answer your questions as follows:\n\n> 1. Would be interesting to see the proposed metric somehow compared with some other context-sensitive evaluation methods or even participants of the WMT metrics task.\n\nThe existing context-sensitive evaluation methods are very limited. We only found the work of Doc-COMET, which has been introduced in our paper and mainly compared with our methods. Moreover, participants in the WMT metrics task almost pay attention to the evaluation of individual sentences, which includes certain limitations. We agree that adding more automatic evaluation metrics can demonstrate the improvements of our method. We will collect the relevant results and add a table in the appendix to present them.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Substantial work has been done in an area where there's still room for improvement, promising results have been reported, at least on the system level compared to the original COMET-21 metric. ", "reasons_to_reject": "Would be interesting to see the proposed metric somehow compared with some other context-sensitive evaluation methods or even participants of the WMT metrics task. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper proposes Cont-COMET a modification of the popular machine translation (MT) evaluation method COMET by adding consideration for previous and following sentence context during evaluation, as well as a context selection method that enables finding the most relevant context. Authors experiment with data from the WMT metrics shared tasks and report improvements in system-level and segment-level evaluations. ", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "EVAeRx2AC0", "reviewer": "Reviewer_7yHH", "summary": "", "questions": "- What kind of sentences were selected as context when cosine similarity was used? Was any manual analysis conducted to check the usefulness of the context?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time and suggestions, and we will answer your questions as follows:\n\n> 1. The improvements demonstrated are marginal, and their advantage over COMET is unclear. COMET still outperforms their proposed metric in several cases. Statistical significance tests are not conducted, so it is unclear if the improvements are significant as claimed.\n\nThe results reported in our paper were obtained by averaging results from multiple experiments with different random seeds, ensuring reliability. Therefore, we think the improvements from our approach can be identified as significant. Moreover, COMET only exhibits better performance in terms of segment-level correlation in the domain of news in WMT21, compared to the metrics we proposed.\n\n> 2. Another claim that does not seem to have sufficient backing is that using a similar number of sentences for preceding and subsequent context performs the best. This conclusion seems to be based on a couple of improved results in Table 1, one of which is marginal, while all the rest of the results seem to show the contrary.\n\nWe agree with your opinion, and conducting experiments with different numbers of preceding and subsequent context sentences can enhance the clarity of our method. We will add more detailed context utilization settings in the appendix, including different combinations of numbers of context sentences.\n\n> 3. One of the assumptions made for context selection is that sentences similar to the current sentence (based on cosine similarity) are more relevant context. There is a good chance this is unlikely, since useful context often contains information that is not present in the current sentence (e.g. antecedents of pronouns) and is therefore less likely to be similar.\n\nWe acknowledge that this idea is somewhat subjective, but our method was based on semantic similarity, which is intuitive and reasonable. It is because, during document-level translation, semantic considerations play an important role in handling context content. And our experiment results also indicate that our context selection method achieves certain improvements.\n\n> 4. There is no language-wise breakdown of results in the paper. It is likely that the usefulness of context differs depending on the source and target languages and it seems to be a fairly big oversight to have a discussion of this missing.\n\nWe apologize for our oversight. Our initial experiment results actually contain the performance across different languages. However, due to space limitations in the short paper, we did not include these detailed results and relevant discussions. We will supplement them in the appendix.\n\n> 5. What kind of sentences were selected as context when cosine similarity was used? Was any manual analysis conducted to check the usefulness of the context?\n\nWe used sentences from the reference to maintain the same setting as our context extension process. And we did not conduct additional manual analysis, which could be considered for further evaluation.\n\n> 6. Missing References and Typos Grammar Style And Presentation Improvements.\n\nThank you very much for your reviews and the valuable suggestions that have helped us enhance the quality of our work. We will carefully address and improve upon these issues.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "As motivated by the paper, more work needs to be done towards having reliable context-aware metrics. So far there have been few model-based trained metrics that have incorporated context, so this work is a welcome step in that direction.", "reasons_to_reject": "- The improvements demonstrated are marginal, and their advantage over COMET is unclear. COMET still outperforms their proposed metric in several cases. Statistical significance tests are not conducted, so it is unclear if the improvements are significant as claimed.\n- Another claim that does not seem to have sufficient backing is that using a similar number of sentences for preceding and subsequent context performs the best. This conclusion seems to be based on a couple of improved results in Table 1, one of which is marginal, while all the rest of the results seem to show the contrary.\n- One of the assumptions made for context selection is that sentences similar to the current sentence (based on cosine similarity) are more relevant context. There is a good chance this is unlikely, since useful context often contains information that is not present in the current sentence (e.g. antecedents of pronouns) and is therefore less likely to be similar. \n- There is no language-wise breakdown of results in the paper. It is likely that the usefulness of context differs depending on the source and target languages and it seems to be a fairly big oversight to have a discussion of this missing.", "questions_to_authors": "- What kind of sentences were selected as context when cosine similarity was used? Was any manual analysis conducted to check the usefulness of the context?", "paper_topic_and_main_contributions": "This work introduces a context-aware machine translation metric, Cont-COMET, built on top of the trained COMET metric used for machine translation evaluation, which has been shown to have better correlations with human judgments compared to string-based matching metrics such as BLEU. In this work, the authors experiment with including previous and next context sentences for both source and target reference sentences, using reference sentences as context for the machine-generated hypothesis translation. All of these are encoded using XLM-R and trained to produce a score, in the same way as COMET. Experiments are conducted using cosine similarity to select relevant sentences from the context since XLM-R can only process up to 512 tokens. Experiments demonstrate some improvements in scores compared to COMET.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "BlonDe: An Automatic Evaluation Metric for Document-level Machine Translation (Jiang et al., NAACL 2022)"}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- It is claimed in the paper that WMT introduced context information for human judgments in 2019. This may need to be double-checked, since the document-level MT task was introduced in 2019; annotators were able to see context during evaluation much earlier.\n- It is not clear that the results in Table 1 are for untrained COMET with additional context sentences. This needs to be mentioned more explicitly before presenting the results.\n- Please also mention the language pairs in the test data in the main paper and perform statistical significance tests. Having a language-pair-wise breakdown will also help analysis.\n- ll 206-208 mention that Vernikos et al (2022) do not provide comprehensive conclusions. It would be helpful to include what conclusions they do provide, and why this paper is more comprehensive in comparison.\n- ll 029 were early proposed -> were earlier proposed\n- 043-044 while the related document content was often involved in practice -> in which practice? Unclear what this sentence means."}}]

Author rebuttal:

Meta-review: Four reviewers gave the Soundness scores 3/3/4/3 and Excitement Scores 3/4/3/2. The authors have responded to all four reviewers and all reviewers acknowledged the author's responses.
All reviewers tend to agree that this paper has positive Soundness in addressing context-aware MT Evaluation, with a minimum score 3 - good level. However, most reviewers do not seem to be excited for this paper to appear in the main conference. It shall appear in the findings if possible.
Decision: Accept-Findings