Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks
Authors: Yichen Huang
Timothy Baldwin
Venue: EMNLP 2023 Findings (2023)
Keywords: Automatic metric
textual adversarial attack
machine translation

Abstract: We investigate MT evaluation metric performance on adversarially-synthesized texts, to shed light on metric robustness. We experiment with word- and character-level attacks on three popular machine translation metrics: BERTScore, BLEURT, and COMET. Our human experiments validate that automatic metrics tend to overpenalize adversarially-degraded translations. We also identify inconsistencies in BERTScore ratings, where it judges the original sentence and the adversarially-degraded one as similar, while judging the degraded translation as notably worse than the original with respect to the reference. We identify patterns of brittleness that motivate more robust metric development.

Robustness Tests for Automatic Machine Translation Metrics with Adversarial Attacks
Yichen Huang
Timothy Baldwin MBZUAI
MBZUAI
yichen..hunngmmbuuii.cccc
The University of Melbourne timothy. bal dwin@mbzuai ac
Abstract
COMFT(crigina)) COMFT(perturhd): 2,081 -0.33 (-2.444)
We investigate evaluation metric per- formance adversarially-synhesizee texts, to shed light on metric robustness. We experiment with word- and character-level attacks on three popular machine transla- tion metrics: BeRe BLEURT, and COMET. Our human experiments validate that automatic metries tend to overpenalize adversarially-degraaeed translations. We also identify inconsistencies in BERTScore ratings, where judges the original sentence and the dddeessariallydeeraaae one similar, while judging the degraded translation notably worse than the original with respect to the ref- erence. We identify patterns of brittleness that motivate more robust metric development.
Human(orignal) - Hunantperturbodli:
1.225 0.274 (-0.551)
Reference:
Notic:: don't have an oven, just large pot.
Original tmansatioon
Note: don't have
Perturbed trnslation:
Note: don't an oven, just large ensserole.
(a)
BERTSconc(original original) - BERTSeonxpertuubo, original:: 2.127 1,800 (-0.268)
BEXTScore(original reference) SerRTSoorepeeuubbed reference):
0.325 -0077 (-0.44()4401))
Referene
There are many Latin American elements its tempo and melodiess
Origisal translation:
There are a for of Latin American ingredients your Tempo and melody.
1 Introduction
Pertured translatoon:
There are good American ingrafients your Tampo and melady.
Automatic evaluation metrics are key tool in modern-day machine translation (MT) quick and inexpensive proxy for human judgements. The most common and direct means evaluate an au- tomatic metric is to test its correlation with human judgements on outputs of MT systems. However, as such metrics are commonly used to inform the development of new MT systems and even used training and decoding objectives (Wieting et al.. 2019; Fernandes et al., 2022), it is inevitable for them to be applied to out-of-distributtin texts that do not frequently occur in existing system outputs. The rapid advancement of MT systems and metries, as well as the prospect of incorporating MT metrics in the training and generation process, motivates investigation into MT metric robustness.
(b)
Figure 1: (a) The metric overpenalizes the per- turbed translation when compared with human ratings. (b) The metric is self-inconsistent as judges the original and perturbed transla- tions to be similar (BERRScore(original, origi- nal) + (BERTScore(peturrbed original)) while judging the perturbed sentence translation (BERTScore(origina,, reference) + BIRRTScoreperurrbe, reference)). All ratings are normalized.
MT metries: BERTScore (Zhang et al., 2020). BLEURT (Sellam et al., 2020), and COMET (Rei et al., 2020). We construct situations where the metrics disproportionatell penalize adversarially- degraded translations. To validate such situations, collect a large set of human ratings both orig- inal and adversarially- BERTScore can also be seen as a measure of seman- tic similarity between any two sentences, we also explore another scenario of inconsistency where BERTScore judges the original and adversarially-
In this work, we examine textual adversarial at- tacks (TAAs) means to synthesize challeng- ing translation hypotheses where automatic metrics systematically underperform. We experiment with word- (Li et al., 2021; Jia al., 2019; Feng et al., 2018) and character-level (Gao et al., 2018) at- tacks on three popular, hegh-peeforming automatic
translations. As

---

perturbed translations as similar while judging the perturbed translation as notably worse than the orig- inal one with regard to the reference translation. Examples are shown Figure
f(a') is notably lower than y with a minimal num- ber of perturbations under only the fluency con- straint. The intuition is that when the number of perturbations small, humans are likely to rate the extent of degradation as less significant than the automatic metric. To validate whether the assump- tion holds, we collect continuous human ratings on meaning preservation against the reference trans- lation following Graham al. (2013, 2014, 2017) and compare the extent of degradation as judged by humans and that as judged by the metrics. We focus on meaning preservation as it is aligned with the training objectives of BLEURT and COMET. We describe further details in Appendix A.
We identify mask-filling and word substitu- tion as effective means to generate perturbed translations where BERTScore, BLEURT, and COMET over-penalize degraded translations and BERTScore is self-inconsistent. In particular, BLEURT and COMET are more susceptible to per- turbations in data with quaaity translations. Our findings serve as a basis for developing more robust automatic MT metrics."
2 Methods
Secondly, we investigate scenario where BERTScore is self-inconsistent by using itself semantic similarity constraint. As BERTScore can be seen as generic distance metric of se- mantic similarity, we can use it to measure the distance between the original and the perturbed translations, the original translation and the refer- ence, as well as the perturbed translation and the reference. When the original and perturbed transla- tions are measured as similar, the robust behavior would be for them have similar ratings with gard to the reference. We search for violations against this where BERTScore(perturbed refer- ence) is notably smaller than BLIRTScooelrirginall reference), but BERTTccorepperturhel, original) is close to BERTScoree original), which we use as a maximum score of similarity."
2.1 Formulation
Most TAA methods probe for the overreaction of the victim model (Wang et al., 2022). Given the original text and associated label y. the methods generate bounded perturbed x' with label y'. The perturbation is assumed be label-preserving (i.e. y = y). Robust behavior would be fax' = y for classification or f(x') s y for regression, and the attack is considered successful iff f(²') is notably different to y. The label-preserving assumption is usually enforced by set of constraints.
In our task, given the original translation and metric rating y. we aim generate perturbed text æ' that misleads the metric such that f(x') is notably different from y. The label-preserving assumption amounts to equivalence in meaning and fluency, which is commonly enforced through sentence embedding distance (Li et al., 2021) and perplexity (Jia et al., 2019; Alzantot et al., 2018). However, semantic equivalence can clearly not be adequately enforeed in our case: the MT metric can roughly be seen as model-based measure of semantic similarity, similar to the sentence em- bedding model enforeing the semantic constraint. When we have a "successful" attack where f(z') is notably different from y. we cannot be cer- tain whether it is because we have faulty metric (where the ground truth y/ is close to but f(x' is notably different from y') or faulty constraint (where the perturbed x' is semantically different from z and thus y' should be different from y).
2.2 Adversarial Attack Setup
We use the German-to-EEnglish system outputs from WMT 12, 17, and 22 (Callison-Burch et al., 2012; Bojar et al., 2017: Koemi et al., 2022), and ran- domly select 500 sentences for each system for each year, totalling 19K (source, translation, ref- erence) tuples. For the sake of efficiency, we use MT outputs whose associated references are longer than words. We normalize each metric such that their outputs on this dataset have a mean of and a standard deviation of 1. When probing for overpenalization, we consider three widely- used metrics: BERTScore (Zhang et al., 2020), BLEURT (Sellam et al., 2020), and COMET (Rei et al., 2020). We constrain the perturbed sentence to have an increase in perplexity of no more than
We explore two approaches with regard to this issue. Firstly, we experiment with forgoing the semantic constraint and searching for x' such that
We attempted similar setup with BLEURT. using a sym- metric variant of BLEURT as the semantic constraint, but preliminary experiments returned no successful attacks. We discuss further details in Appendix C.
Cooee and data are available at https:///githbb i i-need-sleed sleep/eveaaaeeeeeeee ataack

---

BERTS
COMET
Method
WMT 12
17
WMT
WMT 12
WMT 17
WMT
WMI
WMT 17
WMT
CLARR
98, 15.66% (7885)
90 59.44% (5480)
59.5555 (5466)
96. 24% (7809)
90.9155 (3405)
95.025 (5479)
20.265 (1622)
20085 (1148)
20.735 1400
Faster Genetii
75.99%5 (6079)
75.67% (4162)
70.84% (38966)
76.2156 (6997)
4255 (4423)
75.02% (4130)
11.74% (939)
12.20% (074)
11.50% (649)
Input Recuuation
40,0155
1695
(2407)
90,965
94.345 (1888)
(400))
(2777)
$0.825,
DeepWordBag
10.9358 (874)
6.753% (371)
BUBBSS (500)
12.82% (956)
7,0775 (389)
6.63% (360)
17,955 (1436)
10.70% (591)
9.02% (529)
Table 1: The percentages and numbers of perturbations fitting our criteria for each metric for each year. The WMT 12, 17, and 22 splits ontain 8K, 5.5K and 5.5K original system outputs, respectively.
10 as measured by GPT-2 (Radford et al., 2019), and search for cases where the perturbed transla- tion causes decrease of more than 1 standard deviation in the metric rating. When probing for self incoosissistncyy with BERTScore, we constrain the difference between BERTSccoe(perrurhed,, orig- inal) and OERTScoral, original) to be less than 0.3 after normalization, and search for cases where the perturbed translation causes a decrease of more than 0.4 in BERTScore.
ized ratings across annotators, resulting in 2,800 qualifying ratings respectively for original and per- turbed sentences. The Pearson correlations with human ratings are shown in Figure 2. We observe that the attacks lead to worsened correlations in most cases, with CLARE and the Faster Alzantot Genetic Algorithm leading to bigger degradations, suggesting mask-filling and word substitution as effective means of attack. All three metrics are par- ticularly susceptible to perturbations on the WMT 22 data where the original translations of higher quality. Both CLARE and the Faster Alzantot Ge- netic Algorithm lead to degradations of over 0.2 in Pearson correlations for BLEURT and over 0.4 for COMET. This likely because BLEURT and COMET are trained on data from previous years and cannot easily generalize to higher-quality trans- lations with minor modifications.
For both setups, we apply range of black-box search methods to generate perturbations, includ- ing word-level attacks (CLARE (Li et al., 2021), the Faster Alzantot Genetic Algorithm (Jia et al., 2019), Input Reduction (Feng et al., 2018)) and character-level attacks (DeepWordBug (Gao et al., 2018)). CLARE applies word replacements, in- sertions, and merges by mask-filling the Faster Alzantot Genetric Algorithm applies word substitu- tions, Input Reduction applies word deletions, and DeepWordBug applies character swapping, substi- tution, deletion, and insertion. We further describe these methods in Appendix B.
To investigate the cause of the reduced correla- tions, we compare the degradation of translation quality measured by the metrics and as judged by humans. Results are shown in Figure 5, We observe that, in most cases, the metrics assign higher differ- ences between the original and perturbed transla- tions. To quantify this observation, we perform a one-sided Wilcoxon rank-sum test on the subsets of the data corresponding to the 36 combinations of metrics, years, and attack methods. Under signifi- cance level of p < 0.05, for 25 out of the 36 combi- nations, the degradation measured by the metrics significantly larger than that as measured by hu- mans. This confirms our assumption of overpenal- ization. We also find that in most cases, the metrics penalize different perturbation instances more con- sistently than humans. As an exception, BLEURT and COMET are significantly more inconsistent when measuring CLARE-generatee degradations for WMT 22. This, again, suggests vulnerability against perturbed, high-quality translations outside the models' training sets.
3 Results
3.1 Probing for Overpenalization
We generate a total of 102, 176 perturbed transla- tions fitting our criteria. The breakdown across the search methods, metrics, and years is shown in Table 1. All three metrics seem insensitive to character-level perturbations, with WordWor returning a small number of eligible perturbations for each year. The more sophisticated CLARE and Faster Alzantot Genetic Algorithm returns a larger ratio of eligible perturbations for BERTScore and BLEURT. On the contrary. COMET appears more sensitive to word deletions, with Input Reduction returning eligible perturbations for more than 50% of the system outputs. The ratio of eligible pertur- bations fluctuates only slightly for different years.
We collect human ratings for a balanced subset of cligible perturbated translations and correspond- ing original translations. We aggregate the normal-
We also investigate the influence of sentence length on overpenalization. Changing a word in a short sentence may result in a larger score differ-

---

IEMM WMT
SortSiv WMT 17
BEim WMT
36
COMFT, WXTT
COMRY WMF
COMET, WMT
(L400 pessatod
dedeetien aggeter
Frped Aedeete perfected
Ruoe Glundh pestededd
Day argous
therr Ceatie
Gutte
thand
ded
Figure 2: The Pearson correlation (r) with human ratings for different metrics, years, and attack methods original and perturbed translations. The error bars show the standard error as computed through bootstraping with 10K resamples.
Method
WMT
WMT
WMT
5.78%
2.38% (34)
Faster Generic
0.88% (11)
1.58% (21)
0.98% (13)
Inpul Reduction
1.26% (125%((17)
0.23% (3)
(53
Table 2: Percentages and numbers of successful attacks on BERTScore self-inconsistency on 4K randomly- sampled sentences. DeepWordBug returns successful attacks.
21. 93 All (eaherpppe)
22 45
19.48 Kiok disermuuan
prntelvttonn
find that (3) is smaller than (I) and (2) level of statistical significance. This suggests that shorter MT outputs lead more severe over- wver-ennaiizaton. However, the difference in sentence lengths is small and does not fully explain the different degrees penalization.
Figure 3: MT output length in the number of words for: (1) the 19K MT outputs used generate the perturbations (subsampled to be balanced across the years); (2) all MT outputs leading to cligible perturba- tions; and (3) the 500 MT outputs where humans and metrics disagree the most on the degree of penalization.
3.2 Probing for Self-Inconsistency
ence, and this difference might be different between human and metric scores. We compare the length of MT outputs (in the number of words) before perturbation for the following subsets: (1) the 19K MT outputs we used to generate the perturbations (subsampled to be balanced across the years); (2) all MT outputs leading eligible perturbations; and (3) the 500 MT outputs where humans and metrics disagree the most on the degree of penal- ization. Statistics are shown in Figure Using a one-sided Wilcoxon rank-sum test with p < 0.05,
We randomly select a total of 4K system out- puts balanced across years and systems, and search for perturbations fitting our criterion of self- inconsistency. Results are shown in Table 2. While all attack methods return small number of suc- cessful attacks, observe trend that CLARE and the Faster Alzantot Genetic Algorithm have a higher success rate. This, again, suggests the effec- tiveness of mask-filling and word substitution as attack methods.

---

3.3 Implications of this Work
5 Conclusion
The immediate implication of this work is to aug- ment training of learned metrics such as BLEURT and COMET with the data generated in this work, and experiment with incorporating automatically- generated synthetic data based mask-filling and word substitution.
We apply word- and character-level adversar- ial attacks and probe for overpenalization with BERTScore, BLEURT, and COMET, and for self- inconsistencies with BERTScore. We observe that mask-filling and word substitution are more effec- tive at generativing challenging cases, and that BLEURT and COMET are more susceptible to perturbation of high-quality translations.
4 Related Work
Classical MT metrics such as BLEU (Papineni et al., 2002) and ROUGE (Lin, 2004) have been shown to correlate poorly with human judgements (Mathur al., 2020; Koemi et al., 2022), moti- vating the development of model-based metrics. Supervised metrics such BLEURT (Sellam et al., 2020) and COMET (Rei et al., 2020) are trained to mimic human ratings as a regression task. Non- learnalle metrics such as BERTScore (Zhang et al., 2020), XMoverDistance (Zhao et al., 2020), and UScore (Belouadi and Eger, 2023) do not rely on human ratings and instead leverage the embeddings of the source, reference, and translation. Whereas more recent and higher-performing metrics ist, we focus our investigation on BERTScore, BLEURT, and COMET as they are most commonly used and easily adapted to other domains such as text simplification (Maddela et al., 2023).
Our findings motivate more sophisticated data augmentation and training methods to achieve greater metric robustness. In particular, our for- mulation of self-consistency requires no valida- tion against human ratings and can be applied other embedding- metries (Zhao et al., 2020; Reimers and Gurevych, 2020; Belouadi and Eger, 2023) as a regularization term. We leave this to future work.
6 Limitations
We use the high-resource German-to-Engee sub- set of the WMT datasets, which mainly focuses the news domain. How readily results translate across other language pairs, transla- tion systems, metries, or domains requires further investigation. experiment with only word- and character-leve attacks, but other methods exist that generate sentence-level (Ross et al., 2022) or multi- level (Chen et al., 2021) attacks. We leave more comprehensive study of attack methods to future work.
Wang et al. (2022) defines robustness perfor- mance unseen test distributions. Such distribu- tions can occur naturally (Hendrycks et al., 2021) be constructed adversarially, and robustness fail- ures are usually identified through human priors and error analysis. Alves et al. (2022) and Chen et al. (2022) use hand-crafted types of perturbations to create challenge sets where MT metrics underpe- nalize the perturbed translations. Yan et al. (2023) use minumum risk training (Shen et al., 2016) to optimize directly for higher metric scores, resulting in set of translations with overestimated scores, This work complements previous works by investi- gating overpenalization, which is very different behavior to overestimation. We use adversarial at- tacks targeted at each metric that are not limited to pre-defined categories, which allows us to discover particular failure cases specific to each metric. In addition, we consider the same set of metrics different years of WMT data. This allows us to draw connections between adversarial robustness and the quality of MT system outputs, and whether the MT system outputs are used when training the metric.
Ethics Statement
The human ratings are collected from fluent En- glish speakers contracted through work-sharing company. The annotators are paid fairly according to local standards. Prior to annotation, we informed the annotators of the purpose the collected data and provided relevant training. We ensured that the annotators had prompt means of contacting us throughout the annotation process.
Acknowledgements
We extend our gratitude to Teresa Lynn for her as- sistance with experiment design and data collection. We thank Daniel Chin and Yuchen Wang for their feedback on the initial draft of this work. Finally, we thank the anonymous reviewers for their helpful feedback and suggestions.

---

References
Pathologies of neural models make interpretations difficult. In Proceedings the 2018 Conference on Empirical Methods in Natural Language Processing, pages 37 19-3728, Brussels. Belgium. Association for Computational Linguistics.
Duarte Alves, Ricardo Rei, Ana C. Farinha, José G. C. de Souza, and André F. T. Martins. 2022. Robust MT evaluation with sentence-level multilingual augmen- tation. In Proceedings of the Seventh Conference on Machine Trunslation (WMT), pages 469-478 Abu Dhabi, United Arab Emirates (Hybrid). Association for Computational Linguistics.
Patrick Fernandes, António Farinhas, Ricardo Rei, José G. C. de Souza, Perez. Ogayo, Graham Neubig, and Andre Martins. 2022. Quality-aware decoding for neural machine translation. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1396-1412, Seattle, United States. Association for Computational Linguistics.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wa Chang. 2018. Generating natural language adversarial ex- amples. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium. Association for Computational Linguisties.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50-56.
Jonas Belouadi and Steffen Eger. 2023. UScore: An effective approach to fully unsupervised evaluation metrics for machine translation. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 358-374, Dubrovnik, Croatia. Association for Com- putational Linguistics.
Yvette Graham, Timothy Baldwin, Alistair Mofat, and Justin Zobel. 2017. Can machine translation systems be evaluated by the crowd alone. Natural Language Engineering, 23(1):3-30.
Ondřej Bojar, Rajen Chatterjee. Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz. Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017. Findings of the 2017 Conference on Machine Translation (WMT17). In Proceedings of the Second Conference on Machine Translation, pages 169-214, Copenhagen, Denmark. Association for Computa- tional Linguisties.
Yvette Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2013. Continuous measurement scales in human evaluation of machine translation. In Pro- ceedings of the 7th Linguistic Annotation Workshop and interoperability with Discourse, pages 33-41, Sofia, Bulgaria. Association for Computational Lin- guistics.
Yettee Graham, Timothy Baldwin, Alistair Moffat, and Justin Zobel. 2014. Is machine translation getting bet- over time? In Proceedings of the 14th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 443-451, Gothenburg. Sweden. Association for Computational Linguisties.
Chris Callison-Burree Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2012. Findings of the 2012 Workshop on Statistical Ma- chine Translation. In Proceedings of the Seventh Workshop on Statistical Machine Translation, WMT '12, page 10-51, USA. Association for Computa- tional Linguistics.
Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Stein- hardt, and Dawn Song. 2021. Natural adversarial examples. CVPR.
Xiaoyu Chen, Daimeng Wei, Hengchao Shang, Zongyao Li, Zhanglin Wu, Zhengzhe Yu. Ting Zhu, Mengli Zhu, Ning Xie. Lizhi Lei. Shimin Tao, Hao Yang, and Ying Qin. 2022. Exploring robustness of machine tramslation metrics: A study of twenty-two automatic metries in the WMT22 metric task. In Proceedings of the Seventh Conference on Machine Translation (WMT), pages 530-540, Abu Dhabi, United Arab Emirates (Hybrid). Association for Com- putational Linguistics.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLL pages 4129-4142, Hong Kong, China. Association for Computational Linguistics.
Yangyi Chen, Jin Su, and Wei Wei. 2021. Multi- granularity textual adversarial attack with behavior cloning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. pages 151164526, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Tom Kocmi, Rachel Bawden, Ondřej Bojar, Anton Dvorkovich, Christian Federmann, Mark Fishel. Thamme Gowda, Yvette Graham, Roman Grund- kiewicz, Barry Haddow, Rebecca Knowles, Philipp Koehn, Christof Monz. Makoto Morishita, Masaaki Nagata, Toshiaki Nakazawa, Michal Novák. Martin Popel, and Maja Popović. 2022. Findings of the 2022 Conference Machine Translation (WMT22). In Proceedings of the Seventh Conference on Machine Trunslation (WMT), pages 1-45, Abu Dhabi, United
Shi Feng, Eric Wallace, Alvin Grissom II, Mohit Iyyer. Pedro Rodriguez, and Jordan Boyd-Graber. 2018.

---

Arab Emirates (Hybrid). Association for Computa- tional Linguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon Lavie. 2020. COMET: A neural framework for MT evaluation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 2685-270, Online. Association for Computational Linguistics.
Dianqi Li, Yizhe Zhang, Hao Peng, Liqun Chen, Chris Brockett, Ming- Ting Sun, and Bill Dolan, 2021. Con- textualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 5053-5069, Online. Association for Computa- tional Linguistics.
Nils Reimers and Tryna Gurevych. 2020. Making monolingual sentence embeddings multilingual us- ing knowledge distillation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1512-4525. Online, Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for auto-
matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Alexis Ross, Tongshuang Wu, Hao Peng. Matthew Pe- ters, and Matt Gardner. 2022. Tailor: Generating and perturbing text with semantic controls. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics /Volume I: Long Pa- pers), pages 31 3194-3213, Dublin, Ireland. Association for Computational Linguistics.
Mounica Maddela, Yao Dou, David Heineman, and Wei Xu. 2023. LENS: A learnable evaluation metric for text simplification. arXiv cs.CL/2212.09773
Nitika Mathur, Timothy Baldwin, and Trevor Cohn. 2020. Tangled up BLEU: Reevaluating the eval- uation of automatic machine translation evaluation metrics. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics (ACL 2020), pages 4984-4997.
Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020. BLEURT: Learning robust metrics for text genera- tion. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pages 7881-7892, Online. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigshy, Di Jin, and Yanjun Qi. 2020. TextAttack: A frame- work for adversarial attacks, data augmentation, and adversarial training NLP. In Proceedings of the 2020 Conference on Empirical Methods in Natu- ral Language Processing: System Demonstrations, pages 119-126.
Shiqi Shen, Yong Cheng, Zhongjun He, Wei He, Hua Wu, Maosong Sun, and Yang Liu. 2016. Minimum risk training for neural machine translation. In Pro- ceedings of the 54th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 1683-1692, Berlin, Germany, Associ- ation for Computational Linguistics.
Nikola Mrkšić, Diarmuid Ó Séaghdha, Blaise Thomson, Pei-Hao Su, David Vandyke, Tsung-Hsiee Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic con- straints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 142-148, San Diego, California. As- sociation for Computational Linguistics.
Milica Gašić, Lina M.
Xuezhi Wang, Haohan Wang, and Diyi Yang. 2022. Measure and improve robustness in NLP models: A survey. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisticx: Human Language Tech- nologies, pages 4569-4586, Seattle, United States. Association for Computational Linguistics.
John Wieting, Taylor Berg- Kirkpatrick. Kevin Gimpel, and Graham Neubig. 2019. Beyond BLEU: Training neural machine translation with semantic similarity. In Proceedings of the 57th Annual Meeting of the As- sociation for Computational Linguistics, pages 4344- 4355, Florence, Italy, Association for Computational Linguistics.
Kishore Papineni. Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. BLEU: a method for automatic eval- uation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 31 1-1318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Yiming Yan, Tao Wang. Chenggi Zhao, Shujian Huang. Jiajun Chen, and Mingxuan Wang. 2023. BLEURT has universal translations: An analysis of automatic metrics by minimum risk training. In Proceedings of the 61 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 5428 -444, Toronto, Canada. Association for Computational Linguistics.
Amy Pu, Hyung Won Chung, Ankur Parikh, Sebastian Gehrmann, and Thibault Sellam. 2021. Learning compact metrics for MT. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 751-762, Online and Punta Cana, Dominican Republic. Association for Compu- tational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAl blog, 18):9
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q- Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating text generation with BERT. In Interna- tional Conference on Learning Representations.

---

Wei Zhao. Goran Glavaš, Maxime Peyrard, Yang Gao, Robert West, and Steffen Eger. 2020. On the lim- itations of cross-lingual encoders as exposed by reference-free machine translation evaluation. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pages 1656- 1671, Online. Association for Computational Linguis- tics.
tain the z-scores by normalizing annotations from the same annotator, and aggregate the ratings for the same translation by averaging. We use tuples with at least three annotations, resulting in 10,080 annotations for 2,800 tuples. The annotated data is balanced for the different metrics, years, and search methods.
A Annotation
B Implementation Details
To validate whether the perturbed sentences lead to degraded metric performance, we collected human ratings for a subset of perturbed translations and corresponding original system outputs. We largely follow the DA protocol (Graham et al., 2013, 2014, 2017) and task the annotators to rate on a con- tinuous scale to what extent the meaning of the reference is expressed in the translations. We fo- cus on mcaning preservation as it is aligned with the training objectives of BLEURT and COMET. We take more conservative stance by displaying the original and perturbed sentences in parallel and highlighting the perturbed words. Our intuition is that the annotators are more inclined to exaggerate the quality differences between the original and per- turbed translations under this setup. An example of the annotation interface is shown in Figure 4. We randomise the order of the original and perturbed sentences such that it is not immediately clear to the annotator which is which.
We use the Huggingface Evaluate* implemen- tations of BERTScore, BLEURT, and COMET. For BERTScore, we roberta-large as the underlying model and use FI score as the met- ric output. For BLEURT, we use the improved bleurt-20-d1 checkpoint introduced by Pu et al. (2021). For COMET, we use the wmt 1at:200-coett--0 checkpoint.
We consider four search methods for adversar- ial attacks: CLARE, the Faster Alzantot Genetic Algorithm, Input Reduction, and Bug.WordBug. CLARE iteratively applies contextualized word- level replacements, insertions, and merges by mask- ing and bounded infilling, with each perturbation greedily selected by the impact on the victim. The Faster Alzantot Genetic Algorithm modifies the ge- netic algorithm proposed by Alzantot et al. (2018), and iteratively searches for word replacements that are close in counter-fitted embedding space (Mrkšić et al., 2016). Input Reduction iteratively removes the least important word based on its influ- ence on the victim's output. DeepWordBug itera- tively applies a heuristic set of scores to determine the word to perturb, and applies character level swapping, substitution, deletion, and insertion.
For each human intelligence task (HIT) of 100 (reference, original, perturbed) tuples, we use the ratings from 70 tuples and include 30 tuples as control items. 15 of the control items are dupli- cates from the 70 tuples, and the other 15 contain degraded original and perturbed translations from the 70 tuples. where we randomly drop four words. We use the Wilcoxon rank sum test ensure that the score differences from the duplicated pairs are smaller than that of the degraded pairs. We reject HITs where the p value is larger than 0.05.
We use the TextAttack (Morris et al., 2020) implementations of the adversarial attacks. For CLARE, we modify the default implementation by removing the sentence similarity constraint and using beam search with width 2 and maximum of 10 iterations when probing for overpenalization, and with width and a maximum of 15 iterations when probing for self-inconsistency. For the Faster Alzantot Genetic Algorithm, we modify the im- plementation by changing the LM constraint and using population size of 30 and maximum of 15 iterations when probing for overpenalization, and population size of 60 and a maximum of 40 iterations when probing for self-inconsistency. We follow the default implementation otherwise. For the GPT-2 perplexity constraint, we use the
We collect ratings from 10 fluent English speak- e contracted through work-sharing company. We conduct training sessions where we describe the task and annotation interface prior to the annota- tion process. In total, we collected 268 HITs, with 177 (66.01%) HITs passing quality control. The ratio is higher than those reported by Graham et al. (2017) as we work with trained annotators. We ob-
'The annotation team is lead by two native English speak- ers. Each of the remaining members either completed under- graduate education in English or have spent years living in the UK, and have the equivalent of C2 proficiency in English.
"nttps://huggingfan co/ http:://uuggnnggeee.coooooeoeess

---

Completed annotatoon roquesti Of 1001
Please read the the following sentences:
Reference,
Ray rescuer men tuckyyh in
Samtnnce 1
Saving
ludky
Semience 2.
Saving
have
Takingg the hightli nied words into cons deration, rate the above sentences by how much you agree that:
The moaning of the reference sentence adequately expressed Sentence 1.
disagree
agres
Th moaning of tho referonce sontonce adoquatoly oxprossed Sentonco 2.
Strongly deaggrnn
Strangly agren
Nedl
Figure 4: A screenshot of the annotation interface. The slider is initialized at the middle position. The annotator must interact with the slider before proceeding to the next annotation instance and cannot revisit completed annotations.
Huggingface Evaluate implementation.
This makes it difficult find perturbations satisfy- ing the semantic constraint.
C Probing for Self-Inconsisteccy with BLEURT
Our formulation of self-consistency does not immediately apply to BLEURT as it distin- guishes between the hypothesis and the refer- ence and thus cannot be seen distance met- ric. We instead experiment with symmet- ric variant of BLERUT as the semantic con- straint. Given (hytttshypothesss, reference), we define symmetric BLEURT as the aver- age between BLEURT(rriiinal, perturbed) and BLEURT(perurbed, original), We constrain the difference between (LFURi original) and this symmetric measure to be smaller than 0.3. We search for perturbed translations such that the difference between BLERUT(original, reference) and BLERUT(p reference) is larger than 0.4.
None of the search methods returns successful at- tacks for preliminary experiments with 1K ran- domly sampled translations. We find that BLEURT overpenalizes translations with low-quality refer- ences, i.e. (orginl, perturbed) signifi- cantly smaller than BLERUT (prruree,,, original).

---

WOLD
WUIT
COMET, WMI
COMRT,
thy andtie
ClASE eeee
Paic dheedil avrde
Eove sheett Ataede
Clubk wure
Power cheeti Awaldf
Alere Cheale absisme
Spyer didde Awate
or
Reser
dbene Cresde
dedeetia
Aver Audeddey
Appor destralee
they Bogttaal
Figure S: The quality difference between the original and perturbed translations as measured by the metrics and humans. Bars marked with red asterisks the where the degradation for metrics is significantly larger (p < 0.05) than that for humans.


Reviews:
[{"review_id": "LGLFBD12JY", "reviewer": "Reviewer_cxZu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful feedback. In the following comments, we address the raised concerns in a breakdown format.\n\n> I'm not convinced that the over-penalization phenomenon described in L206-210 is real -- in Figure 4, most of the differences between humans & metrics are well within the confidence interval.\n\nWe run a Wilcoxon rank-sum test on the degree of degradation as measured by the metrics vs.  humans, with the null hypothesis being that the differences are drawn from the same distribution and the alternative hypothesis that the differences between metric scores are greater. On all data reported in Figure 4, the test results in a very low p-value of $1.65 \\times 10^{-38}$, suggesting that overall, the difference is highly significant and the over-penalization phenomenon is real. We also run the test on subsets of the data corresponding to different combinations of metrics, years and attack methods under a significance level of 0.05. Out of the 36 combinations, the 11 cases where the null hypothesis is not rejected are as follows.\n\n* BERTScore\n  * 2012: Clare, faster genetic, input reduction\n  * 2017: Clare, faster genetic\n  * 2022: Faster genetic\n* BLEURT\n  * 2012: Input reduction\n  * 2017: Faster genetic, input reduction\n* COMET\n  * 2012: faster genetic\n\nIn the other 25 cases, the over-penalization is statistically significant. These include cases on WMT 22, and cases with CLARE and the Faster Alzantot Genetic Algorithm, where we observed the most significant degradation in the performance of the metrics. This supports our findings and will be included in the paper.\n\n> Even the example in Figure 1 is not very convincing -- the humans are not instructed to give negative scores, so this may just be an effect of different scales.\n\nThe human ratings and metric scores in Figure 1 are normalized to have a mean of 0 and standard deviation of 1 and are of the same scale.\n\n> When compared to some (uncited) prior work on MT metric robustness (e.g. https://aclanthology.org/2022.wmt-1.43.pdf, https://aclanthology.org/2022.wmt-1.46.pdf), the novelty of this work is incremental.\n\nThank you for the suggested papers. We agree that they are relevant and will include them in the revised version. We point out the following important differences with our work:\n\n* The referred works test MT metrics on under-penalization. We complement their findings by investigating over-penalization, which is a very different behaviour.\n* The referred works are based on hand-crafted types of perturbations (named entities, numbers, negations, insertions and deletions for Alves et al. (WMT 2022), and numbers, dates/times, named entities/terminology, unit and affirmation/negations for Chen et al., (WMT 2022)) and do not take into account the response of the metric when generating perturbations. We use adversarial attacks targeted at each metric that are not limited to pre-defined categories (e.g. neither of the referred challenge sets includes free-form substitution). This allows us to discover particular failure cases specific to each metric.\n* The referred works consider different metrics on the same MT dataset, while we consider the same set of metrics on different years of WMT data. This allows us to draw connections between adversarial robustness and the quality of MT system outputs, and whether the MT system outputs are used when training the metric.\n\n> All the experiments are done only on one language pair (de-en).\n\nWe agree that experiments on more language pairs would support more general conclusions, but suggest that such extensive investigations fall outside the scope of our short paper. We point out that it is not uncommon for similar works to include only one language pair (e.g. one of the suggested papers (Chen et al., WMT 2022) also only considers one language pair). As such, we leave investigation of different language pairs and domains to future work.\n\nReferences: \n\n[Robust MT Evaluation with Sentence-level Multilingual Augmentation](https://aclanthology.org/2022.wmt-1.43) (Alves et al., WMT 2022)\n\n[Exploring Robustness of Machine Translation Metrics: A Study of Twenty-Two Automatic Metrics in the WMT22 Metric Task](https://aclanthology.org/2022.wmt-1.46) (Chen et al., WMT 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. MT metric's robustness is an important topic to address.\n2. The paper demonstrated a successful path toward generating adversarial attacks for MT metrics. The resulting attack and human evaluation, if released, will be a great resource for improving metric models.\n3. The experiment, especially the human evaluation, is solid and carefully designed.", "reasons_to_reject": "1. I'm not convinced that the over-penalization phenomenon described in L206-210 is real -- in Figure 4, most of the differences between humans & metrics are well within the confidence interval. Even the example in Figure 1 is not very convincing -- the humans are not instructed to give negative scores, so this may just be an effect of different scales.\n2. When compared to some (uncited) prior work on MT metric robustness (e.g. https://aclanthology.org/2022.wmt-1.43.pdf, https://aclanthology.org/2022.wmt-1.46.pdf), the novelty of this work is incremental.\n3. All the experiments are done only on one language pair (de-en).", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper studies machine translation metrics' robustness to various adversarial attacks. Ideally, we would like to find adversarial attacks that (1) do not change the semantics of the translation and (2) significantly degrade the MT metrics' score. However, (1) is hard to enforce, so the paper proposed two approaches: (1) perturb minimally, then confirm by the human evaluation; (2) for BERTScore, use it to evaluate the semantic similarity between original/perturbed translation, then check if it's consistent with the observed change in metric scores before/after perturbation.\n\nResults show that (1) yields a large number of successful attacks that degrades metric score's correlation with human judgments most of the time, but (2) only returned a small number of successful attacks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- https://aclanthology.org/2022.wmt-1.43.pdf\n- https://aclanthology.org/2022.wmt-1.46.pdf"}, "extra_text": {}}, {"review_id": "aWaMfhzQ65", "reviewer": "Reviewer_1kEg", "summary": "", "questions": "A.. Regarding Figure 1(a)\n* Could this phenomenon be affected by the length of the input sentence?\n\nB. Regarding Figure 1(b)\n* What does it mean by calculating Bert (original, original)?\n* The calculation in the second row seems to be incorrect?\n\nline152 C. What's the threshold for COMET and BLEURT?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback and questions. We respond to the questions and concerns in a breakdown format.\n\n>The length of the sentence may affect the evaluation results, i.e., change a word in a short sentence may lead to large difference in the metric score. Therefore, a standardization term may be needed when calculate the differences between metric scores.\n\nWe agree that the length of the sentence might be an influencing factor for the metric scores, but the same is true for human ratings: changing a word in a short sentence will likely significantly alter its meaning and greatly alter the human ratings for meaning preservation. It is still a fair comparison when we compare the degree of penalization as given by humans and metrics.\n\nTo investigate the influence of sentence length, we report the statistics of the length of MT outputs before perturbation (in the number of words). Note that for the sake of efficiency, we used MT outputs whose associated references are longer than 10 words when collecting (3).\n\n1. The 500 MT outputs where humans and metrics disagree the most on the degree of penalization: Mean: 19.482. Standard deviation: 8.179.\n2. All MT outputs leading to eligible perturbations: Mean: 22.452. Standard deviation: 10.979.\n3. The 19K MT outputs we used to generate the perturbations (subsampled to be balanced across the years): Mean: 21.807. Standard deviation: 11.138.\n\nShould shorter MT outputs lead to more severe over-penalization, (1) should be notably smaller than (2) and (3). Using a one-sided Mann-Whitney U test with a significance level of 0.05, we found that it is indeed statistically significant that (1) is smaller than (2) and (3). However, the difference in sentence lengths is small and cannot fully explain the different degrees of penalization. We do however acknowledge that the potential impact of sentence length deserves further investigation, and will discuss this in the paper.\n\n> Regarding Figure 1(b), what does it mean by calculating Bert (original, original)?\n\nWe calculate BERTScore(original, original) by using the same original sentence as both the hypothesis and reference for BERTScore, resulting in a score of 1 before normalization. This is meant to establish a maximum score of similarity as measured by the metric.\n\n> Regarding Figure 1(b), the calculation in the second row seems to be incorrect?\n\nThis is indeed our mistake. The second row should be BERTScore(original, reference) -> BERTScore(perturbed, reference): 0.325 -> -0.077(-0.402).\n\n> line152 C. What's the threshold for COMET and BLEURT?\n\nOur method of probing for self-inconsistency does not immediately apply to COMET and BLEURT as they are not symmetric measures (e.g. BLEURT(hypothesis, reference) is different from BLEURT(reference, hypothesis)). As described in Appendix C, we experimented with a symmetric variant of BLEURT under the same threshold as BERTScore.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The investigated problem is interesting, and I agree that some more convinced metrics should be applied to investigate the robustness of MT systems.\n2. The experiments lead to some insightful findings.", "reasons_to_reject": "1. The evaluation setting has some problems. \n* The length of the sentence may affect the evaluation results, i.e., change a word in a short sentence may lead to large difference in the metric score. Therefore, a standardization term may be needed when calculate the differences between metric scores.\n* Some points in this paper is not clear enough (see the questions below)\n\n", "questions_to_authors": "A.. Regarding Figure 1(a)\n* Could this phenomenon be affected by the length of the input sentence?\n\nB. Regarding Figure 1(b)\n* What does it mean by calculating Bert (original, original)?\n* The calculation in the second row seems to be incorrect?\n\nline152 C. What's the threshold for COMET and BLEURT?", "paper_topic_and_main_contributions": "This paper investigates the robustness of three prevailing machine translation metrics on sentences with adversarial perturbations. It comes up with two conclusions that 1) these metrics tend to over-penalize adversarially degraded translations; 2) they are inconsistent when evaluating different sentences, through experimenting with several adversarial attack methods on German-English translation dataset.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "line23: from-of"}, "extra_text": {}}, {"review_id": "fNi4mkAlnh", "reviewer": "Reviewer_JiPk", "summary": "", "questions": "- I don't really understand the results in Table 2. There are only at most 1~2% successful attacks for the self-inconsistency problem of BERTScore. Does this mean that the self-inconsistency problem is actually not severe at all for BERTScore and the conclusion doesn't hold?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the insightful feedback and questions. We respond to the questions and concerns in a breakdown format.\n\n> The two differences between BERTScore results in Figure 1(b) seem incorrect.\n\nThe difference on the second row is indeed a mistake on our side. The second row should be BERTScore(original, reference) -> BERTScore(perturbed, reference): 0.325 -> -0.077(-0.402). The numbers on the first row should be correct, with rounding to 1.860 and 2.127.\n\n> As the authors stated, the analyses are limited to De-En. It would be more convincing with more results on other language pairs and domains.\n\nWe agree that experiments on more language pairs would support more general conclusions, but suggest that such extensive investigations fall outside the scope of our short paper. We point out that it is not uncommon for similar works to include only one language pair (e.g. one of the suggested papers (Chen et al., WMT 2022) also only considers one language pair). As such, we leave investigation of different language pairs and domains to future work.\n\n> I don't really understand the results in Table 2. There are only at most 1~2% successful attacks for the self-inconsistency problem of BERTScore. Does this mean that the self-inconsistency problem is actually not severe at all for BERTScore and the conclusion doesn't hold?\n\nIt is true that self-consistency issues only occur for a small number of cases for BERTScore. However, we argue that despite their infrequent occurrence, such failure cases can still cause issues in real-world applications and likely cannot be detected using other methods. In addition, the trend is clear that BERTScore is more likely to be self-inconsistent under mask-filling and word-substitution attacks, which is in agreement with our conclusions. We also stress that this method has no human cost and is easily applicable to other embedding-based metrics, such as Word Mover’s Distance. \n\nFinally, we thank you for raising concerns about the writing of the paper and the reproducibility of the work. May we ask you to elaborate on these issues to help inform further revisions?\n\nReference: \n\n[Exploring Robustness of Machine Translation Metrics: A Study of Twenty-Two Automatic Metrics in the WMT22 Metric Task](https://aclanthology.org/2022.wmt-1.46) (Chen et al., WMT 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This paper aims at an important problem in NMT, i.e., the robustness of the current neural metrics for NMT, and conducts comprehensive analyses on three prevalent metrics.\n- The findings in this paper are interesting and may motivate future work on enhancing the robustness of the neural metrics for NMT.", "reasons_to_reject": "- The two differences between BERTScore results in Figure 1(b) seem incorrect, e.g., $1.860-2.127=-0.268?$ and $-0.077-0.325=-2.444?$\n- The writing of the paper needs to be improved. Some descriptions of the methods and details in the results are somewhat difficult to follow.\n- As the authors stated, the analyses are limited to De-En. It would be more convincing with more results on other language pairs and domains.", "questions_to_authors": "- I don't really understand the results in Table 2. There are only at most 1~2% successful attacks for the self-inconsistency problem of BERTScore. Does this mean that the self-inconsistency problem is actually not severe at all for BERTScore and the conclusion doesn't hold?", "paper_topic_and_main_contributions": "This paper probes the robustness of the current neural MT metrics (i.e., BERTScore, BLEURT, COMET) via adversarial attacks. The authors collect lots of human judgments for comparison and finally conclude that these metrics have overpenalization problem and BERTScore performs inconsistently on perturbed translations. Overall, the findings in this paper are somewhat instructive for the development of the robustness of current neural metrics but some statements still need to be clearer.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper investigates the robustness of machine translation (MT) metrics in the face of adversarial attacks. The primary contributions are the exploration of adversarial attacks that do not change translation semantics and the evaluation of MT metrics' responses to these attacks, particularly focusing on BERTScore, BLEURT, and COMET.

Based on the average scores, the paper is considered sound and has potential, but it is not without weaknesses. It falls short in terms of excitement, mainly due to perceived incremental novelty and the need for revisions.
In conclusion, while the paper addresses an important issue in the field of MT metrics, it should strive to provide stronger evidence for its claims, highlight its unique contributions, consider expanding the scope, and address the reviewers' concerns for a higher chance of acceptance.
Decision: Accept-Findings