Multilingual Machine Translation Evaluation Metrics Fine-tuned on
Pseudo-Negative Examples for WMT 2021 Metrics Task
Kosuke Takahashi1 , Yoichi Ishibashi1 , Katsuhito Sudoh1,2 , Satoshi Nakamura1
1Nara Institute of Science and Technology
2PRESTO, Japan Science and Technology Agency{takahashi.kosuke.th0, ishibashi.yoichi.ir3, sudoh, s-nakamura}@is.naist.jp

Abstract
This paper describes our submission to theWMT2021 shared metrics task. Our metricis operative to segment-level and system-leveltranslations. Our belief toward a better metricis to detect a significant error that cannot be
missed in the real practice cases of evaluation.
For that reason, we used pseudo-negative examples in which attributes of some words aretransferred to the reversed attribute words, andwe build evaluation models to handle such serious mistakes of translations. We fine-tune a
multilingual largely pre-trained model on theprovided corpus of past years’ metric task andfine-tune again further on the synthetic negative examples that are derived from the samefine-tune corpus. From the evaluation resultsof the WMT21’s development corpus, finetuning on the pseudo-negatives using WMT1517 and WMT18-20 metric corpus achieved abetter Pearson’s correlation score than the one
fine-tuned without negative examples. Our
submitted models are named C-SPEC (Crosslingual Sentence Pair Embedding Concatenation) and C-SPECpn, are the plain model using WMT18-20 and the one additionally finetuned on negative samples, respectively.

1

Introduction

Recent studies of automatic evaluation is mostly
based on the family models of BERT (Devlin et al.,2019). BERTscore (Zhang et al., 2020), BLEURT(Sellam et al., 2020), COMET (Rei et al., 2020)
have shown a strong correlation with human judgement scores. However, we reported in our previous
study (Takahashi et al., 2020), it is hard for BERT
based metrics to correctly evaluate the translation
errors that are annotated with low Direct Assessment (DA) score.
Upon the problems of evaluating poor qualitytranslations, Sudoh et al. (2021) has attempted tosolve the problem by creating a different humanannotation set and corpus. Compared to DA, theiridea is to make a clear definition of critical translation errors and let models learn the critical errorsthat can cause a serious misunderstanding.
Following the idea, we used pseudo-negative examples to train the evaluation model. Since, empirically, the cases of the evaluation failure happens frequently with the nouns translation errors, we generated pseudo-negative sentences by transferring
the attribute of nouns with Word Attribute Transfer(Ishibashi et al., 2020). This system is based on ourprevious work (Takahashi et al., 2020), with an extension with fine-tuning with the pseudo-negativeexamples.

2Related Work

BERTscore (Zhang et al., 2020), BERT regressor (Shimanaka et al., 2019), BLEURT (Sellamet al., 2020), and COMET (Rei et al., 2020) areapplications of BERT to the machine translation
evaluation. BERTscore measures the similarity ofreference and hypothesis translation by the cosinesimilarity of the token embeddings for each token in the reference and hypothesis. It uses a pretrained BERT model without fine-tuning on evaluation data. Instead, BERT regressor and BLEURT
are fully parameterized and require a human annotated evaluation corpus to fine-tune the models.
Both metrics have the same model architecture; alinear layer is attached on top of the BERT encoder.
They encode a paired reference and hypothesis sentence with BERT and predict the human evaluation
score. Additionally, BLEURT conducts warm-up
training of BERT before fine-tuning on an evaluation corpus. The model architecture of our submission is similar to BERT regressor and BLEURT,
but its uniqueness comes from using the synthetic
negative data to fine-tune the models to evaluatepoor translations better.

evaluation score

MLPconcatenation vhyp+src, vhyp+refsentence-pair
vector vhyp+src

sentence-pair
vector vhyp+ref

XLMRoBERTa
hypothesis+ source

hypothesis+ reference

Figure 1: Architecture of C-SPEC3

Our system

3.1

Model architecture

We extend the BERT regressor (Shimanaka et al.,2019) and use a cross-lingual language models,
XLMRoBERTa (Conneau et al., 2020), to utilize
a source sentence as a pseudo reference. In orderto obtain a sentence-pair vector from source language and target language sentences together, ourmodel encodes input sentences with a cross-lingual
language model instead of monolingual BERT.
The model procedure is illustrated in the Figure 1. Our metric, called C-SPEC (Cross-lingual
Sentence Pair Embedding Concatenation), uses
paired inputs of hypothesis-source and hypothesisreference.
It introduces another vector for
hypothesis-source (vhyp+src ) in addition to the standard one for hypothesis-reference pair (vhyp+ref )to make an ensemble evaluation. Both sentence
vectors are concatenated and used to predict theevaluation score in multi-layer perceptron (MLP).
At first of the evaluation process, the cross-lingual
language model encodes an input sentence into asentence-pair vector. Then, using the sentence-pairvector, a MLP outputs the final evaluation score inregression manner. In training, we used standardized z score of DA (Direct Assessment; Grahamet al. (2013)) as the ground truth and updated themodel parameters by backpropagation (Rumelhartet al., 1986) with Mean Squared Error (MSE) Loss.
Our model was trained by the following steps.

Firstly, in order to speed up and stabilize the training procedure, our models were trained on the corpus of WMT2015-2016’s DA. Secondly, the models were additionally trained on WMT2015-2017’sDA, WMT2018-2020’s DA, or WMT2015-2020’sDA. Thirdly, they were fine-tuned with the pseudo
negative examples. Lastly, they were fine-tuned
again on the WMT20’s MQM segment-level corpus.
In each step of fine-tuning, we initialize theoutput-layer and only inherit the parameters ofXLMRoBERTa. We tried three different conditions in the second step because DA corpus after
WMT2018 is relatively noisy, and removing thosedata may play out well.
In the system-level evaluation, we simply averaged the segment-level evaluation scores for eachsystem.
3.2

Word Attribute Transfer

We used the reflection-based word attribute transfer Ishibashi et al. (2020) for data augmentation.
This transfer can make conversion of words into acertain word attribute, such as queen to king, usingparameterized mirrors composed of two multi-layer
perceptrons.
For the pseudo-negative hypothesis generation,
we used two types of word attribute transfer in gender (male/female) and antonym. The word attributetransfer was applied onto all the words in an input
sentence, and words having a target attribute wererewritten into their transferred counterparts while
those that were not related to the target attributewere kept unchanged. For example, a sentence “Itis our duty to remain at his sides”, he said, to applause is transferred into “It is our duty to change
at his sides”, he said, to whisper, by the antonymtransfer. Note that the word attribute transfer may
not make any changes on an input sentence when
all the words were identified as non-related words.
We eliminated such sentences from our pseudonegative examples.
3.3

Fine-tuning using pseudo-negative
examplesOur pseudo-negative examples were obtained fromthe reference sentences in the training corpus of allEnglish that was used to firstly fine-tune a model,
because the word attribute transfer model works
only in English. However, we did not have any
DA scores on these pseudo-negative examples. So,
we used them to fine-tune the evaluation models

in classification manner. We introduced a different
output layer on the top of the model illustrated
in Figure 1 to classify an input example into thefollowing categories:
1. A hypothesis is the same as its original systemtranslation.

WMT15-20 with negative examples dropped its
performance in the [-25.0, -5.0) range comparedto the plain model. We assume the reason of thescore drop is that the model was overly fine-tuned
to the high quality translations, as it can be seenthat the Pearson’s correlation score in the [-0.1, 0.0]
improved tremendously.

2. A hypothesis is the same as its reference.
3. A hypothesis is from the pseudo-negative examples
In the fine-tuning, we used three types of inputs
corresponding to the classes above, and the models
were trained to discriminate them. We expected
such fine-tuned models to identify the serious
word choice translation errors given in the pseudonegative examples. We call the metric trainedusing the pseudo-negative examples C-SPECpn(pn:psuedo-negative).

4

Segment-level evaluation experiments

Our experiment was conducted on the development
data for WMT21 metric task, which is randomly
selected 10% of WMT20 MQM segement-levelcorpus. All the results were calculated by the Pearson’s correlation with the MQM segment scores.
4.1

Results

The results of the WMT20 MQM segment-level
corpus are shown in Table 1.
From the results, the models trained on negative examples of WMT15-17 and WMT18-20 overcame the plain models in Pearson’s correlation.
Among the models, the best score was archived
by the one trained on WMT18-20 with the negative examples. Although WMT15-20 is a larger
corpus than WMT18-20, the score of plain models
was negligible at best, and the model trained onWMT18-20 and with negative examples did notovercome the plain one.
In order to figure out whether and how finetuning on the negative examples had impact on
the evaluation performance, we calculated the Pearson’s correlation for each small chunk of segmentlevel MQM scores and visualized the gap between
models’ outputs in Figure 2. Both the models
trained on WMT15-17 and WMT18-20 with negative examples performed better in the MQM rangeof [-25.0, -5.0) and [-0.1, 0.0]. This suggests thatusing negative examples can improve the performance of evaluating high and critically low quality translations. However, the model trained on

5Conclusion

In this paper, we presented a BERT-based multilingual evaluation metric that is boosted by pseudonegative examples to evaluate poor translations
more precisely. Our model leverages our previous work Takahashi et al. (2020) and have shown
an improvement of Pearson correlation when finetuning on the synthetic examples in the WMT15-17
and WMT18-20 corpus settings.

Acknowledgements
This work is supported by JST PRESTO (JPMJPR1856). With the support of JST, we usedRIKEN’s miniRAIDEN in a part of the experimentsin this work.

References
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,Vishrav Chaudhary, Guillaume Wenzek, FranciscoGuzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised
cross-lingual representation learning at scale. In
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8440–8451, Online. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers),
pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yvette Graham, Timothy Baldwin, Alistair Moffat, andJustin Zobel. 2013. Continuous measurement scales
in human evaluation of machine translation. In Proceedings of the 7th Linguistic Annotation Workshopand Interoperability with Discourse, pages 33–41,
Sofia, Bulgaria. Association for Computational Linguistics.
Yoichi Ishibashi, Katsuhito Sudoh, Koichiro Yoshino,and Satoshi Nakamura. 2020. Reflection-based
word attribute transfer. In Proceedings of the58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop,

Table 1: Pearson’s correlation with MQM segment scores in WMT2020. C-SPEC stands for a plain model finetuned without negative examples. C-SPECpn is a model fine-tuned on negative examples.

model
C-SPEC w/ WMT15-17
C-SPEC w/ WMT18-20
C-SPEC w/ WMT15-20
C-SPECpn w/ WMT15-17
C-SPECpn w/ WMT18-20
C-SPECpn w/ WMT15-20

en-de0.609
0.612
0.603
0.626
0.6190.309

zh-en
0.773
0.805
0.798
0.809
0.824
0.715

avg
0.691
0.708
0.700
0.717
0.721
0.512

all
0.787
0.813
0.808
0.817
0.829
0.724

0.800

Pearson's correlation

0.600
0.400
0.200
0.000
-0.200-0.400[-25.0, -5.0)
1582[-5,0 -1.0)
2258[-1.0, -0.1)
1518[-0.1, 0.0)78

segment-level MQM score (human evaluation score)
C-SPEC trained on WMT15-17

C-SPECpn trained on WMT15-17

C-SPEC trained on WMT18-20

C-SPECpn trained on WMT18-20

C-SPEC trained on WMT15-20C-SPECpn trained on WMT15-20

Figure 2: Pearson’s correlation for each small segment-level MQM ranges. The amount of each segment is written
below the range description.

pages 51–58, Online. Association for ComputationalLinguistics.
Ricardo Rei, Craig Stewart, Ana C Farinha, and Alon
Lavie. 2020. COMET: A neural framework for MT
evaluation. In Proceedings of the 2020 Conferenceon Empirical Methods in Natural Language Processing (EMNLP), pages 2685–2702, Online. Association for Computational Linguistics.
David Rumelhart, Geoffrey Hinton, and RonaldWilliams. 1986. Learning Representations by Backpropagating Errors. Nature, 323(6088):533–536.
Thibault Sellam, Dipanjan Das, and Ankur Parikh.
2020. BLEURT: Learning robust metrics for textgeneration. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics,
pages 7881–7892, Online. Association for Computational Linguistics.
Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru
Komachi. 2019.
Machine Translation Evalu-ation with BERT Regressor.
abs/1907.12679.

arXiv preprint,

Katsuhito Sudoh, Kosuke Takahashi, and Satoshi Nakamura. 2021.
Is this translation error critical?:Classification-based human and automatic machinetranslation evaluation focusing on critical errors. InProceedings of the Workshop on Human Evaluationof NLP Systems (HumEval), pages 46–55, Online.
Association for Computational Linguistics.
Kosuke Takahashi, Katsuhito Sudoh, and Satoshi Nakamura. 2020. Automatic machine translation evaluation using source language inputs and cross-lingual
language model. In Proceedings of the 58th Annual
Meeting of the Association for Computational Linguistics, pages 3553–3558, Online. Association for
Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q.
Weinberger, and Yoav Artzi. 2020. Bertscore: Evaluating text generation with bert. In International
Conference on Learning Representations.