Multilingual Machine Translation Evaluation Metrics Fine-tuned on Pseudo-Negative Examples for WMT 2021 Metrics Task
Kosuke Takahashi1, Yoichi Ishibashi 1, Katsuhito Sudoh 1,2, Satoshi Nakamura 1
1Nara Institute of Science and Technology (NAIST), 2PRESTO, Japan Science and Technology Agency{takahashi.kosuke.th0, ishibashi.yoichi.ir3, sudoh, s-nakamura}@is.naist.jp

Abstract• Previous studies (Takahashi et al., 2020 and Sudoh et al., 2021) and empirical experiments show that BERT-family model based metrics suffer from evaluating low quality translations.
• We prepared a pseudo-negative corpus for fine-tuning a metric model beforehand by transferring words’ attributes into reversed ones.
• Experiments on the development set showed that models trained on WMT15-17/WMT18-20 and the pseudo negatives performed better than the plain ones.

Methodology

Training Procedureevaluation score

Fine-tuning on
WMT15-16’s DA

MLP
Evaluation Modelconcatenation vhyp+src, vhyp+refsentence-pair
vector vhyp+src

sentence-pair
vector vhyp+ref

XLMRoBERTa
hypothesis+ source

hypothesis+ reference

C-SPECModel Architecture

Fine-tuned Modelwith Initialized MLP

Fine-tuning on
WMT15-17’s DA,
WMT18-20’s DA, or
WMT15-20’s DA

Fine-tuned Modelwith Initialized MLP

Classification
Fine-tuning on
Pseudo-NegativeExamples

Making of Pseudo-Negative Examples• Apply reflection-based word attribute transfer (Ishibashi et al., 2020) to nounsof the same corpus used to fine-tune beforehand.
• Only nouns with an attribute of sex (male/female) and antonym aretransferred to the target nouns with the reversed attribute.

Fine-tuned Modelwith Initialized MLP

Fine-tuning on
WMT20’sSegment-Level
MQM

C-SPEC

Fine-tuning on
WMT20’s
Segment-Level
MQM

C-SPECpn

Make 3 Categories of Hypothesis for Classification
Example of Attribute Transfer Transfer Words
Original Hypothesis+ Reference

Reference+ Reference

Attribute Transferred Hypothesis+ Reference

w/ Antonym ''It is our duty to change at''It is our duty to remain at
his sides'', he said, to whisper.
his sides'', he said, to applause.

Results on WMT20’s Segment-Level MQM Human Evaluation
0.800

Metricen-dezh-en

ave

allC-SPEC trained on WMT15-170.609

0.7730.691

0.787

C-SPEC trained on WMT18-20

0.612

0.805

0.708

0.813

C-SPEC trained on WMT15-20

0.603

0.798

0.700

0.808

C-SPECpn
trained on WMT15-17C-SPECpn
trained on WMT18-20
C-SPECpn
trained on WMT15-20

0.626

0.809

0.717

0.817

0.619

0.824

0.721

0.829

0.309

0.715

0.512

0.724

• C-SPECpn of WMT15-17 and WMT18-20 overcame the plain models.

Pearson's correlation

Pearson’s correlation with MQM scores

C-SPECpn on 15-17/18-20
had higher correlation thanplain model

0.600
0.400
0.200

0.000-0.200
segment-level MQM score (human evaluation score)-0.400[-25.0, -5.0)
1582[-5,0 -1.0)
2258

C-SPEC trained on WMT15-17
C-SPEC trained on WMT18-20
C-SPEC trained on WMT15-20[-1.0, -0.1)
1518[-0.1, 0.0)78

C-SPECpn trained on WMT15-17
C-SPECpn trained on WMT18-20
C-SPECpn trained on WMT15-20

Pearson’s correlation for each small MQM score range• Among the models, the best score was archived by C-SPECpn trained on WMT18-20.
• C-SPECpn of WMT15-17 and WMT18-20 performed better in the MQM range of [-25.0, -5.0) and [-0.1, 0.0].

This work is supported by JST PRESTO (JP-MJPR1856).