Automatic Machine Translation Evaluation using
Source Language Inputs and Cross-lingual Language Model
Kosuke Takahashi1 , Katsuhito Sudoh1 ,,2 , Satoshi Nakamura1
1
Nara Institute of Science and Technology
2
PRESTO, Japan Science and Technology Agency{takahashi.kosuke.th0, sudoh, s-nakamura}@is.naist.jp

Abstract
We propose an automatic evaluation method ofmachine translation that uses source language
sentences regarded as additional pseudo references. The proposed method evaluates a translation hypothesis in a regression model. The
model takes the paired source, reference, andhypothesis sentence all together as an input. A
pretrained large scale cross-lingual language
model encodes the input to sentence-pair vectors, and the model predicts a human evaluation score with those vectors. Our experiments
show that our proposed method using Crosslingual Language Model (XLM) trained witha translation language modeling (TLM) objective achieves a higher correlation with humanjudgments than a baseline method that uses
only hypothesis and reference sentences. Additionally, using source sentences in our proposed method is confirmed to improve the evaluation performance.

1

Introduction

Automatic machine translation evaluation (MTE)
has been studied to substitute human evaluationin machine translation development because it is
low-cost, handy, and stable to use. Popular automatic MTE metrics such as BLEU (Papineni et al.,2002) calculate the evaluation score based on asurface-level similarity of a paired 1-to-1 reference
and translated hypothesis sentences. BLEU particularly evaluates the sentence similarity with the ngram word matching rate between a reference andhypothesis. However, the evaluation score drops
when a reference and hypothesis are dissimilar in
the surface even if they share the same meaning.
To counter this problem, METEOR (Banerjee
and Lavie, 2005) is proposed to mitigate the word
matching of synonyms with a synonym dictionary.
Yet still, with mitigation of word matching, surfacelevel similarity cannot fully compensate for seman-tics, thus word representation instead of word symbols is used in Word Mover’s Distance (Kusneret al., 2015) and bleu2vec (Tättar and Fishel, 2017).
Besides, sentence representation is known to bean efficient feature instead of word representation
because sentence vectors can represent more global
meanings. RUSE (Shimanaka et al., 2018) and
BERT (Devlin et al., 2019) based MTE, BERT regressor (Shimanaka et al., 2019), utilized sentence
representation and performed well on WMT17 Metric Shared Task (Bojar et al., 2017). The metrics
mentioned above compare a hypothesis translation
to a reference. However, a reference translation
represents only one possible translation and those
MTE metrics are unlikely to correctly evaluate all
candidates that share the same meanings of thereference or have fatally different meanings due
to a few translation errors. This problem can be
mitigated by the use of multiple reference translations as argued by Dreyer and Marcu (2012) andQin and Specia (2015), but preparing such multiplereferences is costly.
Hereby, we propose a method to incorporate
source sentence into MTE as another pseudoreference, since the source and reference sentences should be semantically equivalent. The proposed method uses Cross-lingual Language Model(XLM) (Lample and Conneau, 2019) to handle
source and target languages in a shared sentence
embedding space. The proposed method with
XLM trained with a translation language modeling (TLM) objective showed a higher correlation
with human judgments than a baseline method using hypothesis and reference sentences.

2

Related Work

Recent advances in sentence-level embedding havebeen used in MTE. Shimanaka et al. (2018) proposed an MTE framework called RUSE (Regressor
Using Sentence Embeddings), which uses sentence-Table 1: Available corpus size annotated with human judgments in WMT-2017 Metrics Shared Task (to-English)

WMT-2015
WMT-2016
WMT-2017
ALL

cs-en
500
560
560
1620

de-en
500
560
560
1620

fi-en
500
560
560
1620

lv-en
560
560

ro-en
560
560

level embeddings obtained by a large-scale pretrained model like InferSent (Conneau et al., 2017),
Quick Thought (Logeswaran and Lee, 2018), andUniversal Sentence Encoder (Cer et al., 2018). Its
regressor takes sentence vectors for a reference andtranslation hypothesis as inputs and returns a score,
which is trained to correlate well with human evaluation (Graham et al., 2015). RUSE achieved thebest correlation score with human judgments inthe WMT-2017 Metrics Shared Task (Bojar et al.,2017).
BERT regressor (Shimanaka et al., 2019) isa simple MTE metric based on BERT (Devlin
et al., 2019) encoder. It is composed of BERT
encoder and a multi-layer perceptron (MLP) regressor attached to the last layer of BERT. This
BERT encoder is a 12 layers bi-directional language model, referring to BERTbase (uncased)1 ,trained with masked language model (MLM) andnext sentence prediction (NSP). BERT regressor
surpassed RUSE on the WMT-2017 data.

3

Proposed method: Automatic
evaluation using XLM

ru-en
500
560
560
1620

tr-en
560
560
1120

zh-en
560
560

{de,ru,tr,zh}-en
1000
1680
2240
4920

all-en
2000
3360
3920
9280

The proposed method has two variants for theuse of source language sentences, as illustrated in
Figure 2. The first one called hyp+src/hyp+refuses two sentence-pair vectors for hypothesissource and hypothesis-reference, encoded by across-lingual language model independently. These
sentence-pair vectors are given to an MLP-based
regression model to predict the human evaluation
scores. This can be regarded as an ensemble modelusing a monolingual vector based on the reference
and a cross-lingual vector based on the source sentence. The other one called hyp+src+ref takes
a concatenation of hypotheses, source, and reference sentences as an input to a cross-lingual
language model to obtain a sentence-pair vector.
This sentence-pair vector is expected to be directly
learned to represent the quality of the translation hypothesis given two correct sentences aligned aside.

4

Experiments

We conducted experiments to evaluate the performance of the proposed method in MTE by comparing with some existing methods.
4.1

Setting

We propose an MTE method using source language
sentences as additional pseudo references. We use
cross-lingual language models called XLM (Lample and Conneau, 2019) to encode both source andtarget language sentences into an embedding vector.
XLM has three additional techniques to BERT:
language independent subword based on Byte Pair
Encoding (Sennrich et al., 2016), a language embedding layer, and a translation language modeling (TLM) objective that predicts masked words
from surrounding words or a paired translation.
The brief architecture of XLM is shown in Figure 1. (Lample and Conneau, 2019) reported that
XLM trained with TLM objective obtains better
performance than multilingual BERT(Devlin et al.,2019) on the XNLI cross-lingual classificationtask(Conneau et al., 2018).

The experiments were conducted with a corpusof all language pairs to English translation fromsegment-level WMT2017 Metrics Shared Task (Bojar et al., 2017). We split sentences in WMT15 andWMT16 to training and development data with theratio of 9:1 and whole sentences in WMT17 areused for evaluation of MTE methods. The corpus
size for each language pair is shown in Table 1.
We used two different models from all available
XLM family models2 : XLM15 pretrained by MLM
and TLM, and XLM100 pretrained only by MLM.
XLM15 is expected to perform better by the paired
bilingual training of TLM, but the number of available languages is limited. XLM15 is compatible
with only German, Russian, Turkish, and Chinese
in the corpus, which confines the model to partial
access to the corpus. On the other hand, XLM100

1
https://github.com/google-research/
bert

XLM

2

https://github.com/facebookresearch/sentence-pair vector
hiddenvectors

V1,1

V[/s]

V1, ···

V1,n

V[/s]

V2,1

V[/s]

V2, ···

V2,m

V[/s]

token2, ···token2,m[/s]n layer transformer encoder
input
tokens[/s]positional
embeddinglanguageembeddingtoken1,1[/s]+token1,n+token1, ···

+

+

+

+

+

+

POS0

POS1

POS ···POSn

POSn+1

POSn+3++

POS ···

POSn+m+2++

POSn+2

POSn+m+3

+

+

+

+

+

+

L1

L1

L1

L1

L1

L2

L2

L2

L2

L2

+

[/s]

token2,1+

Figure 1: The architecture of XLM sentence-pair encoder
Table 2: Pearson’s correlation scores in the small corpus ({de,ru,tr,zh}-en)

evaluation score

MLP

concatenation vhyp+src, vhyp+src
sentence-pair
vector vhyp+src

sentence-pair
vector vhyp+src

evaluation score

MLP

sentence-pair vector vhyp+src+refsentence-pair encoder
hypothesis+ source

hypothesis+ reference(a) hyp+src/hyp+refsentence-pair encoder
hypothesis + source + reference(b) hyp+src+refFigure 2: Two variants in the proposed method

is compatible with all language pairs in the corpus,
while it lacks supervised bilingual pretraining.
Thus the experiments had two corpus settings;
One was a small corpus including {German (de),Russian (ru), Turkish (tr), and Chinese (zh)} toEnglish (en) language pairs, and the other was awhole corpus including {Czech (cz), German (de),Finnish (fi), Latvian (lv), Romanian (ro), Russian(ru), Turkish (tr), and Chinese (zh)} to English
language pairs. The evaluation was conducted with
Pearson’s correlation to human judgments in thetest set.
We compared the proposed methods with SentBLEU (Bojar et al., 2017), BERT regressor (Shimanaka et al., 2019) by our implementation. We
also conducted experiments using multilingual
BERT, BERTmulti (cased), to contrast language
models and experiments limiting the model’s input into source-hypothesis only and referencehypothesis only to study the impact of adding
source sentences.
The fine-tuning on the proposed methods and
BERT regressor was based on Mean Squared Error

SentBLEU
BERT regressor
multi-BERT
hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref
XLM15
hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref
XLM100hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref

de-en
0.432
0.729

ru-en
0.484
0.757

tr-en
0.538
0.770

zh-en
0.512
0.702

avg
0.484
0.740

0.661
0.625
0.520
0.627

0.739
0.713
0.558
0.688

0.768
0.725
0.601
0.718

0.735
0.691
0.559
0.685

0.726
0.689
0.559
0.679

0.753
0.729
0.722
0.716

0.795
0.769
0.763
0.787

0.771
0.767
0.761
0.746

0.763
0.725
0.668
0.714

0.771
0.747
0.728
0.741

0.643
0.635
0.464
0.631

0.722
0.695
0.450
0.718

0.725
0.715
0.557
0.695

0.712
0.661
0.449
0.702

0.701
0.677
0.480
0.687

(MSE) loss in the training set, back-propagatedto both MLP and XLM in order. The hyperparameters were selected through grid search forthe following parameters. Since models are affected by randomness in training, we ran ten experiments for each of the settings and report results of
the average scores.
• Optimizer : {Adam}• Learning rate : {3e-5, 1e-5, 9e-6, 7e-6}• Number of epochs : {1, ...,20}• Dropout rate: {0.1}• Batch size : {2, 4, 8, 16}
4.2

Results

The results of each small corpus and whole corpus
experiments are shown in Tables 2 and 3, respectively. Note that XLM15 was not included in theTable 3: Pearson’s correlation scores in the whole corpus (all-en)

SentBLEU
BERT regressor
multi-BERT
hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref
XLM100hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref

cs-en
0.435
0.776

de-en
0.432
0.753

fi-en
0.571
0.863

lv-en
0.393
0.818

ru-en
0.484
0.788

tr-en
0.538
0.803

zh-en
0.512
0.767

avg
0.481
0.795

0.743
0.714
0.599
0.720

0.688
0.670
0.525
0.681

0.824
0.802
0.699
0.823

0.812
0.774
0.681
0.806

0.772
0.754
0.586
0.744

0.796
0.758
0.633
0.768

0.751
0.722
0.571
0.748

0.769
0.742
0.613
0.756

0.712
0.698
0.510
0.692

0.681
0.666
0.531
0.666

0.822
0.818
0.672
0.813

0.810
0.795
0.662
0.788

0.756
0.742
0.543
0.743

0.773
0.765
0.602
0.746

0.745
0.727
0.537
0.714

0.757
0.745
0.580
0.738

whole corpus experiment due to its limited language coverage.
Performance of each language model As we
can see from Table 2, the proposed method using
XLM15 with hyp+src/hyp+ref structure surpassed
BERT regressor in the small corpus. However,XLM100 did not work well in the experiments;
its results were much worse than the others in thesmall corpus condition, and it did not compete with
BERT regressor in the whole corpus condition asshown in Table 3. One possible reason is the lackof TLM objective pretraining in XLM100. Since
the TLM task allows the model for learning semantically equivalent cross-lingual sentences directly,
the TLM task can be concluded to be importantfor using source sentences in MTE. The results ofmultilingual BERT are worse than BERT regressor
and XLM15, but close to XLM100 or slightly better in general. From this comparison of pretraining
objectives and language models, we report that our
proposed method is influenced by the multilingualism of a language model.
hyp+src/hyp+ref VS hyp+src+ref The results from Table 2 and Table 3 shows thathyp+src/hyp+ref structure is better thanhyp+src+ref in most of the conditions, although we expected hyp+src+ref to perform better
because it can access 2 translation answers as
references at the same time. This is probably
because both of XLM and multilingual BERT was
not pretrained to handle 3 sentences in a sequence.
However, it is perhaps possible that hyp+src+refsurpasses hyp+src/hyp+ref when a fine-tuning
corpus is large enough.

Table 4: Pearson’s correlation score in the halved small
corpus {de,ru,tr,zh}-en

BERT regressor
multi-BERT
hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref
XLM15
hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref
XLM100hyp+src/hyp+ref
hyp+src+ref
hyp+src
hyp+ref

de-en
0.686

ru-en
0.731

tr-en
0.753

zh-en
0.691

avg
0.715

0.583
0.563
0.384
0.574

0.670
0.664
0.509
0.651

0.720
0.704
0.629
0.722

0.675
0.698
0.482
0.693

0.662
0.657
0.501
0.660

0.712
0.679
0.570
0.682

0.744
0.748
0.635
0.707

0.740
0.706
0.654
0.708

0.690
0.666
0.616
0.700

0.722
0.700
0.619
0.699

0.594
0.605
0.321
0.559

0.676
0.644
0.408
0.631

0.706
0.676
0.447
0.675

0.686
0.639
0.431
0.668

0.666
0.631
0.402
0.633

Contribution of adding source sentences Every model with hyp+src/hyp+ref achieved a betterscore than both of hyp+src and hyp+ref, whichindicates that source sentences contribute to theimprovement of evaluation.

5

Analysis

Training data size We conducted another experiment to see the effect of the training corpus size
using randomly halved {de, ru, tr, zh}-en small
corpus. From the results in Table 4, BERT regressor stably performed well even when the numberof training data is about 1000 sentences, however,XLM15, XLM100, and multilingual BERT deteriorated their performances. Since our proposed
hyp+src/hyp+ref is an ensemble model and has amore complex network structure than hyp+ref, theTable 5: Pearson’s correlation score for low and high human judgement score range in the small corpus({de,ru,tr,zh}-en)
DA ≥ 0.0

DA < 0.0

0.728
0.728
0.686
0.539
0.672
0.768
0.740
0.679
0.735
0.703
0.662
0.522
0.685

0.553
0.535
0.512
0.339
0.493
0.580
0.560
0.469
0.534
0.535
0.501
0.337
0.521

0.464
0.494
0.423
0.316
0.384
0.529
0.497
0.430
0.458
0.419
0.389
0.292
0.378

6

Conclusion

In this paper, we proposed an MTE framework thatutilizes source sentences using XLM. We show

1.5
evaluation score

0.5
0.0
0.5
1.0
1.5

1.0

0.5 0.0 0.5
DA score

1.0

1.0
0.5
0.0
0.5
1.0

1.5

BERT regressor(a)r = 0.728

(b)

1.0

1.0

0.5

0.5

evaluation score

Evaluation errors In order to see when models make errors to evaluate hypothesis sentences,
we plot scatters of evaluation scores and humanjudgement scores (DA scores) in Figure 3(a), Figure 3(b), Figure 3(c), and Figure 3(d). Although
in comparison, the evaluation scores of our bestmodel XLM15 hyp+src/hyp+ref are set more linearly than the baseline BERT regressor, the scoresof all models seem much dispersed in the low DA
area (DA < 0.0). This indicates that all evaluation
models listed here tend to miss-evaluate when ahypothesis is poor. Furthermore, we show Pearson’s correlation score for each of high and low
DA score range in Table 5. As we confirmed in
the scatter figures, the correlation scores of low
DA is low; evaluation models work poorly when
hypotheses are poor. However, the reduction rateof Pearson’s scores from high DA to low DA issmall with XLM15 hyp+src/hyp+ref and hyp+src.
Therefore adding source sentences has an impact
to stabilize the evaluation performance when hypotheses are low-quality.

Reduction rate of Pearson’s scorefrom DA ≥ 0.0 to DA < 0.0 (%)
16.10
7.77
17.51
6.88
22.05
8.68
11.12
8.46
14.20
21.75
22.29
13.42
27.48

1.0
evaluation score

use of XLM and multi-BERT with the proposed
method requires a certain amount of training data.
Therefore, our proposed method deteriorated in thehalved small corpus setting. On the other hand,monolingual BERT and hyp+ref benefits from thelarge corpus because it has no language limitation
other than English.

evaluation score

BERT regressor
multi-BERT hyp+src/hyp+refmulti-BERT hyp+src+ref
multi-BERT hyp+src
multi-BERT hyp+ref
XLM15 hyp+src/hyp+ref
XLM15 hyp+src+ref
XLM15 hyp+src
XLM15 hyp+ref
XLM100 hyp+src/hyp+ref
XLM100 hyp+src+ref
XLM100 hyp+src
XLM100 hyp+ref

All

0.0
0.5
1.0

1.5

1.0

0.5 0.0 0.5
DA score

1.0

XLM15 hyp+src(c)r = 0.679

1.5

1.5

1.0

0.5 0.0 0.5
DA score

1.0

1.5

XLM15 hyp+src/hyp+refr = 0.768

0.0
0.5
1.0
1.5

1.0

0.5 0.0 0.5
DA score

1.0

1.5

XLM15 hyp+ref(d)r = 0.735

Figure 3: Scatter plots of human judgement scores (DAscores) and evaluation scoresthat the proposed method with TLM-trained XLM
showed a higher correlation with human judgments
than the baseline method in the small corpus condition and stabilize the evaluation performance regardless of the quality of translation sentences byusing additional source sentences. We also investigated why our proposed method worked poorly
in the other conditions and found the importanceof TLM training. In future work, we will workaround the problem of evaluation errors in the low
DA range.

Acknowledgments
This work is supported by JST PRESTO (JPMJPR1856).

References
Satanjeev Banerjee and Alon Lavie. 2005. METEOR:
An Automatic Metric for MT Evaluation with Improved Correlation with Human Judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
Ondřej Bojar, Yvette Graham, and Amir Kamran. 2017.
Results of the WMT17 Metrics Shared Task”. In
Proceedings of the Second Conference on MachineTranslation, pages 489–513, Copenhagen, Denmark.
Association for Computational Linguistics.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua,
Nicole Limtiaco, Rhomni St. John, Noah Constant,
Mario Guajardo-Cespedes, Steve Yuan, Chris Tar,
Brian Strope, and Ray Kurzweil. 2018. Universal
sentence encoder for English. In Proceedings ofthe 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations,pages 169–174, Brussels, Belgium. Association for
Computational Linguistics.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loı̈c
Barrault, and Antoine Bordes. 2017. Supervised
Learning of Universal Sentence Representations
from Natural Language Inference Data. In Proceedings of the 2017 Conference on Empirical Methods
in Natural Language Processing, pages 670–680,
Copenhagen, Denmark. Association for Computational Linguistics.

2015 Conference of the North American Chapterof the Association for Computational Linguistics:
Human Language Technologies, pages 1183–1191,
Denver, Colorado. Association for Computational
Linguistics.
Matt J. Kusner, Yu Sun, Nicholas I. Kolkin, and Kilian Q. Weinberger. 2015. From Word Embeddings
to Document Distances. In Proceedings of the 32nd
International Conference on International Conference on Machine Learning - Volume 37, pages 957–
966.
Guillaume Lample and Alexis Conneau. 2019. Crosslingual language model pretraining. arXiv preprint,
abs/1901.07291.
Lajanugen Logeswaran and Honglak Lee. 2018. An
efficient framework for learning sentence representations. In Proceedings of the 6th International Conference on Learning Representations (ICLR 2018).
Kishore Papineni, Salim Roukos, Todd Ward, and WeiJing Zhu. 2002. B LEU: a Method for Automatic
Evaluation of Machine Translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311–318,
Philadelphia, Pennsylvania, USA. Association for
Computational Linguistics.
Ying Qin and Lucia Specia. 2015. Truly Exploring
Multiple References for Machine Translation Evaluation. In Proceedings of the 18th Annual Conferenceof the European Association for Machine Translation, pages 113–120, Antalya, Turkey.

Alexis Conneau, Ruty Rinott, Guillaume Lample Adina Williams, Samuel R. Bowman, Holger Schwenk,
and Veselin Stoyanov. 2018. Xnli: Evaluating crosslingual sentence representations. pages 2475–2485.
In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing Association for Computational Linguistics.

Rico Sennrich, Barry Haddow, and Alexandra Birch.
2016. Neural Machine Translation of Rare Words
with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational
Linguistics (Volume 1: Long Papers), pages 1715–
1725, Berlin, Germany. Association for Computational Linguistics.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. BERT: Pre-training ofdeep bidirectional transformers for language understanding. In Proceedings of the 2019 Conferenceof the North American Chapter of the Association
for Computational Linguistics: Human Language
Technologies, Volume 1 (Long and Short Papers),
pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.

Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru
Komachi. 2018. RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation. In Proceedings of the Third Conference on Machine Translation: Shared Task Papers,
pages 751–758, Belgium, Brussels. Association for
Computational Linguistics.

Markus Dreyer and Daniel Marcu. 2012. HyTER:
Meaning-Equivalent Semantics for Translation Evaluation. In Proceedings of the 2012 Conference of theNorth American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 162–171, Montréal, Canada. Association
for Computational Linguistics.
Yvette Graham, Timothy Baldwin, and Nitika Mathur.
2015. Accurate Evaluation of Segment-level Machine Translation Metrics. In Proceedings of the

Hiroki Shimanaka, Tomoyuki Kajiwara, and Mamoru
Komachi. 2019.
Machine Translation Evaluation with BERT Regressor.
arXiv preprint,
abs/1907.12679.
Andre Tättar and Mark Fishel. 2017. bleu2vec: thePainfully Familiar Metric on Continuous Vector
Space Steroids. In Proceedings of the Second Conference on Machine Translation, pages 619–622,
Copenhagen, Denmark. Association for Computational Linguistics.