Unbabel’s Participationin the WMT17 Translation Quality Estimation Shared Task
André F. T. Martins
Fabio N. Kepler
Unbabel & Instituto de Telecomunicações
UnbabelLisbon, Portugal
University of Pampa, Alegrete, Brazil
andre.martins@unbabel.com
kepler@unbabel.com
José L. Monteiro
UnbabelLisbon, Portugal
jose@unbabel.com
Abstract
This paper presents the contribution of the
Unbabel team to the WMT 2017 Shared
Task on Translation Quality Estimation.
We participated on the word-level andsentence-level tracks. We describe our
two submitted systems: (i) S TACKED QE,
a “pure” QE system, trained only on
the provided training sets, which is astacked combination of a feature-rich sequential linear model with a neural network, and (ii) F ULL S TACKED QE, which
also stacks the predictions of an automatic
post-editing system, trained on additional
data. When evaluated on the EnglishGerman and German-English datasets,
F ULL S TACKED QE achieved word-level
F1MULT scores of 56.6% and 52.9%, andsentence-level correlation Pearson scoresof 64.1% and 62.6%, respectively. Our
system ranked second in both tracks, being statistically indistinguishable from thebest system in the word-level track.

1

Introduction

Quality estimation is the task of evaluating a translation system’s quality without access to referencetranslations (Blatz et al., 2004; Specia et al., 2013).
This paper describes the contribution of the Unbabel team to the Shared Task on Sentence-Level andWord-Level Quality Estimation (QE Tasks 1 and
2) at the 2017 Conference on Statistical MachineTranslation (WMT 2017).
In the word-level task, the goal is to predict theword-level quality of machine translated text, by
assigning a label of OK or BAD to each word in

the translation. The sentence-level task attempts
to predict the HTER of each sentence, along witha ranking of the sentences. Two language pairs
and domains are considered: English-German (ITdomain) and German-English (medical domain).
Our submission is largely based on the approachthat we have recently proposed in Martins et al.
(2017), which ensembles a “pure” quality estimation system with predictions derived from an automatic post-editing system. The focus was on
developing a word-level system, and to use theword label predictions to predict the sentencelevel HTER.
Our system architecture is described in full detail in the following sections. We first describe
our “pure” QE system (§2), which consists of aneural model (N EURAL QE) stacked into a linear feature-rich classifier (L INEAR QE). Then, we
train an APE system (using a large amount of artificial “roundtrip translations”) and adapt it to predict word-level quality labels (yielding A PE QE,§3). We show that the pure and the APE-based QE
system are highly complementary (§4): our bestsystem is a stacked combination of L INEAR QE,
N EURAL QE, and A PE QE. By employing a simple
word-to-sentence conversion, we adapt our systems to sentence-level QE. Overall, we achieve
word-level F1MULT scores of 56.6% and 52.9% andsentence-level Pearson scores of 64.1% and 62.6%
for English-German and German-English, respectively.
The following external resources were used:
part-of-speech tags and extra syntactic dependency information were obtained with TurboTagger and TurboParser (Martins et al., 2013),1
1
Publicly available on http://www.cs.cmu.edu/
ark/TurboParser/.
˜

2softmax

OK/BAD

2 x FF
BiGRU

100 + 50...
...

2 x FF
BiGRU

100
2 x 200

...
...

200

2 x FF2 x 400

embeddings

trained on the Penn Treebank (for English) and on
the version of the German TIGER corpus used in
the SPMRL shared task (Seddah et al., 2014). For
the neural models, we used pre-trained word embeddings from Polyglot (Al-Rfou et al., 2013).
For our F ULL S TACKED QE submission, we also
use additional data to train the APE-based QE systems: for English-German, the set of 500K artificial roundtrip translations provided by JunczysDowmunt and Grundkiewicz (2016), and, forGerman-English, the UFAL Medical Corpus provided in the WMT17 Biomedical Translation task.

3 x 643 x 643 x 503 x 50target word
source word
target POS
source POS

Figure 1: Architecture of our N EURAL QE system.

Pure Quality Estimation

We use the pure quality estimation system developed by the Unbabel team and described in Martins et al. (2017), which consists of an ensembleof a linear feature-based classifier with a neuralnetwork. We briefly describe the linear (§2.1) andneural (§2.2) components of our system, as well as
their combination (§2.3). Further details are presented in Martins et al. (2016, 2017).
2.1

Linear Sequential Model

The linear component of our model is a discriminative feature-based sequential model (called L IN EAR QE). The system receives as input a tuple
hs, t, Ai, where s = s1 . . . sM is the source sentence, t = t1 . . . tN is the translated sentence, andA ⊆ {(m, n) | 1 ≤ m ≤ M, 1 ≤ n ≤ N } is aset of word alignments. It predicts as output a sequence yb = y1 . . . yN , with each yi ∈ {BAD, OK}.
This is done as follows:
yb = argmaxyNXw> φu (s, t, A, yi )

i=1

+

N
+1
X

w> φb (s, t, A, yi , yi−1 ).

(1)

i=1

Above, w is a vector of weights, φu (s, t, A, yi )are unigram features (depending only on a single output label), φb (s, t, A, yi , yi−1 ) are bigramfeatures (depending on consecutive output labels),
and y0 and yN +1 are special start/stop symbols.
Table 1 shows the unigram and bigram features used in the L INEAR QE system. We include
features that depend on the target word and itsaligned source word, as well as the context surrounding them.2 We include also syntactic fea2

Features involving the aligned source word are replacedtures to detect grammatically incorrect constructions. We use features that involve the dependency relation, the head word, and second-ordersibling and grandparent structures. Features involving part-of-speech (POS) tags and syntactic
information are obtained with TurboTagger andTurboParser (Martins et al., 2013). The feature
weights are learned by running 50 epochs of the
max-loss MIRA algorithm (Crammer et al., 2006),
with regularization constant C ∈ {10−k }4k=1 anda Hamming cost function placing a higher penalty
on false positives than on false negatives (cFP ∈{0.5, 0.55, . . . , 0.95}, cFN = 1 − cFP ), to accountfor the existence of fewer BAD labels than OK labels in the data. These values are tuned on thedevelopment set.
2.2

Neural System

Next, we describe the neural component of ourpure QE system, which we call N EURAL QE.
The architecture of N EURAL QE is depicted in
Figure 1. We used Keras (Chollet, 2015) to implement our model. The system receives as input the source and target sentences s and t, theirword-level alignments A, and their corresponding
POS tags obtained from TurboTagger. The input
layer follows a similar architecture as QUETCH(Kreutzer et al., 2015), with the addition of POS
features. A vector representing each target word is
obtained by concatenating the embedding of thatword with those of the aligned word in the source.3
The immediate left and right contexts for source
and target words are also concatenated. We use
by NIL if the target word is unaligned. If there are multiplealigned source words, they are concatenated into a single feature.
3
For the cases in which there are multiple source wordsaligned to the same target word, the embeddings are averaged.

LabelInput (referenced by the ith target word)

unigramyi ∧ . . .

∗

B IAS
W ORD , L EFT W ORD , R IGHT W ORD∗
S OURCE W ORD , S OURCE L EFT W ORD , S OURCE R IGHT W ORD∗
L ARGEST NG RAM L EFT /R IGHT, S OURCE L ARGEST NG RAM L EFT /R IGHT∗
P OS TAG , S OURCE P OS TAG†
W ORD +L EFT W ORD , W ORD +R IGHT W ORD†
W ORD +S OURCE W ORD , P OS TAG +S OURCE P OS TAG

simple bigramyi ∧ yi−1 ∧ . . .

∗rich bigramsyi ∧ yi−1 ∧ . . .
yi+1 ∧ yi ∧ . . .

all above
W ORD +S OURCE W ORD , P OS TAG +S OURCE P OS TAG

syntactic

yi ∧ . . .

D EP R EL , W ORD +D EP R EL
H EADW ORD /P OS TAG +W ORD /P OS TAG
L EFT S IB W ORD /P OS TAG +W ORD /P OS TAG
R IGHT S IB W ORD /P OS TAG +W ORD /P OS TAG
G RANDW ORD /P OS TAG +H EADW ORD /P OS TAG +W ORD /P OS TAG

Features∗

B IAS

Table 1: Features used in the L INEAR QE system (see Martins et al., 2016 for a detailed description).
Features marked with ∗ are included in the WMT16 baseline system. Those marked with † were proposedby Kreutzer et al. (2015).
the pre-trained 64-dimensional Polyglot word embeddings (Al-Rfou et al., 2013) for English andGerman, and refine them during training. In addition to this, POS tags for each source and target
word are also embedded and concatenated. POS
embeddings have size 50 and are initialized as described by Glorot and Bengio (2010). A dropout
probability of 0.5 is applied to the resulting vector
representations.
The following layers are then applied in sequence:
1. Two feed-forward layers of size 400 with rectified linear units (ReLU; Nair and Hinton(2010));
2. A layer with bidirectional gated recurrent
units (BiGRU, Cho et al. (2014)) of size 200,
where forward and backward vectors are concatenated, trained with layer normalization(Ba et al., 2016);
3. Two feed-forward ReLU layers of size 200;
4. A BiGRU layer of size 100 with identicalconfiguration to the previous BiGRU;
5. Two more feed-forward ReLU layers of sizes
100 and 50, respectively.
As the output layer, a softmax transformation over
the OK/BAD labels is applied. We provide ablation
experiments in Martins et al. (2017) to validate thisarchitecture choice.

We train the model with the RMSProp algorithm (Tieleman and Hinton, 2012) by minimizing the cross-entropy with a linear penalty for BAD
word predictions, as in Kreutzer et al. (2015). We
set the BAD weight factor to 3.0. All hyperparameters are adjusted based on the development set.
Target sentences are bucketed by length and thenprocessed in batches (without any padding or truncation).
Finally, we also trained 5 independent instancesof N EURAL QE with different random initializations and different data shuffles. We ensemble
these systems by taking the averaged probabilityof each word being BAD. Tables 2–3 show the results. We observe that, for both language pairs, theneural model outperforms the linear model, andthe ensemble of 5 neural systems achieves an extra boost (most noticeable in the English-German
dataset).
2.3

Stacking Neural and Linear Models

We now stack the N EURAL QE system (§2.2)
into the L INEAR QE system (§2.1) as an ensemble strategy; we call the resulting system
S TACKED QE.
The individual instances of the neural systems
are incorporated in the stacking architecture as different features, yielding S TACKED QE. In total, we
have 5 predictions (probability values given byeach N EURAL QE system) for every word in the
training, development and test datasets. These predictions are plugged as additional features in theL INEAR QE model. As unigram features, we used
one real-valued feature for every model prediction
at each position, conjoined with the label. As bigram features, we used two real-valued features
for every model prediction at the two positions,conjoined with the label pair.
For the remainder of this paper, we will take
S TACKED QE as our pure QE system.

3

APE-Based Quality Estimation

To develop our APE-based QE system (A PE QE),
we followed a similar approach as the one described in Martins et al. (2017), with a few minor
differences, explained below.
Junczys-Dowmunt and Grundkiewicz (2016)
applied neural translation models to the APE problem, treating different models as components in a
log-linear model, allowing for multiple inputs (thesource s and the translated sentence t) that were
decoded to the same target language (post-editedtranslation p). Two systems were considered, one
using s as the input (s → p) and another usingt as the input (t → p). For English-German, weused the 500K artificial roundtrip translations provided by the shared task organizers, along withthe original data from the shared task (oversampled 20 times, as in Junczys-Dowmunt and Grundkiewicz (2016)). For German-English, we only
considered the s → p machine translation system,
trained from a subset of the UFAL Medical Corpus provided in the WMT17 Biomedical Translation task. This subset was obtained through crossentropy filtering. For this, we built an in-domain
trigram language model from the English postedited training data. We then calculated crossentropy scores for the UFAL corpus according to
the language model. We sorted the corpus by increasing cross-entropy and kept the first 500K sentences to be used as additional training data.
To convert the resulting APE systems into
word-level quality estimators, we need to turn theautomatic post-edited sentences into word quality
labels. This is done in a straightforward way byusing the TERCOM software tool (Snover et al.,2006)4 with the default settings (tokenized, caseinsensitive, exact matching only, shifts disabled).
This tool computes the HTER (the normalized edit
distance) between the translated and post-editedsentence. As a by-product, it aligns the words inthe two sentences, identifying substitution errors,4

http://www.cs.umd.edu/˜snover/tercom.

word deletions (i.e. words omitted by the translation system), and insertions (redundant words inthe translation). This is mapped deterministicallyinto OK and BAD labels.
Our approach for the shared task differs from
Martins et al. (2017) in which we skipped the
QE-tuning step when training the log-linear APE
model; instead, we kept the output of the s →p and the t → p systems, converted each toword-level quality labels, and then include the two
predictions as additional features in the F ULL S TACKED QE system, described below. We denote the individual systems as A PE QE s → p
and A PE QE t → p, and the combined system as
F ULL S TACKED QE s, t → p.

4

Full Stacked System

Finally, we consider a larger stacked system where
we stack both N EURAL QE and A PE QE into L IN EAR QE. This mixes pure QE with APE-based QE
systems; we call the result F ULL S TACKED QE.
The procedure is analogous to that described in§2.3, with extra binary features for the APE-based
word quality label predictions. For training, weused jackknifing to ensure the predictions on the
training set are not biased.
4.1

Word-Level QE

The performance of the F ULL S TACKED QE system on the English-German and German-English
development datasets are shown in Tables 2–3.
For English-German, we compare with the systemfrom Martins et al. (2017).
For both language pairs, we can see that theAPE-based and the pure QE systems are highly
complementary. For English-German, the fullcombination of the linear, neural, and APE-based
systems improves the scores with respect to thebest individual system by about 5.5 points. Thereis a small improvement by including also a feature from APE t → p, in addition to s → p. For
German-English, we observe an improvement of
4.9 points.
4.2

Sentence-Level QE

We followed the same procedure of Martins et al.
(2017) to convert word-level quality predictions toa sentence-level HTER prediction. For the APE
system, we simply measured the HTER between
the translated sentence t and the predicted corrected sentence pb. For a pure QE system, we ap-

F1MULT dev

F1MULT test 2016

Martins et al. (2017)

56.80

57.47

L INEAR QE
N EURAL QE (single)
N EURAL QE (5-avg)
S TACKED QE
A PE QE (s → p)
A PE QE (t → p)
F ULL S TACKED QE (s → p)
F ULL S TACKED QE (s, t → p)

47.71
49.58
51.37
52.72
51.43
35.27
57.18
57.55

48.09
49.95
51.38
52.89
52.47
37.13
58.04
58.36

Table 2: Performance of the several word-level
QE systems on the development and WMT16
English-German test datasets.
F1MULT dev
L INEAR QE
N EURAL QE (single)
N EURAL QE (avg-5)
S TACKED QE
A PE QE (s → p)
F ULL S TACKED QE (s → p)

48.07
49.39
49.58
53.22
45.09
58.08

Table 3: Performance of the several word-level
QE systems on the German-English development
dataset.

plied the following word-to-sentence conversion
technique: (i) run a QE system to obtain a sequence of OK and BAD word quality labels; (ii)
use the fraction of BAD labels as an estimate for
HTER. Finally, to combine the APE and pure QE
systems toward sentence-level QE, we simply takethe average of the two HTER predictions above.
Table 4 shows the results obtained with our pure
QE system (S TACKED QE), with our APE-based
system (A PE QE), and with the combination of thetwo (F ULL S TACKED QE). We report also the performance of the system of Martins et al. (2017) for
English-German, for comparison.

5

Final Results

Finally, we show in Tables 5–6 the results obtained in the test set for our two submitted systems,
S TACKED QE and F ULL S TACKED QE, in wordlevel and sentence-level quality estimation. As
expected, the inclusion of the predictions madeby the APE system gave a significant boost for
the word-level task (>5 F1MULT points for EnglishGerman, and >6 points for German-English) andfor the sentence-level task (>5 Pearson correlation points for English-German, >4 points forGerman-English).

6

Conclusions

We have presented the contribution of the Unbabel team to the WMT 2017 Shared Task on Translation Quality Estimation. Our word-level system combines a pure quality estimation system,
based on stacking a neural and feature-based linear model, and an APE-based quality estimation
system, which uses the predictions of an automatic post-editing system to generate additionalfeatures. We applied a simple conversion strategy
to obtain a sentence-level quality estimator basedon the word-level one. The system is evaluated on
two language pairs, English-German and GermanEnglish.

Acknowledgments
This work was partially supported by the the EXPERT project (EU Marie Curie ITN No. 317471),
and by Fundação para a Ciência e Tecnologia(FCT), through contracts UID/EEA/50008/2013
and UID/CEC/50021/2013, the LearnBig project(PTDC/EEI-SII/7092/2014), and the GoLocal
project (grant CMUPERI/TIC/0046/2014).

References
Rami Al-Rfou, Bryan Perozzi, and Steven Skiena.
2013.
Polyglot: Distributed word representations for multilingual nlp.
arXiv preprint
arXiv:1307.1662 .
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint
arXiv:1607.06450 .
John Blatz, Erin Fitzgerald, George Foster, Simona
Gandrabur, Cyril Goutte, Alex Kulesza, Alberto
Sanchis, and Nicola Ueffing. 2004. Confidence estimation for machine translation. In Proc. of the International Conference on Computational Linguistics.
page 315.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations Using RNN EncoderDecoder for Statistical Machine Translation. In
Proc. of Empirical Methods in Natural Language
Processing.
François Chollet. 2015. Keras. https://github.
com/fchollet/keras.
Koby Crammer, Ofer Dekel, Joseph Keshet, Shai
Shalev-Shwartz, and Yoram Singer. 2006. Online
Passive-Aggressive Algorithms. Journal of Machine Learning Research 7:551–585.

En-DeDe-En

Pearson dev

Pearson test 2016

Spearman dev

Spearman test 2016

Martins et al. (2017)
S TACKED QE
A PE QEF ULL S TACKED QE

64.04
62.18
58.79
64.33

65.56
59.90
56.69
65.25

65.52
64.07
59.41
65.65

65.92
60.23
59.43
66.10

S TACKED QE
A PE QEF ULL S TACKED QE

60.47
59.94
67.97

–
–
–

59.74
57.68
62.45

–
–
–

Table 4: Performance of our sentence-level QE systems on the English-German and German-English
datasets, as measured by the official evaluation script. We show the performance of Martins et al. (2017)
for comparison.
F1OK

F1BAD

F1MULTE N -D E S TACKED QE
E N -D E F ULL S TACKED QE

88.2
90.6

58.1
62.5

51.2
56.6

D E -E N S TACKED QE
D E -E N F ULL S TACKED QE

93.6
94.1

49.7
56.2

46.6
52.9

Table 5: Performance of the submitted word-levelsystems on the test set.
PearsonSpearman

E N -D E S TACKED QE
E N -D E F ULL S TACKED QE

58.9
64.1

61.0
65.2

D E -E N S TACKED QE
D E -E N F ULL S TACKED QE

58.0
62.6

57
61

Table 6: Performance of the submitted sentencelevel systems on the test set.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neuralnetworks. In International Conference on Artificial
Intelligence and Statistics. pages 249–256.
Marcin Junczys-Dowmunt and Roman Grundkiewicz.
2016. Log-linear combinations of monolingual andbilingual neural machine translation models for automatic post-editing. In Proceedings of the First
Conference on Machine Translation. pages 751–
758.
Julia Kreutzer, Shigehiko Schamoni, and Stefan Riezler. 2015.
Quality estimation from scratch(quetch): Deep learning for word-level translation
quality estimation. In Proceedings of the TenthWorkshop on Statistical Machine Translation. pages
316–322.
André F. T Martins, Miguel B. Almeida, and Noah A.
Smith. 2013. Turning on the turbo: Fast third-ordernon-projective turbo parsers. In Proc. of the Annual
Meeting of the Association for Computational Linguistics.
André F. T. Martins, Ramón Astudillo, Chris Hokamp,
and Fábio Kepler. 2016. Unbabel’s participation in

the wmt16 word-level translation quality estimation
shared task. In Proceedings of the First Conference
on Machine Translation.
André F. T. Martins, Marcin Junczys-Dowmunt, Fabio
Kepler, Ramon Astudillo, Chris Hokamp, and Roman Grundkiewicz. 2017. Pushing the limits oftranslation quality estimation. Transactions of the
Association for Computational Linguistics (to appear) .
Vinod Nair and Geoffrey E Hinton. 2010. Rectified
linear units improve restricted boltzmann machines.
In Proc. of the International Conference on Machine
Learning. pages 807–814.
Djamé Seddah, Sandra Kübler, and Reut Tsarfaty.
2014. Introducing the SPMRL 2014 shared task
on parsing morphologically-rich languages. In Proceedings of the First Joint Workshop on Statistical Parsing of Morphologically Rich Languages andSyntactic Analysis of Non-Canonical Languages.
pages 103–109.
Matthew Snover, Bonnie Dorr, Richard Schwartz, Linnea Micciulla, and John Makhoul. 2006. A studyof translation edit rate with targeted human annotation. In Proc. of the 7th Conference of the Association for Machine Translation in the Americas. pages
223–231.
Lucia Specia, Kashif Shah, Jose G.C. de Souza, andTrevor Cohn. 2013. QuEst - a translation quality estimation framework. In Proc. of the Annual Meeting of the Association for Computational
Linguistics: System Demonstrations. pages 79–84.
http://www.aclweb.org/anthology/P13-4014.
Tijmen Tieleman and Geoffrey Hinton. 2012. Rmsprop: Divide the gradient by a running averageof its recent magnitude. COURSERA: Neural Networks for Machine Learning 4(2).