Quality Estimation for Language Output
Applications
Carolina Scarton, Gustavo Paetzold and Lucia Specia
University of Sheffield, UK

COLING, Osaka, 11 Dec 2016

Quality Estimation

I Approaches to predict the quality of a language output

application – no access to “true” output for comparison
I Motivations:

Evaluation of language output applications is hard: no single
gold-standard
I For NLP systems in use, gold-standards are not available
I

I Some work done for other NLP tasks, e.g. parsing

Quality Estimation - Parsing
Task [Ravi et al., 2008]
I Given: a statistical parser and its training data and somechunk of text
I Estimate of the f-measure of the parse trees produced for thatchunk of text
Features
I Text-based, e.g. length, LM perplexity
I Parse tree, e.g. number of certain syntactic labels such as

punctuation
I Pseudo-ref parse tree: similarity to output of another parser

Training
I Training data labelled for f-measure based on gold-standard
I Learner optimised for correlation of f-measureQuality Estimation - Parsing
Very high correlation and low error (in-domain): RMSE = 0.014

Quality Estimation - Parsing
I Very close to actual f-measure:

Baseline (mean of dev set)
Prediction
Actual f-measure
Baseline (mean of dev set)
Prediction
Actual f-measure

In-domain (WSJ)
90.48
90.85
91.13
Out-of-domain (Brown)
90.48
86.96
86.34

I Simpler task: one possible good output; f-measure is very

tellingQuality Estimation - Summarisation

Task: Predict quality of automatically produced summaries
without human summaries as references [Louis and Nenkova, 2013]
I Features:

Distribution similarity and topic words → high correlation
with PYRAMID and RESPONSIVENESS
I Pseudo-references:I

Outputs of off-the-shelf AS systems → additional summary
modelsI High correlation with human scores, even on their own

II

Linear combination of features → regression task

Quality Estimation - Summarisation[Singh and Jin, 2016]:
I Features addressing informativeness (IDF, concreteness,

n-gram similarities), coherence (LSA) and topics (LDA)
I Pairwise classification and regression tasks predicting

RESPONSIVENESS and linguistic quality
I Best results for regression models → RESPONSIVENESS(around 60% of accuracy)

Quality Estimation - Simplification
Task: Predict the quality of automatically simplified versions oftext
I Quality features:Length measures
Token counts/ratios
I Language model probabilities
I Translation probabilities
I

I

I Simplicity ones:

Linguistic relationships
Simplicity measures
I Readability metrics
I Psycholinguistic features
I

I

I Embeddings features

Quality Estimation - Simplification

QATS 2016 shared task
I The first QE task for Text Simplification

I 9 teams
I 24 systems

I Training set: 505 instances
I Test set: 126 instances

Quality Estimation - Simplification

QATS 2016 shared task
I 2 tracks:
I
I

Regression: 1/2/3/4/5
Classification: Good/Ok/Bad

I 4 aspects:

Grammaticality
Meaning Preservation
I Simplicity
I OverallI
IQuality Estimation - Simplification

QATS 2016 shared task: baselines
I Regression and Classification:

BLEU
TER
I WER
I METEORI

I

I Classification only:I
I

Majority class
SVM with all metrics

Quality Estimation - Simplification
Systems:
SystemUoLGP
OSVCMLML
GPs
Forests

SimpleNets
IIT

LSTMs
Bagging

CLaC

Forests

Deep(Indi)Bow
SMH
MS
UoW

MLPs
Misc.
Misc
SVM

Features
QuEst features and embeddings
embeddings, readability, sentiment,
etc
embeddings
language models, METEOR and complexity
language models, embeddings, length,
frequency, etc
bag-of-words
QuEst features and MT metrics
MT metrics
QuEst features, semantic similarity
and simplicity metrics

Quality Estimation - Simplification

Evaluation metrics:
I Regression: Pearson
I Classification: Accuracy

Winners:
Grammaticality
Meaning
Simplicity
Overall

Regression
OSVCML1
IIT-Meteor
OSVCML1
OSVCML2

Classification
Majority-class
SMH-Logistic
SMH-RandForest-b
SimpleNets-RNN2

Quality Estimation - Machine Translation
Task: Predict the quality of an MT system output without
reference translations
I Quality: fluency, adequacy, post-editing effort, etc.
I General method: supervised ML from features + quality

labelsI Started circa 2001 - Confidence Estimation
I
I

How confident MT system is in a translation
Mostly word-level prediction from SMT internal features

I Now: broader area, commercial interest

Motivation - post-editing
MT: The King closed hearings Monday with Deputy Canary Coalition
Ana Maria Oramas González -Moro, who said, in line with the above,that “there is room to have government in the coming months,”
although he did not disclose prints Rey about reports Francesco Manetto.
Monarch Oramas transmitted to his conviction that ‘ soon there will be
an election” because looks unlikely that Rajoy or Sanchez can form agovernment.

Motivation - post-editing
MT: The King closed hearings Monday with Deputy Canary Coalition
Ana Maria Oramas González -Moro, who said, in line with the above,that “there is room to have government in the coming months,”
although he did not disclose prints Rey about reports Francesco Manetto.
Monarch Oramas transmitted to his conviction that ‘ soon there will be
an election” because looks unlikely that Rajoy or Sanchez can form agovernment.
SRC: El Rey cerró las audiencias del lunes con la diputada de CoaliciónCanaria Ana Marı́a Oramas González-Moro, quien aseguró, en la lı́nea delos anteriores, que “no hay ambiente de tener Gobierno en los próximos
meses”, aunque no desveló las impresiones del Rey al respecto, informa
Francesco Manetto. Oramas transmitió al Monarca su convicción de que“pronto habrá un proceso electoral”, porque ve poco probable que Rajoyo Sánchez puedan formar Gobierno.
By Google Translate

Motivation - gisting
Target:site security should be included in sex education
curriculum for students
Source:场地安全性教育应纳入学生的课程
Reference:site security requirements should be included in the
education curriculum for students
By Google Translate

Motivation - gisting
Target:the road boycotted a friend ... indian robin hoodkilled the poor after 32 years of prosecution.
Source:‫ قاطع الطريق صديق‬..‫مقتل روبن هود الهندي‬‫ عاما من الملحقة‬32 ‫الفقراء بعد‬Reference:
death of the indian robin hood, highway robberand friend of the poor, after 32 years on the run.
By Google Translate

Uses
Quality = Can we publish it as is?
Quality = Can a reader get the gist?
Quality = Is it worth post-editing it?
Quality = How much effort to fix it?
Quality = Which words need fixing?
Quality = Which version of the text is more reliable?

General method

General method

Main components to build a QE system:
1. Definition of quality: what to predict and at what level
Word/phrase
Sentence
I DocumentII

2. (Human) labelled data (for quality)
3. Features
4. Machine learning algorithm

Features

Adequacy
indicators

Source text

Complexity
indicators

MT system

Translation

Confidence
indicators

Fluencyindicators

Sentence-level QE

I Most popular levelII

MT systems work at sentence-level
PE is done at sentence-level

I Easier to get labelled data
I Practical for post-editing purposes (edits, time, effort)

Sentence-level QE - Features
MT system-independent features:
I SF - Source complexity features:source sentence length
source sentence type/token ratio
I average source word length
I source sentence 3-gram LM score
I percentage of source 1 to 3-grams seen in the MT training
corpusI depth of syntactic tree
I
I

I TF - Target fluency features:target sentence 3-gram LM scoretranslation sentence length
I proportion of mismatching opening/closing brackets andquotation marks in translation
I coherence of the target sentence
I
ISentence-level QE - Features
I AF - Adequacy features:

ratio of number of tokens btw source & target and v.v.
absolute difference btw no tokens in source & target
I absolute difference btw no brackets, numbers, punctuation
symbols in source & target
I ratio of no, content-/non-content words btw source & target
I ratio of nouns/verbs/pronouns/etc btw source & target
I proportion of dependency relations with constituents aligned
btw source & target
I difference btw depth of the syntactic trees of source & target
I difference btw no pp/np/vp/adjp/advp/conjp phrase labels insource & target
I difference btw no ’person’/’location’/’organization’ (aligned)
entities in source & target
I proportion of matching base-phrase types at different levels ofsource & target parse trees
I
ISentence-level QE - Features

I Confidence features:score of the hypothesis (MT global score)
size of nbest list
I using n-best to build LM: sentence n-gram log-probability
I individual model features (phrase probabilities, etc.)
I maximum/minimum/average size of the phrases in translation
I proportion of unknown/untranslated words
I n-best list density (vocabulary size / average sentence length)
I edit distance of the current hypothesis to the center hypothesis
I Search graph info: total hypotheses, % discarded / pruned /recombined search graph nodes
I

I

I Others:II

quality prediction for words/phrases in sentence
embeddings or other vector representations

Sentence-level QE - Algorithms

I Mostly regression algorithms (SVM, GP)
I Binary classification: good/bad
I Kernel methods perform better
I Tree kernel methods for syntactic trees
I NN are difficult to train (small datasets)

Sentence-level QE - Predicting HTER @WMT16

Languages, data and MT systems
I 12K/1K/2K train/dev/test English → German (QT21)
I One SMT system
I IT domain
I Post-edited by professional translators
I Labelling: HTER

Sentence-level QE - Results @WMT16

System ID
English-German• YSDA/SNTX+BLEU+SVM
POSTECH/SENT-RNN-QV2
SHEF-LIUM/SVM-NN-emb-QuEst
POSTECH/SENT-RNN-QV3
SHEF-LIUM/SVM-NN-both-emb
UGENT-LT3/SCATE-SVM2
UFAL/MULTIVEC
RTM/RTM-FS-SVRUU/UU-SVM
UGENT-LT3/SCATE-SVM1
RTM/RTM-SVR
Baseline SVM
SHEF/SimpleNets-SRC
SHEF/SimpleNets-TGT

Pearson ↑Spearman ↑

0.525
0.460
0.451
0.447
0.430
0.412
0.377
0.376
0.370
0.363
0.358
0.351
0.182
0.182

–
0.483
0.474
0.466
0.452
0.418
0.410
0.400
0.405
0.375
0.384
0.390
–
–

Sentence-level QE - Best results @WMT16I YSDA: features about complexity of source (depth of parsetree, specific constructions), pseudo-reference, backtranslation, web-scale LM, and word alignments. Trained to
predict BLEU scores, followed by a linear SVR to predict
HTER from BLEU scores.
I POSTECH: RNN with two components: (i) two bidirectional

RNNs on the source and target sentences plus (ii) other RNNs
for predicting the final quality: (i) is an RNN-based modified
NMT model that generates a sequence of vectors about targetwords’ translation quality. (ii) predicts the quality at sentencelevel. Each component is trained separately: (i) relies on theEuroparl parallel corpus, (ii) relies on the QE task data.

Sentence-level QE - Challenges

I Data: how to obtain objective labels, for different languages

and domains, which are comparable across translators?
I How to adapt models over time? → online learning[C. de Souza et al., 2015]
I How to deal with biases from annotators (or domains)? →multi-task learning [Cohn and Specia, 2013]

Sentence-level QE - Learning from multiple annotators
I Perception of quality varies
I E.g.: English-Spanish translations labelled for PE effort

between 1 (bad) and 5 (perfect)
I 3 annotators: average of 1K scores: 4; 3.7; 3.3

Sentence-level QE - Learning from multiple annotators[Cohn and Specia, 2013]

Sentence-level QE - Learning from multiple annotators[Cohn and Specia, 2013]

Sentence-level QE - Learning from multiple annotators[Shah and Specia, 2016]

Sentence-level QE - Learning from multiple annotators

Predict en-fr using en-fr & en-es [Shah and Specia, 2016]

Word-level QE

Some applications require fine-grained information on quality:
I Highlight words that need fixing
I Inform readers of portions of sentence that are not reliable

Seemingly a more challenging task
I A quality label is to be predicted for each target word
I Sparsity is a serious issue
I Skewed distribution towards GOOD
I Errors are interdependentWord-level QE - Labels
I Predict binary GOOD/BAD labels
I Predict general types of edits:

Shift
Replacement
I Insertion
I Deletion is an issue
I
I

I Predict specific errors. E.g. MQM in WMT14

Word-level QE - Features
I target token, its left & right token
I source token aligned to target token, its left & right tokens
I boolean dictionary flag: whether target token is a stopword, apunctuation mark, a proper noun, a number
I dangling token flag (null link)
I LM of n-grams with target token ti : (ti−2 , ti−1 , ti ),(ti−1 , ti , ti+1 ), (ti , ti+1 , ti+2 )I order of the highest order n-gram which starts/ends with thesource/target token
I POS tag of target/source token
I number of senses of target/source token in WordNet
I pseudo-reference flag: 1 if token belongs to pseudo-reference,
0 otherwise

Word-level QE - Algorithms
I Sequence labelling algorithms, like CRF

I Classification algorithms: each word tagged independently

I NN:
I
I

MLP with bilingual word embeddings and standard features
RNNs

Word-level QE @WMT16
Languages, data and MT systems
I Same as for T1
I Labelling done with TERCOM:I
I

OK = unchanged
BAD = insertion, substitution

I Instances: <source word, MT word, OK/BAD label>

Training
Dev
Test

Sentences
12, 000
1, 0002, 000Words
210, 958
19, 487
34, 531% of BAD words
21.4
19.54
19.31

New evaluation metric:
F1 -multiplied = F1 -OK × F1 -BAD
Challenge: skewed class distribution

Word-level QE - Results @WMT16
System ID
English-German• UNBABEL/ensemble
UNBABEL/linear
UGENT-LT3/SCATE-RFUGENT-LT3/SCATE-ENS
POSTECH/WORD-RNN-QV3
POSTECH/WORD-RNN-QV2
UAlacant/SBI-Online-baseline
CDACM/RNN
SHEF/SHEF-MIME-1
SHEF/SHEF-MIME-0.3
Baseline CRF
RTM/s5-RTM-GLMd
UAlacant/SBI-Online
RTM/s4-RTM-GLMd
All OK baseline
All BAD baseline

F1 -mult ↑F1 -BADF1 -OK

0.495
0.463
0.411
0.381
0.380
0.376
0.367
0.353
0.338
0.330
0.324
0.308
0.290
0.273
0.0
0.0

0.560
0.529
0.492
0.464
0.447
0.454
0.456
0.419
0.403
0.391
0.368
0.349
0.406
0.307
0.0
0.323

0.885
0.875
0.836
0.821
0.850
0.828
0.805
0.842
0.839
0.845
0.880
0.882
0.715
0.888
0.893
0.0

Word-level QE - Results @WMT16

I Unbabel: linear sequential model with baseline features +dependency-based features (relations, heads, siblings andgrandparents, etc.), and predictions by an ensemble methodthat uses a stacked architecture which combines three neuralsystems: one feedforward and two recurrent ones.
I UGENT: 41 features + baseline feature set to train binary

Random Forest classifiers. Features capture accuracy errors
using word and phrase alignment probabilities, fluency errors
using language models, and terminology errors using abilingual terminology list.

Word-level QE - Challenges

I Data:Labelling is expensive
Labelling from post-editing not reliable → need betteralignment methods
I Data sparsity and skewness are hard to overcome →I

I

I

Injecting errors or filtering positive cases[Logacheva and Specia, 2015]

I Errors are rarely isolated – how to model interdependencies?

→ Phrase-level QE - WMT16

Document-level QE

I Prediction of a single label for entire documents
I Assumption: quality of a document is more than the simple

aggregation of its sentence-level quality scores
While certain sentences are perfect in isolation, their
combination in context may lead to an incoherent document
I A sentence can be poor in isolation, but good in context as it
may benefit from information in surrounding sentencesI

I Application: use as is (no PE) for gisting purposesDocument-level QE - Labels

I Notion of quality is very subjective[Scarton and Specia, 2014]
I

Human labels: hard and expensive to obtain, no datasets
available

I Most work predicts METEOR/BLEU against an

independently created reference. Not ideal:I
I

Low variance across documents
Do not capture discourse issues

I Alternative: task-based labelsII

2-stage post-editing
Reading comprehension tests

Document-level QE - Features
I average or doc-level counts of sentence-level features
I word/lemma/noun repetition in source/target doc and ratio
I number of pronouns in source/target doc
I number of discourse connectives (expansion, temporal,

contingency, comparison & non-discourse)
I number of EDU (elementary discourse units) breaks insource/target doc
I number of RST nucleus relations in source/target doc
I number of RST satellite relations in source/target doc
I average quality prediction for sentences in docs

Algorithms: same as for sentence-level

Document-level QE @WMT16

Languages, data and MT systems
I English → Spanish
I Documents by all WMT8-13 translation task MT systems
I 146/62 documents for training/test
I Labelling: 2-stage post-editing method

1. PE1: Sentences are post-edited in arbitrary order (no context)
2. PE2: Post-edited sentences are further edited withindocument context

Document-level QE @WMT16Label
I Linear combination of HTER values:

w1 · PE1 × MT + w2 · PE2 × PE1
I w1 and w2 are learnt empirically to:I
I

Maximise model performance (MAE/Pearson) and/or
Maximise data variation (STDEV/AVG)

Document-level QE - Results @WMT16

System ID
English-Spanish• USHEF/BASE-EMB-GP• RTM/RTM-FS+PLS-TREE
RTM/RTM-FS-SVR
Baseline SVM
USHEF/GRAPH-DISC

Pearson’s r

Spearman’s ρ ↑

0.391
0.356
0.293
0.286
0.256

0.393
0.476
0.360
0.354
0.285

I USHEF: 17 baseline features + word embeddings from source

documents combined using GP. Document embeddings are theaverage of the word embeddings in the document. GP model
was trained with 2 kernels: one for the 17 baseline features
and another for the 500 features from the embeddings.

Document-level QE - New label

MAE gain (%) compared to “mean” baseline:

Document-level QE - New label

Document-level QE - Challenges

I Quality label still an open issue
I
I

should take into account purpose of translation
should reliably distinguish different documents

I Feature engineering: few tools for discourse processing
I
I

Topic and structure of document
Relationship between its sentences/paragraphs

I Relevance information needed: how to factor it in

Features: QE for sentence + sentence ranking methods[Turchi et al., 2012]
I Labels
IParticipants @WMT16Centre for Development of Advanced Computing, India
Pohang University of Science and Technology,
Republic of Korea
Referential Translation Machines, Turkey
University of Sheffield, UK
University of Sheffield, UK &Lab. d’Informatique de l’Université du Maine, France
University of Alicante, Spain
Nile University, Egypt &Charles University, Czech RepublicGhent University, Belgium
Unbabel, Portugal
Uppsala University, Sweden
Yandex, Russia

WL/PL
X

SL

DL

X
X
X

X
X
X

X
X

X
X

X
X

X
X
X
X

Neural Nets for QE

As features:
I NLM for sentence-level
I Word embeddings for word, sentence and doc-level

As learning algorithm:
I MLP proved effective until 2015
I 2016 submissions use RNNs for sentence-level (POSTECH,

SimpleNets), word-level (Unbabel), phrase-level (CDAC)

QE in practiceDoes QE help?
I Time to post-edit subset of sentences predicted as “low PE

effort” vs time to post-edit random subset of sentences[Specia, 2011]
Language
fr-en
en-es

no QE
0.75 words/sec
0.32 words/sec

QE
1.09 words/sec
0.57 words/sec

QE in practice
I Productivity increase [Turchi et al., 2015]
I Comparison btw post-editing with and without QE
I Predictions shown with binary colour codes (green vs red)QE in practice

I MT system selection: BLEU scores [Specia and Shah, 2016]

en-de
de-en
en-es
es-en

Majority Class
16.14
25.81
30.88
30.13

Best QE-selected
18.10
28.75
33.45
38.73

Best MT system
17.04
27.96
25.89
37.83

QE in practiceI SMT self-learning: de-en SMT enhanced with MT data

’best’ according to QE [Specia and Shah, 2016]
19.6(
19.4(
19.2(
BLEU%

19(
18.8(
18.6(
18.4(
18.2(
18(
Baseline(

Itera,on(1(

Itera,on(2(

Itera,on(3(

Itera,on(4(

Itera,on(5(

Itera,on(
6(
19.42(

SMT(

18.43(

18.78(

19.1(

19.21(

19.46(

19.45(

RBMT(

18.43(

18.62(

18.99(

19.11(

19.29(

19.25(

19.29(

References(

18.43(

18.91(

19.17(

19.33(

19.42(

19.41(

19.43(

Random(18.43(

18.59(

18.91(

19.11(

19.1(

19.21(

19.27(

QE in practiceI SMT self-learning: en-de SMT enhanced with MT data

’best’ according to QE [Specia and Shah, 2016]
14.6(
14.4(
14.2(
BLEU%

14(
13.8(
13.6(
13.4(
13.2(
13(
Baseline(

Itera,on(1(

Itera,on(2(

Itera,on(3(

Itera,on(4(

Itera,on(5(

Itera,on(
6(
14.39(

SMT(

13.31(

13.62(

13.99(

14.4(

14.31(

14.42(

RBMT(

13.31(

13.43(

13.74(

13.99(

14.21(

14.31(

14.25(

References(

13.31(

13.72(

14.09(

14.2(

14.49(

14.44(

14.43(

Random(13.31(

13.4(

13.65(

13.92(

14.2(

14.23(

14.25(

Quality Estimation for Language Output
Applications
Carolina Scarton, Gustavo Paetzold and Lucia Specia
University of Sheffield, UK

COLING, Osaka, 11 Dec 2016

References I
C. de Souza, J. G., Negri, M., Ricci, E., and Turchi, M. (2015).
Online multitask learning for machine translation quality estimation.
In 53rd Annual Meeting of the Association for Computational Linguistics,
pages 219–228, Beijing, China.
Cohn, T. and Specia, L. (2013).
Modelling annotator bias with multi-task gaussian processes: An
application to machine translation quality estimation.
In 51st Annual Meeting of the Association for Computational Linguistics,
ACL, pages 32–42, Sofia, Bulgaria.
Logacheva, V. and Specia, L. (2015).
The role of artificially generated negative data for quality estimation ofmachine translation.
In 18th Annual Conference of the European Association for MachineTranslation, EAMT, Antalya, Turkey.

References IILouis, A. and Nenkova, A. (2013).
Automatically assessing machine summary content without a gold
standard.
Computational Linguistics, 39(2):267–300.
Ravi, S., Knight, K., and Soricut, R. (2008).
Automatic prediction of parser accuracy.
In Conference on Empirical Methods in Natural Language Processing,pages 887–896, Honolulu, Hawaii.
Scarton, C. and Specia, L. (2014).
Document-level translation quality estimation: exploring discourse andpseudo-references.
In 17th Annual Conference of the European Association for MachineTranslation, EAMT, pages 101–108, Dubrovnik, Croatia.

References III
Shah, K. and Specia, L. (2016).
Large-scale multitask learning for machine translation quality estimation.
In Conference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, pages
558–567, San Diego, California.
Singh, A. and Jin, W. (2016).
Ranking Summaries for Informativeness and Coherence without Reference
Summaries.
In The Twenty-Ninth International Florida Artificial Intelligence ResearchSociety Conference, pages 104–109, Key Largo, Florida.
Specia, L. (2011).
Exploiting objective annotations for measuring translation post-editing
effort.
In 15th Conference of the European Association for Machine Translation,
pages 73–80, Leuven.

References IV
Specia, L. and Shah, K. (2016).
Machine Translation Quality Estimation: Applications and Future
Perspectives, page to appear.
Springer.
Turchi, M., Negri, M., and Federico, M. (2015).
Mt quality estimation for computer-assisted translation: Does it reallyhelp?
In 53rd Annual Meeting of the Association for Computational Linguistics,
pages 530–535, Beijing, China.
Turchi, M., Specia, L., and Steinberger, J. (2012).
Relevance ranking for translated texts.
In 16th Annual Conference of the European Association for MachineTranslation, EAMT, pages 153–160, Trento, Italy.