Adequacy-Fluency Metrics (AM-FM) forMachine Translation (MT) Evaluation
Haizhou Li

Invited Talk: WAT 2015, 16 Oct 2015, Kyoto, JapanBanchs R. E., D'Haro L.F., Li H. (2015) "Adequacy - Fluency Metrics: Evaluating MT in the Continuous Space Model Framework",
IEEE/ACM Transactions on Audio, Speech and Language Processing, Vol.23, No.3, pp.472-482

Pg 1

Updated: 18/10/2015

Agenda••
••••

Pg 2

The evaluation of ASR and MT
How do machines evaluate translations today?
How do humans evaluate translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

Automatic Evaluation ofAutomatic Speech Recognition

ASRoutputUNIQUE
transcription

ASR output is compared to areference transcription.
The reference transcription is unique!
Pg 3

Updated: 18/10/2015

Automatic Evaluation of Machine Translation
MT output is compared to reference translations.
… but references are not unique!
文无第一 , 武无第二NONUNIQUEreference

MTPg 4

Updated: 18/10/2015

output?

Agenda•
•••••

Pg 5

The evaluation of ASR and MT
How do machines evaluate translations today?
How do humans evaluate translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

Traditional Evaluation Approach
Compare the output with a set of references
WER1, PER1

Compare words

BLEU2, NIST3

Compare n-grams

1.- C. Tillmann et al., “Accelerated DP Based Search for Statistical Translation”, in Proc. of the 5th European Conf. on
Speech Commun. and Tech., Rhodos, Greece, Sept 1997, pp. 2667–2670.
2.- K. Papineni et al., “BLEU: a method for automatic evaluation of machine translation”, in Proc. of the 40th Annu.
Meeting of the Assoc. for Computational Linguistics, Philadelphia, PA, USA, Jul 2002, pp. 311-318
3.- G. Doddington, “Automatic evaluation of machine translation quality using n-gram co-occurrence statistics”, inProc. of the Human Lang. Tech. Conf., San Diego, CA, USA, Mar 2002

Pg 6

Updated: 18/10/2015

Traditional Approach: Good Scores

Translation Output

This is a toilet.

Reference Translation

This is a toilet.

word matches = 4/4
n-gram matches = 5/5

Pg 7

Updated: 18/10/2015

GoodScoreGoodTranslation

Traditional Approach: Bad ScoresTranslation Output
Reference Translation
word matches = 0/4
n-gram matches = 0/5

Pg 8

Updated: 18/10/2015

It’s the Water Closet.

This is a toilet.
BadScore?

Traditional Approach: Better Scores?

Translation Output

This isn’t a toilet.

Reference Translation

This is a toilet.

word matches = 3/4
n-gram matches = 3/5

Pg 9

Updated: 18/10/2015

BetterScore?

How Machines Evaluate Translations?

• Only look at outputs
and references• Without knowledge
supportPg 10

Updated: 18/10/2015

A Semantic Framework is Needed
Automatic MT evaluation must move beyond
words and n-grams! Some recent proposals:
METEOR1TER2MEANT3Compare stems and synonyms
Compute edit distances
Compare semantic roles

1.- A. Lavie and M.J. Denkowski, “The Meteor metric for automatic evaluation of machine translation”, MachineTranslation, vol. 23, pp. 105-115, May 2009
2.- M. Snover et al., “Study of Translation Edit Rate with Targeted Human Annotation”, in Proc. of the 7th Biennial
Conf. of the Assoc. for Mach. Translation in the Amer., Cambridge, MA, USA, Aug 2006
3.- C.K. Lo and D. Wu, “MEANT: An inexpensive, high-accuracy, semi-automatic metric for evaluating translation
utility based on seman-tic roles”, in Proc. of the 49th Annu. Meeting of the Assoc. for Computational Linguistics,
Portland, OR, USA, Jun 2011, pp. 220-229

Pg 11

Updated: 18/10/2015

Agenda•
•••
•
•

Pg 12

The evaluation of ASR and MT
How do machines evaluation translations today?
How do humans evaluation translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

How Humans Evaluate Translations?* (I)MToutputP(T|S) ≈ P(S|T) P(T)ADEQUACY
How much of the source information is preserved?

FLUENCY
How good is the generated target language quality?
* J.S. White, T. O’Cornell and F. O’Nava, “The ARPA MT evaluation methodologies: evolution, lessons and future
approaches”, in Proc. of the Assoc. for Mach. Translation in the Amer., Oct 1994, pp. 193-205

Pg 13

Updated: 18/10/2015

How Humans Evaluate Translations ? (II)• Look at bothoutputs and inputs• Language andculturalknowledge

Pg 14

Updated: 18/10/2015

Adequacy Evaluation Scale*

How much of the source information ispreserved in the translation?
(Look at both inputs and outputs!)

ScoreDefinition

12
3
45

None of the meaning is preserved
Little of the meaning is preserved
Much of the meaning is preserved
Most of the meaning is preserved
All the meaning is preserved* J.S. White, T. O’Cornell and F. O’Nava, “The ARPA MT evaluation methodologies: evolution, lessons and futureapproaches”, in Proc. of the Assoc. for Mach. Translation in the Amer., Oct 1994, pp. 193-205

Pg 15

Updated: 18/10/2015

Fluency Evaluation Scale*How good is translation regarding thetarget language quality?
(Only look at the outputs!)

Score

Definition

12
3
4

Incomprehensible target language
Disfluent target language
Non-native kind of target language
Good quality target language

5Flawless target language* J.S. White, T. O’Cornell and F. O’Nava, “The ARPA MT evaluation methodologies: evolution, lessons and futureapproaches”, in Proc. of the Assoc. for Mach. Translation in the Amer., Oct 1994, pp. 193-205

Pg 16

Updated: 18/10/2015

Agenda••••••Pg 17

The evaluation of ASR and MT
How do machines evaluate translations today?
How do humans evaluate translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

The Proposed Evaluation Framework*• Approximate adequacy and fluency by means of
independent models:– Use a “semantic approach” for adequacy– Use a “syntactic approach” for fluency• Combine both evaluation metrics into a single
evaluation score* Banchs R.E., D'Haro L.F., Li H. (2015) "Adequacy - Fluency Metrics: Evaluating MT in the Continuous Space ModelFramework", IEEE/ACM Transactions on Audio, Speech and Language Processing, Special issue on continuous spaceand related methods in NLP, Vol.23, No.3, pp.472-482

Pg 18

Updated: 18/10/2015

State of the Art in MT Evaluation*Assessment
Level

Need for
References

Cross-Language
Approach

Humans inthe Loop

Words

WER, PER--

Word n-grams

BLEU, NIST--

Stems & Synonyms

METEOR--

Edit Distances

TER-HTER

Semantic RolesMEANT

XMEANT

HMEANT

Continuous SpacemAM-FM

xAM-FM-* Banchs R.E., D'Haro L.F., Li H. (2015) "Adequacy - Fluency Metrics: Evaluating MT in the Continuous Space ModelFramework", IEEE/ACM Transactions on Audio, Speech and Language Processing, Special issue on continuous spaceand related methods in NLP, Vol.23, No.3, pp.472-482

Pg 19

Updated: 18/10/2015

Properties of Continuous Spaces

The Distributional Hypothesis“a word is characterized for the company it keeps” (Firth 1957)
meaning is mainly determined by the context rather than fromindividual language units• Continuous spaces represent semantic similarities bymeans of the geometric concept of proximity• Offer much “better” smoothing capabilities• Not constrained to the Markovian assumption
Pg 20

Updated: 18/10/2015

The Term-Document Matrix• A model representing joint distributions between
words and documents
.

D1

D2

D3

D4

D5

D6

D7

D8…

DN

T1
Non-zero row values for those
documents containing a given word

T2
T3
T4

vij

T5
T6…
TM

Pg 21

Updated: 18/10/2015

Non-zero column values for those words
occurring within a given document

Document Vector Spaces
Pay attention to the columns of the term-document matrix
documentvector.
D1variables

T1
T2
T3
T4T5T6…
TMPg 22

Updated: 18/10/2015

D2

D3

observations
D4

D5

D6

D7

D8…

DN

Semantic Association in Vector Spaces
Association scores and similarity metrics can be used toassess the degree of semantic relatedness among
documents.DISSIMILAR DOCUMENTS

Pg 23

Updated: 18/10/2015

SIMILAR DOCUMENTS

Semantic Map for Data Collection (1)Opinionated content from rating website

Positive

Negative

Automotive

Pg 24

Updated: 18/10/2015

FinancialSemantic Map for Data Collection (2)66 Books from The Holy Bible: English version(vocabulary size: 8121 words)Pg 25

Updated: 18/10/2015

Agenda•
••
•••

Pg 26

The evaluation of ASR and MT
How do machines evaluate translations today?
How do humans evaluate translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

AM: Adequacy-oriented Metric• Compare sentences in a semantic space– Monolingual AM (mAM): compare output vs. reference– Cross-language AM (xAM): compare output vs. input

MToutput

LSI

CL-LSIinput

Pg 27

Updated: 18/10/2015

referenceLatent Semantic Indexing (LSI)*SVD: MM×N = UM×M S

T
M×N VN×N

Documents projected intoword spaceT
UM×M MM×N = DM×N

Translation output (to) andtranslation reference (tr)compared in reduced vector space

T

T...

...

...

um1...umk ...umm…

UK×M =

u11...u1k ...um1
u21...u2k ...um2

T

T<UK×M toM×1 , UK×M trM×1>

T
UK×M MM×N = DK×N
Documents projected into
reduced word (semantic) space* Deerwester, S., Dumais, S.T., Furnas, G.W., Landauer, T.K. and Harshman, R. (1990), Indexing by latent semantic
analysis, Journal of the American Society for Information Science, 41, pp.391-407
Pg 28

Updated: 18/10/2015

Cross-Language LSI*
Multilingual
term-document matrix

X(Ms+Mt)×N =SVD: X = U S V T

MMs×N
MMt×N

Term-document matrix
in source language
Term-document matrix
in target language

Translation output (to) and translation input (ti)compared in cross-language vector space

0Ms×1
tiMs×1T<UK×(Ms+Mt) to, UK×(Ms+Mt) 0>
Mt×1
Mt×1
T

* Dumais S.T., Letsche T.A., Littman M.L. and Landauer T.K. (1997), Automatic Cross-Language Retrieval Using
Latent Semantic Indexing, in AAAI-97 Spring Symposium Series: Cross-Language Text and Speech Retrieval, pp. 1824
Pg 29

Updated: 18/10/2015

FM: Fluency-oriented Metric• Measures the quality of the target language with a language
model• Uses a compensation factor to avoid effects derived fromdifferences in sentence lengths

MTinput

Pg 30

Updated: 18/10/2015

outputreferencen-gram LM

Compensated Language Modeln-gram probabilities

1FM  exp  n1:N log  p( wn | wn1 ,...)
N
compensation factor

Pg 31

Updated: 18/10/2015

AM-FM Combined Score
Both components can be combined into a single metric according
to different criteria• Weighted Harmonic Mean:• Weighted Mean:

Updated: 18/10/2015

a AM + (1–a) FM

M-AM-FM = (1–a) AM + a FM

• Weighted L2-norm:

Pg 32

H-AM-FM =AM ∙ FM

N-AM-FM =(1–a) AM2 + a FM2

Agenda•
••
•
•
•

Pg 33

The evaluation of ASR and MT
How do machines evaluate translations today?
How do humans evaluate translations?
The Adequacy-Fluency Metrics (AM-FM)
The mathematical formulation
The experiments

Updated: 18/10/2015

WMT-2007 Dataset*• Fourteen tasks:– five European languages (EN, ES, DE, FR, CZ) and– two different domains (News and EPPS).

• Systems outputs available from 14 teams that had participated in
the evaluation. In total, 86 system outputs.
• Overall 172,315 individual sentence translations, from which atotal of 10,754 were rated for both adequacy and fluency byhuman judges.
* Callison-Burch C., Fordyce C., Koehn P., Monz C. and Schroeder J. (2007), (Meta-) evaluation of machinetranslation, in Proceedings of Statistical Machine Translation Workshop, pp. 136-158

Pg 34

Updated: 18/10/2015

WMT-2007 Translation Task Details

Pg 35

Task

DomainSourceTargetSystems

Sentences

T1

News

CZ

EN

3727

T2

News

EN

CZ

2806

T3

EPPS

EN

FR

7577

T4

News

EN

FR

8561

T5

EPPS

EN

DE

6924

T6News

EN

DE

6892

T7

EPPS

EN

ES

6703

T8

News

EN

ES

7832

T9

EPPS

FR

EN

7624

T10

News

FR

EN

7740

T11

EPPS

DEEN

7949

T12News

DEEN

5939

T13

EPPS

ES

EN

8812

T14

NewsESEN

7668Updated: 18/10/2015

Metric Correlation with Human Scores
Pearson’s correlation coefficients between the mAM-FM Weighted
Mean (left) and xAM-FM Weighted Mean (right) components andhuman-generated scores for adequacy

Pg 36

Updated: 18/10/2015

mAM-FM and Adequacy

Pg 37

Updated: 18/10/2015

mAM-FM and Fluency

Pg 38

Updated: 18/10/2015

xAM-FM and Adequacy

Pg 39

Updated: 18/10/2015

xAM-FM and Fluency

Pg 40

Updated: 18/10/2015

Comparative Evaluation Results

All coefficients (except those marked with ‘*’) are significant with p<0.01

Pg 41

Updated: 18/10/2015

Human Adequacy and Fluency

Pg 42

Updated: 18/10/2015

AM and FM Metrics

Pg 43

Updated: 18/10/2015

Conclusions• We have proposed a new evaluation framework for MTevaluation operating on a continuous space• mAM-FM achieve better correlations with humanevaluations for both adequacy and fluency than otherconventional metrics• xAM-FM allows for quality assessment without theneed for a set of reference translations, its
performance is still comparable to other state-of-the-art automatic evaluation metrics
Pg 44

Updated: 18/10/2015

Thank You

Online:www.i2r.a-star.edu.sg
www.facebook.com/i2r.research