Attainable Text-to-text MT vs. Translation:
Issues Beyond Linguistic Processing
Atsushi Fujita
NICT, JapanGiven a document, how would you translate it?
…
So the following statement should have been taken into account, among
others:States Parties particularly condemn racial segregation andapartheid and undertake to prevent, prohibit and eradicate allpractices of this nature in territories under their jurisdiction.
Justice Baltasar Garzón decided to indict Augusto Pinochet in 1998, he was…

2

Given a document, how would you translate it?
Register? Text type? Readers?

Purpose? Usages?

…
So the following statement should have been taken into account, among
others:States Parties particularly condemn racial segregation andapartheid and undertake to prevent, prohibit and eradicate allpractices of this nature in territories under their jurisdiction.
Justice Baltasar Garzón decided to indict Augusto Pinochet in 1998, he was…

Quote from an international treaty

Person names

Does official translation exist?

Existing transliteration?

3Translation
Producing a document in the target languagethat plays the same role as the given document
The production workflow refers tol

Not only documents
Text segments (as in MT)
n Other modals within the given documentnl

Extra-document and non-linguistic information
Designated specifications [ISO/TC37 15]
n Norm [Toury 78]
nConventions and criteria in the target language and registerl Register: determined by the subject field, expected readers, mode, etc.
ln

Skopos [Vermeer 89/04]: objective and intended usages of final product4

Our aim
Clarify the gap between MT and translation
through analyzing translation issues that MT systems causel (1) Collecting issues and their revisions through PEl (2) Clarifying the types of necessary informationlDiscuss
Future research directionsl How to properly reflect necessary information in MT
l5

Collecting and analyzing
revision examples of translation issues

A strategy to fill the gap between MT and translation
MT+PE workflow
lISO 18587 [ISO/TC37 17]
Source document

Segment-leveltext-to-text process(1) MT

Translation process
Translation
Target document(draft)Revision & review(2)
Full PE••

Remaining issues of text-to-text MT
Issues beyond text-to-text MT

Target document(final)Non-MT workflow

MT+PE
7

Our 2-stage PE workflow
Given MT outputs (1), we perform(2) Segment-level minimal PE: referring only to each segment, only necessary editingl (3) Document-level full PE: referring to any information, perform all necessary editing
lSource document

Segment-leveltext-to-text process(1) MT

Translation process
Translation(2) Minimal PE

Target document(draft)Revision & review

Full PE(3)
Full PE

Target document(final)Non-MT workflow

MT+PEOurs8

(1) Segment-level text-to-text MT
A case study: English-to-Japanese news translation task
Asian Language Treebank [Riza+ 16]
l English-to-Japanese NMT systemlTransformer Base [Vaswani+ 17]
n Trained via a 3-phase domain adaptation[Chu+ 17][Dabre+ 19]n

1. Pre-training (gigantic in-house data)
2. Mixed fine-tuning(gigantic in-house data + 18k ALT training data)
3. Pure fine-tuning (18k ALT training data)

Starting with translations
with reasonably high BLEU scorel See the paper for detailsl

ALT-Standard-Split

#Doc.

#Seg.

Training

1,698

Development
Test

#Tok.
EnJa

18,088472.1k

610.8k981,000

25.6k

33.0k

97

1,018

26.3k

33.4k

Translation

BLEU

HTER

Output of NMT trained only on ALT

14.673.9

Output of NMT in phase 1

29.055.5

Output of NMT in phase 2

35.8

47.6

Output of NMT in phase 3

36.0

47.6

9

(2) Segment-level minimal PE
Shuffle all the 1,018 segments in the ALT test data
Perform minimal PE for each segment
at an experienced, ISO-certified TSP [ISO/TC37, 15;17]
l referring only to the segment itself
l “minimal PE” (cf. [Scarton+ 15])lnTER(mt à pe) ≦ TER(mt à ref)mt: MT output, pe: PE result, ref: reference translationl Restart from mt if pe does not satisfy the constraintlnProhibit subjective stylistic changes

Results
95% (970/1,018) of the segments were revisedl The closest goal of segment-level text-to-text MT
l10

(3) Document-level full PE
Reorder the text segments into the original 97 documents
Perform full PE for each document
at an experienced, ISO-certified TSP [ISO/TC37, 15;17]
l as usual MT+PE workflowlnregarding the segment-level PE results as the raw MT outputResultsl

31% (320/1,018) of the segments were revised

11An example (DocID=139760, SntID=6023)
SourceJohn Paul’s six-day tour was hugely popular, attracting hundreds of thousandsof people from the Catholic religion. It became the first ever UK visit from thePope, which will make Pope Benedict’s visit the second papal one.
MT output(1) Segment-level text-to-text MT

ジョン・ポールの６日間のツアーは非常に人気があって、カトリックの宗教から何十万もの人々を引きつけた。それはローマ教皇からの初めての英国訪問となり、ローマ教皇ベネディクトの第二の教皇訪問となる。

Result of segment-level minimal PE(2) Segment-level minimal PEジョン・ポールの６日間のツアーは非常に人気があり、カトリックを信仰する何十万もの人々を引きつけた。それはローマ教皇の初めての英国訪問となり、ローマ教皇ベネディクトの訪問は２度目の教皇訪問となる。

Result of document-level full PE(3) Document-level full PE

法王ヨハネ・パウロの６日間の訪問は非常に人気があり、カトリックを信仰する何十万もの人々を引きつけた。この時がローマ法王の初めての英国訪問となり、法王ベネディクトの訪問は２度目の法王訪問となる。
12

Error analysis
Objective: clarify the types of necessary information
ProcedurelExtract revision examples produced during stage (3)Identify the corresponding text spans, also referring to the source text
n 529 examples
nlAnnotate each example (an annotator ≠ the PE worker)Need for document-level textual informationn Need for extra information
nlnand the type(s) of such information

Issue type among 16 types [Fujita+ 17]

13

A simplified example (Figure 2 in the paper)
SourceClemens (3-0, 1.90 ERA in seven World Series starts) will make his 33rd career postseason
start Saturday, at least for a day matching Pettitte for the most ever.
(1) Segment-level MT & (2) Segment-level PE

Result of segment-level minimal PE

クレメンス（ワールドシリーズ７回出場で３対０、防御率１．９０）は、少なくとも１日はペティットと組んで、土曜日に３３回目のポストシーズンのスタートを切る。

(3) Document-level full PE

Result of document-level full PE

クレメンス（ワールドシリーズ７回出場で３勝０敗、防御率１．９０）は、少なくとも１日はペティットと史上最多で並び、土曜日に生涯でポストシーズン33回目の先発登板を行う。
３対０

３勝０敗

3 points vs. 0 points

3 wins and 0 losses[]

生涯でNo, Yes, X3:distortion

No, No, X1:omission

in one’s life[]〜と組んで〜と並び

pairedranked the same

３３回目のポストシーズンの33rd postseason

史上最多で No, No, X1:omission
most ever

No, Yes, X3:distortion

ポストシーズン33回目の No, Yes, X3:distortion
33rd time in postseason

スタートを切る

先発登板を行うstart

to be the first pitcher of the game

No, Yes, X3:distortion
14

Necessity of extra information
Extra info.

Necessary info.
Document-leveltextual info.

No needNecessaryNo need196

168

Necessary116

4940% (217/529) of revisions relied on extra informationl(A) Fine-grained style specifications (121/217)
nlll

John Paul

Incorrect term translations, untranslated Latin names(C) Domain-specific knowledge (31/217)n

抽選

ジョン・ポール

ヨハネ・パウロ/dʒ’ɔn p’ɔ:l//johʌnɛ pʌʊlə/

3-0

Misunderstandings of the contents, inappropriate lexical choices(D) Reference documents (8/217)
n抽せん

Violation of circumstances: vocabulary, character, usages of symbols(B) Terminology (80/217)
ndrawing

Wrong disambiguations, remaining ambiguities

brother

３対０

３勝０敗

3 points vs. 0 points

3 wins and 0 losses

兄弟

兄elder andyounger brothers

elder brother15

Discussion on future research directions

What is the appropriate goal of MT (and researchers)
Full automatic MT is infeasible [Hutchins+ 92]
Human-Machine cooperation is vital (e.g., MT+PE)
lHumans (users of MT)
Determine the best practices in using MT
n Acquire translation literacy [Klitgård 18] and MT literacy [Bowker+ 19]
nll

Our data could be a useful learning material

Machine (to be used by humans in production workflows)nResolve the remaining issues of text-to-text MTl

60% of document-level PE rely only on text

Enable MT conform to the given criteria
n Enable MT confirm the entities and factualityn17

Toward properly reflecting extra information
How to realize perfect adherence
to the given criteria?
l



ii?

in]

Constrained decodingh/cp

A variety of rules and constraints
npmip:6odoopo
8 ? :5cmimcmpu+-[)d*ap

Vocabulary filtering [Ha+ 17]
l Terminology guided decoding[Hasler+ 18, Post+ 18, Zhang+ 18]+gln

Post-processing
llppeb? 5: 8bcc

[rblue.g., rules/pattern in CAT tools

cf. Indirect supervision throughcorpus-level adaptationol*tuybosl

4q(lltDomain adaptation [Chu+ 17]
n Style transfer [Niu+ 17, Michel+ 18]"cp'%(n!
l?

12llo:ob:18

Conclusion and future work
Error analysis of NMT
Manual PE for filling the gap between MT and translationl Types of necessary informationlDiscussion
Future research directionsl How to properly reflect necessary information?
lFuture & ongoing work
Evaluate the impact of enforcing decodingl Compile educational materials
l19