Context-aware Decoder for Neural Machine Translation Using a Target-side Document-Level Language Model
Amane Sugiyama (U.Tokyo, sugi@tkl.iis.u-tokyo.ac.jp)

Document-level Machine Translationand the Standard Approach
Context may be necessary for correct translation

Approximate the objective function by sentence-level translation model,
document-level language model, and sentence-level language model scores.

Settings𝑦ො = arg max log 𝑝 𝑦 𝑥, 𝑐 𝑦= arg max log 𝑝 𝑐 𝑦 𝑥, 𝑦 𝑝 𝑦 𝑥≈ arg max log 𝑝 𝑐 𝑦 |𝑦 𝑝 𝑦 𝑥𝑦

堤防 riverのそばで発見されたHe (φ) was looking for you.

(φ)君を探していたよ= arg max log 𝑝 𝑦 𝑥 + log 𝑝 𝑦 𝑐

Split it into four pieces

Divisez-le en quatre.
Divisez-la en quatre.

Standard approach to context-aware translation
Directly optimize using parallel data with context
attached (document-level parallel data)
Source𝑐𝑥
End-to-End
Neural Network

Target𝑐𝑦
Prev. sent.
(prev. output)𝑦Curr. Output𝑦ො = arg max 𝑝 𝑦 𝑥, 𝑐 𝑥 , 𝑐 𝑦𝑦𝑥
Source Sent.

Most of existing parallel data are built from only
reliable sentence alignments in parallel/comparable
documents.
Can we perform document-level translation
without using document-level parallel data?

− log 𝑝 𝑦𝑐𝑦
Prev. output

Document-level LM𝑦ොOutput

Sentence-level TM

Sentence-level LM

Data

OpenSubtitles2018 (English -> Russian, parallel: 6M, monolingual: 30M)

ModelsSentence-level TM: Transformer base [Vaswani+ 2017]
Document/sentence-level LM: Decoder of Transformer

Overall translation performance measured by BLEU score
Model
Transformer w/ BT
Sentence-level TM
DocTransformer
Multi-encoder document-level translation [Zhang+ 18]
DocRepairsequence-to-sequence post-editing [Voita+ 18]
Bayes DocReranker
Reranking based on scores of S-TM, backward S-TM,
w/o context and D-LM [Yu+ 20]
Ours (Context-aware beam search)
Ours (Reranking with C-Score)para only
32.36
32.50
n/an/an/an/an/a(The actual model can be the same as D-LM)• Document-level parallel is not required for training• 𝑃𝑀𝐼 𝑐 𝑦 , 𝑦 = log 𝑝 𝑦 𝑐 𝑦 − log 𝑝 𝑦 represents association between 𝑦 and 𝑐 (𝑦)• A hyper param. 𝑇 [Guo+ 2017] is used to scale PMI: C−Score = log 𝑝 𝑦 𝑥 + log 𝑝 𝑦 𝑐 𝑦 − log 𝑝 𝑦 /𝑇

Only ours (rerank) and Bayes DocReranker achieved significant improvements overTransformer. Bayes DocReranker performed almost as well without context.
A brief discussion of inference speed is provided in the paper.

Decoding Strategy

Evaluation of the ability to capture context [Voita+ 2019]

Generate n-best hypotheses by sentence-level decoding and select the one that maximizes C-Score
Context (previous translation)

Source(φ)君を探していたよN-best Hypotheses
I was looking for you.

TMHe was looking for you.
log 𝑝(𝑦|𝑥)

Did you see Mary?
log 𝑝 𝑦 𝑐 𝑦(Step 2)
Compute PMI for each hypotheses,
D-LM
Get 𝑦ො = arg max C−Score 𝑦; 𝑥, 𝑐 𝑦
S-LM

She was looking for you.
−log 𝑝 𝑦 𝑐 𝑦(Step 1) Generate hypotheses+30M mono
32.40
31.59
32.35
33.75**
33.67**
32.27
32.93*

Models guess the correct translation out of several candidates based on the translation score

Reranking with C-Score (§2.2.1)

𝑦= “She was looking for you”

Deixis (person deixis), lex.c (lexical cohesion), ell.infl (inflection of Russian nouns caused by ellipsis), and ell.vp(verb ellipsis in English text not allowed in Russian)ModelDocTransformer
DocRepair
Bayes DocReranker
C-ScorePMIdeixis
50.0
89.1
65.2
86.9
96.8

lex.c
45.9
75.8
72.2
94.9
97.8

ell.infl
56.0
82.2
59.6
78.2
75.8

ell.vp
57.2
67.2
44.6
77.0
90.6

C-Score achieves better lex.c and ell.vp
socres and comparable deixis andell.infl scores to the DocRepair

Analysis: How models change source-target PMI correlation?

Context-aware Beam Search (§2.2.2)
Decompose the sentence C-Score into token-wise C-Score and perform beam search
C−Score 𝑦; 𝑥, 𝑐 𝑦

Problem: Lack of document-level parallel data𝑦Objective C−Score 𝑦; 𝑥, 𝑐 𝑦 = log 𝑝𝑆−𝑇𝑀 𝑦 𝑥 + log 𝑝𝐷−𝐿𝑀 𝑦 𝑐 𝑦 − log 𝑝𝑆−𝑇𝑀 𝑦She (φ) was looking for you

Grammatical Gender (En-Fr)

𝑝 𝑐 𝑦 |𝑥, 𝑦 ≈ 𝑝 𝑐 𝑦 𝑦Assuming x and y aresemantically similar𝑦𝑦

Zero pronoun (Ja-En)

𝑥
Curr. Sent.

Experiments

銀行 financialのそばで発見された

It was found near the bankPrev. sent.

Decoding with a Document-level Language Model𝑦Homonymy (En-Ja)

Naoki Yoshinaga (IIS, U.Tokyo, ynaga@iis.u-tokyo.ac.jp)

Each point stands for a pair of PMI: PMI 𝑐 𝑦𝑖 , 𝑦𝑖𝑚𝑜𝑑𝑒𝑙 , PMI 𝑐 𝑥𝑖 , 𝑥𝑖 for the i-th sentence pair in a dev
set where model is the reference 𝑦𝑖 and outputs of S-TM, proposed beam search with 𝑇 = 0, and proposed
beam search with 𝑇 = 4.

= ෍ log 𝑝𝑆−𝑇𝑀 𝑦𝑖 𝑥, 𝑦<𝑖 + log 𝑝𝐷−𝐿𝑀 𝑦𝑖 𝑐 𝑦 , 𝑦<𝑖 − log 𝑝𝑆−𝐿𝑀 𝑦𝑖 𝑦<𝑖𝑖(φ) 君を 探していたよ.

<S>

Encoderlog 𝑝 𝑦𝑖 𝑥, 𝑦<𝑖Decoder
She φ was looking foryou

Sentence-level LM

Document-level LM
log 𝑝 𝑦 𝑐𝑦, 𝑦<𝑖 − log 𝑝 𝑦|𝑦<𝑖(PMI)
Did

you see Mary ?

AA
The hypotheses for reranking have high reliability but may suffer from the low diversity problem

BAB

B

B

A: Sentence-level Transformer fails to reflect context in translation (low target-side PMI).
B: Decoding with C-Score with T=1 suffers from over-correction

Conclusion
You can perform document-level translation using sentence-level translation model and language model.
Experiment results for En-Fr and En-Ja are available in the v1 paper on arXiv.org (https://arxiv.org/abs/2010.12827v1)