Improving Context-aware
Neural Machine Translation
with Target-side Context
Hayahide Yamagishi and Mamoru Komachi
Tokyo Metropolitan University, Tokyo, Japan@PACLING2019, Hanoi, Vietnam

Abstract• Neural Machine Translation has become popular.
• It is said NMT can consider sentence-level contexts.

• Recently, some researches expand the context window
from sentence-level to document-level→ Context-aware neural machine translation (CNMT)• Previous researches of CNMT found the source-sidecontext improves the performance.
• There are few researches using the target-side context.

• This research: how can we use the target-side context?
Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

1

NMT does not employ the context.
• Human translation can use the document context.
• Can keep the coherence of words and styles.

• The sentence in a pro-drop language such as Japaneseoften omits pronouns if it is apparent from a context.
• Google Translate （used in 9/30/2019）• Second sentence mistakes the possessor of “poster”, because
Japanese sentence omits the subject noun, “彼 (He)”.
• Todayʼs NMT cannot use the inter-sentential context.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

2

LSTM-based attentional NMT [Luong+, EMNLP15]• Encoder• Change the word sequence xm into hidden states sm andsentence representation.

• Attention• Calculate the attention vector ct which refers the source-side
information, using the hidden states of the encoder.

• Decoder• Generate the word yt using the attention vector ct , hidden
state of decoder ht and previous result yt-1 .

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

3

Multi-Encoder [Bawden+, NAACL18]• Most popular model of Context-aware NMT [Müller+,WMT18]• English-German [Jean+, arXiv 2017.04], [Müller+, WMT18]• English-French [Bawden+, NAACL18]• English-Russian [Voita+, ACL18]• Chinese-English [Zhang+, EMNLP18]• In this slide, we call this model “Separated model”

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

4Separated modelGet the information of two sentence with two encoders.
• Context Encoder: read a previous sentence as a context.
,-.
𝑣+,-. = LSTM4567897 𝑊; 𝑦+,-., 𝑣+-.
• Encoder: read an input sentence, = LSTM, , 𝑠,𝑠?
𝑊𝑥
8645@8A
B ? ?-.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

5Two patterns of context encoder• Separated source: use the source-side context.
,-.
𝑢+,-. = LSTM4567897 𝑊B 𝑥+,-., 𝑢+-.
• Separated target: use the target-side context.
,-.
𝑣+,-. = LSTM4567897 𝑊; 𝑦+,-., 𝑣+-.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

6

Separated model
The attention 𝒄,F and context attention 𝒄,-.
F vectors arecalculated with results of each encoder.
IJ𝒄,F = G softmax 𝒔,? O 𝒉,F 𝒔,? ,?H.

QJRS,-. O 𝒉, 𝒗,-.
𝒄,-.
F = G softmax 𝒗+F++H.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

7Separated model• The objective is maximizing 𝑝 for i = 1, … L, .
QJ,
𝑝 𝑌 , 𝑋 , , 𝑍 ,-. = Y 𝑝 𝑦F, 𝑦ZF, 𝑋 , , 𝑍 ,-.
FH.
,𝑝 𝑦F, 𝑦ZF, 𝑋 , , 𝑍 ,-. = softmax 𝑊[ ℎ\F, ,ℎ\F, = 𝑊^ [𝒉,F ; 𝒄,F ; 𝒄,-.
F ]Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

8The knowledge of the CNMT
• The context-aware NMT tackles coreference resolution.
[Tiedemann+, DiscoMT17, De-En], [Voita+, ACL18, En-Ru]• En-Ru MT: Pronouns in Russian are inflected by nouns incontext sentences.
• There is a possibility that multi-encoder-based CNMT can use
the context information.

• [Bawden+, NAACL18] say ...
• The source-side context is useful for CNMT.
• The target-side context is not useful for CNMT.
(context = previous sentence)

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

9Hypothesis• [Bawden+, NAACL18] only tried En-Fr translation.
• Is this tendency in distant language pairs, such as En-Ja?

• Is the model consistent?
• When the model uses the source-side context→ the source-side sentence is incorporated into an encoder• When the model uses the target-side context→ the target-side sentence is also incorporated into an encoder• In this research, we hypothesized thatthe target-side sentence should be incorporated intoa decoder.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

10

Proposed method: shared model• The shared model saves hidden states of the decoder.
• When translating the current sentence, this model uses
the saved states as an output of the context encoder.
• This model doesnʼt require many additional parametersand much computational times.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

11

Shared source and shared target• Shared source: using the source-side context.
• Shared target: using the target-side context.
• Saving the hidden states of decoder as a context.
• The target-side context can be incorporated into a decoder
instead of an encoder.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

12

Shared mix• If we use the context of both sides, what will happen?
→ Shared Mix model• Saving the both sides of hidden states, and using them.
,-. + 𝑐 ,-. .
• For keeping the parameter size, attention is 𝑐cde+dg

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

13

Related works of context-aware NMT• Hierarchical Encoder [Wang+, EMNLP17, Zh-En task]• First encoder: word embedding → sentence embedding• Second encoder: sentence embedding → document embedding• This model can use the context of several sentences.

• Cache [Tu+, TACL18, Zh-En task]• Saving generated words and hidden states in a cache.
• Calculating the attention with the cache.
• They found, “more than 5 contexts is not needed.”→ Both models need a huge neural architecture.
Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

14

Experiment

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

15

Datasets• TED Corpus（Subtitles on TED Talks）• 6 language pairs: De-En, En-De, Zh-En, En-Zh, Ja-En, En-Ja• Recipe Corpus (User-posted recipes on popular website in Japan)• 2 language pairs: Ja-En, En-Ja
Language Family

Word orderContext

En-De

Same

Similar (SVO - SOV (V2) )Low context

En-Zh

Different

Same (SVO)

Low context

En-Ja

Different

SVO - SOVHigh context (Ja)

CorpusTokenizer

Train

DevTestTED De-EnMoses Tokenizer

203,9988881,305

TED Zh-En

Jieba / Moses Tokenizer

226,196

879

1,297

TED Ja-En MeCab / Moses Tokenizer

194,170

871

1,285

Recipe Ja-En MeCab / Moses Tokenizer

108,990

3,303

2,804

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

16

Experimental Setup• Baseline: Global Attention NMT [Luong+, EMNLP15]• Hyper-parameters are as follows.

• BPE (the algorithm of subword tokenization [Sennrich+, ACL16])
was applied to the vocabulary of each language.
• Metrics: BLEU [Papineni+, ACL02]
Hyper-parameter

setup

Dimension of embedding

512

Dimension of hidden state

512

Vocabulary (subword level)

TED: 32,000, Recipe: 8,000

Minibatch size

128 documents

Encoder / Decoder

2-layer Bi-LSTM / 2-layer Uni-LSTM

Beam size

5Optimization

AdaGrad (Initial learning rate: 0.01)

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

17

Experimental Setup (for proposed method)• Pretrained with baseline models, except for the context
encoder.
• When the model translates the first sentence ofdocument, the context attention 𝑐 h = 𝟎• For separated models• The context encoder is initialized with random values• The reference is used as an input of the context encoder
during training of the separated target model.

• For shared models• The saved hidden states are used during training and test.

• Each experiment is executed three times.
• We show the average, the SD, and statistical significancecalculated by bootstrap resampling.
Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi18Result• The shared target improves the performance in all pairs.
• Improvement of the separated target is less comparedto the baseline.
• The target-side context should be introduced to adecoder.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

19Result• The shared source also improves the performances.
• Is weight sharing more efficient than learning contexts?
• Weight sharing between the layers keeps performance.
[Dabre+, AAAI19]• It can be seen as an instance of multitask learning between
translation and skip-thought.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

20Result• In Ja-En / En-Ja results, both sides of context are
equally important• In other cases, target-side context is more important.
• Does the word order have an important role in CNMT?
• High context language (Japanese) has unique characteristics?

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

21Convergence of training• Graph: the BLEU scores on development sets
• The separated target is unstable.
• This is due to the exposure bias between a decoder and acontext encoder• The shared target is stable because of no exposure biases.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

22

Sentence (the upper sentences represent context sentence.)Input

わかめはよく洗って塩を落とし、10分ほど水に浸けておいてからざく切りにする。長ねぎは小口切りにする。
熱した鍋にごま油をひき、わかめと長ねぎを入れて30秒ほど軽く炒める。

Wash the wakame well to remove the salt, put into a bowl ofwater for 10 minutes and drain. Cut into large pieces. Slice theReference Japanese leek.
Heat a pan and pour the sesame oil. Stir fry the wakame and leekfor 30 seconds.

Baseline

Wash the wakame seaweed well and remove the salt. Soak inwater for 10 minutes, then roughly chop. Cut the Japanese leek
into small pieces.
Heat sesame oil in a heated pot, add the wakame and leek, andlightly sauté for about 30 seconds.

Wash the wakame well, soak in water for about 10 minutes. Cut
Separated into small pieces. Cut the Japanese leek into small pieces.
TargetHeat the sesame oil in a frying pan, add the wakame and leek, andstir-fry for about 30 seconds.
Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

23

Sentence (the upper sentences represent context sentence.)Input

わかめはよく洗って塩を落とし、10分ほど水に浸けておいてからざく切りにする。長ねぎは小口切りにする。
熱した鍋にごま油をひき、わかめと長ねぎを入れて30秒ほど軽く炒める。

Wash the wakame well to remove the salt, put into a bowl of
water for 10 minutes and drain. Cut into large pieces. Slice theReference Japanese leek.
Heat a pan and pour the sesame oil. Stir fry the wakame and leekfor 30 seconds.

Baseline

Wash the wakame seaweed well and remove the salt. Soak inwater for 10 minutes, then roughly chop. Cut the Japanese leek
into small pieces.
Heat sesame oil in a heated pot, add the wakame and leek, andlightly sauté for about 30 seconds.

SharedTargetWash the wakame well, remove the salt, soak in water for about
10 minutes, then roughly chop. Chop the Japanese leek into smallpieces.
Heat sesame oil in a heated pan, add the wakame and Japaneseleek, and lightly stir-fry for about 30 seconds.

Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

24Conclusion• We reported how CNMT effectively employs thetarget-side context.
• The shared model can incorporate the target-sidecontext into a decoder instead of an encoder.
• The shared model achieves the high performance, eventhough it does not need additional costs.
• We found that the importance of context is different
between the language pairs.
• Our code is available: https://github.com/hargon24/Context̲aware̲NMT
Improving Context-aware Neural Machine Translation with Target-side Context, presented by Hayahide Yamagishi

25