On the Relation between Position Information andSentence Length in Neural Machine Translation
Masato Neishi

Naoki Yoshinaga

The University of Tokyo
neishi@tkl.iis.u-tokyo.ac.jp

Institute of Industrial Science, the University of Tokyoynaga@iis.u-tokyo.ac.jp

Code available: https://github.com/nem6ishi/conll19̲relative̲transformer

Summary

NMT has difficulty in translating long sentences.
Problem:Hypothesis: Word position encoding significantly affects the performance.
Relative (ex. RNN) vs. Absolute (ex. Positional Encodings)
Conclusion: Relative position is better and prevents overfitting to the sentence length.

1. Backgroundu Long sentence: A major problem in NMT• Attention mechanism helps RNN-based NMT model tomitigate this problem. [Bahdanau+, 2015; Luong+, 2015]• RNN-based NMT < Phrase-based SMT in translating
very long (>80) sentences. [Koehn and Knowles, 2017]
Does Transformer [Vaswani+ 17], NMT model superior
to RNN-based one, work well for long sentences?
- No, it is worse. (cf. 5)
2. Preliminary: Type of position information

Ex. Encoder of Transformerℎ"ℎ#ℎ$)(Next to thepredecessor((!"!#!$((()!%ℎ%2nd)((!"!$!#!%

Need to learn to process theposition vector.
Less chance to learn large
positions.

No explicit position
representations to learn.

Hypothesis*The number of parameters set to be almost equal.

u Datasets (preprocessed):• WMT2014 English-to-German (3.7M sentences)• ASPEC English-to-Japanese (1.2M sentences)*Sentences longer than 49 tokens are filtered out.

5. Result & Analysis

Transformer and RNN-based NMT differ inposition information to handle variable-length input.
u Absolute positionu Relative position
Ex. Encoder of RNN-based modelℎ#ℎ$ℎ"ℎ%

4. Experimental Settingsu Models and their types of position information:• RNN-NMT[Luong+, 2015], (Relative)• Transformer (Absolute) and its three variants (Relative)u BLEU score [Papineni+, 2002]
WMT2014 En-De
19.95
21.00
22.51
22.35
23.01

RNN-NMT
Transformer
Rel-Transformer
RNN-Transformer
RR-Transformer• Among Transformers, Relative beats Absolute.
• RR-Transformer performs the best.

[Analysis on WMT2014] (See our paper on ASPEC)u Evaluation on test data split by input length
BLEU score

The type of position information significantlyaffects the translation of long sentences.
3. Approach: Transformer with Relative Position
Compare the types of position information
using Transformer.
Position information customizable!

u [Shaw+. 2018]: Self-attention with relative position• Introduce relative position vectors into self-attention
process (and remove positional encoding layer).
Need to learn to process the position vector,
but more chance to learn large position.

ASPEC En-Ja
36.67
38.44
39.58
39.17
40.34

Gap between reference
and NMT's output

No training data

No training data

• Transformer fails to translate long sentences, andoverfits to short input sentences in the training data.
• Relative position avoids this overfitting.
Does Transformer overfit to short input
sentences only?
u Results when trained on length-controlled data

[The modified self-attention process]Input sentence length: 34-49

u Proposal: RNN as a Relative Positional Encoder

BLEU score• Replace positional encoding layers by RNN.

Almost No training data

[Original]

Gap between reference
and NMT's output
Almost No training data

Fixed vector using
sine & cosine functions.

[Proposed]

Variants of Transformer for comparison,

881211888--1+1 8,8-/ 180-1/,71+0++---1/

118• Transformer overfits tothe lengths of input sentences in the training data.

888++ - 10/,710/

0*Modifications are applied to both encoder & decoder.

6. Conclusion• Relative position shows better translation quality
while Absolute position causes overfitting.
ü TAKE AWAY: Use Relative position in NMT.