A Bag of Useful Tricks For Neural Machine Translation:
Embedding Layer Initialization and Large Batch Size
*1
Masato Neishi ,*1
Jin Sakuma ,*1
Satoshi Tohda ,1
Shonosuke Ishiwatari ,2Naoki Yoshinaga ,{neishi, tohda, jsakuma, ishiwatari, ynaga, toyoda}@tkl.iis.u-tokyo.ac.jp

1The University of Tokyo

2
Masashi Toyoda

2Institute of Industrial Science (IIS), the University of Tokyo*contributed equally

System Overview of Team UT-IIS
Task:

Seq2seq with Attention:

ASPEC English - Japanese

Encoder

Seq2seq model with attention[Bahdanau+, 2015]+ Model independent tricks

Embedding Layer
Bi-LSTM
Attention

Decoder

Evaluation Metric
BLEU (KyTea)
Human Evaluation

Training Phase• Adam Optimization [Kingma and Ba, 2015]• Subword Translation (SentencePiece)• Embedding Layer Initialization• Large Batch Size
Prediction Phase• Ensemble of 8 Models• Beam Search (width=256)

𝑥% , 𝑥' , … , 𝑥) , … , 𝑥*+

Approach:Result:Tricks:

Score
38.93
68.000

LSTM
Embedding Layer

𝑦% , 𝑦' , … , 𝑦) , … , 𝑦*-Large Batch Size

Embedding Layer Initialization
Background:

Background:

Proposal:

Proposal:

Ramachandran+ (2017) obtained a significant BLEU gain
by using LSTM to pretrain layers of a network on a large
external corpus (one week on 32 GPUs).
Is external data necessary?

Morishita+ (2017) tested larger batch sizes, up
to 64, for improvements in (mini-batch) training
NMT.
How about even larger batch sizes?

Pretrain word embeddings from the training data only
• No additional resources• Very quick pretraining (less than 30 min on a CPU)
ParallelCorpus

Word Embedding
Training

Embedding
Layer
Word Embedding

Test whether translation quality will continue toimprove with batch sizes larger than 64
Mini-batch

Model

Experiments:

Experiments:
MethodRandom(Gaussian)
CBOW
Skip-gram
SI-Skip-gram
GloVe

Faster startup

Gradient𝜕𝐿𝜕𝑤
𝜕𝐿𝜕𝑤

Noisy
Less Noisy

BLEU

Diff.

Tradeoff

34.20-Pros:• Better optimalCons:• More memory needed• Slower update

35.50
34.44
34.44
34.58

+1.30
+0.24
+0.24+0.38• +1.30 BLEU (34.20 w/ Gaussian à 35.50 w/ CBOW)• CBOW performed best among initialization methods

Overall ResultTricks

ModelEmb. layer init.
remains effective• +5.19 BLEU (30.31 @64 à 35.50 @256)• Effect of large batch size saturates at 256

Translation Examples
SourceDoping induced a noticeable change at the lower boundary of the three-dimensional orderly
vortex phase.

BLEU

Gain

Baseline (existing tricks)

23.83-+ Embedding Layer Initialization

27.05

+3.22

Reference ドーピングにより三次元規則的渦糸相の下界に著しい変化が生じた。
No tricks ドーピングドーピングは3次元の秩序渦相の低下で注目可能な変化を誘導する。

+ Larger Batch Size

35.50+11.67

ドーピングは三次元秩序渦相の下部境界で顕著な変化を誘起した。

+ Ensemble of 8 Models

38.00+14.17

W/ tricks+ Beam Search (width=256)

38.93+15.10SourceThe outline of the 23rd white paper which is used to be issued yearly from Ministry of Posts andTelecommunications is specified.

Training Tricks
Prediction Tricks• +3.43 BLEU by prediction tricks• +15.10 BLEU against vanilla seq2seq• The tricks are simple and applicable
to any NMT model
Code available: https://github.com/nem6ishi/wat17

Reference 郵政省が毎年発表し，今回２３回目に当たる標記白書の概要を述べた。
No tricks st上から行われているまでに使用されるまでに使用されている23rd白色紙の概要を特定した。
W/ tricks

郵政省から発行されている第23回白書の概要を述べた。

SourceIt has been already entering into the ubiquitous society, and the diffusion of the portable
telephone is over 70% of the total population.

Reference すでに，ユビキタス社会の入り口にあり，携帯電話は総人口の７０％以上の普及率である。
No tricks 既にユビキタス社会に入院し,携帯電話の普及は全人口の70%以上である。
W/ tricks 既にユビキタス社会に入り,携帯電話の普及率は全人口の70%以上である。