Softmax Alternatives in Neural MT

Softmax Alternatives inNeural MT
Graham Neubig
5/24/20171

Softmax Alternatives in Neural MT

Neural MT Models
kouen wookonaimasu
giveatalkP(e1∣F ) P(e2∣F , e1 ) P(e3∣F , e21 ) P(e4∣F , e31 )argmaxgiveatalk
2Softmax Alternatives in Neural MT

How we Calculate Probabilitiesp(ei|hi) = softmax( W * hi + b )Next word prob. Weights Hidden context BiasbWb1[w*,1, w*,2, w*,3, ...]
b2In other words, the score is:s(ei|ci) = w*,k ・ ci + bkb3...

Closeness of output embedding and context+ bias. Choose word with highest score3

Softmax Alternatives in Neural MT

A Visual Example

p = softmax(Wh + b )4

Softmax Alternatives in Neural MT

Problems w/ Softmax●

Computationally inefficient at training time●

Computationally inefficient at test time●

Many parameters●

Sub-optimal accuracy

5Softmax Alternatives in Neural MT

Calculation/Parameter Efficient
Softmax Variants

6Softmax Alternatives in Neural MT

Negative Sampling/
Noise Contrastive Estimation●

Calculate the denominator over a subset

Wc+bW'Negative samples according to distribution qc+b'7

Softmax Alternatives in Neural MT

Lots of Alternatives!
●Noise contrastive estimation: train a model todiscriminate between true and false examples●

Negative sampling: e.g. word2vec●BlackOut
Used in MT:Eriguchi et al. 2016: Tree-to-sequence attentional neural machine translation 8Ref: Chris Dyer, 2014. Notes on Noise Contrastive Estimation and Negative SamplingSoftmax Alternatives in Neural MT

GPUifying Noise ContrastiveEstimation●●Creating the negative samples and arranging memoryis expensive on GPU
Simple solution: sample the negative samples once foreach mini-batch
Zoph et al. 2016. Simple, Fast Noise-Contrastive
Estimation for Large RNN Vocabularies

9Softmax Alternatives in Neural MT

Summary of Negative SamplingApproaches●

Train time efficiency: Much faster!

●

Test time efficiency: Same●

Number of parameters: Same●

Test time accuracy: A little worse?

●

Code complexity: Moderate

10Softmax Alternatives in Neural MT

Vocabulary Selection●

Select the vocabulary on a per-sentence basis
Mi 2016. Vocabulary Manipulation for NMT
L'Hostis et al. 2016. Vocabulary Selection Strategies for NMT

11

Softmax Alternatives in Neural MT

Summary of Vocabulary Selection●Train time efficiency: A little faster●

Test time efficiency: Much faster!

●

Number of parameters: Same●

Test time accuracy: Better or a little worse●

Code complexity: Moderate

12Softmax Alternatives in Neural MT

Class-based Softmax●●

Predict P(class|hidden), then P(word|class,hidden)
Because P(w|c,h) is 0 for all but one class, efficientcomputation

softmax(Wch + bc )softmax(Wwh + bw )13

Goodman 2001. Classes for Fast Maximum Entropy Training

Softmax Alternatives in Neural MT

Hierarchical Softmax●Tree-structured prediction of word ID●

Usually modeled as a sequence of binary decisions

01 110→ word 14

Morin and Bengio 2005: Hierarchical Probabilistic NNLM

14

Softmax Alternatives in Neural MT

Summary of Class-basedSoftmaxes●

Train time efficiency: Faster on CPU, Pain to GPU●

Test time efficiency: Worse●

Number of parameters: More●

Test time accuracy: Slightly worse to slightly better●

Code complexity: High15

Softmax Alternatives in Neural MT

Binary Code Prediction●

Just directly predict the binary code of the word IDσ(Wh + b )=01110↓

word 14●

Like hierarchical softmax, but with shared weights atevery layer → fewer parameters, easy to GPU

Oda et al. 2017: NMT Via Binary Code Prediction

16Softmax Alternatives in Neural MT

Two Improvements
Hybrid model

Error correcting codes

17Softmax Alternatives in Neural MT

Summary of Binary CodePrediction●

Train time efficiency: Faster●

Test time efficiency: Faster (12x on CPU!)●

Number of parameters: Fewer●

Test time accuracy: Slightly worse●Code complexity: Moderate18Softmax Alternatives in Neural MT

Parameter Sharing

19

Softmax Alternatives in Neural MT

Parameter Sharing●

We have two |V| x |h| matrices in the decoder:
Input word embeddings, which we look up and feed intothe RNN●
Output word embeddings, which are the weight matrix
W in the softmax●●

Simple idea: tie their weights together

Press et al. 2016: Using the output embedding to improve
language models
Inan et al. 2016: Tying Word Vectors and Word Classifiers:
A Loss Framework for Language Modeling20Softmax Alternatives in Neural MTSummary of Parameter Sharing●

Train time efficiency: Same●

Test time efficiency: Same●

Number of parameters: Fewer●

Test time accuracy: Better●Code complexity: Low

21

Softmax Alternatives in Neural MT

Incorporating ExternalInformation

22

Softmax Alternatives in Neural MT

Problems w/ Lexical Choice in
Neural MT

Arthur et al. 2016:
Incorporating Discrete Translation Lexicons in NMT

23

Softmax Alternatives in Neural MT

When Does Translation Succeed?
(in Output Embedding Space)
I come from Tunisia

w*,eatw*,swedenw*,tunisia
h1w*,nigeria

w*,consumew*,norway24

Softmax Alternatives in Neural MT

When Does Translation Fail?
Embeddings Version
I come from Tunisia

w*,eatw*,swedenw*,tunisiaw*,nigeria

h1w*,norwayw*,consume25

Softmax Alternatives in Neural MT

When Does Translation Fail?
Bias Version
I come from Tunisia

w*,eatw*,chinaw*,swedenw*,tunisia

w*,consume

h1w*,nigeria

w*,norway

btunisia = -0.5

bchina = 4.5

26

Softmax Alternatives in Neural MT

What about Traditional Symbolic
Models?
his father likes Tunisia

P(kare|his) = 0.5
P(no|his) = 0.5
P(chichi|father) = 1.0
P(chunijia|Tunisia) = 1.0
P(suki|likes) = 0.5
P(da|likes) = 0.5

karenochichi
wachunijia
gasukida1-to-1 alignment27

Softmax Alternatives in Neural MT

Even if We Make a Mistake...
his father likes Tunisia
karenochichi
wachunijia
gasukida☓☓

Different mistakes
than neural MT

P(kare|his) = 0.5
P(no|his) = 0.5
P(chichi|Tunisia) = 1.0
P(chunijia|father) = 1.0
P(suki|likes) = 0.5
P(da|likes) = 0.5
Soft alignment
possible28

Softmax Alternatives in Neural MT

Calculating Lexicon Probabilities
Attention I come from Tunisia
watashiore
…
kuru
kara
…
chunijia
oranda0.05 0.01

0.02

0.93

0.6
0.2
…
0.01
0.02
…
0.0
0.0

0.01
0.02
…
0.01
0.5
…
0.0
0.0

0.0
0.0
…
0.0
0.01
…
0.96
0.0

0.03
0.01
…
0.3
0.1
…
0.0
0.0

Word-by-wordlexicon prob

0.03
0.01
…
0.00
0.02
…
0.89
0.00

Conditional
lexicon prob
29

Softmax Alternatives in Neural MT

Incorporating w/ Neural MT●softmax bias:p(ei|hi) = softmax( W * hi + b + log (lexi + ε))
To prevent -∞ scores●Linear interpolation:p(ei|hi) = γ * softmax( W * hi + b ) + (1-γ) * lexi

30

Softmax Alternatives in Neural MT

Summary of External Lexicons●

Train time efficiency: Worse●

Test time efficiency: Worse●

Number of parameters: Same●

Test time accuracy: Better to Much Better●

Code complexity: High31

Softmax Alternatives in Neural MT

Other Varieties of Biases●Copying source words as-is
Gu et al. 2016. Incorporating copying mechanism insequence-to-sequence learning
Gulcehre et al. 2016. Pointing the unknown words●Remembering and copying target words
Were called cache models, now called ★pointersentinel models★ :)Merity et al. 2016. Pointer Sentinel Mixture Models
32Softmax Alternatives in Neural MT

Use of External Phrase Tables

Tang et al. 2016. NMT with External Phrase Memory33Softmax Alternatives in Neural MT

Conclusion

34

Softmax Alternatives in Neural MT

Conclusion●●

Lots of softmax alternatives for neural MT→ Consider them in your systems!
But there is no fast at train, fast at test, accurate,
small, and simple method→ Consider making one yourself!

35