Neural Networks Language Models
Philipp Koehn
28 September 2023

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

N-Gram Backoff Language Model1• Previously, we approximatedp(W ) = p(w1, w2, ..., wn)• ... by applying the chain rulep(W ) =Xp(wi|w1, ..., wi−1)i• ... and limiting the history (Markov order)p(wi|w1, ..., wi−1) ' p(wi|wi−4, wi−3, wi−2, wi−1)• Each p(wi|wi−4, wi−3, wi−2, wi−1) may not have enough statistics to estimate→ we back off to p(wi|wi−3, wi−2, wi−1), p(wi|wi−2, wi−1), etc., all the way to p(wi)– exact details of backing off get complicated — ”interpolated Kneser-Ney”

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Refinements2• A whole family of back-off schemes• Skip-n gram models that may back off to p(wi|wi−2)• Class-based models p(C(wi)|C(wi−4), C(wi−3), C(wi−2), C(wi−1))⇒ We are wrestling here with– using as much relevant evidence as possible– pooling evidence between words

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

First Sketch

3

wiSoftmax

Output WordhFF

Hidden Layer

wi-4

Philipp Koehn

wi-3

wi-2

wi-1

Machine Translation: Neural Networks

History

28 September 2023

Representing Words4• Words are represented with a one-hot vector, e.g.,– dog = (0,0,0,0,1,0,0,0,0,....)– cat = (0,0,0,0,0,0,0,1,0,....)– eat = (0,1,0,0,0,0,0,0,0,....)• That’s a large vector!
• Remedies– limit to, say, 20,000 most frequent words, rest are OTHER√– place words in n classes, so each word is represented by∗ 1 class label∗ 1 word in class label– splitting rare words into subwords– character-based models
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

5word embeddings

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Add an Embedding Layer

6wiSoftmax

Output Word

hFF

Hidden Layer

Ew

Embed

Embed

Embed

Embed

Embedding

wi-4

wi-3

wi-2

wi-1History• Map each word first into a lower-dimensional real-valued space• Shared weight matrix E
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Details (Bengio et al., 2003)7• Add direct connections from embedding layer to output layer• Activation functions– input→embedding: none
– embedding→hidden: tanh
– hidden→output: softmax• Training– loop through the entire corpus– update between predicted probabilities and 1-hot vector for output word

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Word Embeddings
Word8EmbeddingC• By-product: embedding of word into continuous space• Similar contexts → similar embedding• Recall: distributional semantics

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Philipp Koehn

Word Embeddings

9Machine Translation: Neural Networks

28 September 2023

Philipp Koehn

Word Embeddings

10Machine Translation: Neural Networks

28 September 2023

Are Word Embeddings Magic?

11• Morphosyntactic regularities (Mikolov et al., 2013)– adjectives base form vs. comparative, e.g., good, better– nouns singular vs. plural, e.g., year, years– verbs present tense vs. past tense, e.g., see, saw• Semantic regularities– clothing is to shirt as dish is to bowl
– evaluated on human judgment data of semantic similarities
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

12

recurrent neural networks

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Recurrent Neural Networks

013

Softmax

Output Wordtanh

Hidden Layer

Embed

Embedding

w1History• Start: predict second word from first• Mystery layer with nodes all with value 1
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Recurrent Neural Networks

Softmax

Output Word

tanh

Hidden Layer

Embed

Embed

Embedding

w1

w2

History

Softmax

tanh0

Philipp Koehn

14

copy

Machine Translation: Neural Networks

28 September 2023

Recurrent Neural Networks

Softmax

tanh0

Philipp Koehn

Softmax

copytanhcopy

15

Softmax

Output Word

tanh

Hidden Layer

Embed

Embed

Embed

Embedding

w1

w2

w3

History

Machine Translation: Neural Networks

28 September 2023

Training

16

CostytSoftmax

Output Word

RNN

Hidden Layer

Ewt

Embed

Embeddingwt

w1

History

ht0• Process first training example• Update weights with back-propagation

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Training

17

Cost

yt

Softmax

Output Word

RNN

Hidden Layer

Ewt

Embed

Embeddingwt

w2

Historyht

RNN• Process second training example• Update weights with back-propagation• And so on...
• But: no feedback to previous history
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Back-Propagation Through Time

18

Cost

Cost

Cost

Softmax

Softmax

Softmax

Output Word

RNN

RNN

RNN

Hidden Layer

Ewt

Embed

EmbedEmbedEmbedding

wt

w1

w2

w3

Historyytht0• After processing a few training examples,
update through the unfolded recurrent neural network

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Back-Propagation Through Time19• Carry out back-propagation though time (BPTT) after each training example– 5 time steps seems to be sufficient– network learns to store information for more than 5 time steps• Or: update in mini-batches– process 10-20 training examples– update backwards through all examples– removes need for multiple steps for each training example

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

20

long short term memory

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Vanishing Gradients21• Error is propagated to previous steps• Updates consider– prediction at that time step
– impact on future time steps• Vanishing gradient: propagated error disappears

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Recent vs. Early History

22• Hidden layer plays double duty
– memory of the network– continuous space representation used to predict output words• Sometimes only recent context important
After much economic progress over the years, the country → has• Sometimes much earlier context important
The country which has made much economic progress over the years still → has

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Long Short Term Memory (LSTM)

23• Design quite elaborate, although not very complicated to use• Basic building block: LSTM cell
– similar to a node in a hidden layer– but: has a explicit memory state• Output and memory state change depends on gates– input gate: how much new input changes memory state– forget gate: how much of prior memory state is retained– output gate: how strongly memory state is passed on to next layer.
• Gates can be not just be open (1) and closed (0), but slightly ajar (e.g., 0.2)

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

LSTM Cell

24

LSTM Layer Time t-1m⊗

Xi

input gate⊗ ⊕moutput gate

Preceding Layerforget gate⊗

Next LayerohY

LSTM Layer Time t

Philipp Koehn

Machine Translation: Neural Networks28 September 2023

LSTM Cell (Math)25• Memory and output values at time step t
memoryt = gateinput × inputt + gateforget × memoryt−1outputt = gateoutput × memoryt• Hidden node value ht passed on to next layer applies activation function f
ht = f (outputt)• Input computed as input to recurrent neural network node
– given node values for prior layer ~xt = (xt1, ..., xtX )t−1– given values for hidden layer from previous time step ~ht−1 = (ht−1,...,h1H )– input value is combination of matrix multiplication with weights wx and whand activation function g!
XHXXtx t
input = gwi xi +wihht−1i
i=1Philipp Koehn

i=1

Machine Translation: Neural Networks

28 September 2023

Values for Gates

26• Gates are very important• How do we compute their value?
→ with a neural network layer!
• For each gate a ∈ (input, forget, output)– weight matrix W xa to consider node values in previous layer ~xt– weight matrix W ha to consider hidden layer ~ht−1 at previous time stept−1~– weight matrix W ma to consider memory at previous time step memory– activation function hgatea = hX
X
i=1

Philipp Koehn

wixaxti +HX
i=1+
wihaht−1iHX!
wimamemoryt−1i

i=1Machine Translation: Neural Networks

28 September 2023

Training27• LSTM are trained the same way as recurrent neural networks• Back-propagation through time
• This looks all very complex, but:– all the operations are still based on∗ matrix multiplications∗ differentiable activation functions→ we can compute gradients for objective function with respect to all parameters→ we can compute update functions

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

What is the Point?

28(from Tran, Bisazza, Monz, 2016)• Each node has memory memoryi independent from current output hi• Memory may be carried through unchanged (gateiinput = 0, gateimemory = 1)
⇒ can remember important features over long time span
(capture long distance dependencies)

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Visualizing Individual Cells

29

Karpathy et al. (2015): ”Visualizing and Understanding Recurrent Networks”
Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Philipp Koehn

Visualizing Individual Cells

30

Machine Translation: Neural Networks

28 September 2023

Gated Recurrent Unit (GRU)31

GRU Layer Time t-1h⊗

Preceding Layer
Xx

update gate

reset gate⊕

Next LayerhY

GRU Layer Time t

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Gated Recurrent Unit (Math)32• Two Gates
updatet = g(Wupdate inputt + Uupdate statet−1 + biasupdate)resett = g(Wreset inputt + Ureset

statet−1 + biasreset)• Combination of input and previous state(similar to traditional recurrent neural network)
combinationt = f (W inputt + U (resett ◦ statet−1))• Interpolation with previous state
statet =(1 − updatet) ◦ statet−1 +updatet

Philipp Koehn◦ combinationt) + bias

Machine Translation: Neural Networks

28 September 2023

33

deeper models

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Deep Learning?

34

ytSoftmax

Softmax

Softmax

Output

ht

RNN

RNN

RNN

Hidden
Layer

E xt

Embed

Embed

Embed

Input Word
Embedding

Shallow• Not much deep learning so far• Between prediction from input to output: only 1 hidden layer• How about more hidden layers?

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023

Deep Models

35

yt

Softmax

Softmax

SoftmaxytSoftmax

Softmax

Softmax

Output

ht,3

RNN

RNN

RNN

ht,3

RNN

RNN

RNN

HiddenLayer 1ht,2

RNN

RNN

RNN

ht,2

RNN

RNN

RNN

HiddenLayer 2ht,1

RNN

RNN

RNN

ht,1

RNN

RNN

RNN

HiddenLayer 3E xi

Embed

Embed

Embed

E xiEmbedEmbed

Embed

Input Word
Embedding

Deep Stacked

Philipp Koehn

Deep Transitional

Machine Translation: Neural Networks

28 September 2023

36

questions?

Philipp Koehn

Machine Translation: Neural Networks

28 September 2023