A trip down long short-term memory lane
Petar Veličković
Artificial Intelligence Group
Computer Laboratory, University of Cambridge, UK
Research Students Lecture Series

21 February 2017

Introduction

I In this lecture, I will introduce recurrent neural networks fromessential first concepts.
I This will cover the application of simple RNNs to solve

sequential problems. . .
I . . . followed by a deep sweep through long short-term

memories (LSTMs). . .
I . . . concluding with a variety of modern tips ‘n’ tricks fordeploying LSTMs, along with a neat demo (time permitting).
I Let’s get started!

The three “flavours” of machine learning

I Unsupervised learning

I Supervised learning(the only kind of learning neural networks can directly do!)I Reinforcement learningSupervised learning

I The environment gives you labelled data (∼ known input/outputpairs)—and asks you to learn the underlying function.

y’(x~1 , y1 ), . . . , (x~n , yn )AgentEnvironment
x~0I Example: determining whether a person will be likely to return

their loan, given their credit history (and a set of previous dataon issued loans to other customers).

Classification
I Specifically, we will focus on classification—assuming ouroutputs to come from a discrete set of classes.

Neural networks

I To make life simpler (esp. notationally!), let’s start with an

introduction to simple neural networks.
I Neural networks are structures of interconnected processing

units (neurons).
I Each neuron computes a linear combination of its inputs,

afterwards potentially applying an activation function, toproduce its output.
I Occasionally, I will illustrate how to specify neural networks ofinterest using Keras (keras.io). (highly recommended!)A single neuron
Within this context sometimes also called a perceptron (. . . )x2x3..
.
xnw2

nP~)=σ b+wi xih(~x ; wΣσ

i=1w3
n

x1bw1

w

+1

Popular choices for the activation function σ:
I Identity: σ(x) = x;
I Rectified linear unit (ReLU): σ(x) = max(0, x);
1I Sigmoid functions: σ(x) =1+exp(−x) (logistic); σ(x) = tanh x.

Activation functions

2Common activation functions

1.51σ(z)0.50−0.5−1Identity
LogisticTanh
ReLU−1.5−2−2 −1.8−1.6−1.4−1.2 −1 −0.8−0.6−0.4−0.2 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2z

Neural networks and deep learning

I It is easy to extend a single neuron to a neural network—simply

connect outputs of neurons to inputs of other neurons.
I We may do this in two ways:
II

Feedforward: the computation graph does not have cycles;
Recurrent: the computation graph has cycles.

I Typically we organise neural networks in a sequence of layers,

such that a single layer only processes output from theprevious layer. Everything with > 1 hidden layer is “deep”!

A few details on training

I Neural networks are trained from known (input, output)

samples. The training algorithm adapts the neurons’ weights tomaximise predictive power on the training examples.
I This is done, for a single training example (~x , y), by:~ );Computing the output of the network y 0 = h(~x ; wDetermining the loss of this output L(y , y 0 );I Computing partial derivatives of the loss with respect to eachweight, ∂∂L~ , and using these to update weights.
wI Key words: backpropagation, stochastic gradient descent.
IIA simple classifier
Let’s ignore the activation functions and “deep learning” fornow. . . here is a simple, shallow, 4-class classifier.
+1x1
x2x3..
.
xnΣΣΣΣ

Choose the class which has the maximal output:
P
C = argmaxj bj + ni=1 wij xi

Block notation
Note that this layer is essentially doing a matrix multiplication. . .

~xW×+~bC = argmaxj W~x + ~bN.B. W of size 4 × n, ~b of size 4!

jSoftmax

I Problem: what should the targets be?
I Outputs are unbounded! For an example of the second class,the targets should be ~y = −∞ +∞ −∞ −∞ . . .
I Solution: transform the outputs monotonically to the [0, 1]

range, using the softmax function:exp(zi )softmax(~z )i = P
j exp(zj )Probabilistic classification

I This conveniently also makes the outputs add up to 1, so wecan interpret yi0 = softmax(h(~x ))i = P(~x in class i).
I Now the target for an example of the second class should be~y = 0 1 0 0 (∼ one-hot encoding).
I Typically express the loss function as the cross-entropy:

L(~y , ~y 0 ) =KX
i=1

where K is the number of classes.

yi log yi0

Back in business~xW×+~bsoftmax

Integrating into our simple classifier:



~~
C = argmaxj softmax Wx + bj

Going deeper with LEGOTM~x

W1 ×+~b1
C = argmaxj

W2 ×softmax

Making things deep is now easy. . .

+~b2
~~
softmax W2 ReLU W1~x + b1 + b2




jN.B. the ReLU is important! A composition of linear functions isitself a linear function. . .

Fully connected layers
The “matrix-multiply–bias–activation” (sometimes also called fully
connected or Dense) layer is a common building block of neuralnetworks.

Dense(7)Dense(4)~xsoftmax

Keras code:x = Input(shape=(7,))
h = Dense(7, activation=’relu’)(x)y = Dense(4, activation=’softmax’)(h)Sequential inputs

I Now, consider a classification problem where the input issequential—a sequence consisting of arbitrarily many steps,
wherein at each step we have n features.

~x1 ~x2 ~x3 ~x4 ~x5 . . . ~xt . . .

I The fully connected layers will no longer suffice, as they expect

a fixed-size input!

Making it work

Key ideas:
I Summarize the entire input into m features (describing themost important patterns for classifying it);
I Exploit relations between adjacent steps—process the input ina step-by-step manner, iteratively building up the features, ~h:~ht = f (~xt , ~ht−1 )I If we declare a pattern to be interesting, then it does not matter

when it occurs in the sequence =⇒ employ weight sharing!

An RNN cell

~yt

~yt−1

RNN~xtUnrolling the cell. . .

Compute ~yT iteratively, then feed it into the usual
fully connected network to get the final answer.

~y0RNN~x1~y1RNN~x2~y2RNN~y3~x3

N.B. Every RNN block has the same parameters!

~yT...

...

~yT −1

RNN~xTSimpleRNN

I Initial versions introduced by Jordan (1986), Elman (1990).

I Simply apply a fully-connected layer on both ~xt and ~yt−1 , andadd the results together before applying the activation.

SimpleRNNDense(m)~xt+σDense(m)
~yt−1



~yt = σ W~xt + U~yt−1 + ~bN.B. W of size n × m, U of size m × m! ~b = ~bx + ~bu .

~ytThe “combine” block

This operation (linearly extract m features each out of twovectors–add them–apply activation) will be a very important building
block for LSTMs—let’s call it the “combine” block.

~xt

Combine(m)

~yt−1

σ~yt

Let’s look into what we should choose for our SimpleRNN’s σ. . .

SimpleRNN activations

I Identity: not useful (want to model nonlinear problems). . .

I ReLU: should be a natural first choice.

BUT: exploding gradients!

I Sigmoid: tanh preferred to the logistic function (for symmetry).

BUT: vanishing gradients!

A brief intro to gradient descent

I Weights of a neural network are updated in the direction of thenegative gradient of the loss with respect to them:~t ← w~ t−1 − η∇L(w~)w~)~)∂L(ww~ ) = ∂L(where ∇L(w.
.
.
∂w1
∂wn
I Gradients are computed in an iterative fashion, starting fromoutput neurons backwards to the inputs, using the chain rule.

A simple example

I Consider a very simple “path” neural network, where each layer

has only one neuron, each computing the same activation σ:x

w1σ

w2σ

w3σ

w4σ

w5...

I Although this is a “toy” example, the conclusions will naturally

carry over to wider networks (can be decomposed into manysuch paths, over which we “accumulate” gradient updates).
I For a recurrent neural network, these paths can grow at least

as long as the number of steps in the input!

Gradient updates
I Let ai be the i-th neuron’s activation, and zi be its output:

z0 = x,ai = wi zi−1 ,zi = σ(ai )I Now, consider the partial derivative of the loss function withrespect to w2 , by repeatedly applying the chain rule:~ ) ∂a2~ ) ∂a3 ∂z2~)∂L(w∂L(w∂L(w== σ(a1 )∂w2∂a2 ∂w2∂a3 ∂z2 ∂a2~∂L(w)∂a4 ∂z3= σ(a1 )σ 0 (a2 )w3∂a4 ∂z3 ∂a3~ ) ∂a5 ∂z4∂L(w= σ(a1 )σ 0 (a2 )w3 σ 0 (a3 )w4∂a5 ∂z4 ∂a4= . . . (you see where this is going. . . )Vanishing gradients

I In general, the gradient with respect to w2 will be:σ(a1 ) × σ 0 (a2 ) × w3 × σ 0 (a3 ) × w4 × σ 0 (a4 ) × . . .
I For RNNs applied to very long sequences, this productincludes a lot of σ 0 s when considering the first RNN block!
I When σ is a sigmoid activation:

Logistic: σ 0 (x) = σ(x)(1 − σ(x)) =⇒ |σ 0 (x)| ≤ 0.25.
I tanh: σ 0 (x) = 1 − σ(x)2 =⇒ |σ 0 (x)| ≤ 1.

I

I When you multiply many values with magnitudes less thanone. . . the gradient vanishes!

Exploding gradients

I In general, the gradient with respect to w2 will be:σ(a1 ) × σ 0 (a2 ) × w3 × σ 0 (a3 ) × w4 × σ 0 (a4 ) × . . .
I ReLUs solve the vanishing gradient problem by having aderivative of 1 when a neuron is “alive”, and 0 otherwise.
I However, the value of σ(a1 ) may grow without bound!
I This is not overly troublesome for feedforward networks, butrecurrent networks share weights, so this update is applied
once for each starting position.
I When you add up many updates that may grow withoutbound. . . the gradient easily explodes!

Handling exploding gradients

I To make exploding gradients manageable we apply gradientclipping: making the gradient’s norm no larger than a threshold:∇L = ∇L∇maxmax(||∇L||, ∇max )I While this is convenient, it will stop important long-term

dependencies from fully coming through—as was the case withvanishing gradients.
I Handling vanishing gradients leads us to the central theme ofthis lecture. . .

Long Short-Term Memory

I Proposed by Hochreiter and Schmidhuber (1997).
I Introduce a memory cell, ~c , which maintains features between

time steps, and explicitly control, based on the ~xt and ~yt−1 :What proportion of newly computed features enters the cell;
What proportion of the previously stored cell state is retained;
I What proportion of the new cell contents exits the LSTM.
III This architecture solves the vanishing gradient problem:
IWe dynamically learn the rate at which we want the gradient tovanish, with respect to the current inputs.

An LSTM block
LSTM
M
~ct−1
forget gate~xt

input gate

~yt−1

new fts.
output gate××~ct+σ×~yt

The “new features” block

I Compute the new features based on ~xt and ~yt−1 —essentially aSimpleRNN/“combine” block!
I Since the LSTMs are designed to handle vanishing gradients,

best to use the more stable sigmoid (tanh) as the activation.
~xtCombine(m)~ft t~yt−1
~ft t = tanh Wft ~xt + Uft ~yt−1 + ~bft
I Similarly, we should use tanh as the output activation (σ).

The input/output/forget gates

I Compute the required proportions based on ~xt and ~yt−1 —yetanother “combine” block!
I Want a value in the [0, 1] range per feature (0/1–block/passcompletely), so the logistic sigmoid is appropriate here.
~xt

Combine(m)~it~yt−1
~it = logistic Wi ~xt + Ui ~yt−1 + ~biI (Similarly for ~ot and ~ft . . . )Putting it all together~it = logistic Wi ~xt + Ui ~yt−1 + ~bi~ft = logistic Wf ~xt + Uf ~yt−1 + ~bfgates~ot = logistic Wo ~xt + Uo ~yt−1 + ~bo
~ft t = tanh Wft ~xt + Uft ~yt−1 + ~bft
new features~ct = ~ft t ⊗ ~it + ~ct−1 ⊗ ~ft update cell
~yt = tanh ~ct ⊗ ~ot output

Creating deep LSTMs~yTUse the intermediate outputs to form a new sequence,~ht , that gets fed into a second LSTM.
~y0

LSTM2~y1~h1~h0

LSTM1~x1

LSTM2~y2

LSTM1~x2~y3...

~yT −1~h2

LSTM1~x3

LSTM2~hT~h3~h2~h1

LSTM2~h3...

...

~hT −1

LSTM1~xT

Tips ‘n’ Tricks: Initialisation and optimisers

I It is important to choose the initial parameter values to help theLSTM learn effectively in the early stages!
I Sensible initialisations are (Keras does these automatically):U∗ as orthonormal matrices (help combat vanishing gradientseven further; eigenvalues ∼ 1).
I ~bf = ~1 (to encourage long-term dependencies early on);I Xavier initialisation (Glorot and Bengio (2010)) for all otherweights (recommended for sigmoid activations).

II Gradient descent algorithms that automatically tune thelearning rate are now common—for RNNs, Adam (Kingma andBa (2014)) and RMSProp (Tieleman and Hinton (2012)) work
particularly well.

Tips ‘n’ Tricks: Dropout
I Randomly “kill” neurons during training only.

(Srivastava et al. (2014))

dropout×××××××

I Forces network to not rely on existence of some neuron. . .

Tips ‘n’ Tricks: Dropout in LSTMs

Typically, it is only safe to dropout non-recurrent connections.
(Zaremba et al. (2014))~y0

LSTM2×~h~y11~h0

LSTM1×~x1~h1

LSTM2×~h~x2

LSTM2×~h~y3...

~yT −132

LSTM1×~y2~h2

LSTM1×~x3~yT×

LSTM2×~hT~h3...

...

~hT −1

LSTM1×~xT

Tips ‘n’ Tricks: Bidirectional LSTM

I Very often, the dependencies within sequential data are notjust in one direction, but may be observed in both!
I Examples: DNA strands, words in a sentence, . . .
I Bidirectional layers exploit this by combining the features

obtained going in both directions simultaneously.
I Very simple to deploy in Keras (Bidirectional wrapper

around recurrent layers).

Tips ‘n’ Tricks: Bidirectional LSTM in action~h3~h2...

LSTM←

...

LSTM→

...

~x2~h←
3~h→
2

LSTM←

LSTM→

~x3~h4~h←4~h→3LSTM←

LSTM→~x4~h5~h←5~h→4LSTM←

...

LSTM→

...

~x5...

Demo: The Josef K. LSTM

I Character-level language model: based on the previous ncharacters observed in a text, predict the next one.
I Two-layer LSTM (with 512 features in each layer), trained on

Franz Kafka’s The Trial (Der Prozess):
inp = Input(shape=(seq len, vocab len))h 1 = LSTM(512, return sequences=True)(inp)
h 2 = LSTM(512, dropout W=0.5)(h 1)out = Dense(vocab len, activation=’softmax’)(h 2)
I Once trained, we can repeatedly sample the obtained

probability distribution to generate our own text, one characterat a time!

Demo: Samples from the Josef K. LSTM. . .

"I’m not sure that’s worrible?" asked K. "Yes," said thebusinessman, "and that’s what they could do," said K.,
and was able to make things some place and he would need
to be able to deal with the court which were the old man
with a sigh for in the hallway which was already for him
to de and darkness without being didned the bed. "I’msure this court would have been meant to be a little
while to him in the court. They were all asside thepainter and looked at her with some way on his way andleave. "It’s nothing about your face when I had been
given the first attention foreary, there was a lithle
like that are not the time he remained standing with hishead so far from the bank.

Taking the idea further

I The idea of repeatedly sampling the LSTM output and feedingit back to the input is a common one. For example, we mightlearn a classifier to predict a translation of a given sentence,
word by word.

I Or, if we feed in features extracted from an image (e.g. byusing a convolutional neural network). . .

Neural Talk
I Vinyals et al. (2014), Karpathy and Fei-Fei (2015).

Future prospects. . .

I Although it seems like recurrent networks are the de facto

standard for processing sequential data, this may well change
in the future.
I Feedforward models are significantly easier to train, control

and parallelise (fewer interdependencies).
I Recent work using à trous (dilated) convolutions on machinetranslation (Kalchbrenner et al. (2016)) and WaveNets (vanden Oord et al. (2016)) outperforms or matches recurrent
models in domains where they historically dominated.

O1O2O3O4O5O6O7O8O9O10O11O12O13O14O15O16I1I2I3I4I5I6I7I8I9I10I11I12I13I14I15I16Regular 1 × 3 convolution in 1D (Convolution1D)O1O2O3O4O5O6O7O8O9O10O11O12O13O14O15O16

I1I2I3I4I5I6I7I8I9I10I11I12
I13
I14
I15
I16À trous convolutions in 1D (AtrousConvolution1D)O1O2O3O4O5O6O7O8O9O10O11O12O13O14O15O16I1I2I3I4I5I6I7I8I9I10
I11
I12
I13
I14
I15
I16

Temporal dependencies captured by O16

Thank you!

Questions?
petar.velickovic@cl.cam.ac.uk
https://github.com/PetarV-/a-trip-down-lstm-lane