Be nice to your neurons
Initialisation, Normalisation, Regularisation and Optimisation
Petar Veličković
Artificial Intelligence Group
Department of Computer Science and Technology, University of Cambridge, UK
COMPGI23—Introduction to Deep Learning, University College London

14 November 2017

Introduction

I In this lecture, I will cover several essential techniques that canhelp simplify neural network training.
I These will primarily be techniques for making the network’s

actions more robust and generalisable, across every stage of
its layer pipeline. We will cover simple, commonly-usedmethods as well as some ongoing bleeding-edge research.
I The title holds a double meaning: if you treat your networks

well, you’ll require less effort to properly optimise them! :)A (rough) outline for today

1. Regularisation: Forcing our networks to find good parameters
while simultaneously being constrained in some other way(hopefully, one that discourages parameter choices that overfit!)
2. Initialisation: Choosing initial parameters (“starting points” fortraining) for the network in order to encourage good behaviourin early stages of training.
3. Normalisation: Making sure that our network consistently
receives well-behaved signals (in terms of magnitudes andstatistics) at all stages of its processing pipeline.
4. Optimisation: Actually deciding how to use the computed
gradients to update the network parameters in a stable andadaptive way.

Combatting overfitting
Learning the sine function
4.5
4
3.5
3
2.5
2
1.5
1

y

0.5
0
−0.5
−1
−1.5
−2
−2.5

Training set
TargetLinear fit
Degree-3
Degree-14

−3
−3.5
−4
−4.5
−2 −1.8 −1.6 −1.4 −1.2 −1 −0.8 −0.6 −0.4 −0.2

0

x

0.2

0.4

0.6

0.8

1

1.2

1.4

1.6

1.8

2

Last time I just said “beware”. . . now let’s see what we can do.

Regularisation

I Overfitting can often be alleviated using regularisation.

I Broadly speaking, regularisation corresponds to making themodel optimise the same problem, under additional constraints
that somehow restrict the set of parameters available to themodel during training.
I With proper regularisation, many deep learning models canlearn to fit the provided training data in a controllable fashion,
regardless of their nominal parameter count.

Early stopping

I Hyperparameters in a neural network are commonly optimisedby using the obtained loss on a validation set as a proxy.
I For arguably the least intrusive form of regularisation, we can(re)use the validation set to learn the number of training
iterations as a hyperparameter.
I This leads us to the technique of early stopping, which is now

near-ubiquitously used across deep learning applications.

Early stopping: the essentials

I While training, keep track of the model parameters that

achieved the best validation loss so far.
I Stop training when the loss hasn’t improved for a set number of

iterations (the patience hyperparameter).
I Restore the best-performing model (and potentially evaluate it

on a testing dataset).

Early stopping in action
Training loss
Validation loss

1

Loss

0.8
0.6
0.4
0.2
0

0

200

400
600
Iteration

800

1,000

Early stopping in action
Training loss
Validation loss
Stopping point

1

Loss

0.8
0.6
0.4
0.2
0

0

200

400
600
Iteration

800

1,000

Early stopping: analysis

I We are basically assuming that validation performance will bea good surrogate for testing performance.
I

Very simple, and usually very effective (esp. if large valid. set)!

I Often desirable to use, as it does not require modifying

anything else about the training or model setup itself!
I Downsides include the cost of storing a copy of the parameters(often negligible) and separating out a validation set (but canretrain on it afterwards).

Early stopping: regularisation

I How exactly does early stopping regularise the model?
I Assume, for simplicity:

Initial model parameters w;
A fixed learning rate of η;I That the learning is performed over τ iterations;
I Early stopping at the τ 0 th iteration;
I That the maximal weight gradient is bounded by |gmax |.
II

I This means that, initially, each parameter is allowed to vary in

the range wi ± ητ |gmax |.
I However, an early stopped model’s parameters can only vary in
the range wi ± ητ 0 |gmax |—this is a regularising constraint,

limiting the set of parameters reachable from w!

Early stopping in Keras

Simple to do using the EarlyStopping callback!

model.fit(X train, Y train, batch size=batch size,
epochs=num epochs, verbose=1, validation split=0.1,
callbacks=[EarlyStopping(monitor=’val loss’, patience=5)])

L2 regularisation

I Now, we turn our attention to a regulariser that directly restricts

the desirable values of weights.
I We do this by further constraining our loss function, L (e.g.

cross-entropy or squared error), by adding a penalty term,
which penalises weights that deviate too much from zero:~ ) = L(~x , ~y , w~)+
L̃(~x , ~y , w

λ
~ k2
kw
2

where λ is a hyperparameter.
I We have arrived at the L2 regularisation technique (also called~.
weight decay), as we seek to restrict the L2 norm of w

Implication on training dynamics
I Recall the gradient descent update rule:

~ ←w
~ −η
w

∂L
~
∂w

I The gradient of the added L2 penalty is λw~ . Therefore:



∂L
~
+ λw
~
∂w
∂L
~ ← (1 − ηλ)w
~ −η
w
~
∂w



~ ←w
~ −η
w

I This forces the weight vector to shrink in magnitude at each

step, prior to applying the gradient update!

L2 regularisation: notes

I Nowadays, L2 regularisation has been somewhat abandoned infavour of more recent ideas—but is still often very useful.
I Properly choosing λ is important!
I
I

Too low: negligible effects;
Too high: weights set to ~0.

I N.B. The regularisation effect would still be present if wetended to a point other than the origin, i.e. if we optimised~ ) = L(~x , ~y , w~)+
L̃(~x , ~y , w

λ~ −w~ 0 k2kw
2

but this is rarely done in practice (why?).

L2 regularisation in Keras

Simply attach a kernel regularizer to the model!
(N.B. we often do not regularise the bias, as it is less overfit-prone)y = Dense(64, activation=’relu’,kernel regularizer=regularizers.l2(0.0001))(x)Aside: L1 regularisation
I While the L2 norm might have seemed like a natural choice tooptimise, nothing stops us from using a different (Lp ) norm.
I Especially, L1 regularisation is another popular approach. In

this scenario we penalise the L1 norm:~ ) = L(~x , ~y , w~)+λL̃(~x , ~y , w

X
i

|wi |

The gradient update is no longer as simple to analyse—but it
can be shown (details omitted) that this will force several
weights to be (near-)zero, sparsifying the model.
I This makes L1 regularisation useful for, among other things,

feature selection (cf. LASSO).

Ensemble methods

I While a final trained model may not perfectly generalise, we

may find that the exact nature of the faults it makes will dependon the (stochastic) progress of the training procedure.
I This means that we could be able to extract better performance

if we train several models and average their predictions!
I This technique is known as ensembling.

An overview of the potential of ensembling

I Assume, for simplicity, that we are optimising a squared error
loss, and that model i makes error i (with variance E(2i ) = v

and covariance E(i j ) = c).
I The error made by an average of k such models is 1kPk

i=1 i .
The expected squared error is then:




!2 
k
k
X
X
X
1
1
1
k −1
2i +
E
i  = 2 E i j  = v +
c
k
k
k
k
i=1

i=1

j6=i

I If errors are not fully correlated (c < v ), this causes animproved generalisation performance!

Bagging

I If we want to decorrelate our errors while ensembling a modelwith the same hyperparameters (desirable!), a good techniqueto use is bootstrap aggregating (bagging).
I Here we expose each copy of the model to a different dataset,by sampling k examples from the training set with replacement.
I This means that, if the dataset size is unchanged (k = n),approximately two-thirds of the examples will be present in theresampled dataset (with the remainder replaced by duplicates).

Bagging in action (Goodfellow et al., 2016)

Notes on using ensembles

I Ensembling is a remarkably powerful technique—often

required for winning machine learning competitions.
I However, it should usually be avoided when benchmarking

machine learning algorithms (for scientific publications),
especially when comparing against single-model approaches.
I Lastly, note that not all ensembling methods are

regularisers—e.g. a technique called boosting will createmodels with increased capacity.

Reliance on neurons

I When trained without additional constraints, neural networks

will tend to encourage highly specialised neurons.
I Even worse, there might be neurons that perform poorly, withseveral other neurons there just to correct for its mistakes.
I Both of these make the network overly reliant on actions ofspecific neurons, which generalises poorly (if only one neuron
fails, it might compromise many others!).
I=⇒ could be useful to force the network not to rely on theexistence of a neuron. . .

Dropout (Srivastava et al., 2014)

dropout

×
×

×
×
×

×
×

I Randomly “kill” each neuron in a layer with probability p, during

training only. Also scale the output of remaining neurons by
1
1−p , to preserve expected value.

Dropout in action (Srivastava et al., 2014)

An alternate look at dropout

I When using dropout, at each iteration a different model is used

(depending on which hidden units are dropped), by samplingmask elements from a Bernoulli(p) distribution.
I This means that dropout can also be seen as a computationallycheap way of creating a large ensemble of neural networks(with 2n possibilities for n neurons).
I Although there is no formal proof for the fact that running thefull model at evaluation time preserves the required ensemble
properties, this technique performs really good in practice!

The ensembled networks (Goodfellow et al., 2016)

Notes on dropout

I Dropout is a very desirable method, given its high effectivenessas a regulariser and favourable computational cost.
I Typically, good values of p lie in 0 ≤ p ≤ 0.5, but it issometimes possible to require even higher values.

I Rough rule-of-thumb:

Fully-connected layers are the most prone to overfitting, andapplying dropout with p = 0.5 to their output is often appropriate.
I Overfitting is not as common with convolutional layers, but aweaker dropout (p ≈ 0.25) applied there could still be helpful.
I Occasionally, it becomes useful to dropout inputs to the network.
I

I As dropout is turned off for validation/testing, note that thetraining loss may not initially be better than validation loss!

Dropout in Keras

Extremely simple—just apply the Dropout layer!

x drop = Dropout(0.5)(x)Aside: DropConnect (Wan et al., 2013)
I Generalises dropout by dropping weights rather than outputs.

I Harder to do both training (as it requires a different mask foreach example) and inference (requiring Gaussian moment
matching). However, some impressive results (MNIST SOTA)!

Concrete dropout (Gal et al., 2017)

I It is possible to relax the z ∼ Bernoulli(p) distribution to acontinuous version (the Concrete distribution):1z̃ = σ· (log p − log(1 − p) + log u − log(1 − u))twhere p is the dropout probability, u ∼ U(0, 1), t is atemperature parameter, and σ is the logistic sigmoid function.
I This is now differentiable by p—can learn the hyperparameter

directly by gradient descent!
I The technique is known as Concrete dropout.

Concrete dropout (Gal et al., 2017)

I The authors provide a ConcreteDropout wrapper for Keras,

which allows one to trivially apply it to any layer’s output:
y = ConcreteDropout(Dense(32, activation=’relu’))(x)I The reference implementation may be found at:https://github.com/yaringal/ConcreteDropout

Concrete dropout (Gal et al., 2017)

I Directly learning p from training data (requiring no validation

dataset) makes this technique very attractive for small-data
problems as well as reinforcement learning!
I The learnt values of the concrete dropout’s p do not always

coincide with manual tuning results! (e.g. for an
encoder-decoder network for image segmentation:)

Data augmentation

I As mentioned in my Week 2 lecture, the best way to improve a

network’s generalisation performance is to gather more training
data—but this is not always feasible!
I A simple manner of gathering more data is randomlytransforming the training inputs in a way that makes predictablechanges to the ground-truth outputs.
I

Most commonly, for classification problems we can choosetransformations that do not change the class.

I We arrive at the common technique of data augmentation.

Data augmentation in computer vision

I This technique is best visualised (and most commonly applied)

on an image classification dataset:
Applying random shifts/scales/rotations/crops/. . . to the input
image will usually not change the object class of the image!
I Careful with some transformations (such as horizontal flips).
I

I Now the network will never see the same example twice during

training—significantly lowering the potential for overfitting.
I Furthermore, it will be encouraged to learn to be resistant tosuch perturbations in the input!

Data augmentation in action

(Image) Data augmentation in Keras

Make advantage of the ImageDataGenerator class!
datagen = ImageDataGenerator(width shift range=0.1,
height shift range=0.1, rotation range=10)
model.fit(...)
model.fit generator(datagen.flow(
X train, y train, batch size=32),
steps per epoch=len(X train) / 32, epochs=100)

Being nice to our signals

I We will now focus on techniques that are primarily designed tomake our data representations more convenient to work withas the data passes through the network.

I As such, they’re not designed with regularisation in mind—but

many of them end up being useful regularisers nonetheless!

Input normalisation
I To avoid numerical issues with signals of overly large

magnitude or variance, it is almost-always helpful to normalise
the input of the network to have more controlled statistics.
I Typically, this can be done by computing the sample means, µ~,and sample standard deviations, ~σ , across each feature within
the training set, and transforming the input data, x, byx̃i =xi − µiσi

I Note that we cannot observe the test data, so the samevalues of µ
~ and ~σ are used for evaluating!
I Many of the techniques to follow will assume that we’ve

normalised our inputs!

Network initialisation

I We now revisit a problem explored in Week 2—appropriately

initialising network weights.
I We’ve shown how unsupervised pre-training can be a goodinitialiser (albeit costly in terms of the added training time).
I Since then, such methods have largely been replaced withdrawing initial weights from carefully crafted probability
distributions.
I

These often perform equivalently or better to unsupervised
pre-training, for only a fraction of the effort.

Initialisation in Keras

I We will usually draw each weight from the uniform or normal

distribution centered at zero, i.e. U(−x, x) or N (0, σ).
I
I

Thus far, no significant difference in the two has been found.
We only need to specify the standard deviation, σ!

I Usually initialise biases to zero.
I Keras will allow us to specify the initialiser using thekernel initializer parameter:y = Dense(64, activation=’relu’,kernel initializer=’ones’)(x)LeCun initialisation (LeCun et al., 1998)
I An early development towards such initialisers is the LeCun

initialisation. Consider what happens to the variance of a linear
neuron’s output, assuming its weights wi and inputs xi areuncorrelated and zero-mean:
!
nin
nin
X
X
Var
wi xi =
Var (wi xi )
i=1

=

i=1
nin
X

Var (W )Var (X ) = nin Var (W )Var (X )i=1where nin is the fan-in—the number of inputs to the neuron.
I This approximation is appropriate for sigmoid activations aswell (as they’re roughly linear in their unsaturated region).

LeCun initialisation (LeCun et al., 1998)

I Typically, we will want to choose our weights to preserve thevariance of the input signal as it passes through the network.
I This implies that we should set

Var (W ) =1nin

I Use ’lecun uniform’ or ’lecun normal’ in Keras.
I To make it work nicely for deeper networks, we will need

another constraint. . .

Xavier initialisation (Glorot and Bengio, 2010)

I Signals do not just go forward in a network—they are also

propagated backwards when computing gradients!
I During this procedure, a neuron will accumulate gradients fromthe neurons it directly sends output to, scaled by the relevant
weights. Let nout be the fan-out—the number of such neurons.
I A similar argument as before now mandates we should set

Var (W ) =1
nout

in order to preserve the variance of the gradient update signals.

Xavier initialisation (Glorot and Bengio, 2010)

I Typically, we cannot satisfy both constraints simultaneously.

Xavier initialisation instead aims for their average:
Var (W ) =2
nin + nout

I Now a ubiquitously used initialisation scheme.
I Use ’glorot uniform’ (default!) or ’glorot normal’ inKeras.

He initialisation (He et al., 2015)

I We’ve thus far assumed a linear layer. This was somewhat

okay for sigmoid activations, but not for ReLU (which is mostcommonly used)!
I He initialisation rectifies the analysis to take into account the

behaviour of ReLUs (details omitted), obtaining:
Var (W ) =2nin

I Enabled surpassing human performance on ImageNet!
I Use ’he uniform’ or ’he normal’ in Keras.

Orthogonal initialisation (Saxe et al., 2014)

I An alternate approach to initialising is to not initialise weights

independently from one another—this may be helpful inobtaining some desirable properties.
I The best-known example of this is orthogonal initialisation,

which constrains the initial weight matrix W to be orthogonal,
i.e. WT W = I.
I This has two desirable properties:

The transformation is norm-preserving, i.e. kW~x k = k~x k—thishelps combat vanishing/exploding gradients early on.
I Its columns are orthonormal, i.e. w~ iT w~ j = δij —this encourages
the network to learn varied input features.

I

I Use ’orthogonal’ in Keras.

LSUV (Mishkin and Matas, 2015)

I Perhaps the best approach to initialisation is one that isdata-driven—tuned to the features of the input data. One
interesting development in this direction is Layer-sequential
unit variance (LSUV) initialisation.

I Initialise layer-by-layer, starting with an orthogonally-initialised

weight matrix, then tuning on minibatches of training data untilthe variance of the layer’s output signals becomes close to 1.

LSUV (Mishkin and Matas, 2015)

Internal covariate shift

I Thus far, we’ve focussed our attention at making the input

signal well-behaved, and, initially, making it propagate throughthe network nicely in both directions.
I However, as training progresses, especially for deeper

networks, it is to be expected that weights are pushed in a
direction that will change the intermediate layer statistics—aneffect known as the internal covariate shift.
I This internal covariate shift is an important problem in deep

networks, since any neuron deeper in the network needs not
only to do its job properly, but also to adapt to inputs that arechanging over time.

Internal covariate shift, visualised

The solution is quite simple—renormalise periodically!

Batch normalisation (Ioffe and Szegedy, 2015)
I The first—and still by far most widely used—approach to

reducing internal covariate shift is renormalising the signal
across each training minibatch (batch normalisation).
I Let the outputs of the current layer across the current batch be
B = {x1 , . . . , xm }. Then:
m

1 X
xi
µB =
m
i=1

x − µBx̂i = qiσB2 + ε1
σB2 = mmP

(xi − µB )2

i=1

yi = γ x̂i + β

where γ and β are trainable (allowing the network to revert)!
I Now ubiquitously used for training deep architectures.

Notes on batch normalisation
I At test time, use the entire training set’s statistics to fix

parameters of µ and σ (usually as a moving average):µ ← µ + α(µB − µ)σ ← σ + α(σB − σ)

I Regularises the network! Similarly to data augmentation, thesignals the network will see will rarely repeat (as signals for
one example now depend on its entire minibatch)!
I Sometimes applied to the inputs of the network directly as a
substitute for preprocessing!
I Simple to include in a Keras model, by leveraging the
BatchNormalization layer:y = BatchNormalization()(x)

Limitation of batch normalisation

I One issue of batch normalisation is its dependence on

minibatch statistics.
I This makes the technique harder to apply in, e.g.:

online learning, where batch size is forced to be 1;
working with variable-length inputs and RNNs;
I noise-sensitive applications such as reinforcement learning.

I
I

I We will now survey a few recently proposed techniques forovercoming this issue.
I There does not seem to be a clear consensus on which

technique is best to use in these circumstances—attempting
several of them is the best approach for a new problem.

Layer normalisation (Ba et al., 2016)

I Under the intuition that changes in output of one layer will tend

to cause highly correlated changes in the summed inputs forthe next layer, layer normalisation proposes normalising all
signals across the layer, for a single training example:
H

1X
ai
µl =
H
i=1

1
σl2 =

H

H
X
(ai − µl )2
i=1

where H is the number of neurons in the layer, and ai is theoutput of the i-th neuron.
I This is followed by a learnable scale and shift, as in batchnormalisation. N.B. each example is normalised independently!

Weight normalisation (Salimans and Kingma, 2016)

I Weight normalisation proposes to speed up the optimisation

procedure of neural networks by making the norm of theirweights explicitly trainable.
I This is achieved through the following reparametrisation:

~ =
w

g
~v
k~v k

where g and ~v are trainable by gradient descent.
I A data-driven methodology is also suggested to initialiseg = r 1  , after fixing an initial ~v , across a single training
Var k~v~v·~xk

minibatch of layer inputs ~x .

Batch renormalisation (Ioffe, 2017)

I Proposes an extension to break the discrepancy of training andinference transformations of batchnorm.
I As before, compute µ and σ as moving averages of µB and σB ,and use those for inference.
I During training, introduce a correcting scale and shift, which

compensates for the discrepancies of µ and σ to µB and σB :x − µBxi − µ
= i
·r +d
σ
σB
This implies that r = σσB , d = µBσ−µ . In practice, the values of r
and d are clipped (initially fixing them to 1 and 0, respectively,
and progressively widening the range of allowed values).

Residual connections (He et al., 2015)

I Training very deep networks has been very problematic formany years. Batch normalisation is only one of the manytechniques that have helped significantly simplify the process inrecent years.
I Another remarkable technique are residual (skip) connections,which enabled a 152-layer network to win the ImageNet
competition in 2015.
I Key observation: stacking too many layers limits not only

validation, but also training performance!
I But, if the added layers are useless, the network could just be

encouraged to make them compute the identity function!

Residual connections (He et al., 2015)
I Essentially, allows the input to additively shortcut to a latterstage of the network’s pipeline.

I Lets the network effectively choose its own depth. . . a very

powerful and versatile concept!

Residual networks (He et al., 2015)

https://github.com/fchollet/keras/blob/master/keras/
applications/resnet50.py

Advanced ReLU activations: PReLU

I To conclude this section, we will cover two kinds of activation

functions that are more general versions of the ReLU function,and have seen considerable usage in recent years.
I The parametric ReLU (PReLU), proposed by He et al., allows

for some (learnable) amount of output signal for negative input
values:(xi
xi > 0PReLU(xi ) =ai xi xi ≤ 0Here, ai are learnable parameters—one per each feature.

Advanced ReLU activations: ELU

I Another substantial development in the field of activation

functions is the exponential linear unit (ELU), proposed byClevert et al.
I The ELU extends the ReLU into the negative values:

(
x
x >0
ELU(x) =α(exp(x) − 1) x ≤ 0Here, α is a hyperparameter; often set to 1.
I Critically, unlike the PReLU, here the output of the function

saturates for too low values—keeping the negative signals
under control.

Advanced ReLU activations (Clevert et al., 2015)

Gradient descent optimisers

I We turn our attention to the learning algorithm employed totrain our neural networks.
I This area has seen significant improvement over the past

years, and has now completely simplified the training of many
common neural network models (often in conjunction with theother regularising techniques already enumerated here).
I I will, for the most part, follow the brilliant overview by

Sebastian Ruder, which I highly recommend:
http://ruder.io/optimizing-gradient-descent/Vanilla minibatch SGD

I Recall the standard minibatch stochastic gradient descentframework:~)∂LB (w~∂w
where LB is the loss on a sampled minibatch B of training
examples, and η the learning rate.
~ ←w
~ −η
w

I N.B. SGD and backpropagation are different concepts!

Backpropagation is used to efficiently compute the gradients~)∂LB (w~ , whereas SGD is used to update the weights, assuming∂w
the gradients are already computed.

Limitation of vanilla SGD

I Choosing a proper value of η can be notoriously difficult—withpoorly chosen values either significantly slowing down
convergence or hindering it.
I One way to address this issue is learning rate annealing, whereη starts with a larger value and is gradually reduced—but thisschedule needs to be set-up upfront, and therefore is unable
to adapt to the characteristics of the loss function’s surface.
I Furthermore, the same learning rate is used for updating all theweights, which may be inappropriate (different weights may
require different scales)!

Recall: getting trapped in saddle points

Momentum (Qian, 1999)

I One very simple approach to improving the performance of

SGD is momentum—take a step in an accumulated direction:~vt = γ ~vt−1 + η~)∂LB (w~∂w~ ←w~ − ~vtw

where γ is a hyperparameter, often set to a value near 0.9.
I This has the effect of amplifying gradient updates in the

directions that are consistent across timesteps, and reducing
them in oscillating directions—speeding up convergence andimproving stability.

Effects of momentum

Without momentum

With momentum

Vanilla SGD + momentum is still widely used in research, as itprovides a lot of control (at the expense of having to put in more
effort in optimising the learning rate schedule).

Nesterov accelerated gradient (Nesterov, 1983)

I Nesterov’s NAG method allows us to further improve the“look-ahead” capability of momentum.
I Since we’re going to apply the γ ~vt−1 update no matter what, wemight as well use this position to evaluate the gradient on:~vt = γ ~vt−1 + η~ − γ ~vt−1 )∂LB (w~∂w~ ←w~ − ~vt
w

I This prevents us from making too eager jumps, and increases

responsiveness.

Effects of NAG

Standard momentum: blue vector—compute the local gradient,then take a big step in the direction of accumulated gradient.
Nesterov momentum: green vector—first take the step in the
direction of the accumulated gradient (brown vector), then correct
based on the local gradient there (red vector).

Momentum methods in Keras

I All the methods described so far can be deployed to a Keras

model by modifying the model’s compilation:model.compile(loss=’categorical crossentropy’,
optimizer=SGD(lr=0.01, momentum=0.9, nesterov=True))

Adagrad (Duchi et al., 2011)

I First widely used method that adapts the learning rate for eachparameter individually.
I Let ~
gt be a vector that holds the gradient at time t, i.e.
~)w
gt,i = ∂L(
wi . Then AdaGrad modifies SGD by adapting thelearning rate for wi using historical gradient values:η
wt+1,i ← wt,i − p· gt,iGt,ii + ε
where Gt is a diagonal matrix where Gt,ii is the sum of squaresof gradients wrt wi until timestep t.

Adagrad: notes

I To use in Keras, set optimizer=’adagrad’.
I Intuition: weights that receive strong gradients less frequently

should receive a stronger learning rate to compensate.
I Benefit: no longer need to explicitly worry about the value of η;

often, setting it to a value like 0.01 is fine.
I Main weakness: values of G can only grow—learning rate

eventually vanishes!

Adadelta (Zeiler, 2012)

I Designed to address the aggressive learning rate decay of

Adagrad, Adadelta smooths the sum of squares of gradientswith exponential averaging:E(~g 2 )t = γE(~g 2 )t−1 + (1 − γ)~gt2
where γ is once again chosen to be around 0.9.
I Replacing Gt with E(g 2 )t , we arrive at the following parameter

update (denoting the root mean square metric by RMS):ηη~gt = −~g~t = −p∆w
2~g )t tRMS(~
E(g )t + εAdadelta (Zeiler, 2012), cont’d
I The authors note that the parameters in the update rule do nothave the same units, so another exponential average (this timeover the parameter updates) is defined:~ 2 )t = γE(∆w~ 2 )t−1 + (1 − γ)∆w
~ t2
E(∆w
I Replacing the learning rate with the RMS of these updates up

to time t − 1 (as the update at time t is not immediately known)
yields the Adadelta update rule:~t = −∆w~ )t−1
RMS(∆w~gt
RMS(~g )t~ t+1 ← w
~ t + ∆w
~tw

I To use in Keras, set optimizer=’adadelta’.

RMSprop (Tieleman and Hinton, 2012)

I A method developed concurrently with Adadelta, with the sameaim of rectifying the issues of Adagrad. Presented for the firsttime on Geoffrey Hinton’s Coursera course on Neural networks!
I In fact, equivalent to the first update rule of Adadelta!

η~gt~ t+1 ← w~t − pw
E(g 2 )t + ε

I To use in Keras, set optimizer=’rmsprop’.

Adam (Kingma and Ba, 2014)
I Adaptive moment estimation (Adam) is currently the mostpopular adaptive optimisation algorithm.
I Computes two exponential averages: one of the gradients aswell as the squared gradients (as Adadelta and RMSprop did):~ t = β1 m
~ t−1 + (1 − β1 )~gtm

~vt = β2~vt−1 + (1 − β2 )~gt2

where β1 and β2 are hyperparameters (with default values of
0.9 and 0.999, respectively).
I These tend to be biased towards zero early on, so the following
bias-corrected values are recorded:
~ =
m̂
t

~t
m
1 − β1t~v̂t =~vt
1 − β2t

Adam (Kingma and Ba, 2014), cont’d~ as the gradient direction (similar to momentum!)
I Now, use m̂tand ~v̂t to scale the learning rate (similar to RMSprop!).
η~~ t+1 ← w~t − qwm̂
t~v̂t + ε
We have arrived at the Adam update rule!
I To use in Keras, set optimizer=’adam’.

AdaMax (Kingma and Ba, 2014)
I Adam has used the L2 norm of the gradient updates tocompute ~vt —but using the Lp norm is also possible:~vt = β2p ~vt−1 + (1 − β2p )|~gt |p~ut = (~vt )1/p

I While values of p > 2 are generally unstable, stable behaviour

re-emerges at L∞ , yielding the simple AdaMax algorithm:~u0 = ~0~ut = max(β2 · ~ut−1 , |gt |)~ t+1 ← w
~t −
w

η ~
m̂
~ut t

I The max operator does not bias towards zero, and therefore

there is no need to compute a bias-corrected ~ut .
I To use in Keras, set optimizer=’adamax’.

Nadam (Dozat, 2016)

I As we noted, Adam leverages the ideas from both momentum

and RMSprop—how about adding Nesterov momentum?
I To simplify, we modify the steps of Nesterov~~ t − β1 m̂momentum—rather than using w
t−1 as a position to~ t (as with ordinary
compute gradients, we compute them at w~
momentum), but take an additional step in the direction of β2 m̂tat the end!
I Integrating this into Adam’s update rule. . .

Nadam (Dozat, 2016)
I Expanding Adam’s update rule, we obtain:~ t−1 (1 − β1 )~gt
β1 m
~ t+1 ← w~t − qw+t
1 − β1t~v̂t + ε 1 − β1
η
(1 − β1 )~gt~
~t − q=w
β1 m̂t−1 +1 − β1t~v̂t + εη~~I Now, just replace m̂
t−1 with m̂t (to do the “look-ahead” step as
per the previous slide) to obtain the Nadam update rule!
(1 − β1 )~gtη~
~ t+1 ← w~t − qβ1 m̂t +w
1 − β1t~v̂t + εVisualisation (credit: Alec Radford)

Adaptive methods generally outperform SGD—especially when it
comes to escaping saddle points.

Being nice to RNNs

I Recurrent neural networks are somewhat special:

They have extensive weight sharing through time;
They often have to cope with variable-length sequences;
I They may not have access to large batch sizes of data;
I ...

I

I

I This introduces a specific set of tricks for optimising them. . .

I I will focus on the long short-term memory (LSTM) model here;

but many of these ideas will work on other cell types as well.

Recall: LSTM computational graph
LSTM
M
~ct−1
forget gate~xt

input gate

~yt−1

new fts.
output gate

×

×

~ct

+

σ

×

~yt

Recall: LSTM equations~it = logistic Wi ~xt + Ui ~yt−1 + ~bi~ft = logistic Wf ~xt + Uf ~yt−1 + ~bf
gates


~ot = logistic Wo ~xt + Uo ~yt−1 + ~bo~ft t = tanh Wft ~xt + Uft ~yt−1 + ~bft
new features~ct = ~ft t ⊗ ~it + ~ct−1 ⊗ ~ft update cell
~yt = tanh ~ct ⊗ ~ot output

LSTM initialisation (Jozefowicz et al., 2015)

I It is important to choose the initial parameter values to help the

LSTM learn effectively in the early stages!
I Sensible initialisations are (Keras does these automatically):U∗ with orthogonal initialisation (help combat vanishinggradients even further; eigenvalues ∼ 1).
I ~bf = ~1 (to encourage long-term dependencies early on);I Xavier initialisation (Glorot and Bengio (2010)) for all otherweights (recommended for sigmoid activations).
I

I For RNNs, the Adam (Kingma and Ba (2014)) and RMSProp(Tieleman and Hinton (2012)) optimisation algorithms work
particularly well.

Dropout in LSTMs

Typically, it is safest to dropout non-recurrent connections only.
(Zaremba et al. (2014))
~y0

LSTM2

×~h

~y1

1

~h0

LSTM1

×
~x1

~h1

LSTM2

×~h
~x2

LSTM2

×~h

~y3

...

~yT −1

3

2

LSTM1

×

~y2

~h2

LSTM1

×
~x3

~yT

×

LSTM2

×~h

T

~h3

...

...

~hT −1

LSTM1

×
~xT

Normalisation techniques

I Generally, batch normalisation may be less applicable, if

sequences will have a wide variety of lengths (as many entries
will end up missing, leading to weak approximations of the step
statistics).
I Techniques such as layer normalisation and weightnormalisation seem to rectify this situation well (albeit while
being a bit harder to control).
I Recurrent batch normalisation (Cooijmans et al., 2017)batch-normalises the hidden-to-hidden connections of an RNN,
achieving solid results.

Bidirectional LSTM

I Very often, the dependencies within sequential data are notjust in one direction, but may be observed in both!
I Examples: DNA strands, words in a sentence, . . .
I Bidirectional layers exploit this by combining the features

obtained going in both directions simultaneously.
I Very simple to deploy in Keras (Bidirectional wrapper

around recurrent layers).

Bidirectional LSTM in action
~h3

~h2

...

LSTM←

...

LSTM→

...

~x2

~h←
3

~h→
2

LSTM←

LSTM→

~x3

~h4
~h←
4

~h→
3

LSTM←

LSTM→

~x4

~h5
~h←
5

~h→
4

LSTM←

...

LSTM→

...

~x5

...

Thank you!

Questions?
petar.velickovic@cst.cam.ac.uk
Special thanks:César Laurent (Montréal Institute for Learning Algorithms)
Reading material:‘Deep Learning’, Chapter 7 (Regularisation)‘Deep Learning’, Chapter 8(Initialisation, Optimisation, Normalisation)‘Deep Learning’, Chapter 10 (RNN pointers)