CSE 587: Deep Learning for Natural
Language Processing
Lecture 3. Neural Networks and Backpropagation

Rui Zhang
Spring 2023

Many Slides from CS224n

1

Outline - Key Concepts
NLP
N/AML
Neural Networks
Forward Propagation
Gradient Descent
Backpropagation
Computational Graph
Optimizers

2Neural Networks

3

A NeuronA neuron is the fundamental building block of neural networks.
e.g., a neuron with sigmoid function

4Two layer neural networks

5Activation Functions

Leaky ReLU

6

How to train two-layer Perceptron?
Forward propagation:

Backward propagation:

7

How to train two-layer Perceptron?
Forward propagation: In this phase, the inputs for a training instance are fed into the neuralnetwork. This results in a forward cascade of computations across the layers, using the current
set of weights. The output is the prediction of current weights.

Backward propagation:

8

How to train two-layer Perceptron?
Forward propagation: In this phase, the inputs for a training instance are fed into the neuralnetwork. This results in a forward cascade of computations across the layers, using the current
set of weights. The output is the prediction of current weights.

Backward propagation: The main goal of the backward phase is to learn the gradient of theloss function with respect to the different weights by using the chain rule of differential
calculus. The weights are just adjusted based on error.

9Forward propagation

We use the sigmoid activation
function which introduces
better non-linearity:
10Forward propagation11

Forward propagation

12

Forward propagation

13

Forward propagation - Compact Form

14

Beyond two layers: multilayer neural networks
Why not add more and more layers to learn more complex decision boundaries?

15

Discriminative ModelThe discriminative model is parameterized by

16

Discriminative Model Objective Function
The discriminative model is parameterized by

We often use negative log likelihood over training data as our objective function
or loss function. It is a function of17

Discriminative Model Objective Function
The discriminative model is parameterized by

We often use negative log likelihood over training data as our objective function
or loss function. It is a function ofWe then optimize the parameters to minimize the loss. Better model has lower loss
18

Optimize Objective Function by Gradient DescentCalculate the gradient of the loss function with respect to the parameter

Updateby moving a small step in the gradient direction to decrease the loss

is the learning rate
19

Gradient Descent20

Gradient Descent
The key operation in gradient descent is to calculate the gradient.

21

Gradient of univariate scalar-valued functions

22

Gradient of multivariate scalar-valued functions

23

Gradient of multivariate vector-valued functions: Jacobian Matrix

24

Example Jacobian: Elementwise activation Function

25

Other Jacobians

Left as exercise. Will be used later.
26

Chain Rule
Chain Rule tells us how to calculate gradients of composite functions.

27

Example
Calculate

for the following feed-forward neural network.

28

Example
Calculate29

Example
Calculate30

Example
Suppose we now want to compute
Using the chain rule again:

The same! Let’s avoid
duplicated computation …
31

Example
Suppose we now want to compute
Using the chain rule again:

32

Example
How to compute, whereandn outputs, nm inputs, by definition, the Jacobianis33

ExampleNow we can calculateof shape

34

Shape Convention
The shape of the parameter is
By definition of Jacobian, 1 output, nm inputs, so the shape ofis

To make it more convenient for make gradient update, we use theshape convention: the shape of the gradient is the shape of the parameters!

35

Example
For shape convention, we reshape, so it has the same shape as

36

Example
For shape convention, we reshape, so it has the same shape as

A more direct way: compute the outer product ofandand this directly makes the shape37

Backpropagation
We use gradient descent to optimize the parameters in neural networks.
The key question: How can we efficiently calculate the gradients in neuralnetworks (or any computational graph)?
The key intuition: View neural networks (or any computational graph) ascompositions of functions and use chain rules to calculate the gradient.
This is Backpropagation = Gradient Descent + Chain Rule

38

Computation Graph
In software library, neural networks are implemented as Computation Graph (CG).

39

Forward Propagation in CG
In software library, neural networks are implemented as Computation Graph (CG).

40

Backpropagation in CG
In software library, neural networks are implemented as Computation Graph (CG).

41

Backpropagation avoids duplicated computationBackpropagation follows the computation graph in reverse order, so you avoid
dupdated computation.

42

Backpropagation: Single Node
Node receives an upstream gradient.
Each node has a local gradient.
Goal is to pass on the correct downstream gradient.
By Chain Rule: downstream gradient = upstream gradient x local gradient.

43

Backpropagation: Multiple Inputs
With multiple inputs, each input has a local gradient.

44

Backpropagation: Multiple Outputs
If a node has multiple outgoing edges during forward propagation, we sum thegradients during the backpropagation.

45Example

46

Example47

Example48

Example49

Example50

Example

51

Example52

Node Intuitions+ “distributes” the upstream gradient to each summand

53

Node Intuitions+ “distributes” the upstream gradient to each summandmax “routes” the upstream gradient54

Node Intuitions+ “distributes” the upstream gradient to each summandmax “routes” the upstream gradient* “switches” the upstream gradient55

Backpropagation in General Computation Graph●●

F-prop: visit nodes in topological sortorder - Compute value of node given
predecessors.
B-prop:○ initialize output gradient = 1○ visit nodes in reverse order:
Compute gradient wrt each nodeusing gradient wrt successors

56

AutoDiff: Automatic Differentiation
The gradient computation can be automatically inferred from the symbolic
expression of the fprop.
Each node type needs to know how to compute its output and how to compute thegradient wrt its inputs given the gradient wrt its output.

57

Check Your Implementation with Numeric Gradient
Following the definition of gradient:

Use a very small h to compute this value numerically.
Useful for checking your implementation.

58Optimizers
Once we calculate the gradient with respect to the objective function, we optimize
objective function by gradient descent.
The very basic one is standard gradient descent. But there are many otherchoices of optimizers:
●
●
●
●
●
●
●
●
●
●

Batch Gradient Descent
Stochastic Gradient Descent (SGD)
Mini-batch Gradient Descent
SGD with Momentum
Nesterov accelerated gradient
AdagradAdadelta
RMSPropAdam; AdamW (most commonly used)
Adafactor

59

Batch Gradient Descent

New Parameters

Old Parameters

Learning Rate

Gradient60

Mini-batch Gradient Descent

Gradient for a batch of training examples; batch size n61

Stochastic Gradient Descent (SGD)

Gradient for each training example

62

SGD with Momentum
Momentum parameter, usually 0.9 or a similar value

63

Nesterov Accelerated Gradient (NAG)

calculating the gradient not w.r.t. to our current
parameters θ but w.r.t. the approximate future
position of our parameters64

Nesterov Accelerated Gradient (NAG)

Momentum● first computes the current gradient (small blue
calculating the gradient not w.r.t. to our currentvector)
parameters θ but w.r.t. the approximate future● then takes a big jump in the direction of theposition of our parameters
updated accumulated gradient (big blue vector)
NAG● first makes a big jump in the direction of theprevious accumulated gradient (brown vector)● measures the gradient and then makes a correction(red vector)● results in the complete NAG update (green vector).
65

AdagradAdagrad adapts the learning rate to the parameters: larger updates for
infrequent parameters, smaller updates for frequent parameters.
It uses a different learning rate for every parameter θi at every time step t,
A diagonal matrix where eachdiagonal element is the sum of thesquares of the gradients w.r.t. θi up
to time step tsmoothing term that avoids divisionby zero66

AdagradAdagrad adapts the learning rate to the parameters: larger updates for
infrequent parameters, smaller updates for frequent parameters.
It uses a different learning rate for every parameter θi at every time step t,
A diagonal matrix where eachdiagonal element is the sum of thesquares of the gradients w.r.t. θi up
to time step tsmoothing term that avoids divisionby zeroProblem: learning rate continuously decreases, and training can stall -- fixed byusing rolling average in AdaDelta, RMSProp, Adam.

67

Adaptive Moment Estimation (Adam)
m_t and v_t are estimates of the first moment (the mean) and the second moment (theuncentered variance) of the gradients respectively, hence the name of the method.
like momentum
rolling average!

68Adaptive Moment Estimation (Adam)
m_t and v_t are estimates of the first moment (the mean) and the second moment (theuncentered variance) of the gradients respectively, hence the name of the method.
like momentum
rolling average!

As m_t and v_t are initialized as vectors of 0’s, the authors of Adam observe that they arebiased towards zero, especially during the initial time steps, and especially when the decay
rates are small (i.e. β1 and β2 are close to 1). They counteract these biases by computing
bias-corrected first and second moment estimates,

Then update the parameters
69Training Tricks
Shuffling the Training Data Every Epoch: avoid providing the training examples in ameaningful order to our model as this may bias the optimization algorithm.
Regularization: L2 regularization adds L2 norm of parameters to the loss function.
Dropout: randomly zero-out nodes in the hidden layer with probability p at training time only
Learning Rate Decay: gradually reduce learning rate as training continues
Early stopping● You should thus always monitor error on a validation set during training and stop (withsome patience) if your validation error does not improve enough.
● Use Patience70Parameter Initialization
Very Important for neural networks to achieve good performance.
Uniform Initialization: Initialize weights in some range, such as [-0.1, 0.1] forexampleXavier Initialization: n(l) is the number of input units to W (fan-in) and n(l+1) is
the number of output units from W (fan-out).

71