Introduction to Deep Learning
MIT 6.S191
Alexander Amini

What is Deep Learning?
ARTIFICIALINTELLIGENCE
Any technique thatenables computers to
mimic human behavior

MACHINE LEARNING
Ability to learn without
explicitly being programmed

6.S191 Introduction to Deep Learning
introtodeeplearning.com

DEEP LEARNING
Learn underlying features in datausing neural networks

1/29/18

Deep Learning Success: Vision
Image Recognition

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Deep Learning Success: Vision

6.S191 Lab!

Detect pneumothorax in real X-Ray scans

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Deep Learning Success: Audio

6.S191 Lab!

Music Generation

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Deep Learning SuccessAnd so many more…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

6.S191 Goals

Fundamentals

Practical Skills

Advancements

Community @MIT

Knowledge, intuition, know-how, and community to do
deep learning research and development
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Administrative Information• Meets Mon Jan 29 – Fri Feb 2• 10:30 am – 1:30pm• Lecture + Lab Breakdown• Graded P/D/F; 3 Units• 1 Final Assignment

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 1 slide, 1 minute (strict)• List of example proposals on
website: introtodeeplearning.com• Presentations on Friday, Feb 2• Submit groups by Wednesday
5pm to be eligible• Submit slide by Thursday 9pm
to be eligible• Judged by a panel of industry judges• Top winners are awarded:
2x NVIDIA Titan TX2
MSRP: $600

3x Google Home
MSRP: $300

1x NVIDIA Titan Xp
MSRP: $1200

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 1 slide, 1 minute (strict)• List of example proposals on
website: introtodeeplearning.com• Presentations on Friday, Feb 2• Submit groups by Wednesday
5pm to be eligible• Submit slide by Thursday 9pm
to be eligible

Option 2: Write a 1-page review
of a deep learning paper• Suggested papers listed on
website: introtodeeplearning.com• Grade is based on clarity ofwriting and technicalcommunication of main ideas• Due Friday 10:30am (before
lecture)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Class Support• Piazza: https://piazza.com/class/iwmlwep2fnd5uu• Course Website: http://introtodeeplearning.com• Lecture slides: http://introtodeeplearning.com/#schedule• Email us: introtodeeplearning-staff@mit.edu• Office Hours by request

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Course Staff

Alexander Amini
Lead Organizer

Harini Suresh
Co-Chair

Shraman

Yilun

Abhimanyu

Jose Javier

Alex

NikhilFaraazTim

Bhaktipriya

Ava Soleimany
Lead Organizer

Lex Fridman
Co-Chairi n tr otode e pl e a r n i n g-sta f f @mi t.e du6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Thanks to Sponsors!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Why Deep Learning and Why Now?

Why Deep Learning?
Hand engineered features are time consuming, brittle and not scalable in practice
Can we learn the underlying features directly from data?
Low Level Features

Mid Level Features

High Level Features

Lines & Edges

Eyes & Nose & Ears

Facial Structure

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Why Now?
1952
1958

1986

Stochastic Gradient
Descent
Perceptron•

Backpropagation•

1995

Learnable Weights

Neural Networks date back decades, so why the resurgence?
1. Big Data

2. Hardware

3. Software• Larger Datasets• Easier
Collection &Storage• Graphics
Processing Units(GPUs)• Massively
Parallelizable• Improved
Techniques• New Models• Toolboxes

Multi-Layer Perceptron

Deep Convolutional NN•

Digit Recognition

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron

The structural building block of deep learning

The Perceptron: Forward Propagation
Output

𝑥#
𝑥%

𝜃#
𝜃%

$

𝑦) = 𝑔Σ𝑦)𝜃$

Weights, 𝑥 - 𝜃- .#Non-linear
activation function𝑥$

InputsLinear combinationof inputs

Sum

Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Forward Propagation1𝑥#

Output𝜃/$𝑦) = 𝑔𝜃#𝜃%𝑥%Linear combinationof inputsΣ𝜃/ + , 𝑥 - 𝜃-.#𝑦)Non-linear
activation function

𝜃$

Bias

𝑥$
Inputs

WeightsSum

Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Forward Propagation1𝑥#𝑦) = 𝑔

𝜃#
𝜃%

𝑥%

$

𝜃/

-.#

Σ

𝑦)

𝜃$

Weights𝑦) = 𝑔 𝜃 / + 𝑿 3 𝜽
where: 𝑿 =𝑥$Inputs𝜃/ + , 𝑥 - 𝜃-

Sum

𝑥#
𝜃#
⋮
and 𝜽 = ⋮𝑥$𝜃$

Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Forward Propagation
1
𝑥#

Activation Functions
𝜃/
𝜃#
𝜃%

𝑥%

𝑦) = 𝑔Σ𝑦)𝜃/ + 𝑿3 𝜽• Example: sigmoid function
𝑔 𝑧 = 𝜎 𝑧 =𝜃$1
1 + 𝑒 9:𝑥$
Inputs

WeightsSum

Non-Linearity Output
𝑧
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Common Activation Functions
Sigmoid Function

Hyperbolic Tangent

Rectified Linear Unit (ReLU)

1𝑔 𝑧 =1 + 𝑒 9:𝑒 : − 𝑒 9:𝑔 𝑧 = :
𝑒 + 𝑒 9:𝑔 𝑧 = max ( 0 , 𝑧 )𝑔 ′ 𝑧 = 𝑔(𝑧) 1 − 𝑔(𝑧)𝑔 ′ 𝑧 = 1 − 𝑔(𝑧)%tf.nn.sigmoid(z)tf.nn.tanh(z)𝑔′(𝑧) = D

1,
0,

𝑧> 0
otherwise

tf.nn.relu(z)NOTE: All activation functions are non-linear
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

What if we wanted to build a Neural Network to
distinguish green vs red points?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

Linear Activation functions produce linear
decisions no matter the network size
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

Linear Activation functions produce linear
decisions no matter the network size

Non-linearities allow us to approximate
arbitrarily complex functions

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Example
1𝑥#

We have: 𝜃 / = 1 and 𝜽 =
1
3

Σ

𝑦)

−2

3
−2

𝑦) = 𝑔 𝜃 / + 𝑿 3 𝜽𝑥# 3 3= 𝑔 1+ 𝑥%−2𝑦) = 𝑔 ( 1 + 3 𝑥 # − 2 𝑥 % )𝑥%This is just a line in 2D!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Example𝑦) = 𝑔 ( 1 + 3 𝑥 # − 2 𝑥 % )
𝑥%

1
𝑥#

1
3

Σ

𝑦)

−2

𝑥#

𝑥%

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Example𝑦) = 𝑔 ( 1 + 3 𝑥 # − 2 𝑥 % )
𝑥%

1
𝑥#

1
3

−1
2

Σ

𝑦)

−2

𝑥#

𝑥%
−1
2
𝑦) = 𝑔 1 + 3 ∗ −1 − 2 ∗ 2= 𝑔 −6 ≈ 0.002
Assume we have input: 𝑿 =6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Example𝑦) = 𝑔 ( 1 + 3 𝑥 # − 2 𝑥 % )
𝑥%

1
𝑥#

1
3

𝑧 < 0
𝑦 < 0.5

Σ

𝑦)

−2

𝑥#

𝑥%
𝑧 > 0
𝑦 > 0.5

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Building Neural Networks with Perceptrons

The Perceptron: Simplified
1
𝑥#

𝜃/
𝜃#

Σ

𝜃%

𝑥%

𝑦)

𝜃$

𝑥$
Inputs

WeightsSum

Non-Linearity Output

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

The Perceptron: Simplified

𝑥#
𝑥%

𝑦=𝑔 𝑧

𝑧

𝑥$

𝑧 = 𝜃/ + ,

$
V.#

𝑥V 𝜃V

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Multi Output Perceptron

𝑥#
𝑧#
𝑥%
𝑧%

𝑦# = 𝑔 𝑧#
𝑦% = 𝑔 𝑧%𝑥$𝑧- = 𝜃/,- + ,$V.#

𝑥V 𝜃V,-

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Single Layer Neural Network
(%)

(#)

𝜽

𝑔 𝑧#

𝑧#

𝑥#
𝑧%

𝜽

𝑔 𝑧%

𝑦)#

𝑥%
𝑧W
𝑥$
𝑧XY
Inputs

𝑔 𝑧W

𝑦)%

𝑔 𝑧XY

HiddenFinal Output

$
(#)
(#)
𝑧- = 𝜃/,- + , 𝑥V 𝜃V,V.#

XY
(%)
(%)
𝑦)- = 𝜃/,- + , 𝑧V 𝜃V,V.#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Single Layer Neural Network
𝑧#
𝑥#

(#)

𝜃#,%
𝜃%,%

(#)

𝑧%

𝑦)#

(#)

𝑧W

𝑦)%

𝑥%
𝑥$

𝜃$,%

𝑧XY
(#)
𝑧% = 𝜃/,% + ,
(#)

$
V.#

(#)

𝑥V 𝜃V,%

(#)

(#)

(#)

= 𝜃/,% + 𝑥# 𝜃#,% + 𝑥% 𝜃%,% + 𝑥$ 𝜃$,%6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Multi Output Perceptron
𝑧#
𝑥#
𝑧%

𝑦)#

𝑧W

𝑦)%

𝑥%
𝑥$
𝑧XY
Inputs

Hidden6.S191 Introduction to Deep Learning
introtodeeplearning.com

Output1/29/18

Deep Neural Network
𝑧Z,#
𝑥#
𝑥%

⋯

𝑧Z,%

⋯

𝑧Z,W

𝑦)#
𝑦)%

𝑥$
𝑧Z,X[
Inputs

OutputHidden(Z)𝑧Z,- = 𝜃/,+,

X[\Y
V.#

(Z)
𝑔(𝑧Z9#,V ) 𝜃V,-

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Applying Neural Networks

Example Problem
Will I pass this class?
Let’s start with a simple two feature model𝑥 # = Number of lectures you attend
𝑥 % = Hours spent on the final project

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Example Problem: Will I pass this class?
𝑥 % = Hours
spent on the
final project
LegendPassFail𝑥 # = Number of lectures you attend
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Example Problem: Will I pass this class?
𝑥 % = Hours
spent on the
final project
Legend
?
4
5

Pass
Fail

𝑥 # = Number of lectures you attend
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Example Problem: Will I pass this class?
𝑧#𝑥#𝑥 # = 4 ,5

𝑧%

𝑦)#

Predicted: 0.1

𝑥%𝑧W

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Example Problem: Will I pass this class?
𝑧#
𝑥#

𝑥#

= 4 ,5𝑧%𝑥%𝑦)#Predicted: 0.1
Actual: 1

𝑧W

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Quantifying Loss
The loss of our network measures the cost incurred from incorrect predictions
𝑧#
𝑥#

𝑥#

= 4 ,5𝑦)#𝑧%𝑥%Predicted: 0.1
Actual: 1𝑧Wℒ 𝑓 𝑥 (-) ; 𝜽 , 𝑦 (-)
Predicted

Actual

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Empirical Loss
The empirical loss measures the total loss over our entire dataset𝒙 =Also known as:• Objective function• Cost function• Empirical Risk

4, 5
2, 1
5, 8
⋮ ⋮

𝑧#
𝑥#
𝑦)#

𝑧%
𝑥%
𝑧W

𝑓(𝑥)

𝑦

0.1 ✖
0.8 ✖
0.6 ✔
⋮

1
0
1
⋮

e
1
𝑱 𝜽 = , ℒ 𝑓 𝑥 (-) ; 𝜃 , 𝑦 (-)
𝑛 -.#Predicted
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Actual
1/29/18

Binary Cross Entropy Loss
Cross entropy loss can be used with models that output a probability between 0 and 1

𝒙 =

4, 5
2, 1
5, 8
⋮ ⋮

𝑧#
𝑥#
𝑦)#

𝑧%
𝑥%
𝑧W

e
1
𝑱 𝜽 = , 𝑦 (-) log 𝑓 𝑥 - ; 𝜃𝑛-.#ActualPredicted

𝑓(𝑥)

𝑦

0.1 ✖
0.8 ✖
0.6 ✔
⋮

1
0
1
⋮

+ (1 − 𝑦 (-) ) log 1 − 𝑓 𝑥 - ; 𝜃ActualPredicted

loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(model.y, model.pred) )Mean Squared Error Loss
Mean squared error loss can be used with regression models that output continuous real numbers

𝒙 =

4, 5
2, 1
5, 8
⋮ ⋮

𝑓(𝑥)

𝑧#
𝑥#
𝑦)#

𝑧%
𝑥%
𝑧W
e
1
𝑱𝜽 = ,
𝑦 - − 𝑓 𝑥 - ;𝜃𝑛-.#Actual%𝑦

30 ✖ 9080 ✖ 2085 ✔ 95⋮⋮

Final Grades(percentage)

Predicted

loss = tf.reduce_mean( tf.square(tf.subtract(model.y, model.pred) )Training Neural Networks

Loss Optimization
We want to find the network weights that achieve the lowest losse1𝜽∗ = argmin , ℒ 𝑓 𝑥 (-); 𝜽 , 𝑦(-)
𝑛𝜽-.#𝜽∗ = argmin 𝐽(𝜽)𝜽

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Optimization
We want to find the network weights that achieve the lowest losse1𝜽∗ = argmin , ℒ 𝑓 𝑥 (-); 𝜽 , 𝑦(-)
𝑛𝜽-.#𝜽∗ = argmin 𝐽(𝜽)𝜽

Remember :𝜽 = 𝜃 (/) , 𝜃 (#) , ⋯6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Optimization𝜽∗ = argmin 𝐽(𝜽)𝜽

Remember:
Our loss is a function of
the network weights!

𝐽(𝜃/, 𝜃# )𝜃/𝜃#
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Optimization
Randomly pick an initial (𝜃/, 𝜃#)𝐽(𝜃/, 𝜃# )𝜃/𝜃#
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Optimization
Compute gradient,

jk(𝜽)
j𝜽

𝐽(𝜃/, 𝜃# )𝜃/𝜃#
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Optimization
Take small step in opposite direction of gradient𝐽(𝜃/, 𝜃# )𝜃/𝜃#
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Gradient Descent
Repeat until convergence𝐽(𝜃/, 𝜃# )𝜃/𝜃#
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)weights = tf.random_normal(shape, stddev=sigma)

2. Loop until convergence:
3.

jk(n)
Compute gradient, jn

4.

Update weights, 𝜃 ← 𝜃 − 𝜂

grads = tf.gradients(ys=loss, xs=weights)

jk(n)
jn

weights_new = weights.assign(weights – lr * grads)

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)weights = tf.random_normal(shape, stddev=sigma)

2. Loop until convergence:
3.

jk(n)
Compute gradient, jn

4.

Update weights, 𝜃 ← 𝜃 − 𝜂

grads = tf.gradients(ys=loss, xs=weights)

jk(n)
jn

weights_new = weights.assign(weights – lr * grads)

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝑧#

𝜃%

𝑦)

𝐽(𝜽)

How does a small change in one weight (ex. 𝜃%) affect the final loss 𝐽(𝜽)?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝜃%

𝑧#

𝑦)

𝐽(𝜽)

𝜕𝐽(𝜽)
=
𝜕𝜃%

Let’s use the chain rule!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝑧#

𝜃%

𝑦)

𝐽(𝜽)

𝜕𝐽(𝜽)
𝜕𝑦)
𝜕𝐽(𝜽)
∗
=
𝜕𝑦)
𝜕𝜃%
𝜕𝜃%

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝜃%

𝑧#

𝑦)

𝐽(𝜽)

𝜕𝐽(𝜽)
𝜕𝑦)
𝜕𝐽(𝜽)
∗
=
𝜕𝑦)
𝜕𝜃#
𝜕𝜃#

Apply chain rule!

Apply chain rule!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝑧#

𝜃%

𝜕𝐽(𝜽)
𝜕𝑦)
𝜕𝐽(𝜽)
∗
=
𝜕𝑦)
𝜕𝑧#
𝜕𝜃#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

𝑦)

𝐽(𝜽)

𝜕𝑧#
∗
𝜕𝜃#

1/29/18

Computing Gradients: Backpropagation
𝑥

𝜃#

𝑧#

𝜃%

𝜕𝐽(𝜽)
𝜕𝑦)
𝜕𝐽(𝜽)
∗
=
𝜕𝑦)
𝜕𝑧#
𝜕𝜃#

𝑦)

𝐽(𝜽)

𝜕𝑧#
∗
𝜕𝜃#

Repeat this for every weight in the network using gradients from later layers
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Neural Networks in Practice:
Optimization

Training Neural Networks is Difficult“Visualizing the loss landscapeof neural nets”. Dec 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Functions Can Be Difficult to Optimize
Remember:
Optimization through gradient descent𝜕 𝐽 (𝜃 )𝜃←𝜃−𝜂𝜕𝜃

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Loss Functions Can Be Difficult to Optimize
Remember:
Optimization through gradient descent𝜕 𝐽 (𝜃 )𝜃←𝜃−𝜂𝜕𝜃How can we set thelearning rate?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Setting the Learning Rate
Small learning rate converges slowly and gets stuck in false local minima

𝐽(𝜽)Initial guess𝜃
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Setting the Learning Rate
Large learning rates overshoot, become unstable and diverge

𝐽(𝜽)Initial guess𝜃
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Setting the Learning Rate
Stable learning rates converge smoothly and avoid local minima

𝐽(𝜽)Initial guess𝜃
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

How to deal with this?
Idea 1:
Try lots of different learning rates and see what works “just right”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

How to deal with this?
Idea 1:
Try lots of different learning rates and see what works “just right”

Idea 2:
Do something smarter!
Design an adaptive learning rate that “adapts” to the landscape

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Adaptive Learning Rates• Learning rates are no longer fixed• Can be made larger or smaller depending on:• how large gradient is• how fast learning is happening• size of particular weights• etc...

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Adaptive Learning Rate Algorithms• Momentum

tf.train.MomentumOptimizer

Qian et al. “On the momentum term in gradientdescent learning algorithms.” 1999.

• Adagrad

tf.train.AdagradOptimizer

Duchi et al. “Adaptive Subgradient Methods for Online
Learning and Stochastic Optimization.” 2011.

• Adadelta

tf.train.AdadeltaOptimizer

Zeiler et al. “ADADELTA: An Adaptive Learning Rate
Method.” 2012.

• Adamtf.train.AdamOptimizer

Kingma et al. “Adam: A Method for Stochastic
Optimization.” 2014.

• RMSProp

tf.train.RMSPropOptimizer

Additional details: http://ruder.io/optimizing-gradient-descent/6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Neural Networks in Practice:
Mini-batches

Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:
3.

jk(n)
Compute gradient, jn

4.

Update weights, 𝜃 ← 𝜃 − 𝜂

jk(n)
jn

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:
3.

jk(n)
Compute gradient, jn

4.

Update weights, 𝜃 ← 𝜃 − 𝜂

jk(n)
jn

5. Return weights
Can be very
computational tocompute!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:3.

Pick single data point 𝑖

4.

Compute gradient,

5.

jk(n)
Update weights, 𝜃 ← 𝜃 − 𝜂 jn

jks (n)jn

6. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:3.

Pick single data point 𝑖

4.

Compute gradient,

5.

jk(n)
Update weights, 𝜃 ← 𝜃 − 𝜂 jn

6. Return weightsjks (n)jn

Easy to compute butvery noisy(stochastic)!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:3.

Pick batch of 𝐵 data points

4.

Compute gradient,

5.

jk(n)
# u jk[ (n)∑=jnu Z.# jnjk(n)Update weights, 𝜃 ← 𝜃 − 𝜂 jn

6. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~𝒩(0, 𝜎 %)2. Loop until convergence:3.

Pick batch of 𝐵 data points

4.

Compute gradient,

5.

jk(n)
# u jk[ (n)∑=jnu Z.# jnjk(n)Update weights, 𝜃 ← 𝜃 − 𝜂 jn

6. Return weights
Fast to compute and a much better
estimate of the true gradient!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Mini-batches while training
More accurate estimation of gradient
Smoother convergence
Allows for larger learning rates

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Mini-batches while training
More accurate estimation of gradient
Smoother convergence
Allows for larger learning rates

Mini-batches lead to fast training!
Can parallelize computation + achieve significant speed increases on GPU’s

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Neural Networks in Practice:OverfittingThe Problem of Overfitting

Underfitting
Model does not have capacity
to fully learn the data

Ideal fit

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Overfitting
Too complex, extra parameters,
does not generalize well

1/29/18

Regularization
What is it?
Technique that constrains our optimization problem to discourage complex models

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization
What is it?
Technique that constrains our optimization problem to discourage complex models

Why do we need it?
Improve generalization of our model on unseen data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 1: Dropout• During training, randomly set some activations to 0

𝑧#,#

𝑧%,#

𝑧#,%

𝑧%,%

𝑦)#

𝑧#,W

𝑧%,W

𝑦)%

𝑧#,w

𝑧%,w

𝑥#
𝑥%
𝑥W

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 1: Dropout• During training, randomly set some activations to 0
• Typically ‘drop’ 50% of activations in layer• Forces network to not rely on any 1 node

tf.nn.dropout(hiddenLayer, p=0.5)

𝑧#,#

𝑧%,#

𝑧#,%

𝑧%,%

𝑦)#

𝑧#,W

𝑧%,W

𝑦)%

𝑧#,w

𝑧%,w

𝑥#
𝑥%
𝑥W

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 1: Dropout• During training, randomly set some activations to 0
• Typically ‘drop’ 50% of activations in layer• Forces network to not rely on any 1 node

tf.nn.dropout(hiddenLayer, p=0.5)

𝑧#,#

𝑧%,#

𝑧#,%

𝑧%,%

𝑦)#

𝑧#,W

𝑧%,W

𝑦)%

𝑧#,w

𝑧%,w

𝑥#
𝑥%
𝑥W

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

Loss

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Stop training
here!

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Regularization 2: Early Stopping• Stop training before we have a chance to overfit
Under-fitting

Over-fitting
LegendLoss

Stop training
here!

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Core Foundation Review
The Perceptron

Neural Networks• Structural building blocks• Nonlinear activation
functions• Stacking Perceptrons toform neural networks• Optimization throughbackpropagation

𝑥#
𝑥%
𝑥$

𝑦)

𝑥%
𝑥$

• Adaptive learning• Batching• Regularization

𝑧Z,#

𝑥#

Σ

Training in Practice

⋯

𝑧Z,%
𝑧Z,W

⋯

𝑦)#
𝑦)%

𝑧Z,X[

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/18

Questions?