Introduction to Deep Learning
MIT 6.S191

Alexander Amini
January 28, 2019

The Rise of Deep Learning

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

What is Deep Learning?
ARTIFICIALINTELLIGENCE
Any technique that enables
computers to mimic
human behavior

MACHINE LEARNING
Ability to learn without
explicitly being programmed

6.S191 Introduction to Deep Learning
introtodeeplearning.com

DEEP LEARNING
Extract patterns from data usingneural networks

1/28/19

Lecture Schedule• Mon Jan 28 – Fri Feb 1• 1:00 pm – 4:00pm• Lecture + Lab Breakdown• Graded P/D/F; 3 Units• 1 Final Assignment
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 3 minutes (strict)• List of example proposals on
website: introtodeeplearning.com• Presentations on Friday, Feb 1• Submit groups by Wednesday
5pm to be eligible• Submit slide by Thursday 9pm
to be eligible• Judged by a panel of industry judges• Top winners are awarded:

3x NVIDIA RTX 2080 Ti
MSRP: $4000

6.S191 Introduction to Deep Learning
introtodeeplearning.com

4x Google Home
MSRP: $400

1/28/19

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 3 minutes (strict)• List of example proposals on
website: introtodeeplearning.com• Presentations on Friday, Feb 2• Submit groups by Wednesday
5pm to be eligible• Submit slide by Thursday 9pm
to be eligible

Option 2: Write a 1-page review
of a deep learning paper• Grade is based on clarity ofwriting and technicalcommunication of main ideas• Due Friday 1:00pm (before
lecture)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Class Support• Piazza: http://piazza.com/mit/spring2019/6s191• Useful for discussing labs

• Course Website: http://introtodeeplearning.com• Lecture schedule• Slides and lecture recordings• Software labs
• Grading policy• Email us: introtodeeplearning-staff@mit.edu• Office Hours by request
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Course Staff
Alexander Amini
Lead Organizer

Ava Soleimany
Lead Organizer

Thomas

Mauri

Harini

Houssam

Julia

Felix

Jacob

Rohil

Gilbertintrotodeeplearning-staff@mit.edu
6.S191 Introduction to Deep Learning
introtodeeplearning.com+ Ravi A.
1/28/19

Thanks to Sponsors!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Why Deep Learning and Why Now?

Why Deep Learning?
Hand engineered features are time consuming, brittle and not scalable in practice
Can we learn the underlying features directly from data?
Low Level Features

Mid Level Features

High Level Features

Lines & Edges

Eyes & Nose & Ears

Facial Structure

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Why Now?
Neural Networks date back decades, so why the resurgence?
1952
1958

1986

Stochastic Gradient
Descent
Perceptron•

2. Hardware

3. Software• Larger Datasets• Easier Collection& Storage• Graphics
Processing Units(GPUs)• Massively
Parallelizable• Improved
Techniques• New Models• Toolboxes

Backpropagation•

1995

Learnable Weights

1. Big Data

Multi-Layer Perceptron

Deep Convolutional NN•

Digit Recognition

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron

The structural building block of deep learning

The Perceptron: Forward Propagation
Output

&"
&$

!"
!$

#

(' = *

Σ('!#

Weights+ &, !,,-"

Non-linear
activation function&#

InputsLinear combinationof inputs

Sum

Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Forward Propagation
1&"

Output!%#)( = +!"!$&$

Linear combinationof inputsΣ!%+ - &. !.
./")(Non-linear
activation function

!#

Bias

&#
Inputs

WeightsSum

Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Forward Propagation1&")( = +

!"
!$

&$

#

!%

./"

Σ

)(

!#

Weights)( = + ! % + 1 2 3
where: 1 =&#Inputs!%+ - &. !.

Sum

!"
&"
⋮
and 3 = ⋮!#&#Non-Linearity Output
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Forward Propagation
1&"

Activation Functions
!%
!"
!$

&$

*) = ,

Σ*)!% + . / 0• Example: sigmoid function, 1 = 2 1 =!#1
1 + 3 45&#
Inputs

WeightsSum

Non-Linearity Output
1
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Common Activation Functions
Sigmoid Function

Hyperbolic Tangent

Rectified Linear Unit (ReLU)

1! " =1 + & '(& ( − & '(! " = (
& + & '(

! " = max ( 0 , " )! ′ " = !(") 1 − !(")

! ′ " = 1 − !(")-

tf.nn.sigmoid(z)tf.nn.tanh(z)!′(") = 3

1,
0,

" > 0
otherwise

tf.nn.relu(z)NOTE: All activation functions are non-linear
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

What if we wanted to build a Neural Network to
distinguish green vs red points?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

Linear Activation functions produce linear
decisions no matter the network size
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Importance of Activation Functions
The purpose of activation functions is to introduce non-linearities into the network

Linear Activation functions produce linear
decisions no matter the network size

Non-linearities allow us to approximate
arbitrarily complex functions

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Example

1&'

We have: + , = 1 and . =
1
3

Σ

*)

−2

3
−2

*) = / + , + 1 2 .
&' 2 3= / 1+ &(−2*) = / ( 1 + 3 & ' − 2 & ( )&(This is just a line in 2D!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Example*) = , ( 1 + 3 & ' − 2 & ( )

( =

1

*)

3&

Σ

1+

&'

3

' −

2&

1

0

&(

−2

&'

&(

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Example*) = , ( 1 + 3 & ' − 2 & ( )

Σ

*)

( =

2&
' −

3

−1
2

3&

&'

1

1+

1

0

&(

−2

&'

&(
−1
2
*) = , 1 + 3 ∗ −1 − 2 ∗ 2= , −6 ≈ 0.002
Assume we have input: 0 =6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Example*) = , ( 1 + 3 & ' − 2 & ( )

Σ

*)

( =

2&
' −

3

1 < 0
* < 0.5

3&

&'

1

1+

1

0

&(

−2

&'

&(
1 > 0
* > 0.5

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Building Neural Networks with Perceptrons

The Perceptron: Simplified
1
&"

!%
!"

Σ

!$

&$

*)

!#

&#
Inputs

WeightsSum

Non-Linearity Output

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

The Perceptron: Simplified

!$
!"

%

&=( %

!#

#

% = )* + ,

-.$

!- )-

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Multi Output Perceptron

!$
%$
!"
%"

&$ = ( %$&" = ( %"!##%) = *+,) + .

/0$!/ */,)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Single Layer Neural Network
7

($)
6 %$

%$

!$
%"

7

(")

6 %"

('$

!"
%&
!#
%)*
Inputs

('"

6 %&

6 %)*

Hidden#($)($)%+ = -.,+ + 3 !4 -4,+
45$
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Final Output('+ = 6)*(")(")-.,+ + 3 %4 -4,+
45$
1/28/19

Single Layer Neural Network
%$
!$

($)

5$,"
5","

($)

%"

('$

($)

%&

('"

!"
!#

5#,"

%)*
#
($)
($)
%" = ,-," + 2 !3 ,3,"
34$
($)

($)

($)

($)

= ,-," + !$ ,$," + !" ,"," + !# ,#,"
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Multi Output Perceptron
from tf.keras.layers import *inputs = Inputs(m)hidden = Dense(d1)(inputs)
outputs = Dense(2)(hidden)
model = Model(inputs, outputs)

%$
!$
%"

('$

%&

('"

!"
!#
%)*
Inputs

Hidden6.S191 Introduction to Deep Learning
introtodeeplearning.com

Output1/28/19

Deep Neural Network
%&,$
!$
!"

⋯

%&,"
%&,(

⋯

*)$
*)"

!#
%&,+,
Inputs

HiddenOutput

+,78
(&)
(&)
%&,- = /0,- + 49(%&:$,5 ) /5,56$
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Applying Neural Networks

Example Problem
Will I pass this class?
Let’s start with a simple two feature model! " = Number of lectures you attend! $ = Hours spent on the final project

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example Problem: Will I pass this class?
! " = Hours
spent on the
final project
LegendPassFail! $ = Number of lectures you attend
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example Problem: Will I pass this class?
! " = Hours
spent on the
final project
Legend
?

4
5

Pass
Fail

! $ = Number of lectures you attend
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example Problem: Will I pass this class?

$#!#! # = 4 ,5

$"

'&#

Predicted: 0.1!"$%

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Example Problem: Will I pass this class?

$#

!

#

!#

= 4 ,5$"!"'&#Predicted: 0.1
Actual: 1$%

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Quantifying Loss
The loss of our network measures the cost incurred from incorrect predictions
$#

!

#

!#

= 4 ,5

$"

'&#

!"

Predicted: 0.1
Actual: 1$%ℒ , ! (.) ; 1 , ' (.)
Predicted

Actual

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Empirical Loss
The empirical loss measures the total loss over our entire dataset)=

Also known as:• Objective function• Cost function• Empirical Risk

4, 5
2, 1
5, 8
⋮ ⋮

$#
!#
$"

'&#

!"
$%

+(!)

'

0.1
0.8
0.6
⋮

1
0
1
⋮

5
1
. / = 2 ℒ + ! (3) ; / , ' (3)134#Predicted
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Actual
1/28/19

Binary Cross Entropy Loss
Cross entropy loss can be used with models that output a probability between 0 and 1

) =

4, 5
2, 1
5, 8
⋮ ⋮

$#
!#
'&#

$"
!"
$%

5
1
. / = 2 ' (3) log + ! 3 ; /1
34#ActualPredicted

+(!)

'

0.1
0.8
0.6
⋮

1
0
1
⋮

+ (1 − ' (3) ) log 1 − + ! 3 ; /ActualPredicted

loss = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits(model.y, model.pred) )Mean Squared Error Loss
Mean squared error loss can be used with regression models that output continuous real numbers

) =

4, 5
2, 1
5, 8
⋮ ⋮

$#
!#
'&#

$"
!"
$%
5
1
. / = 2' 3 − + ! 3 ;/1
34#

Actual

"

+(!)

'

30
80
85
⋮

90
20
95
⋮

Final Grades(percentage)

Predicted

loss = tf.reduce_mean( tf.square(tf.subtract(model.y, model.pred) )Training Neural Networks

Loss Optimization
We want to find the network weights that achieve the lowest loss
01!∗ = argmin , ℒ 2 3 (-) ; ! , 8 (-)+!
-./!∗ = argmin 9(!)!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Loss Optimization
We want to find the network weights that achieve the lowest loss
01!∗ = argmin , ℒ 2 3 (-) ; ! , 8 (-)+!
-./!∗ = argmin 9(!)!

Remember:! = !(:) , !(/) , ⋯6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Loss Optimization!∗ = argmin *(!)!

Remember:
Our loss is a function of
the network weights!

*(-., -0)-.

-0

Loss Optimization
Randomly pick an initial ("#, "%)'("#, "%)"#"%

Loss Optimization
Compute gradient,

!"($)
!$

&('(, '*)'('*

Loss Optimization
Take small step in opposite direction of gradient!(#$, #&)#$#&

Gradient Descent
Repeat until convergence!(#$, #&)#$#&

Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')weights = tf.random_normal(shape, stddev=sigma)

2. Loop until convergence:
3.

)*(+)
Compute gradient, )+4.

Update weights, + ← + − .

grads = tf.gradients(ys=loss, xs=weights)

)*(+)
)+

weights_new = weights.assign(weights – lr * grads)

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')weights = tf.random_normal(shape, stddev=sigma)

2. Loop until convergence:
3.

)*(+)
Compute gradient, )+4.

Update weights, + ← + − .

grads = tf.gradients(ys=loss, xs=weights)

)*(+)
)+

weights_new = weights.assign(weights – lr * grads)

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

'

!)

()

!"

+*

#(%)

How does a small change in one weight (ex. !") affect the final loss #(%)?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

)

&+

&'

*+

-,

"($)

!"($)
=
!&'

Let’s use the chain rule!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

,

&.

-.

&'

*)

"($)

!"($)
!*)
!"($)
=
∗
!&'
!*)
!&'

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

,

&'

&.

-'

*)

"($)

!"($)
!*)
!"($)
=
∗
!&'
!*)
!&'

Apply chain rule!

Apply chain rule!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

-

&'

,'

&.

*)

"($)

!"($)
!*)
!"($)
!,'
=
∗
∗
!&'
!*)
!,'
!&'

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Computing Gradients: Backpropagation

-

&'

,'

&.

*)

"($)

!"($)
!*)
!"($)
!,'
=
∗
∗
!&'
!*)
!,'
!&'

Repeat this for every weight in the network using gradients from later layers
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Neural Networks in Practice:
Optimization

Training Neural Networks is Difficult“Visualizing the loss landscapeof neural nets”. Dec 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Loss Functions Can Be Difficult to Optimize

Remember:
Optimization through gradient descent%& (!)!←!−$%!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Loss Functions Can Be Difficult to Optimize

Remember:
Optimization through gradient descent%& (!)!←!−$%!
How can we set thelearning rate?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Setting the Learning Rate
Small learning rate converges slowly and gets stuck in false local minima"(!)

Initial guess!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Setting the Learning Rate
Large learning rates overshoot, become unstable and diverge"(!)

Initial guess!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Setting the Learning Rate
Stable learning rates converge smoothly and avoid local minima"($)

Initial guess!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

How to deal with this?
Idea 1:
Try lots of different learning rates and see what works “just right”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

How to deal with this?
Idea 1:
Try lots of different learning rates and see what works “just right”

Idea 2:
Do something smarter!
Design an adaptive learning rate that “adapts” to the landscape

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Adaptive Learning Rates• Learning rates are no longer fixed• Can be made larger or smaller depending on:• how large gradient is• how fast learning is happening• size of particular weights• etc...

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Adaptive Learning Rate Algorithms• Momentum

tf.train.MomentumOptimizer

Qian et al. “On the momentum term in gradientdescent learning algorithms.” 1999.

• Adagrad

tf.train.AdagradOptimizer

Duchi et al. “Adaptive Subgradient Methods for Online
Learning and Stochastic Optimization.” 2011.

• Adadelta

tf.train.AdadeltaOptimizer

Zeiler et al. “ADADELTA: An Adaptive Learning Rate
Method.” 2012.

• Adamtf.train.AdamOptimizer

Kingma et al. “Adam: A Method for Stochastic
Optimization.” 2014.

• RMSProp

tf.train.RMSPropOptimizer

Additional details: http://ruder.io/optimizing-gradient-descent/6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Neural Networks in Practice:
Mini-batches

Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:
3.

)*(+)
Compute gradient, )+4.

Update weights, + ← + − .

)*(+)
)+

5. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:
3.

)*(+)
Compute gradient, )+4.

Update weights, + ← + − .

)*(+)
)+

5. Return weights
Can be very
computational tocompute!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:3.

Pick single data point )4.

Compute gradient,

5.

*+(-)
Update weights, - ← - − 0 *-*+, (-)*-

6. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:3.

Pick single data point )4.

Compute gradient,

5.

*+(-)
Update weights, - ← - − 0 *-6. Return weights*+, (-)*-

Easy to compute butvery noisy(stochastic)!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:3.

Pick batch of ) data points

4.

Compute gradient,

5.

*+(,)
. / *+3 (,)∑=*,/ 12. *,*+(,)
Update weights, , ← , − 6 *,6. Return weights

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Stochastic Gradient Descent
Algorithm
1. Initialize weights randomly ~"(0, & ')2. Loop until convergence:3.

Pick batch of ) data points

4.

Compute gradient,

5.

*+(,)
. / *+3 (,)∑=*,/ 12. *,*+(,)
Update weights, , ← , − 6 *,6. Return weights
Fast to compute and a much better
estimate of the true gradient!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Mini-batches while training

More accurate estimation of gradient
Smoother convergence
Allows for larger learning rates

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Mini-batches while training

More accurate estimation of gradient
Smoother convergence
Allows for larger learning rates

Mini-batches lead to fast training!
Can parallelize computation + achieve significant speed increases on GPU’s

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Neural Networks in Practice:OverfittingThe Problem of Overfitting

Underfitting
Model does not have capacity
to fully learn the data

Ideal fit

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Overfitting
Too complex, extra parameters,
does not generalize well

1/28/19

Regularization

What is it?
Technique that constrains our optimization problem to discourage complex models

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization

What is it?
Technique that constrains our optimization problem to discourage complex models

Why do we need it?
Improve generalization of our model on unseen data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 1: Dropout• During training, randomly set some activations to 0

'$,$

'",$

'$,"

'","

&%$

'$,#

'",#

&%"

'$,)

'",)

!$
!"
!#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 1: Dropout• During training, randomly set some activations to 0
• Typically ‘drop’ 50% of activations in layer• Forces network to not rely on any 1 node

tf.keras.layers.Dropout(p=0.5)

'$,$

'",$

'$,"

'","

&%$

'$,#

'",#

&%"

'$,)

'",)

!$
!"
!#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 1: Dropout• During training, randomly set some activations to 0
• Typically ‘drop’ 50% of activations in layer• Forces network to not rely on any 1 node

tf.keras.layers.Dropout(p=0.5)

'$,$

'",$

'$,"

'","

&%$

'$,#

'",#

&%"

'$,)

'",)

!$
!"
!#

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

Loss

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

LegendLoss

Stop training
here!

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Regularization 2: Early Stopping• Stop training before we have a chance to overfit

Under-fitting

Over-fitting
LegendLoss

Stop training
here!

Testing
Training

Training Iterations
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Core Foundation Review
The Perceptron

Neural Networks• Structural building blocks• Stacking Perceptrons toform neural networks• Adaptive learning• Optimization throughbackpropagation• Regularization• Nonlinear activation
functions

"%
"#
"$

'&

"#
"$

• Batching

(),%

"%

Σ

Training in Practice

⋯

(),#
(),+

⋯

'&%
'&#

(),,-

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/28/19

Questions?