Deep Learning Limitations andNew Frontiers
Alexander AminiT-shirts!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 1 slide, 1 minute• List of example proposals on
website: CLICK HERE
• Presentations on Friday, Feb 2• Submit groups by TODAY at
9pm to be eligible• Submit slide by Thursday 9pm
to be eligible• Judged by a panel of industry judges• Top winners are awarded:
2x NVIDIA Titan TX2MSRP: $600

3x Google Home
MSRP: $300

1x NVIDIA Titan Xp
MSRP: $1200

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Final Class Project
Option 1: Proposal Presentation• Groups of 3 or 4
• Present a novel deep learning
research idea or application• 1 slide, 1 minute• List of example proposals onwebsite: CLICK HERE
• Presentations on Friday, Feb 2• Submit groups by TODAY at
9pm to be eligible• Submit slide by Thursday 9pm
to be eligible

Option 2: Write a 1-page reviewof a deep learning paper (singlespaced)• Suggested papers listed on
website: CLICK HERE
• Grade is based on clarity ofwriting and technicalcommunication of main ideas

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Thursday: Deep Learning in Industry
Urs Muller
Chief Architect Autonomous Driving• End-to-End Learning for Self
Driving Cars

D Sculley• Issues with Image Classification
Shanqing Cai• Faster TensorFlow Developmentwith TF Debugger and Eager Mode

Industry sponsors recruitment booths setup in front of class

NVIDIAGoogleIBM

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Tencent
1/31/18

Friday: Project Presentations
Lisa Amini
Director of IBM Research
Cambridge & Acting Directorof MIT/IBM AI Lab

Lin MaTechnical Lead, Manager• Computer Vision Meets
Social Networks

Afternoon Session:
Final Project Presentations
Pizza and Awards!
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

So far in 6.S191…

So far in 6.S191…

Data• Signals• Images• Sensors…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

So far in 6.S191…

DataDecision• Signals• Images• Sensors…

• Prediction• Detection• Action…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

So far in 6.S191…

DataDecision• Signals• Images• Sensors…

• Prediction• Detection• Action…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Power of Neural Nets
Universal Approximation Theorem
A feedforward network with a single layer is sufficient to approximate, toan arbitrar y precision, any continuous function.

Hornik, K., et al. "Multilayer feedforward
networks are universal approximators." (1989)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18Power of Neural Nets
Universal Approximation Theorem
A feedforward network with a single layer is sufficient to approximate, toan arbitrar y precision, any continuous function.

Caveats:
The number ofhidden units may
be infeasibly large

The resulting
model may not
generalize
Hornik, K., et al. Neural Networks. (1989)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

History of Artificial Intelligence Hype

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Limitations

Rethinking Generalization“Understanding Deep Neural Networks requires rethinking generalization”

dogbananadogtree

Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Rethinking Generalization“Understanding Deep Neural Networks requires rethinking generalization”

dogbananadog

tree

Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Rethinking Generalization“Understanding Deep Neural Networks requires rethinking generalization”

dogbananadog

treebanana

dog

tree

dog
Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Rethinking Generalization“Understanding Deep Neural Networks requires rethinking generalization”

dogbananadogtreebananadogtree

dog
Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Capacity of Deep Neural Networks
100%

Training Set

accuracy0%

Testing Set

originallabelsrandomization
6.S191 Introduction to Deep Learning
introtodeeplearning.com

completelyrandom
1/31/18

Capacity of Deep Neural Networks
100%

Training Set

accuracy0%

Testing Set

originallabelsrandomization
6.S191 Introduction to Deep Learning
introtodeeplearning.com

completelyrandom
1/31/18

Capacity of Deep Neural Networks
Modern deep networks
can perfectly fit torandom data

100%

Training Set

accuracy0%

Testing Set

originallabels

randomization
6.S191 Introduction to Deep Learning
introtodeeplearning.com

completelyrandom
1/31/18

Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Function Approximators
Neural networks are excellent function approximators?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Function Approximators
Neural networks are excellent function approximators…when they have training data

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Function Approximators
Neural networks are excellent function approximators…when they have training data

How do we know
when our network
doesn’t know?

6.S191 Introduction to Deep Learningintrotodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks

Despois. “Adversarial
examples and theirimplications”. 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent𝜕𝐽(𝜃, 𝑥, 𝑦)𝜃 ← 𝜃−𝜂𝜕𝜃“How does a small change in weights increase our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent𝜕𝐽(𝜃, 𝑥, 𝑦)𝜃 ← 𝜃−𝜂𝜕𝜃“How does a small change in weights decrease our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent𝜕𝐽(𝜃, 𝑥, 𝑦)𝜃 ← 𝜃−𝜂𝜕𝜃

Fix your image 𝑥,and true label 𝑦“How does a small change in weights decrease our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Adversarial Image:Modify image to increase error
𝜕𝐽(𝜃, 𝑥, 𝑦)𝑥 ←𝑥+𝜂
𝜕𝑥“How does a small change in the input increase our loss”
Goodfellow et al.
2014
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Adversarial Image:
Modify image to increase error
𝜕𝐽(𝜃, 𝑥, 𝑦)𝑥 ←𝑥+𝜂
𝜕𝑥“How does a small change in the input increase our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Adversarial Attacks on Neural Networks
Adversarial Image:
Modify image to increase error
𝜕𝐽(𝜃, 𝑥, 𝑦)𝑥 ←𝑥+𝜂
𝜕𝑥

Fix your weights 𝜃,and true label 𝑦“How does a small change in the input increase our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Neural Network Limitations…
• Very data hungry (eg. often millions of examples)• Computationally intensive to train and deploy (tractably requires GPUs)• Easily fooled by adversarial examples• Poor at representing uncertainty (how do you know what the model knows?)• Uninterpretable black boxes, difficult to trust• Finicky to optimize: non-convex, choice of architecture, learning parameters• Often require expert knowledge to design, fine tune architectures

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Neural Network Limitations…
• Very data hungry (eg. often millions of examples)• Computationally intensive to train and deploy (tractably requires GPUs)• Easily fooled by adversarial examples• Poor at representing uncertainty (how do you know what the model knows?)• Uninterpretable black boxes, difficult to trust• Finicky to optimize: non-convex, choice of architecture, learning parameters• Often require expert knowledge to design, fine tune architectures

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

New Frontiers 1:

Bayesian Deep LearningWhy Care About Uncertainty?

ℙ(cat)ORℙ(dog)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Why Care About Uncertainty?
ℙ cat = 0.2ℙ dog = 0.8
Remember :ℙ cat + ℙ dog = 1
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Bayesian Deep Learning for Uncertainty
Network tries to learn output, 𝒀, directly from raw data, 𝑿Find mapping, 𝑓, parameterized by weights 𝜽 such thatmin ℒ(𝒀, 𝑓 𝑋; 𝜽 )Bayesian neural networks aim to learn a posterior over weights,ℙ 𝜽 𝑿, 𝒀 :ℙ 𝒀 𝑿, 𝜽 ℙ(𝜽)ℙ 𝜽 𝑿, 𝒀 =ℙ(𝒀|𝑿)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Bayesian Deep Learning for Uncertainty
Network tries to learn output, 𝒀, directly from raw data, 𝑿
Find mapping, 𝑓, parameterized by weights 𝜽 such thatmin ℒ(𝒀, 𝑓 𝑋; 𝜽 )Bayesian neural networks aim to learn a posterior over weights,ℙ 𝜽 𝑿, 𝒀 :Intractable!

ℙ 𝒀 𝑿, 𝜽 ℙ(𝜽)ℙ 𝜽 𝑿, 𝒀 =ℙ(𝒀|𝑿)

Approximate the posterior ℙ 𝜽 𝑿, 𝒀 by sampling
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Elementwise Dropout for Uncertainty
Evaluate 𝑇 stochastic forward passes through the network 𝜽G JGHI
Dropout as a form of stochastic sampling 𝑧L,G ~ 𝐵𝑒𝑟𝑛𝑜𝑢𝑙𝑙𝑖 𝑝
Unregularized Kernel𝜽⊙Stochastic Sampled𝜽GBernoulli Dropout𝑧𝜽,G=∀𝑤 ∈𝜽J

1\ 𝑿 = ] 𝑓 𝑿 𝜽G𝔼 𝒀𝑇GHIJ1\ 𝑿 = ] 𝑓(𝑿)` − 𝔼 𝒀\𝑿 `𝑉𝑎𝑟 𝒀𝑇GHIGal and Ghahramani, ICML, 2016.
Amini et al., NIPS Workshop on Bayesian Deep Learning, 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Model Uncertainty Application

Input image

Predicted Depth

Model Uncertainty

Kendall, Gal, NIPS 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

New Frontiers II:
Learning to Learn

Motivation
Standard deep neural networks are optimized for a single task

Greater the need for specializedengineers

Complexity of models increases

Often require expert knowledge to build an architecture for a given task
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Possible Solution
AutoML: Learning to Learn
Build a learning algorithm that learns which model to use tosolve a given problem

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

AutoML: Learning to Learn

Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Model Controller
At each step, the model samples a brand new network

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

The Child Network

Training Data

Sampled networkfrom RNN

Predicted Labels

Compute final accuracy on this dataset
Update RNN controller based on the accuracy of the child network after training
Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Learning to Learn: A level deeper

Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

This Spawns a Very Powerful Idea
• Design an AI algorithm that can build new models capable of solving atask• Reduces the need for experienced engineers to design the networks• Makes deep learning more accessible to the public
The connection to Artificial
General Intelligence:
the ability to intelligently
reason about how we learn
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/31/18

Questions?