Deep Learning:
Limitations and New Frontiers
MIT 6.S191

Ava Soleimany
January 30, 2019

T-shirts! Today!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Course Schedule

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Final Class Project
Option 1: Proposal Presentation• Present a novel deep learning
research idea or application• Groups of 1 welcome• Listeners welcome• Groups of 2 to 4 to be eligible
for prizes, incl. 1 for-credit student• 3 minutes• Proposal instructions:goo.gl/JGJ5E7• Judged by a panel of industry judges• Top winners are awarded:

3x NVIDIA RTX 2080 Ti
MSRP: $4000

6.S191 Introduction to Deep Learning
introtodeeplearning.com

4x Google HomeMSRP: $400

1/30/19

Final Class Project
Option 1: Proposal Presentation

Proposal Logistics• Present a novel deep learning
research idea or application• Groups of 1 welcome• Listeners welcome• Groups of 2 to 4 to be eligible
for prizes, incl. 1 for-credit student• 3 minutes• Proposal instructions:goo.gl/JGJ5E7• >= 1 for-credit student to be eligible
for prizes• Prepare slides on Google Slides• Group submit by today 10pm:
goo.gl/rV6rLK• In class project work: Thu, Jan 31• Slide submit by Thu 11:59 pm:
goo.gl/7smL8w• Presentations on Friday, Feb 1

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Final Class Project
Option 1: Proposal Presentation• Present a novel deep learning
research idea or application• Groups of 1 welcome• Listeners welcome• Groups of 2 to 4 to be eligible
for prizes, incl. 1 for-credit student• 3 minutes• Proposal instructions:goo.gl/JGJ5E7

Option 2: Write a 1-page review
of a deep learning paper• Grade is based on clarity ofwriting and technicalcommunication of main ideas• Due Friday 1:00pm (beforelecture)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Thursday: Visualization in ML +Biologically Inspired Learning
Fernanda Viegas,
Co-Director Google PAIR
Data Visualization forMachine Learning

Final project work
Ask us questions!
Open office hours!

Dmitry Krotov,
MIT-IBM Watson AI Lab
Biologically Inspired DeepLearning

Work with group members!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19Friday: Learning and Perception +Project Proposals + Awards + Pizza
Project Proposals!
Jan Kautz,
VP of Research
Learning and Perception

Judging and Awards!
Pizza Celebration!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

So far in 6.S191…

The Rise of Deep Learning

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

So far in 6.S191…

Data• Signals• Images• Sensors…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

So far in 6.S191…

DataDecision• Signals• Images• Sensors…

• Prediction• Detection• Action…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

So far in 6.S191…

DataDecision• Signals• Images• Sensors…

• Prediction• Detection• Action…

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Power of Neural Nets
Universal Approximation Theorem
A feedforward network with a single layer is sufficient to approximate, toan arbitrary precision, any continuous function.

Hornik et al. Neural Networks. (1989)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Power of Neural NetsUniversal Approximation Theorem
A feedforward network with a single layer is sufficient to approximate, toan arbitrary precision, any continuous function.

Caveats:
The number ofhidden units may
be infeasibly large

The resulting
model may not
generalize
Hornik et al. Neural Networks. (1989)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Artificial Intelligence “Hype”: Historical Perspective

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Limitations

Rethinking Generalization“Understanding Deep Neural Networks Requires Rethinking Generalization”

dogbanana

dogtreeZhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Rethinking Generalization“Understanding Deep Neural Networks Requires Rethinking Generalization”

dogbananadogtreeZhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Rethinking Generalization“Understanding Deep Neural Networks Requires Rethinking Generalization”

dogbananadog

treebanana

dog

tree

dog
Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Rethinking Generalization“Understanding Deep Neural Networks Requires Rethinking Generalization”

dogbanana

dogtreebananadogtreedog
Zhang et al. ICLR. (2017)6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Capacity of Deep Neural Networks
100%

accuracy

0%originallabelsrandomization
Training Set

Testing Set

6.S191 Introduction to Deep Learning
introtodeeplearning.com

completely
randomZhang et al. ICLR. (2017)
1/30/19

Capacity of Deep Neural Networks
100%

accuracy0%
originallabelsrandomization
Training Set

Testing Set

6.S191 Introduction to Deep Learning
introtodeeplearning.com

completelyrandom
Zhang et al. ICLR. (2017)
1/30/19

Capacity of Deep Neural Networks
Modern deep networks canperfectly fit to random data

100%

accuracy0%
originallabels

randomization
Training Set

Testing Set

6.S191 Introduction to Deep Learning
introtodeeplearning.com

completelyrandom
Zhang et al. ICLR. (2017)
1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Networks as Function Approximators
Neural networks are excellent function approximators…when they have training data

How do we know when ournetwork doesn’t know?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks

Despois. “Adversarial examples and their implications” (2017).
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent%&(!, ), *)! ←!−$%!
“How does a small change in weights decrease our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent%&(!, ), *)! ←!−$%!
“How does a small change in weights decrease our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Remember:
We train our networks with gradient descent%&(!, ), *)! ←!−$%!

Fix your image ),and true label *“How does a small change in weights decrease our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Adversarial Image:
Modify image to increase error%&((, !, *)! ←!+$%!
“How does a small change in the input increase our loss”

Goodfellow et al. NIPS (2014)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Adversarial Image:
Modify image to increase error%&((, !, *)! ←!+$%!
“How does a small change in the input increase our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Adversarial Attacks on Neural Networks
Adversarial Image:
Modify image to increase error%&((, !, *)! ←!+$%!

Fix your weights (,and true label *“How does a small change in the input increase our loss”

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Synthesizing Robust Adversarial Examples

Athalye et al. ICML. (2018)
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Network Limitations…
• Very data hungry (eg. often millions of examples)• Computationally intensive to train and deploy (tractably requires GPUs)• Easily fooled by adversarial examples• Can be subject to algorithmic bias
• Poor at representing uncertainty (how do you know what the model knows?)• Uninterpretable black boxes, difficult to trust• Finicky to optimize: non-convex, choice of architecture, learning parameters• Often require expert knowledge to design, fine tune architectures

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Neural Network Limitations…
• Very data hungry (eg. often millions of examples)• Computationally intensive to train and deploy (tractably requires GPUs)• Easily fooled by adversarial examples• Can be subject to algorithmic bias
• Poor at representing uncertainty (how do you know what the model knows?)• Uninterpretable black boxes, difficult to trust• Finicky to optimize: non-convex, choice of architecture, learning parameters• Often require expert knowledge to design, fine tune architectures

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

New Frontiers 1:

Bayesian Deep LearningWhy Care About Uncertainty?
ℙ(cat)ORℙ(dog)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Why Care About Uncertainty?
ℙ cat = 0.2ℙ dog = 0.8
Remember: ℙ cat + ℙ dog = 1
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Bayesian Deep Learning for Uncertainty
Network tries to learn output, !, directly from raw data, "Find mapping, #, parameterized by weights $ such thatmin ℒ(!, # +; $ )Bayesian neural networks aim to learn a posterior over weights,ℙ $ ", ! :ℙ ! ", $ ℙ($)
ℙ $ ", ! =ℙ(!|")

6.S191 Introduction to Deep Learningintrotodeeplearning.com

1/30/19

Bayesian Deep Learning for Uncertainty
Network tries to learn output, !, directly from raw data, "Find mapping, #, parameterized by weights $ such thatmin ℒ(!, # +; $ )Bayesian neural networks aim to learn a posterior over weights,ℙ $ ", ! :ℙ ! ", $ ℙ($)
Intractable! ℙ $ ", ! =ℙ(!|")

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Elementwise Dropout for Uncertainty
Evaluate ! stochastic forward passes through the network "# &#$%
Dropout as a form of stochastic sampling '(,# ~ +,-./0112 3
Unregularized Kernel"

Stochastic Sampled"#Bernoulli Dropout'",#⊙∀5 ∈"&1: < = > ? < "#9 ;!

=#$%

&1
: < = > ?(<)D − 9 ;:< D@A- ;!
#$%

Gal and Ghahramani, ICML, 2016.
Amini, Soleimany, et al., NIPS Workshop on Bayesian Deep Learning, 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Model Uncertainty Application

Input image

Predicted Depth

Model Uncertainty

Kendall, Gal, NIPS, 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com1/30/19

Multi-Task Learning Using Uncertainty

Kendall, et al., CVPR, 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Multi-Task Learning Using Uncertainty

Kendall, et al., CVPR, 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Multi-Task Learning Using Uncertainty

Kendall, et al., CVPR, 2018.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

New Frontiers II:
Learning to Learn

Motivation: Learning to Learn
Standard deep neural networks are optimized for a single task

Complexity of models increases

Greater need for specialized engineers

Often require expert knowledge to build an architecture for a given task

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

Motivation: Learning to Learn
Standard deep neural networks are optimized for a single task

Complexity of models increases

Greater need for specialized engineers

Often require expert knowledge to build an architecture for a given task
Build a learning algorithm that learns which model to use to solve a given problem
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19Motivation: Learning to Learn
Standard deep neural networks are optimized for a single task

Complexity of models increases

Greater need for specialized engineers

Often require expert knowledge to build an architecture for a given task
Build a learning algorithm that learns which model to use to solve a given problem

AutoML
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

AutoML: Learning to Learn

Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

AutoML: Model Controller
At each step, the model samples a brand new network

Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

AutoML: The Child Network

Training Data

Sampled networkfrom RNNPrediction

Compute final accuracy on this dataset.
Update RNN controller based on the accuracy of the child network after training.

Zoph and Le, ICLR 2017.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

AutoML on the Cloud

Google Cloud.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/19

AutoML Spawns a Powerful Idea
• Design an AI algorithm that can build new models
capable of solving a task
• Reduces the need for experienced engineers todesign the networks• Makes deep learning more accessible to the public

Connection to
Artificial General Intelligence:
the ability to intelligently
reason about how we learn

Questions?