Deep Learning:
State of the Art (2020)

Deep Learning Lecture Series

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

“AI began with an ancient wish to forge the gods.”- Pamela McCorduck, Machines Who Think, 1979
Frankenstein (1818)

Ex Machina (2015)

Visualized here are 3% of the neurons and 0.0001% of the synapses in the brain.
Thalamocortical system visualization via DigiCortex Engine.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Deep Learning & AI in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago• Civilization
12,000 years ago

1700s and beyond: Industrial revolution, steam
engine, mechanized factory systems, machine tools• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

Dreams, mathematical foundations, and engineering in reality.
Alan Turing, 1951: “It seems probable that once the machine
thinking method had started, it would not take long to outstrip
our feeble powers. They would be able to converse with eachother to sharpen their wits. At some stage therefore, we shouldhave to expect the machines to take control."https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago• Civilization
12,000 years ago

Dreams, mathematical foundations, and engineering in reality.
Frank Rosenblatt, Perceptron (1957, 1962): Early description andengineering of single-layer and multi-layer artificial neuralnetworks.

• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago

Kasparov vs Deep Blue, 1997

• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago

Lee Sedol vs AlphaGo, 2016

• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago

Robots on four wheels.

• Modern humans
300,000 years ago• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Artificial Intelligence in Context of Human History
We are here

Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago

Robots on two legs.

• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

History of Deep Learning Ideas and Milestones*• 1943: Neural networks
We are here

• 1957-62: Perceptron• 1970-86: Backpropagation, RBM, RNN• 1979-98: CNN, MNIST, LSTM, Bidirectional RNN• 2006: “Deep Learning”, DBN
Perspective:• Universe created
13.8 billion years ago• Earth created
4.54 billion years ago• Modern humans
300,000 years ago• 2009: ImageNet + AlexNet• 2014: GANs
• 2016-17: AlphaGo, AlphaZero• 2017: 2017-19: Transformers* Dates are for perspective and not as definitive historical
record of invention or credit• Civilization
12,000 years ago• Written record
5,000 years ago
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Turing Award for Deep Learning• Yann LeCun• Geoffrey Hinton• Yoshua Bengio

Turing Award given for:• “The conceptual and engineering breakthroughs that have madedeep neural networks a critical component of computing.”• (Also, for popularization in the face of skepticism.)
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Early Key Figures in Deep Learning(Not a Complete List by Any Means)• 1943: Walter Pitts and Warren McCulloch
Computational models for neural nets
• 1957, 1962: Frank Rosenblatt
Perceptron (Single-Layer & Multi-Layer)• 1965: Alexey Ivakhnenko and V. G. Lapa
Learning algorithm for MLP• 1970: Seppo Linnainmaa
Backpropagation and automatic differentiation• 1979: Kunihiko Fukushima
Convolutional neural networks• 1982: John Hopfield
Hopfield networks (recurrent neural networks)
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

People of Deep Learning and Artificial Intelligence• History of science is a story of both people and ideas.
• Many brilliant people contributed to the development of AI.
Schmidhuber, Jürgen. "Deep learning inneural networks: An overview." Neuralnetworks 61 (2015): 85-117
https://arxiv.org/pdf/1404.7828.pdf

My (Lex) hope for the community:• More respect, open-mindedness, collaboration, credit sharing.
• Less derision, jealousy, stubbornness, academic silos.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Limitations of Deep Learning• 2019 is the year it became coolto say that “deep learning” haslimitations.
• Books, articles, lectures, debates,
videos were released thatlearning-based methods cannot
do commonsense reasoning.
Prediction from Rodney Brooks:“By 2020, the popular press starts having stories that the era of
Deep Learning is over.”
http://rodneybrooks.com/predictions-scorecard-2019-january-01/
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[3, 4]

https://deeplearning.mit.edu 2020

Deep Learning Research Community is Growing

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[2]

https://deeplearning.mit.edu 2020

Deep Learning Growth, Celebrations, and Limitations

Hopes for 2020
• Less Hype & Less Anti-Hype: Less tweets on how
there is too much hype in AI and more solid
research in AI.
• Hybrid Research: Less contentious, counterproductive debates, more open-mindedinterdisciplinary collaboration.
• Research topics:• Reasoning• Active learning and life-long learning• Multi-modal and multi-task learning• Open-domain conversation• Applications: medical, autonomous vehicles• Algorithmic ethics• Robotics
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Competition and Convergence of Deep Learning Libraries

TensorFlow 2.0 and PyTorch 1.3• Eager execution by default(imperative programming)• Keras integration + promotion• Cleanup (API, etc.)• TensorFlow.js• TensorFlow Lite
• TensorFlow Serving• TorchScript(graph representation)• Quantization• PyTorch Mobile(experimental)• TPU support

Python 2 support ended on Jan 1, 2020.

>>> print “Goodbye World”
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Reinforcement Learning Frameworks• TensorFlow• OpenAI Baselines• Stable Baselines – the one I recommend for beginners• TensorForce• Dopamine (Google)• TF-Agents• TRFL• RLLib (+ Tune) – great for distributed RL & hyperparameter tuning• Coach - huge selection of algorithms• PyTorch• Horizon• SLM-Lab• Misc• RLgraph• Keras-RL

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Reinforcement Learning Frameworks• “Stable Baselines” (OpenAI Baselines Fork)• A2C, PPO, TRPO, DQN, ACKTR, ACER and DDPG
• Good documentation (and code commenting)• Easy to get started and use

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[5]https://deeplearning.mit.edu 2020

Deep Learning and Deep RL Frameworks

Hopes for 2020
• Framework-agnostic Research: Make it eveneasier to translate a trained PyTorch model toTensorFlow and vice-versa.
• Mature Deep RL frameworks: Converge to fewer,
actively-developed, stable RL frameworks thatare less tied to TensorFlow or PyTorch.

• Abstractions: Build higher and higher
abstractions (i.e. Keras) on top of deep learning
frameworks that empower researchers, scientists,
developers outside of machine learning field.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Transformer

Vaswani et al. "Attention is all you need." Advances in Neural Information Processing Systems. 2017.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[7, 8]

https://deeplearning.mit.edu 2020

BERT

Devlin, Jacob, et al. "Bert: Pre-training of deep bidirectional transformers for language understanding." (2018).
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[9, 10]https://deeplearning.mit.edu 2020

BERT Applications
Now you can use BERT:• Create contextualized wordembeddings (like ELMo)• Sentence classification• Sentence pair classification• Sentence pair similarity• Multiple choice• Sentence tagging• Question answering

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[9, 10]https://deeplearning.mit.edu 2020

Transformer-Based Language Models• BERT (Google)• XLNet (Google/CMU)• RoBERTa (Facebook)• DistilBERT (HuggingFace)• CTRL (Salesforce)• GPT-2 (OpenAI)• ALBERT (Google)• Megatron (NVIDIA)

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[12]https://deeplearning.mit.edu 2020

Megatron
Shoeybi et al. (NVIDIA)• Megatron-LM is a 8.3 billion parameter transformer language model with 8way model parallelism and 64-way data parallelism trained on 512 GPUs
(NVIDIA Tesla V100)• Largest transformer model ever trained. 24x the size of BERT and 5.6x thesize of GPT-2.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[13]

https://deeplearning.mit.edu 2020

XLNET
Yang et al. (CMU, Google AI)• Combine bidirectionality of BERT and the relative positional embeddings and therecurrence mechanism of Transformer-XL.
• XLnet outperforms BERT on 20 tasks, often by a large margin.
• The new model achieves state-of-the-art performance on 18 NLP tasks including question
answering, natural language inference, sentiment analysis & document ranking.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[24, 25]https://deeplearning.mit.edu 2020

ALBERT
Lan et al. (Google Research & Toyota Technological Institute at Chicago)

Machine performanceon the RACE challenge(SAT-like reading
comprehension).

• Idea: Reduces parameters via cross-layer parameter sharing• Results: An upgrade to BERT that advances the state-of-the-art performanceon 12 NLP tasks (including SQuAD2.0)• Code: Open-source TensorFlow implementation, including a number ofready-to-use ALBERT pre-trained language models

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[11]https://deeplearning.mit.edu 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Transformers Model Language
They Do Not Understand Language. Far from it (for now).

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020• Key takeaways in the report:• Coordination during model release between organization is difficult but
possible• Humans can be convinced by synthetic text
• Machine-based detection is difficult.

• My takeaways• Conversations on this topic are difficult, because the model of sharing
between ML organizations and experts is mostly non-existent• Humans are still better at deception (disinformation) and detection in
text and conversation (see Alexa prize slides)
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[26]

https://deeplearning.mit.edu 2020

Transferable Multi-Domain State Generator for TaskOriented Dialogue Systems
Wu et al. (Honk Kong UST, Salesforce) – ACL 2019 Outstanding Paper• Task: Dialogue state tracking• Problem: Over-dependence on domain
ontology and lack of knowledge sharing
across domains• Details:• Share model parameters across domains• Track slot values mentioned anywhere
in a dialogue history with a contextenhanced slot gate and copy mechanism• Don’t require a predefined ontology• Results: State-of-the-art joint goal

accuracy of 48.62% on MultiWOZ,
a challenging 5-domain humanhuman dialogue dataset.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[34]

https://deeplearning.mit.edu 2020

Explain Yourself! Leveraging Language Models for
Commonsense Reasoning
Rajani et al. (Salesforce)

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[15]

https://deeplearning.mit.edu 2020

Alexa Prize and Open Domain Conversations• Amazon open-sourced the Topical-Chat dataset.
• Alexa Prize (like the Loebner Prize, etc) are teaching us valuablelessons (from the Alquist 2.0 team):• Parts: Break dialogue into small parts• Tangents: Create an interconnected graph of topics. Be ready to jumpfrom context to context and back.
• Attention: Not everything that is said is important. E.g.: “You know, I’ma really terrible cook. But I would like to ask you, what’s your favorite
food?”• Opinions: Create opinions.
• ML• Content: ML is okay for generic chitchat, but nothing more for now.
• Classification: ML classifies intent, finds entities or detects sentiment.

• Goal: Goal is to maximize entertainment not information.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[17, 18, 19]https://deeplearning.mit.edu 2020

Alexa Prize and Open Domain Conversations

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[19]

https://deeplearning.mit.edu 2020

code2seq: Generating Sequences from Structured
Representations of CodeAlon et al. (Technion) – ICLR 2019
• Instead of treating source code as a sequence of tokens, code2seq leverages
the syntactic structure of programming languages to better encode sourcecode as paths in its abstract syntax tree (AST).

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[14]

https://deeplearning.mit.edu 2020

Natural Language Processing:

Hopes for 2020
• Reasoning: Combining (commonsense) reasoning
with language models• Context: Extending language model context tothousands of words.
• Dialogue: More focus on open-domain dialogue• Video: Ideas and successes in self-supervised
learning in visual data.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

OpenAI & Dota 2• Dota 2 as a testbed for the messiness and continuous nature of
the real world: teamwork, long time horizons, and hidden
information.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[21]

https://deeplearning.mit.edu 2020

OpenAI & Dota 2 Progress• Aug, 2017: 1v1 bot beats top professional Dota 2 players.
• Aug, 2018: OpenAI Five lost two games against top Dota 2 players at The
International. “We are looking forward to pushing Five to the next level.”• Apr, 2019: OpenAI Five beats OG team (the 2018 world champions)

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[21]

https://deeplearning.mit.edu 2020

OpenAI & Dota 2 Progress• The Difference: OpenAI Five’s victories in 2019 as compared to its losses in2018 are due to: 8x more training compute (training for longer)• Compute: Current version of OpenAI Five has consumed 800 petaflop/s-daysand experienced about 45,000 years of Dota self-play over 10 realtime months• Performance: The 2019 version of OpenAI Five has a 99.9% win rate versus
the 2018 version.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[21]

https://deeplearning.mit.edu 2020

DeepMind Quake III Arena Capture the Flag

• “Billions of people inhabit the planet, each with their own individual goals
and actions, but still capable of coming together through teams,
organisations and societies in impressive displays of collective intelligence.
This is a setting we call multi-agent learning: many individual agents must
act independently, yet learn to interact and cooperate with other agents.
This is an immensely difficult problem - because with co-adapting agents
the world is constantly changing.”
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[23]https://deeplearning.mit.edu 2020

DeepMind Quake III Arena Capture the Flag
• The agents automatically figure out the game rules, importantconcepts, behaviors, strategies, etc.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[23]https://deeplearning.mit.edu 2020

DeepMind Quake III Arena Capture the Flag
• The agents automatically figure out the game rules, importantconcepts, behaviors, strategies, etc.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[23]https://deeplearning.mit.edu 2020

DeepMind AlphaStar• Dec, 2018: AlphaStar beats MaNa, one of the world’s strongest professional
StarCraft players, 5-0.
• Oct, 2019: AlphaStar reaches Grandmaster level playing the game under
professionally approved conditions (for humans).

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[22]

https://deeplearning.mit.edu 2020

DeepMind AlphaStar“AlphaStar is an intriguing and unorthodox player – one with the reflexes andspeed of the best pros but strategies and a style that are entirely its own. The
way AlphaStar was trained, with agents competing against each other in aleague, has resulted in gameplay that’s unimaginably unusual; it really makes
you question how much of StarCraft’s diverse possibilities pro players havereally explored.”- Kelazhur, professional StarCraft II player
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[22]

https://deeplearning.mit.edu 2020

Pluribus: Six-Player No-Limit Texas Hold’em Poker
Brown et al. (CMU, Facebook AI)• Six-Player No-Limit Texas Hold’em Poker• Imperfect information• Multi-agent• Result: Pluribus won in six-player no-limit Texas Hold’em poker• Offline: Self-play to generate coarse-grained “blueprint” strategy• Iterative Monte Carlo CFR (MCCFR) algorithm• Self-play allows for counterfactual reasoning• Online: Use search to improve blueprint strategy based onparticular situation• Abstractions• Action abstractions: reduce action space• Information abstraction: reduce decision space based on what
information has been revealed
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[28]

https://deeplearning.mit.edu 2020

Pluribus: Six-Player No-Limit Texas Hold’em Poker
Brown et al. (CMU, Facebook AI)• Chris Ferguson: “Pluribus is a very hard opponent to play against. It’s really hard to pin
him down on any kind of hand. He’s also very good at making thin value bets on theriver. He’s very good at extracting value out of his good hands.”• Darren Elias: “Its major strength is its ability to use mixed strategies. That’s the samething that humans try to do. It’s a matter of execution for humans — to do this in aperfectly random way and to do so consistently. Most people just can’t.”
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[28]

https://deeplearning.mit.edu 2020

OpenAI Rubik’s Cube Manipulation• Deep RL: Reinforcement learning approach from OpenAI Five
• ADR: Automatic Domain Randomization (ADR) – generate progressively
more difficult environment as the system learns (alternative for self-play)• Capacity: Term of “emergent meta-learning” is used to described the factthat the network is constrained and the ADR process of environment
generation is not
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[20]

https://deeplearning.mit.edu 2020

Deep RL and Self-Play:

Hopes for 2020
• Robotics: Use of RL methods in manipulation andreal-world interaction tasks.
• Human Behavior: Use of multi-agent self-play to
explore naturally emerging social behaviors as away to study equivalent multi-human systems.
• Games: Use RL to assist human experts indiscovering new strategies at games and othertasks in simulation.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

The Lottery Ticket Hypothesis:
Finding Sparse, Trainable Neural Networks
Frankle et al. (MIT) - Best Paper at ICLR 20191.

Randomly initialize a neural network.

2.

Train the network until it converges.

3.

Prune a fraction of the network.

4.

Reset the weights of the remaining network to initialization values from step 1

5.

Train the pruned, untrained network. Observe convergence and accuracy.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[29, 30]

https://deeplearning.mit.edu 2020

The Lottery Ticket Hypothesis:
Finding Sparse, Trainable Neural Networks
Frankle et al. (MIT) - ICLR 2019 Best Paper• Idea: For every neural network, there is a subnetwork that can achieve thesame accuracy in isolation after training.
• Iterative pruning: Find this subset subset of nodes by iteratively training
network, pruning its smallest-magnitude weights, and re-initializing theremaining connections to their original values. Iterative vs one-shot is key.
• Inspiring takeaway: There exist architectures that are much more efficient.
Let’s find them!
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[29, 30, 31]https://deeplearning.mit.edu 2020

Challenging Common Assumptions in the Unsupervised
Learning of Disentangled Representations
Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper• The goal of disentangled representations is to build models that can capture
explanatory factors in a vector.
• The figure above presents a model with a 10-dimensional representation vector.
• Each of the 10 panels visualizes what information is captured in one of the 10
different coordinates of the representation.
• From the top right and the top middle panel we see that the model has successfully
disentangled floor color, while the two bottom left panels indicate that object colorand size are still entangled.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[32, 33]

https://deeplearning.mit.edu 2020

Challenging Common Assumptions in the Unsupervised
Learning of Disentangled Representations
Locatello et al. (ETH Zurich, Max Plank Institute, Google Research) - ICML 2019 Best Paper• Proof: Unsupervised learning of disentangled representations without inductive
biases is impossible.
• Takeaway: Inductive biases (assumptions) should be made explicit• Open problem: Finding good inductive biases for unsupervised model selection thatwork across multiple data sets persists is a key open problem.
• Open Experiments: Open source library with implementations of the considereddisentanglement methods and metrics, a standardized training and evaluation
protocol, as well as visualization tools to better understand trained models.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[32, 33]

https://deeplearning.mit.edu 2020

Deep Double Descent:
Where Bigger Models and More Data Hurt
Nakkiran et al. (Harvard, OpenAI)• Double Descent Phenomena: As we increase the number ofparameters in a neural network, the test error initiallydecreases, increases, and, just as the model is able to fit thetrain set, undergoes a second descent.
• Applicable to model size, training time, dataset size.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[36]

https://deeplearning.mit.edu 2020

Science of Deep Learning and Interesting Directions

Hopes for 2020
• Fundamentals: Exploring fundamentals of model
selection, training dynamics, and representation
characteristics with respect to architecture
characteristics.
• Graph Neural Networks: Exploring use of graphneural networks for combinatorial optimization,
recommendation systems, etc.
• Bayesian Deep Learning: Exploring Bayesianneural networks for estimating uncertainty andonline/incremental learning.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Level 2

Level 4

Human is Responsible

Machine is Responsible

WaymoOctober, 2018:

January, 2020:• On-road: 20 million miles• Simulation: 10 billion miles• Testing & Validation:
20,000 classes ofstructured tests• Initiated testing without asafety driver

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Tesla Autopilot

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[37]

https://deeplearning.mit.edu 2020

Tesla Autopilot

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[37]

https://deeplearning.mit.edu 2020

Active Learning Pipeline(aka Data Engine)
Neural Network(Version N)

Perform Task
(Perception, Prediction, Planning)

Discover Edge Case
Search for Others Like It
Annotate
Retrain Network
Neural Network(Version N+1)

Human helps annotate tricky situations
Human helps design data mining procedures
Human annotates

Collaborative Deep Learning(aka Software 2.0 Engineering)
Neural Network(Version N)

Data Engine
for Task 1

Data Engine
for Task 2

Data Engine
for Task 3

Retrain Network
Deploy Updated Neural Network

Neural Network(Version N+1)…

Data Engine
for Task 100

Vision vs Lidar
L2 vs L4
• Primarily: Vision Sensors + Deep Learning• Pros:••••Highest resolution information
Feasible to collect data at scale and learn
Roads are designed for human eyes
Cheap

Example L2 System:

Tesla Autopilot
2+ billion miles• Cons:• Needs a huge amount of data to be accurate• Less explainable• Driver must remain vigilant• Primarily: Lidar + Maps
• Pros:• Explainable, consistent• Accurate with less data

• Cons:• Less amenable to machine learning• Expensive (for now)
• Safety driver or teleoperation fallback
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

Example L4 System:Waymo
20+ million miles
https://deeplearning.mit.edu 2020

Open Questions for Tesla Autopilot• Deep learning question:
Problem Difficulty: How difficult is driving? How many edgecases does it have? Can it be learned from data?
• Perception (detection, intention modeling, trajectory prediction)• Action (in a game-theoretic setting)• Balancing enjoyability and safety• Human supervision of deep learning system question:
Vigilance: How good can Autopilot get before vigilance
decrements significantly?
• And … will this decrement nullify the safety benefits of automation?

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Open Questions for Waymo• When we have maps, lidar, and geo-fenced routes:
Problem Difficulty: How difficult is driving? How many edgecases does it have? Can it be learned from data?
• Perception (detection, intention modeling, trajectory prediction)• Action (in a game-theoretic setting)• Balancing enjoyability and safety• Simulation question:
How much can be learned from simulation?

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Autonomous Vehicles and AI-Assisted Driving

Hopes for 2020
• Applied deep learning innovation: Life-long learning,
active learning, multi-task learning• Over-the-air updates: More level 2 systems begin bothdata collection and over-the-air software updates.
• Public datasets of edge-cases: More publicly available
datasets of challenging cases.
• Simulators: Improvement of publicly available simulators(CARLA, NVIDIA DRIVE Constellation, Voyage Deepdrive)• Less hype: More balanced in-depth reporting (byjournalists and companies) on successes and challenges ofautonomous vehicle development.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

AI in Political Discourse: Andrew Yang
• First presidential candidate to discuss artificial
intelligence extensively as part of his platform• Proposals• Department: Create a new executive department –the Department of Technology – to work withprivate industry and Congressional leaders tomonitor technological developments, assess risks,and create new guidance.
• Focus on AI: The new Department would be basedin Silicon Valley and would initially be focused on
Artificial Intelligence.
• Companies: Create a public-private partnership
between leading tech firms and experts withingovernment to identify emerging threats andsuggest ways to mitigate those threats while
maximizing the benefit of technological innovation
to society.
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[27]

https://deeplearning.mit.edu 2020

American AI Initiative• In February, 2019, the president signed Executive
Order 13859 announcing the American AI Initiative• Goals• Investment in long-term research• Support research in academia and industry• Access to federal data
• Promote STEM education• Develop AI in “a manner consistent with our Nation’svalues, policies, and priorities.”• AI must also be developed in a way that does notcompromise our American values, civil liberties, or
freedoms.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Tech Leaders Testifying Before Congress(Ethics of Recommender Systems)

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

DeepMind + Google Research:

Play Store App Discovery• Candidate app generation:
LSTM → Transformer → efficient addition attention model• Candidate app unbiasing:• The model learns a bias that favors the apps that are shown – and thus installed– more often.
• To help correct for this bias, impression-to-install rate weighting is introduced.

• Multiple objectives: relevance, popularity, or personal

preferences
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020[35]

https://deeplearning.mit.edu 2020

Government, Politics, Policy

Hopes for 2020
• Less fear of AI: More balanced, informed
discussion on the impact of AI in society.
• Experts: Continued conversations by government
officials about AI, privacy, cybersecurity withexperts in academia and industry.

• Recommender system transparency: More open
discussion and publication behind recommendersystems used in industry.

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Online Deep Learning Courses• Deep Learning• Fast.ai: Practical Deep Learning for Coders• Jeremy Howard et al.

• Stanford CS231n: Convolutional Neural Networks for Visual Recognition• Stanford CS224n: Natural Language Processing with Deep Learning• Deeplearning.ai (Coursera): Deep Learning• Andrew Ng• Reinforcement Learning• David Silver: Introduction to Reinforcement Learning• OpenAI: Spinning Up in Deep RL

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Tutorials: Over 200 of the Best Machine Learning, NLP,
and Python Tutorials (by Robbie Allen)• Link: http://bit.ly/36skFE7• Topics• Machine learning• Activation and Loss Functions• Bias• Perceptron• Regression• Gradient descent• Generative learning• Support vector machines• Backpropagation• Deep Learning• Optimization

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020• Long Short Term Memory• Convolutional Neural Networks• Recurrent Neural Nets (RNNs)• Reinforcement Learning• Generative Adversarial Networks• Multi-task Learning• NLP• Word Vectors• Encoder-Decoder• TensorFlow• PyTorchhttps://deeplearning.mit.edu 2020

Deep Learning Books

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Outline• Deep Learning Growth, Celebrations, and Limitations• Deep Learning and Deep RL Frameworks• Natural Language Processing• Deep RL and Self-Play• Science of Deep Learning and Interesting Directions• Autonomous Vehicles and AI-Assisted Driving• Government, Politics, Policy• Courses, Tutorials, Books• General Hopes for 2020

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Deep Learning Growth, Celebrations, and Limitations

Hopes for 2020
• Reasoning• Active learning and life-long learning• Multi-modal and multi-task learning• Open-domain conversation• Applications: medical, autonomous vehicles• Algorithmic ethics• Robotics• Recommender systems

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020

Hope for 2020: Recipe for Progress (in AI)

Perseverance
Skepticism(Never Give Up)

Hard Work
Criticism
OpenMindedness“The future depends onsome graduate student
who is deeply suspiciousof everything I have said.”- Geoffrey Hinton
For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

Crazy

https://deeplearning.mit.edu 2020

Thank You

Videos and slides are posted on the website:

deeplearning.mit.edu

For the full list of references visit:
http://bit.ly/deeplearn-sota-2020

https://deeplearning.mit.edu 2020