DEEP LEARNING –A PERSONAL PERSPECTIVEUrs Muller
MIT 6.S191 Introduction To Deep Learning

NEURAL NETWORKMAGIC

Training dataRandom
networkarchitecture

Learning
algorithm

Excellent
results2

THE REAL MAGIC OF DEEP LEARNING

Let’s us solve problems wedon’t know how to program3

NEURAL NETWORKS RESEARCH AT
BELL LABS, HOLMDEL 1985 – 1995

4

BELL LABS BUILDING IN HOLMDELNow called Bell Works
Around 1990:
6,000 employees in Holmdel~300 in Research~30 in machine learning, including:
Larry Jackel, Yann LeCun, Leon Bottou, JohnDenker, Vladimir Vapnik, Yoshua Bengio,Hans Peter Graf, Patrice Simard, CorinnaCortes, and many others

5

ORIGINAL DATABASE ~300 DIGITS
Demonstrated need forlarge database forbenchmarking→ USPS database→ MNIST database (60,000digits)

6

USING PRIORKNOWLEDGE
What class is point “X”?
If representation is north, south, east, west,
then choose green
If better representation is elevation above
sea level choose red
Using prior knowledge beats requiring
tons of training datax

Example: use convolutions for OCR

7

CONVOLUTIONAL NEURAL NETWORK(CNN)

8LENET 1993
9HOW WE CAME TO VIEW LEARNING
Largely Vladimir Vapnik’s infuence

Choose the right structure – “Structural Risk Minimization”→ Bring prior knowledge to the learning machine

Capacity control – matching learning machine complexity to available data
→ Examine learning curves

10LEARNING CURVESError

If test error >> training error:
get more training examples ordecrease capacity
If test error ~ training error:
increase capacity

Test set error

Smart structure allows lowerrors with lower capacity
Training set error
Number of training examples
Cortes, Jackel, Wan-Ping Chiang: Limits on Learning Machine Accuracy Imposed by Data Quality
http://papers.nips.cc/paper/918-limits-on-learning-machine-accuracy-imposed-by-data-quality.pdf

11

PILOTNET LEARNING CURVE

12

THE (FRIENDLY) RECOGNIZER WAR
Who can read the digits best?

Year 1994
http://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=576879
13

RESULTS ON MNIST DATA SETClassification
error on test set

Year 1994

Larry Jackel: “In hindsight, the similar test errors are not so surprising since
all the researchers were using variants of the same learning principles(everyone talked to Vladimir and studied the learning curves).”
The Optimal Margin (SVM) was remarkable because, unlike theother top performers, it did not include knowledge about thegeometry of the problem.
14

Classification
error on test set

Inference time insec on a Sparc 10

Training time in days
on a Sparc 10
Year 1994
15

LESSONS LEARNED
Look at the data during all processing steps
Solid debugging tools are critical
Validate the training data
The work is experimental in nature
Work with real data

16

EPILOG
1995 Deployed Holmdel neural nets at Wachovia Bank – Eventually ourtechnology was processing 20% of the checks written in the US
1995 AT&T Bell Labs fractured
2002 AT&T Labs – mass layofs
2003 Bell Labs work led to new programs at DARPA –LAGR, Learning Locomotion, Deep Learning, Challenges
2012 Deep Learning becomes popular triggered by availability of data,
compute power and ready commercial applications

17

LEARNED AUTONOMY18

DRIVE-AV
A full self-driving car stack

DRIVE AVDRIVEWORKS SDKDRIVE OSDRIVE PX – AI CAR COMPUTER
19

Solve the hard and unsolved autonomousvehicles (AV) problems using learning
To provide enhanced safety throughalgorithmic diversity

NEW JERSEYTEAM GOAL

To create functionality that may not be
achievable otherwise, such as
perception-based turns or merges onto
a busy highway
Additional AV labs in California, Boulder,
Seattle, and Europe

20NVIDIA OFFICE AT BELL WORKSIN HOLMDEL, NJ
21

An ideal location forautonomous driving R&D
Private roads for easy
testing
Quick access to diverse
public roads
Space to store and work onthe cars22

New York CityCONVENIENTLOCATION
Close to New York City(45 miles)
Accessible by commuter
train, ferry, bus, and car

Local train

Ferry

Family friendly, yet neara major metropolitan
area
Close to popular beaches
along the Jersey Shore
23

WHERE DO RULES FAIL?
Hand crafted features and rules work wellwhen the problem domain is well understood
Simple rules, such as “follow the lane lines”
fail quickly→ Can you list every feature you look atwhile driving?
It is extremely hard to write down every single
rule required for safe driving – it is relatively
easy to collect lots of training data (just drive)24

LEARNING TO PREDICT THE PATH
nealgft
Le rkin ilRama

Record data from lots of humans
driving their cars:
 Sensor data
 Human driven pathnealht nggi
R rkima

Rear axle

Human driven path
Training
dataCar

CNN predictedpath

SensordataCNN

Error (training) signal
25

EXAMPLE ARCHITECTURE
Single frame input with 200x66 pixel
resolution
Input is from a single camera with a small
patch cropped and scaled from the full image

Output: Turn radius
Fully
connectedlayersConvolution
64@1x18
Convolution
64@3x20~250,000 neuron weights, not a huge network

Convolution
48@5x22

The network grows as we use more sensors
and handle harder tasks

Convolution
36@14x47
Convolution
24@31x98
Normalization
3@66x200
YUV Input
3@66x20026

https://www.youtube.com/watch?v=-96BEoXJMs0&t=39s
27

DRIVING WITH LEARNED PATH PREDICTION
28

LEARNED LANE CHANGES
29

LEARNED TURNS
https://www.youtube.com/watch?v=Sm-NBdSzP6E

30

VISUALIZATION
Where does the network
look?

31

VISUALIZATION
32

ATYPICAL VEHICLE CLASS

33

OPEN CHALLENGESDeal with ambiguous situations
There is often more than one correct answer
Learn from imperfect behavior
Several observations in the same situation, most are correct, some are notcorrect

34

DRIVING IN THE SNOW35

Thank You