Deep Learning for Computer Vision
MIT 6.S191

Ava Soleimany
January 29, 2019

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

What Computers “See”

Images are Numbers

6.S191 Introduction to Deep Learning
introtodeeplearning.com[1] 1/29/19

Images are Numbers

6.S191 Introduction to Deep Learning
introtodeeplearning.com[1] 1/29/19

Images are Numbers
What the computer sees

An image is just a matrix of numbers [0,255]!
i.e., 1080x1080x3 for an RGB image
6.S191 Introduction to Deep Learning
introtodeeplearning.com[1] 1/29/19

Tasks in Computer Visionclassification

Input Image

Lincoln

0.8

Washington

0.1

Jefferson

0.05

Obama

0.05

Pixel Representation- Regression: output variable takes continuous value- Classification: output variable takes class label. Can produce probability of belonging to a particular class

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

High Level Feature Detection
Let’s identify key features in each image category

Nose,
Eyes,
Mouth

Wheels,License Plate,Headlights
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Door,
Windows,
Steps
1/29/19

Manual Feature ExtractionDomain knowledge

Define features

Detect features
to classify

Problems?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Manual Feature Extraction
Domain knowledge

Define features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Detect features
to classify[2] 1/29/19

Manual Feature Extraction
Domain knowledge

Define features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Detect features
to classify[2] 1/29/19

Learning Feature Representations
Can we learn a hierarchy of features directly from the data
instead of hand engineering?
Low level features

Mid level features

High level features

Edges, dark spots

Eyes, ears, nose

Facial structure

6.S191 Introduction to Deep Learning
introtodeeplearning.com[3] 1/29/19

Learning Visual Features

Fully Connected Neural Network

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Fully Connected Neural Network
Input:• 2D image• Vector of pixel values

Fully Connected:• Connect neuron in hidden
layer to all neurons in input
layer• No spatial information!
• And many, many parameters!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Fully Connected Neural Network
Input:• 2D image• Vector of pixel values

Fully Connected:• Connect neuron in hidden
layer to all neurons in input
layer• No spatial information!
• And many, many parameters!

How can we use spatial structure in the input to inform the architecture of the network?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Using Spatial Structure

Idea: connect patches of input
to neurons in hidden layer.

Input: 2D image.
Array of pixel values

Neuron connected to region ofinput. Only “sees” these values.

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Using Spatial Structure

Connect patch in input layer to a single neuron in subsequent layer.
Use a sliding window to define connections.
How can we weight the patch to detect particular features?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Applying Filters to Extract Features
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter(features that matter in one part of the input should matter elsewhere)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Feature Extraction with Convolution- Filter of size 4x4 : 16 different weights- Apply this same filter to 4x4 patches in input- Shift by 2 pixels for next patch
This “patchy” operation is convolution
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Feature Extraction and Convolution
A Case StudyX or X?

Image is represented as matrix of pixel values… and computers are literal!
We want to be able to classify an X as an X even if it’s shifted, shrunk, rotated, deformed.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/29/19

Features of X

6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/29/19

Filters to Detect X Features
filters

6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/29/19

The Convolution Operation
1X

1=1element wise
multiply

add outputs= 96.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/29/19

The Convolution Operation
Suppose we want to compute the convolution of a 5x5 image and a 3x3 filter:

filterimage

We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs…
6.S191 Introduction to Deep Learning
introtodeeplearning.com[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learningintrotodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[5] 1/29/19

Producing Feature Maps

Original

Edge Detect

Sharpen6.S191 Introduction to Deep Learning
introtodeeplearning.com“Strong” Edge
Detect1/29/19

Feature Extraction with Convolution

1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Convolutional Neural Networks (CNNs)

CNNs for Classification

1. Convolution: Apply filters with learned weights to generate feature maps.
2. Non-linearity: Often ReLU.
3. Pooling: Downsampling operation on each feature map.
Train model with image data.
Learn weights of filters in convolutional layers.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Convolutional Layers: Local Connectivity

For a neuron in hidden layer:- Take inputs from patch- Compute weighted sum- Apply bias

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Convolutional Layers: Local Connectivity

For a neuron in hidden layer:- Take inputs from patch- Compute weighted sum- Apply bias

4x4 filter: matrixof weights !"#''$ $ !"# (")*,#), + .
"%& #%&for neuron (p,q) in hidden layer1) applying a window of weights
2) computing linear combinations
3) activating with non-linear function

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

CNNs: Spatial Arrangement of Output Volume
depthLayer Dimensions:ℎ"#"$heightwhere h and w are spatial dimensions
d (depth) = number of filters

Stride:Filter step size

Receptive Field:
width

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Locations in input image thata node is path connected to[3] 1/29/19

Introducing Non-Linearity- Apply after every convolution operation (i.e., afterconvolutional layers)- ReLU: pixel-by-pixel operation that replaces all negativevalues by zero. Non-linear operation!

Rectified Linear Unit (ReLU)! " = max ( 0 , " )6.S191 Introduction to Deep Learning
introtodeeplearning.com[5] 1/29/19

Pooling

1) Reduced dimensionality2) Spatial invariance

How else can we downsample and preserve spatial invariance?
6.S191 Introduction to Deep Learning
introtodeeplearning.com[3] 1/29/19

Representation Learning in Deep CNNs
Low level features

Mid level features

High level features

Edges, dark spots

Eyes, ears, nose

Facial structure

Conv Layer 1

Conv Layer 2

Conv Layer 3

6.S191 Introduction to Deep Learning
introtodeeplearning.com[3] 1/29/19

CNNs for Classification: Feature Learning

1. Learn features in input image through convolution
2. Introduce non-linearity through activation function (real-world data is non-linear!)
3. Reduce dimensionality and preserve spatial invariance with pooling
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

CNNs for Classification: Class Probabilities- CONV and POOL layers output high-level features of input+ ,softmax () =- Fully connected layer uses these features for classifying input image∑/ + ,0- Express output as probability of image belonging to a particular class
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

CNNs: Training with Backpropagation

Learn weights for convolutional filters and fully connected layers
Backpropagation: cross-entropy loss,(%)! " = $ & (%) log %6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

CNNs for Classification: ImageNetImageNet Dataset
Dataset of over 14 million images across 21,841 categories“Elongated crescent-shaped yellow fruit with soft sweet flesh”

1409 pictures of bananas.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[6,7] 1/29/19

ImageNet Challenge

Classification task: produce a list of object categories present in image. 1000 categories.
“Top 5 error”: rate at which the model does not output correct label in top 5 predictions
Other tasks include:
single-object localization, object detection from video/image, scene classification, scene parsing6.S191 Introduction to Deep Learning
introtodeeplearning.com[6,7] 1/29/19

ImageNet Challenge: Classification Task
28.2

25.820

16.4
11.7

10

6.7

5.1

3.57

an

2012: AlexNet. First CNN to win.
- 8 layers, 61 million parameters
2013: ZFNet- 8 layers, more filters
2014:VGG- 19 layers
2014: GoogLeNet- “Inception” modules- 22 layers, 5million parameters
2015: ResNet- 152 layers

Hum

1520

1420

1320

1220

1120

10020

classification error %30

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6,7] 1/29/19

ImageNet Challenge: Classification Task
150

25.820

16.4
11.7

10

7.3

6.7

5.1

3.57
0

number of layers

28.2

100501520

1420

1420

1320

1220

1120

20

10anum

H

20
1520
1420
1420
1320
12020
1120
10

classification error %30

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6,7] 1/29/19

An Architecture for Many Applications

An Architecture for Many Applications

Object detection with R-CNNs
Segmentation with fully convolutional networks
Image captioning with RNNs
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Beyond Classification
Semantic Segmentation

Object Detection

Image Captioning

CATCAT, DOG, DUCK

The cat is in the grass.

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Semantic Segmentation: FCNs
FCN: Fully Convolutional Network.
Network designed with all convolutional layers,
with downsampling and upsampling operations

6.S191 Introduction to Deep Learning
introtodeeplearning.com[3,8,9] 1/29/19

Driving Scene Segmentation

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Fix reference[10] 1/29/19

Driving Scene Segmentation

6.S191 Introduction to Deep Learningintrotodeeplearning.com[11, 12] 1/29/19

Object Detection with R-CNNs
R-CNN: Find regions that we think have objects. Use CNN to classify.

6.S191 Introduction to Deep Learning
introtodeeplearning.com[13] 1/29/19

Image Captioning using RNNs

6.S191 Introduction to Deep Learning
introtodeeplearning.com[14,15] 1/29/19

Image Captioning using RNNs

6.S191 Introduction to Deep Learning
introtodeeplearning.com[14,15] 1/29/19

Deep Learning for Computer Vision:
Impact and Summary

Data, Data, Data
Airplane
Automobile
BirdCatDeerDog
Frog

MNIST: handwritten digits

HorseImageNet:
22K categories. 14M images.

ShipTruck

CIFAR-10
6.S191 Introduction to Deep Learning
introtodeeplearning.com

places: natural scenes
1/29/19

Deep Learning for Computer Vision: Impact

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/29/19

Impact: Face Detection

6.S191 Introduction to Deep Learning
introtodeeplearning.com

6.S191 Lab!

1/29/19

Impact: Self-Driving Cars

6.S191 Introduction to Deep Learning
introtodeeplearning.com[16] 1/29/19

Impact: Healthcare
Identifying facial phenotypes of genetic disorders using deep learning
Gurovich et al., Nature Med. 20196.S191 Introduction to Deep Learning
introtodeeplearning.com[17] 1/29/19

Deep Learning for Computer Vision: SummaryFoundations• Why computer vision?
• Representing images• Convolutions forfeature extraction

CNNs• CNN architecture• Application toclassification• ImageNet

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Applications• Segmentation, object
detection, image
captioning• Visualization

1/29/19References
goo.gl/hbLkF6

End of Slides