Deep Learning for Computer Vision
MIT 6.S191
Ava Soleimany

Vision: Evolutionary Origins
Vision: 540 million years of data
vs.
Bipedal movement: 230+ million years of data
Human language: ~100 thousand years of data
Opabinia, 540 mil. years ago

Can we understand neural basis of vision?
How is visual information processed?
Can we use that structure to inform computer vision?
Hubel & Wiesel, Harvard. 1960s
6.S191 Introduction to Deep Learning
introtodeeplearning.com[1] 1/30/18

The Visual Cortex
1) Spatial Invariance2) Receptive Field
3) Hierarchy

6.S191 Introduction to Deep Learning
introtodeeplearning.com[2] 1/30/18

The Visual Cortex

6.S191 Introduction to Deep Learning
introtodeeplearning.com[3] 1/30/18

What Computers “See”

Images Are Numbers

6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/30/18

Images Are Numbers

6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/30/18

Images are Numbers
What the computer sees

An image is just a matrix of numbers [0,255]!
i.e., 1080x1080x3 for an RGB image
6.S191 Introduction to Deep Learning
introtodeeplearning.com[4] 1/30/18

Tasks in Computer Visionclassification

Input Image

Lincoln

0.8

Washington

0.1

Jefferson

0.05

Obama

0.05

Pixel Representation- Regression: output variable takes continuous value- Classification: output variable takes class label. Can produce probability of belonging to a par ticular class

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

High Level Feature Detection
Let’s identify key features in each image category

Nose,Eyes,
Mouth

Wheels,License Plate,Headlights
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Door,
Windows,
Steps
1/30/18

Manual Feature Extraction
Domain knowledge

Define features

Detect features
to classify

Problems?

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Manual Feature Extraction
Domain knowledge

Define features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Detect features
to classify[5] 1/30/18

Manual Feature Extraction
Domain knowledge

Define features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Detect features
to classify[5] 1/30/18

Learning Feature Representations
Can we learn hierarchy of features directly fromthe data instead of hand engineering?
Low level features

Mid level features

High level features

Edges, dark spots

Eyes, ears, nose

Facial structure

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6] 1/30/18

Learning Visual Features

Fully Connected Neural Network

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Fully Connected Neural Network
Input:• 2D image• Vector of pixel
valuesFully connected:• Connect neuron in hidden
layer to all neurons in
input layer• No spatial information!
• And many, many
parameters!

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Fully Connected Neural Network
Input:• 2D image• Vector of pixelvalues

Fully connected:• Connect neuron in hidden
layer to all neurons ininput layer• No spatial information!
• And many, manyparameters!

How can we use spatial structure in the input to inform the architecture of the network?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Using Spatial Structure

Idea: connect patches of input toneurons in hidden layer.

Input: 2D image.
Array of pixel values

Neuron connected to region ofinput. Only “sees” these values.

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Using Spatial Structure

Connect patch in input layer to a single neuron in subsequent layer.
Use a sliding window to define connections.
How can we weight the patch to detect particular features?
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Applying Filters to Extract Features
1) Apply a set of weights – a filter – to extract local features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Applying Filters to Extract Features
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Applying Filters to Extract Features
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter(features that matter in one par t of the input should matter elsewhere)

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Feature Extraction with Convolution- Filter of size 4x4 : 16 different weights- Apply this same filter to 4x4 patches in input- Shift by 2 pixels for next patch
This “patchy” operation is convolution
1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Feature Extraction and Convolution
A Case StudyX or X?

Image is represented as matrix of pixel values… and computers are literal!
We want to be able to classify an X as an X even if it’s shifted, shrunk, rotated, deformed.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[7] 1/30/18

Features of X

6.S191 Introduction to Deep Learning
introtodeeplearning.com[7] 1/30/18

Filters to Detect X Features
filters

6.S191 Introduction to Deep Learning
introtodeeplearning.com[7] 1/30/18

The Convolution Operation
1 X 1 =1= 9Element wise multiply
6.S191 Introduction to Deep Learning
introtodeeplearning.com

Add outputs[7] 1/30/18

The Convolution Operation
Suppose we want to compute the convolution of a 5x5 image and a 3x3 filter:

filterimage

We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs…
6.S191 Introduction to Deep Learning
introtodeeplearning.com[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learningintrotodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

The Convolution Operation
We slide the 3x3 filter over the input image, element-wise multiply, and add the outputs:

filter

6.S191 Introduction to Deep Learning
introtodeeplearning.com

feature map[8] 1/30/18

Producing Feature Maps

Original

SharpenEdge Detect

6.S191 Introduction to Deep Learning
introtodeeplearning.com“Strong” EdgeDetect

1/30/18

Feature Extraction with Convolution

1) Apply a set of weights – a filter – to extract local features
2) Use multiple filters to extract different features3) Spatially share parameters of each filter
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Convolutional Neural Networks (CNNs)

CNNs for Classification

1. Convolution: Apply filters with learned weights to generate feature maps.
2. Non-linearity: Often ReLU.
3. Pooling: Downsampling operation on each feature map.
Train model with image data.
Learn weights of filters in convolutional layers.
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Convolutional Layers: Local Connectivity
For a neuron in hidden layer:- Take inputs from patch the neuron “sees”- Compute weighted sum- Apply bias

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Convolutional Layers: Local Connectivity
For a neuron in hidden layer:- Take inputs from patch the neuron “sees”- Compute weighted sum- Apply bias

0

4x4 filter: matrix
of weights 𝜃&'0

( ( 𝜃&' 𝑥&*+,'*- + 𝑏&12 '12

applying a window of weights
computing linear combinations
activating with non-linear functionfor neuron (p,q) in hidden layer
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

CNNs: Spatial Arrangement of Output Volume
depthLayer Dimensions:ℎ𝑥𝑤𝑥𝑑
heightwhere h and w are spatial dimensions
d (depth) = number of filters

Stride:Filter step size

Receptive Field:
width

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Locations in input image thata node is path connected to[5] 1/30/18

Introducing Non-Linearity- Apply after every convolution operation (i.e., afterconvolutional layers)- ReLU: pixel-by-pixel operation that replaces all negativevalues by zero. Non-linear operation

Rectified Linear Unit (ReLU)

𝑔 𝑧 = max ( 0 , 𝑧 )6.S191 Introduction to Deep Learning
introtodeeplearning.com[8] 1/30/18

Pooling

1) Reduced dimensionality2) Spatial invariance

How else can we downsample and preserve spatial invariance?
6.S191 Introduction to Deep Learning
introtodeeplearning.com[5] 1/30/18

Representation Learning in Deep CNNs
Low level features

Mid level features

High level features

Edges, dark spots

Eyes, ears, nose

Facial structure

Conv Layer 1Conv Layer 2

Conv Layer 3

6.S191 Introduction to Deep Learning
introtodeeplearning.com[6] 1/30/18

CNNs for Classification: Feature Learning

1. Learn features in input image through convolution
2. Introduce non-linearity through activation function (real-world data is non-linear!)
3. Reduce dimensionality and preserve spatial invariance with pooling
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

CNNs for Classification: Class Probabilities-CONV and POOL layers output high-level features of input
Fully connected layer uses these features for classifying input imageExpress output as probability of image belonging to a particular class
6.S191 Introduction to Deep Learning
introtodeeplearning.com

𝑒 DE
softmax 𝑦& =∑' 𝑒 DG

1/30/18

CNNs: Training with Backpropagation

Learn weights for convolutional filters and fully connected layers
Backpropagation: cross-entropy loss
𝑱 𝜽 = ( 𝑦 (&) log 𝒚M(&)&6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

CNNs for Classification: ImageNetImageNet Dataset
Dataset of over 14 million images across 21,841 categories“Elongated crescent-shaped yellow fruit with soft sweet flesh”

1409 pictures of bananas.
6.S191 Introduction to Deep Learning
introtodeeplearning.com[9,10] 1/30/18

ImageNet Challenge

Classification task: produce a list of object categories present in image. 1000 categories.
“Top 5 error”: rate at which the model does not output correct label in top 5 predictions
Other tasks include:
single-object localization, object detection from video/image, scene classification, scene parsing
6.S191 Introduction to Deep Learning
introtodeeplearning.com[9,10] 1/30/18

ImageNet Challenge: Classification Task
28.2

25.8

2013: ZFNet- 8 layers, more filters

2016.4
2014: VGG
- 19 layers

11.7
10

6.7

5.1

3.57

anHum

15201420

1320

1220

1120

10020

classification error %30

2012: AlexNet. First CNN to win.
- 8 layers, 61 million parameters

6.S191 Introduction to Deep Learning
introtodeeplearning.com

2014: GoogLeNet- “Inception” modules- 22 layers, 5million parameters
2015: ResNet- 152 layers[9,10] 1/30/18

ImageNet Challenge: Classification Task
150

25.820

16.4
11.7

10

7.3

6.7

5.1

3.57
0

number of layers

28.2100501520142014201320122011202010anumH201520142014201320
12020112010classification error %30

6.S191 Introduction to Deep Learning
introtodeeplearning.com[9,10] 1/30/18

An Architecture for Many Applications

An Architecture for Many Applications

Different image classification domains
Object detection with R-CNNs
Segmentation with fully convolutional networks
Image captioning with RNNs
6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Beyond Classification
Semantic Segmentation

Object Detection

Image Captioning

CATCAT, DOG, DUCK

The cat is in the grass.

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Beyond Classification
Semantic Segmentation

Classification + Localization

Object Detection

Instance Segmentation

CATCAT

CAT, DOG, DUCK

CAT, DOG, DUCK

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Semantic Segmentation: FCNs
FCN: Fully Convolutional Network
Network designed with all convolutional layers,
with downsampling and upsampling operations

6.S191 Introduction to Deep Learning
introtodeeplearning.com[5,11,12] 1/30/18

Driving Scene Segmentation

Object Detection with R-CNNs
R-CNN: Find regions that we think have objects. Use CNN to classify.

6.S191 Introduction to Deep Learning
introtodeeplearning.com[13] 1/30/18

Image Captioning using RNNs

6.S191 Introduction to Deep Learning
introtodeeplearning.com[14,15] 1/30/18

Image Captioning using RNNs

6.S191 Introduction to Deep Learning
introtodeeplearning.com[14,15] 1/30/18

Visualizing Convolutional Neural Networks

Peering Inside CNNs

6.S191 Introduction to Deep Learning
introtodeeplearning.com[16] 1/30/18

Class Activation Maps (CAMs)

6.S191 Lab!

A class activation map (CAM) for a given classhighlights the image regions used by the CNN to identify that class

6.S191 Introduction to Deep Learning
introtodeeplearning.com[17] 1/30/18

Class Activation Maps (CAMs)

6.S191 Lab!

A class activation map (CAM) for a given classhighlights the image regions used by the CNN to identify that class

6.S191 Introduction to Deep Learning
introtodeeplearning.com[17] 1/30/18

Deep Learning for Computer Vision:
Impact and Summary

Data, Data, Data
Airplane
Automobile
BirdCatDeerDog

MNIST: handwritten digits

Frog
HorseShipImageNet:
22K categories. 14M images.

Truckplaces: natural scenes
6.S191 Introduction to Deep Learning
introtodeeplearning.com

CIFAR-10
1/30/18

Deep Learning for Computer Vision: Impact

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Impact: Face Recognition

6.S191 Introduction to Deep Learning
introtodeeplearning.com

1/30/18

Impact: Self-Driving Cars

[18]

Impact: Medicine
Dermatologist-level classification of skin cancer
with deep neural networks

6.S191 Introduction to Deep Learning
introtodeeplearning.com[19] 1/30/18

Deep Learning for Computer Vision: ReviewFoundationsCNNs• Why computer vision?
• Representing images• Convolutions for feature
extraction• CNN architecture• Application toclassification: ImageNet

6.S191 Introduction to Deep Learning
introtodeeplearning.com

Applications• Segmentation, object
detection, image
captioning• Visualization

1/30/18

References:https://goo.gl/ZuBkGx