1

Efficient Computing for
Deep Learning, AI and Robotics
Vivienne Sze ( @eems_mit)
Massachusetts Institute of Technology
In collaboration with Luca Carlone, Yu-Hsin Chen, Joel Emer, Sertac Karaman, Tushar Krishna,
Thomas Heldt, Trevor Henderson, Hsin-Yu Lai, Peter Li, Fangchang Ma, James Noraky, Gladynel
Saavedra Peña, Charlie Sodini, Amr Suleiman, Nellie Wu, Diana Wofk, Tien-Ju Yang, Zhengdong Zhang

Slides available athttps://tinyurl.com/SzeMITDL2020
Vivienne Sze (@eems_mit)2Compute Demands for Deep Neural Networks
AlexNet to AlphaGo Zero: A 300,000x Increase in Compute
Petaflop/s-days(exponential)

Year
Source: Open AI (https://openai.com/blog/ai-and-compute/)
Vivienne Sze (@eems_mit)

3Compute Demands for Deep Neural Networks[Strubell, ACL 2019][Strubell, ACL 2019]

Vivienne Sze (@eems_mit)

4

Processing at “Edge” instead of the “Cloud”

Communication

Vivienne Sze (@eems_mit)

Privacy

Latency5Computing Challenge for Self-Driving Cars

(Feb 2018)
Cameras and radar generate~6 gigabytes of data every 30 seconds.

Self-driving car prototypes use
approximately 2,500 Watts ofcomputing power.
Generates wasted heat and some
prototypes need water-cooling!
Vivienne Sze (@eems_mit)

6Existing Processors Consume Too Much Power< 1 Watt

Vivienne Sze (@eems_mit)> 10 Watts

7

Transistors are NOT Getting More Efficient
Slow down of Moore’s Law and Dennard Scaling
General purpose microprocessors not getting faster or more efficient• Need specialized hardware for significant improvement inspeed and energy efficiency• Redesign computing hardware from the ground up!

Slowdown

Vivienne Sze (@eems_mit)

8

Popularity of Specialized Hardware for DNNs

Big Bets On A.I. Open a New Frontier for
Chips Start-Ups, Too. (January 14, 2018)“Today, at least 45 start-ups are working
on chips that can power tasks like speechand self-driving cars, and at least five ofthem have raised more than $100 million
from investors. Venture capitalists
invested more than $1.5 billion in chip
start-ups last year, nearly doubling theinvestments made two years ago, according
to the research firm CB Insights.”

Vivienne Sze (@eems_mit)

9Power Dominated by Data Movement

Operation:

Energy

(pJ)

Area
(µm2)

8b Add0.03

36

16b Add

0.05

67

32b Add

0.1

137

16b FP Add

0.4

1360

32b FP Add

0.9

4184

8b Mult

0.2

282

32b Mult

3.1

3495

16b FP Mult

1.1

1640

32b FP Mult

3.7

7700

32b SRAM Read (8KB)

5N/A

32b DRAM Read

640

N/A

Relative Energy Cost

1

10 102 103

104

Relative Area Cost

1

10 102 103

Memory access is orders of magnitude higher energy than compute
Vivienne Sze (@eems_mit)[Horowitz, ISSCC 2014]10

Autonomous Navigation Uses a Lot of Data
• Semantic Understanding- High frame rate
- Large resolutions- Data expansion
2 million pixels• Geometric Understanding- Growing map size

Vivienne Sze (@eems_mit)[Pire, RAS 2017]

10x-100x more pixels

11

Understanding the Environment
Depth Estimation

Semantic Segmentation

Vivienne Sze (@eems_mit)

State-of-the-art approaches
use Deep Neural Networks,
which require up to several
hundred millions ofoperations and weights tocompute!
>100x more complex thanvideo compression

12

Deep Neural Networks
Deep Neural Networks (DNNs) have become a cornerstone of AI

Vivienne Sze (Computer Vision

Speech Recognition

Game Play

Medical@eems_mit)13

What Are Deep Neural Networks?
Low Level Features

High Level Features

Output:“Volvo XC90”

Input:
Image

Modified Image Source: [Lee, CACM 2011]

Vivienne Sze (@eems_mit)14

Weighted Sum
W11

Y1

X1

⎞
Nonlinear ⎛ 3
Y j = Activation
activation ⎜ Wij × Xi ⎟⎠Function ⎝ i=1∑

Sigmoid

Y2
X2

1

Rectified Linear Unit (ReLU)100
y=1/(1+e-x)

Y3

-1

X3

-1

0

y=max(0,x)
1

-1
-1

1

Image source: Caffe tutorial

W34

Y4

Output Layer

Input Layer
Hidden Layer

Key operation is multiply and accumulate (MAC)
Accounts for > 90% of computation
Vivienne Sze (0@eems_mit)

15

Popular Types of Layers in DNNs

• Fully Connected Layer– Feed forward, fully connectedFeedback

Feed
Forward– Multilayer Perceptron (MLP)• Convolutional Layer– Feed forward, sparsely-connected w/ weight sharing– Convolutional Neural Network (CNN)– Typically used for images• Recurrent Layer– Feedback

Output Layer

Input Layer
Hidden Layer

Fully
Connected

Sparsely
Connected– Recurrent Neural Network (RNN)– Typically used for sequential data (e.g., speech, language)• Attention Layer/Mechanism– Attention (matrix multiply) + feed forward, fully connected– Transformer [Vaswani, NeurIPS 2017]

Vivienne Sze (@eems_mit)

Output Layer
Input Layer
Hidden Layer

16

High-Dimensional Convolution in CNN
a plane of input activations
a.k.a. input feature map (fmap)

filter (weights)
H

R
S

Vivienne Sze (W@eems_mit)

17High-Dimensional Convolution in CNN
input fmap

output fmap
an output
activation

filter (weights)
H

R
S

E
W

Element-wiseMultiplication

Vivienne Sze (@eems_mit)

FPartial Sum (psum)
Accumulation

18

High-Dimensional Convolution in CNN
input fmap

output fmap
an output
activation

filter (weights)
H

R
S

E
W

Sliding Window Processing

Vivienne Sze (@eems_mit)F19

High-Dimensional Convolution in CNN…

output fmap

H

E
…

…

R

…

…

…

C

…

C

filter…

input fmap

S

W

Many Input Channels (C)AlexNet: 3 – 192 Channels (C)Vivienne Sze (@eems_mit)

FHigh-Dimensional Convolution in CNN

…

…

…

…

E
W

…

F

…

Many
Output Channels (M)

M

…

…

C

…

…

1

M
…

H
S

R

output fmap

…

…

C

R

C

…

input fmap

manyfilters (M)…

20

AlexNet: 96 – 384 Filters (M)S
Vivienne Sze (@eems_mit)

High-Dimensional Convolution in CNN
Many
Input fmaps (N)…

…

C

filters

Many
Output fmaps (N)

…

H

…

S

H

W

…

…

…
…

E
…

N
@eems_mit)

F

…

…

…

…

C

…
…

…

W

R

Vivienne Sze (

1

…

S

C

E

1

…

R

N

…

…

C

…

…

M

…

21

F

Image batch size: 1 – 256 (N)Define Shape for Each Layer
Input fmaps

Filters

…

…

M
E

…

1

N
W

…

…

… …

…

S

M
E

H

……

…

M

C

…

F

…

C

…

…

…

W

N

…

1

…

1

…

H
S

R

Output fmaps

…

…

C

R

C

…

22

F

H – Height of input fmap (activations)
W – Width of input fmap (activations)C – Number of 2-D input fmaps /filters(channels)
R – Height of 2-D filter (weights)
S – Width of 2-D filter (weights)
M – Number of 2-D output fmaps (channels)
E – Height of output fmap (activations)
F – Width of output fmap (activations)
N – Number of input fmaps/output fmaps(batch size)

Shape varies across layers

Vivienne Sze (@eems_mit)

23

Layers with Varying Shapes
MobileNetV3-Large Convolutional Layer Configurations
Filter Size (RxS)
3x3

3
3
3

1x1
3x3
1x1

6
6
6

1x1
5x5
1x1

# Filters (M)16# Channels (C)
3

…

Block
1

64
64
24

16
1
64

…
120
120
40

40
1
120

…
[Howard, ICCV 2019]
Vivienne Sze (@eems_mit)

24

Popular DNN Models

Metrics

LeNet-5

AlexNet

VGG-16

GoogLeNet(v1)

ResNet-50

EfficientNet-B4

Top-5 error(ImageNet)

n/a

16.4

7.4

6.7

5.3

3.7*

Input Size

28x28

227x227224x224224x224224x224380x380# of CONV Layers

2

5

16

21 (depth)

49

96# of Weights

2.6k

2.3M

14.7M

6.0M

23.5M

14M

# of MACs

283k

666M

15.3G

1.43G

3.86G

4.4G

# of FC layers

2

3

3

1

1

65**

# of Weights

58k

58.6M

124M

1M

2M

4.9M

# of MACs

58k

58.6M

124M

1M

2M

4.9M

Total Weights

60k

61M

138M

7M

25.5M

19M

Total MACs

341k

724M

15.5G

1.43G

3.9G

4.4G

Reference

Lecun,PIEEE 1998

Krizhevsky,
NeurIPS 2012

Simonyan,
ICLR 2015

Szegedy,
CVPR 2015

He,
CVPR 2016

Tan,
ICML 2019

DNN models getting larger and deeper* Does not include multi-crop and ensemble** Increase in FC layers due to squeeze-and-excitation layers (much smaller than FC layers for classification)

Vivienne Sze (@eems_mit)

25

Efficient Hardware Acceleration
for Deep Neural Networks

Vivienne Sze (@eems_mit)

26

Properties We Can Leverage• Operations exhibit high parallelism
à high throughput possible• Memory Access is the Bottleneck
Memory Read
DRAM

MAC*

filter weight
image pixel
partial sum

200x

Memory Write

ALU
updatedpartial sum
1x

DRAM

* multiply-and-accumulate

Worst Case: all memory R/W are DRAM accesses• Example:
Vivienne Sze (@eems_mit)

AlexNet has 724M MACs
à 2896M DRAM accesses required

27

Properties We Can Leverage• Operations exhibit high parallelism
à high throughput possible• Input data reuse opportunities (up to 500x)

FilterInput Fmap

Filters

Input Fmap

Input Fmaps

Filter1

1
2
Convolutional Reuse(Activations, Weights)
CONV layers only
(sliding window)
Vivienne Sze (@eems_mit)

Fmap Reuse(Activations)
CONV and FC layers

2
Filter Reuse(Weights)
CONV and FC layers(batch size > 1)

28

Exploit Data Reuse at Low-Cost Memories
DRAM

GlobalBuffer

PE

PE

PE

ALU

Specializedwith
Reg 0.5
File– 1.0hardwarekB
small (< 1kB)
fetch data to low
run cost memory
Controla MAC
here near compute

Normalized Energy Cost*
ALU

1× (Reference)

ALU

1×

NoC: 200 – 1000 PEs PE

ALU

2×

100 – 500 kB Buffer

ALU

DRAM

ALU

0.5 – 1.0 kB RF

6×
200×

* measured from a commercial 65nm process
Vivienne Sze (Fartherand larger memories consume more power@eems_mit)29

Weight Stationary (WS)Global Buffer
Activation

Psum
W0

W1

W2

W3

W4

W5

W6

W7 PE

Weight• Minimize weight read energy consumption− maximize convolutional and filter reuse of weights• Broadcast activations and accumulate partial sums
spatially across the PE array• Examples: TPU [Jouppi, ISCA 2017], NVDLA
Vivienne Sze (@eems_mit)[Chen, ISCA 2016]

30

Output Stationary (OS)Global Buffer
WeightActivation
P0

P1

P2

P3

P4

P5

P6

P7 PE

Psum

• Minimize partial sum R/W energy consumption− maximize local accumulation• Broadcast/Multicast filter weights and reuse activations
spatially across the PE array• Examples: [Moons, VLSI 2016], [Thinker, VLSI 2017]
Vivienne Sze (@eems_mit)[Chen, ISCA 2016]

31

Input Stationary (IS)Global Buffer

Weight

Psum
I0

I1

I2

I3

I4

Act

I5

I6

I7

PE

• Minimize activation read energy consumption− maximize convolutional and fmap reuse of activations• Unicast weights and accumulate partial sums spatiallyacross the PE array• Example: [SCNN, ISCA 2017]
Vivienne Sze (@eems_mit)[Chen, ISCA 2016]

32

Row Stationary Dataflow
Row 1
PE 1

Row 1 *

Row 1

• Maximize row
convolutional reuse in RF− Keep a filter row and fmap
sliding window in RF• Maximize row psum
accumulation in RF*
Vivienne Sze (=@eems_mit)[Chen, ISCA 2016] Select for Micro Top Picks

33

Row Stationary Dataflow
Row 1

Row 2

Row 3

PE 1

Row 1 *

Row 1

PE 4

Row 1 *

Row 2

PE 2

Row 2 *

Row 2

PE 7

Row 1 *

Row 3

PE 5

Row 2 *

Row 3

PE 3

PE 8

Row 2 *

Row 4

PE 6

PE 9

Row 3 *Row 3

Row 3 *Row 4

Row 3 *

Row 5

*

=

*

=

*

=

Vivienne Sze (Optimize for overall energy efficiency insteadfor only a certain data type@eems_mit)[Chen, ISCA 2016] Select for Micro Top Picks

34

Dataflow Comparison: CONV Layers
2
1.5

Normalized
Energy/MAC

psums
weights

1

pixels
0.5
0

WS

OSA

OSB

OSC

NLR

RS

CNN Dataflows
RS optimizes for the best overall energy efficiency
Vivienne Sze (@eems_mit)[Chen, ISCA 2016]

35

Exploit Sparsity
Method 1. Skip memory access and computation

Zero Data Skipping

No R/WNo Switching

RegisterPad
File
Scratch

== 0

Zero
Buff

Enable 45% power reduction

Method 2. Compress data to reduce storage and data movement
DRAM
Access(MB)

DRAM Access (MB)Uncompressed

66
5

44

1.4×

Compressed

1.7×

1.8×

3

1.9×

Uncompressed
Fmaps + Weights

55

RLE Compressed
Fmaps + Weights

22
1

00

Vivienne Sze (

1.2×

11
22
33
44
AlexNet Conv Layer
AlexNet Conv Layer@eems_mit)[Chen, ISSCC 2016]

36

Eyeriss: Deep Neural Network Accelerator

On-chip Buffer

4mm

4mm

SpatialPE Array[Chen, ISSCC 2016]

Exploits data reuse for 100x reduction in memory accesses from global
buffer and 1400x reduction in memory accesses from off-chip DRAM

Overall >10x energy reduction compared to a mobile GPU (Nvidia TK1)
Eyeriss Project Website: http://eyeriss.mit.edu
Vivienne Sze (@eems_mit)[Joint work with Joel Emer]

Results for AlexNet

37

Features: Energy vs. Accuracy
Exponential

10000

VGG162

1000

AlexNet2

Energy/100
Pixel (nJ)10

Measured in 65nm*
4mm

1 [Suleiman, VLSI 2016]

On-chip Buffer

4mm

4mm

4mm

Vivienne Sze (1

SpatialPE Array@eems_mit)

HOG1

Linear0.1

2 [Chen, ISSCC 2016]* Only feature extraction. Does
not include data, classificationenergy, augmentation andensemble, etc.

Video
Compression

0

20

40

60

Accuracy (Average Precision)
Measured in on VOC 2007 Dataset
1. DPM v5 [Girshick, 2012]
2. Fast R-CNN [Girshick, CVPR 2015][Suleiman*, Chen*, ISCAS 2017]

80

38

Energy-Efficient Processing of DNNs
A significant amount of algorithm and hardware researchon energy-efficient processing of DNNs

http://eyeriss.mit.edu/tutorial.html

V. Sze, Y.-H. Chen,
T-J. Yang, J. Emer,“Efficient Processing of
Deep Neural Networks:
A Tutorial and Survey,”
Proceedings of the IEEE,
Dec. 2017
Book Coming
Spring 2020!

We identified various limitations to existing approaches
Vivienne Sze (@eems_mit)39

Design of Efficient DNN Algorithms• Popular efficient DNN algorithm approaches
Network Pruning
before pruning

Efficient Network Architectures

after pruning

pruning
synapsesRR

pruning
neurons

C

S

S

1

Examples: SqueezeNet, MobileNet... also reduced precision• Focus on reducing number of MACs and weights• Does it translate to energy savings and reduced latency?
Vivienne Sze (@eems_mit)[Chen*, Yang*, SysML 2018]

1

1

C

40

Data Movement is Expensive
DRAM

GlobalBuffer

PE

PE

PE

ALU

fetch data to run
a MAC here

Normalized Energy Cost*
ALU

1× (Reference)

ALU

1×

NoC: 200 – 1000 PEs PE

ALU

2×

100 – 500 kB Buffer

ALU

DRAM

ALU

0.5 – 1.0 kB RF

6×
200×

* measured from a commercial 65nm process

Energy@eems_mit)
of weight depends on memory hierarchy and dataflow

Vivienne Sze (41Energy-Evaluation Methodology

DNN Shape Configuration(# of channels, # of filters, etc.)

Hardware Energy Costs of each
MAC and Memory Access# acc. at mem. level 1# acc. at mem. level 2# of MACs
Calculation# of MACs…

MemoryAccesses
Optimization[0.3, 0, -0.4, 0.7, 0, 0, 0.1, …]

Tool available at: https://energyestimation.mit.edu/@eems_mit)

Ecomp

EnergyDNN Weights and Input Data

Vivienne Sze (Edata# acc. at mem. level n[Yang, CVPR 2017]

L1 L2 L3 …
DNN Energy Consumption

42

Key Observations• Number of weights alone is not a good metric for energy• All data types should be considered
Computation
10%

Energy Consumptionof GoogLeNet

Input Feature Map
25%

Weights22%

Output Feature Map
43%

Vivienne Sze (@eems_mit)[Yang, CVPR 2017]

43

Energy-Aware Pruning
Directly target energy andincorporate it into the
optimization of DNNs toprovide greater energy savings

Normalized Energy (AlexNet)
4.5

x109

4
3.5
3

• Sort layers based on energy andprune layers that consume mostenergy first

2.5• EAP reduces AlexNet energy by
3.7x and outperforms theprevious work that uses
magnitude-based pruning by 1.7x

1

2.1x

3.7x

2
1.5
0.5
0
Ori.

Magnitude
DC
Based Pruning

EnergyAware
EAP
Pruning

Pruned models available at
http://eyeriss.mit.edu/energy.html
Vivienne Sze (@eems_mit)[Yang, CVPR 2017]

# of Operations vs. Latency

44•# of operations (MACs) does not approximate latency well

Source: Google (https://ai.googleblog.com/2018/04/introducing-cvpr-2018-on-device-visual.html)
Vivienne Sze (@eems_mit)

45

NetAdapt: Platform-Aware DNN Adaptation• Automatically adapt DNN to a mobile platform to reach atarget latency or energy budget• Use empirical measurements to guide optimization (avoid
modeling of tool chain or platform architecture)
BudgetEnergy10.5

MetricProposal A…

Proposal Z

Latency

15.6

…

14.3

Energy

41

…

3.8

…

Latency…

Budget…

MetricPlatform

Empirical Measurements…

Pretrained
Network…

46

NetAdapt

MeasureA

Network Proposals
B
C
D

Z

…
AdaptedNetwork
Vivienne Sze (Code available at http://netadapt.mit.edu@eems_mit)[Yang, ECCV 2018]

In collaboration with Google’s Mobile Vision Team46

Simplified Example of One Iteration
1. Input

2. Meet Budget

3. Maximize
Accuracy

4. Output

Layer 1
100ms

90ms

80ms

Acc: 60%

Network from
Previous Iteration

Network for
Next Iteration

Selected

Latency: 100ms
Budget: 80ms

100ms

80ms

Selected

Vivienne Sze (@eems_mit)[Yang, ECCV 2018]…

…
Layer 4

Selected

Acc: 40%

Latency: 80ms
Budget: 60ms

47

Improved Latency vs. Accuracy Tradeoff• NetAdapt boosts the real inference speed of MobileNetby up to 1.7x with higher accuracy+0.3% accuracy
1.7x faster+0.3% accuracy
1.6x faster*Tested on the ImageNet dataset and a Google Pixel 1 CPU
Reference:
MobileNet: Howard et al, “Mobilenets: Efficient convolutional neural networks for mobile vision applications”, arXiv 2017
MorphNet: Gordon et al., “Morphnet: Fast & simple resource-constrained structure learning of deep networks”, CVPR 2018

Vivienne Sze (@eems_mit)[Yang, ECCV 2018]

48

FastDepth: Fast Monocular Depth Estimation
Depth estimation from a single RGB image desirable, due tothe relatively low cost and size of monocular cameras.
RGB

Prediction

Auto Encoder DNN Architecture (Dense Output)

Reduction(similar to classification)
Vivienne Sze (@eems_mit)

Expansion[Joint work with Sertac Karaman]

49

FastDepth: Fast Monocular Depth Estimation
Apply NetAdapt, compact network design, and depth wise decomposition
to decoder layer to enable depth estimation at high frame rates on anembedded platform while still maintaining accuracy> 10x~40fps onan iPhone

Configuration: Batch size of one (32-bit float)

Models available at http://fastdepth.mit.edu
Vivienne Sze (@eems_mit)[Wofk*, Ma*, ICRA 2019]

50

Many Efficient DNN Design Approaches
Network Pruning
before pruning

Efficient Network Architectures

after pruning

ChannelGroups G

pruning
synapsesR

pruning
neurons

R
C
S

Reduce Precision…

Convolutional
Layer

S
Depth-Wise
Layer

1

1

1

Point-Wise
Layer

32-bit float 10100101000000000101000000000100
8-bit fixed 0 1 1 0 0 1 1 0
BinaryVivienne Sze (0@eems_mit)

No guarantee that DNN algorithm
designer will use a given approach.
Need flexible hardware!
[Chen*, Yang*, SysML 2018]

C

51

Existing DNN Architectures• Specialized DNN hardware often rely on certain properties of
DNN in order to achieve high energy-efficiency• Example: Reduce memory access by amortizing across MAC array
Activation
MemoryWeightreuse
WeightMemory

MAC array

Activation
reuse
Vivienne Sze (@eems_mit)

52

Limitation of Existing DNN Architectures• Example: Reuse and array utilization depends on # of channels,feature map/batch size
– Not efficient across all network architectures (e.g., compact DNNs)

Number of
input channels

Number of filters(output channels)

Vivienne Sze (MAC array(spatial
accumulation)@eems_mit)

feature mapor batch size

Number of filters(output channels)

MAC array(temporal
accumulation)

53

Limitation of Existing DNN Architectures• Example: Reuse and array utilization depends on # of channels,feature map/batch size
– Not efficient across all network architectures (e.g., compact DNNs)
Example mapping for
depth wise layer
Number of
input channels

Number of filters(output channels)

Vivienne Sze (MAC array(spatial
accumulation)@eems_mit)

1

R
S

1

C

1

Number of filters(output channels)feature mapor batch size

MAC array(temporal
accumulation)

54

Limitation of Existing DNN Architectures• Example: Reuse and array utilization depends on # of channels,feature map/batch size
– Not efficient across all network architectures (e.g., compact DNNs)– Less efficient as array scales up in size
– Can be challenging to exploit sparsity

Number of
input channels

Number of filters(output channels)

Vivienne Sze (MAC array(spatial
accumulation)@eems_mit)

feature mapor batch size

Number of filters(output channels)

MAC array(temporal
accumulation)

55

Need Flexible Dataflow• Use flexible dataflow (Row Stationary) to exploit reuse in any
dimension of DNN to increase energy efficiency and arrayutilization

Example: Depth-wise layer
Vivienne Sze (@eems_mit)Need Flexible NoC for Varying Reuse

56• When reuse available, need multicast to exploit spatial datareuse for energy efficiency and high array utilization• When reuse not available, need unicast for high BW for weights
for FC and weights & activations for high PE utilization• An all-to-all satisfies above but too expensive and not scalable

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

Unicast Networks

Vivienne Sze (@eems_mit)

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

1D Systolic Networks

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

1D Multicast Networks[Chen, JETCAS 2019]

Global BufferPE

Low Bandwidth, High Spatial Reuse

Global BufferPE

Global Buffer

Global BufferHigh Bandwidth, Low Spatial Reuse

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

PE

Broadcast Network

57

Hierarchical Mesh…

RouterCluster

…

…

Mesh

GLB
Cluster…

PE
Cluster…

All-to-all
Network…

…

Mesh
Network(a)

High Bandwidth(b)Vivienne Sze (@eems_mit)

All-to-All(b)High Reuse

Grouped Multicast

Interleaved Multicast

(c)

(d)

(e)

[Chen, JETCAS 2019]

58

Eyeriss v2: Balancing Flexibility and Efficiency
Efficiently supports

12.6
10.9
5.6

• Wide range of filter shapes– Large and Compact• Different Layers– CONV, FC, depth wise, etc.

• Wide range of sparsity– Dense and Sparse• Scalable architecture

Speed up over Eyeriss v1 scales with number of PEs# of PEs

256

1024

16384

AlexNet

17.9x

71.5x

1086.7x

GoogLeNet

10.4x

37.8x

448.8xMobileNet

15.7x

57.9x

873.0xOver an order of magnitude faster andmore energy efficient than Eyeriss v1
Vivienne Sze (@eems_mit)[Chen, JETCAS 2019][Joint work with Joel Emer]

59

Looking Beyond the DNN
Accelerator for Acceleration

Vivienne Sze (@eems_mit)

60Super-Resolution on Mobile Devices
Low
Resolution
Streaming

High
Resolution
Playback

Transmit low resolution for lower bandwidth

Screens are getting larger

Use super-resolution to improve the viewing experience oflower-resolution content (reduce communication bandwidth)
Vivienne Sze (@eems_mit)

61

FAST: A Framework to Accelerate SuperRes

SR algorithm

FAST

Compressed videoSR
15x faster

Real-time

A framework that accelerates any SR algorithm by up to
15x when running on compressed videos
Vivienne Sze (@eems_mit)[Zhang, CVPRW 2017]

62

Free Information in Compressed Videos

DecodeCompressed video

PixelsVideo as a stack of pixels

Block-structure

Motion-compensation

Representation in compressed video

This representation can help accelerate super-resolution
Vivienne Sze (@eems_mit)[Zhang, CVPRW 2017]

63

Transfer is LightweightTransfer

SR

SR
SR
SR
SRSR

Low-res video

Low-res video
High-res video

High-res video

Transfer allows SR to run on only a subset of frames

Fractional
Bicubic
Interpolation Interpolation

Skip Flag

The complexity of the transfer is comparable to bicubic interpolation.
Transfer N frames, accelerate by N
Vivienne Sze (@eems_mit)[Zhang, CVPRW 2017]

64

Evaluation: Accelerating SRCNN

Vivienne Sze (@eems_mit)[Zhang, CVPRW 2017]

65

Visual Evaluation

SRCNN

FAST +SRCNN

Bicubic

Look beyond the DNN accelerator for opportunities to accelerate
DNN processing (e.g., structure of data and temporal correlation)
Code released at www.rle.mit.edu/eems/fast
Vivienne Sze (@eems_mit)[Zhang, CVPRW 2017]

66

Beyond Deep Neural
Networks

Vivienne Sze (@eems_mit)

67

Visual-Inertial Localization
Determines location/orientation of robot from images and IMU…

Localization

Image sequence
IMU

Inertial Measurement Unit

Visual-Inertial
Odometry(VIO) *

*Subset of SLAM algorithm(Simultaneous Localization And Mapping)
Vivienne Sze (@eems_mit)

Mapping

Slide 28

68

Localization at Under 25 mW

First chip that performs
complete Visual-Inertial Odometry
Front-End for camera(Feature detection, tracking, andoutlier elimination)

Front-End for IMU(pre-integration of accelerometerand gyroscope data)

Back-End Optimization of Pose
Graph

NavionConsumes 684× and 1582×
less energy thanmobile and desktop CPUs,
respectively
Navion Project Website: http://navion.mit.edu
Vivienne Sze (@eems_mit)[Zhang et al., RSS 2017], [Suleiman et al., VLSI 2018][Joint work with Sertac Karaman]

69

Key Methods to Reduce Data Size
Navion: Fully integrated system – no off-chip processing or storage
Vision Frontend (VFE)Line Buffers

Apply Low
Cost
Frame
Compression

FeatureDetection(FD)

Undistort& Rectify
(UR)

Left
Frame

Backend (BE)Previous
Frame

Current
Frame

Undistort& Rectify(UR)

FeatureTracking
(FT)

Right
Frame

Backend Control
Data & Control Bus
Floating
Point
Arithmetic

Build
Graph

MatrixOperations

Linearize

CholeskySparse Stereo (SS)Back
SubstituteVision Frontend Control

Rodrigues
Operations

Data & Control BusRANSAC

Fixed Point
Arithmetic

LinearSolverMarginal
Retract

IMU Frontend (IFE)
Point Cloud

Floating Point
Arithmetic

Pre-Integration

Graph
LinearSolverHorizonStatesSharedMemoryExploitSparsity inGraph andLinear Solver

RegisterFile
IMU
memoryUse compression and exploit sparsity to
reduce memory down to 854kB
Vivienne Sze (@eems_mit)[Suleiman, VLSI-C 2018] Best Student Paper Award

70

Where to Go Next: Planning and Mapping

Robot Exploration: Decide where to go by computing Shannon Mutual Information
Select candidate
scan locations

Compute Shannon MI andchoose best location

Where to scan?

Mutual Information

Occupancy map

Exploration with a mini
race car using motion
capture for localization

Vivienne Sze (@eems_mit)

Move tolocationand scanUpdateOccupancy
Map
Updated Map

Mutual information map

Occupancy map withplanned path

MI surface[Zhang, ICRA 2019]

71

Challenge is Data Delivery to All Cores
Process multiple beams in parallel
Core N
Core 1
Core 3

Core 2

Core 2
Core 1

Core N

Data delivery from memory is limited
Read Port 1

Core 1
Core 2

Read Port 2

Core N
Vivienne Sze (@eems_mit)

Specialized Memory Architecture

72

Break up map into separate memory banks and novel storage pattern tominimize read conflicts when processing different beams in parallel.
Memory Access Pattern

Diagonal Banking Pattern
8

Y

X

Y

Y

8

7

7

6

6

5

5

8
7

Bank 2

5

4

3

3

3

2

2

3

4

1

2

3

4

Bank 34

8
5

6

Bank 4
Bank 5

7

Bank 6
5

X

X

Bank 1

6

4

Bank 06

7

8Bank 7[Li, RSS 2019]

Compute the mutual information for an entire map of 20m x 20m at 0.1m resolutionin under a second à a 100x speed up versus CPU for 1/10th of the power.
Vivienne Sze (@eems_mit)[Joint work with Sertac Karaman]

73

Monitoring Neurodegenerative Disorders
Dementia affects 50 million people worldwide today(75 million in 10 years) [World Alzheimer’s Report]
Mini-Mental
State Examination (MMSE)
Q1. What is the year? Season? Date?
Q2. Where are you now? State? Floor?
Q3. Could you count backward from100 by sevens? (93, 86, …)

Clock-drawing test

Agrell et al.
Age and Ageing, 1998.

• Neuropsychological assessments are time consuming and require a
trained specialist• Repeat medical assessments are sparse, mostly qualitative, andsuffer from high retest variability
Vivienne Sze (@eems_mit)[Joint work with Thomas Heldt and Charlie Sodini]

74

Use Eye Movements for Quantitative Evaluation
Eye movements can be used to quantitatively evaluate severity,
progression or regression of neurodegenerative diseases
High-speed camera

Substantial head support

IR illumination

Phantom v25-11

SR EYELINK 1000 PLUS

Reulen et al., Med. & Biol. Eng. &Comp, 1988.

Clinical measurements of saccade latency are done in constrained
environments that rely on specialized, costly equipment.
Vivienne Sze (@eems_mit)

75

Measure Eye Movements Using Phone
SmartphoneCount

Eye movements

Eye movement
featureDevelop algorithm to measure eye
movement using a consumer-grade
camera rather than high-costresearch-grade camera.

iPhone 6(< $1k)Enable low-cost in-homelongitudinal measurements.
Reaction Time (milliseconds)
Vivienne Sze (@eems_mit)[Saavedra Peña, EMBC 2018] [Lai, ICIP 2018]

Phantom($100k)

76

Looking For Volunteers for Eye Reaction TimeIf you are near or on
MIT Campus and interested
in volunteering your eye
movements for this study,
please contact us atvolunteer-eye-movement@mit.edu

Vivienne Sze (@eems_mit)

77

Low Power 3D Time of Flight Imaging• Pulsed Time of Flight: Measure distance using round trip timeof laser light for each image pixel– Illumination + Imager Power: 2.5 – 20 W for range from 1 - 8 m• Use computer vision techniques and passive images toestimate changes in depth without turning on laser– CMOS Imaging Sensor Power: < 350 mW

Estimated Depth Maps
Real-time Performance on Embedded Processor
VGA @ 30 fps on Cortex-A7 (< 0.5W active power)
Vivienne Sze (@eems_mit)[Noraky, ICIP 2017]

78

Results of Low Power Depth ToF Imaging

RGB Image

Depth Map
Ground Truth

Depth Map
Estimated

Mean Relative Error: 0.7%
Duty Cycle (on-time of laser): 11%
Vivienne Sze (@eems_mit)[Noraky, ICIP 2017]

79

Summary• Efficient computing extends the reach of AI beyond
the cloud by reducing communication requirements,
enabling privacy, and providing low latency so that
AI can be used in wide range of applications ranging
from robotics to health care.
• Cross-layer design with specialized hardware
enables energy-efficient AI, and will be critical to theprogress of AI over the next decade.
Today’s slides available athttps://tinyurl.com/SzeMITDL2020

Vivienne Sze (@eems_mit)

80

Additional Resources
Overview Paper
V. Sze, Y.-H. Chen, T-J. Yang, J. Emer,“Efficient Processing of Deep Neural
Networks: A Tutorial and Survey,”
Proceedings of the IEEE, Dec. 2017
Book Coming Spring 2020!
More info about
Tutorial on DNN Architectures
http://eyeriss.mit.edu/tutorial.html

Vivienne Sze (@eems_mit)

For updates
EEMS Mailing List

81

Additional Resources

MIT Professional Education Course on“Designing Efficient Deep Learning Systems”
http://shortprograms.mit.edu/dls
Next Offering: July 20-21, 2020 on MIT Campus
Vivienne Sze (@eems_mit)

82

Additional Resources
Talks and Tutorial Available Online
https://www.rle.mit.edu/eems/publications/tutorials/

YouTube Channel
EEMS Group – PI: Vivienne Sze
Vivienne Sze (@eems_mit)

83

Acknowledgements

Joel Emer

Sertac Karaman

Thomas Heldt

Research conducted in the MIT Energy-Efficient Multimedia Systems Group would not
be possible without the support of the following organizations:

Vivienne Sze (@eems_mit)

Mailing List: http://mailman.mit.edu/mailman/listinfo/eems-news

84

References• Energy-Efficient Hardware for Deep Neural Networks– Project website: http://eyeriss.mit.edu– Y.-H. Chen, T. Krishna, J. Emer, V. Sze, “Eyeriss: An Energy-Efficient Reconfigurable Accelerator for Deep
Convolutional Neural Networks,” IEEE Journal of Solid State Circuits (JSSC), ISSCC Special Issue, Vol. 52,
No. 1, pp. 127-138, January 2017.
– Y.-H. Chen, J. Emer, V. Sze, “Eyeriss: A Spatial Architecture for Energy-Efficient Dataflow for
Convolutional Neural Networks,” International Symposium on Computer Architecture (ISCA), pp. 367379, June 2016.
– Y.-H. Chen, T.-J. Yang, J. Emer, V. Sze, “Eyeriss v2: A Flexible Accelerator for Emerging Deep Neural
Networks on Mobile Devices,” IEEE Journal on Emerging and Selected Topics in Circuits and Systems(JETCAS), June 2019.
– Eyexam: https://arxiv.org/abs/1807.07928• Limitations of Existing Efficient DNN Approaches– Y.-H. Chen*, T.-J. Yang*, J. Emer, V. Sze, “Understanding the Limitations of Existing Energy-Efficient
Design Approaches for Deep Neural Networks,” SysML Conference, February 2018.
– V. Sze, Y.-H. Chen, T.-J. Yang, J. Emer, “Efficient Processing of Deep Neural Networks: A Tutorial andSurvey,” Proceedings of the IEEE, vol. 105, no. 12, pp. 2295-2329, December 2017.
– Hardware Architecture for Deep Neural Networks: http://eyeriss.mit.edu/tutorial.html

Vivienne Sze (@eems_mit)

85

References• Co-Design of Algorithms and Hardware for Deep Neural Networks– T.-J. Yang, Y.-H. Chen, V. Sze, “Designing Energy-Efficient Convolutional Neural Networks using EnergyAware Pruning,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
– Energy estimation tool: http://eyeriss.mit.edu/energy.html– T.-J. Yang, A. Howard, B. Chen, X. Zhang, A. Go, V. Sze, H. Adam, “NetAdapt: Platform-Aware Neural
Network Adaptation for Mobile Applications,” European Conference on Computer Vision (ECCV), 2018.
– D. Wofk*, F. Ma*, T.-J. Yang, S. Karaman, V. Sze, “FastDepth: Fast Monocular Depth Estimation on
Embedded Systems,” IEEE International Conference on Robotics and Automation (ICRA), May 2019.
http://fastdepth.mit.edu/• Energy-Efficient Visual Inertial Localization– Project website: http://navion.mit.edu– A. Suleiman, Z. Zhang, L. Carlone, S. Karaman, V. Sze, “Navion: A Fully Integrated Energy-Efficient
Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones,” IEEE Symposium on
VLSI Circuits (VLSI-Circuits), June 2018.
– Z. Zhang*, A. Suleiman*, L. Carlone, V. Sze, S. Karaman, “Visual-Inertial Odometry on Chip: An
Algorithm-and-Hardware Co-design Approach,” Robotics: Science and Systems (RSS), July 2017.
– A. Suleiman, Z. Zhang, L. Carlone, S. Karaman, V. Sze, “Navion: A 2mW Fully Integrated Real-Time
Visual-Inertial Odometry Accelerator for Autonomous Navigation of Nano Drones,” IEEE Journal of Solid
State Circuits (JSSC), VLSI Symposia Special Issue, Vol. 54, No. 4, pp. 1106-1119, April 2019.
Vivienne Sze (@eems_mit)

86

References• Fast Shannon Mutual Information for Robot Exploration– Z. Zhang, T. Henderson, V. Sze, S. Karaman, “FSMI: Fast computation of Shannon Mutual Information for
information-theoretic mapping,” IEEE International Conference on Robotics and Automation (ICRA), May 2019.
– P. Li*, Z. Zhang*, S. Karaman, V. Sze, “High-throughput Computation of Shannon Mutual Information on Chip,”
Robotics: Science and Systems (RSS), June 2019.

• Low Power Time of Flight Imaging– J. Noraky, V. Sze, “Low Power Depth Estimation of Rigid Objects for Time-of-Flight Imaging,” IEEE Transactions
on Circuits and Systems for Video Technology (TCSVT), 2019.
– J. Noraky, V. Sze, “Depth Estimation of Non-Rigid Objects For Time-Of-Flight Imaging,” IEEE International
Conference on Image Processing (ICIP), October 2018.
– J. Noraky, V. Sze, “Low Power Depth Estimation for Time-of-Flight Imaging,” IEEE International Conference on
Image Processing (ICIP), September 2017.

• Monitoring Neurodegenerative Disorders Using a Phone– H.-Y. Lai, G. Saavedra Peña, C. Sodini, T. Heldt, V. Sze, “Enabling Saccade Latency Measurements with
Consumer-Grade Cameras,” IEEE International Conference on Image Processing (ICIP), October 2018.
– G. Saavedra Peña, H.-Y. Lai, V. Sze, T. Heldt, “Determination of saccade latency distributions using video
recordings from consumer-grade devices,” IEEE International Engineering in Medicine and Biology Conference(EMBC), 2018.
– H.-Y. Lai, G. Saavedra Peña, C. Sodini, V. Sze, T. Heldt, “Measuring Saccade Latency Using Smartphone
Cameras,” IEEE Journal of Biomedical and Health Informatics (JBHI), March 2020.
Vivienne Sze (@eems_mit)