Invocation-driven Neural Approximate Computing with a
Multiclass-Classifier and Multiple Approximators
Haiyue Song, Chengwen Xu, Qiang Xu, Zhuoran Song, Naifeng Jing, Xiaoyao Liang, and Li Jiang
Advanced Computer Architecture Laboratory
Shanghai Jiao Tong University

1

Background2

Related works and Motivation3

Proposed Method

4

Experiment Results

5Conclusion

Approximate Computing

Machine Learning

Robotics

Original

10%ErrorImage Processing

Data Mining• Many applications are error tolerant• Neural network (NN) is suitable to approximate a code block/function• Amdahl law: performance limited by serial code
• NN has high parallelism, e.g., FPGA, ASIC, GPU• An interesting facts: Neural network can approximate any continuous function

1Background2

Related works and Motivation3

Proposed Method

4

Experiment Results

5Conclusion

Related works▪ Model based quality control for Approximate Computing [ISCA’15, ISLPED’16, DATE’16]• Classifier：predict the data is “approximatable” or not• Approximator (Accelerator)：approximately compute data at
fast speed and low power consumption• Error：the gap between the output of approximator and that oforiginal program

NN-basedError < 𝜃Error > 𝜃

NN-based

With quality control architectureRelated works▪ Model based quality control for Approximate Computing [ISCA’15, ISLPED’16, DATE’16]• Classifier：predict the data is “approximatable” or not• Approximator (Accelerator)：approximately compute data at
fast speed and low power consumption• Error：the gap between the output of approximator and that oforiginal program▪ Question:

NN-basedError < 𝜃

Error > 𝜃How to train NN-based classifier and approximator?
NN-based

With quality control architectureRelated works▪ One-pass training[ISCA’16]▪ Train Approximator and Classifier separately▪ Ignore the correlation between the two NNs

One-pass training method

Related works▪ Iterative training[DAC’17]▪ Train Approximator and Classifier together using iterative training▪ Classifier correlate with Approximator▪ Data with low error is easy to predict

Iterative training

Motivation▪ Problems▪ Even iterative training, some data still fail to be approximated (red part in the figure)▪ Single Approximator may overfit one cluster/distribution of input sample

Do we really have to give up those data?

Motivation▪ Problems▪ Even iterative training, some data still fail to be approximated (red part in the figure)▪ Single Approximator may overfit one cluster/distribution of input sample▪ Motivation▪ Multiple approximators may be complementary, and make invocation higher

Do we really have to give up those data?

1

Background2

Related works and Motivation3

Proposed Method

4

Experiment Results

5

Conclusion

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Training Process▪ The original input samples are used to train classifier C1 and approximator A1.

Training Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Training Process▪ The original input samples are used to train classifier C1 and approximator A1.
▪ Feed the remaining input samples not yet to be recognized by C1 (Data nC) to classifier C2 andapproximator A2.

Training Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Training Process▪ The original input samples are used to train classifier C1 and approximator A1.
▪ Feed the remaining input samples not yet to be recognized by C1 (Data nC) to classifier C2 andapproximator A2.
▪ Repeat until a specific pair of Cn and An cannot converge.

Training Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Inference Process▪ If C1 approves, the input data are sent to A1.

Inference Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Inference Process▪ If C1 approves, the input data are sent to A1.
▪ If C1 disapproves, the input data are sent to the next classifier C2.

Inference Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Inference Process▪ If C1 approves, the input data are sent to A1.
▪ If C1 disapproves, the input data are sent to the next classifier C2.
▪ Repeat until Cn approves.

Inference Process

Multiple Cascaded Classifiers and Approximators (MCCA)▪ Inference Process▪ If C1 approves, the input data are sent to A1.
▪ If C1 disapproves, the input data are sent to the next classifier C2.
▪ Repeat until Cn approves.

▪ Demerit▪ The time spending on inference is too long

Inference Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Inference Process▪ The multiclass-classifier predicts which approximator can approximate theinput data.

Inference Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Complementary training▪ Test A1 with all data, produce thethe label C1 for any input sample
that A1 can safely approximate

Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Complementary training▪ Test A1 with all data,
produce the the label C1 forany input sample that A1
can safely approximate▪ Test A2 with the remaining
data, produce the the label
C2 for any input sample thatA2 can safely approximate

Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Complementary training▪ Test A1 with all data, produce the thelabel C1 for any input sample that A1can safely approximate▪ Test A2 with the remaining data,
produce the the label C2 for any input
sample that A2 can safely approximate▪ Repeat until test An, the remaining
input samples without any label arelabeled as nC.
Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Complementary training▪ Test A1 with all data, produce the thelabel C1 for any input sample that A1can safely approximate▪ Test A2 with the remaining data,
produce the the label C2 for any input
sample that A2 can safely approximate▪ Repeat until test An, the remaining
input samples without any label arelabeled as nC.
▪ Train the multiclass-classifier andapproximators using iterative training.
Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Competitive training▪ Test A1 with all data, obtain theapproximation error.
▪ Test A2 with all data, obtain theapproximation error.
▪ …
▪ Test An with all data, obtain theapproximation error.

Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Competitive training▪ Test A1 with all data, obtain theapproximation error.
▪ Test A2 with all data, obtain theapproximation error.
▪ …
▪ Test An with all data, obtain theapproximation error.
▪ Generate the label for each dataaccording to the lowest
approximation error.
Training Process

Multiclass-classifier and Multiple Approximators (MCMA)▪ Competitive training▪ Test A1 with all data, obtain theapproximation error.
▪ Test A2 with all data, obtain theapproximation error.
▪ …
▪ Test An with all data, obtain theapproximation error.
▪ Generate the label for each data according
to the lowest approximation error.
▪ Train the multiclass-classifier andapproximators using iterative training.

Training Process

Hardware design▪ Add a Controller to control the weight buffer inside the PE.

ApproximatorClassifierBus
Scheduler

BusScheduler
PEPE

PE

PE

PE

PE

PEPE

Input FIFO

Input FIFO
Output FIFO

CacheController

Output FIFOCacheCPU

The overall NPU design

Hardware design▪ Add Controller to control multiple approximators.

Layer2

Layer1

Layer2

Address

Layer1

Layer2

Cache

Layer1▪ Weight buffer receives the signal from the controller, and then sechedule
approximators.

Weight Buffer
A1
A2
A3Fetch
W reg

I reg

MAC
Accumulator
Registers

Output FIFO
Sigmoid

The detail PE design

Input
FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PE

PE

PE

PE

Cache

PEPE

PE

Address

Fetch
W reg

I reg

Input FIFO

Input FIFO
1

Output FIFOPE

Weight Buffer
A2 A3

case1: A1

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1▪ Read data from
Input FIFO.

CacheController

Output FIFO

Cache

CPU

Data flow

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

Input
FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PE2

PE

PE
PE

Cache

PEPE

PE

Address

Fetch
W reg

I reg

Input FIFO

Input FIFO
Output FIFOPE

Weight Buffer
A2 A3

case1: A1

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1▪ Conduct vector
multiplication in PE.

Cache

Controller

Output FIFO

Cache

CPU

Data flow

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

Input
FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PEPE

PE

PE

3CachePEPEPE

AddressFetch
W reg

I reg

Input FIFO

Input FIFO
Output FIFOPE

Weight Buffer
A2 A3

case1: A1

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1▪ Controller send
signal to CPU orApproximator.

Cache

Controller

Output FIFO

Cache

CPU

Data flow

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

Input
FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PEPE

PE

Weight Buffer
A2 A3

case1: A1CachePE

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1▪ If approximator
invoked, fetchcorresponding
approximator’s
weight.

Address
4PE

PE

PE

W reg

I reg

Input FIFO

Input FIFO
Output FIFO

PE

Fetch

Cache

Controller

Output FIFO

4CacheCPU

Data flow

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

Input
FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PE
PE

PE
PE

Cache

Controller

Cache

PE

PE

PE

Input FIFO

Input FIFO
Output FIFOPE

Weight Buffer
A2 A3

case1: A1

Layer2
Layer1
Layer2Layer1
Layer2
Layer1▪ Conduct vector
multiplication in PE.

Address

Fetch
W reg

I reg

5Output FIFO

Cache

CPU

Data flow

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

Input
5FIFO

Hardware design
ApproximatorClassifierBus
Scheduler

Bus
Scheduler
PE
PE

PE
PE

PE

Cache

PE

PE

PE

Address

Fetch
W reg

I reg

Input FIFO

Input FIFO
Output FIFO

6

Weight Buffer
A2 A3

case1: A1

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1▪ Send back the result
from PE to output
FIFO.

Cache

Controller

Output FIFO

Cache

CPU

Data flowMACAccumulatorRegisters
Output FIFO
Sigmoid

Input
FIFO

Hardware design▪ Load the weights layer by layer.

Weight Buffer
A2 A3

case2:

case1: A1

Layer2
Layer1
Layer2
Layer1
Layer2
Layer1

Cache

Address

CacheFetch
W reg

I reg

A1

Layer1

Input
FIFO

MAC
AccumulatorRegisters
Output FIFO
Sigmoid

The detail PE design

1

Background2

Related works and Motivation3

Proposed Method

4

Experiment Results

5Conclusion

Experimental setup• Compared with One-pass[ISCA’16] and Iterative training[DAC’17]• 8 benchmark applications

Experiment Results• Invocation increase 20%~30% on average.
• Invocation increase 40%+ in sobel or kmeans benchmark.

Experiment Results• The approximation error is below the error bound in most benchmarks.

Experiment Results• The average speedup is 1.23x compared with one-pass method.

Experiment Results• The average energy reduction is 1.15x compared with one-pass method.

Experiment• Almost all samples have a corresponding approximator that can approximate it1

Background2

Related works and Motivation3

Proposed Method4Experiment Results

5Conclusion

Thanks for listening!
Invocation-driven Neural Approximate Computing with a
Multiclass-Classifier and Multiple Approximators
Zhuoran Song (宋卓然)
Professor Li Jiang (蒋力)Advanced Computer Architecture Laboratory
Shanghai Jiao Tong University