Measuring Inference Performance of Machine-Learning Frameworks on Edgeclass Devices with the MLMark™ Benchmark
Peter Torelli
President, EEMBC
peter.j.torelli@eembc.org

Mohit Bangale
Senior Engineer - ML, Ignitarium
mohit.bengale@ignitarium.com

This paper explains how EEMBC—a non-profit
consortium of embedded technology companies—
developed the MLMark benchmark for characterizing
machine-learning inference on edge devices, anddiscusses results obtained running the benchmark on
multiple accelerators.

Flex, and several other member companies. Lastly, we
will review data that has been collected to date, withsome preliminary conclusions.

1

Introduction
In the past five years, machine learning (ML) as apractical application has consumed the vast majority of
research in computer engineering. Despite 50 years of
research on this topic, only recently has thetechnological environment exposed researchers toaffordable, high-performance computers designed for
ML acceleration. The cloud-edge paradigm is entirely
suited to ML development: training done in the cloud,with its bottomless-well of compute resources, andinference performed on the edge, at a billion locations ina space once referred to as “ubiquitous computing.” In
this paper we are interested in the characteristics of thebillions of tiny devices rapidly emerging at the edge.
These devices have significant constraints on energy use,
size and cost; constraints which point back to a need for
effective performance analysis, which in turn requires aneffective benchmark.
In this paper, we will first provide a short overview
of benchmarking from our perspective, followed by the
motivation for exploring the machine learning domain.
Next the paper covers how we applied our traditional
methodology to the problem by developing theMLMark™ benchmark1 with a team comprised ofengineers from Intel, NVIDIA, Arm, TI, Ignitarium,
1

2

EEMBC history and motivation
EEMBC was founded by EDN in 1997 with thegoal of addressing the lack of benchmarks in theembedded microprocessor industry. While desktop
CPUs had plenty of options due to popularity of theWindows operating system, embedded platform
benchmarks were limited to a few choices, and theindustry had no standards body to represent its needs.
After accomplishing its initial goal, EEMBC went on toaddress the changing needs of the industry by expanded
into areas such as multicore processing; energy
consumption of the MCU core and its peripherals (e.g.,
SPI, BLE); mobile phones; automated driving; and mostrecently, machine learning.
In the past, machine learning research depended onlarge and esoteric custom architectures, or in the last
decade, huge arrays of GPUs or symmetric-core CPUs.
Recently, an extensive R&D investment and academic
research in this space has optimized two key variables:
software efficiency, which has produced smaller, more
effective neural nets requiring fewer resources (plusbetter-realized APIs to make research more accessible);and dedicated hardware acceleration that requires lesspower and has a lower cost. The combined effect of these
optimizations has pushed these machine-learning
devices down into EEMBC’s benchmarking domain of
embedded computing.

https://www.mlmark.org-1-

3

Benchmarks
A benchmark is a tool for distilling the behavior of
a complex system into a single number, preferably one
that increases to indicate the improvement of said
system in response to an input change. This simple
measurement capability enables the entire foundation ofsystem-performance analysis because every model, every
decision, every engineering tradeoff requires empiricalgrounding. Here are a few observations EEMBC has
fostered over the years.
3.1

Benchmark qualities
For a benchmark to be effective, it must be
reproducible, transparent, and constrained.

Reproducible

BenchmarkQualities
Constrained

Transparent

Reproducible means that the same inputs must
generate the same output, within a small degree oftolerance for systemic or stochastic variation. No
random components exist unintentionally in thebenchmarks
themselves,andtolerancesarerecommended when comparing scores or certifying
them. MLMark has no random variation, the sameimages are used for each iteration, and the amount ofruntime must exceed a certain minimum value to avoid
timing resolution and operating system asynchronous
behavior from interfering. MLMark also reports values
to three significant digits, helping to further filter out
these variations.
Transparent means anyone may view the means bywhich a score is generated, including code, algorithms,
and run-rules. Some EEMBC benchmarks are licensed
to help fund support and research, while others areavailable at no cost under an augmented Apache 2.0
license. MLMark is available on GitHub, and in orderfor someone to upload scores to the website database,the target source-code implementation must be firstincluded in the repository for all to see.
Constrained, the most nuanced of the threerequirements, refers to the how the benchmark enables
flexibility while at the same time facilitating meaningfulcomparisons between very different systems.
One classic example of trying to maintainbenchmarking constraints can be seen in the MCU
CoreMark® benchmark. Consider the execution loops
within parts of the benchmark, which are always thesame length. If a compiler were to unroll the loops
entirely, an MCU might perform better due to lack ofbranch misprediction. This optimization would
illustrate a best-case scenario for that architecture underthose fixed-loop conditions. However, if the number of
iterations were to change by one, the performance would
drastically change. If an outside observer was not privy
to the compiler optimization, this discontinuity would
seem jarring.
A good benchmark responds to small changes at
the input with commensurate changes at the output. By
constraining what is permissible in terms of
optimizations, this kind of non-intuitive discontinuity
vanishes. This is generally done with run-rules, which
are procedural do’s and don’ts, but can also be done byin many other ways such as limiting what portions of
the benchmark may be altered and how, or by selecting
specific input stimuli used in all measurements.
Going back to the CoreMark example: this doesn’tmean that a developer or manufacturer cannot perform
loop unrolling to show benefit, but it needs to be clearlystated the score was obtained out-of-spec (and thus is
not valid for upload to the EEMBC database). MLMark
addresses constraints by putting limiting the input
datasets and models to specific versions. However, what
the underlying framework actually does to optimize themodel is unrestricted. We will discuss this more later
under accuracy measurement.
The MLMark benchmark meets all three criteria:
reproducibility by pre-selecting the exact stimuli andmodels that must be used and accounting formeasurement precision; transparency by providing thesource code and conversion scripts used to generate
published scores, and constrained by only allowing theuse of certain datasets and rigid run-rules policies whilestill allowing the framework to optimize at its discretion.

-2-

Benchmark components
Embedded benchmarks generally consist of a test
harness and a workload (see Figure 1). The test harness
has two jobs: interface the user to the benchmark, andinterface the benchmark to the hardware. The workload
is the actual operation(s) of interest to the benchmark.

TEST HARNESSTARGET

Parse Parameters& Config Files

Convert Reference
Model

3.2

Test harness
Historically, embedded devices have lacked thecapacity to run a highly-advanced operating system.
Many run “bare metal”, where the workload is the only
code present in the firmware, with no operating system.
In these cases, the test harness usually runs on a remote
operating system and communicates to the device via a
serial debug port or a USB port. If the hardware has thecapability, the test harness may also run on the target
device.
MLMark is a set of interpreted Python scripts
intended to run on Linux in version 1.0. MLMarkprovides an API abstraction layer to multiple targets.
With this abstraction layer, the target may be the hostCPU, a USB device, an FPGA, or any other type ofaccelerator: it is not relevant to the test harness if the
DUT is local or remote. Some of the targets requireinterface layers to their frameworks. These are provided
as libraries which have been pre-compiled for Ubuntu
16.04, but source code is provided for recompilation ondifferent target platforms.

Optimize Model

3.2.1

3.2.2

Workload
A workload consists of a behavioral model defining
what the device under test actually does during thebenchmark. This region of execution is fenced bytimestamps to ensure the measurement happens as close
to the execution as possible. In earlier benchmarks, theworkload consisted of C-code that computed a specifictask, like an FFT kernel, or an emulation of a real-world
application, like the combination of a state-machine,
XML parser, and a compression algorithm (seeCoreMark-PRO). More recent EEMBC benchmarks
include wireless transfers, security handshakes, etc.

2

Preprocess Images
with OpenCV

Process Results

Throughput, Latency
Accuracy (mAP, TopN)

Infer

Figure 1. The MLMark software architecture API between
test harness and target falls largely on the frameworkboundary.

In MLMark, the workloads consist of neural-net
graphs, also known as models. In a similar analogy toolder C-based benchmarks where the compiler
optimized the C-code, models are optimized by theframework before execution on the hardware. For
example, in MLMark the model for ResNet-50 isprovided as a TensorFlow graph. When this is run on
NVIDIA hardware, the TensorFlow graph is converted
to a UFF format for NVIDIA hardware. During theconversion process, the NVIDIA framework, TensorRT,
looks for hardware specific optimizations which may
include a wide variety of operations such as node
pruning or scheduling to various heterogeneous computeelements in the accelerator. The same thing happens
with Intel hardware through OpenVINO and Google
TPUs through their TPU compiler.
In the first version of MLMark, three vision models
were chosen based on their popularity among academics
and support from the industry: MobileNet V1.0, a small
image classification network; ResNet-50 V1.5, a larger,
more accurate image classifier; and SSDMobileNet v1.0,
a single-shot detector using MobileNet for classification.
Refer to the MLMark GitHub repository2 for specificcopyright information about the authors who graciously
made these models available to the public.

https://github.com/eembc/mlmark-3-Call target’s
apiRun()

The workloads also require input datasets ofimages. For the classification models, we used theILSVRC2012 dataset which contains over 6.4 GB ofimages. For the object detection model, we used the
COCO2017 dataset, which is slightly smaller, about 1.3
GB. Due to copyright licensing limitations, the images
are not part of the repository, but instructions areprovided to obtain them.
Scoring and accuracy
Performance is generally measured in units of work
per second, where units of work may be iterations of aloop, CPU instructions retired, or in the case of
MLMark V1.0, image inferences. The timing points
inside MLMark occur on the boundaries of the inference
operation, after dataset loading, and after retrieving
inference results. Dataset loading was excluded from thetiming due the many ways an inference engine may load
data. Retrieving results was included in the timing loop
because in order to assess the inference, the data must
be read back to the host system. However, given theamount of time spent on inference, reading back results
is a very fast operation compared to loading the dataset.
See Figure 2.
MLMark produces two different performance
metrics: throughput and latency. Throughput is simply
the average inference performance computed by the
total number of inferences performed within the sum of
all timing windows. Since version 1.0 of MLMark
contains image-based inferences, the throughput is thesame as frames-per-second.
Latency is measured as the 95th percentile of eachinference’s timing window, or how long it takes for one
inference to complete. This statistical designation means
95% of the time the actual latency will be equal to or
better than this reported value. This value often differsfrom datasheets, which may report the best-caselatency.

Load Images

session.predict()
Timing
Window

3.2.3

read output
Tensor(s)Log results

Figure 2. Performance is measured at the start of inference
and the end of result retrieval (TensorFlow example).

For the workloads used in MLMark, accuracy isexpressed in reference to ground-truths, which are a setof human-annotated results for each input datum. For
example, during object classification the ground truth
for an image would be one (or more) categories
associated with that image (dog, tree, car). For object
detection, a collection of regions of interest (or ROIs)
and their associated categories would make up theground truth for that image (e.g., the sub-image atrectangle [(10,10),(100,100)] is a dog, with 50%
confidence).
The accuracy score depends on the type of
inference. For classification, it is defined as Top-1 andTop-5, which indicate if the ground-truth category
matched the most confident prediction of the system, or
if it was one of the Top-5 predictions. For a large
number of inferences, a perfect score would be a Top-1of 100%, meaning for every inference, the system’s top
prediction was always the ground-truth.
For object detection, the accuracy measurement ismuch more sophisticated, since not only must thesystem classify an object, it must also detect theboundaries of the object in an image. There are manydegrees of error here: false object detections, incompletebounding boxes, wrong classification for a correctbounding box. The method for computing this is known
as “mean average-precision”, or mAP for short. The-4-

details are beyond the scope of this document, but are
documented in many papers, see: (Hui, 2018),(Wikipedia, n.d.), and (Shah, 2018). Since there is alsoa prediction threshold for each detection, MLMark
limits the threshold to 30% confidence.
MLMark does not mandate an accuracy threshold.
Instead, the run rules state that performance numbers
must always be reported simultaneously with theaccuracy of the inference. In this way, the benchmark
explicitly exposes the performance vs. accuracy tradeoff.

All of these components in the framework work
together to make sure the model’s graph is converted toan optimal format for the underlying hardware.

4

5.1

ML frameworks
Each accelerator manufacturer provides aframework that enables the user to perform an inference
on a model. The framework is usually one or morelibraries. Every accelerator company provides their own
frameworks, with some integrated into Caffe or
TensorFlow. All frameworks consist of roughly the same
API functions: load a graph, optimize it, perform an
inference, and fetch results.
Here are a few examples of frameworks found in
the MLMark target area:

•

•
•
•
•

•

TensorFlow: a self-contained framework created by
Google which natively runs on a CPU (and canrun on a GPU with extensions)TensorFlow Lite (TFLite): a somewhat optimized
version of TensorFlow targeting edge devices
Intel OpenVINO: an expansive API that interfaces
with Intel CPUs and other accelerators
NVIDIA TensorRT: a library that builds on
CUDA and cuDNN
Arm NN: a library built on the Arm Compute
Library that provides acceleration for Arm CPUs
with NEON, and Arm GPUs with Mali.
Google TPU: the TPU compiler which compiles amodel into a native binary for their TPU
accelerator

5

Configuration variables that impact performance
Simply loading the model with the framework andproviding an input stimulus for inference is half thechallenge. There are also a number of variables that may
impact performance. MLMark exposes three of these:precision, concurrency, and batching.
Precision
The baseline models provided by MLMark encode
their weights in IEEE 32-bit floating-point format, also
known as FP32. Since the majority of the math in CNN
inference involves matrix operations, it is theoreticallypossible to double the efficiency of the network by using
16-bit floating-point, FP16. This may be IEEE FP16 or
bfloat16 (a format invented by Google. Optimizing evenfurther) or an 8-bit integer format, INT8, which has
been gaining popularity because some models show little
decrease in accuracy when reducing the precision of theweights. Typically, converting an FP32 to an INT8
requires quantization, which means adjusting theweights of the model to accommodate the reduced
fidelity. As a result, MLMark provides pre-quantizedversions of the models in INT8 format using posttraining integer quantization, or PTIQ (as well as the
scripts required to do this for oneself).
The tradeoff here seems straightforward: reducing
the number of bits in the weight may boost performancebut also reduce precision of the predictions. However, as
we will see this is not always the case. Further
complicating matters, not all frameworks and hardware
combinations support all precisions. MLMark allows
configuration to support all available formats on the
hardware.

-5-

One
Inference,
33ms
1

2

1

3

2

1 second

1 second

Batch = 1, Concurrency = 1, Iterations = 3, Throughput = 3 FPS

Batch = 2, Concurrency = 1, Iterations = 3, Throughput = 6 FPS

1
1

3
2

5
4

3

3

2

6

1 second

5

4

6

1 second

Batch = 1, Concurrency = 2, Iterations = 6,
Steady-state Throughput = 6 FPS

Batch = 2, Concurrency = 2, Iterations = 6,
Steady-state Throughput = 12 FPS

Figure 3. An illustration of the subtle differences between batching and concurrency. When batching, all inputs must be presented
to the input of the model at the same time; when running concurrent inferences, each inference may begin once the first stage
in the pipeline is available.

platform dependent because it requires both APIs in
the framework and hardware features to facilitate
scheduling.
The framework and hardware determine which
method of parallelization are available. In either case,significant gains may be realized by utilizing idle
resources. This is where smaller models have an
advantage, as more resources may be engaged toincrease performance.

5.2

Batching and concurrency
Multiple images may be inferred at the sametime either through batching or concurrency, or insome cases, both. Not all hardware supports thesecapabilities. Figure 3 illustrates the subtle difference
between the two from an execution timing
perspective.
Batching refers to performing multiple inferences
simultaneous by increasing the dimensionality of thelayers’ tensors to accommodate more images. For
example, an input tensor may be of dimension [1, 32,
32, 24], which would mean one image, 32x32 with 8bits per-pixel. By expanding this input tensor to [4,32, 32, 24], each successive operation in the graphperforms four simultaneous inferences since thetensors will reshape to accommodate the new data;
the mathematic operations remain the same.
Concurrency refers to exploiting the parallel
nature of the graph as a pipeline. Unlike batching,concurrency feeds successive inputs to the model after
each layer has completed calculation. Concurrency
may also refer to having multiple instances of thesame model active at one time, as many as theavailable resources allow. Concurrency is very

6

Methodology
This section describes how the decisions we madeaffect the constraints on data collection, input format,
and measurement.
6.1

Input models file formats
Here we are dealing with a number of different
frameworks. Some frameworks are designed to read aparticular file format. For example, TensorRT
framework has parsers for neural network models in
Caffe, UFF and ONNX file formats while Google TPU
can read only INT8 TFLite, and only after being
compiled for the Edge TPU. We will go throughdetails as follows:-6-input layer and one output layer. The user can findthe name of last layer using the command:

6.1.1

Native TensorFlow PB format
TensorFlow PB refers to “protobuf” or “protocol
buffers.” This format is from Google which is language
and platform-neutral. (Google, n.d.) It is a mechanism
for serializing structured data. Both the graphdefinition and weights are stored together in this fileformat. Thus, this is a convenient way to store aneural network model.
TensorFlow’s model zoo hosts many standard
neural network models in this format. We areconsidering models in this file format as “golden”
models. In case a framework cannot read the model inthis format, a converter is used to convert the modelin PB format into the appropriate format.

$ convert_to_uff input_file.pb -lThis command is also used to convert the UFF
file:$ convert_to_uff input_file.pb -ooutput_file.uff -O name_of_output_node

6.1.4

TFLite file format
TFLite is specially designed for inference onembedded devices. The serialization format used in
TFLite is different from TensorFlow. TensorFlow uses
“Protocol Buffers” while TFLite makes use of“FlatBuffers.”
As stated by Google CodeLabs: “The primary
benefit of FlatBuffers comes from the fact that they
can be memory-mapped, and used directly from disk
without being loaded and parsed. This gives much
faster startup times, and gives the operating system
the option of loading and unloading the required pages
from the model file, instead of killing the app when it
is low on memory.” (CodeLabs, n.d.)

6.1.2

UFF file format
UFF stands for “Universal Framework Format,”
Not to be confused with Universal File Format which
is widely used in CAD domain. This is an NVIDIA
format “designed to encapsulate trained neuralnetworks so that they can be parsed by TensorRT.
It’s also designed in a way of storing the information
about a neural network that is needed to create aninference engine based on that neural network.”(NVIDIA, n.d.)6.1.5

TensorFlow to TFLite conversion without
quantization
TFLite converter is part of the TensorFlow
installation. The converter takes the output file name,input model in PB format, and the names of the input
and output layers, e.g.:

The V1.0 MLMark release TensorRT target uses
an UFF parser. However, per NVIDIA’s
documentation, this format will be deprecated in the
future. (NVIDIA, 2019)
6.1.3

Conversion process : TensorFlow PB to
UFF
TensorFlow PB to NVIDIA UFF conversion
needs to be done on an x86 machine (laptop or adesktop with NVIDIA GPU). This limitation exists
because TensorRT’s Python API was not supported
on their Jetson platform and the converter script “convert_to_uff” - needs this Python API.
Prerequisites are CUDA, cuDNN and TensorRT. The“convert_to_uff” utility is part of the TensorRT
installation.
The following command is sufficient for simpler
models like ResNet-50 and MobileNet which have one% tflite_convert \--output_file=foo.tflite \--graph_def_file=Mobilenetfrozen_graph.pb \--input_arrays=input \--output_arrays=\
MobilenetV1/Predictions/Reshape_1

The
TFLiteconverter
appliesmany
optimizations apart from quantization which
improves performance of the model, such as pruning
unused graph-nodes, and joining operations into more
efficient composite operations.
In practice, it is observed that just the format
conversion from PB to TFLite gives significant-7-performance improvement without even quantization.
See the results section on TensorFlow versus TFLite.
6.2

6.2.3

Post-training full-integer quantization:
PTIQ
TFLite now supports converting all model values(weights and activations) to 8-bit integers whenconverting from TensorFlow to TFLite’s flat-buffer
format. This results in four times reduction in modelsize. It also boosts performance 3-4x on a CPU.
Furthermore, this fully quantized network model canbe deployed on integer-only hardware accelerators.
The post-training quantization method stores
only the weights as 8-bit integer, but this full-integer
quantization method statically quantizes all weights
and activations. (TensorFlow, n.d.)Eight-bit integer (INT8) datatypes

6.2.1

Why quantize?
Typically, training happens in FP32 where all of
the weights and biases are 32-bit IEEE floats. It is
possible to tradeoff some accuracy and reduce model
size to one-fourth of its original by quantizing themodel.
INT8 quantization is trending in the embedded
inference world. The methodology for generating, andthe usage of INT8 quantified models vary fromframework to framework. For example, TensorRT
contains the code to create an optimized INT8
inference engine on the fly, while the Google Edge
TPU needs a pre-quantized INT8 model as an input.

6.2.4

TFLite compiled for Edge TPU
Google’s Edge TPU is able to run only INT8
TFLite models. A model also needs to be compiled
with edge TPU compiler. The compiler creates asingle custom operation (binary) for all Edge TPU
compatible ops, until it reaches an unsupported op.
The remaining layers stay in their non-binary format
and are run on the CPU or host.
The Edge TPU compiler is a command-line tool
which requires very little setup, e.g.:

6.2.2

Model
optimizationbyframework:
TensorRT
TensorRT needs a small image dataset of around1000 images for INT8 calibration. This calibration
image dataset is usually a subset of the validation
dataset. Calibration is a one-time process, meaning
once the engine has been created it can be reused for
subsequent inference engine generation. The output of
this process is called a “Calibration Table File” and isgenerally quite small (few KB).

$ edgetpu_compiler [options] model.tflite-8-

in performance, the data is presented in log format.
The units can interchange frames and inferences, sincea single inference call to the API requires one image
frame as an input, and produces one inference
summary (classifications, detections, etc.). For devices
that reported scores at more than one batch or
concurrency setting, the highest score from the setting
was chosen per workload. Raw data can be found at
the MLMark website, www.mlmark.org.
It should come as no surprise that there areorders of magnitude difference in this chart. After all,
we are comparing quad-core Cortex-A5X/A7X
devices to hundred-core dedicated neural accelerators.
The purpose of this paper is not to make marketingclaims, but to take a snapshot in time of the state of
the industry to serve as the leftmost point on a trend
graph that will span decades.

7

Results and observations
After the initial release of MLMark, several
dozen scores were collected on available edge
hardware. During the coming months we will continue
to add new scores and targets to the repository. Notethat many new accelerators are not represented due
to MLMark’s first rule of transparency: theimplementation must be published along with thescore. The following sections discuss the first round ofobservations.
Figure 4 combines several results on different
hardware, workloads and precision formats into one
chart. For each device, all three workloads arepresented, color coded by the weight precision. (Notethat some of the Arm devices do not have
SSDMobileNet scores, this is due to an API limitation
at the time of collection.) Due to the extreme range

1

10

Inferences (frames) per Second
100

1000

10000

Coral Dev Board, Edge TPU
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
Coral Dev Board, i.MX8M Cortex-A53
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
HiKey970, Arm Cortex-A73
MobileNet 1.0
ResNet-50 1.0
HiKey970, Arm Mali G72 M12
MobileNet 1.0
ResNet-50 1.0
Jetson AGX Xavier, Volta
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
Jetson Nano, Maxwell
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
Neural Compute Stick 2, Myriad X
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0

FP16

FP32

INT8

Figure 4. MLMark overall results (as of 2019-11-05); Each cluster of scores was produced by a specific device; three workloads
are illustrated, at three different precisions at a single batch & concurrency setting.

-9-

7.3

Batching and concurrency
In Figure 6, the batch size was increased on the
NVIDIA Jetson Xavier from 1 (a single image) to 32
simultaneous images per call to the inference engine
for the MobileNet V1.0 workload. TensorRT also
supports concurrent streams, which are plottedagainst the batch data for similar input sizes. Latency
at the 95th percentile is plotted on the secondary
vertical axis. While batching shows a significant
increase in performance at the start of the curve, ROI
begins to decrease rapidly after 16 images. Latency forbatching is relatively linear. Concurrency shows noincrease in throughput and even worse latency.

7.1

Performance by class
Figure 5 below plots all performance scoresagainst three basic technology classes: CPU, GPU andaccelerator. When grouped by technology class, theresults show CPU as being the lowest performing,
GPU the highest, and accelerators spanning a middle
range. Note that the only CPUs used in the results
were Arm Cortex A5X/A7X-class.

1000
Accelerator

GPU

5000

100
CPU

Performance (fps)

Throughput (Frames per second)

10000

10

1

40
35

4000

30
25

3000

20
2000

15
10

1000

5

0

Performance gains from decreasing precision
One would expect moving from FP32 to FP16
would yield a doubling of performance. This was notobserved (Table 1) in the three devices that supported
both formats.
FP16
(fps)
547
291
171

FP32
(fps)
367
128
128

% Increase

62.7
38.4
25.3

55.3
21.2
22.4

12%
45%
11%

68.0
16.7

45.6
10.5

33%
37%

10

20

30

40

# of batch images, or # of concurrent inferences

7.2

Jetson Xavier AGX
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
Jetson Nano
MobileNet 1.0
ResNet-50 1.0
SSDMobileNet 1.0
HiKey970, Mali G72
MobileNet 1.0
ResNet-50 1.0

00

Figure 5. Performance grouped by technology class.

Batch Throughput
Concurrency Throughput
Batch Latency
Concurrency Latency

Figure 6. Batching and concurrency using TensorRT on
the NVIDIA Xavier AGX platform.

33%
56%
25%

Table 1. Comparison of FP32 vs. FP16 performance acrosstargets that support both.

- 10 -

268%
Nano

250%

Xavier
200%

150%

283%
Accuracy (both mAP and Top-1)

Relative increase in throughput

300%
237%

187%

129%

100%

80
70
60
50
40
30
20
10
0
1

10

100

FP32

50%
0%

5%

1

2

4

10%

12%

12%

12%

8

16

32

64

10000

Performance (fps)

67%

8%

1000

FP16

INT8

Figure 8. Change in precision versus performance for all
hardware, targets, models and configurations.

0%Batch size

7.5

Model format optimizations
The Arm NN API provides a single-line change
in the code to switch between TensorFlow (protobuf)
and TensorFlow Light (FlatBuffer) models. This
enabled us to look at the same hardware, same
precision, and the same graph, but with a different
architectural representation of that graph. A 5-10%
improvement can be seen.

Figure 7. Batch performance depends on available computeresources; Xavier AGX has more available resources thanthe Nano, hence it obtains better batch performance.

7.4

Precision type impact to accuracy
When using lower precision datatypes forweights, conventional wisdom expects accuracy todecrease by some amount. Figure 8 plots all accuracy
values (mAP and Top-1%) against all throughputs fordifferent datatype precision; hardware, model andtarget are intentionally excluded simply to see larger
trends. The results indicate that there isapproximately 5% or less variation in accuracy across
32-, 16- and 8-bit datatypes when viewed as precision
clusters, and the inverse relationship isn’t an inherent
property of the models used.

ResNet-50

MobileNet

0

2

4

6

8

10

12

Performance (fps)TensorFlow Lite

TensorFlow

Figure 9. Observing the difference in performance between
model formats.

7.6

Economical value
In an attempt to assess cost and performance, we
compare the lowest selling price of the hardware
platform to their best score on MobileNet (since thisgenerally is the best performance model on all
hardware). If we consider a metric of frames-per-- 11 -

second, per dollar spent, the accelerators exceed theonly CPU in the study by a large margin. There arethree distinct strata in Table 2, low [0,0.1], medium(0.1,0.7], and high [0.7,2.4]. Additional data will bevery insightful over time.

some sort of SIMD. Still this code path remains unoptimized.” (tehtea, 2019)
7.7.2

FP32 TFLite outperforms INT8 on x86_64
architecture
There is a huge performance gap between FP32
and INT8, in the opposite direction of expectations,when running on x86_64 compared to otherarchitectures. The following comparison was made onan Intel i3 laptop running Ubuntu 16.04 andTensorFlow 1.15.0rc2:

Platform & Accelerator

fps/US$

Jetson Xavier AGX, Volta

2.4

Coral Dev Board, Edge TPU

2.3

Neural Compute Stick 2, Myriad X

0.7

Jetson Nano, Maxwell

0.6

HiKey970, Arm Mali G72 M12

0.2

HiKey970, Arm Cortex-A73

0.1

Workload, Precision

FPS on x86_64

0.1

MobileNet, FP32
MobileNet, INT8

30.2
0.331

Coral Dev Board, i.MX8M Cortex-A53

Table 2. Performance-per-dollar comparison, where cost is
the price of the entire platform, not simply the processor.

This could be related to TFLite optimizations forArm but not x86, as found in this TensorFlow issue:“[This is likely because quantized INT8 requires an
ARM NEON to be faster than float. On a PC float
runs better. This is because quantized int relies on
special instructions that have not been emphasized on
intel x86_64.]” (abhi-rf, 2018)

7.7

Other unusual observations
Every component in the benchmark chain—
except the test-harness—is experiencing a rapid stateof revision: some frameworks have monthly releases,
and others like TensorFlow have nightly patches. As
a result, we often found ourselves re-running
experiments with multiple versions of frameworks in
an attempt to be as fair as possible. Here are some of
the more puzzling instances we observed.

8

Conclusions
First and foremost, the wide range ofperformance values measured and the fragile natureof the software during the process is indicative of a
nascent industry on the left-hand side of the“innovation-optimization” pendulum. In one case,simply updating software to a minor version release ofa mature framework exposed a 40% performanceincrease. It is not clear that the results in this paper
will even be the same in six months on the very same
hardware.
However, we can conclude that some early
assumptions are correct, such as performance gains
made from decreasing the precision of the neural-net
weights have had only minimal impact on accuracy.
The accuracy decrease is not as severe as one would
expect, a 5% decrease in accuracy compared to a 75%
decrease in data size. Sensitivity to precision still
remains a function of other variables besides the
hardware, it remains to be seen if these other formats
will continue to persist.

7.7.1

TensorFlow Light lags TensorFlow on
x86_64 for the same precision
It is observed that TFLite runs slower thanTensorFlow on a x86-64 machine. The following
comparison was made on an Intel i3 laptop running
Ubuntu 16.04 and TensorFlow 1.15.0rc2:
Workload, Precision
MobileNet, FP32
MobileNet, FP32

Target Framework
TensorFlow
TFLiteFPS on x86_64
31.7
30.2

Following comparison was made on a i3 laptop
running ubuntu 16. TF version is 1.15.0rc2. The
actual reason is explained in this stack overflow
answer: “Existing TensorFlow Lite op kernels are
optimized for ARM processor by using NEON
instruction set. If SSE is available, it will try to adapt
NEON calls to SSE, so it should be still running with- 12 -

We’ve also seen that neural nets pose aninteresting scalability artifact: parallelism is inherent
in tensor math and can be accessed through batching
simultaneous operations on the same model, however
there needs to be enough resources available to see
this gain.
In terms of performance per dollar, the small
number of CPU and GPU scores in the results don’t
make a clear case for accelerator dominance, but wecan see trends emerging in dollars that favors theaccelerators.

to historical data and power analysis. Converting themodel performance numbers into TOPS (trillions-ofoperations per second) would enable us to map
performance on to historical trends. Powerconsumption was excluded since most of the boards
did not have power-plane isolation, meaning overall
platform scores would include NIC, USB and otherancillary power unrelated to the accelerator.
10

Thanks
The authors would like to thank the members of
EEMBC who helped make the hard decisions required
to complete this benchmark, as well as the countless
people staffing the GitHub accounts at Tensorflow
and ARM for answering our questions. Further, thisresearch would not have been possible without theacademics publishing the actual neural-net models.

9

FutureWe will continue to add scores, targets, andmodels. Already we have observed huge performanceincreases in the latest versions of drivers which havenot been captured in this document. Future studies
we would like to pursue include performance mapping

11 Bibliography
abhi-rf. (2018, August 18). INT TFLITE very much slower than FLOAT TFLITE #21698. Retrieved from
GitHub: https://github.com/tensorflow/tensorflow/issues/21698
CodeLabs. (n.d.). TensorFlow for Poets. Retrieved from codelabs.developers.google.com:
https://codelabs.developers.google.com/codelabs/tensorflow-for-poets-2-ios/#2
Google. (n.d.). Protocol buffers are a language-neutral, platform-neutral extensible mechanism for serializing
structured data. Retrieved from developers.google.com: https://developers.google.com/protocol-buffers
Hui, J. (2018, March 7). mAP (mean Average Precision) for Object Detection. Retrieved from medium.com:
https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173
NVIDIA. (2019, October). TensorRT Release Notes v6.01. Retrieved from docs.nvidia.com:
https://docs.nvidia.com/deeplearning/sdk/pdf/TensorRT-Release-Notes.pdf
NVIDIA. (n.d.). Deep Learning SDK Documentation. Retrieved from docs.nvidia.com:
https://docs.nvidia.com/deeplearning/sdk/tensorrt-archived/tensorrt_301/tensorrt-releasenotes/rel_3.html
Shah, T. (2018, January 26). Measuring Object Detection models — mAP — What is Mean Average Precision?
Retrieved from medium.com: https://towardsdatascience.com/what-is-map-understanding-the-statistic-ofchoice-for-comparing-object-detection-models-1ea4f67a9dbd
tehtea. (2019, January 8). Why is TensorFlow Lite slower than TensorFlow on desktop? Retrieved fromstackoverflow: https://stackoverflow.com/questions/54093424/why-is-tensorflow-lite-slower-thantensorflow-on-desktop
TensorFlow. (n.d.). Post-training Quantization. Retrieved from www.tensorflow.org:
https://www.tensorflow.org/lite/performance/post_training_quantization
Wikipedia. (n.d.). Evaluation measures for information retrieval. Retrieved from wikipedi.org:
https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)- 13 -