Embench 0.5: A Free Benchmark Suite for IoT - Jeremy Bennet - ORConf 2019
Channel: FOSSi Foundation
Date: 12/10/2019

In June we announced Embench™ (for Embedded Benchmark), a benchmark for embedded IoT devices created by experts from both academia and industry. We first looked at past benchmark efforts to try to incorporate their good ideas and to avoid their mistakes, and drew the following seven lessons:

-    Embench must be free (no cost).
-    Embench must be easy to port and run.
-    Embench must be a suite of real programs.
-    Embench must have a supporting organization that maintains its relevance over time.
-    Embench must report a single summarizing performance score.
-    Embench should report geometric mean and standard deviation as the summarizing score.
-    Embench must involve both academia and industry.

Our plan is to follow “Agile Benchmark Development” by bringing out an initial 0.5 version for evaluation and feedback, with a 1.0 version to be released after a few iterations of preliminary Embench versions (0.5, 0.6, 0.7,...) to debug both the suite and the ground rules. We initially aim to evaluate microcontrollers that support ≤64KiB of code and read only data—which would typically be stored in Flash or ROM—and ≤16KiB of data.

In this talk we describe Embench 0.5. Unlike the popular Dhrystone and CoreMark benchmarks that are single synthetic programs, Embench 0.5 suite has 20 real programs. They do little or no floating point computation, but some are branch intensive, some memory intensive, and some integer compute intensive. Like SPEC, the single performance number that people will promote is the geometric mean of performance relative to a reference platform of the 20 programs plus the geometric standard deviation to indicate the significance of differences between processors. Novel features of Embench include reporting code size as well as program performance and to measure interrupt latency and context switching. These novel features play an important role in embedded computing but have not yet appeared in embedded benchmarks. The Free and Open Source Silicon (FOSSi) Foundation organization is the official sponsor of Embench. The Embench organizing committee now has more than 10 volunteers to evolve the benchmark.

In addition to announcing Embench 0.5, we will demonstrate its value by contrasting its performance evaluation of several embedded systems to Dhrystone and CoreMark.

Our hope is that henceforth is that publications of performance for embedded computers will include Embench along with Dhrystone and CoreMark, and that Embench will eventually retire these much older, single, synthetic benchmarks to the dustbin of history.
Presenter: Jeremy Bennett

Jeremy Bennett is Vice-chair and convener of the Embench working group, which is developing a new set of benchmarks for Embedded systems.