An Overview of Common
Benchmarks
Reinhold P. Weicker
Siemens Nixdorf Information Systems

he mdin redwn for using computers is to perform ta\ks taster Thiris why performance measurement
is taken so seriously by computer customers. Even though performance measurement
usually compares only one aspect of computers (speed), this aspect is often dominant.
Normally, a mainframe customer can run
typical applications on anew machine before
buying it. With microprocessor-based
systems, however, original equipment manufacturers must make decisions without
detailed knowledge of the end user’s code,so performance measurements with standard
benchmarks become more important.
Performance is a broad area, and traditional benchmarks cover only part of it.
This article is restricted to benchmarks
measuring hardware speed, including
compiler code generation; it does not cover
the more general area of system benchmarks(for example, operating system performance). Still, manufacturers use traditional
benchmarks in their advertising, and customers use them in making decisions, so it
is important to know as much as possible
about them. This article characterizes themost often used benchmarks in detail andwarns users about a number of pitfalls.

,

*

.
*

,

“Fair benchmarking”
would be less of an
oxymoron if thoseusing benchmark
results knew whattasks the benchmarks
really perform andwhat they measure.

nearly all its significance. This became
obvious when reduced instruction-set
computer architectures appeared.’ Operations that can be performed by one CISC
(complex instruction-set computer) instruction sometimes require several RISC
instructions. Consider the example of ahigh-level language statement

load
load
add
store

mem ( B J ,reg ( B )mem ( C ) ,reg ( C )reg ( E ) ,reg ( C ) ,reg (A)reg (A), mem (A)If both machines need the same time to
execute (not unrealistic in some cases),should the RISC then be rated as a 4-MIPS
machine if the CISC (for example, a VAX
11) operates at 1 MIPS? The MIPS number
in its literal meaning is still interesting forcomputer architects (together with the CPI
number - the average number of cycles
necessary for an instruction), but it loses its
significance for the end user.
Because of these problems, “MIPS” hasoften been redefined, implicitly or explicitly, as “VAX MIPS.” In this case MIPS isjust a performance factor for a given machine relative to the performance of a VAX
1 1/780. If a machine runs some program or
set of programs X times faster than a VAX
11/780, it is called an X-MIPS machine.
This is based on computer folklore sayingthat for typical programs a VAX 11/780
performs one million instructions per second. Although this is not true,* the belief is

A = B + C /* Assume mem operands */The ubiquitous MIPS
numbers
For comparisons across different instruction-set architectures, the unit MIPS,
in its literal meaning of millions of instructions per second (native MIPS), has lost

With a CISC architecture, this can becompiled into one instruction:addmem ( B ) ,mem (C), mem (A)On a typical RISC, this requires four instructions:*Some time ago I ran the Dhrystone benchmark program on VAX I1/780s with different compilers. With
Berkeley Unix (4.2) Pascal, the benchmark was translated into 483 instructions executed in 700 microseconds, yielding 0.69 (native) MIPS. With DEC VMS
Pascal (V. 2.4),226 instructions were executed in 543
microseconds, yielding 0.42 (native) MIPS. Interestingly, the version with the lower MIPS rating executed
the program faster.

widespread. When VAX MIPS are quoted,
it is important to know what programs
form the basis for the comparison and whatcompilers are used for the VAX 111780.
Older Berkeley Unix compilers produced
code up to 30 percent slower than VMS
compilers, thereby inflating the MIPS rating of other machines.
The MIPS numbers that manufacturers
give for their products can be any of thefollowing:
MlPS numbers with noderivation. Thiscan mean anything, and flippant interpretations such as “meaningless indication ofprocessor speed’ are justified.
Native MIPS, or MIPS in the literalmeaning. To interpret this you must knowwhat program the computation was basedon and how many instructions are generated
per average high-level language statement.
Peak MIPS. This term sometimes appears in product announcements of new
microprocessors. It is largely irrelevant,since it equals the clock frequency for mostprocessors (most can execute at least one
instruction in one clock cycle).
EDN MIPS, Dhrystone MIPS, or similar. This could mean native MIPS, when aparticular program is running. More often
it means VAX MIPS (see below) with a
specific program as the basis for comparison.
VAX MIPS. A factor relative to theVAX 11/780, which then raises the following questions: What language? Whatcompiler (Unix or VMS) was used for the
VAX? What programs have been measured?
(Note that DEC uses the term VUP, for
VAX unit of performance, in making
comparisons relative to the VAX 111780.
These units are based on a set of DEC
internal programs, including some floatingpoint programs.)
In short, Omri Serlin2 is correct i n saying, “There are no accepted industry standards for computing the value of MIPS.”

Benchmarks
Any attempt to make MIPS numbers
meaningful (for example, VAX MIPS)
comes down to running a representative
program or set of programs. Therefore, we
can drop the notion of MIPS and justcompare the speed for these benchmark
programs.
It has been said that the best benchmark
is the user’s own application. But this isoften unrealistic, since it is not always
66

possible to run the application on eachmachine in question. There are other considerations, too: The program may havebeen tailored to run optimally on an older
machine; original equipment manufacturers must choose a microprocessor for awhole range of applications; journalists
want to characterize machine speed independent of a particular application program.
Therefore, the next best benchmark (1) iswritten in a high-level language, making it
portable across different machines, (2) isrepresentative for some kind of programming style (for example, systems programming, numerical programming, orcommercial programming), (3) can bemeasured easily, and (4) has wide distribution.
Obviously, some of these requirements
are contradictory. The more representative
the benchmark program - in terms ofsimilarity to real programs - the more
complicated it will be. Thus, measurement
becomes more difficult, and results may beavailable for only a few machines. This
explains the popularity of certain benchmark
programs that are not complete application
programs but still claim to be representative for a given area.
This article concentrates on the mostcommon “stone age” benchmarks (CPU/memorykompiler benchmarks only) -in
particular the Whetstone, Dhrystone, andLinpack benchmarks. These are thebenchmarks whose results are most often
cited in manufacturers’ publications and in
the trade press. They are better thanmeaningless MIPS numbers, but readers
should know their properties - that is,what they do and don’t measure.
Whetstone and Dhrystone are synthetic
benchmarks: They were written solely forbenchmarking purposes and perform nouseful computation. Linpack was distilled
out of a real, purposeful program that is
now used as a benchmark.
Tables A-D in the sidebar on pages 6869 give detailed information about the highlevel language features used by thesebenchmarks. Comparing these advantages
with the characteristics of the user’s own
programs shows how meaningful the resultsof a particular benchmark are for the user’s
own applications. The tables containcomparable information for all threebenchmarks, thereby revealing their differences and similarities.
All percentages in the tables are dynamic percentages, that is, percentages obtainedby profiling or, for the language-feature
distribution, by adding appropriate counterson the source level and executing the pro-gram with counters. Note that for all programs, even those normally used in the
Fortran version, the language-feature-related statistics refer to the C version of thebenchmarks; this was the version for which
the modification was performed. However, since most features are similar in the
different languages, numbers for otherlanguages should not differ much. The
profiling data has been obtained from the
Fortran version (Whetstone, Linpack) or
the C version (Dhrystone).

Whetstone
The Whetstone benchmark was the firstprogram in the literature explicitly designed
for benchmarking. Its authors are H.J.
Curnow and B.A. Wichmann from the
National Physical Laboratory in GreatBritain. It was published in 1976, with
Algol 60 as the publication language. Today it is used almost exclusively in its
Fortran version, with either single precisionor double precision for floating-point
numbers.
The benchmark owes its name to the
Whetstone Algol compiler system. Thissystem was used to collect statistics about
the distribution of “Whetstone instructions,”
instructions of the intermediate language
used by this compiler, for a large number ofnumerical programs. A synthetic program
was then designed. It consisted of several
modules, each containing statements ofsome particular type (integer arithmetic,
floating-point arithmetic, “if‘ statements,
calls, and so forth) and ending with astatement printing the results. Weights wereattached to the different modules (realized
as loop bounds for loops around the individual modules’ statements) such that the
distribution of Whetstone instructions forthe synthetic benchmark matched the distribution observed in the program sample.
The weights were chosen in such a way thatthe program executes a multiple of one
million of these Whetstone instructions;thus, benchmark results are given as KWIPS(kilo Whetstone instructions per second)or MWIPS (mega Whetstone instructions
per second). This way the familiar term
“instructions per second” was retained but
given a machine-independent meaning.
A problem with Whetstone is that only
one officially controlled version exists the Pascal version issued with the Pascal
Evaluation Suite by the British Standards
Institution - Quality Assurance (BSIQAS). Versions in other languages canbe registered with BSA-QAS to ensure
COMPUTER

comparability.
Many Whetstone versions copied informally and used for benchmarking have theprint statements removed, apparently with
the intention of achieving better timing
accuracy. This is contrary to the authors’
intentions, since optimizing compilers may
then eliminate significant parts of the
program. If timing accuracy is a problem,the loop bounds should be increased insuch a way that the time spent in the extra
statements becomes insignificant.
Users should know that since 1988 there
has been a revised (Pascal) version of theb e n ~ h m a r k .Changes~
were made to modules 6 and 8 to adjust the weights and topreclude unintended optimization bycompilers. The print statements have been
replaced by statements checking the valuesof the variables used in the computation.
According to Wichmann,’ performance
figures for the two versions should be very
similar; however, differences of up to 20
percent cannot be ruled out. The Fortranversion has not undergone a similar revision, since with the separate compilation
model of Fortran the danger of unintended
optimization is smaller (though it certainly
exists if all parts are compiled i n one unit).
All Whetstone data in this article is basedon the old version; the language-feature
statistics are almost identical for bothversions.

Size, procedure profile, and languagefeature distribution. The static length of
the Whetstone benchmark (C version) as
compiled by the VAX Unix 4.3 BSD C
compiler* is 2,117 bytes (measurement
loops only). However, because of the program’s nature, the length of the individual
modules is more important. They are between40 and 527 bytes long; all except one
are less than 256 bytes long. The weights(upper loop bounds) of the individual
modules number between 12 and 899.
Table 1 shows the distribution ofexecution time spent in the subprograms of
Whetstone (VAX 11/785, BSD 4.3 Fortran, single precision). The most important,and perhaps surprising, result is that Whetstone spends more than half its time inlibrary subroutines rather than in the compiled user code.
The distribution of language features isshown in Tables A-D in the sidebar on*With the Unix 4.3 BSD language systems, it waseasier to determine the code size for the C version. The
numbers for the Fortran version should be similar.

December 1990

Table 1. Procedure Drofile for Whetstone.*
Procedure

Percent

What is done there

Main program
P3
PO
Pa
User code

18.9
14.4
11.6
1.9
46.8

FP arithmetic
Indexing
FP arithmetic

Trigonometric functions
Other math functions
Library functions

21.6
31.7

Sin, cos, atanExp, log, sqrt

Total

53.3
100

*Because of rounding, all percentages can add up to a number slightly below or above 100.

pages 68-69. Some properties of Whetstone are probably typical for most numeric applications (for example, a high number of loop statements); other properties
belong exclusively to Whetstone (for example, very few local variables).

Whetstone characteristics. Some important characteristics should be kept inmind when using Whetstone numbers for
performance comparisons.
( I ) Whetstone has a high percentage offloating-point data and floating-point operations. This is intentional, since thebenchmark is meant to represent numeric
programs.
(2) As mentioned above, a high percentage of execution time is spent inmathematical library functions. This
property is derived from the statistical data
forming the basis of Whetstone; however,it may not be representative for most of
today’s numerical application programs.
Since the speed of these functions (realized
as software subroutines or microcode)
dominates Whetstone performance to a highdegree, manufacturers can be tempted tomanipulate the runtime library for Whetstone performance.
(3) As evident from Table D in the sidebar, Whetstone uses very few local variables.
When Whetstone was written, the issue of
local versus global variables was hardly
being discussed i n software engineering,
not to mention in computer architecture.
Because of this unusual lack of local variables, register windows (in the Sparc RISC,
for example) or good register allocation
algorithms for local variables (say, in the

MIPS RISC compilers) make no difference in Whetstone execution times.
(4) Instead of local variables, Whetstone
uses a handful of global data (several scalarvariables and a four-element array of constant size) repeatedly. Therefore, a compiler
in which the most heavily used global
variables are allocated in registers (an optimization usually considered of secondary
importance) will boost Whetstone performance.
( 5 ) Because of its construction principle(nine small loops), Whetstone has an extremely high code locality. A near 100
percent hit rate can be expected even for
fairly small instruction caches. For the same
reason, a simple reordering of the sourcecode can significantly alter the execution
time in some cases. For example, it has
been reported that for the MC68020 withits 256-byte instruction cache, reorderingof the source code can boost performance
up to 15 percent.

LinpackAs explained by its author, Jack Dongarra4 from the University of Tennessee(previously Argonne National Laboratory),
Linpack didn’t originate as a benchmark.
When first published in 1976, it was just acollection (a package, hence the name) of
linear algebra subroutines often used in
Fortran programs. Dongarra, who collects
and publishes Linpack results, has now
distilled what was part of a “real life”
program into a benchmark that is distributed in various version^.^The program operates on a large matrix
67

(two-dimensional array); however, the inner subroutines manipulate the matrix as a
one-dimensional array, an optimization
customary for sophisticated Fortran programming. The matrix size in the version
distributed by standard mail servers is 100
x 100 (within a two-dimensional arraydeclared with bounds 200), but versions
for larger arrays also exist.
The results are usually reported in millions of floating-point operations per second (Mflops); the number of floating-point
operations the program executes can bederived from the array size. This terminol-ogy means that the nonfloating-point operations are neglected or, stated anotherway, that their execution time is includedin that of the floating-point operations.
When floating-point operations become
increasingly faster relative to integer operations, this terminology becomes some-

Tables covering more than one benchmark
Table A. Statement distribution in percentages. *Statement

Dhry stone

Whetstone

Assignment of a variable
Assignment of a constant
Assignment of an expression (one operator)
Assignment of an expression (two operators)
Assignment of an expression (three operators)
Assignment of an expression (>three operators)

20.4
11.7
17.5
1.0
1 .0

14.4
8.2
1.4
24.3
1.6
6.8

One-sided if statement, “then” part executedOne-sided if statement, “then” part not executedTwo-sided if statement, “then” part executedTwo-sided if statement, “else” part executed

2.9
3.9
4.9
1.9

0.5
0.1
4.0
4.0

For statement (evaluation)
Goto statement
Whilehepeat statement (evaluation)
Switch statement
Break statement

6.8

17.3
0.5

Return statement (with expression)

4.9

Call statement (user procedure)Call statement (user function)
Call statement (system procedure)Call statement (system function)

9.7
4.9
1.0

11.9

1
.
0

4.7
100

Linpackkaxpy

48.5

2.2

49.3

4.9
I .0
1 .0

100

100

*Because of rounding, all percentages can add up to a number slightly below or above 100.

Table C. Operand data-type distribution in percentages.
Operand Data Type
IntegerChar
Float/double
Enumeration
BooleanArray
StringPointer

68

Dhry stone

Whetstone

LinpacWsaxpy

57.0
19.6

55.7

67.2

44.3

32.8

~

10.9
4.2
0.8
2.3
5.3
100

100

~

100

COMPUTER

what misleading.
For Linpack, it is important to knowwhat version is measured with respect tothe following attribute pairs:
Single/double -Fortran single precision or double precision for the floating-point data.
Rolled/unrolled --In the unrolled version, loops are optimized at the source
level by “loop unrolling”: The loop index(say, i) is incremented in steps of four, andthe loop body contains four groups ofstatements, for indexes i, i + I , i + 2, and i

Table B. Operator distribution in percentages.
Operator

Dhrystone

Whetstone

Linpack/saxpy+ (intkhar)

21.0
5.0
2.5
0.8
29.3

11.9
6.0
6.0

14.1

- (int)* (int)I (int)Integer arithmetic~+ (float/double)- (floddouble)* (float/double)/ (float/double)
Floating-point arithmetic<, <= (incl. loop control)
Other relational operators
Relational

23.9

14.1

14.9
2.1
9.3

14.1

4.6

~

10.1
11.7
21.8

Logical3.3

Indexing (one-dimensional)
Indexing (two-dimensional)
Indexing

5.9
3.4
9.3

Record selection
Record selection via pointerRecord selection

15.1

Address operator (C)Indirection operator (C)C-specific operators

5.0
8.4
13.4

Total

100

14.1
~

30.9

28.2

10.7

14.5

2.8

0
.
6

13.5

15.1

Size, procedureprofile, and languagefeature distribution. The Linpack data

0.2
24.5

42.3

24.5

42.3

7.6
22.1

~

3.6
3.6
7.2

IO0

100

Tab%D. Operand locality distribution in percentages.
~iOperand Locality
Local
GlobalParameter (value)
Parameter (reference)
Function result
Constant

December 1990-__

+ 3. This technique saves execution time
for most machines and compilers; however, more sophisticated vector machines,
where loop unrolling is done by the compiler generating code for vector hardware,
usually execute the standard (rolled) version faster.
Coded BLASIFortran BLAS - Linpack relies heavily on a subpackage ofbasic linear algebra subroutines (BLAS).
Coded BLAS (as opposed to Fortran BLAS)
means that these subroutines have beenrewritten in assembly language. Dongarra
has stopped collecting and publishing results for the coded BLAS version and requires that only the Fortran version of thesesubroutines be used unchanged. However,some results for coded BLAS versions arestill cited elsewhere. Computing the execution-time ratio between coded BLAS and
Fortran BLAS versions for the same machine offers insights about the Fortrancompiler’s code optimization quality: Forsome machines the ratio is 1.2 to I ; forothers it can be as high as 2 to 1 .

Dhrystone

Whetstone

Linpacklsaxpy

48.7
8.3
10.6
6.8
2.3
23.4
IO0

0.4
56.3
18.6
1.9
I .3
21.6
IO0

49.5
17.0
24.6
8.8
100

reported here is for the rolled version, single
precision, with Fortran BLAS; code sizes
have been measured with VAX Unix BSD
4.3 Fortran.
The static code length for all subprograms
is 4,537 bytes. The length for individual
subprograms varies between 11 1 and 1,789
bytes; the most heavily used subprogram,
saxpy, is 234 bytes long. Data size, in thestandard version, is dominated by an arrayof 100 x 100 real numbers. For 32-bit
machines, this means that with single precision, 40 Kbytes are used for data (80Kbytes with double precision).
Table 2 shows the distribution of execution time in the various subroutines. The
most important observation from the table
is that more than 75 percent of Linpack’s
execution time is spent in a 15-line subroutine (called saxpy in the single-precision version and daxpy in the doubleprecision version). Dongarra4 reports thaton most machines the percentage is evenhigher (90 percent). Because of this extreme concentration of execution time inthe saxpy subroutine, and because of thetime-consuming instrumentation method
for obtaining the measurements, languagefeature distribution has been measured
only for the saxpy subroutine (rolled
version).
Table A in the sidebar shows that very
few statement types (assignment withmultiplication and addition, and “for”
69

Table 2. Procedure profile for Linpack.
Procedure

Percent

What is done there

Main programmatgensgefa=xPY
isamaxMiscellaneous
User code

0.0
13.8
6.2
77.1
1.6
1.2
100

y[il = y [ i ] + a*x[i]Library functions~

0.0

statements) make up the bulk of thesubroutine and, therefore, of Linpack itself. The data is mostly reference parameters (array values) or local variables(indexes); there are hardly any global
variables.

Linpack characteristics. To interpret
performance characterizations by Linpack
Mflops, it helps to know the benchmark’s
main characteristics:
As expected for a numeric benchmark,Linpack has a high percentage of floatingpoint operations, though only a few are
actually used. For example, the program
has no floating-point divisions. In striking
contrast to Whetstone, no mathematical
functions are used at all.
The execution time is spent almost
exclusively in one small function. Thismeans that even a small instruction cache
will show a very high hit rate.
Contrary to the high locality for code,
Linpack has a low locality for data. A
larger size for the main matrix leads depending on the cache size -to significantly more cache misses and therefore toa lower Mflops rate. So, in making comparisons, it is important to know whether
Linpack Mflops for different machines havebeen computed using the same array dimensions. Also, Linpack can be highly
sensitive to the cache configuration: A
different array alignment (201 x 200 instead of 200 x 200 for the global arraydeclaration) can lead to a different mappingof data to cache lines and therefore to aconsiderably different execution time. The
program, as distributed by the standard
mail servers, delivers Mflops numbers fortwo choices of leading dimension, 200 and201; we can assume that manufacturers
report the better number.
70

Dhrystone
As the name indicates, Dhrystone wasdeveloped much as Whetstone was; it is asynthetic benchmark that I published in
1984. The original language of publication
is Ada, although it uses only the Pascal
subset of Ada and was intended for easytranslation to Pascal and C. It is used mainlyin the C version.
The basis for Dhrystone is a literature
survey on the distribution of source language
features in nonnumeric, system-type programming (operating systems, compilers,
editors, and so forth). In addition to the
obvious difference in data types (integral
versus floating-point), numeric and systemtype programs have other differences, too:
System programs contain fewer loops,
simpler computational statements, and more
“if’ statements and procedure calls.
Dhrystone consists of 12 procedures
included in one measurement loop with 94
statements. During one loop (one Dhrystone), 101 statements (103 in the C version)
are executed dynamically. The results areusually given in Dhrystones per second.
The program (currently Version 2.1) has
been distributed mainly through Usenet,
the Unix network; I also make it available
on a floppy disk. Rick Richardson has
collected and posted results for the Dhrystone benchmark regularly to Usenet (thelatest list of results is dated April 29,1990).

Size, procedure profile, and languagefeature distribution. The static length of
the Dhrystone measurement loop, as compiled by the VAX Unix (BSD 4.3) Ccompiler, is 1,039 bytes. Table 3 shows the
distribution of execution time spent in its
subprograms.
The percentage of time spent in string
operations is highly language dependent; itdrops to 10 percent instead of 16 percent if
the Pascal (or Ada) version is used (measurement for Berkeley Unix 4.3 Pascal).
On the other hand, the number is higher for
newer RISC machines with optimizing
compilers, mainly because they spend muchless time in procedure calls than the VAX.
Consistent with usage in system-type
programming, arithmetic expressions are
simpler than in the other benchmarks; there
are more “if’ statements and fewer loops.
Dhrystone was the first benchmark to
explicitly consider the locality of operands:
Local variables and parameters are used
more often than global variables. This is
not only consistent with good software
engineering practices but also importantfor modern CPU architectures (RISC architectures). On older machines with few
registers, local variables and parameters
are allocated in memory in the same way as
global variables; on RISC machines they
typically reside in registers. The resulting
difference in access time is one of themost important advantages of RISC architectures.

Dhrystone characteristics. Familiarity
with the benchmark’s main characteristics,
described below, is important when interpreting Dhrystone performance characterizations.
As intended, Dhrystone contains nofloating-point operations in its measurement
loop.
A considerable percentage of execution
time is spent in string functions; this number
should have been lower. In extreme cases(MIPS architecture and C compiler), thisnumber goes up to 40 percent.
Unlike Whetstone, Dhrystonecontains
hardly any loops within the main measurement loop. Therefore, for microprocessors with small instruction caches (below 1,000 bytes), almost all instructionaccesses are cache misses. But as soon as
the cache becomes larger than the measurement loop, all instruction accesses arecache hits.
Only a small amount of global data is
manipulated, and the data size cannot be
scaled as in Linpack.
No attempt has been made to thwart
optimizing compilers. The goal was for the
program to reflect typical programmingstyle; it should be just as optimizable asnormal programs. An exception is the
optimization of dead-code removal. Since
in Version 1 the computation results were
not printed or used, optimizing compilers
were able to recognize many statements as
COMPUTERdead code and suppress code generation
for these statements. In Version 2, this has
been corrected.

Table 3. Dhrystone procedure profile.

Ground rules for Dhrystone number
comparisons. Because of Dhrystone’s
peculiarities, users should be sure to observe
certain ground rules when comparing
Dhrystone results. First, the version usedshould be 2.1 ;the earlier version, 1.1, leaves
too much room for distortion of results bydead-code elimination.
Second, the two modules must be compiled separately, and procedure merging(in-lining) is not allowed for user procedures. ANSI C, however, allows in-liningof library routines (relevant for string
routines in the C version of Dhrystone).
Third, when processors are compared,
the same programming language must be
usedon both. For compilers of equal quality,
Pascal and Ada numbers can be about 10
percent better because of the string semantics. In c , the length of a string isnormally not known at compile time, andthe compiler needs -at least for the string
comparison statement in Dhrystone - togenerate code that checks each byte for thestring terminator byte (null byte). With
Pascal and Ada the compiler can generate
word instructions (usually in-line code) for
the string operations.
Therefore, for a meaningful comparisonof C-version results, it helps to be able toanswer certain questions:(1) Are the string routines written inmachine code?
(2) Are the string routines implemented
as in-line code?
(3) Does the compiler use the fact thatin the “strcpy” statement the source operand has a fixed length? If it does (legal
according to ANSI C), this statement canbe compiled in the same way as a record
assignment, which can result in considerable savings.
(4) Is a word alignment assumed for thestring routines? This is acceptable for the
strcpy statement only, not for the “strcmp”
statement.
Language systems are allowed to optimize for cases 1 through 3 above, just as
they can for programs in general. For processor comparisons, however, it is important that the compilers used apply the sameamount of optimization; otherwise, optimization differences may overshadow CPU
speed differences. This usually requires aninspection of the generated machine codeand the C library routines.
December 1990

Procedure

Percent

Main program
User procedures
User code

18.3
65.7
84.0
8.0

strcpy8.1

strcmp~Library functions

I

Total

What is done there

String copy
(string constant)String comparison(string variables)

16.1
100

Other benchmarks
In addition to the most often quoted
benchmarks explained above, several otherprograms are used as benchmarks, including
Livermore Fortran Kernels,
Stanford Small Programs Benchmark
Set,
EDN benchmarks,
Sieve of Eratosthenes,
Rhealstone, andSPEC benchmarks.
These range from small, randomly chosen
programs such as Sieve, to elaborate
benchmark suites such as Livermore Fortran Kernels and SPEC benchmarks.

Livermore Fortran Kernels. The Livermore Fortran Kernels, also called theLawrence Livermore Loops, consist of 24
kernels, or inner loops, of numeric computations from different areas of the physicalsciences. The author, F.H. McMahon of
Lawrence Livermore National Laboratory,
has collected them into a benchmark suiteand has added statements for time measurement. The individual loops range froma few lines to about one page of sourcecode. The program is self-measuring andcomputes Mflops rates for each kernel, forthree different vector lengths.
As we might expect, these kernels contain many floating-point computations anda high percentage of array accesses. Several kernels contain vectorizable code; somecontain code that is vectorizable if rewritten.
(Feo6 provides a detailed discussion of theLivermore Loops.) McMahon characterizes
the representativity of the Livermore Loops
as follows:

The net Mflops rate of many Fortran programs
and work loads will be in the subrangebetween
the equi-weighted harmonic and arithmetic
means, depending on the degree of code
parallelism and optimization. The Mflops
metric provides aquick measure of the average
efficiency of a computer system, since its
peak computing rate is well known.

Stanford Small Programs Benchmark
Set. Concurrent with development of thefirst RISC systems at Stanford University
and the University of California, Berkeley,John Hennessy and Peter Nye at Stanford’s
Computer Systems Laboratory collected a
set of small programs (one page or less ofsource code for each program). These
programs became popular mainly because
they were the basis for the first comparisonsof RISC and CISC processors. They havenow been packed into one C program
containing eight integer programs - Permutations, Towers of Hanoi, Eight Queens,
Integer Matrix Multiplication, Puzzle,
Quicksort, Bubble Sort, and Tree Sort and two floating-point programs - Floating-point Matrix Multiplication and Fast
Fourier Transformation.
Characteristics of the individual programs
vary; most contain a high percentage ofarray accesses. There seems to be no official publication of the source code. The
only place I have seen the C code in print isin a manufacturer’s performance report.
There is no standardized method forgenerating an overall figure of merit fromthe individual execution times. In one
version, a driver program assigns weights
between 0.5 and 4.44 to the individual
execution times. Perhaps a better alternative, used by Sun and MIPS, is to compute
the geometric mean of the individual programs’ execution times.
71

Table 4. SPEC benchmark Droerams.
~~

~~~~

~

~

~

~

~~~~~~~

Short Characterization

Language

Main Data Types

gcc
espresso
spice 2g6
doduc
nasa7
li
eqntottmatrix300
fPPPP
tomcatv

GNU C compiler
PLA simulator
Analog circuit simulation
Monte Carlo simulation
Collection of several numerical “kernels”
Lisp interpreter
Switching-function minimization, mostly sorting
Various matrix multiplication algorithms
Maxwell equations
Mesh generation, highly vectorizable

C
C
FortranFortranFortranC
C
FortranFortranFortranInteger
Integer
Floating point
Floating point
Floating point
Integer
Integer
Floating point
Floating point
Floating point

EDN benchmarks. The program collection now known as the EDN benchmarks was developed by a group at Carnegie Mellon University for the MilitaryComputer Family project. EDN published
it in 1981. Originally, the programs were
written in several assembly languages (LSI11/23, 8086, 68000, and ZSOOO); the intention was to measure the speed of microprocessors without also measuring thecompiler’s quality.
A subset of the original benchmarks isoften used in a C version:
Benchmark E: String search
Benchmark F: Bit testhetheset
Benchmark H: Linked list insertion
Benchmark I: Quicksort
Benchmark K: Bit matrix transformation
This subset of the EDN benchmarks has
been used in Bud Funk’s comparison of
RISC and CISC processor^.^ There seems
to be no standard C version of the EDN
benchmarks; the programs are disseminatedinformally.

Sieve of Eratosthenes. One of the mostpopular programs for benchmarking small
PCs is the Sieve of Eratosthenes, sometimes called “Primes.” It computes all prime
numbers up to a given limit (usually 8,192).
The program has some unusual characteristics. For example, 33 percent of the dynamically executed statements are assignments of a constant; only S percent areassignments with an expression at the righthand side. There are no “while” statements
and no procedure calls; S O percent of thestatements are loop control evaluations.
72~~

Acronym

All operands are integer operands, and 58
percent of them are local variables.
The program is mentioned here not because it can be considered a good benchmark but because, as one author put it,“Sieve performance of one compiler overanother has probably sold more compilers
for some companies than any otherbenchmark in history.”

SPEC benchmarks. Probably the mostimportant current benchmarking effort isSPEC - the systems performance evaluation cooperative effort. It started because
benchmarking experts at various companies
felt that most previously existing benchmarks (usually small programs) were inadequate. Small benchmarks can no longer
be representative for real programs when it
comes to testing the memory system, because with the growing size of cachememories and the introduction of on-chipcaches for high-end microprocessors, thecache hit ratio comes close to 100 percent
for these benchmarks. Furthermore, once asmall program becomes popular as abenchmark, compiler writers are inclined(or forced) to “tweak” their compilers into
optimizations particularly beneficial to thisbenchmark - for example, the string optimizations for Dhrystone.
SPEC’S goal is to collect, standardize,and distribute large application programs
that can be used as benchmarks. This is a
nontrivial task, since realistic programs
previously used in benchmarking (for example, the Unix utilities “yacc” or “nroff’)
often require a license and are therefore notfreely distributable.
The founding members of SPEC were
Apollo, Hewlett-Packard, MIPS, and Sun;

subsequently, AT&T, Bull, CDC, Compaq, Data General, DEC, Dupont, Fujitsu,
IBM, Intel, Intergraph, Motorola, NCR,
Siemens Nixdorf, Silicon Graphics, Solbourne, Stardent, and Unisys became
members.
In October 1989, SPEC released its first
set of 10 benchmark programs. Table 4
contains only a rough characterization of
the programs; J. Uniejewski* provides amore detailed discussion. Because a licensemust be signed, and because of its size
(lS0,OOO lines of source code), the SPECbenchmark suite is distributed via magnetic
tape only.
Results are given as performance relative
to a VAX 1 U780 using VMS compilers.
Results for several computers of SPECmember companies are contained in theregular SPEC Newsletter (see Additional
reading and address information). A comprehensive number, the “SPECmark,” isdefined as the geometric mean of the relative performance of the 10 programs.
However, SPEC requires a reporting form
that gives, in addition to the raw data, therelative performance for each benchmark
program separately. Thus, users can selectthe subset of performance numbers forwhich the programming language and/or
the application area best matches theirapplications.

Non-CPU influences inbenchmarkperformance
In trade journals and advertisements,
manufacturers usually credit good benchCOMPUTERmark numbers only to the hardware system’s speed. With microprocessors, this is
reduced even more to the CPU speed.
However, the preceding discussion makes
it clear that other factors also have an
influence -for example, the programming
language, the compiler, the runtime libraryfunctions, and the memory and cache size.

Table 5. Performance ratio for different languages (larger is better, C = 1): Stanford programs.

I

ProeramBliss

C

Pascal

SearchSieve
PuzzleAc kermann

1.24
0.63
0.77
1.20

1.o
1.o
1.o
1.o

0.70
0.80
0.73
0.80

.o

1.32

Programming-language influence.
Table 5 (numbers from Levy and Clark9
and my own collection of Dhrystone results)
shows the execution time of several programs on the same machine (VAX, 1982
and 1985). Properties of the languages(calling sequence, pointer semantics, andstring semantics) obviously influence execution time even if the source programs
look similar and produce the same results.

Compiler influence. Table 6, taken from
the MIPS Performance Brief,lo gives
Dhrystone results (as of January 1990) for
the MIPS M/2000 with the MIPS C compiler cc2.0. The table shows how the different levels of optimization influence execution time.
Note that optimization “ 0 4 ” performs
procedure in-lining, an optimization notconsistent with the ground rules and included in the report for comparison only.
On the other hand, the “strcpy” optimization for Dhrystone is not included in any of
the optimization levels for the MIPS C
compiler. If it is used, the Dhrystone rateincreases considerably.
Runtime library system. The role of
the runtime library system is often overlooked when benchmark results are compared. As apparent fromTable 1, Whetstone
spends 40 to 50 percent of the execution
time in functions of the mathematicalsubroutines library. The C version of
Dhrystone spends 16 percent of the execution time in the string functions (VAX,
Berkeley Unix 4.3 C); with other systems,
the percentage can be higher.
Some systems have two flavors of themathematical floating-point library: The
first is guaranteed to comply with the IEEE
floating-point standard; the second is fasterand may give less accurate results undersome circumstances. Customers who must
rely on the accuracy of floating-pointcomputations should know which library
was used for benchmark measurements.
Cache size. It is important to look for thebuilt-in performance boost when the cache
size reaches the relevant benchmark size.
Depending on the difference between acDecember 1990

Dhrystone (1.1)

1

I

Ada

1.02

Table 6. Compiler optimization levels in Dhrystonedsec.

I Optimization Level
No opt., no “register” attribute
No opt., with “register” attribute
Optimization “0,”no “register” attribute
Optimization “0,”with “register” attribute
Optimization “ 0 3 ”
Optimization “ 0 4 ”

cess times for the cache and the main memory, cache size can have a considerable
effect.
Table 7 summarizes the code sizes (sizeof the relevant procedureshnner loops) anddata sizes (of the main array) for some
popular benchmarks. All sizes have beenmeasured for the VAX 11 with the Unix
BSD 4.3 C compiler, with optimization“-0”(code compaction). Of course, thesizes will differ for other architectures andcompilers. Typically, RISC architectures
lead to larger code sizes, whereas the data
size remains the same.
If the cache is smaller than the relevantbenchmark, reordering the code can, forsome benchmarks and cache configurations,lead to considerable savings in execution
time. Such savings have been reported for
Whetstone on MC 68020 systems (reordering the source program) as well as for
Dhrystone on NS 32532, where just a different linkage order can lead to a differenceof up to 5 percent in execution time. It is
debatable whether the “good case” or the“bad case” better represents the system’strue characteristics. In any event, customers should be aware of these effects andknow when the standard order of the code
hasbeenchanged.

Iv. 1.1v . 2.1

30,70032,60039,70039,70043,10046,7003 1,000
32,40036,70036,70039,40043,200

Table 7. Size in bytes for some popularbenchmarks.

I Program

Code

Data

-

Whetstone
256
Dhrystone
1,039
234
Linpack (saxpy)( 1OOx 100 version)
Sieve
160
(standard version)
174
Quicksort(standard version)
Puzzle1,642
Ac kermann
52

I

16
40,000
8,192
20,000511

Small, synthetic
benchmarks versus
real-life programs
It should be apparent by now that withthe advent of on-chip caches and sophisticated optimizing compilers, small benchmarks gradually lose their predictive value. This is why current efforts like SPEC’S
73

Obtaining benchmark sources via e-mail
Most of the benchmarks discussed in this article can be obtained via electronic
mail from several mail servers established at large research institute^.^ The major mail servers and their electronic mail addresses are shown below. Users can
get information about the use of these mail servers by sending electronic mailconsisting of the line “send index” to any of the mail servers.
The SPEC benchmarks are available only via magnetic tape.

North America
uucp:Internet:
Internet:

uunet!research!netlib
netlib@research.att.com
netlibOornl.gov

Murray Hill, New Jersey

EuropeEUNEThucp:
Internet:
EARN/Bitnet:
X.400:

nac!netlib
netlib @ nac.no
netlib%nac.no@norunix.bitnet
s=netlib; o=nac; c=no;

Oslo, Norway

netlib@draci.cs.uow.edu.au

Univ. of Wollongong,
NSW. Australia

Pacific
Internet:

Oak Ridge, Tennessee

References
1. D.A. Patterson, “Reduced Instruction-Set
Computers,” Comm. ACM, Vol. 28, No. I ,Jan. 1985, pp. 8-21.
2. 0. Serlin, “MIPS, Dhrystones, and Other
Tales,” Daramarion, June 1986, pp. 112118.
3. B.A. Wichmann, “Validation Code for the
Whetstone Benchmark,” Tech. Report NPLDITC 107/88, National Physical Laboratory, Teddington, UK, Mar. 1988.
4. J.J. Dongarra, “The Linpack Benchmark:An Explanation,” in Evaluating Supercomputers, Aad J. Van der Steen, ed., Chapman
and Hall, London, 1990, pp. 1-21.
5. J. Dongama and E. Grosse, “Distribution of

Mathematical Software viaElectronic Mail,”Comm. ACM, Vol. 30, No. 5, May 1987, pp.
403-407.
6. J.T. Feo, “An Analysis of the Computationaland Parallel Complexity of the Livermore
Loops,” Parallel Computing, Vol. 7, No. 2,
June 1988, pp. 163-185.
7. B. Funk, “RISC and CISC Benchmarks andInsights,” Unisys World, Jan. 1989, pp. 11-

14.
activities concentrate on collecting large,
real-life programs. Why, then, should thisarticle bother to characterize in detail these“stone age” benchmarks? There are several
reasons:

example, it won’t be as easy to tweak
compilers for the SPEC benchmarks as it isfor the small benchmarks; but if it happens,it also will be harder to detect.

8. J. Uniejewski, “Characterizing System( 1 ) Manufacturers will continue to usethem for some time, so the trade press willkeep quoting them.
(2) Manufacturers sometimes base their
MIPS rating on them. An example is IBM’s(unfortunate) decision to base the published (VAX-relative) MIPS numbers for
the IBM 6000 workstation on the old 1 . I
version of Dhrystone. Subsequently, DEC
and Motorola changed the MIPS computation rules for their competing products,
also basing their MIPS numbers on Dhrystone 1.1.
(3) For investigating new architecturaldesigns -via simulations, for example the benchmarks can provide a useful first
approximation.
(4) For embedded microprocessors withno standard system software (the SPEC
suite requires Unix or an equivalent operating system), nothing else may be available.
( 5 ) We can expect that larger benchmarks will not be completely free of distortions from unforeseen effects either. Experience gained with smaller benchmarks
can help us be aware of such effects. For

dvice for users looking at benchmark numbers to characterize
machine performance should begin with a warning not to trust MIPS numbers unless their derivation is clearly explained. Here are some other things towatch for:9. H. Levy and D.W. Clark, “On the Use of
Benchmarks for Measuring System Performance,” Computer Architecture News, Vol.
10, No. 6 , Dec. 1982, pp. 5-8.

74

Check whether Mflops numbers relate
to a standard benchmark. Does thisbenchmark match your applications?
Know the properties of the benchmarks
whose results are advertised.
Be sure you know all the relevant facts
about your system and the manufacturer’s
benchmarking system. For hardware thisincludes clock frequency, memory latency, and cache size; for software it includes
programming language, code size, data size,
compiler version, compiler options, andruntime library.
Check benchmark code listings to makesure apples are compared with apples andthat no illegal optimizations are applied.
Ask for a well-written performance
report. Good companies provide all relevant details. W

Performance Using Application-Level
Benchmarks,”Proc. Buscon,Sept. 1989,pp.
159-167. Partial publication in SPEC
Newsletter, Vol. 2, No. 3, Summer 1990, pp.
3-4.

10. MIPS Computer Systems, Inc., Performance
Brief; CPUBenchmarks,Issue 3.9, Jan. 1990,

p. 35.

Additional reading andaddress information
Following are the main reference sources
for each of the benchmarks discussed inthis article, together with a short characterization. A contact person is identified
for each of the major benchmarks so thatreaders can get additional information. For
information about access to the benchmark
sources via electronic mail, see the sidebar“Obtaining benchmark sources via e-mail.”
COMPUTER

Whetstone
Curnow, H.J., and B.A. Wichmann, “A SynthelicBenchmark,”TheComputerJ.,Vol. 19,No. 1,
1976, pp. 43-49. Original publication, explanation of the benchmark design, program (Algol60) in the appendix.
Wichmann, B.A., “Validation Code for the
Whetstone Benchmark,” see Reference 3. Discussion of comments made to the original program, explanation of the revised version. Paper
contains a program listing of the revised version, in Pascal, including checks for correct
execution.
Contact: Brian A. Wichmann, National PhysicalLaboratory, Teddington, Middlesex, England
TWl1 OLW; phone 44 (81) 943-6976, fax 44(8 1) 977-7091, Internet baw@seg.npl.co.uk.
Registration of other versions: J.B. Souter,
BenchmarkRegistration, BSI-QAS, PO Box 375,
Milton Keynes, Great Britain MK14 6LL.

Linpack

Dongarra, J.J.,etal., LinpackUsers’Guide, SIAM
Publications, Philadelphia, Pa., 1976. Original
publication (not yet as a benchmark), contains
the benchmark program as an appendix.
Dongarra, J.J., “Performance of Various Computers Using Standard Equations Software in a
Fortran Environment,” Computer Archirecrure
News, Vol. 18, No. 1 , Mar. 1990, pp. 17-31.
Latest published version of the regularly maintained list of Linpack results, rules for Linpackmeasurements.
Dongarra, J.J., “The Linpack Benchmark: An
Explanation,” see Reference 4. Explanation of
Linpack, guide to interpretation of Linpack results.
Contact: Jack J. Dongarra, Computer Science
Dept., Univ. of Tennessee, Knoxville, TN 379961301; phone (615) 974-8295, fax (615) 9748296, Internet dongarra@cs.utk.edu.

of Dhrystone are compared; reiteration of measurement rules.
Contact: Reinhold P. Weicker, Siemens Nixdorf
Information Systems, STM OS 3 2 , Otto-HahnRing 6, W-8000 Munchen 83, Germany; phone
49 (89) 636-42436, fax 49 (89) 636-48008, Internet: weicker@ztivax.siemens.com; Eunet:
weicker%ztivax.uucp @unido.uucp.
Collection of results: Rick Richardson, PC Research, Inc., 94 Apple Orchard Dr., Tinton Falls,
NJ07724;phone(201) 389-8963.e-mail (UUCP)...!uunet!pcrat!rick.

Livermore Fortran Kernels

Feo, J.T., “An Analysis of the Computationaland Parallel Complexity of the Livermore Loops,”
see Reference 6. Analysis of the Livermore
Fortran Kernels with respect to the achievableparallelism.

Weicker, R.P., “Dhrystone Benchmark: Rationale for Version 2 and Measurement Rules,”
SIGPlan Notices, Vol. 23, No. 8, Aug. 1988, pp.
49-62. Version 2.0 of Dhrystone (in C), measurement rules. For the Ada version, a similar
article appeared in Ada Letters, Vol. 9, No. 5,
July 1989, pp. 60-82.
Weicker, R.P., “Understanding Variations in
Dhrystone Performance,” Microprocessor Reporr, Vol. 3, No. 5, May 1989, pp. 16-17. Whatcustomers should know when C-version results

December 1990

SPEC benchmarks“Benchmark Results,”SPECNewsletrer, Vol. I ,No. 1, Fall 1989, pp. 1-15. First published list of
results, in the report form required by SPEC.
Uniejewski, J., “Characterizing System Performance Using Application-Level Benchmarks,”
see Reference 8. This paper includes a shortcharacterization of each SPEC benchmark program.
Contact: SPEC -System Performance Evaluation Cooperative (Kim Shanley, Secretary), c/o
Waterside Associates, 39 150PaseoPadre Pkwy.,
Suite 350, Fremont, CA 94538; phone (415)792-2901, fax (41 5) 792-4748, Internet
shanley @cup.portal.com.

McMahon, F.H., “The Livermore Fortran Kernels: A Computer Test of the Numerical Performance Range,” Tech. Report UCRL-53745,
Lawrence Livermore National Laboratory,
Livermore, Calif., Dec. 1986, p. 179. Original
publication ofthe benchmark with sampleresults.
McMahon, F.H., “The Livermore Fortran Kernels Test of the Numerical Performance Range,”
in Performance Evaluarion of Supercomputers,
J.L. Martin, ed., North Holland, Amsterdam,
1988, pp. 143.186. Reprint of main part of theoriginal publication.
Contact: Frank H. McMahon, Lawrence Livermore National Laboratory, L-35, PO Box 808,
Livermore, CA 94550; phone (415) 422-1647,
Internet mcmahon@ocfmail.ocf .Ilnl.gov,

Stanford Small Programs Benchmark Set

Appendix 2 -Stanfordcomposite Sourcecode,
Appendix to “Performance Report 68020/68030
32-bit Microprocessors,” Motorola, Inc., BR705/
D, 1988, pp. A2-1 - A2-IS. This is the only
place I have seen this benchmark in print; it isnormally distributed via informal channels.

Dhrystone

Weicker, R.P., “Dhrystone: A Synthetic Systems Programming Benchmark,” Comm. ACM,
Vol. 27, No. I O , Oct. 1984, pp. 1,013-1,030.
Original publication, literature survey on theuse of programming language features, basestatistics and benchmark program in Ada.

gram listings in Pascal, C, Forth, Fortran IV,
Basic, Cobol, Ada, and Modula-2.

EDN benchmarks

Grappel, R.D., and J.E. Hemenway, “A Tale of
Four pPs: Benchmarks Quantify Performance,”
EDN, Apr. 1, 198I , pp. 179-265. Original publication with benchmarks described in assembler(code listings for LSI-11/23, 8086, 68000, and28000).

Reinhold P. Weicker is a senior staff engineer

with Siemens Nixdorf Information Systems AG
in Munich, Germany. His research interests include performance evaluation with benchmarks
and its relation toCPU architecture and compiler
code generation. He wrote the often-used
Dhrystone benchmark while working on the
CPU architecture team for the i80960 microprocessor. Previously, he performed research intheoretical computer science at the Universityof Hamburg, Germany, and was a visiting assistant professor at Pennsylvania State University.
Weicker received a diploma degree in mathematics and a PhD in computer science from the
University of Erlangen-Numberg. He is a memberof the IEEE Computer Society, the ACM, and
the Gesellschaft fur Informatik.

Patstone, W., “16-bit-yP Benchmarks - An
Update withExplanations,”EDN, Sept. 16,1981,
pp. 169.203. Discussionof results, updatedcode
listings (assembler).
Sieve

Gilbreath, J., and G. Gilbreat’l, “Eratosthenes
Revisited,” B y t e , Jan. 1983, pp. 283-326. Pro-

The author can be contacted at Siemens
Nixdorf Information Systems AG, Otto-HahnRing 6, W-8000 Munchen 83, Germany; Internet: weicker@ztivax.siemens.com; Eunet:
weicker%ztivax.uucp@unido.uucp.

15