From: clc5q@hemlock.cs.Virginia.EDU (Clark L. Coleman)
Subject: Re: Geometric Mean or Median
Organization: University of Virginia Computer Science Department
Distribution: comp.benchmarks
Date: Wed, 26 Aug 1992 16:02:40 GMT

In article <1992Aug23.114309.3643@nosc.mil> aburto@nosc.mil (Alfred A. Aburto) writes:
>However, even the SPEC results will vary, as will any measure of 
>performance, when the underlying test parameters change. See SunWorld 
>magazine, Mar 1992, pg 48, "SPEC: Odyssey of a benchmark", where 
>geometric mean SPECmark ratings of 17.8, 20.0, 20.8, and 25.0 were 
>measured on the same machine using different compilers.

This is given as an example of "noisiness" or "error proneness" in SPEC,
but SPEC was intended to measure the SYSTEM, including the compilers. If
one vendor has better compilers than another, it matters to the end users.
Conversely, trying to benchmark the raw hardware in some way that filters
out compiler differences would not be interesting to people who have to
purchase the systems and use the compilers, not just the hardware.

The only real question in my mind about SPEC's approach is allowing vendors
to use different compiler switch settings for each individual benchmark,
if it produces better numbers. I don't think many users can compile every
program that many different times and run timing tests on each one.

>I have heard alot of Dhrystone 1.1 bashing in the past. I tried to 
>understand just how 'bad' Dhrystone was several years ago (it seems)
>by correlating Dhrystone 1.1 results with SPECint89 results. I had  
>20 or so data points to work with. I thought perhaps there would be
>little correlation and that Dhrystone really needed to be cast out 
>as a measure of performance, because of the greater confidence placed 
>in the SPEC results. Instead, they were highly correlated (0.92 
>correlation or so).  Well, I had to revise my thinking.

This kind of bogus correlation was debunked long ago by SPEC. As soon as
more data points are added, the correlation gets worse. Try adding the
SparcStation 10 numbers to your test, for example. For that matter, just
look at the HP 9000/710 versus the HP 9000/720.  The only differences in
the hardware are the larger caches on the 720.  Since the 710 has large
enough caches for the Dhrystone code, but not for some SPECint codes, it
produces the same Dhrystones as the 720 but significantly lower SPECint.

Similar poor correlations will be obtained for two different systems
with very different cache sizes. Compare the HP9000/720 to a smaller
cache machine like an IBM RS/6000 or Sun SS2. For example, here are some
Spring, 1991, numbers:

                 SPECint89    Dhrystone 1.1 MIPS      MIPS/SPECint89
                 ---------    ------------------      --------------
HP 9000/720       39.0            57                      1.46
DEC 5000/200      19.0            24.2                    1.27
IBM RS6000/550    34.5            56                      1.62

If I didn't have SPECint89 numbers, but wanted to derive them from available
Dhrystone MIPS numbers, the third column above would indicate that I have
a tough job ahead of me.

I would expect there would be some reasonable relationship between any two
integer benchmarks. That doesn't make one a good substitute for the other,
however. I think there are specific reasons why Dhrystone is not as good
at predicting integer performance on my real applications as SPECint is.
They have been listed in this group before. And, as I pointed out a few
months ago, superscalar machines change the results radically. The Dhrystone
code has less instruction to instruction data dependence than normal code,
as it was designed in a scalar processor era to execute a certain group
of instructions according to a measured frequency. So, for example, you
might see statements like:

a = b + c;   /* do an addition so we can have so many additions in the loop */
x = y - z;   /* similarly, do a subtraction */

There is no data dependence, so a superscalar machine can just do them
simultaneously.  This happens less frequently in real code than in Dhrystone,
so SPECint92 and Dhrystone 1.1 are destined to diverge drastically in the
near future. For example:


                 SPECint92      Dhrystone MIPS    MIPS/SPECint92
                 ---------      --------------    --------------
Sun IPX           21.8             28.5             1.31
Sun SS10 (36MHz)  44.2             86.1             1.95

So, does moving from an IPX to a 36MHz SS10 improve your integer performance
by 103%, or by 202% ?  Before I buy, I need to know --- there's a pretty big
difference.  When I move from an HP 9000/710 to an HP 9000/730, does my
integer performance improve by 35% (Dhrystone 1.1) or by 51% (SPECint92) ?
When I move from the 710 to the 720, does my integer performance stay the
same (Dhrystone 1.1 --- so why spend any extra money on the 720?) or does it
improve by about 13% (SPECint92) ?

Let's put poor old Dhrystone to rest. It served its purpose in a
scalar CPU world of similar CISC architectures with similar sized
caches. In a world of widely differing approaches to cache design,
with different degrees of superscalarity in the CPU, it is more
misleading than useful. In order to make a synthetic benchmark mimic
the data dependencies, instruction frequencies, basic block sizes,
branch prediction accuracy, etc. --- all of which are important
parameters in modern CPU design, and which do differ across the
various vendors' approaches --- AND make the synthetic code have the
same cache hit/miss ratio as real code --- would take herculean
effort. And if you succeeded, how would it be better than the real
codes in SPECint92?

Synthetic benchmarks are dead outside of marketing departments and ill informed
buyers of equipment.


-- 
-----------------------------------------------------------------------------
"It is seldom that any liberty is lost all at once." David Hume
|||  clc5q@virginia.edu (Clark L. Coleman)
Newsgroups: comp.benchmarks