From: aburto@nosc.mil (Alfred A. Aburto)
Subject: Re: Dhrystone and SPECint Correlation
Organization: Naval Ocean Systems Center, San Diego
Distribution: comp.benchmarks
Date: Tue, 15 Sep 1992 01:18:21 GMT

In article <1992Sep10.161125.6482@news.eng.convex.com> patrick@convex.COM (Patrick F. McGehearty) writes:
>In article <1992Sep10.012358.5084@nosc.mil> aburto@nosc.mil (Alfred A. Aburto) writes:
>some solid discussion about the need for realizing measurement have
>errors.

Thanks, from email I've received I was beginning to feel rather 
defensive relative to Dhrystone. Mention anything remotely good about 
Dhrystone, seems to cause some peoples hair to rise, finger nails 
suddenly grow longer, fangs appear where there were none before, toe 
nails become claws, a smile turns into a snarl, common sense and logic 
go flying out the window ... :-)

>>
>>We stand on very shaky ground when making
>>comparisons (of performance) when we have no idea of the magnitude of
>>the error involved in the things (measurements) we are comparing. It
>>is time we stopped thinking of benchmark results (system measures of
>>performance) as having zero error. The error could be small or it may
>>be quite large ...
>>        

>
>I would like to elaborate on this theme.  A common error is to assume
>different benchmarks are random samples from a data space which are
>independent from each other.  In truth, they are not independent.
>Thus, all of our developed intuition about the normal distribution,
>standard deviations, etc, etc, can be very misleading.  
>

Yes, those measures can be misleading because they need to be calculated
relative to the probability density, but generally in benchmarking our
samples are drawn from unknown distributions. But there are measures of
performance ('non-parametric') that are independent of a particular 
distribution. The cumulative distribution and median are examples. The
Spearman rank correlation is another. 

>
>I would suggest we think of errors in terms of modeling errors instead 
>of sampling errors.  The model (the benchmark) is likely to be 
>systematically different from the reality (a particular user's workload) 
>in ways that are different for each user.  If a variety of benchmarks 
>all indicate the same system is superior, and there is some reason to
>believe that those benchmarks are in the same domain as target use of 
>the system, then selection decisions are easy.  When different 
>benchmarks give different predictions about the best choice, then
>further study is necessary to determine which benchmarks most closely
>model the intended use of the machine.  To do that, we need to 
>understand what a benchmark is really measuring, not what is claims to 
>be measuring.
>

That is a good point about modeling errors. An important point. Linpack
for example tests specific algorithms and a specific instruction mix,
flops examines performance from a different perspective using different
algorithms and instruction mixes, and the Livermore Loops yet another.
Linpack MFLOPS rating are widely quoted, but I'm absolutely sure that
one should not use Linpack to ascertain relative performance gains for
codes or instruction mixes substantially different than that used in
Linpack. The user needs to be careful and should not blindly accept
a given benchmark result as truth for all cases. I don't know if you
noticed but when Frank McMahnon posted a summary of Livermore Loops
results the HP 9000/730 had an LLOOPS rating of 16 MFLOPS while
Linpack gives 24 MFLOPS, and flops gives a rating of 27 MFLOPS. When
it comes to the IBM RS/6000 540 though, Linpack gives a high rating
while both LLOOPS and flops give much lower ratings (11 MFLOPS). We 
must be careful and we must know what the programs are measuring, as 
you said.

There are real errors too. These happen when a test program fails due
to some optimization usually. It has happened to Whetstone, Dhrystone,
Livermore Loops, and SPEC too. The failure mode is such that the
programs measure of performance has been compromised and it no longer
reflects real performance gains for a particular system. In the 
Livermore Loops and SPEC these failures have been mended but not so 
for Whetstone and Dhrystone.

[Some Dhrystone remarks about optimization left out]

I want to add however that all programs and benchmarks are sensitive
to compiler optimizations of one form or another and to different 
degrees of optimization (some more sensitive than others). Dhrystone
is particularly sensitive to optimization unfortunately. That is, on
a particular system with no optimization one may get X Dhrys/sec
while with full optimization turned on one may get 3X Dhrys/sec. I
have a number of examples showing this. IBM and some HP compilers 
are particularly good at optimizing Dhrystone.

The SPEC results are also sensitive to optimization. As a reference
point and to get a 'flavor' of how much compilers add to performance
I would like to see SPEC results reported with all optimizations
disabled and also, in addition, with full optimizations turned on.

>
>In so far as Dhrystone correlates with SPECint for some machines, I 
>believe that it does so because subroutine call overhead correlates 
>with integer performance for these machines.  Vector machines, VLIW 
>machines, superscalar machines all have potential for executing 
>integer operations much faster than subroutine calls.  Using 
>Dhrystones and its correlation with SPECint for traditional 
>architectures to predict performance for current advanced 
>architectures is as likely to be correct as throwing darts 
>blindfolded.
>
><<End of Dhrystone discussion>>
>

I'm not sure it is the subroutine call overhead that does it. After
all so many compilers nowadays inline the small subroutines removing
the call overhead altogether. It is simply that the results track
one another. The fastest systems tend to give both the highest 
SPECint and Dhrystone ratings (roughly). The slowest systems also
tend to give both the lowest SPECint and Dhrystone ratings (again 
roughly). The middle performers also tend to be in the middle for 
both SPECint and Dhrystone (again roughly so). It is in the details
where things fall apart in a hurry. The SPECint89 results (the only 
numbers I have --- about 70 different system results) appear to be 
very stable. The arithmetic mean agrees very closely with the 
geometric mean and the standard deviation is relatively small (less
than +/- 3). The distribution of results appears to be bi-modal but
I think that is just an artifact of an insufficient sample size. The
individual programs (GCC, ESP, LI, and EQN) are well correlated with
each other and also with SPECint89. The ESP results seem to show the
highest correlation with SPECint89, but not by much. The Dhrystone
results on the other hand, despite the apparent correlation with 
SPECint89, show a huge 'spread' in performance as compared to 
SPECint89. A plot of these results along with SPECint89 very clearly 
shows that it is unsafe to use Dhrystone in making performance 
comparisons across different systems (too much scatter). That's it 
in a nutshell. And thats why I think we need to quantify the spread 
in our measures of performance a bit better than we have --- to help 
prevent us from drawing incorrect conclusions.

The apparent (because I still need more samples) correlation between 
SPECint89 and Dhrystone is disproved not by words, but by actually
doing it with a sufficiently large data set. I'm not advocating 
anything, I am merely pointing out an unexpected and curious result 
and I'd like to learn more about it, but I need more data. Also, as
I've indicated previously, it is not only Dhrystone but all the
integer programs I've looked at that seem to show this correlation.
GCC SPECratio89, LI SPECratio89, ESP SPECratio89, EQN SPECratio89,
nsieve MIPS, Dhrystone MIPS, and SPECint89 are all (apparently)
showing relatively high correlations with one another.

[I'm leaving out some good points you brought out about 'ratio games',
and how we can get misleading and biased results by not having a good 
mix of different types of programs]

Al Aburto
aburto@marlin.nosc.mil
14 Sep 1992

-------


Newsgroups: comp.benchmarks