From: aburto@nosc.mil (Alfred A. Aburto)
Newsgroups: comp.benchmarks
Subject: Re: Dhrystone and SPECint Correlation
Date: 10 Sep 92 01:23:58 GMT
Distribution: comp.benchmarks
Organization: Naval Ocean Systems Center, San Diego

-------
There are several points to be made with regards to the correlation
of Dhrystone 1.1 and SPECint89:

(1)     You posted an article with an example of Dhrystone 1.1 and 
SPECint89 results for 3 systems. You used the MIPS/SPECint89 ratio
to 'prove' that there was no correlation.

(2)     I looked at the same results and saw a correlation. I showed
qualitatively that the results were roughly correlated (Dhrystone1.1 
and SPECint89 results). I went through the mathematics too and that
result showed (for the data we were looking at) a strong correlation 
between Dhrystone1.1 MIPS and SPECint89 (0.982). I also added in the 
nsieve program MIPS results and there we found an even stronger 
correlation between nsieve MIPS and SPECint89 (0.999). Similarly, if
we had examined the individual SPEC integer programs (GCC, ESP, LI,
and EQN) we would have found (I did this later) that GCC, ESP, LI, 
and EQN are all correlated amongst themselves and also with SPECint89,
Dhrystone1.1 MIPS, and nsieve MIPS. The degree of correlation varied,
but in all cases it was relatively high (greater than 0.90).

(3)     The results in (2) clearly showed that the conclusion drawn
in (1) was incorrect. The conclusion in (1) was incorrect because the
ratio "MIPS/SPECint89" destroyed the correlation and it mislead the
author causing an incorrect conclusion to be drawn. This was an almost
classic example of how numbers, WITH NO REGARD FOR THE ERRORS INVOLVED
IN THE MEASUREMENT OF THOSE NUMBERS, can lead us astray and cause us to
draw inappropriate, incorrect, and erroneous conclusions. This was the
most important point I wanted to make. It really has nothing to do with
correlation, Dhrystone1.1 MIPS or niseve MIPS versus SPECint89, or 
anything like that. I want to repeat what I said before because I think 
it is important to do so. We stand on very shaky ground when making
comparisons (of performance) when we have no idea of the magnitude of
the error involved in the things (measurements) we are comparing. It
is time we stopped thinking of benchmark results (system measures of
performance) as having zero error. The error could be small or it may
be quite large ...
        
        I know there is some intrinsic error involved because I see it
when plotting various results. For example the Dhrystone1.1 MIPS results
versus system show alot more scatter (fluctuation, variance, peaks,
error, spread, or whatever one may want to call it) than the SPECint89, 
GCC, ESP, LI, or EQN SPECratio, results versus the same systems. In my 
view this makes Dhrystone1.1 more dangerous to use when comparing 
system A with system B --- dangerous because this 'error' (or degree of 
fluctuation if you will) appears to be rather large (maybe 2 to 3 times 
larger than I see in the other program results). The fact that Dhrystone
shows more error than the SPECratio's or SPECint appears to be the main
difference between them. Otherwise they all pretty much paint the same
picture in terms of system performance (i.e., they are all correlated).
By the way, I know what causes the larger fluctuation in the Dhrystone
results. It is merely that some systems (including compiler) just do a
better job of optimizing the Dhrystone program than other systems. If
for example we were to somehow force all systems to optimize Dhrystone
the same way (to the same degree) then the fluctuation would be greatly
reduced. 

        I'm not sure right now exactly how to quantify the errors in
general. It is definitely not easy to do, but if we are going to advance 
further in the benchmarking field I think it must be done. I do know 
however if we time M different programs on N systems we can in fact 
determine the error due to the M different programs for each of the N 
systems. Certainly I think SPEC should start paying more attention to the 
error involved in their estimates of system performance (SPECint and 
SPECfp).


(4)     Now here comes the real clincher. In (1) you picked 3 systems
with Dhrystone1.1 MIPS and SPECint89 results and attempted to show via
the 'MIPS/SPECint89' ratio that there was no correlation. Well, I came
along and in (2) showed clearly that there was in fact correlation,
and that the conclusions drawn in (1) were erroneous. You did not pick
your 3 examples wisely, because I can pick 3 other different system 
results and prove mathematically ('rigorously') for these samples that 
there is in fact NO correlation between Dhyrstone1.1 MIPS and SPECint89
(in fact for the 3 different systems I'm looking at, the correlation is 
terrible at -0.191)!  I will post these results if you wish, but I hope 
for now you can just believe I can in fact prove NO correlation by 
appropriately selecting a different set of 3 system results than you 
chose in (1).

        I hope now you can see why we can just about prove _anything_
by selectively, or randomly, picking isolated (or small groups of) 
'benchmark' numbers out of a rather large basket of ad hoc results.

The only way out of this quagmire is to gain an understanding of the
errors associated with our measures (estimates) of system performance 
and to use sufficiently large data bases so that the measures of 
performance are stable and relatively independent of sample size.

Personally, I would never use Dhrystone alone or SPECint alone. 
Instead I would gather as much information as possible, using as many
different programs as possible and as many different systems as
possible. If for a given program and system I had alot of data of the
same type (a SPEC value for example from the vendor and 4 other 
different SPEC values from other independent sources) I would select 
the median (or maybe peak value) as the representative measure of 
performance for that program and system. I would then sort the results
in order of increasing performance and plot them for each program. I
might even do a least squares curve fit to the results to gain a 
clearer picture of the relative ranking of the various system results.
After this I would feel 'somewhat safe' in drawing a conclusion and
in considering in other factors such as cost, etc.

First thing though is I need good data bases of results (well
documented results). We really need good data bases and we could
certainly use more good test programs of all types.

(5)     Well, what about the correlation of Dhrystone1.1 and SPECint?
I have extended my own data base of results to 32 different systems
(previously 18 different systems) and the high correlation still holds. 
It is even somewhat better than before. I am happy about this result
because there seems to be at least something in benchmarking that makes
sense. However because of the dangers of unknown errors and inadequate
sample sizes I discussed previously I would never use Dhrystone1.1 or 
SPECint alone in drawing any conclusions. I would use Dhrystone1.1,
SPECint, nsieve and any other information I can get ahold of ...

Al Aburto
aburto@marlin.nosc.mil

-------