From: Patrick F. McGehearty <patrick@convex.COM>
Subject: Re: Dhrystone and SPECint Correlation
Originator: patrick@mozart.convex.com
Date: Thu, 10 Sep 1992 16:11:25 GMT
Distribution: comp.benchmarks
Reply-To: patrick@convex.COM (Patrick F. McGehearty)
Organization: Engineering, CONVEX Computer Corp., Richardson, Tx., USA
X-Disclaimer: This message was written by a user at CONVEX Computer
              Corp. The opinions expressed are those of the user and
              not necessarily those of CONVEX.

In article <1992Sep10.012358.5084@nosc.mil> aburto@nosc.mil (Alfred A. Aburto) writes:
some solid discussion about the need for realizing measurement have
errors.
>We stand on very shaky ground when making
>comparisons (of performance) when we have no idea of the magnitude of
>the error involved in the things (measurements) we are comparing. It
>is time we stopped thinking of benchmark results (system measures of
>performance) as having zero error. The error could be small or it may
>be quite large ...
>        
I would like to elaborate on this theme.  A common error is to assume
different benchmarks are random samples from a data space which are
independent from each other.  In truth, they are not independent.
Thus, all of our developed intuition about the normal distribution,
standard deviations, etc, etc, can be very misleading.  

I would suggest we think of errors in terms of modeling errors instead of
sampling errors.  The model (the benchmark) is likely to be systematically
different from the reality (a particular user's workload) in ways that
are different for each user.  If a variety of benchmarks all indicate
the same system is superior, and there is some reason to believe that
those benchmarks are in the same domain as target use of the system,
then selection decisions are easy.  When different benchmarks give different
predictions about the best choice, then further study is necessary to
determine which benchmarks most closely model the intended use of the
machine.  To do that, we need to understand what a benchmark is really
measuring, not what is claims to be measuring.

<The following is as aside about Dhrystone, unrelated to the rest of
 the discussion.  Skip it if you are tired of hearing about Dhrystone.>

Dhrystone is particularly flawed relative to today's technology, which is
why any attempt to justify it's continued use provokes widespread cries of
dismay amoung those who have studied it carefully.  A single optimization
which has no importance for any real application that I have seen (at least
in the scientific market segment) makes a 20% performance difference on
Convex machines.  Which Dhrystone number would you use?  The one with or
without the optimization?  Your real application performance will be the
same in either case.
    [That optimization is doing an inline copy of a fixed length string
     in C instead of calling the strcpy routine.  The inline version
     moves data 8 bytes at a time, while the library routine must examine
     each byte of the source string for a possible zero to terminate the copy].
In addition, Dhrystone is heavily dominated by subroutine calls.  The
typical Dhrystone subroutine call only executes a few operations before
returning, which is less than most C code I have looked at, and far less
than most Fortran code I have seen.  Further, the few loops it does contain
are only executed once.  If an optimizing compiler spends two operations
outside a loop to save one operation inside a loop, it will perform less
well on Dhrystones than not doing this optimization.  It was a reasonable
attempt for it's time, but we have much better benchmarks available to us
now, such as those provided by SPEC and Perfect.

In so far as Dhrystone correlates with SPECint for some machines, I believe
that it does so because subroutine call overhead correlates with integer
performance for these machines.  Vector machines, VLIW machines, superscalar
machines all have potential for executing integer operations much faster
than subroutine calls.  Using Dhrystones and its correlation with SPECint
for traditional architectures to predict performance for current advanced
architectures is as likely to be correct as throwing darts blindfolded.

<<End of Dhrystone discussion>>


>        I'm not sure right now exactly how to quantify the errors in
>general. It is definitely not easy to do, but if we are going to advance 
>further in the benchmarking field I think it must be done. I do know 
>however if we time M different programs on N systems we can in fact 
>determine the error due to the M different programs for each of the N 
>systems. Certainly I think SPEC should start paying more attention to the 
>error involved in their estimates of system performance (SPECint and 
>SPECfp).
>
I agree that error quantification is not easy, and any particular approach
may be very misleading for some classes of benchmark users.  A major area
for biasing the results is in determining a base value for each program.
In "The Art of Computer Systems Performance Analysis", Raj Jain devotes an
entire chapter to "Ratio Games", followed by several chapters on what
to do about it.  Another area is in the selection of benchmarks.  For
example, if there are 9 benchmarks which focus on one aspect of performance
(like subroutine calls), and 1 benchmark which focuses on another aspect
(like block copies), then machines which are better at the first will look
much better on the benchmark aggregate than those which are better at the
second.  If the actual application mix uses those two aspect equally, the
average use of total benchmark set should also use each equally.
That does not mean each benchmark should use each equally, just be careful
not to bias results by too many benchmarks measuring a single aspect of
performance.
Newsgroups: comp.benchmarks