From: mash@mash.wpd.sgi.com (John R. Mashey)
Newsgroups: comp.benchmarks
Subject: Re: SPEC (was Re: wanted: benchmark results for DEC alpha)
Date: 13 Oct 92 18:57:25 GMT
Organization: Silicon Graphics, Inc.

In article <1992Oct12.194333.5676@ornl.gov>, de5@ORNL.GOV (Dave Sill) writes:
|> In article <1992Oct12.175359.18023@odin.corp.sgi.com>, mash@mash.wpd.sgi.com (John R. Mashey) writes:
|> >
|> >7) From past experience & simulation, we know that changing R4000 scache
|> >from 256KB to 1MB makes little (5%?) difference on SPEC; however, we think
|> >it makes a noticable difference for a lot of real applications, which spend
|> >more time inside the OS, doing context-switching, I/O, etc.
|> 
|> This challenges SPEC's claim that the SPEC Suites consist of real applications,
|> or at least implies that the "real applications" in the SPEC Suites aren't real
|> enough.  Certainly, no matter how many and varied the SPEC applications are,
|> there will be plenty of applications with widely different performance
|> characteristics.  I thought the intent of the SPEC suites was to choose real,
|> large benchmarks that would, on average, represent the average application
|> pretty closely.  You seem to be saying they haven't succeeded.

Nope.  What I'm saying is that SPEC does the best job it can (remember,
I'm one of the original SPEC founders, & I support SPEC 100%).
SPEC tries to get the most realistic benchmarks it can; guess what:
3rd-party ISVs don't give away their code that they make their livings
with.   SPEC has been  pretty careful to say what it thinks these
benchmarks do and don't measure, and you can get a lot of useful
information from them [I hope so, or I wouldn't keep giant spreadsheets
full of them, up-to-date].  My opinion is that the SPECfp92 gives
a pretty good indication of the FP compute portion of many kinds of
code, and the relative performance pattersn can be interesting.
If much of you work is FP-compute-intensive, with little time
spent in OS or doing I/O, then SPECfp92 is at least a reasonable
starting point, especially for some subset of traditional computationally
intense work.  In fact, it you're using your machine as a compute
server, and it has only a few % kernel time, this may be good match.
[And I beleive this opinion matches what SPEC says: make sure the
benchmarks match what you do.] 


|> >Note that UNIX kernel
|> >has a much bigger footprint in the cache than do most of the SPEC 
|> >CPU benchmarks.
|> 
|> Do the SPEC applications not spend as much time in the kernel as the average
|> "real" application?
There are "real" applications that look just like the SPECs; after all, most of
them are "real" applications.  There are certainly real applications that
spend very little time inteh kernel.
|> 
|> >We *think* that many real applications that are FP-intensive
|>                  ~~~~
|> >use integer code more than do SPECfps...  (This is not to say *all*,
|> >and I wouldn't know how to quantify *most*; however, we've seen enough
|>                                       ~~~~
|> >cases where SGI Crimsons are about equal (on average, with wild variations)
|> >against IBM 560s or HP 730s) on FP-intense applications, despite what
|> >SPECfp92 alone would tell you... hence, the various conjectures above.
|> 
|> "Many" I can believe, but to say that "most" real fp apps differ from the SPEC
|> apps is a little harder to swallow.  Not impossible, but I'd have to see some
I explicitly did not claim *most*; I said, I don't know how to quantify that.
|> data before I'd believe it.  Especially coming from a hardware vendor trying to
|> tell me I'll get better performance from his product than SPEC would predict.
Skepticism forever!  Good! 
	Note that benchmarking is like science, it's a series of successive
	approximations.  With SPEC89, we got some numbers that seemed to
	correlate with other benchmarks better than things like Dhrys & Whets.
	Then, as matrix300 got crazy, we had to do better.
	ideally, you'd like one simple benchamr, with a few knobs and dials,
	that would easily precit relative performance across many
	applications ... but we don't know how to do that yet.
The only reason this comes up now is:
	a) It has taken a while to build up SPECfp92 numbers, to get enough
	data.
	b) Only since I've gotten to SGI have I gotten much useful data on
	the FP-intense applications from ISVs; in many cases ISVs will only
	provide data if their name is omitted... (I don't blame them).

If I can figure out how to post what I have in a useful form, I'll do it.
(Right now, a lot of it is, and has to be, for any insight, in graphical
form).  One good consistent set of data was having 20 data points of
Crimsons versus IBM 500s, where SPECfp92 of Crimson was .9 of the 550,
but the mean of the 20 FP-intensive applications was 1.2X Crimson over 550
(with of course, a very wide variance, as we've said time and time again is
waht to expect in FP-intense codes).  The codes included finite-element
analysis, computational chemistry, and simulations [and names withheld,
I don't even know them, just have the data points.]  

Anyway, none of this is intended to downgrade the SPEC numbers, which are
still quite useful.  Just a caution that we don't necesarily have our
hands around every application out there ... bu there's nothing wrong with
that.  As SPEC *always* says: your own benchmark is teh best test, and if
you can find subsets or combinations of the individual SPEC #s that
correlate well with your own, use that subset and ignore the rest.

|> 
|> >BOTTOM LINE: whenever anyone publishes SPEC numbers, do a careful comparison
|> >across the board, not just on the mean numbers, to get some diea of what's
|> >going on.  In particular, always watch out for "peaks", like what happened
|> >to SPEC89's matrix300 (although perhaps not so extreme).
|> >However, *expect* that performance ratios on individual benchmarks
|> >will vary widely among these machines, and that if you pick your benchmarks,
|> >you'll probably be able to prove superiority of anything you like,
|> >among machines whose means are similar. 
|> 
|> Good advice.
|> 
|> -- 
|> Dave Sill (de5@ornl.gov)           For every Bill Joy there is a Kirk McKusick.
|> Martin Marietta Energy Systems     For every Bill Gates there is a Richard
|> Workstation Support                Stallman.                     --Paul Graham

-- 
-john mashey    DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP:    mash@sgi.com 
DDD:    415-390-3090
USPS:   Silicon Graphics 7U-005, 2011 N. Shoreline Blvd, Mountain View, CA 94043