From: mash@mash.wpd.sgi.com (John R. Mashey)
Subject: Re: wanted: benchmark results for DEC alpha
Keywords: Dec alpha
Organization: Silicon Graphics, Inc.
Date: Mon, 12 Oct 1992 17:53:59 GMT

In article <1992Oct12.100235.10060@irfu.se>, bt@irfu.se (Bo Thide') writes:

|> >   I have seen one set of benchmark results for some quite large 
|> >fortran programs, Alpha came out more than twice as fast as an HP 720 snake.
|> >It was a non-production machine with early compiliers so the ratio
|> >should get bigger....
|
|> I have seen benchmarks results comparing the Alpha with HP's of the same
|> generation (the new machines based on the HP PA RISC 7110 Thunderbird
|> CPU; the first workstation model in this new series is due any day now)
|> that show that the HP's are up to 50% *faster* than the Alpha.

Note: if you look at SPECfp92 benchmarks, you find that the relative
performance of 2 machines with the same SPECfp92 overall numbers,
can easily vary up to about 2:1 in either direction....

Below, "fits" means has a reasonable hit rate in the given cache size,
not that the total size of code or data ius that size.

In particular, this can strongly depend on the benchmark size,
that is:
	PA 7100 uses split I & D caches, up to 256KB each in announced systems.
	Alpha & MIPS R4000 have on-chip 8K I cache & 8K Dcache (well, 16K
		each soon, for R4000A), plus (usually) 1MB secondary cache.

1) Now, if each of code & data "fits" in the (small) on-chip caches,
	Alpha & R4000A may well win.

2) If each of code & data fits into off-chip 256KB HP caches, it probably wins,
	as 1-level caches have less overhead than 2-level ones.

3) If either code or data does *not* fit into the 256KB HP cache, but does
fit into 1MB, then Alpha/R4000 probably win.

4) If hardly anything fits, then we pretty rapidly get down to DRAM memory
bandwidth, and *nobody* wins :-)  (actually, of existing system out there,
IBM RS/6000s may win, given 128-bit memory paths in bigger systems.)

5) If the code is mostly integer, R4000As will probably win [since they'll
have 2X larger on-chip caches compared to Alphas].  If the code is
mostly FP [i.e., SPECfp], then either Alpha or HP will probably be ahead,
subject to the memory effects above.

6) From past experience, R4000s seem better at turning SPECint92 into multi-
tasking (like AIMs), although it is hard to get enough data.

7) From past experience & simulation, we know that changing R4000 scache
from 256KB to 1MB makes little (5%?) difference on SPEC; however, we think
it makes a noticable difference for a lot of real applications, which spend
more time inside the OS, doing context-switching, I/O, etc.  Note that UNIX kernel has a much bigger footprint in the cache than do most of the SPEC
CPU benchmarks.  We *think* that many real applications that are FP-intensive
use integer code more than do SPECfps...  (This is not to say *all*,
and I wouldn't know how to quantify *most*; however, we've seen enough
cases where SGI Crimsons are about equal (on average, with wild variations)
against IBM 560s or HP 730s) on FP-intense applications, despite what
SPECfp92 alone would tell you... hence, the various conjectures above.

BOTTOM LINE: whenever anyone publishes SPEC numbers, do a careful comparison
across the board, not just on the mean numbers, to get some diea of what's
going on.  In particular, always watch out for "peaks", like what happened
to SPEC89's matrix300 (although perhaps not so extreme).
However, *expect* that performance ratios on individual benchmarks
will vary widely among these machines, and that if you pick your benchmarks,
you'll probably be able to prove superiority of anything you like,
among machines whose means are similar. 


-john mashey    DISCLAIMER: <generic disclaimer, I speak for me only, etc>
UUCP:    mash@sgi.com 
DDD:    415-390-3090
USPS:   Silicon Graphics 7U-005, 2011 N. Shoreline Blvd, Mountain View, CA 94043