From: eric@snark.thyrsus.com (Eric S. Raymond)
Newsgroups: comp.unix.sysv386,comp.arch,comp.unix.wizards,comp.benchmarks,alt.folklore.computers
Subject: The Great 1992 UNIX Dream Machines Bake-Off --- Progress Report #1
Date: 11 Apr 92 05:49:36 GMT

In <1g0YV6#50vCGv87xMXj64cm3J2Jp3ly=eric@snark.thyrsus.com> I wrote:
>Having secured the willing cooperation of all the major UNIX vendors and
>several of the leading clonemakers, I'm now ready to go to the next stage; a
>full competitive benchmark of representative high-end clone systems running
>UNIX.

This is a progress report on the Bake-Off preparations.  It also answers a
number of questions raised by various people in email, and summarizes the
helpful advice I've received so far.

1) WHO'S IN, WHO'S OUT

I've just sent the formal proposal mailing to the following vendors:  ALR,
Ares Microdevelopment, Basic Time, Blue Dolphin, Compaq, Dell, Everex, E.P.S.,
Gateway, Keydata, Naga, Northgate, Swan Technologies, TriStar, Zenon, and Zeos.
Decision-makers at all of these have indicated an interest in participating;
one (Swan) is already officially committed.  AST Research is also on my list,
but I haven't been able to reach their UNIX honcho yet to verify that they
want in.

Microport, UHC and Esix have already volunteered to donate copies of their
UNIXes for the test (Esix's will be an advance 4.0.4).

MetroLink will be donating a copy of its X server product for ATI and S3-based
graphics accelerator cards.  I'm in touch with SGCS too and will probably end
up benching the Metrolink and SGCS servers head-to-head against each other.

Note to other hardware and software vendors: if you want in, send me email.
I'll at least test-install everything I get.  If the number of combinations
doesn't break into three digits I'll bench all the OS/machine combinations
separately.  Move quickly on this, because if the number of combinations *does*
explode beyond what I can handle, the vendors who get left out will be the
ones who were slowest to get on board....

Intel is now officially supporting the Bake-Off and will supply the SPECmark
tape for the benching runs.

2) A BALANCE OF QUESTIONS

In <1992Apr7.140644.27039@bilver.uucp> Bill Vermillion wrote:
> One BIG question.  Why is this limited to EISA machines.  Why not the
> MCA architecture.   I remember reading specs on an ALR machine (One of
> the few (if not only) that makes both and EISA and MCA, and the MCA
> outperformed EISA in some respects.    And what about the NCR machines,
> or any of the manfacturers of MCA bus based machines other than IBM.

Because MCA isn't commodity hardware.  So it's got a whole two manufacturers.
And yes, it's technically nifty.  But it's expensive, IBM effectively controls
it, and there's no reason to believe either of those things will ever change.

In email, hiday_john_d@lilly.com writes:
> I was really looking forward to your bake-off, but now that I've
> read your proposal I am somewhat disheartened.  I agree that the
> focus should be on commodity hardware, but I am disappointed to see
> that you seem to be trying to challenge the big boys' latest toys.
> What I'd like to know is can a 486dx/33 ISA compete with the big
> boys' low to middle price range machines?

There's an un-obvious reason I'm doing it this way.  It's because prices
are dropping so fast in the clone market.  If I'd gone with vanilla 33s,
the info would be stale in three months as the 50s and clock-doubler 66s
and 100s come in.  These expensive high-end machines, on the other hand,
should reach the price point of a vanilla 33 in six months or so.  The
workstation market is moving much more slowly.  Thus, by deliberately
testing *ahead* of the cheap hardware I'll get comparative results that are
useful for a longer time into the future.

He continued:
> I'm no PC expert (I don't do DOS or MSwindows), but I do keep
> up with the mags just for prices.  I would estimate that the 486-50/DX
> EISA system you propose must cost at least $7000 from a reputable clone
> dealer, then you have to add the OS.  I can get a personal DECstation
> 3100 or a NeXTstation for less than that and both include the OS.  I
> imagine that people with good edu connections could do even better.

$5500 is closer to the mark --- and that's retail, not wholesale.  Also, you
have to think about life-cycle costs.  Parts and service for Intel-based
hardware is *always* going to be cheaper than traditional workstation stuff,
because the size of the market over which vendors can amortize costs is so much
bigger.  Like it or not (and I don't, particularly; I haven't forgiven them for
the 8086 and 286) Intel has won the processor wars.

3) BENCHMARK WARS

I've snarfed the entire comp.benchmarks archive from wustl and read it all.
I'm under no delusions that this makes me an expert on benchmarking, but at
least my ignorance has been somewhat lessened. :-) 

In jonathan@hoffmann.ph.utexas.edu (Jonathan Thornburg) wrote:
>I strongly suggest using the SPEC92 suite.  This will make it much
>easier to compare the results to those of "real" workstations.

This is only the most succinct of many, many notes I received urging that
SPEC be the Bake-off's primary benchmark.  Their will be done.

Nhuan Doduc <ndoduc@framentec.fr> wrote urging me to also use SSBA, largely on
the grounds that it's popular in Europe.  I may do this as well, but the
consensus for SPEC as number-one number was very strong.

In email, msf@rotary.East.Sun.COM (Mike Fischbein)
>MUSBUS would probably be the best frame work for measuring multiuser
>or just multitasking performance.

There's a modified (and allegedly improved) MUSBUS included in SPEC's SDM
suite.  Those results will be quoted.

In email, exu.ericsson.se!exudnw (Dave Williams) wrote:
>Make sure you also measure network performance in a TCP/IP environment.
>NFS (where suplied) is of particular interest to me.  Try LADDIS or 
>nhfsstone.

I have no experience at setting up or administering networks.  Also, there are
nontrivial logistics involved in pulling together physical hardware like
cabling and terminators that the vendors won't automatically supply.  However,
one of my co-judges is a TCP/IP wizard.  We'll see what we can do.

In email, hiday_john_d@lilly.com writes:
> No matter what kind of machines you decide to test on, please include
> X performance as one judging criteria.  Slow X performance is one
> common complaint I have seen on the net.

I intend to weigh X performance heavily, as promised in the rules.

In email, jato.Jpl.Nasa.Gov!dave (Dave Hayes) wrote:
>I think you definately want some PERL benchmarks. Perhaps the perl
>test suite is a benchmark of compatability itself...but lots of netters
>use perl for lots of stuff. 

Gack.  I guess this means I have to learn perl...

My present plan is to use SPEC, bonnie, iozone, and some
yet-to-be-determined X benchmark.  And no Dhrystone!

Also, I will report the time required to TeX the 433-page master of
_The_New_Hacker's_Dictionary_.

4) OTHER WORDS FROM THE WISE

In email, msf@rotary.East.Sun.COM (Mike Fischbein)
>I'd also suggest some portion of the hardware cuteness award (forgot
>your name for it) include the ease/difficulty of adding hardware --
>stuff like extra RAM, an additional disk, etc, can be VERY difficult
>to figure out on some systems.  Maybe that's a documentation test,
>but it is very real.

I do plan to weigh upgradability heavily in the scores for the Warm Fuzzy award
for Best System Design.

In email morton.rain.com!joe (Joe Moss) wrote:
>I'm not sure if it deserves it's own award or not, but I think it might
>be good to make a decision as to which hardware/software incantation
>would be best for a business environment (where support/compatibilty
>issues would have a heavier weighting than they would for a hacker's
>system).

I'm frankly not sure I'm qualified to do this.  I know too much --- my slant
on "quality of support" is never going to be the same as J. Random Luser's.
Also, if by "compatibility" you mean the ability to run old binaries from
shrink-wrapped UNIX software, I'm the wrong person to do that; I've never used
any such software!  I will, however, work hard to provide enough information
for commercial users to apply their own criteria.

He also said:
>Cute.  While you're at it could you maybe send one of the left over
>not-quite-the-wizbangiest-but-still-a-good-Unix-box machines my way?

You and about 1,723 other people.  Sorry; leftover hardware, if any, will
go to compensate the other judges for their time.

In email, jato.Jpl.Nasa.Gov!dave (Dave Hayes) wrote:
>How about the "Gee! It compiled!" award to those systems that can compile
>the most GNU/X11/perl/etc public domain software with the least amount
>of difficulty?

This is not a function of hardware but of OS.  I don't expect there will be
any differences at all in this among the SVr4 ports, and probably none worth
noting even with the BSD ones (as all are advertised POSIX-compatible).

5) ON THE HOME FRONT

I have recruited three other technical-expert judges.  At a netter's
suggestion, these gentlemen will remain anonymous until *after* the test.

The spacious third-floor loft of the Eric Conspiracy Secret Labs here
in sunny Malvern, PA is being readied for the expected flood of
incoming hardware.  Translation: I'm getting rid of that ratty old couch,
stacking the overflow from my SF collection's bookshelves in one corner,
and buying a new power bar.  It's gonna be wall-to-wall machines up here
soon...
--
      Eric S. Raymond = eric@snark.thyrsus.com  (mad mastermind of TMN-Netnews)