From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 4/17/92] SPEC info sources	c.be. FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Thu, 16 Apr 92 11:25:27 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

16	SPEC	<This panel>
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to the FAQ chain and netiquette
2
3	PERFECT Club
4
5	Performance Metrics
6
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Measurement Environments
13
14
15	12 Ways to Fool the Mass with Benchmarks

Systems Performance Evaluation Cooperative
[SPEC]
c/o NCGA
2722 Merrilee Drive
Suite 200
Fairfax, VA, 22031 

Ph:  703-698-9600 x318
FAX: 703-560-2752
spec-ncga@cup.portal.com. (NOT FREE)
SPEC is a trademark of the Standard Performance Evaluation Cooperative.

SPEC, the System Performance Evaluation Cooperative, is a non-profit
corporation formed to "establish, maintain and endorse a standardized
set of relevant benchmarks that can be applied to the newest generation
of high-performance computers".
The founders of this organization believe that the user community
will benefit greatly from an objective series of applications-oriented
tests, which can serve as common reference points and be considered
during the evaluation process.  While no one benchmark can fully
characterize overall system performance, the results of a variety
pf realistic benchmarks can give valuable insight into expected
real performance.
The members of SPEC (at current annual dues of $5000) are
Amdahl, AT&T, Bull S.A., Compaq, CDC, DG, DEC, Fujitsu, HP, IBM,
Intel, Integraph, MIPS, Motorola, NCR, Prime, Siemens, Silicon Graphics,
Solbourne, Sun, Unisys.

SPEC basically does 2 things:

1)  Puts together suites of benchmarks that are generally available
    in source form.
    These benchmarks are intended to measure something meaningful and
    are extensively tested for portability before release.
    There are strict rules on how these benchmarks must be run
    and how results must be reported for the trademarked results
    (SPECmark, SPECint, and SPECfp for the current CPU benchmarks).
2)  Publishes SPEC Benchmark results in a quarterly newsletter.

There are currently two (now three) suites of benchmarks.
1)  SPEC Benchmark suite 1.2.
    This is a suite of 10 CPU (and memory system) intensive benchmarks
    intended to measure "engine horsepower" on some applications of
    realistic size and content.
    The results are expressed as the ratio of the wall clock time to
    execute the benchmark compared to a (fixed) "SPEC reference time"
    (which was chosen early-on as the execution time on a VAX 11/780).
    The results are reported as the geometric mean of the individual ratios:
    SPECmark -- all 10
    SPECint -- the 4 benchmarks that are integer-operation intensive
    SPECfp -- the 6 benchmarks that are floating-point intensive
	      (in this case that means > 1% of executed instr are FP instr).
    A separate measure known as SPECthruput may be reported for
    multiprocessor systems.
    This attempts to measure the actual CPU/memory performance
    available from the multiple processors.
    In effect, it shows up any benchmark slowdown due to resource
    contention in the MP system.

2)  SPEC SDM 1.0 (Systems Development Multitasking) (announced 14-May-91).
    This consists of two benchmarks that present a multiprocess
    load to the system.  The results are more graphical than the
    CPU benchmark and are not as easily reduced to single numbers,
    but the marketing ad numbers used will be the two (different)
    peak throughput numbers measured in scripts/hour.

The benchmark sources are generally available -- but not free.
SPEC is charging separately for the two benchmark suites.
The cost of the source tapes is intended to support the administrative
costs of the corporation -- making tapes, answering questions about
the benchmarks, and so on.

SPEC membership costs:

	Initiation	$10,000
	Annual Dues	$ 5,000

SPEC Associate:

	Initiation	$2,500
	Annual Dues	$1,000

To qualify as a SPEC associate, you must be an accredited educational
institution or a non-profit organization.  An associate has no voting
privileges.  An associate will receive the newsletter and the benchmark
tapes as they are available.  In addition, an associate will have early
access to benchmarks under development so that an associate may act in an
advisory capacity to SPEC.

SPEC Benchmark Suite Tape/Newsletter Subscriptions

Release 1 of the SPEC Benchmark Suite is available in
QIC 24 tape format for $300.
A one-year subscription (4 issues) to the "SPEC Newsletter" is $399.
A combination order (both the tape and a newsletter subscription)
is $699.  There are no discounts for the SPEC tape/newsletter.  

Should probably place contents here:

Q. Where can I get SPEC Results?

A. Some SPEC results are tabulated (very unofficially) and maintained
   for anonymous ftp access on perelandra.cms.udel.edu (128.175.74.1)
   in the directory bench/spec/ .
   The file spec.sc contains the raw data and some derived quantities.
   This is an `sc' format spreadsheet file, and is not easy to read
   without using the `sc' spreadsheet calculator.
   The file spec.print contains a formatted subset of the data.

   If you need to get `sc' as well, it is available in pub/Lang/sc614.tar.Z

   REQUEST:  I have no official source for these results.  Your help
	in keeping them up to date is requested.  Just send e-mail to
	"mccalpin@perelandra.cms.udel.edu" with the following info:
		(1) Exact model of machine
		(2) SPECIFIC citation of where you got the data
		(3) Either the 10 SPEC ratios or the 10 SPEC times
	Unless you request otherwise, I will cite your personal
	communication in the notes for the new results.


                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
 
From jason@cs.utexas.edu  Sun Nov 24 10:46:02 1991
Received: from cs.utexas.edu by amelia.nas.nasa.gov (5.61/1.34)
	id AA19212; Sun, 24 Nov 91 10:46:02 -0800
Message-Id: <9111241845.AA15491@cs.utexas.edu>
Received: by cs.utexas.edu (5.64/1.116) 
From: jason@cs.utexas.edu (Jason Martin Levitt)
Date: Sun, 24 Nov 1991 12:45:56 -0600
X-Mailer: Mail User's Shell (7.2.3 5/22/91)
To: eugene@amelia.nas.nasa.gov
Subject: spec
Status: RO

   You might want to mention LADDIS in your SPEC listing even
though LADDIS is not yet available from SPEC. 
[See the 11/25/91 issue of "Unix Today!" for details. :-)))  ]

   Might you still want to write an editorial for UT! ?

      ---jason   jason@cs.utexas.edu