HDTenEightyP/NetNews huggingface dataset:
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/17/91] Ways to fool the masses with benchmarks	(15/28) c.be. FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Sat, 15 Aug 92 11:25:16 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

15	12 Ways to Fool the Masses with Benchmarks	<This panel>
16	SPEC
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to the FAQ chain and netiquette
2
3	PERFECT Club
4
5	Performance Metrics
6
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Measurement Environments
13
14

From David Bailey et al.

Quote only 32-bit performance results, not 64-bit results.

Present performance figures for an inner kernel as the performance of the
entire application.

Quietly employ assembly code and other low-level language constructs.

Scale up the problem size with the number of processors, but omit any
mention of this fact.

Quote performance results projected to a full system.

Compare your results against scalar, unoptimized code on Crays.

When direct run time comparisons are required, compare with an old code
on an obsolete system.

If megaFLOPS rates must be quoted, base the operation count on the parallel
implementation, not on the best sequential implementation.

Quote performance in terms of processor utilization, parallel speedup or
megaFLOPS per dollar.

Multilate the algorithm used in the parallel implementation to match
the architecture.

Measure parallel run times on a dedicated system, but measure conventional
run times in a busy environment.

If all else fails, show pretty pictures and animated videos, and don't talk
about performance.

Ref. 1
NAS TR #

Ref. 2
%A David Bailey
%T Twelve Ways to Fool the Masses when Giving Performance Results on
Parallel Computers
%J Supercomputing Review
%V 4
%N 8
%D August 1991
%P 54-55

References
Darrell Huff, How to Lie with Statistics
	How to Lie with Maps

Gordon Bell's 11 rules of supercomputer design:

1) Performance, performance, performance.
2) Everything matters.
3) Scalar matter most.
4) Provide vectors as price allows.
5) Avoid holes in performance.
6) Place peaks in performance.

7) Provide a decade of addressing.
8) Make it easy to use.
9) Build on others work.
10) Design for the next one (and do it again) [three times]
11) Have slack resources.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
 
Newsgroups: comp.benchmarks
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/17/92] WPI Benchmark Suite (1.1)	(19/28) c.be.FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Wed, 19 Aug 92 11:25:12 GMT
Reply-To: mach@cs.wpi.edu

19 	WPI Benchmark 					<This panel>
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to the FAQ chain and netiquette
2	Benchmarking concepts
3	PERFECT Club/Suite
4
5	Performance Metrics
6
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Benchmark Environments
13
14
15	12 Ways to Fool the Masses with Benchmarks
16	SPEC
17	Benchmark invalidation methods
18

The WPI Benchmark Suite, version 1.1, is now available for download
via anonymous ftp.  The WPI Benchmark Suite, developed with the support
of the Research Institute of the Open Software Foundation, is a series
of programs designed to investigate operating system performance of Unix
and Unix-like operating systems.  We have used the Benchmark Suite
extensively to study the performance of various versions of the Mach
operating system.

The Benchmark Suite consists of six programs, each of which is designed
to mimic the operating system resource usage of a user application.
Each program can be run independently, and reports the execution time
for the measured portion of the benchmark, as well as other resource
usage.

The programs are in the public domain, and may be downloaded via
anonymous ftp from wpi.wpi.edu (130.215.24.1) in the "/benchmarks"
directory.  Get the file WPI_Benchmark_Suite_1.1.readme and either
BenchmarkSuite1.1.tar.Z or BenchmarkSuite1.1.lzh.

In addition to the benchmarks, there are two technical reports available
on-line, describing the benchmarks and giving some sample results of
running them.  The technical reports are in the files benchmarks_tr1.ps
and benchmarks_tr2.ps.  The files are in PostScript format.

If ftp is not available at your site, please send a message requesting
the Suite by e-mail. This is a much slower process, so please ftp the 
benchmarks if you can.

If you download the benchmarks, please send mail to mach@cs.wpi.edu
to have your named placed on our mailing list for information about
future releases.  In addition, we are interested in your comments
about the benchmarks and in hearing about your experiences in using
them.


 Mach Research Group
 Computer Science Department                Internet:  mach@cs.wpi.edu
 Worcester Polytechnic Institute            Telephone: (508) 831-5357
 Worcester, MA 01609 USA                    FAX:       (508) 831-5776 

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
 
Newsgroups: comp.benchmarks
From: clc5q@hemlock.cs.Virginia.EDU (Clark L. Coleman)
Subject: Re: HP AIM Configuration and pricing
Organization: University of Virginia Computer Science Department
Date: Thu, 20 Aug 1992 21:15:09 GMT

In article <1992Aug13.231403.4605@odin.corp.sgi.com> ricky@benchmaster.esd.sgi.com (Ricky Cheng) writes:
>Does anyone know the Prices and configuration of
>
>HP 710C
>HP 720CRX
>
>when running AIM ?
>Here is the results which were posted in this news group earlier!
>
>		AIM	MAX USER
>HP 710C	29.6	180
>HP 720CRX	35.5	229
>
>Does anyone knows the milestone results of all HP machines ?

The tested configuration for the HP 710 was with two 424MB disks, 6MB of RAM
ised for I/O buffers, 16 MB total RAM, $14,490 price as tested. The HP 720
was tested with 32MB of RAM, 16MB for I/O buffers, two 4242MB disks, $21,490.

The MPH (milestones) test exercises Unix utilities. The AIM people apparently
have a nonstandard idea of what the "standard" Unix utilities are: they say
"utilities missing" instead of giving an MPH number for all systems tested
from HP, IBM, Encore and Data General.


-- 
-----------------------------------------------------------------------------
"It is seldom that any liberty is lost all at once." David Hume
|||  clc5q@virginia.edu (Clark L. Coleman)
From: dhinds@leland.Stanford.EDU (David Hinds)
Subject: Re: Geometric Mean or Median
Organization: DSG, Stanford University, CA 94305, USA
Distribution: comp.benchmarks
Date: Fri, 14 Aug 92 16:52:47 GMT

In article <1992Aug14.155857.6561@riacs.edu> lamaster@pioneer.arc.nasa.gov (Hugh LaMaster) writes:
>In article <Aug14.142126.38458@yuma.ACNS.ColoState.EDU>, shafer@CS.ColoState.EDU (spencer shafer) writes:
>|> 
>|> 
>|> Having joined this in midstream, I risk repeating some information.  If
>|> so, my apologies in advance.  A discussion of this, and an offered proof
>|> of the geometric mean as preferred method is in the March 1986 issue of 
>|> Communications of the ACM, "How Not to Lie With Statistics: The Correct
>|> Way to Summarize Benchmark Results," by Fleming and Wallace.

Has anyone tried to take all the available SPEC numbers, and do a factor
analysis, to see if there is a statistically meaningful small set of
numbers that can be used to predict the performance on all the tests?
One would hope that the factors that fell out would naturally fit
different architectural parameters -- scaler integer speed, scaler 
floating point speed, vector performance, etc.  You would also get the
weights of each factor for each SPEC test, and users could estimate
performance of their own codes on new machines, by running on a few old
machines, and using the published SPEC factors for those machines to
calculate the weights for their codes.

        - David Hinds
          dhinds@allegro.stanford.edu
From: lamaster@pioneer.arc.nasa.gov (Hugh LaMaster)
Newsgroups: comp.benchmarks
Subject: Re: Geometric Mean or Median
Date: 14 Aug 92 15:58:57 GMT
Distribution: comp.benchmarks
Organization: RIACS, NASA Ames Research Center

In article <Aug14.142126.38458@yuma.ACNS.ColoState.EDU>, shafer@CS.ColoState.EDU (spencer shafer) writes:
|> 
|> 
|> Having joined this in midstream, I risk repeating some information.  If
|> so, my apologies in advance.  A discussion of this, and an offered proof
|> of the geometric mean as preferred method is in the March 1986 issue of 
|> Communications of the ACM, "How Not to Lie With Statistics: The Correct
|> Way to Summarize Benchmark Results," by Fleming and Wallace.



Yes, and there was a rebuttal to this "proof" in CACM by, I believe,
J.E. Smith, in October of 1988.  {If I have the reference correct,}
it is proved that the harmonic mean is the correct measure of rates, 
if you want to examine a fixed workload and characterize the performance
on that workload.  Fleming and Wallace missed the point, IMHO, of a lot
of things, including Amdahl's law.  Always use the harmonic mean of rates
unless you have a specific reason not to.



I was glad to see that Gordon Bell provided the "LFK(hm)" column in the
performance summary of his recent article in CACM, since this is the
harmonic mean of an untuned workload of Livermore Loops.  This is the one
of the many possible measures, which I find most meaningful when comparing 
raw floating point performance.  It tends to be pessimistic with respect
to tuned rates on vector machines by perhaps a factor of 3 or 4, but it
is rather realistic when looking at untuned rates.

("raw" -- that is, a first cut, before you do some "real" benchmarking :-)



-- 
  Hugh LaMaster, M/S 233-9,     UUCP:      ames!lamaster
  NASA Ames Research Center     Internet:  lamaster@ames.arc.nasa.gov
  Moffett Field, CA 94035-1000  Or:        lamaster@george.arc.nasa.gov 
  Phone:  415/604-1056                     #include <usenet/std_disclaimer.h> 
Newsgroups: comp.benchmarks
From: patrick@convex.COM (Patrick F. McGehearty)
Newsgroups: comp.benchmarks
Subject: Re: Disk storage size?
Date: 19 Aug 92 17:22:46 GMT
Article-I.D.: news.1992Aug19.172246.8264
Reply-To: patrick@convex.COM (Patrick F. McGehearty)
Organization: Engineering, CONVEX Computer Corp., Richardson, Tx., USA
Originator: patrick@mozart.convex.com
X-Disclaimer: This message was written by a user at CONVEX Computer
              Corp. The opinions expressed are those of the user and
              not necessarily those of CONVEX.

>In article <0001@umd.edu>, pc@umd.edu writes:
>>When people talk about 200MB disk, do they mean  200x1,000,000 byte, or
>>200x1024x1024 ?  When talk about the RAM, people seems use 1k=1024, but
>>when go to disk, it is metric. Is this always the case?
>
...stuff deleted
>In article <1992Aug19.152525.15741@ornl.gov> de5@ORNL.GOV (Dave Sill) writes:
...stuff deleted
>
>In practice, as when calculating transfer rates or repartitioning a disk, I'm
>careful to calculate in terms of true K, M, and G, but when someone asks me how
>big a file is or how much space is left on a partition, I'm sloppier and
>sometimes use K as an abbreviation for "thousand", etc.
>
>Bottom line: if it matters, make sure it's clear which meaning is being used.
>
Ironic point.  From your posting, I can't tell whether you consider
metric or binary to be "true K, M, and G".  Just confirms the
ambiquity.  When I can't tell which type of mega a user means, I
try to assume it was whatever is most favorable to the user of the term.

I have noticed that almost anything marketing or sales oriented uses metric
Mega and Giga terms.  To do otherwise is to give up 5 or 7% apparent
relative value (recalling 1024*1024=1,048,576, and 1024**3=1,073,741,824).
This has not happened in RAMs because they are almost always sold in
increments which are a power of 2 or a small integer constant times a power
of two.  So most people can tell that 1 Mbyte of RAM is the same as 1.047
Mbytes of RAM.  Disks are not forced to be so regular.

The case of nominal disk space is made even worse by another practice.  The
difference between unformatted and formatted disks is quite large.  A nomial
1.1 Gbyte disk (unformatted, G=10**9) might convert to 0.91 Gbytes of
formatted space.  The Berkeley Unix fast file system traditionally reserves
10% of the space to insure reasonable performance and minimize
fragmentation.  (Most OS's use some disk space for their own purposes, I
just happen to be most familar with Berkeley Unix).  So a user only gets
0.82 Gbytes (G=10**9) or 0.75 Gbytes (G=2**30) of real usable disk space,
considerable less than he might have expected.  Very much a buyer beware
scenario.  Of course, if you can buy twice as much for half the price
next year, it may not hurt as much as it would if disk technology were
on a slower growth curve.

- Patrick McGehearty
Newsgroups: comp.benchmarks
From: ajn@physics.wm.edu (Alastair Neil)
Subject: Specrate Multiprocessor Benchmark
Reply-To: ajn@physics.wm.edu
Organization: Rio de Caca Illuminati
Date: Wed, 19 Aug 1992 18:11:41 GMT

anyone have any preliminary specrate measurements for SPARC 10 systems,
IBM rs6000, and HP systems?

---
+-----------------------------------------------------------------------------+
|..Alastair Neil................................|                             |
|..(804)-221-3533..[ajn@physics.wm.edu].........|       None Shall Sleep      |
+-----------------------------------------------------------------------------+
Newsgroups: comp.benchmarks
From: cutler@equator.com (Ben Cutler)
Subject: Convex SPECmarks
Summary: Looking for numbers
Organization: Equator Technologies, Inc., Seattle, WA
Date: Mon, 17 Aug 1992 17:53:07 GMT

A few weeks back, somebody posted some Convex SPECmarks.

If that person reads this, I'd be grateful if they could
e-mail me the numbers at cutler@equator.com.

Thanks in advance.

                                Ben Cutler
                                cutler@equator.com
From: davidsen@ariel.crd.GE.COM (william E Davidsen)
Newsgroups: comp.benchmarks
Subject: Re: SPECmarks and Multiprocessor Machines
Date: 12 Aug 92 17:32:27 GMT
Reply-To: davidsen@crd.ge.com (bill davidsen)
Organization: GE Corporate R&D Center, Schenectady NY

In article <BsuAp1.KLK@cup.hp.com>, raj@cup.hp.com (Rick Jones) writes:

| Finally, SPEC is always looking for a few good benchmarks, so if you do 
| have a benchmark that is "MP," you might want to consider proposing it. 
| I imagine that as MP becomes more prevalent outside of the circles of 
| supercomputing, that SPEC's desire for such a benchmark would increase.

As Jeff mentioned, SPEC thinks the current programs are paralellizable,
although not a lot for most of them IMHO. My person MP benchmark is

	soelim textbook.t | tbl | eqn | troff -Tpost -mm | dpost >tb.ps

which is then compared to the time it took (~4hr cpu) on an 11/780, back
when we had such things.
-- 
bill davidsen, GE Corp. R&D Center; Box 8; Schenectady NY 12345
    I admit that when I was in school I wrote COBOL. But I didn't compile.
From: shafer@CS.ColoState.EDU (spencer shafer)
Newsgroups: comp.benchmarks
Subject: Re: Geometric Mean or Median
Date: 14 Aug 92 14:21:26 GMT
Distribution: comp.benchmarks
Organization: Colorado State University, Computer Science Department



Having joined this in midstream, I risk repeating some information.  If
so, my apologies in advance.  A discussion of this, and an offered proof
of the geometric mean as preferred method is in the March 1986 issue of 
Communications of the ACM, "How Not to Lie With Statistics: The Correct
Way to Summarize Benchmark Results," by Fleming and Wallace.

DrLove
-- 
/\__________________________________________________________________/\
||                     SPENCER SHAFER                               ||
|| email:shafer@handel.cs.colostate.edu    Work Phone: 303-498-0901 ||
\/------------------------------------------------------------------\/
Newsgroups: comp.benchmarks
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/23/92] TPC Transaction Processing Council   (21/28) c.be FAQ
Keywords: business, online transactions processing, database,
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Fri, 21 Aug 92 11:25:12 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

21	TPC						<This panel>
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to FAQ chain and netiquette
2
3	PERFECT
4
5	Performance Metrics
6	Temporary scaffold of New FAQ material
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Benchmark Environments
13
14
15	12 Ways to Fool the Masses with Benchmarks
16	SPEC
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence

OLTP (On-Line Transaction Processing): Jim Gray, DEC, was Tandem, was IBM
	Has a new book out on the subject.  Also O. Serlin follows this.

TPC A and TPC B benchmarks are well documented, audited by outside firms,
and the results kept by the Transaction Processing Council.
The TPC administrator is:

   Shanley Public Relations
   177 N. First Street
   Suite 600
   San Jose, CA 95112-6311

TPC/A TPC/B
Location: FTP from DG.COM (128.222.1.2), login anonymous, directory tpc.

Name            Description

Results-text    Results spreadsheet
                (format: tab separated text)
                (size: 15KB)

Results-sylk    Results spreadsheet
                (format: SYLK spreadsheet interchange format)
                (size: 26KB)

TPCARev1.1      TPC Benchmark A Standard Specification Rev 1.1
                (format: Macintosh Microsoft Word)
                (size: 153KB)

TPCARev1.1-PS   TPC Benchmark A Standard Specification Rev 1.1
                (format: Postscript)
                (size: 816KB)

TPCARev1.1-RTF  TPC Benchmark A Standard Specification Rev 1.1
                (format: Rich Text Format - interchange format)
                (size: 243KB)

TPCBRev1.1      TPC Benchmark B Standard Specification Rev 1.1
                (format: Macintosh Microsoft Word)
                (size: 126KB)

TPCBRev1.1-PS   TPC Benchmark B Standard Specification Rev 1.1
                (format: PostScript)
                (size: 993KB)

TPCBRev1.1-RTF  TPC Benchmark B Standard Specification Rev 1.1
                (format: Rich Text Format - interchange format)
                (size: 194KB)

As for the code, the benchmark specs have sample SQL codes, but they may not
help you very much. Every implementation is going to be different. 

After all:
	what is a "transaction?" (but an arbitrary definition.)

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
From: pc@umd.edu
Newsgroups: comp.benchmarks
Subject: Disk storage size?
Date: 18 Aug 92 19:14:21 GMT
Followup-To: comp.sys.ibm.pc.hardware
Organization: UM
Originator: qian@rac2.wam.umd.edu



When people talk about 200MB disk, do they mean  200x1,000,000 byte, or
200x1024x1024 ?  When talk about the RAM, people seems use 1k=1024, but
when go to disk, it is metric. Is this always the case?
From: jimh@chips.com (Jim Hornback)
Newsgroups: comp.benchmarks
Subject: Re: flops.c results update (long)
Summary: where is flops.c source file?
Date: 12 Aug 92 20:52:07 GMT
Distribution: comp.benchmarks
Organization: Chips and Technologies, San Jose, CA

Can someone please tell me where to find the source file flops.c?
Please reply with e-mail.
Newsgroups: comp.benchmarks
Subject: Weitek 4167: Speed in MFLOPS?
From: fantini%boot.decnet@edwards-tems.af.mil
Date: 14 Aug 92 09:11:24 -0700
Organization: T&E Mission Simulator - Edwards AFB, CA

Hello Net,

I have a question about the Weitek 4167 math co-processor:
I know this much: It costs >$500.00 and it is reported to be faster
than the co-processor in the 486DX.

Can anyone tell me the advantages/disadvantages of this device for use
in a 486DX/33? How fast (in MFLOPS) are double precision calculations?
It is sometype of array processor that assist in vector type calculations
but not in scalar? What benchmarks have been run on this device
and what are the figures? How does it compare to (for example) a VAX 6610?

It is worth the cost? Any info and comments (pro and con) are welcome.

						-Thanks
-- 
---------------------------------------------------------------------------
Fantini@edwards-tems.af.mil   | 6510 TSWS U.S.A.F. Simulation Lab
Mathematican & Loose Cannon   | "Zu viel Rechnerkraft? Unmoeglich!" - Me
---------------------------------------------------------------------------
From: eugene@wilbur.nas.nasa.gov (Eugene N. Miya)
Newsgroups: comp.benchmarks
Subject: Re: Geometric Mean or Median
Date: 12 Aug 92 17:22:09 GMT
Distribution: comp.benchmarks
Organization: NAS, NASA Ames Research Center, Moffett Field, CA

In article <1992Aug12.012620.3441@nosc.mil> aburto@nosc.mil
(Alfred A. Aburto) writes:
>Ah, that was a good point!  I was not thinking of the Median as being
>TOO insensitive to extreme data values. I was thinking more from the 
>sense of reliability of the measure of performance, from the consumer 

Well this is why a person should NOT
1) rely on one number like a mean or a median
and should
2) consider other statistics like variance, or the mean (arithmetic,
harmonic, or geometric) of a completely different measure.

Thanks for the defense while I was on vacation.  Now if I can get some time
to work on the FAQ.....

--eugene miya, NASA Ames Research Center, eugene@orville.nas.nasa.gov
  Resident Cynic, Rock of Ages Home for Retired Hackers
  {uunet,mailrus,other gateways}!ames!eugene
Second Favorite email message: Returned mail: Cannot send message for 3 days
A Ref: Mathematics and Plausible Reasoning, vol. 1, G. Polya
From: arperez@mtecv2.mty.itesm.mx (Arnulfo Perez)
Newsgroups: comp.benchmarks
Subject: FAQ
Keywords: FAQ
Date: 12 Aug 92 22:20:55 GMT
Reply-To: arperez@mtecv2.mty.itesm.mx (Arnulfo Perez)
Organization: Centro de Inteligencia Artificial, ITESM, Mexico

Is there a FAQ file for this group?
if not can sombody point to a good review paper on benchmarking?
Thanks.

Arnulfo Perez


----------------------------------------------------------
Pablo Ramirez        |   Internet: pramirez@mtecv1.mty.itesm.mx
Tech Support              | or:    pramirez@mtecv2.mty.itesm.mx
ITESM/IA Center          |     BITNET: pramirez@tecmtyvm.bitnet
Monterrey, NL  64789  |        Voice: (83) 582-00 ext 5132
----------------------------------------------------------

From: pc@hpcuhe.cup.hp.com (Paul Chan)
Date: Tue, 18 Aug 1992 21:57:23 GMT
Subject: Re: SPECmarks and Multiprocessor Machines
Organization: Hewlett Packard, Cupertino
Newsgroups: comp.benchmarks


   So far, there seem to be opposing views on how much SPECmark and SPEC92 
will scale with the number of processors.  However, there are no concrete data
provided by people who made the claim.  Can someone who has experience with
parallelized compilers (from SGI, Sequent, DEC, ConVex) provide data or 
insights? 

  -Paul Chan

Newsgroups: comp.benchmarks