HDTenEightyP/NetNews huggingface dataset:
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/17/92] Measurement environments		(12/28) c.be FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Sat, 12 Sep 92 11:25:05 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

12	Measurement environments			<This panel>
13	SLALOM
14
15	12 Ways to Fool the Masses with Benchmarks
16	SPEC
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to FAQ chain and netiquette
2
3	PERFECT
4
5	Performance Metrics
6	Temporary scaffold of New FAQ material
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11

Benchmarking environments
Taxonomy described by: A, B, C, D, E
Examples:
	A Cray Y-MP or better with integrated hardware performance monitor
	B Cray-2, Convex C-1/C-2...
	C VAX-11/780, IBM PC, IBM 370-class
	D Apple II, Timex Sinclair
	E Incomplete hardware, chip, module, board-level

An A environment should have state of the art hardware and
software to aid benchmarking.  Hardware should be minimally or non-intrusive.
This category should change with time.  The SS-1 should have a
more programmable HPM, this would lower the Y-MP down into
a B environment (can stay A if no other HPM type hardware becomes
predominant).  This is akin to high-precision Cesium clocks.

A B environment is a "good" environment to do benchmarking.
It has as a minimal, a near cycle-time clock, profiling software,
etc.  The environment at this level or higher may also be more hostile
for measurement.  Good performance is likely to be an issue, so the
compilers will have good optimizers.

A C environment is an "average" benchmarking environment.
This is just typical: a 60 or 100 Hz clock.  Some profiling software.

A D environment is a difficult environment to do benchmarking.
This environment may not have a clock or software to help the poor
benchmarker.  Timing by wrist watch or maybe oscilliscope.
Software? What software?

An E environment:
Another difficult environment to do benchmarking.  It might have software.
It might be an incomplete computer: a processing using of a larger
multiprocessor: it touches on the issue of composition.

If dealing with a simulator:
there maybe an F category for simulated timing.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/23/92] Linpack				c.be FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Wed, 9 Sep 92 11:25:10 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

9	Linpack					<This panel>
10
11
12	Measurement Environments
13
14
15	12 Ways to Fool the Masses with Benchmarks
16	SPEC
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to FAQ chain and netiquette
2	Benchmarking Concepts
3	PERFECT
4
5	Performance Metrics
6	Temporary scaffold of New FAQ material
7	Music to benchmark by
8	Benchmark types

The LINPACK benchmark is a very simple LU decomposition of a dense linear
system (Gaussian elimination) by Jack Dongarra the developer of the
LINPACK library and the netlib numerical software server.

Ref: Dongarra's article in CACM on Netlib.
And ACM SIGARCH Computer Architecture News and SIGNUM Newsletter.

Advantages:
Simple fairly portable FORTRAN.  One of the shorter benchamrks.
It is small enough to be carried on disk or in Jack's laptop without
consuming too much porting time.
A good attempt at experiment control, has stringent execution requirements.

Dongarra also records the compiler options used to invoke the Fortran
compilers.  Record keeping is good.  Reports are quickly available
electronically and published with some frequency in Supercomputing Review.

Disadvantages:
Diminishing parallelism during the decomposition (as in all Gaussian
elimination).  It is perhaps too simple.  Execution time is on the shorter
side.  It cannot characterize less numeric applications.

NETLIB benchmark index (Linpack benchmark)
netlib@ornl.gov
	send index from benchmark
includes linpack (100x100,300x300,1000x1000).

What is interesting about these sizes is that they are not
the powers of 2 which characterize many benchmarks.  Powers of 2
can bias in favor of some architectures and bias against other
architectures.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
From: eoahmad@ntuix.ntu.ac.sg (Othman Ahmad)
Subject: Re: What is AIM stand for ?
Organization: Nanyang Technological University - Singapore
X-Newsreader: Tin 1.1 PL4
Date: Fri, 11 Sep 1992 03:40:46 GMT

rec@aim.com (Bob Cousins) writes:
: 
: However, for the Aim Performance Report (the first page of which is
: published in the Price Performance Guide), to guarantee an
: apples-to-apples comparison, we use the following mix for the Suite
: III run:
: 
: 	20% main memory
: 	10% fp (single and double)
: 	20% integer
: 	10% IPC
: 	20% function calls
: 	20% disk I/O
:        ----
:        100%
: 
: This mix has been in use for more than two years and has proven to be
: highly satisfactory. We arrived at this mix by working with our
: customers, some key industry experts and consultants. Amazingly, we
: reached consensus on this mix quite quickly. (Some vendors have their
: own agendas which can make consensus slow or impossible to reach.) The
It shows that you must have separate programs testing those portions that you
want. You miss the "cache" evaluation. 

Instead of trying to predict how a software behaves, why don't you just
characterise the "requirements" of typical software.
	Use your test suites to predict the performance of that software.

I believe this is the way engineers work. They introduce concepts such as
complex impedence, noise, distortion, to a machine, and characterise how a
signal behave in those environments.
	It is just too hard to predict the "typical" resource mix. Let the
user determine it himself.
	If we have a tool that can measure the characteristics of a software,
in terms of those "propertise", better still.

--
Othman bin Ahmad, School of EEE,
Nanyang Technological University, Singapore 2263.
Internet Email: eoahmad@ntuix.ntu.ac.sg
Bitnet Email: eoahmad@ntuvax.bitnet
From: ricky@benchmaster.esd.sgi.com (Ricky Cheng)
Newsgroups: comp.benchmarks
Subject: What is AIM stand for ?
Date: 8 Sep 92 22:04:01 GMT
Organization: Silicon Graphics, Inc.

Does anyone know what AIM stand for ?
"AIM" is the AIM benchmark from AIM Technology.

Ricky
From: rec@aim.com (Bob Cousins)
Subject: Re: What [does] AIM stand for ?
Keywords: AIM
Organization: /usr/lib/news/organization
Date: Wed, 9 Sep 1992 17:39:02 GMT

In article <laqeqgINNchc@utkcs2.cs.utk.edu> larose@thud.cs.utk.edu (Brian LaRose) writes:
>
>>>  What [does] AIM stand for ?
>
>  I have the Spring 1992 Price/Performance Guide in my hands
>and I can't find out what AIM stands for.   

"AIM" is not an acronym for anything (if that is what you are asking).
AIM Technology, established in 1981, is an *INDEPENDENT* organization
that specializes in UNIX performance products and services. AIM's goal
is to provide customers reliable information for UNIX system purchases.

All AIM ratings and reports are *INDEPENDENTLY CERTIFIED* by AIM Technology.
AIM Technology is not obligated to any outside commercial interest.

AIM products are used worldwide by virtually every computer manufacturer and
are considered standards nationally at hundreds of Fortune 1000 companies and 
internationally government agencies throughout the world.

>It seems they only refer to themselves as:
>
>       AIM Technology
>       4699 Old Ironsides Drive, Suite 150
>       Santa Clara, CA 95054
>       Phone:   408-748-8649
>                1-800-848-UNIX
>                         ^
>                         |
>                       hmmmm..
>
>     one has to wonder if they are associated with AT&T.  Berkeley Systems

AIM is not in any way associated with AT&T.

> Design, Inc  got a MAJOR lawsuit from AT&T by trying to put a 1(800)-ITS-UNIX
> number in it's publications.  I bet either AT&T is intentionally trying to 
> kill BSDI, OR owns some interest in AIM technology...

AT&T does not own any portion of AIM. (AIM is, in fact, privately held.)
Note that *BOTH* of our numbers end in 8649 or "UNIX" depending on how you
interpret them. These numbers have been in use for more than 6 years. AT&T
knows of the numbers (since AT&T is one of our customers and calls us from
time to time).

It is important to note that AIM does not offer any version of UNIX (licensed
or not) and that there is a big difference between a phone number which claims
"its UNIX" versus a phone number which *CONTAINS* "UNIX" as a memory device.

>
>Personally, I'd stick to public domain benchmarks.

This is your right. However, if you are interested in *STATE OF THE ART*
and *INDEPENDENT* performance measurement, you will have difficulty.
>-- 
>    -------------------------------------
>    | Brian LaRose |  974-3647          |
>    | Ayres 217    |  larose@cs.utk.edu |
>    -------------------------------------
>    


-- 
________________________________________________________________________________
Robert Cousins                    rec@aim.com                  (408)748-8649x212
Consultant to Aim Technology                          Speaking for myself alone.
Newsgroups: comp.benchmarks
From: eugene@wilbur.nas.nasa.gov (Eugene N. Miya)
Subject: Re: What is AIM stand for ?
Organization: NAS, NASA Ames Research Center, Moffett Field, CA
Date: Wed, 9 Sep 92 06:16:31 GMT

I don't that AIM stood for anything.  Sort of like Unix (which is closer
to a pun more than anything else [news.announce.newusers and certain
Usenix and Bell System Tech. Jour.]).

The person to ask of course is Gene Dronek who wrote the excellent:

%A Gene Dronek
%Z AIM Technology
%T Relating Benchmarks to Performance Projections or
What do you do with 20 pounds of benchmark data?
%J Summer 1984 Addendum to the Conference Proceedings
%I Usenix, Association
%C Salt Lake City, UT
%D June 1984
%P 12-15
%K VIEWS, Kiviat diagrams, Q-BENCH, Unix,
%X This paper offers some useful ideas.

Gene was a consult here briefly.  I trust he's doing well.
You can ask AIM of course. 8^)
I still have to catch up reading this group.

--eugene miya, NASA Ames Research Center, eugene@orville.nas.nasa.gov
  Acting Associate Editor, Software and Publication Reviews
  Scientific Programming
  {uunet,mailrus,other gateways}!ames!eugene
Ref: Mathematics and Plausible Reasoning, vol. 1, G. Polya
Newsgroups: comp.benchmarks
From: andr@elvis.sovusa.com (Andrei Arkhipov)
Subject: SPECmarks
Reply-To: andr@elvis.sovusa.com
Organization: ELVIS Corp.
Date: Sun, 6 Sep 92 21:04:03 GMT

I wonder if the sources of SPECmarks program available?

Answer to e-mail address, please.

Andrei.
From: larose@thud.cs.utk.edu (Brian LaRose)
Newsgroups: comp.benchmarks
Subject: Re: What [does] AIM stand for ?
Date: 9 Sep 92 21:17:40 GMT
Organization: University of Tennessee, Knoxville - Department of Computer Science
Distribution: world
Reply-To: larose@thud.cs.utk.edu (Brian LaRose)
NNTP-Posting-Host: thud.cs.utk.edu
Keywords: AIM

>>  AT&T
>>   knows of the numbers (since AT&T is one of our customers and calls us from
>>   time to time).

>> It is important to note that AIM does not offer any version of UNIX (licensed
>> or not) and that there is a big difference between a phone number which claims
>> "its UNIX" versus a phone number which *CONTAINS* "UNIX" as a memory device.

ok.  I'll buy that. It even sounds reasonable.

I guess AT&T is only concerned with competition, and from your point of view
you only augment the Unix product with some performance measures.  
Poor ole' BSDI, they weren't so lucky.

>> This is your right. However, if you are interested in *STATE OF THE ART*
>> and *INDEPENDENT* performance measurement, you will have difficulty.

I guess we'll have to agree to disagree on this point, but that's ok.

 I believe that there are still a few independent, scientific, 
 public domain benchmarks out there. 


------------
brian
From: eoahmad@ntuix.ntu.ac.sg (Othman Ahmad)
Subject: Re: Surprised:gcc&dhryston benchmarks
Organization: Nanyang Technological University - Singapore
X-Newsreader: Tin 1.1 PL4
Date: Fri, 11 Sep 1992 03:30:40 GMT

patrick@convex.COM (Patrick F. McGehearty) writes:
: >
: >Some of those small programs are 
: >dhryston for integer small programs,
: >lorenze for floating point small programs,
: >iozone for portable I/O,
: >
: lorenze can be extremely misleading for predicting floating point
: performance.  As written, there is a subroutine call for every few
: floating point operations.  If you have a compiler which inlines

That is exactly what we want. We want to benchmark a program that is most
likely to be written by a user.
	I call this the introduction of redundancies. I have attempted to
measure these redundancies in cc1 and dhry-1.1 in my previous article.

: that subroutine, AND you use the right options to get that inlining,
: your performance will be dramatically different (a factor of 2 on some
: machines) than if you don't inline the results.  A factor of 2 is bit more
: than should be waved off as background noise.  If you must use something
	I do not consider this as noise, rather "affordability". The optimising
compiler must be available for a machine. The speed of a machine is the one
that a user can extract. No point in giving a theoretical maximum speed if the
user has no access to these optimising compilers.

: that simple, try Dongarra's 100x100 Linpack benchmark (there are C versions
: floating around).  Results are reported regularly for a long list of
: machines, and scrutinized by many people.  Much better (but also much more
: work to get running) is to use the Perfect Club benchmarks. Another
These programs are better but also bigger and more difficult to set up and
transport. The results are not useful in some cases e.g for small user 
programs. The cache requirements are different. We still need these small
programs to measure the cache characteristics.

: alternative would be to write some loops for common floating operations such
: as matrix multiply, correlation, sum reduction, etc.  Do not include
: unnecessary subroutine calls in the inner loops as that just causes the
: program to be measuring something besides floating point performance.
This is too synthetic. It does not reflect they way a user writes programs.
dhryston and lorenze try to emulate the way a user writes programs.

--
Othman bin Ahmad, School of EEE,
Nanyang Technological University, Singapore 2263.
Internet Email: eoahmad@ntuix.ntu.ac.sg
Bitnet Email: eoahmad@ntuvax.bitnet

Newsgroups: comp.benchmarks
From: Patrick F. McGehearty <patrick@convex.COM>
Subject: Re: Surprised:gcc&dhryston benchmarks
Originator: patrick@mozart.convex.com
Date: Thu, 10 Sep 1992 16:57:14 GMT
Reply-To: patrick@convex.COM (Patrick F. McGehearty)
Organization: Engineering, CONVEX Computer Corp., Richardson, Tx., USA
X-Disclaimer: This message was written by a user at CONVEX Computer
              Corp. The opinions expressed are those of the user and
              not necessarily those of CONVEX.

In article <1992Sep10.131908.17807@ntuix.ntu.ac.sg> eoahmad@ntuix.ntu.ac.sg (Othman Ahmad) writes:
...
>Conclusion:
>dhryston 1.1 is not yet useless because it actually gives a reasonable
>processor speed indication for small programs. Sometimes this is what we
>want.
>
In another posting I give some reasons which Dhrystone has the potential
to be misleading.  Those comments were for Dhrystone-2.1.  Dhrystone-1.1 is
even worse.  A sufficiently advanced optimizing compiler can eliminate
almost all of the code in Dhrystone-1.1.  What do the measurements mean
then?  Nothing interesting.

>My recommendations to Spec Mark consortium is to find small programs that
>characterise a computer system characteristics, e.g. cache, RAM, disk,
>Character, Graphic, serial, interrupt-response etc.
>
>Then for each software, what portions of those characteristics do they 
>consume. Cross-correlation with predicted results using those basic 
>characteristics for a software, and the actual results of running user 
>software should be done, to validate the "usefulness of those programs".
>
>Some of those small programs are 
>dhryston for integer small programs,
>lorenze for floating point small programs,
>iozone for portable I/O,
>
lorenze can be extremely misleading for predicting floating point
performance.  As written, there is a subroutine call for every few
floating point operations.  If you have a compiler which inlines
that subroutine, AND you use the right options to get that inlining,
your performance will be dramatically different (a factor of 2 on some
machines) than if you don't inline the results.  A factor of 2 is bit more
than should be waved off as background noise.  If you must use something
that simple, try Dongarra's 100x100 Linpack benchmark (there are C versions
floating around).  Results are reported regularly for a long list of
machines, and scrutinized by many people.  Much better (but also much more
work to get running) is to use the Perfect Club benchmarks. Another
alternative would be to write some loops for common floating operations such
as matrix multiply, correlation, sum reduction, etc.  Do not include
unnecessary subroutine calls in the inner loops as that just causes the
program to be measuring something besides floating point performance.
Newsgroups: comp.benchmarks
From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Subject: [l/m 3/17/91] Ways to fool the masses with benchmarks	(15/28) c.be. FAQ
Keywords: who, what, where, when, why, how
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA
Date: Tue, 15 Sep 92 11:25:18 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)

15	12 Ways to Fool the Masses with Benchmarks	<This panel>
16	SPEC
17	Benchmark invalidation methods
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25	Ridiculously short benchmarks
26	Other miscellaneous benchmarks
27
28	References
1	Introduction to the FAQ chain and netiquette
2
3	PERFECT Club
4
5	Performance Metrics
6
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Measurement Environments
13	SLALOM
14

From David Bailey et al.

Quote only 32-bit performance results, not 64-bit results.

Present performance figures for an inner kernel as the performance of the
entire application.

Quietly employ assembly code and other low-level language constructs.

Scale up the problem size with the number of processors, but omit any
mention of this fact.

Quote performance results projected to a full system.

Compare your results against scalar, unoptimized code on Crays.

When direct run time comparisons are required, compare with an old code
on an obsolete system.

If megaFLOPS rates must be quoted, base the operation count on the parallel
implementation, not on the best sequential implementation.

Quote performance in terms of processor utilization, parallel speedup or
megaFLOPS per dollar.

Multilate the algorithm used in the parallel implementation to match
the architecture.

Measure parallel run times on a dedicated system, but measure conventional
run times in a busy environment.

If all else fails, show pretty pictures and animated videos, and don't talk
about performance.

Ref. 1
NAS TR #

Ref. 2
%A David Bailey
%T Twelve Ways to Fool the Masses when Giving Performance Results on
Parallel Computers
%J Supercomputing Review
%V 4
%N 8
%D August 1991
%P 54-55

References
Darrell Huff, How to Lie with Statistics
	How to Lie with Maps

Gordon Bell's 11 rules of supercomputer design:

1) Performance, performance, performance.
2) Everything matters.
3) Scalar matter most.
4) Provide vectors as price allows.
5) Avoid holes in performance.
6) Place peaks in performance.

7) Provide a decade of addressing.
8) Make it easy to use.
9) Build on others work.
10) Design for the next one (and do it again) [three times]
11) Have slack resources.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language
 
Newsgroups: comp.benchmarks
From: hoelzle@Xenon.Stanford.EDU (Urs Hoelzle)
Subject: Re: A little malloc speed test.
Organization: CS Department, Stanford University, California, USA
Date: 15 Sep 92 20:30:06 GMT

bt@irfu.se (Bo Thide') writes:

>I was curious to see the improvement in malloc speed between HP-UX 8.05
>and HP-UX 8.07 on our Snakes (720 and 730) and ran the test below.  It
>seems that the 8.07 malloc compares favourably to the GNU malloc on the
>HP and to the Suns I had available.  I would be interested in
>comparisons with other platforms and malloc's.  For those systems that
>support mallopt uncommenting (and possible modification of) the mallopt
>line in the code can be elucidating.

Try these:

%T Custo-Malloc: efficient synthesized memory allocators
%A Dirk Grunwald
%A Benjamin Zorn
%J CU-CS-602-92
%D July 1992
%X trace program, then generate customized malloc for it; faster than all
standard methods on set of 7 large C programs

%T Empirical measurements of 6 allocation-intensive C programs
%A Benjamin Zorn
%A Dirk Grunwald
%J CU-CS-604-92
%D July 1992
%X useful data (size distrib., mean time between allocs, life times)

%T The measured cost of conservative garbage collection
%A Benjamin Zorn
%J U Col Boulder CU-CS-573-92
%D April 1992
%X measures Boehm 1.6 (non-generational) against various mallocs; cache misses,
page fault measurements.  Overall result: pretty comparable (but ignores 
optimization problem)

-Urs

----------------------------------------------------------------------------
Urs Hoelzle                                              urs@cs.stanford.EDU
Computer Systems Laboratory, CIS 57, Stanford University, Stanford, CA 94305
Newsgroups: comp.benchmarks