From: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Newsgroups: comp.benchmarks
Subject: [l/m 3/17/92] Performance metrics -- terminology (5/28)	c.be FAQ
Keywords: who, what, where, when, why, how
Date: 5 Apr 92 11:25:51 GMT
Reply-To: eugene@amelia.nas.nasa.gov (Eugene N. Miya)
Organization: NAS Program, NASA Ames Research Center, Moffett Field, CA

5	Performance Metrics terminology				<This panel>
6	Temporary scaffold of New FAQ material
7	Music to benchmark by
8	Benchmark types
9	Linpack
10
11
12	Benchmark Environments
13
14
15	12 Ways to Fool the Masses with Benchmarks
16	SPEC
17
18
19	WPI Benchmark
20	Equivalence
21	TPC
22
23
24
25
26
27
28	References
1	Introduction to FAQ chain and netiquette
2
3	PERFECT
4


Performance/benchmark metric terminology

The usual important quote is:
	What's important is the time it takes to solve MY problem.
This does not help the architect designing tne next machine.
It is a arrogant closed minded, Gestaltist statement which conflicts
with the analytic/reductionist needs for science.

Synthetic problems/benchmarks have some if limited value.
We walk before we run, and we crawl before we walk.  Similarly,
right now, there is more benchmarking noise than signal.

Perhaps the only, certainly best measure, is the second (time):
one of the best studied metrics see the atomic clocks of the NIST.
Subject to relativistic effects: the Lorentz time contraction.
Don't laugh, this is becoming more important at the pico-second level

Less reliable measures include:

MIP, GIP, TIP	:
MIPS, GIPS, TIPS:	Million (Giga, billions; tera, trillion) Instructions
	Per Second
	:	Meaningless Indicator of Performance
	:	"Marketing's" Indicator of Performance
What's an "instruction?"
	An instruction is an event.  It is frequently a minunt change in the
	state of a CPU (and the computer).  Frequently, an instruction if
	synonymous with the clock rate of a machine: that ignores instructions
	requiring more than one clock pulse tick to execute.

	A common fallacy by naive benchmarkers is that a CPU determines the
	speed of a computation.  This is frequently false.  The people in
	the know these days understand 
	Amdahl's "other law:"
	1 MIPS for each 1 MB main memory at 1 MB/S transfer to disk

MFLOPS, GFLOPS, TFLOPS: Million (Giga, billions; tera, trillion) Floating-Point
	Operations Per Second
	: The measure ignores non-floating-point instructions.  Particularly
	bad for numeric codes transitioning from 2-D to 3-D since additional
	time is required for array address calculation, and for algorithms
	requiring big non-numeric steps like matrix transposition.
	: the original program name for Frank McMahon's Livermore Loops
	program.
	: one of the metrics used by Dongarra's LINPACK benchmark.

LIPS, KLIPS, MLIPS: Logical "inferences" Per Second --
	from the Logic Programming community (Gabriel LISP benchmarks).
	Also available in Prolog (Evan Tick).  LIPS roughly correspond to
	"calls per second" for very simple predicates.

Packets Per Second:	Unit of measure used by the networking, communications
	community.  Sometimes useful.
	: What do they do make consistent packets?

MHz, GHz, Bits per Second, Bytes per Second, Words per Second:
	: Frequently used to mismeasure the performance of computer networks
	like Ethernet (tm).  It confuses the base band carrier frequency
	with the data trasnfer rate.  It's not truth, but not complete false.
	: Also sometimes call Null or Wait instructions.

TPS	:	Transactions per second, agree on metric by transaction
	processing council.
	: What's a transaction?

Stones	: An arbitrary unit of computation based on the Whetstone
	(or Dhrystone or other *stone) which is subject to the influences
	like compiler optimization or cache metrics.
	: What's a transaction?

Normalized metrics

SPECmark: A normalized metric based on the performance against a
	DEC VAX-11/780.  Based on a SPEC workload on a 780 under glass.

Speed up:

Efficiency:

Our problems aren't counting seconds (intervals or days), it's not counting
instructions, operations, floating point operations.

Events counts like instructions or operations are best done by non-instrusive
instruction/operating counting hardware.  These are expensive to say the least.
Software profilers/event counters are also some times useful, but they are
subject to optimization.

We need to distingush "virtual" operations or instructions from
real or actual instructions.

Prefixes:
kilo, mega, giga, tera, eka, peta,
milli, micro, nano, pico, femto,


Performance metrics are unlike conventional mathematics.
You can't make mathematical inferences (excepting "guaranteed not to
exceed numbers"), you can't apply all mathematical operators.  The basis
for metric theory is that for a metric space X and a metric function d()
which maps elements of X to the real number system, then 
	a)
	b) 
	c) d(A + B) <= d(A) + d(B) [triangle inequality]

You might have a benchmrk sized for 128 elements.  A program might not test
well if it used 127 or 129 elements instead.  It is not possible in
infer or interpolate between values because of benchmarking "gotchas."
This is especially bad when dealing with powers of two: an artifact
of computer architecture, but sometimes also due to software (in a base-10
world).

Mathematics derives a large portion of its power because of assumptions of
continuity.  Computers are very discrete objects.  What works for case n might
not work for case n-1 or n+1 (vector architectures for instance).
Some interesting thing are learned by simply modifying the size of a benchmark
by one (remember Kernighan and Plauger: beware off-by-1 errors).

Can you even be assured of consistent measures?
Most benchmarks try to run their tests in standalong conditions to
attain consistency.  This is an artifact of not being able to have a
non-intrusive measurement environment.

=====

Im memoriam to Rear Adm. Grace Murray Hopper, for all the "nano seconds"
and "pico seconds" she passed out (30 cm/1 ft copper wires or salt grains).
She will be missed.

                   ^ A  
                s / \ r                
               m /   \ c              
              h /     \ h            
             t /       \ i          
            i /         \ t        
           r /           \ e      
          o /             \ c    
         g /               \ t  
        l /                 \ u
       A /                   \ r
        <_____________________> e   
                Language