12 Ways to Fool the Masses:
Fast Forward to 2011
David H. Bailey
Lawrence Berkeley National Laboratory
http://crd.lbl.gov/~dhbailey

Example from Physics:
Measurements of Speed of Light

Why the discrepancy between pre-1945 and post-1945 values? Probably
due to biases and sloppy experimental methods.

Example from Psychology:
The “Blank Slate”
The “blank slate” paradigm (1920-1975):
The human mind at birth is a “blank slate.”

Heredity and biology play no significant role in human psychology – allpersonality and behavioral traits are socially constructed.

Current consensus, based on latest research:Humans at birth universally possess sophisticated facilities for socialinteraction, language acquisition, pattern recognition, navigation, etc.

Heredity, evolution and biology are major factors in human personality -some personality traits are more than 50% heritable.

How did the early 20th century scientists get it so wrong?
Sloppy experimental methodology and analysis.

Pervasive wishful thinking and “politically correct” biases.

Ref: Steven Pinker, The Blank Slate: The Modern Denial of Human Nature

Example from Anthropology:
The “Noble Savage”
Anthropologists, beginning with Margaret Mead in the 1930s, painted
an idyllic picture of primitive societies (e.g., Pacific Islanders):
Virtually no violence, jealousy or warfare.
 Happy, uninhibited – few of the psychological or social problems thatafflict much of Western society.
Beginning in the 1970s, a new generation of anthropologists revisitedthese societies and did more careful studies. They found:
Crime rates higher than most U.S. and European cities.
 Death rates from inter-tribe conflicts typically exceeding those of warfare
among Western nations by factors of 10 or more.
 Complex, jealous taboos surrounding courtship and marriage:
◊◊

Violent reprisals were condoned in cases of adultery or non-virgin brides.
Ornamentation worn by males of one tribe, earlier thought to be signs ofmale-female role reversal, were actually honor badges from warfare.

Conclusion: Some advantages, but hardly the Garden of Eden as
earlier depicted.
Why so many errors? “Anthropological malpractice” – Pinker.

Lessons From History

Research must be based on solid empirical tests and careful,objective analysis of data.

Researchers must be willing to provide all details of theexperimental environment, so others can reproduce their results.

Rigorous peer review is essential.

Good intentions or “politically correct” conclusions are no excuses
for poor scholarship.

Erudite technical terminology and mathematical formulas are nosubstitutes for sound reasoning.

Hype has no place in the scientific enterprise.

High standards of honesty and rigor must be vigilantly enforced.

History of Parallel Computing

1976-1986: Initial research studies and demonstrations.

1986-1990: First large-scale systems deployed.

1990-1994: Shoddy measurements and questionable performance
claims; faults generally ignored.



1994-1998: Numerous firms failed; government agencies cut funds.



1998-2002: Reassessment.



2002-2009: Recovery.



2010: 1 Pflop/s (1015 floating-point operations per second) demonstrated
on a few large scientific computations.



2011: Hetergeneous architectures introduced; researchers andgovernment agencies set sights on exascale computing.

Have lessons been learned? Or are we slipping into hype and sloppiness?

Parallel System PerformancePractices, circa 1993


Performance results on small-sized parallel systems were linearly
scaled to full-sized systems.
◊Example: 8,192-CPU results were linearly scaled to 65,536-CPU results,
simply by multiplying by eight.

◊Rationale: “We can’t afford a full-sized system.”◊

Sometimes this was done without any clear disclosure in the paper orpresentation.

Parallel System Performance
Practices, circa 1993
Highly tuned parallel implementations were compared withuntuned implementations on other systems.
◊In comparisons of distributed memory systems with vector systems,
often little or no effort was made to tune the vector code.

◊This was the case even for comparisons between SIMD parallelsystems and vector systems -- here the SIMD code could havebeen converted rather easily to efficient vector code, but typicallythis was not done.

Parallel System Performance
Practices, circa 1993


Inefficient algorithms were employed, requiring many moreoperations, which resulted in artificially high Mflop/s rates.
◊Some scientists employed explicit PDE schemes for applications
where implicit schemes were known to be much better.

◊One paper described doing a 3D discrete Fourier transform by direct
evaluation of the defining formula, rather than by using a fast Fouriertransform (i.e., 8n2 operations rather than 5n log2n, where n is thetotal number of points).

Parallel System Performance
Practices, circa 1993


Performance rates on 32-bit floating-point data on one systemwere compared with rates on 64-bit data on another system.
◊Using 32-bit data instead of 64-bit data effectively doubles data
bandwidth, thus yielding artificially high performance rates.

◊Some computations can be done safely with 32-bit floating-pointarithmetic, but many cannot.

◊In some emerging applications, even 64-bit (15-digit) floating-point
arithmetic is not enough – many more digits are required.

Parallel System PerformancePractices, circa 1993


In some cases, performance experiments claimed in published papers
were not actually performed. Example:◊◊◊

Abstract of published paper:“The current Connection Machine implementation runs at 300-800 Mflop/s
on a full [64K] CM-2, or at the speed of a single processor of a Cray-2 on
1/4 of a CM-2.”
Excerpt from text:“This computation requires 568 iterations (taking 272 seconds) on a 16KConnection Machine.”
In other words, the computation was run on a 16K system, not on a 64Ksystem. The figures cited in the abstract were multiplied by four.
Another excerpt from text:“In contrast, a Convex C210 requires 909 seconds to compute thisexample. Experience indicates that for a wide range of problems, a C210 isabout 1/4 the speed of a single processor Cray-2, …”
In other words, the comparison computation mentioned in the abstract was
not actually run on a Cray-2. Instead, it was run on a Convex system, and aquestionable scaling factor was used to produce the Cray-2 rate.

Performance Plot A
3.5

3Time (Hours)

2.5

2

1.5

1

0.5

001000

2000

3000

4000

5000

6000

Number of Objects

7000

8000

9000

10000

Data for Plot A
Problem size(x axis)
20
40
80
160
990
9600

Parallel system
run time
8:18
9:11
11:59
15:07
21:32
31:36

Vector system
run time
0:16
0:26
0:57
2:11
19:00
3:11:50*

Details in text of paper: In last entry, the 3:11:50 figure is an “estimate.”
 The vector system code is “not optimized.”
Note that the vector system performance is better in each run, except for
the last (estimated) entry.

Performance Plot B
10 3

CPU Seconds per Iteration

10 2

10 1

10 0

10 -1
10 3

10 4

10 5
Number of Grid Cells

10 610 7

Facts for Plot B32-bit performance rates on a parallel system are comparedwith 64-bit performance on a vector system.

Parallel system results are linearly extrapolated to a full-sized
system from a small system (only 1/8 size).

The vector version of code is “unvectorized.”

The vector system “curves” are straight lines – i.e., they arelinear extrapolations from a single data point.

Only a handful of the points in these graphs clearly represent realtimings.

Parallel System Performance
Practices, circa 1993
The examples in my files were written by professional scientists, andmost were published in peer-reviewed journals and conferenceproceedings.
 One example is from an award-winning paper.
 In some cases, researchers accepted free computer time or otherbenefits from vendors, but did not disclose this fact in their papers.
Twelve Ways to Fool the Masses(1991)
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

Quote only 32-bit performance results, not 64-bit results.
Present performance figures for an inner kernel, and then represent these figures
as the performance of the entire application.
Quietly employ assembly code and other low-level language constructs.
Scale up the problem size with the number of processors, but omit any mention ofthis fact.
Quote performance results projected to a full system.
Compare your results against scalar, unoptimized code on conventional systems.
When direct run time comparisons are required, compare with an old code on anobsolete system.
If Mflop/s rates must be quoted, base the operation count on the parallel
implementation, not on the best sequential implementation.
Quote performance in terms of processor utilization, parallel speedups or Mflop/sper dollar.
Mutilate the algorithm used in the parallel implementation to match thearchitecture.
Measure parallel run times on a dedicated system, but measure conventional runtimes in a busy environment.
If all else fails, show pretty pictures and animated videos, and don't talk aboutperformance.

New York Times, 22 Sept 1991

Excerpts from NY Times Article“Rival supercomputer and work station manufacturers are prone tohype, choosing the performance figures that make their own
systems look better.”“It’s not really to the point of widespread fraud, but if people aren’t
somewhat more circumspect, it could give the field a badname.”Fast Forward to 2011:
Five New Ways to Fool the Masses
1. Cite performance rates for a run with only one processor core
active in a shared-memory multi-core node. For example, cite
performance on 1024 cores, even though the code was run on1024 nodes, wasting 15 out of 16 cores on each node.
2. Cite performance rates only for a core algorithms (such as FFT orLU decomposition), even though the paper mentions one or more
full-scale applications that were done on the system.
3. List only the best performance figure in the paper, even though therun was made numerous times.
4. Employ special hardware, operating system or compiler settings
that are not appropriate for real-world usage.
5. Define “scalability” as successful execution on a large number ofCPUs, regardless of performance.

Benchmarks Help Prevent Abuse
Benchmark tests must be well-designed, rigorous and scalable.
 Benchmark codes are problematic, because architectures andlanguages change, but “reference implementations” are useful.
 Well-thought-out and well-enforced “ground rules” are essential.
 A rational scheme must be provided for calculating performancerates.
 Tests must be specified to validate the correctness of the results.
 A repository of results must be maintained.
Examples: The NAS Parallel Benchmarks (still useful, but a bit dated).
 The LBNL-UCB “Torch” Reference Kernels (also known as the“Dwarfs” or “Motifs”). See:
http://crd.lbl.gov/~dhbailey/dhbpapers/dwarfs09_intro.pdf

General Guidelines to Prevent Abuse Direct comparisons of run times on real applications are preferred.
 If results are presented for a well-known benchmark, established ground rules
must be followed.
 Only actual performance results should be presented, not projections orextrapolations (unless very clearly disclosed and justified).
 Performance figures should be based on comparable levels of tuning.
 Mflop/s, Gflop/s, Tflop/s rates should be computed from operation counts
based on the best practical serial algorithms.
 When computing parallel speedup figures, the denominator rate should bebased on an efficient single-processor implementation.
 Any ancillary information that would affect the interpretation of the results
should be fully disclosed (e.g., the use of 32-bit instead of 64-bit data, etc.).
 Special care should be taken for figures and graphs.
 Whenever possible, full background information should be provided:algorithms, hardware and software configuration, language, compiler flags,tuning, timing method, basis for operation counts, etc.