Twelve ways to fool the masses:
Fast forward to 2013
David H Bailey
Lawrence Berkeley National Laboratory, USA (recently retired)
University of California, Davis, Department of Computer Science
This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-12ways-2013.pdf!

1

The need for reproducibility inscientific computing
Computing is now widely recognized as the third leg of science, with theory
and experiment. But the scientific computing culture has not kept pace: The notion that a computation should be reproducible by otherresearchers is still not part of the research culture.
 Physicists, chemists and biologists routinely keep notebooks of every
detail, but few computational experiments are performed with such care.
 Scientific computing has its own concerns:





Is numeric precision adequate? Are algorithms appropriate?
Are there difficulties with numerical reproducibility? How many digits?
What independent validity checks have been employed?
Has the experiment been performed with different hardware/software?
Can results be reproduced with code written by other researchers?

For further information, see:V. Stodden, D.H. Bailey, J.M. Borwein, R.J. LeVeque, W. Rider and W. Stein, “Setting thedefault to reproducible: Reproducibility in computational and experimental mathematics,” 2 Feb2013, available at: http://www.davidhbailey.com/dhbpapers/icerm-report.pdf
2

Example from physics:
Measurements of speed of light

Why is there such a discrepancy between pre-1945 and post-1945 values?
This is probably due to biases and sloppy experimental methods.
3

Example from psychology:
The “blank slate” paradigm
The “blank slate” paradigm (1920-1980):
 The human mind at birth is a “blank slate.” Heredity and biology play no significant role in human psychology – allpersonality and behavioral traits are socially constructed.

Current consensus, based on latest research: Humans at birth possess sophisticated facilities for social interaction,
language acquisition, pattern recognition, navigation and counting.
 Heredity, evolution and biology are major factors in human personality.
 Some personality traits are more than 50% heritable.

How did the early 20th century scientists get it so wrong?
 Sloppy experimental methodology and analysis.
 Pervasive wishful thinking and “politically correct” biases.
Ref: Steven Pinker, The Blank Slate: The Modern Denial of Human Nature
4

Example from anthropology:
The “noble savage” paradigm
Anthropologists, beginning with Margaret Mead in the 1930s, painted
an idyllic picture of primitive societies (e.g., Pacific Islanders):
 Virtually no violence, jealousy or warfare.
 Happy, uninhibited – few of the psychological or social problems that afflictmuch of Western society.

But beginning in the 1970s, a new generation of anthropologists
revisited these societies and did more careful studies. They found: Crime rates higher than most U.S. and European cities.
 Death rates from inter-tribe conflicts typically that exceed those of warfare
among Western nations by factors of 10 or more.
 Psychological problems comparable to Western societies.
 Complex, jealous taboos surrounding courtship and marriage:


Violent reprisals in cases of adultery or non-virgin brides.
Ornamentation worn by males of one tribe, earlier thought to be signs of malefemale role reversal, were actually badges of honor from warfare.

Why so many errors? “Anthropological malpractice” – Pinker.
5Lessons from history
 Research must be based on solid empirical tests and careful,objective analysis of data.
 Researchers must provide all details of the experimental
environment, so others can reproduce their results.
 Rigorous peer review is essential.
 Good intentions or “politically correct” conclusions are no excuses
for poor scholarship.
 Erudite technical terminology and mathematical formulas are nosubstitutes for sound reasoning.
 Hype has no place in the scientific enterprise.
 High standards of honesty and rigor must be vigilantly enforced.
Unless great care is taken, we can fool ourselves as well as others!
6History of parallel computing
 1976-1986: Initial research studies and demonstrations.
 1986-1990: First large-scale systems deployed.
 1990-1994: Shoddy measurements and questionable performance claims;
faults generally ignored.
 1994-1998: Numerous firms failed; government agencies cut funds.
 1998-2002: Reassessment.
 2002-2009: Recovery.
 2010: 1 Pflop/s (1015 floating-point operations per second) demonstrated
on a few large scientific computations.
 2011: Hetergeneous architectures introduced; researchers andgovernment agencies set sights on exascale computing.
 2013: A broad international community is engaged in high-end computing.
Have lessons been learned? Or are we slipping back to hype and distortion?

7

Parallel system performance practices(1991)
 Performance results on small-sized parallel systems were linearly scaled
to full-sized systems.


Example: 8,192-CPU results were linearly scaled to 65,536-CPU results,
simply by multiplying by eight.
Rationale: “We can’t afford a full-sized system.”
Sometimes this was done without any clear disclosure in the paper orpresentation.

8

Parallel system performance practices(1991), continued
 Highly tuned parallel implementations were compared with untuned
implementations on other systems.


In comparisons of distributed memory systems with vector systems, often
little or no effort was made to tune the vector code.
This was the case even for comparisons between SIMD parallel systems andvector systems -- here the SIMD code could have been converted rather
easily to efficient vector code, but typically this was not done.

9Parallel system performance practices(1991), continued
 Inefficient algorithms were employed, requiring many more operations,which resulted in artificially high Mflop/s rates.
Some scientists employed explicit PDE schemes for applications whereimplicit schemes were known to be much better.
One paper described doing a 3D discrete Fourier transform by direct
evaluation of the defining formula, rather than by using a fast Fouriertransform (i.e., 8n2 operations rather than 5n log2n, where n is the total
number of points).

10Parallel system performance practices(1991), continued
 Performance rates on 32-bit floating-point data on one system werecompared with rates on 64-bit data on another system.


Using 32-bit data instead of 64-bit data effectively doubles data bandwidth,
thus yielding artificially high performance rates.
Some computations can be done safely with 32-bit floating-point arithmetic,
but many cannot.
In some emerging applications, even 64-bit (15-digit) floating-point arithmetic
is not enough – many more digits are required.

11

Parallel system performance practices(1991), continued In some cases, performance experiments claimed in published papers
were not actually performed. Example:


Abstract of published paper:“The current Connection Machine implementation runs at 300-800 Mflop/s ona full [64K] CM-2, or at the speed of a single processor of a Cray-2 on 1/4 of aCM-2.”
Excerpt from text:“This computation requires 568 iterations (taking 272 seconds) on a 16K
Connection Machine.”
In other words, the computation was run on a 16K system, not on a 64K
system. The figures cited in the Abstract were merely multiplied by four.
Another excerpt from text:“In contrast, a Convex C210 requires 909 seconds to compute this example.
Experience indicates that for a wide range of problems, a C210 is about 1/4
the speed of a single processor Cray-2, …”
In other words, the computation mentioned in the Abstract was not actually
run on a Cray-2. Instead, it was run on a Convex system, and a questionablescaling factor was used to produce the Cray-2 rate.
12Performance plot A
3.5

3Time (Hours)

2.5

2

1.5

1

0.5

00

1000

2000

3000

4000

5000

6000

Number of Objects
137000

8000

9000

10000

Data for plot AProblem size(x axis)
2040
80
1609909600

Parallel system
run time
8:18
9:11
11:59
15:07
21:32
31:36

Vector system
run time
0:16
0:26
0:57
2:11
19:00
3:11:50*

Details in text of paper: In last entry, the 3:11:50 figure is an “estimate.”
 The vector system code is “not optimized.”
Note that the vector system performance is better in each run, except for
the last (estimated) entry.

14Performance plot B
10 3

CPU Seconds per Iteration

10 2

10 1

10 0

10 -1
10 3

10 4

10 5
Number of Grid Cells
15

10 610 7

Facts for plot B 32-bit performance rates on a parallel system are compared with 64-bit
performance on a vector system.
 Parallel system results are linearly extrapolated to a full-sized systemfrom a small system (only 1/8 size).
 The vector version of code is “unvectorized.” The vector system “curves” are straight lines – i.e., they are linear
extrapolations from a single data point.
Conclusion: Most likely, only a handful of the points in these graphs
represent real timings.

16

Parallel system performance practices(1991): other facts
 The examples in my files were written by professional scientists, andmost were published in peer-reviewed journals and conferenceproceedings.
 One example is from an award-winning paper.
 In some cases, researchers accepted free computer time or otherbenefits from vendors, but did not disclose this fact in their papers.

17

DHB’s “Twelve Ways to Fool the Masses”
from Supercomputing Review, 1991
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.

Quote only 32-bit performance results, not 64-bit results.
Present performance figures for an inner kernel, and then represent these figures
as the performance of the entire application.
Quietly employ assembly code and other low-level language constructs.
Scale up the problem size with the number of processors, but omit any mention ofthis fact.
Quote performance results projected to a full system.
Compare your results against scalar, unoptimized code on conventional systems.
When direct run time comparisons are required, compare with an old code on an
obsolete system.
If Mflop/s rates must be quoted, base the operation count on the parallel
implementation, not on the best sequential implementation.
Quote performance in terms of processor utilization, parallel speedups or Mflop/s
per dollar.
Mutilate the algorithm used in the parallel implementation to match thearchitecture.
Measure parallel run times on a dedicated system, but measure conventional runtimes in a busy environment.
If all else fails, show pretty pictures and animated videos, and don't talk aboutperformance.

18

New York Times, 22 Sep 1991

19

Excerpts from NY Times article“Rival supercomputer and work station manufacturers are prone to hype,choosing the performance figures that make their own systems lookbetter.”“It’s not really to the point of widespread fraud, but if people aren’t somewhat
more circumspect, it could give the field a bad name.”

20Fast forward to 2013:
Some new ways to fool the masses
1. Cite performance rates for a run with only one processor core active in ashared-memory multi-core node. For example, cite performance on1024 cores, even though the code was run on 1024 nodes, wasting 15
out of 16 cores on each node.
2. Cite performance rates only for a core algorithms (such as FFT or LU
decomposition), even though the paper mentions one or more full-scale
applications that were done on the system.
3. List only the best performance figure in the paper, even though the run
was made numerous times.
4. Employ special hardware, operating system or compiler settings that are
not appropriate for real-world usage.
5. Define “scalability” as successful execution on a large number of CPUs,regardless of performance.

21

Gerhard Willein’s 14 new tricks
1. Report speedup instead of absolute performance.
2. Slow down code execution.
3. The log scale is your friend.
4. Quietly employ weak scaling to show off.
5. Instead of performance, plot absolute runtime versus CPU count.
6. Ignore affinity and topology issues.
7. Be creative when comparing scaled performance.
8. Impress your audience with awe-inspiring accuracy.
9. Boast massive speedups with accelerators.
10. Always emphasize the “interesting” part of your work.
11. Show data! Plenty. And then some.
12. Redefine “performance” to suit your needs.
13. If they get you cornered, blame it all on OS jitter.
14. Secretly use fancy hardware setups and software tricks.
-- Gerhard Wellein, University of Erlangen, Germany
Source: http://blogs.fau.de/hager/category/fooling-the-masses/
22

Benchmarks help prevent abuse Benchmark tests must be well-designed, rigorous and scalable.
 Benchmark codes are problematic, because architectures andlanguages change, but “reference implementations” are useful.
 Well-thought-out and well-enforced “ground rules” are essential.
 A rational scheme must be provided for calculating performance
rates.
 Tests must be specified to validate the correctness of the results.
 A repository of results must be maintained.
Examples: The NAS Parallel Benchmarks (still useful, but a bit dated).
 The LBNL-UCB “Torch” Reference Kernels. See:
http://crd.lbl.gov/~dhbailey/dhbpapers/dwarfs09_intro.pdf

23

General guidelines to prevent abuse Direct comparisons of run times on real applications are preferred.
 If results are presented for a well-known benchmark, established ground rules
must be followed.
 Only actual performance results should be presented, not projections or
extrapolations (unless very clearly disclosed and justified).
 Performance figures should be based on comparable levels of tuning.
 Mflop/s, Gflop/s, Tflop/s rates should be computed from operation counts basedon the best practical serial algorithms.
 When computing parallel speedup figures, the denominator rate should be basedon an efficient single-processor implementation.
 Any ancillary information that would affect the interpretation of the results should
be fully disclosed (e.g., the use of 32-bit instead of 64-bit data, etc.).
 Special care should be taken for figures and graphs.
 Whenever possible, full background information should be provided: algorithms,
hardware and software configuration, language, compiler flags, tuning, timing
method, basis for operation counts, etc.
This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-12ways-2013.pdf
24