Fooling the masses in 2017:
Why the scientific computing community must still be vigilant
David H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and Univ. of California, Davis

May 12, 2017

1 / 20DANGER AHEAD
Supercomputers operating on big data cangenerate nonsense faster than ever before!

Key concerns:
I Are the results statistically sound?
I Are the results numerically reliable?
I Have the results been validated using independent rigorous tests?
I Are the algorithms, data sources and processing methods well documented?
I Are performance results accurately and honestly reported?

2 / 20Reproducibility crises in physics and cosmology
I In 2011, an international team of researchers at the Gran Sasso Laboratory in Italy

announced that neutrinos had exceeded the speed of light, thus directly
challenging Einstein’s relativity. However, after months of careful checking, asubtle flaw was found in the measurement apparatus (a major embarrassment).
I In 2013, CERN researchers confirmed the discovery of the long-sought Higgs

boson. But more recently, scientists have raised questions as to whether theparticle discovered is really the Higgs – it might be some other particle or particles
masquerading as the Higgs; additional research studies are required.
I In March 2014, researchers announced with considerable fanfare that they had

detected the fingerprint of the long-hypothesized inflationary epoch, a tiny
fraction after the big bang. Sadly, within a few weeks critics pointed out that
their experimental results might well be due to dust in the Milky Way, pending
better data.
3 / 20Reproducibility crises in biomedicine, psychology, economics and finance
I In 2011, Bayer researchers reported that they wereable to reproduce only 17 of 67 pharma studies.
I In 2012, Amgen researchers reported that they wereable to reproduce only 6 of 53 cancer studies.
I In August 2015, the Reproduciblity Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.
I In 2014-2015, “backtest overfitting” emerged as a

Reproducibility Project staff
Credit: NY Times

major problem in computational finance.
These experiences have exposed at least one common flaw:
Only publicizing the results of successful trials introduces a bias into the results.
4 / 20Reproducibility in scientific computing
A December 2012 workshop on reproducibility in computing, held at Brown University
in Rhode Island, USA, noted that
Science is built upon the foundations of theory and experiment validated andimproved through open, transparent communication. With the increasingly
central role of computation in scientific discovery this means communicating
all details of the computations needed for others to replicate the experiment.
... The “reproducible research” movement recognizes that traditional
scientific research and publication practices now fall short of this ideal, andencourages all those involved in the production of computational science ...
to facilitate and practice really reproducible research.
I V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting thedefault to reproducible: Reproducibility in computational and experimental mathematics,”
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

5 / 20Documentation: A glaring deficiency in the scientific computing field
ICERM Report: To aid in reproducibility, computing papers should document:
I A precise statement of assertions to be made in the paper.
I A statement of the computational approach, and why it constitutes a test of hypothesis.
I Complete statements of (or references to) every algorithm employed.
I Details of auxiliary software (both research and commercial software) used.
I Details of the test environment, including hardware, system software and the number ofprocessors utilized.
I Details of data reduction and statistical analysis methods.
I Discussion of the adequacy of parameters such as precision level and grid resolution.
I Full statement of experimental results.
I Verification and validation tests performed by the author(s).
I Availability of computer code, input data and output data.
I Instructions for repeating computational experiments described in the paper.
6 / 20Parallel computing, 1991
I Many new parallel systems were offered; users

were excited about potential.
I Each vendor claimed theirs was best, citing one

or two selected applications.
I There were few standard benchmarks or testing

methodologies — mostly Livermore Loops andoriginal Linpack-100.
I Overall, the level of rigor and peer review inthe field was disappointingly low.
I In 1991 DHB published a humorous essay

Twelve Ways to Fool the Masses, poking fun
at some of the abuses (authored by DHB, butwith contributions from the NPB team).
7 / 20Twelve ways to fool the masses (paraphrased)
1. Quote 32-bit performance results, not 64-bit results, but don’t mention this in paper.
2. Present performance figures for an inner kernel, then represent these figures as theperformance of the entire application.
3. Quietly employ assembly code and other low-level language constructs.
4. Scale up the problem size with the number of processors, but omit any mention of this.
5. Quote performance results projected to a full system.
6. Compare your results against scalar, unoptimized code on conventional systems.
7. When run times are compared, compare with an old code on an obsolete system.
8. Base Mflop/s rates on the operation count of the parallel implementation, instead of thebest practical serial algorithm.
9. Quote performance as processor utilization, parallel speedups or Mflop/s per dollar.
10. Mutilate the algorithm used in the parallel implementation to match the architecture.
11. Measure parallel run times on a dedicated system, but measure conventional run times ina busy environment.
12. If all else fails, show pretty pictures and animated videos, and don’t discuss performance.

8 / 20NY Times (22 Sep 1991): “Measuring How Fast Computers Really Are”
Excerpts:
I “Rival supercomputer and work

station manufacturers are prone tohype, choosing the performance
figures that make their own machines
look best.”
I “It’s like the Wild West.” [quotingDavid J. Kuck of UIUC].
I “It’s not really to the point ofwidespread fraud, but if people aren’t
a little more circumspect, the entire
field could start to get a bad name.”[quoting DHB].
9 / 20

Examples of abuses: Scaling performance results to full-sized system
In some published papers and conference presentations, runs were performed on
smaller systems, then performance rates were linearly scaled to full-sized systems, insome cases without even clearly disclosing this fact.
Example: 8,192-CPU performance results were linearly scaled to 65,536-CPU results,
simply by multiplying by eight.
Typical excuse: “We can’t afford a full-sized system.”

I D. H. Bailey, “Misleading performance reporting in the supercomputing field,” ScientificProgramming, vol. 1., no. 2 (Winter 1992), 41–151.

10 / 20

Using inefficient algorithms on highly parallel systems
In many cases, inefficient algorithms were employed for highly parallel implementations,requiring many more operations, thus producing artificially high speedup figures andMflop/s rates. Examples:
I Some researchers cited parallel PDE performance based on explicit schemes, forproblems where implicit schemes were known to be significantly more efficient.
I One paper cited performance for computing a 3D discrete Fourier transform bydirect evaluation of the defining formula (8n2 operations), rather than by using afast Fourier transform (5n log2 n).
Both examples violate a basic rule of parallel computing, namely to base parallel
implementations and operation counts (when computing Mflop/s or Gflop/s rates) on
the best practical serial algorithm.
Typical excuse: Other algorithms are “more appropriate” for our parallel system.
11 / 20

Abstract versus details in paper
Abstract of published paper: “The current Connection Machine implementation runs
at 300-800 Mflop/s on a full [64K] CM-2, or at the speed of a single processor of aCray-2 on 1/4 of a CM-2.”
I Excerpt from text: “This computation requires 568 iterations (taking 272

seconds) on a 16K Connection Machine.”
In other words, the computation was run on a 16K system, not on a 64K system;
the figures cited in the abstract were merely multiplied by four.
I Excerpt from text: “In contrast, a Convex C210 requires 909 seconds to compute

this example. Experience indicates that for a wide range of problems, a C210 isabout 1/4 the speed of a single processor Cray-2.”
In other words, the computation mentioned in the abstract was actually run on aConvex system, and a rule-of-thumb scaling factor was used to produce theCray-2 rate.
12 / 20

Performance plot: parallel run times (lower) vs vector (upper)
3.5

3Time (Hours)

2.5

2

1.5

10.5

00

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Number of Objects
13 / 20

Data for performance plotProblem size(x axis)
20
40
80
160
990
9600

Parallel system
run time
8:18
9:1111:59
15:07
21:32
31:36

Vector system
run time
0:16
0:26
0:57
2:11
19:00
3:11:50*

Details in text of paper:
I In last entry, the 3:11:50 figure is an estimate.
I The vector system code is not optimized.

Note that the parallel system is actually slower than the vector system for all cases,
except for the last (estimated) entry. Also, except for the last entry, all real data in thegraph is in the lower left corner. A log-log plot should have been used instead.
14 / 20

Origins of the NAS Parallel Benchmarks (NPB)I In 1991, a team of 12 researchers at the Numerical Aerodynamic Simulation(NAS) facility (now known as the NASA Advanced Supercomputing facility)
formulated the “NAS Parallel Benchmarks.”
I The original plan was to be a basis for an upcoming supercomputer procurementfor NAS.
I A central goal of the NPB was to test algorithms of importance in computational
aeronautics.
I However, the NPB team recognized from the beginning that the benchmarks
should be designed to have wider usage — hopefully to help clear the hype andconfusion in the field.
I The resulting paper (below) was recently awarded the 2015 “Test of Time Award”
from the ACM / IEEE Supercomputing Conference.
I D. H. Bailey, E. Barszcz, J. T. Barton, D. S. Browning, R. L. Carter, L. Dagum, R. A. Fatoohi,

P. O. Frederickson, T. .A. Lasinski, R. S. Schreiber, H. D. Simon, V. Venkatakrishnan and
S. Weeratunga, “The NAS Parallel Benchmarks: Summary and preliminary results,” Proceedingsof Supercomputing 1991, ACM/IEEE Computer Society, 1991, 158–165.

15 / 20

New ways to fool the masses in high-performance computing
I Cite performance rates for a run with only one processor core active in a shared-memory multi-core node, producing artificially inflated performance (since there is
no shared memory interference) and wasting resources (since most cores are idle).
I

Example: Cite performance on “1024 cores,” even though the code was run on 1024
multicore nodes, one core per node, with 15 out of 16 cores idle on each node.

I Claim that since one is using a graphics processing unit (GPU) system, the mostefficient algorithms must be discarded in favor of “more appropriate” algorithms(recall the experience with parallel algorithms).
I Run the test code many times, but only include the best performance rate in thepaper (recall the experience of recent pharmaceutical trials).
I Employ special hardware, operating system or compiler settings that are notappropriate for real-world production usage (recall the recent Volkswagen scandal).

16 / 20

Numerical reproducibility
The ICERM reproducibility report further noted:
Numerical round-off error and numerical differences are greatly magnified ascomputational simulations are scaled up to run on highly parallel systems. As
a result, it is increasingly difficult to determine whether a code has beencorrectly ported to a new system, because computational results quickly
diverge from standard benchmark cases. And it is doubly difficult for otherresearchers, using independently written codes and distinct computersystems, to reproduce published results.
I V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting thedefault to reproducible: Reproducibility in computational and experimental mathematics,”
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

17 / 20

The growing problem of numerical reliability
Many applications routinely use either 32-bit or 64-bit IEEE arithmetic, and employ
fairly simple algorithms, assuming that all is well. But problems can arise.
Particularly vulnerable are:
1. Large-scale, highly parallel simulations, running on systems with hundreds ofthousands or millions of processors — numerical sensitivities are greatly magnified.
2. Certain applications with highly ill-conditioned linear systems.
3. Large summations, especially those involving cancellations.
4. Long-time, iterative simulations (such as molecular dynamics or climate models).
5. Computations to resolve small-scale phenomena.
6. Studies in computational physics or experimental mathematics often require hugeprecision levels.
I D. H. Bailey, R. Barrio and J. M. Borwein, “High precision computation: Mathematical physics

and dynamics,” Applied Mathematics and Computation, vol. 218 (2012), pg. 10106–10121.
18 / 20

Analysis of collisions at the Large Hadron Collider
I The 2012 discovery of the Higgs boson at the ATLAS experiment in the LHC

relied crucially on the ability to track charged particles with exquisite precision (10microns over a 10m length) and high reliability (over 99% of roughly 1000
charged particles per collision correctly identified).
I Software: 5 millions line of C++ and python code, developed by roughly 2000

physicists and engineers over 15 years.
I Recently, in an attempt to speed up the calculation, researchers found that merely

changing the underlying math library resulted in some collisions being missed ormisidentified.
Questions:
I How serious are these numerical difficulties?
I How can they be tracked down?
I How can the library be maintained, producing numerically reliable results?
19 / 20Have we learned anything?
Progress has been made, but a renewed focus is needed on:
I Supporting rigorous peer review.
I Focusing on end-to-end application run time.
I Reporting full details of computational environment, so results can be reproduced.
I Reporting all results — not selecting the best run out of many.
I Basing operation counts (for performance reporting) on the best practical serial
algorithm.
I Reporting performance in a realistic operating environment, with no special
settings inappropriate for common production usage.
I Being careful with numerical reproducibility. With the exploding size and scope ofcomputations, numerical round-off error is much more severe. Have the authors
fully justified the numerical reliability of their results?
THANKS!
This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-ipdps-2017.pdf

20 / 20