Quantum computer performance:
Standards now or scandals later?
David H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and Univ. of California, DavisThis talk: http://www.davidhbailey.com/dhbtalks/dhb-dwave-2019.pdf

1 / 30Reproducibility crises in psychology and economics
I In August 2015, the Reproducibility Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In November 2018, a separate consortium of

researchers was above to reproduce only 15 of 28
psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.

Reproducibility Project staff
Credit: NY Times

1. A. P. Taylor, “Half the time, psychology results not reproducible,” The Scientist, 20 Nov 2018,https://www.the-scientist.com/news-opinion/
half-the-time--psychology-results-not-reproducible--study-65117.
2. A. C. Chang and P. Li, “Is economics research replicable? Sixty published papers from thirteen
journals Say ‘Usually Not’,” U.S. Federal Reserve, Washington, 2015,
https://www.federalreserve.gov/econresdata/feds/2015/files/2015083pap.pdf.
2 / 30Reproducibility crises in the social sciences, cont.
The “blank slate” paradigm (1920–1990):
I The human mind at birth is a tabula rasa (“blank slate”).
I Heredity and biology play no significant role in human psychology; all personality
and behavioral traits are socially constructed.
I Pre-modern societies were peaceful, devoid of psychological and social problems.
Current consensus, based on latest research in behavioral genetics and anthropology:
I Humans at birth possess sophisticated facilities for social interaction, language
acquisition, pattern recognition, navigation and counting.
I Heredity, evolution and biology are major factors in human personality.
I Some personality traits are more than 50% heritable.
I Pre-modern societies had more crime, war and social problems than today.
How did the 20th century social scientists get it so wrong?
I Sloppy experimental methodology, with little concern for reproducibility.
I Ignoring or dismissing data that runs counter to predisposition.
3. S. Pinker, The Blank Slate: The Modern Denial of Human Nature, Penguin Books, 2003.
3 / 30Reproducibility crises in biomedicine
The biomedical field has been stung by numerous cases where pharma products lookgood based on clinical trials, but later disappoint in real-world usage, or the resultscannot be reproduced in separate studies. Examples:
I In 2004, GlaxoSmithKline acknowledged that while some trials of Paxil found it

effective for depression in children, other unpublished studies showed no benefit.
I In 2011, Bayer researchers reported that they were able to reproduce the results of

only 17 of 67 published studies they examined.
I In 2012, Amgen researchers reported that they were able to reproduce the resultsof only 6 of 53 published cancer studies.
I In 2014, a review of Tamiflu found that while it made flu symptoms disappear a

bit sooner, it did not stop serious complications or keep people out of the hospital.
Only publicizing the results of successful trials introduces a bias into the results.
The AllTrials movement requires all results to be public: http://www.alltrials.net.
4 / 30Reproducibility crises in biomedicine (references for previous page)
4. S. Foley, “GlaxoSmithKline pays $3bn for illegally marketing depression drug,”
U. K. Independent, 3 Jul 2012, http://www.independent.co.uk/news/business/news/
glaxosmithkline-pays-3bn-for-illegally-marketing-depression-drug-7904555.html.
5. E. S. Reich, “Cancer trial errors revealed,” Nature, 11 Jan 2011,
http://www.nature.com/news/2011/110111/full/469139a.html.
6. F. Prinz, T. Schlange and K. Asadullah, “Believe it or not: How much can we rely on publisheddata on potential drug targets?,” Nature Reviews Drug Discovery, vol. 10 (Sep 2011), pg. 712,
https://www.nature.com/articles/nrd3439-c1.
7. C. G. Begley and L. M. Ellis, “Drug development: Raise standards for preclinical cancer research,”
Nature, vol. 483 (29 Mar 2012), pg. 531-533, https://www.nature.com/articles/483531a.
8. B. Goldacre, “What the Tamiflu saga tells us about drug trials and big pharma,” U. K. Guardian,
9 Apr 2014, http://www.theguardian.com/business/2014/apr/10/tamiflu-saga-drug-trials-big-pharma.

5 / 30Reproducibility crises in physics
In March 2014, the BICEP2 team announced that they had
discovered a “twisting” pattern in cosmic microwave
background data, which fits the most common hypothesizedmodel of the inflation era just after big bang.
Press reports trumpeted this result as the first experimental
evidence of the inflationary big bang.
But other researchers had difficulty reconstructing theclaimed results. Finally, two teams challenged the BICEP2
findings, saying that the results could more readily be
explained by dust in the Milky Way.
Now the consensus is that the detection was false.
9. Ron Cowen, “Doubt grows about gravitational waves detection,” Scientific American, 2 Jun
2014, https://www.scientificamerican.com/article/
doubt-grows-about-gravitational-waves-detection/.

6 / 30Reproducibility crises in finance
Finance has been stung with many instances of investment strategies that look greaton paper, but fall flat in practice. A primary cause is backtest overfitting – statistical
overfitting of historical market data.
When a computer can analyze thousands or millions of variations of a given strategy orfund design, it is almost certain that the best such strategy, measured by backtests,
will be overfit and thus of dubious value.
In two 2014 papers by myself and colleagues, we show that a broad range of financial
strategies and fund designs are compromised by backtest overfitting.
10. D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics andfinancial charlatanism: The effects of backtest over fitting on out-of-sample performance,”
Notices of the American Mathematical Society, May 2014, pg. 458–471,
http://ssrn.com/abstract=2308659.
11. D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability of backtest
overfitting,” 12 Feb 2014, http://ssrn.com/abstract=2326253.
7 / 30Email exchange between DHB and a finance colleague
Email from DHB to finance colleague, 10 June 2013:
One thing that has always puzzled me about the financial world is the following sort of thing:[press examples cited]. Excuse me for being “dumb,” but this sort of thing seems to me to beoutright nonsense. ... When people like those above say that they “know” where the stock
market is heading, this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?

Response from finance colleague to DHB, 17 June 2013:
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. ...
I completely agree with your assessment. The amount of nonsense ... is incredible.
8 / 30Financial news commentary
Excerpt from a recent article on a widely read financial news site:
As I highlighted during the past week, the bottom we struck this past week
seems to best count as a third wave bottom in the (c) wave of the a-wave.
That still has me looking for a lower low yet to come. Moreover, there were
no divergences on any of the technicals present on the 60-minute chart when
we struck that bottom. The great majority of the time, that suggests that
only the third wave of the (c) wave has completed. For this reason, I waslooking for this bounce to be a fourth wave, which we began to anticipate
when we identified the bottoming in the market in our chat room on Tuesdaynight.

9 / 30Why the silence in the mathematical finance community?
Historically scientists have led the way in exposing those who utilize pseudoscience toextract a commercial benefit.
Yet financial mathematicians in the 21st century have remained disappointingly silent
with the regards to those in the community who, knowingly or not:1. Fail to disclose the number of models or variations that were used to develop an
investment strategy or fund.
2. Make vague predictions that do not permit rigorous testing and falsification.
3. Misuse probability theory, statistics and stochastic calculus.
4. Suggest in countless press reports and promotions that investors can achieve
above-market returns via unsophisticated products and chart-watching strategies.
5. Use dubious technical jargon: “stochastic oscillators,” “Fibonacci ratios,”“cycles,” “Elliott waves,” “golden ratio,” “parabolic SAR,” “pivot point,”“momentum,” etc.
Message to community: Our silence is consent, making us accomplices in these abuses.
10 / 30

Increasing performance of the top 500 supercomputers (1994 – present)

Orange = #1
Blue = #500
Green = sum #1 thru #500
Courtesy: Top500.org

11 / 30DANGER AHEAD
Supercomputers can generate nonsense faster than ever before!

Are computational results reproducible?
I Are the algorithms, data sources and processing methods well documented?
I Are all source code files, make files and data files available in a secure public

repository?
I Are the results statistically sound?
I Are the results numerically reliable?
I Have the results been validated using separate tests?
I Have the results been validated by independent researchers?

12 / 30

Reproducibility in scientific computing
A December 2012 workshop on reproducibility in computing, held at Brown University
in Rhode Island, U.S.A., found that
Science is built upon the foundations of theory and experiment validated andimproved through open, transparent communication. With the increasingly
central role of computation in scientific discovery, this means communicating
all details of the computations needed for others to replicate the experiment.
...
The “reproducible research” movement recognizes that traditional scientificresearch and publication practices now fall short of this ideal, and encourages
all those involved in the production of computational science ... to facilitate
and practice really reproducible research.

12. V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting thedefault to reproducible: Reproducibility in computational and experimental mathematics,” Jan2013, http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.
13 / 30

Reproducibility in scientific computing, continued
Issues identified in the ICERM report and other studies include:
I Researchers must carefully document the full context of computational

experiments—system environment, input data, code used, computed results, etc.
I Researchers must save the code and output data in a permanent repository.
I Reviewers, research institutions and funding agencies need to recognize theimportance of computing and computing professionals, and to allocate funding forafter-the-grant support and repositories.
I Researchers need to be more careful with numerical reproducibility.
I The community must encourage the publication of negative results — otherresearchers can often learn from them.
I The community must ensure the responsible reporting of performance.

14 / 30

Numerical reproducibility in high-performance computing
The report mentioned above on reproducibility in high-performance computing noted:
Numerical round-off error and numerical differences are greatly magnified ascomputational simulations are scaled up to run on highly parallel systems. As
a result, it is increasingly difficult to determine whether a code has beencorrectly ported to a new system, because computational results quickly
diverge from standard benchmark cases. And it is doubly difficult for otherresearchers, using independently written codes and distinct computer
systems, to reproduce published results.

15 / 30

Analysis of collisions at the Large Hadron Collider
I The 2012 discovery of the Higgs boson at the ATLAS experiment in the LHC

relied crucially on the ability to track charged particles with exquisite precision (10microns over a 10m length) and high reliability (over 99% of roughly 1000
charged particles per collision correctly identified).
I Software: 5 millions line of C++ and python code, developed by roughly 2000

physicists and engineers over 15 years.
I Recently, in an attempt to speed up the calculation, researchers found that merely

changing the underlying math library resulted in some collisions being missed or
misidentified.
Questions:
I How serious are these numerical difficulties?
I How can they be tracked down?
I How can the library be maintained, producing numerically reliable results?
16 / 30

Are performance reports reproducible?
I Are the algorithms, data sources, and complete system configuration fully

documented?
I Are all source code files, make files and data files available in a secure public

repository?
I Is the exact benchmark being tested fully defined?
I Is the performance comparison a fair, apples-to-apples comparison, withcomparable levels of precision, accuracy, tuning and analysis?
I Are the performance results statistically sound (e.g., are only the most favorable
results reported)?
I Have the performance results been validated using separate tests?
I Have the performance results been validated by independent researchers?
The same standards apply for performance results as for other scientific results.

17 / 30

Reproducibility crisis in HPC performance: 1990-1994
Background:
I Many new parallel systems had been introduced; each claimed theirs was best.
I Many researchers were excited about the potential of highly parallel systems.
I Few standard benchmarks and testing methodologies had been established.
I It was hard to reproduce published performance results; much confusion reigned.
I Overall, the level of rigor and peer review in the field was rather low.

In response, DHB published a humorous essay “Twelve ways to fool the masses whengiving performance results on parallel computers,” poking fun at some of the abuses.
Since abuses continued, DHB presented a talk at Supercomputing 1992 and publisheda paper with specific examples culled from peer-reviewed papers.

18 / 30

1991 paper: “Twelve ways to fool the masses”
1. Quote 32-bit performance results, not 64-bit results, but don’t mention this in paper.
2. Present performance figures for an inner kernel, then represent these figures as theperformance of the entire application.
3. Quietly employ assembly code and other low-level language constructs.
4. Scale up the problem size with the number of processors, but omit any mention of this.
5. Quote performance results projected to a full system.
6. Compare your results against scalar, unoptimized code on conventional systems.
7. When run times are compared, compare with an old code on an obsolete system.
8. Base Mflop/s rates on the operation count of the parallel implementation, instead of thebest practical serial algorithm.
9. Quote performance as processor utilization, parallel speedups or Mflop/s per dollar.
10. Mutilate the algorithm used in the parallel implementation to match the architecture.
11. Measure parallel run times on a dedicated system, but measure conventional run times ina busy environment.
12. If all else fails, show pretty pictures and animated videos, and don’t discuss performance.

19 / 30

NY Times (22 Sep 1991): “Measuring How Fast Computers Really Are”
Excerpts:
I “Rival supercomputer and work

station manufacturers are prone tohype, choosing the performance
figures that make their own machines
look best.”
I “It’s like the Wild West.” [quotingDavid J. Kuck of UIUC].
I “It’s not really to the point ofwidespread fraud, but if people aren’t
a little more circumspect, the entire
field could start to get a bad name.”[quoting DHB].
20 / 30

Example 1: Scaling performance results to full-sized system
In some published papers and conference presentations, performance results on
small-sized parallel systems were linearly scaled to full-sized systems, often withouteven clearly disclosing this fact.
Example: 8,192-CPU performance results were linearly scaled to 65,536-CPU results,
simply by multiplying by eight.
Excuse: “We can’t afford a full-sized system.”This and the other examples mentioned in the next few viewgraphs are presented in:
13. D. H. Bailey, “Misleading performance reporting in the supercomputing field,” Scientific
Programming, vol. 1., no. 2 (Winter 1992), pg. 141–151,
https://www.davidhbailey.com/dhbpapers/mislead.pdf.
21 / 30

Example 2: Using inefficient algorithms on highly parallel systems
In many cases, inefficient algorithms were employed for the highly parallel
implementation, requiring many more operations, thus producing artificially highMflop/s rates:
I Numerous researchers cited parallel PDE performance based on explicit schemes,

where implicit schemes were known to be much more efficient.
Excuse: Explicit schemes “run better” on the researchers’ parallel system.
I One paper cited performance for computing a 3D discrete Fourier transform bydirect evaluation of the defining formula (8n2 operations), rather than by using afast Fourier transform (5n log2 n).
Excuse: Direct computation of FFT was “more appropriate” for the architecture
being analyzed.
Both examples violate a rule of professional performance reporting, namely to base theoperation count (when computing Mflop/s or Gflop/s rates) on the best practical serial
algorithm, no matter what scheme was actually used on the parallel system.
22 / 30

Example 3: Not actually performing a computation on the claimed system
Abstract of published paper: “The current Connection Machine implementation runs
at 300-800 Mflop/s on a full [64K] CM-2, or at the speed of a single processor of a
Cray-2 on 1/4 of a CM-2.”
I Excerpt from text: “This computation requires 568 iterations (taking 272

seconds) on a 16K Connection Machine.”
In other words, the computation was run on a 16K system, not on a 64K system;
the figures cited in the Abstract were merely multiplied by four.
I Excerpt from text: “In contrast, a Convex C210 requires 909 seconds to compute

this example. Experience indicates that for a wide range of problems, a C210 isabout 1/4 the speed of a single processor Cray-2.”
In other words, the computation mentioned in the Abstract was not actually run
on a Cray-2; instead, it was run on a Convex system, and a questionable
rule-of-thumb scaling factor was used to produce the Cray-2 rate.
23 / 30

Example 4: Performance plot — parallel (lower) vs vector (upper)
3.5

3Time (Hours)

2.5

2

1.5

1

0.5

0

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

Number of Objects
24 / 30

Data for performance plot
Problem size(x axis)
20
40
80
160
990
9600

Parallel system
run time
8:18
9:11
11:59
15:07
21:32
31:36

Vector system
run time
0:16
0:26
0:57
2:11
19:00
3:11:50*

Details in text of paper:
I In last entry, the 3:11:50 figure is an “estimate.”
I The vector system code is “not optimized.”
Note that the parallel system is actually slower than the vector system for all cases,
except for the last (estimated) entry. Except for the last entry, all real data in thegraph is in the lower left corner (i.e., a log-log plot should have been used instead).
Also, it is not fair to compared tuned vs. untuned performance.
25 / 30

Graveyard of failed HPC firms in 1990s
I Thinking Machines, Inc.: Founded 1983; popular in early 1990s; bankrupt 1994.
I Intel: Marketed Paragon system in late 1980s and early 1990s; exited 1994.
I Kendall Square Research: Founded 1986; bankrupt 1994.
I Convex Computers: Founded 1982; popular in early 1990s; sold to Hewlett-Packard in 1995.
I Cray Computer Corp.: Founded 1989 by Seymour Cray; bankrupt 1995.
I Cray Research, Inc.: Founded in 1970s by Seymour Cray; industry leader in 1970s

and 1980s; declined in early 1990s; sold to SGI in 1995; then sold to Tera
Computers in 2000, which changed name back to Cray; still in HPC market today.

26 / 30

Fast forward to 2019: Fooling the masses with multicore/GPU systems
I Cite performance rates for a run with only one processor core active in a shared-

memory multi-core node, producing artificially inflated performance (since there is
no shared memory interference) and wasting resources (since most cores are idle).
I

Example: Cite performance on “1024 cores,” even though the code was run on 1024
multicore nodes, one core per node, with 15 out of 16 cores idle on each node.

I Claim that since one is using a graphics processing unit (GPU) system, that

efficient parallel algorithms must be discarded in favor of “more basic algorithms.”
I Cite performance rates only for a core algorithm (such as FFT or linear systemsolution), even though full-scale applications have been run on the system.
I List only the best performance figure in the paper, omitting numerous lessfavorable results (recall the experience of pharma tests).
I Employ special hardware, operating system or compiler settings that are notappropriate for real-world production usage.
I Redefine “scalability” as successful execution on a large number of CPUs,
regardless of performance.
27 / 30

Fooling the masses with quantum computing
A 2014 study (14) discussed comparisons between quantum computers and classicalcomputers. However,I Runtimes reported in Fig. 3 for the classical solvers SA and SQA are in factmeasured runtimes divided by the problem size N.
I Thus the reported runtimes at the rightmost points of Fig. 3 are 512 times lower
than actual measured CPU times.
I The analysis of the classical algorithms involved an extensive search of theparameter space to find the best combination of parameters, and then the results
for the best cherry-picked combination were reported as measured performance ateach problem size.
I The time to find the best parameter combination was not included in total
runtimes.
The standards and expectations of the two communities differ. But why can’t we agree
on a single standard?
14. T. F. Ronnow, Z. Wang, J. Job and six others, “Defining and detecting quantum speedup,”
arXiv, 13 Jan 2014, https://arxiv.org/pdf/1401.2910.pdf.

28 / 30

Fooling the masses with quantum computing, cont.
A 2018 study (15) compared the Coherent Ising Machine (CIM) with a D-Wave
2000Q, asserting that there is an “exponential penalty” for the D-Wave system on
dense graph Ising problems. However (16),I The exponential performance gap is based on two regression models extrapolating
the algorithm scaling on the systems.
I The resulting extrapolations overestimate the scaling of the 2000Q processor andunderestimate the scaling of the CIM.
I If actual measured data are used, the CIM is approximately 10X and 8000X fasteron the SK and MC problems.
I In a stricter apples-to-apples comparison, the CIM advantage drops to 5X on SK
and 364X on MC.
15. R. Hamerly, T. Inagaki and 20 others, “Scaling advantages of all-to-all connectivity in physicalannealers: the Coherent Ising Machine vs. D-Wave 2000Q,” arXiv, 14 May 2018,https://authors.library.caltech.edu/86965/1/1805.05217.pdf.
16. C. C. McGeoch, W. Bernoudy and J. King, “Comment on ‘Scaling advantages ... 2000Q’,”
arXiv, 3 Jul 2018, https://arxiv.org/pdf/1807.00089.pdf.

29 / 30

Standards now or scandals later?
Now is the time to establish performance standards for the quantum computing field:
I Benchmarks: The community carefully selects a set of kernels as well as full
applications, representative of real-world, state-of-the-art usage.
I Thorough documentation: Test reports include algorithms, software, compilers,
system environment, precision used, accuracy achieved and details of performance
calculations.
I Full disclosure: Test reports clearly state any information that may affect theinterpretation of the performance results.
I Fair comparisons: Tests reflect comparable tuning, comparable precision,comparable accuracy and comparable system environments.
I Public availability: All relevant files and input data are saved on a publicly
available, persistent data repository.
An IEEE-sponsored group has begun work on benchmarking standards and metrics forquantum computing. All players should support this activity.
This talk is available at: http://www.davidhbailey.com/dhbtalks/dhb-dwave-2019.pdf.

30 / 30