Present-day petascale supercomputers
The memory capacity of supercomputers is measured in “Pbytes” or “petabytes”(namely one quadrillion bytes), and their performance is measured in “Pflop/s” or“petaflops” (namely one quadrillion 64-bit floating-point operations per second).
It is amusing to consider for a moment how big a quadrillion is:
I The volume of Sydney Harbor is roughly 5 × 1011 litres, or 1/10,000 of one

quadrillion litres.
I The volume of Lake Tahoe, California is roughly 1.51 × 1014 litres, or 1/7 of one

quadrillion litres.
I The distance to Alpha Centauri is roughly 4.15 × 1013 kilometres, or 41.5

quadrillion meters.
I The number of minutes since the big bang is roughly 7.25 × 1015 minutes, i.e.,

about 7 quadrillion minutes.

Increasing performance of the top 500 supercomputers (1994 – present)

Red = #1
Orange P= #500Blue =#1 thru #500

I “Performance development,” Top500.org, available at http://top500.org/statistics/perfdevel.

Declining cost of data storage (1955 – 2012)

I John C. McCallum, “Disk drive storage price decreasing with time (1955-2012),” available at
http://www.jcmit.com/disk2012.htm.

Systems at the U.S. National Energy Research Scientific Computing(NERSC) Center at the Lawrence Berkeley National Laboratory
Current system (“Edison”):

To be installed in 2016 (“Cori”):

I 5576 Intel “Ivy Bridge” nodes.

I 9300 Intel “Knight’s Landing” nodes.

I Memory per node: 64 Gbyte.

I Main memory per node: 64 Gbyte.

I Performance per node: 460.8 Gflop/s.

I Performance per node: 3 Tflop/s.

I Total main memory: 357 Tbyte.

I Total main memory: 595 Tbyte.

I Total performance: 2.57 Pflop/s.

I Total peak performance: 28 Pflop/s.

I Total disk storage: 7.56 Pbyte.

I Total disk capacity: 28 Pbyte.

Future archival storage requirements at NERSC

Big data applications: The accelerating universe
In 1998, two teams of astronomers (one led
by Saul Perlmutter of LBNL, and the otherled by Brian P. Schmidt of ANU), came tothe paradoxical conclusion that the
expansion of the universe is accelerating,not slowing down.
Both teams based their results on carefulmeasurements of distant supernovas, which
in turn were found by sifting through reams
of digital telescope data. The U.S. team inparticular relied heavily on large computer
systems at the NERSC facility, coupledwith a worldwide network of collaborating
astronomers.

Big data applications: Discovery of supernova in the Pinwheel Galaxy
In 2011, Peter Nugent of LBNL, working
within a worldwide consortium of fellow
astronomers, discovered a Type Ia
supernova in the Pinwheel Galaxy, which is“only” 21 million light-years from earth. It
is the closest and brightest supernova of
this type seen in the last 30 years.
Nugent and his team utilized the Palomar
Transient Facility (PTF), a
robot-controlled telescope that produces
digital images, plus a worldwide network ofdata processing and storage facilities.

The Palomar Transient Facility (PDF) data pipeline

Big data applications: The Materials Project
Researchers at LBNL and Harvard
University started the “Materials Project”:
I Reduce lag time between materials

science advances and real-worldcommercialization.
I Invert the conventional materials

science paradigm: Ask “Whatproperties do I want?,” then ”Whichmaterials have them?”
I Method: Perform ab initio calculationsof tens of thousands of potentiallyinteresting compounds, then save insearchable database.
I Now in operation:

http://www.materialsproject.org.

Today’s Li ion
batteries
Interestingmaterials…

!

Voltage vs. capacity for over 20,000 Li-ion
cathode compounds using high-throughput ab
initio methods.

Implications of big data computing

1. Scientific computing is in the midst of a paradigm shift, from data-poor todata-rich computing.
2. Nowadays most data being processed on supercomputers is experimental, notsimulation data.
3. Advanced visualization facilities are no longer optional — they are absolutely
essential. A significant portion of a researcher’s time is now spent analyzing andvisualizing data.
4. Sophisticated statistical machine learning techniques are now being applied toclassify data and to focus on “interesting” items in the data.

Machine learning for supernova research
Recently machine learning techniques were applied to help automate the supernova
detection process at the PTF. Challenges include:
I Raw data must be processed in real time. Future, higher-resolution telescopes will

exacerbate this problem.
I Observations must be classified as supernovas with very high reliability; false

negatives or false positives both cause problems.
Preliminary results:
I Using a statistical rule ensemble method, we were above to find reliableclassifications using only 21 of the 39 attributes now being used.
I Work is currently underway to develop a new preprocessing method that extracts

even better features, and to decrease the false negative rate.

DANGER AHEAD
In spite of the successes of big data computing, danger lies ahead:
Supercomputers operating on big data can generate nonsense faster than ever before!
Key concerns:
I Are the algorithms, data sources and processing methods well documented?
I Are the results reproducible by other researchers, or even by the same team ofresearchers?
I Are the results statistically sound?
I Are the results numerically reliable?
I Have the results been validated using tests designed by the researchers or others?

Reproducibility in scientific computing
A December 2012 workshop on reproducibility in computing, held at Brown University
in Rhode Island, USA, noted that
Science is built upon the foundations of theory and experiment validated andimproved through open, transparent communication. With the increasingly
central role of computation in scientific discovery, this means communicating
all details of the computations needed for others to replicate the experiment.
... The “reproducible research” movement recognizes that traditional
scientific research and publication practices now fall short of this ideal, andencourages all those involved in the production of computational science ...
to facilitate and practice really reproducible research.

I V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting the default toreproducible: Reproducibility in computational and experimental mathematics,”
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

Reproducibility in scientific computing, continued
Issues identified in the ICERM report and other studies include:
I The need to carefully document the full context of computational

experiments—system environment, input data, code used, computed results, etc.
I The need to save the code and output data in a permanent repository.
I The need for reviewers, research institutions and funding agencies to recognize theimportance of computing and computing professionals, and to allocate funding forafter-the-grant support and repositories.
I The increasing importance of numerical reproducibility, and the need for tools toensure and enhance numerical reliability.
I The need to encourage publication of negative results—other researchers canoften learn from them.
I The re-emergence of the need to ensure responsible reporting of performance.

Reproducibility in biomedicine
The biomedical field has been stung by numerous cases where pharma products lookgood based on clinical trials, but later disappoint in real-world usage, or the resultscannot be reproduced in separate studies. Examples:
I In 2004, GlaxoSmithKline acknowledged that while some trials of Paxil found it

effective for depression in children, other unpublished studies showed no benefit.
I In 2011, Bayer researchers reported that they were able to reproduce the results of

only 17 of 67 published studies they examined.
I In 2012, Amgen researchers reported that they were able to reproduce the resultsof only 6 of 53 published cancer studies.
I In 2014, a review of Tamiflu found that while it made flu symptoms disappear a

bit sooner, it did not stop serious complications or keep people out of the hospital.
These experiences have exposed a fundamental flaw in methodology:
Only publicizing the results of successful trials introduces a bias into the results.
The AllTrials movement would require all results to be public: http://www.alltrials.net

Reproducibility in physics
In March 2014, a team of researchers from Harvard
University made the dramatic announcement thatthey had discovered an interesting “twisting”
pattern in cosmic microwave background data,measured using their BICEP2 experimental system.
This pattern fit very well with the hypothesizedpattern of the most commonly assumed model of“inflation” in the first tiny fraction of a second after
the big bang, and thus has been trumpeted as thefirst experimental evidence of the inflationary
cosmology.
But other researchers had difficulty reconstructing
the claimed results. Finally, two teams challenged
the BICEP2 findings, saying that the results could
more readily be explained by dust in the Milky Way.

I Ron Cowen, “Doubt grows about
gravitational waves detection,”
Scientific American, 2 Jun 2014.

Reproducibility in social science
The “blank slate” paradigm (1920–1990):
I The human mind at birth is a tabula rasa (“blank slate”).
I Heredity and biology play no significant role in human psychology; all personality
and behavioral traits are socially constructed.
I Pre-modern societies were peaceful, devoid of psychological and social problems.
Current consensus, based on latest research:
I Humans at birth possess sophisticated facilities for social interaction, language
acquisition, pattern recognition, navigation and counting.
I Heredity, evolution and biology are major factors in human personality.
I Some personality traits are more than 50% heritable.
I Pre-modern societies had more crime, war and social problems than today.
How did the 20th century social scientists get it so wrong?
I Sloppy experimental methodology and analysis.
I Pervasive wishful thinking and politically correct biases.
I Ignoring or dismissing data that runs counter to predisposition.
I S. Pinker, The Blank Slate: The Modern Denial of Human Nature, Penguin Books, 2003.

Reproducibility in finance
Finance, like the pharmaceutical world, has been stung with numerous instances ofinvestment strategies that look great on paper, but fall flat in practice. A primarycause is statistical overfitting of backtest (historical market) data.
When a computer can analyze thousands or millions of variations of a given strategy, it
is almost certain that the best such strategy, measured by backtests, will be overfit andthus of dubious value.
In two 2014 papers by DHB, JMB, Marcos Lopez de Prado and Jim Zhu, we derive (a)a formula relating the number of trials to the minimum backtest length, and (b) aformula for the probability of backtest overfitting. We also show that under theassumption of memory in markets, overfit strategies are actually prone to lose money.
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest over fitting on out-of-sample performance,” Notices of theAmerican Mathematical Society, May 2014, pg. 458–471.
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability of backtest overfitting,”
manuscript, 12 Feb 2014, http://ssrn.com/abstract=2326253.

How easy is it to overfit a backtest? Answer: Very easy!
I If only 2 years of daily backtest data are available, then no more than 7 strategy

variations should be tried.
I If only 5 years of daily backtest data are available, then no more than 45 strategy

variations should be tried.
I A backtest that does not report the number of trials N makes it impossible toassess the risk of overfitting.
I Given any desired performance level, a financial researcher just needs to keep

trying alternative parameters for that strategy!

MinBTL ≈




 !2
(1 − γ)Z −1 1 − N1 + γZ −1 1 − N1 e −1
E [maxN ]An absurd investment strategy
I A financial advisor sends letters to 10, 240 = 10 × 210 potential clients, with 5120

letters predicting a certain security will go up, and the other half predicting it will
go down.
I One month later, the advisor sends letters only to the 5120 investors who werepreviously sent the correct prediction, with 2560 letters predicting a certain
security will go up, and the other half predicting it will go down.
I The advisor continues this process for 10 months.
I The remaining ten investors, so impressed by the advisor’s ten consecutive spot-on

predictions, will entrust to him/her all of their assets!
This strategy is absurd; even fraudulent.
But why is marketing a statistically overfit strategy, where potential investors are notinformed of the number of trials behind the strategy, any different?

Optimizing an investment strategy to fit pseudorandom time series
I The following 23 viewgraphs present the results of different steps in an attempt tofind an “optimal” investment strategy, based on a fixed market price dataset.
I The underlying dataset was generated by a pseudorandom number generator!
I As you can see, by tweaking some very basic parameters (entry price, sell price,stop-loss, etc), we can fit and “predict” the underlying dataset quite well.
I The final (24th) viewgraph presents the results of implementing the resulting

strategy on a continuation of the underlying (pseudorandom) dataset.

I Code due to Stephanie Ger, Harvard University, modified from earlier code by

Marcos Lopez de Prado.

Optimizing an investment strategy to fit pseudorandom time series, pg. 01

Optimizing an investment strategy to fit pseudorandom time series, pg. 02

Optimizing an investment strategy to fit pseudorandom time series, pg. 03

Optimizing an investment strategy to fit pseudorandom time series, pg. 04

Optimizing an investment strategy to fit pseudorandom time series, pg. 05

Optimizing an investment strategy to fit pseudorandom time series, pg. 06

Optimizing an investment strategy to fit pseudorandom time series, pg. 07

Optimizing an investment strategy to fit pseudorandom time series, pg. 08

Optimizing an investment strategy to fit pseudorandom time series, pg. 09

Optimizing an investment strategy to fit pseudorandom time series, pg. 10

Optimizing an investment strategy to fit pseudorandom time series, pg. 11

Optimizing an investment strategy to fit pseudorandom time series, pg. 12

Optimizing an investment strategy to fit pseudorandom time series, pg. 13

Optimizing an investment strategy to fit pseudorandom time series, pg. 14

Optimizing an investment strategy to fit pseudorandom time series, pg. 15

Optimizing an investment strategy to fit pseudorandom time series, pg. 16

Optimizing an investment strategy to fit pseudorandom time series, pg. 17

Optimizing an investment strategy to fit pseudorandom time series, pg. 18

Optimizing an investment strategy to fit pseudorandom time series, pg. 19

Optimizing an investment strategy to fit pseudorandom time series, pg. 20

Optimizing an investment strategy to fit pseudorandom time series, pg. 21

Optimizing an investment strategy to fit pseudorandom time series, pg. 22

Optimizing an investment strategy to fit pseudorandom time series, pg. 23

Deploying the resulting strategy on a continuation of the time series

John von Neumann on the danger of statistical overfitting

On the danger of statistical overfitting, Enrico Fermi recalled John von Neumann’s
warning:
I remember my friend Johnny von Neumann used to say, with four parameters
I can fit an elephant, and with five I can make him wiggle his trunk.

Why the silence in the mathematical finance community?
I Historically scientists have led the way in exposing those who utilize pseudoscience

to extract a commercial benefit: e.g., in the 18th century, physicists exposed thenonsense of astrologers.
I Yet financial mathematicians in the 21st century have remained disappointingly
silent with the regards to those in the community who, knowingly or not:1. Fail to disclose the number of models or variations that were used to develop an
investment strategy.
2. Make vague predictions that do not permit rigorous testing and falsification.
3. Misuse probability theory, statistics and stochastic calculus.
4. Use dubious technical jargon: “stochastic oscillators,” “Fibonacci ratios,” “cycles,”“Elliot wave,” “Golden ratio,” “parabolic SAR,” “pivot point,” “momentum,” etc.

As we recently wrote in our paper “Pseudo-Mathematics and Financial Charlatanism”:“Our silence is consent, making us accomplices in these abuses.”
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest over fitting on out-of-sample performance,” Notices of theAmerican Mathematical Society, May 2014, pg. 458–471.

Numerical reproducibility in high-performance computing
The report mentioned above on reproducibility in high-performance computing noted:
Numerical round-off error and numerical differences are greatly magnified ascomputational simulations are scaled up to run on highly parallel systems. As
a result, it is increasingly difficult to determine whether a code has beencorrectly ported to a new system, because computational results quickly
diverge from standard benchmark cases. And it is doubly difficult for otherresearchers, using independently written codes and distinct computer
systems, to reproduce published results.

I V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting the default toreproducible: Reproducibility in computational and experimental mathematics,” Jan 2013, available at
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

Numerical reliability
Many applications routinely use either 32-bit or 64-bit IEEE arithmetic, and employ
fairly simple algorithms, assuming that all is well. But problems can arise:
1. Large-scale, highly parallel simulations, running on systems with hundreds ofthousands or millions of processors — numerical sensitivities are greatly magnified.
2. Certain applications with highly ill-conditioned linear systems.
3. Large summations, especially those involving +/- terms and cancellations.
4. Long-time, iterative simulations (such as molecular dynamics or climate models).
5. Computations to resolve small-scale phenomena.
6. Studies in computational physics or experimental mathematics often require hugeprecision levels.
I D. H. Bailey, R. Barrio, and J. M. Borwein, “High precision computation: Mathematical physics anddynamics,” Applied Mathematics and Computation, vol. 218 (2012), pg. 10106–10121.

Analysis of collisions at the Large Hadron Collider
I The 2012 discovery of the Higgs boson at the ATLAS experiment in the LHC

relied crucially on the ability to track charged particles with exquisite precision (10microns over a 10m length) and high reliability (over 99% of roughly 1000
charged particles per collision correctly identified).
I Software: 5 millions line of C++ and python code, developed by roughly 2000

physicists and engineers over 15 years.
I Recently, in an attempt to speed up the calculation, researchers found that merely

changing the underlying math library resulted in some collisions being missed or
misidentified.
Questions:
I How serious are these numerical difficulties?
I How can they be tracked down?
I How can the library be maintained, producing numerically reliable results?

Enhancing reproducibility with selective usage of high-precision arithmetic
Problem: Findof the irregular function
P the arc length−kg (x) = x + 0≤k≤10 2 sin(2k x), over the interval (0, π)(using 107 abscissa points).
3.0

I If this computation is done with ordinary double

precision arithmetic, the calculation takes 2.59 seconds
and yields the result 7.073157029008510.
I If it is done using all double-double arithmetic (31-digit

2.5

2.0

1.5

1.0

0.5

accuracy), it takes 47.39 seconds seconds and yields theresult 7.073157029007832.
I But if only the summation is changed to double-double,

the result is identical to the double-double result (to 15
digits), yet the computation only takes 3.47 seconds.

0.5

1.0

1.5

2.0

2.5

3.0

Graph
P of g (x) =x+ 0≤k≤10 2−k sin(2k x),
over (0, π).

U.C. Berkeley’s CORVETTE project and the “Precimonious” toolObjective: Develop software facilities to find and ameliorate numerical anomalies inlarge-scale computations:
I Facilities to test the level of numerical accuracy required for an application.
I Facilities to delimit the portions of code that are inaccurate.
I Facilities to search the space of possible code modifications.
I Facilities to repair numerical difficulties, including usage of high-precision

arithmetic.
I Facilities to navigate through a hierarchy of precision levels (32-bit, 64-bit, 80-bit

or higher as needed).
The current version of this tool is known as “Precimonious.”

I C. Rubio-Gonzalez, C. Nguyen, H. D. Nguyen, J. Demmel, W. Kahan, K. Sen, D. H. Bailey and C. Iancu,“Precimonious: Tuning assistant for floating-point precision,” manuscript, May 2013.

Reproducibility in highly parallel computing: History from 1990-1994
Background:
I Many new parallel systems had been introduced; each claimed theirs was best.
I Many researchers were excited about the potential of highly parallel systems.
I Few standard benchmarks and testing methodologies had been established.
I It was hard to reproduce published performance results; much confusion reigned.
I Overall, the level of rigor and peer review in the field was rather low.

In response, in 1991 DHB published a humorous essay ”Twelve Ways to Fool the
Masses,” poking fun at some of the abuses.
Since abuses continued, DHB presented a talk at Supercomputing 1992 and publisheda paper with specific examples.
I D. H. Bailey, “Misleading performance reporting in the supercomputing field,” Scientific Programming,
vol. 1., no. 2 (Winter 1992), pg. 141–151.

1991 paper: “Twelve ways to fool the masses in highly parallel computing”
1. Quote 32-bit performance results, not 64-bit results, but don’t mention this in paper.
2. Present performance figures for an inner kernel, then represent these figures as theperformance of the entire application.
3. Quietly employ assembly code and other low-level language constructs.
4. Scale up the problem size with the number of processors, but omit any mention of this.
5. Quote performance results projected to a full system.
6. Compare your results against scalar, unoptimized code on conventional systems.
7. When run times are compared, compare with an old code on an obsolete system.
8. Base Mflop/s rates on the operation count of the parallel implementation, instead of thebest practical serial algorithm.
9. Quote performance as processor utilization, parallel speedups or Mflop/s per dollar.
10. Mutilate the algorithm used in the parallel implementation to match the architecture.
11. Measure parallel run times on a dedicated system, but measure conventional run times ina busy environment.
12. If all else fails, show pretty pictures and animated videos, and don’t discuss performance.

1992 paper: Scaling performance results to full-sized system
In some published papers and conference presentations, performance results on
small-sized parallel systems were linearly scaled to full-sized systems, without evenclearly disclosing this fact.
Example: 8,192-CPU performance results were linearly scaled to 65,536-CPU results,
simply by multiplying by eight.
Excuse: “We can’t afford a full-sized system.”This and the other examples mentioned in the next few viewgraphs are presented in:
I D. H. Bailey, “Misleading performance reporting in the supercomputing field,” Scientific Programming,
vol. 1., no. 2 (Winter 1992), pg. 141–151.

1992 paper: Using inefficient algorithms on highly parallel systems
In many cases, inefficient algorithms were employed for the highly parallel
implementation, requiring many more operations, thus producing artificially highMflop/s rates:
I Numerous researchers cited parallel PDE performance based explicit schemes,

where implicit schemes were known to be much better.
Excuse: Explicit schemes “run better” on the researchers’ parallel system.
I One paper cited performance for computing a 3D discrete Fourier transform bydirect evaluation of the defining formula (8n2 operations), rather than by using afast Fourier transform (5n log2 n).
Excuse: Direct computation was “more appropriate” for the architecture being
analyzed.
Both examples violate a rule of professional performance reporting, namely to base theoperation count (when computing Mflop/s or Gflop/s rates) on the best practical serial
algorithm, no matter what scheme was actually used on the parallel system.

1992 paper: Not actually performing a claimed computation
Abstract of published paper: “The current Connection Machine implementation runs
at 300-800 Mflop/s on a full [64K] CM-2, or at the speed of a single processor of a
Cray-2 on 1/4 of a CM-2.”
I Excerpt from text: “This computation requires 568 iterations (taking 272

seconds) on a 16K Connection Machine.”
In other words, the computation was run on a 16K system, not on a 64K system;
the figures cited in the Abstract were merely multiplied by four.
I Excerpt from text: “In contrast, a Convex C210 requires 909 seconds to compute

this example. Experience indicates that for a wide range of problems, a C210 isabout 1/4 the speed of a single processor Cray-2.”
In other words, the computation mentioned in the Abstract was not actually run
on a Cray-2; instead, it was run on a Convex system, and a questionable
rule-of-thumb scaling factor was used to produce the Cray-2 rate.

1992 paper: Performance plot [parallel (lower) vs vector (upper)]3.5

3Time (Hours)

2.5

2

1.5

1

0.5

0

0

1000

2000

3000

4000

5000

6000

Number of Objects

7000

8000

9000

10000

1992 paper: Data for performance plot
Problem size(x axis)
20
40
80
160
990
9600

Parallel system
run time
8:18
9:11
11:59
15:07
21:32
31:36

Vector system
run time
0:16
0:26
0:57
2:11
19:00
3:11:50*

Details in text of paper:
I In last entry, the 3:11:50 figure is an “estimate.”
I The vector system code is “not optimized.”

Note that the parallel system is actually slower than the vector system for all cases,
except for the last (estimated) entry. Also, except for the last entry, all real data in thegraph is in the lower left corner.

Fast forward to 2014: New ways to fool the masses
I Cite performance rates for a run with only one processor core active in a shared-

memory multi-core node, producing artificially inflated performance (since there is
no shared memory interference) and wasting resources (since most cores are idle).
I

Example: Cite performance on “1024 cores,” even though the code was run on 1024
multicore nodes, one core per node, with 15 out of 16 cores idle on each node.

I Claim that since one is using a graphics processing unit (GPU) system, that

efficient parallel algorithms must be discarded in favor of more basic algorithms.
I Cite performance rates only for a core algorithm (such as FFT or linear systemsolution), even though full-scale applications have been run on the system.
I List only the best performance figure in the paper, even though the run was madenumerous times (recall the experience of pharmaceutical tests).
I Employ special hardware, operating system or compiler settings that are notappropriate for real-world production usage.
I Define “scalability” as successful execution on a large number of CPUs, regardlessof performance.

Summary
I Super-powerful, highly parallel computer systems, and correspondingly powerfulsoftware, are tools of unprecedented power for researchers and engineers.
I Many fields, long starved for reliable data, are now drowning in data.
I Dangers lie ahead, particularly in reproducibility and statistical reliability.
I Potential dangers are illustrated in mathematical finance, where almost any

desired performance can be achieved by massaging a model long enough.
I Numerical reproducibility is of particular concern, especially for very large-scalecomputations. Do results have any numerical validity?
I The issue of questionable performance reporting practices is again emerging inscientific computing.
Science or pseudoscience? That is the question.
This talk is available at http://www.davidhbailey.com/dhbtalks/dhb-pseudoscience.pdf.