Scientific Integrity in Mathematical FinanceDavid H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and Univ. of California, Daviswith collaborators
Jonathan M. Borwein, CARMA, University of Newcastle, Australia
Marcos Lopez de Prado, Lawrence Berkeley National Lab (affiliate)
Qiji Jim Zhu, University of Western Michigan

11 July 2014

The dangers of reading too much into data
Another thing I must point out is that you cannot prove a vague theory
wrong. [...] Also, if the process of computing the consequences is indefinite,then with a little skill any experimental result can be made to look like theexpected consequences. [Richard Feynman, 1964]

I remember my friend Johnny von Neumann used to say, with fourparameters I can fit an elephant, and with five I can make him wiggle histrunk. [Enrico Fermi]

High-performance computing meets mathematical finance

The advent of highly parallel scientific computing is revolutionizing the world finance,
just as it has almost every other field of modern science and engineering.
Researchers and analysts can:I Access and analyze petabytes of past stock and bond market data.
I Analyze hundreds, thousands or millions of different strategies or portfolio options.
I Apply extremely sophisticated mathematical algorithms to investment and financedata — software is widely available, albeit at a fee.
I Implement trading algorithms that can perform thousands of transactions on

multiple trading venues, in a flash.

Increasing performance of the top 500 supercomputers (1994 – present)

Red = #1
Orange P= #500Blue =#1 thru #500

I “Performance development,” Top500.org, available at http://top500.org/statistics/perfdevel.

DANGER AHEAD
In spite of the successes of computing in the finance field, danger lies ahead, just as inevery other field of modern science:Supercomputers can generate nonsense faster than ever before!
Key concerns:
I Are the algorithms, data sources and processing methods well documented?
I Are the results reproducible by other researchers, or even by the same team ofresearchers?
I Are the results statistically sound?
I Are the results numerically reliable?
I Have the results been validated using tests designed by the researchers or others?

Reproducibility in scientific computing
A December 2012 workshop on reproducibility in computing, held at Brown University
in Rhode Island, USA, noted that
Science is built upon the foundations of theory and experiment validated andimproved through open, transparent communication. With the increasingly
central role of computation in scientific discovery, this means communicating
all details of the computations needed for others to replicate the experiment.
... The “reproducible research” movement recognizes that traditional
scientific research and publication practices now fall short of this ideal, andencourages all those involved in the production of computational science ...
to facilitate and practice really reproducible research.

I V. Stodden, D. H. Bailey, J. Borwein, R. J. LeVeque, W. Rider and W. Stein, “Setting the default toreproducible: Reproducibility in computational and experimental mathematics,”
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

Reproducibility in scientific computing, continued
Issues identified in the ICERM report and other studies include:
I The need to carefully document the full context of computational

experiments—system environment, input data, code used, computed results, etc.
I The need to save the code and output data in a permanent repository.
I The need for reviewers, research institutions and funding agencies to recognize theimportance of computing and computing professionals, and to allocate funding forafter-the-grant support and repositories.
I The increasing importance of numerical reproducibility, and the need for tools toensure and enhance numerical reliability.
I The need to encourage publication of negative results—other researchers canoften learn from them.
I The re-emergence of the need to ensure responsible reporting of performance.

Reproducibility in biomedicine
The biomedical field has been stung by numerous cases where pharma products lookgood based on clinical trials, but later disappoint in real-world usage, or the resultscannot be reproduced in separate studies. Examples:
I In 2004, GlaxoSmithKline acknowledged that while some trials of Paxil found it

effective for depression in children, other unpublished studies showed no benefit.
I In 2011, Bayer researchers reported that they were able to reproduce the results ofonly 17 of 67 published studies they examined.
I In 2012, Amgen researchers reported that they were able to reproduce the resultsof only 6 of 53 published cancer studies.
I In 2014, a review of Tamiflu found that while it made flu symptoms disappear abit sooner, it did not stop serious complications or keep people out of the hospital.
These experiences have exposed a fundamental flaw in methodology:
Only publicizing the results of successful trials introduces a bias into the results.
The AllTrials movement would require all results to be public: http://www.alltrials.net

Reproducibility in physics
In March 2014, a team of researchers from Harvard
University made the dramatic announcement thatthey had discovered an interesting “twisting”
pattern in cosmic microwave background data,measured using their BICEP2 experimental system.
This pattern fit very well with the hypothesizedpattern of the most commonly assumed model of“inflation” in the first tiny fraction of a second after
the big bang, and thus has been trumpeted as thefirst experimental evidence of the inflationary
cosmology.
But other researchers had difficulty reconstructing
the claimed results. Finally, two teams challenged
the BICEP2 findings, saying that the results could
more readily be explained by dust in the Milky Way.

I Ron Cowen, “Doubt grows about
gravitational waves detection,”
Scientific American, 2 Jun 2014.

Reproducibility in social science
The “blank slate” paradigm (1920–1990):
I The human mind at birth is a tabula rasa (“blank slate”).
I Heredity and biology play no significant role in human psychology; all personality

and behavioral traits are socially constructed.
I Pre-modern societies were serene, devoid of most psychological and social

problems that afflict modern societies.
As Ashley Montagu wrote in 1973,
Man is man because he has no instincts, because everything he is and hasbecome he has learned, acquired, from his culture, from the man-made partof the environment, from other human beings.
Even today, one often hears this philosophy expressed, particularly in popular media.

The fall of the blank slate
The current consensus in the field, based on latest research:
I Humans at birth possess sophisticated facilities for social interaction, language

acquisition, pattern recognition, navigation and counting.
I Heredity, evolution and biology are major factors in human personality.
I Some personality traits are more than 50% heritable.
I Pre-modern societies had more crime, war and social problems than modern

western societies.
How did the early 20th century social scientists err so badly?
I Sloppy experimental methodology and analysis.
I Pervasive wishful thinking and politically correct biases.
I Ignoring or dismissing data that runs counter to predisposition.
I S. Pinker, The Blank Slate: The Modern Denial of Human Nature, Penguin Books, 2003.

Email from DHB to a financial colleague, 10 June 2013
One thing that has always puzzled me about financial news and commentary is the following
sort of thing. Look at these two examples:
This fellow says that “we are heading over 1700 before the summer swoon”, and makes
number of other rather specific predictions: [URL given]. Or this fellow “tells you where hethinks the stock market is headed, and why” [URL given]. ... Excuse me for being “dumb”,
but this sort of thing seems to me to be outright nonsense — callous attempts to fleece moneyout of credulous, ignorant investors.
After all, the stock market, by definition, contains the consensus of all available information,
including the tens of thousands of stock market analysts and economists worldwide who scour
every morel of information in the business world, and then advise the leading mutual funds andpension funds. Thus the big players have already made their moves long before individual
investors hear anything. In addition, as is amply documented even in semi-popular books suchas Dark Pools, there are thousands more very bright mathematicians using program-trading
schemes, plying every trick of time series analysis, machine learning, stealth and anti-stealththat money can buy, to wriggle every conceivable angle out of the market and beat theircompetitors to the punch with trades.

Email from DHB to a financial colleague, continued
How can amateurs possibly compete with these outfits? Obviously, they can’t — the only
rational strategy for an amateur individual investor is simply to regularly buy, say via a 401K
program, into one or a handful of broad-market index funds, or, with the help of a qualified
professional, buy and hold a balanced, diversified portfolio.
So when people like those above assure their audiences that they “know” the stock market isheading up, or down, or up for a month or two, then down, or that by following their strategies,
John Q Public can enjoy reliable, above-market returns, this cannot have any scientific basis.
After all, if these pundits really had a substantive, scientifically defensible basis for making suchpredictions, they wouldn’t be selling their advice online, but instead they would be making
millions advising institutional clients with many billions of dollars.
So why doesn’t somebody blown this whistle on this sort of thing? Am I missing something?

Email from financial colleague to DHB
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. They aremarket wizards. Some of them made a lot of money and therefore claim to have magic powers.
But this is a zero sum game, someone has to make a lot of money, as a matter of probability
distribution, not magic.
I completely agree with your assessment. The amount of nonsense in the airways is incredible.
The good news is, the quants are silently taking over Wall Street, thanks to high frequency andbig data. For the same reason that alchemists and astrologers fought the chemists andastronomers, the market wizards are fighting the quants. So all this media nonsense is in partthe tug of that war. An attempt of the wizards to squeeze out a few more dimes.

Reproducibility in finance

Finance, in analogy to the pharmaceutical world, has been stung with numerous
instances of investment strategies that look great on paper, but fall flat in practice. A
primary cause is statistical overfitting of backtest (historical market) data.
When a computer can analyze thousands or millions of variations of a given strategy, it
is almost certain that the best such strategy, measured by backtests, will be overfit andthus of dubious value.

Paper on “Pseudo-mathematics and financial charlatanism”

In two 2014 papers by DHB, JMB, Marcos Lopez de Prado and Jim Zhu, we derive (a)a formula relating the number of trials to the minimum backtest length, and (b) aformula for the probability of backtest overfitting.
We also show that under the assumption of memory in markets, overfit strategies are
actually prone to lose money.
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest over fitting on out-of-sample performance,” Notices of theAmerican Mathematical Society, May 2014, pg. 458–471.
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability of backtest overfitting,”
manuscript, 12 Feb 2014, http://ssrn.com/abstract=2326253.

How easy is it to overfit a backtest? Answer: Very easy!
I If only 2 years of daily backtest data are available, then no more than 7 strategy

variations should be tried.
I If only 5 years of daily backtest data are available, then no more than 45 strategy

variations should be tried.
I A backtest that does not report the number of trials N makes it impossible toassess the risk of overfitting.
I Given any desired performance level, a financial researcher just needs to keep

trying alternative parameters for that strategy!

MinBTL ≈
 !2(1 − γ)Z −1 1 − N1 + γZ −1 1 − N1 e −1
E [maxN ]An absurd investment strategy
I A financial advisor sends letters to 10, 240 = 10 × 210 potential clients, with 5120

letters predicting a certain security will go up, and the other half predicting it willgo down.
I One month later, the advisor sends letters only to the 5120 investors who werepreviously sent the correct prediction, with 2560 letters predicting a certain
security will go up, and the other half predicting it will go down.
I The advisor continues this process for 10 months.
I The remaining ten investors, so impressed by the advisor’s ten consecutive spot-on

predictions, will entrust to him/her all of their assets!
This strategy is absurd; even fraudulent.
But why is marketing a statistically overfit strategy, where potential investors are notinformed of the number of trials behind the strategy, any different?

Optimizing an investment strategy to fit pseudorandom time series
I The following 23 viewgraphs present the results of different steps in an attempt tofind an “optimal” investment strategy, based on a fixed market price dataset.
I The underlying dataset was generated by a pseudorandom number generator!
I As you can see, by tweaking some very basic parameters (entry price, sell price,stop-loss, etc), we can fit and “predict” the underlying dataset quite well.
I The final (24th) viewgraph presents the results of implementing the resulting

strategy on a continuation of the underlying (pseudorandom) dataset.

I Code due to Stephanie Ger, Harvard University, modified from earlier code byMarcos Lopez de Prado.

Optimizing an investment strategy to fit pseudorandom time series, pg. 01

Optimizing an investment strategy to fit pseudorandom time series, pg. 02

Optimizing an investment strategy to fit pseudorandom time series, pg. 03

Optimizing an investment strategy to fit pseudorandom time series, pg. 04

Optimizing an investment strategy to fit pseudorandom time series, pg. 05

Optimizing an investment strategy to fit pseudorandom time series, pg. 06

Optimizing an investment strategy to fit pseudorandom time series, pg. 07

Optimizing an investment strategy to fit pseudorandom time series, pg. 08

Optimizing an investment strategy to fit pseudorandom time series, pg. 09

Optimizing an investment strategy to fit pseudorandom time series, pg. 10

Optimizing an investment strategy to fit pseudorandom time series, pg. 11

Optimizing an investment strategy to fit pseudorandom time series, pg. 12

Optimizing an investment strategy to fit pseudorandom time series, pg. 13

Optimizing an investment strategy to fit pseudorandom time series, pg. 14

Optimizing an investment strategy to fit pseudorandom time series, pg. 15

Optimizing an investment strategy to fit pseudorandom time series, pg. 16

Optimizing an investment strategy to fit pseudorandom time series, pg. 17

Optimizing an investment strategy to fit pseudorandom time series, pg. 18

Optimizing an investment strategy to fit pseudorandom time series, pg. 19

Optimizing an investment strategy to fit pseudorandom time series, pg. 20

Optimizing an investment strategy to fit pseudorandom time series, pg. 21

Optimizing an investment strategy to fit pseudorandom time series, pg. 22

Optimizing an investment strategy to fit pseudorandom time series, pg. 23

Deploying the resulting strategy on a continuation of the time series

Why the silence in the mathematical finance community?
I Historically scientists have led the way in exposing those who utilize pseudoscience

to extract a commercial benefit: e.g., in the 18th century, physicists exposed thenonsense of astrologers.
I Yet financial mathematicians in the 21st century have remained disappointingly
silent with the regards to those in the community who, knowingly or not:1. Fail to disclose the number of models or variations that were used to develop aninvestment strategy.
2. Make vague predictions that do not permit rigorous testing and falsification.
3. Misuse probability theory, statistics and stochastic calculus.
4. Use dubious technical jargon: “stochastic oscillators,” “Fibonacci ratios,” “cycles,”“Elliot wave,” “Golden ratio,” “parabolic SAR,” “pivot point,” “momentum,” etc.

As we recently wrote in our paper “Pseudo-Mathematics and Financial Charlatanism”:“Our silence is consent, making us accomplices in these abuses.”
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest over fitting on out-of-sample performance,” Notices of theAmerican Mathematical Society, May 2014, pg. 458–471.

Mathematicians Against Fraudulent Financial and Investment Advice(MAFFIA)
Main site: http://www.financial-math.org
or
http://www.m-a-f-f-i-a.org
Blog: http://www.financial-math.org/blog
If you share our concerns, join the MAFFIA! Send email todavid@davidhbailey.com, and we’ll add you to our mailing list.

This talk is available at http://www.davidhbailey.com/dhbtalks/dhb-integrity.pdf.