Backtesting in the world of quantitative finance:
Limitations, opportunities and scrutiny neededDavid H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and Univ. of California, Davis
Collaborators: Jonathan M. Borwein (Univ. of Newcastle, Australia), Marcos Lopez dePrado (Guggenheim and LBNL), Jim Q. Zhu (Univ. of Western Michigan) and others

September 29, 2015

1 / 21DANGER AHEAD

Supercomputers operating on big data cangenerate nonsense faster than ever before!

Key concerns:
I Are the results statistically sound?
I Are the results numerically reliable?
I Have the results been validated using independent rigorous tests?
I Are the algorithms, data sources and processing methods well documented?

2 / 21

Reproducibility crises in other fields
I In 2011, Bayer researchers reported that they wereable to reproduce the results of only 17 of 67
published pharmaceutical studies.
I In 2012, Amgen researchers reported that they wereable to reproduce the results of only 6 of 53
published cancer studies.
I In 2015, the Reproduciblity Project at the Univ. ofVirginia reported that they were able to reproduce
only 39 of 100 published psychology studies.

Reproducibility Project staff
Credit: NY Times

These experiences have exposed at least one fundamental flaw in methodology:
Only publicizing the results of successful trials introduces a bias into the results.
3 / 21

Email from DHB to a financial colleague, 10 June 2013
One thing that has always puzzled me about [the financial world] is the following sort of thing:[examples cited]. Excuse me for being “dumb,” but this sort of thing seems to me to beoutright nonsense. ...
After all, the stock market, by definition, contains the consensus of all available information,
including the tens of thousands of stock market analysts and economists worldwide who scour
every morsel of information in the business world, and then advise the leading mutual funds
and pension funds. ...
In addition, ... there are thousands more very bright mathematicians using program-trading
schemes, plying every trick of time series analysis, machine learning, stealth and anti-stealththat money can buy, to wriggle every conceivable angle out of the market and beat theircompetitors to the punch with trades. ...
So when people like those above [say] that they “know” [where] the stock market is heading,... or that by following their strategies, John Q Public can enjoy reliable, above-market returns,
this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?
4 / 21

Response from colleague to DHB, 17 June 2013
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. ...
I completely agree with your assessment. The amount of nonsense ... is incredible.
The good news is, the quants are silently taking over Wall Street, thanks to high frequency andbig data. For the same reason that alchemists and astrologers fought the chemists andastronomers, the market wizards are fighting the quants. So all this ... nonsense is in part thetug of that war. An attempt of the wizards to squeeze out a few more dimes.

5 / 21

Email from another colleague to DHB, 5 May 2015
You have written about economics and risk assessment and so I’d like to know if you have any
ideas about protecting personal wealth.
I thought of you while reading Janet Tavakoli’s Decisions: Life and Death on Wall Street. Have
you read it? I turned to the book after noting it was [promoted] by Nomi Prins, another WallStreet ex-exec like Tavakoli who’s been spilling the beans about Wall Street shenanigans.
Economists like Simon Johnson, Anat Admati and Joseph Stiglitz have been writing similar
stories from a broader theoretical perspective, but all-in-all, all five (and they are hardly alone)
describe a rigged game.
So what to do about it at the personal level? This comes down to wondering about specificthings like savings accounts, CDs, stocks, bonds and annuities, life insurance andhome-ownership vs renting.

6 / 21

A credibility crisis in finance?
I Rightly or wrongly, many believe that the quantitative finance world was at least

partly to blame for the 2007-2009 worldwide financial crisis.
I Rightly or wrongly, many individual investors believe that the financial system isrigged against them.
I Many are skeptical of the countless new investment funds and financial strategies

claiming to be supported by historical market data.
I Financial news is replete with dubious technical jargon: “Fibonacci ratios,”“cycles,” “Elliott waves,” “golden ratios,” “parabolic SARs,” “pivot points,”“technical analysis,” etc.
What should the quant community do?
First and foremost, the quantitative finance field needs to ensure that its own
published research and methods are mathematically and statistically sound.
7 / 21

Backtest overfitting
Finance has been stung with many instances of investment
strategies that look great on paper, but fall flat in practice. A
leading factor is backtest overfitting:
I Proposing a model for a dataset that inherently possesses ahigher level of complexity than the historical data; orI Using a computer to try many variations of a model orstrategy on the historical data, and then only presenting
results from the variation that works best.

432

1

1

2

3

4

5

Fitting six data points(almost perfectly!) with afourth-degree function.

I Constructing an exchange-traded fund by exploring millions

of weighting factors, then only marketing the one with thehighest backtest score.
When a computer can analyze thousands, millions or even billions of variations of agiven strategy on a fixed backtest dataset, it is almost certain that the best suchstrategy will be overfit and thus of dubious value.
8 / 21

How easy is it to overfit a backtest? Very!
I If only 2 years of daily backtest data are available, then no more than 7 strategy

variations should be tried.
I If only 5 years of daily backtest data are available, then no more than 45 strategy

variations should be tried.
A backtest that does not report the number of trials N makes it impossible to assess
the risk of overfitting.

MinBTL ≈ !2
(1 − γ)Z −1 1 − N1 + γZ −1 1 − N1 e −1
E [maxN ]I “Pseudo-mathematics and financial charlatanism: The

effects of backtest overfitting on out-of-sample
performance,” Notices of the AMS, May 2014, pg.
458–471, http://www.financial-math.org.
9 / 21

Letters to clients: An absurd investment scheme
I A financial advisor sends letters to 10, 240 = 10 × 210

prospective clients, with 5120 predicting a certain stock will
go up, and 5120 predicting it will go down.
I One month later, the advisor sends letters only to the 5120

investors who were previously sent the correct prediction,with 2560 letters predicting a certain stock will go up, and2560 predicting it will go down.
I After ten months, the final ten investors will have been sent

ten consecutive spot-on predictions!
This strategy is absurd, even fraudulent, because the final ten investors are not told of
the thousands of other letters with different predictions.
But why is marketing a statistically overfit strategy, where potential investors are notinformed of the millions of computer trials behind the strategy, any different?
10 / 21

A not-so-absurd investment strategy
Suppose an investor believes that there are daily, weekly ormonthly patterns in stock market data, and she seeks toexploit them. Sample strategies:
I Basic strategy: Buy a set of stocks each Monday, thensell on Wednesday; buy on the 6th of the month, thensell on the 19th; sell in May and go away, etc.
I Refinements: Sell the portfolio if it drops more thanApple stock price

10% from start; purchase shares only when they
increase in value more than 10% from start; etc.

31 Aug 2014 – 31 Aug 2015

Even with these very simple strategies, there are literally millions of variations (bychanging various parameters), which can be quickly explored by computer.
Selecting only the best combination of parameters (and not mentioning the manyothers that were tried) is a classic selection bias statistical error.
11 / 21

Backtest overfitting: An interactive example
An online demonstration is backtest overfitting is now available:
I The user can select either pseudorandom data or real S&P500 historical data.
I The program then runs a simple monthly-cycle strategy with parameters (day in,holding period, stop-loss percentage, side, etc.), adjusting the parameters to findan optimal strategy.
I The final optimal strategy is then tried on a new (out-of-sample) dataset.
I This software is now available in an online demo (try it yourself!):

http://www.financial-math.org

I Credits: Stephanie Ger, Marcos Lopez de Prado, Amir Salehipour, Alex Sim and Kesheng Wu.

12 / 21

Initial strategy on input data (S&P500, 1960–1980): Sharpe ratio = -0.23

13 / 21

Improved strategy: Sharpe ratio = 0.73

14 / 21

Final (optimal) strategy: Sharpe ratio = 1.04

15 / 21

Final strategy on new data (S&P500, 1980–2013): Sharpe ratio = 0.07

16 / 21

Analysis
I After exploring the very large space of strategy variations, the computer program

found a strategy that achieved a Sharpe ratio of 1.04 on the input (backtest) data
(S&P500, 1960–1980).
I However, this optimal strategy, when applied to new (out-of-sample) data

(S&P500, 1980–2013), failed miserably — the Sharpe ratio was 0.07.
I In other words, the “optimal” strategy found by the computer search only fit

idiosyncrasies of the input (backtest) dataset — it has no “intelligence.”
The software demo program is now available online:
http://www.financial-math.org
For additional analysis, aimed at a fairly basic level, see:
I D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting and backtest

performance,” manuscript, 07 Oct 2014, http://www.financial-math.org.
17 / 21

New papers on backtest overfitting by DHB and colleagues
I Presents formulas relating size of dataset to likelihood of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics
and financial charlatanism: The effects of backtest over fitting on out-of-sample
performance,” Notices of the American Mathematical Society, May 2014, pg. 458–471.
I Presents formulas for calculating the probability of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability ofbacktest overfitting,” Journal of Computational Finance, to appear, 27 Feb 2015.
I Introduces backtest overfitting for a general audience:

D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting andbacktest performance,” manuscript, 07 Oct 2014.
I Defines a “deflated Sharpe ratio,” correcting for some forms of distortion:

D. H. Bailey and M. Lopez de Prado, “The deflated Sharpe ratio: Correcting for selection
bias, backtest overfitting and non-normality,” Journal of Portfolio Management, toappear, 31 Jul 2014.

Full references and preprints are available at: http://www.financial-math.org
18 / 21

Why the silence in the mathematical finance community?
Historically scientists have exposed those who utilize pseudoscience for commercial
gain, e.g., in the 18th century, physicists exposed the nonsense of astrologers.
Yet financial mathematicians in the 21st century have remained disappointingly silent
with the regards to those in the community who, knowingly or not:1. Fail to disclose the number of models or variations that were used to develop aninvestment strategy.
2. Make vague predictions that do not permit rigorous testing and falsification.
3. Misuse probability theory, statistics and stochastic calculus.
4. Use pseudomathematical jargon: “Fibonacci ratios,” “cycles,” “Elliott waves,”“golden ratios,” “parabolic SARs,” “pivot points,” “technical analysis,” etc.
As we wrote in a recent paper:“Our silence is consent, making us accomplices in these abuses.”
I “Pseudo-mathematics and financial charlatanism: The effects of backtest overfitting onout-of-sample performance,” Notices of the AMS, May 2014, pg. 458–471,
http://www.financial-math.org.
19 / 21

One financial colleague’s recommendation
Empirical finance is in crisis: Our most important discovery tool is historicalsimulation, and yet, most backtests and time series analyses published in journals areflawed. The problem is well-known to professional organizations of statisticians andmathematicians, who have publicly criticized the misuse of mathematical tools amongfinance researchers. ...
In an attempt to overcome the challenges posed by multiple testing and selection bias,
I emphasize the need to move from an individual-centric to a community-driven
research paradigm. ... Stronger theoretical foundations and closer ties [betweenacademics and] financial firms would help prevent false discoveries.

I M. Lopez de Prado, “The future of empirical finance,” Journal of Portfolio Management, 41(4),2015, http://www.financial-math.org.

20 / 21

Visit our website and blog
Mathematicians Against Fraudulent Financial and Investment Advice (MAFFIA):
http://www.financial-math.org
or
http://www.m-a-f-f-i-a.org
This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-london-quant.pdf

21 / 21