HOW TO SPOT BACKTEST
OVERFITTINGDavid H. Bailey
Lawrence Berkeley National Lab (retired), andUniversity of California, DavisMarcos López de Prado
Guggenheim Partners, LLC
in collaboration withJonathan M. Borwein, University of Newcastle, Australia
Qiji Jim Zhu, Western Michigan University

1

Key points· Backtests (i.e., historical simulations of performance) are widely employed
to test and operate investment strategies.
· If the researcher tries a large enough number of strategy configurations, abacktest can always be fit to any desired performance for a fixed samplelength. Thus, there is a minimum backtest length (MinBTL) that should be
required for a given number of trials.
· Standard statistical techniques designed to prevent regression overfitting,such as hold-out, are ineffective in the context of backtest evaluation.
· Under memory effects, overfitting may lead to systematic losses.
· Overfitting is just one example of the misuse of mathematical and statisticalmethods applied to finance.
· Since most published backtests do not report the number of trials involved,
many are overfit.
“I remember my friend Johnny von Neumann used to say, with fourparameters I can fit an elephant, and with five I can make him wiggle histrunk.” [Enrico Fermi, 1953]
2

Backtesting· A backtest is a historical simulation of an algorithmic investment
strategy.
· Among other results, it computes the series of profits and losses thatsuch strategy would have generated, should that algorithm had been run
over a specified time period.
Example of a backtested strategy è""The green line plots the performance of a
tradable security, while the blue line plotsthe performance achieved by buying andselling that security. Sharpe ratio is 1.77,
with 46.21 trades per year. Note the low
correlation between the strategy’s
performance and the security’s."

3Reasons for backtesting investment strategies· The information contained in the reported series of profits and losses
may be summarized in popular performance metrics, such as theSharpe Ratio (SR).
· These metrics are essential to select optimal parameter
combinations: Calibration frequency, risk limits, entry thresholds,
stop losses, profit taking, etc.

Optimizing two parameters generates a 3Dsurface, which can be plotted as a heat-map– see graph è""The x-axis tries different entry thresholds,
while the y-axis tries different exit thresholds. ""The spectrum closer to green indicates theregion of optimal in-sample Sharpe Ratio."4

DANGER AHEAD
Supercomputers and high-tech mathematical finance algorithms cangenerate nonsense faster than ever before!
The principal danger is statistical overfitting of backtest data:· When a computer can analyze thousands or millions of variations of agiven strategy, it is almost certain that the best such strategy, measured
by backtests, will be overfit (and thus of dubious value).
· Many studies claim profitable investment strategies, but their results are
based only on in-sample (IS) statistics, with no out-of-sample (OOS)
testing.
· Overfitting is the most common reason that mathematical investment
schemes look great in backtests, but then fall flat in the real world.
· … and yet, most backtesting software does not control for the probabilityof backtest overfitting!

5The hold-out method to test an investment
strategy (not very good)Training
Set"

Unusedsample"

Model
Configurations"
Investment
strategy"No"

IS Perform.
Evaluation"

Is Perform.
Optimal?"Yes"

Unusedsample"

Testing"
Set"

Investment
strategy"
OOS
Perform.
Evaluation"
6

Optimal Model
Configuration"… single point of⇒High Variance ofdecisionError!

How easy is it to overfit a backtest?
10Minimum	  Backtest	  Length	  (in	  Years)

Unfortunately, too easy!""
For instance, if only 5 years ofdata are available, no morethan 45 independent modelconfigurations should be tried.
For that number of trials, theexpected max IS SR = 1,
whereas the expected OOS
SR = 0."1286M inBT L ⇡4(1⇥)Z 1 11N⇤⇥+ Z 1 1

11Ne

E[maxN ]

⇤ !2

20
0

100

200

300

400

500

600

700

800

900

1000

Number	  of	  Trials	  (N)After trying only 7 independent strategy configurations, the expected maximum IS SR= 1 for a 2-year-long backtest, while the expected OOS SR = 0."!
Therefore, a backtest that does not report the number of trials N used to identify theselected configuration makes it impossible to assess the risk of overfitting.""
Overfitting makes any Sharpe ratio achievable in-sample: The researcher just needs
to keep trying alternative parameters for that strategy!!
7

Overfitting in the absence of memory

The left figure shows the relation between SR IS (x-axis) and SR OOS (y-axis).
Because the process follows a random walk, the scatter plot has a circular shapecentered in the point (0,0).""
The right figure illustrates what happens once we add a “model selection”
procedure. Now the SR IS ranges from 1.2 to 2.6, and it is centered around 1.7.
Although the backtest for the selected model generates the expectation of a 1.7
SR, the expected SR OOS is unchanged around 0."8Overfitting in the absence of memory (cont.)
This figure shows what
happens when we select therandom walk with highest SR
in-sample (IS).""The performance of the firsthalf was optimized (IS), andthe performance of the second
half is what the investor
receives out-of-sample (OOS).""The good news is, in theabsence of memory there is
no reason to expect overfittingto induce negative
performance."
In-Sample (IS)"9Out-Of-Sample (OOS)"

Overfitting in the presence of memory
Memory effects may cause OOS
performance to be negative, eventhough the underlying process wastrendless.""Also, a strongly negative linear
relation between performance IS andOOS may arise, indicating that themore we optimize in-sample, theworse is OOS performance."""
Conclusion:"
When financial analysts do not control
for overfitting, “Past performance is
not an indicator of futureperformance” is too optimistic! Goodbacktest performance may be anindicator of negative future results."

The p-values associated with the intercept and the in-sample
performance (SR a priori) are respectively 0.5005 and 0,indicating that the negative linear relation between IS andOOS Sharpe ratios is statistically significant."
10

Tools to prevent backtest overfitting
1. Compute the probability of backtest overfitting, using a formula given in
our paper “The probability of backtest overfitting,” available at
http://ssrn.com/abstract=2326253 or http://www.financial-math.org.
2. Compute performance degradation and probability of loss (also given in
the above paper).
3. Apply the theory of stochastic dominance, which allows us to rank
investment strategies without having to make assumptions regarding anindividual’s utility function (see above paper for details).
4. Perform model sequestration: Announce a proposed investment strategy
to others (either publicly, or within a firm), then subsequently publish theresults of using this strategy for a pre-specified period of time.
§See D. Leinweber and K. Sisk, “Event Driven Trading and the ‘New News’,”
Journal of Portfolio Management, vol. 38(1), pg. 110-124.

11Reproducibility in financeUsing rigorous methods in mathematical finance (e.g., to prevent backtestoverfitting) enhances reproducibility and reliability:· Many other scientific disciplines are facing similar issues ofreproducibility, to overcome the bias of only publishing “good” results.
· There is a growing movement in the pharmaceutical industry to require
the results of all prototype drug tests to be made public. See
http://www.alltrials.net.
· Johnson & Johnson recently announced it will make all test results public.
· Mathematicians and computer scientists are setting standards forreproducibility in the field of scientific computing. See:§V. Stodden, D. Bailey, J. Borwein, E. LeVeque, W. Rider, and W. Stein,“Setting the default to reproducible: Reproduciblity in computational andexperimental mathematics,” February 2013, available at
http://www.davidhbailey.com/dhbpapers/icerm-report.pdf.

12

An absurd investment program· An investment advisor initially sends 10,240 letters to prospective clients. In 5120 ofthese letters, she predicts that a certain set of securities will go up; in the other 5120she predicts they will go down.
· One month later, if the securities have gone up, she sends another letter to the first
5120 and ignores the second 5120 (or the reverse if the securities have gone down).
In 2560 of these letters, she predicts the securities will go up; in the other 2560, she
predicts the securities will go down.
· One month later, if the securities have gone up, she sends another letter to the first
2560 and ignores the second 2560 (or the reverse if the securities have gone down).
In 1280 of these letters, she predicts the securities will go up; in the other 1280, shepredicts the securities will go down. This is repeated for ten months.
· After ten months, the remaining 10 investors, astounded by the advisor’s uncanny
prophetic powers to date, will entrust all their money to her.

Clearly this is an absurd, even fraudulent investment program, because
investors are never told of the many other failed recommendations.
But why is backtest overfitting, where one does not disclose how many models
were tested, any different?
13Why the silence in the mathematicalfinance community?
· Historically scientists have led the way in exposing those who utilize
pseudoscience to extract a commercial benefit – i.e., in the 18th century,physicists exposed the nonsense of astrologers.
· Yet financial mathematicians in the 21st century have remained
disappointingly silent with those who, knowingly or not:
§
§
§
§
§

Fail to disclose the number of models that were used to develop a scheme(i.e., backtest overfitting).
Make vague predictions that do not permit rigorous testing and falsification.
Misuse charts and graphs: “Beware of fund managers bearing double y-axes.”
See Matthew Obrien’s article on the “scary chart” in the Atlantic (11 Feb 2014).
Misuse probability theory, statistics and stochastic calculus.
Misuse technical jargon: “stochastic oscillators,” “Fibonacci ratios,” “cycles,”“Elliot wave,” “Golden ratio,” “parabolic SAR,” “pivot point,” “momentum”, andothers in the context of finance.

· Our silence is consent, making us accomplices in these abuses.
“One has to be aware now that mathematics can be misused and that wehave to protect its good name.” – Andrew Wiles, New York Times, 4 Oct 2013.
14

Mathematicians Against Fraudulent Financial andInvestment Advice (MAFFIA)
http://www.financial-math.org(main site)
http://www.m-a-f-f-i-a.org(alias to main site)
http://www.financial-math.org/blog/(blog)The principal purpose is education, not confrontation – helping readersrecognize and avoid fallacies and abuses in mathematical finance.
For full technical details on the material in this talk:· “Pseudo-mathematics and financial charlatanism: The effects of backtest
overfitting on out-of-sample performance,” Notices of the American MathematicalSociety, to appear (May 2014), http://ssrn.com/abstract=2308659.
· “The probability of backtest overfitting,” manuscript, 10 Feb 2014, available at
http://ssrn.com/abstract=2326253.
· These papers (and this talk) are also available at http://www.financial-math.org.

THANK YOU!
15