Stock portfolio design and backtest overfittingDavid H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and University of California, Davis
Collaborators: Jonathan M. Borwein (Univ. of Newcastle, Australia, deceased),
Marcos Lopez de Prado (Guggenheim Partners and LBNL), Jim Q. Zhu (Univ. of WesternMichigan) and others

September 29, 2017

1 / 37

Reproducibility crises in biomedicine, psychology, economics
I In 2011, Bayer researchers reported that they wereable to reproduce only 17 of 67 pharma studies.
I In 2012, Amgen researchers reported that they wereable to reproduce only 6 of 53 cancer studies.
I In August 2015, the Reproduciblity Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.

Reproducibility Project staff
Credit: NY Times

2 / 37

A credibility crisis in finance?
I Many individual investors believe that the financial

system (high-frequency trading, “dark pools,” etc.) isrigged against them.
I Many are skeptical of the claims of numerous financial

gurus and forecasters.
I Many are skeptical of the hundreds of new investment

funds and strategies that are marketed each year.
I Financial news is replete with pseudomathematicalcharts and jargon: “Fibonacci ratios,” “cycles,” “Elliottwaves,” “golden ratios,” “parabolic SARs,” “technicalanalysis,” “pivot points,” “symmetrical triangles,”“rising wedges,” etc.
What should the mathematical finance community do? First and foremost, ensure that
our own published research and strategies are mathematically and statistically sound.
3 / 37

Email from a mathematical colleague to DHB, 5 May 2015
You have written about economics and risk assessment and so I’d like to know if you have any
ideas about protecting personal wealth.
I thought of you while reading Janet Tavakoli’s Decisions: Life and Death on Wall Street. Have
you read it? I turned to the book after noting it was [promoted] by Nomi Prins, another WallStreet ex-exec like Tavakoli who’s been spilling the beans about Wall Street shenanigans.
Economists like Simon Johnson, Anat Admati and Joseph Stiglitz have been writing similar
stories from a broader theoretical perspective, but all-in-all, all five (and they are hardly alone)
describe a rigged game.
So what to do about it at the personal level? This comes down to wondering about specificthings like savings accounts, CDs, stocks, bonds and annuities, life insurance andhome-ownership vs renting.

4 / 37

Email from DHB to a financial colleague, 10 June 2013
One thing that has always puzzled me about the financial world is the following sort of thing:[examples cited]. Excuse me for being “dumb,” but this sort of thing seems to me to beoutright nonsense. ...
After all, the stock market, by definition, contains the consensus of all available information,
including the tens of thousands of stock market analysts and economists worldwide who scour
every morsel of information in the business world, and then advise the leading mutual funds
and pension funds. ...
In addition, ... there are thousands more very bright mathematicians using program-trading
schemes, plying every trick of time series analysis, machine learning, stealth and anti-stealththat money can buy, to wriggle every conceivable angle out of the market and beat theircompetitors to the punch with trades. ...
So when people like those above say that they “know” where the stock market is heading, ...
or that by following their strategies, John Q Public can enjoy reliable, above-market returns,
this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?
5 / 37

Response from colleague to DHB, 17 June 2013
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. ...
I completely agree with your assessment. The amount of nonsense ... is incredible.
The good news is, the quants are silently taking over Wall Street, thanks to high frequency andbig data. For the same reason that alchemists and astrologers fought the chemists andastronomers, the market wizards are fighting the quants. So all this ... nonsense is in part the
tug of that war. An attempt of the wizards to squeeze out a few more dimes.

6 / 37

Mathematicians against fraudulent financial and investment advice(MAFFIA)In 2013, myself, Jonathan Borwein, Marcos Lopez de Prado and Jim Zhu formed“MAFFIA,” with the goal of doing research in financial mathematics, and, inparticular, to highlight the abuses of mathematics in the financial field:
I We wrote the paper Pseudo-mathematics and financial charlatanism: The effectsof backtest overfitting on out-of-sample performance, which was published in the
Notices of the American Mathematical Society.
I We wrote some additional papers and studies, continuing to the present day.
I We started the Mathematical Investor blog, with new articles posted roughly every
3–4 weeks: http://www.mathinvestor.org.
I With help of some students, we developed online demonstrations of backtest
overfitting, one of the chief abuses in the field.
The remainder of the talk discusses some of our results.
I “Pseudo-mathematics and financial charlatanism: The effects of backtest overfitting on

out-of-sample performance,” Notices of the AMS, May 2014, 458–471.

7 / 37

What is backtest overfitting?
4

I Proposing a model for a dataset that inherently possesses a3higher level of complexity than the historical data.
2

I Using a computer to try millions or billions of variations of amodel or strategy on the historical data, and then only
presenting results from the variation that works best.
I Constructing an exchange-traded fund by exploring millions

or billions of weighting factors, then only marketing the one
with the highest backtest score.

1

1

2

3

4

5

Fitting six data points(almost perfectly!) with afourth-degree function.

When a computer can analyze millions or billions of variations of a fund or strategy on
a fixed backtest dataset, it is almost certain that the optimal fund or strategy will bebadly overfit and thus of dubious value.

8 / 37

How easy is it to overfit a backtest? Very!
I If only 2 years of daily backtest data are available, then no more than 7 strategy

variations should be tried.
I If only 5 years of daily backtest data are available, then no more than 45 strategy

variations should be tried.
A backtest that does not report the number of trials N makes it impossible to assess
the risk of overfitting.

MinBTL ≈


 !2


(1 − γ)Z −1 1 − N1 + γZ −1 1 − N1 e −1
E [maxN ]I “Pseudo-mathematics and financial charlatanism: The

effects of backtest overfitting on out-of-sample
performance,” Notices of the AMS, May 2014, 458–471.
9 / 37

Letters to clients: An absurd investment scheme
I A financial advisor sends letters to 5, 120 = 5 × 210

prospective clients, with 2560 predicting a certain stock will
go up, and 2560 predicting it will go down.
I One month later, the advisor sends letters only to the 2560

investors who were previously sent the correct prediction,with 1280 letters predicting a certain stock will go up, and1280 predicting it will go down.
I After ten months, the final five investors will have been sent

ten consecutive spot-on predictions!
This strategy is absurd, even fraudulent, because the final five investors are not told of
the 10,235 other letters with different predictions.
But why is promoting a statistically overfit strategy, where potential investors are notinformed of the millions of failed computer trials behind the strategy, any different?
10 / 37

A not-so-absurd investment strategy
Suppose an investor believes that there are daily, weekly ormonthly patterns in stock market data, and she seeks to
exploit them. Sample strategies:
I Basic strategy: Buy a set of stocks each Monday, thensell on Wednesday; buy on the 6th of the month, thensell on the 19th; sell in May and go away, etc.
I Refinements: Sell the portfolio if it drops more thanApple stock price

10% from start; purchase shares only when they
increase in value more than 10% from start; etc.

31 Aug 2014 – 31 Aug 2015

Even with these very simple strategies, there are literally millions of variations (bychanging various parameters), which can be quickly explored by computer.
Selecting only the best combination of parameters (and not mentioning the manyothers that were tried) is a classic selection bias statistical error.
11 / 37

Backtest overfitting: An interactive example
An online demonstration is backtest overfitting is now available:
I The user can select either pseudorandom data or real S&P500 historical data.
I The program then runs a simple monthly-cycle strategy with parameters (day in,holding period, stop-loss percentage, side, etc.), adjusting the parameters to findan optimal strategy.
I The final optimal strategy is then tried on a new (out-of-sample) dataset.
I This software is now available in an online demo (try it yourself!):

http://www.mathinvestor.org

I Credits: Stephanie Ger, Marcos Lopez de Prado, Amir Salehipour, Alex Sim and Kesheng Wu.

12 / 37

Initial strategy on input data (S&P500, 1960–1980): Sharpe ratio = -0.23

13 / 37

Improved strategy on input data: Sharpe ratio = 0.73

14 / 37

Final (optimal) strategy on input data: Sharpe ratio = 1.04

15 / 37

Final strategy on new data (S&P500, 1980–2013): Sharpe ratio = 0.07

16 / 37

Analysis
I After exhaustively exploring the space of strategy variations, the computer

program found a strategy that achieved a Sharpe ratio of 1.04 on the input(backtest) pseudorandom time series (i.e., 1.04 standard deviations above zero).
I However, this optimal strategy, when applied to a new (pseudorandom) timeseries, failed miserably — the Sharpe ratio was 0.07 (i.e., no significant gain).
I In other words, the “optimal” strategy found by the computer search only fit

idiosyncrasies of the input (backtest) dataset — it has no fundamental“intelligence” whatsoever.
For additional analysis (aimed at a fairly elementary audience), see:
I D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting andbacktest performance,” manuscript, 07 Oct 2014,
http://ssrn.com/abstract=2507040.
I The software demo program is NOW AVAILABLE online:

http://www.mathinvestor.org.
17 / 37

Additional details on backtest overfitting
I Presents formulas relating size of dataset to likelihood of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics
and financial charlatanism: The effects of backtest over fitting on out-of-sample
performance,” Notices of the American Mathematical Society, May 2014, pg. 458–471.
I Presents formulas for calculating the probability of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability ofbacktest overfitting,” Journal of Computational Finance, to appear, 2015.
I Introduces backtest overfitting for a general audience:

D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting andbacktest performance,” manuscript, 2014.
I Defines a “deflated Sharpe ratio,” correcting for some forms of distortion:

D. H. Bailey and M. Lopez de Prado, “The deflated Sharpe ratio: Correcting for selection
bias, backtest overfitting and non-normality,” Journal of Portfolio Management, toappear, 2014.

Preprint copies are available at: http://www.davidhbailey.com
18 / 37

Proliferation of new stock funds
I Roughly USD$2.1 trillion is held in U.S.-listed exchange-traded funds (ETFs),

with hundreds minted each year.
I In a 2012 study, researchers found that the median time between the definition ofa new index and the inception of a new exchange-traded fund based on the index
dropped from almost three years in 2000 to only 77 days in 2011.
I As a result, the report concludes, “most indexes have little live performance
history for investors to assess in the context of a new ETF investment.”
I Out of 370 new indexes, 87% of the indexes outperformed the broad U.S. stock
market over the time period used for the backtest, but only 51% outperformed
the broad market after inception of the index.
I The study found an average 12.25% annualized excess return above the broad
U.S. stock market for a five-year backtest, but -0.26% excess return in the five
years following the inception of the index.

I Vanguard Research, “Joined at the hip: ETF and index development,” July 2012,

http://www.vanguard.com/pdf/s319.pdf.

19 / 37

How difficult is it to design a stock portfolio to achieve a desired
performance profile?
I Given some desired performance profile (a time series), we construct a weighted

subset of S&P500 stocks whose performance matches, as closely as possible, thatof the profile over the specified backtest time period.
I The design minimizes the sum of squares deviation of the weighted portfolio timeseries from the given profile time series.
I In a typical run, some of the resulting weights are negative, corresponding toshorted positions in certain stocks. This potentially exposes the portfolio to losses.
I As an alternate option, the weights are calculated subject to the constraint that
all weights must be greater than or equal to zero.

I D. H. Bailey, J. M. Borwein and M. Lopez de Prado, “Stock portfolio design and backtest

overfitting,” Journal of Investment Management, vol. 17 (2017), no. 1, preprint available athttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739335.
20 / 37

Constructing a weighted portfolio to achieve a desired performance profile
Given a target time series (vj ) and a collection of m stocks (zi ), each with a time series(zi (tj )), we wish to find m weights (wi ) that minimize the objective function
!2
n
m
X
X
R(w1 , w2 , · · · , wm ) =wi zi (tj ) − vj .
j=1

i=1

Pm

Since i=1 wi zi (tj ) is the weighted portfolio time series, this expression is the sum-of-squares
deviation of the weighted portfolio from the target time series. The function R is minimized
when the following are satisfied:
!
n
m
X
X
∂R
=2
wi zi (tj ) − vj z1 (tj ) = 0,
∂w1
j=1
i=1
!
n
m
X
X
∂R
=2
wi zi (tj ) − vj z2 (tj ) = 0,
∂w2
j=1

i=1

···
n
m
X
X
∂R
wi zi (tj ) − vj
=2
∂wm
j=1

i=1

!
zm (tj ) = 0.
21 / 37

Constructing a weighted portfolio (continued)
This can be rewritten as
m
X
i=1
m
X

wi

wi

n
X
j=1
n
X

i=1

j=1

m
X

n
X

zi (tj )z1 (tj ) =zi (tj )z2 (tj ) =

n
X
j=1
n
X

vj z1 (tj ),vj z2 (tj ),

j=1

···

i=1

wi

j=1

zi (tj )zm (tj ) =nX

vj zm (tj ),j=1

which can be solved for the W vector by using conventional linear system solver software.
Note that it is not essential that n > m; if n < m this scheme produces a best least-squares fit
to the target profile, although the quality of this fit degrades when the ratio n/m falls much
below one.
22 / 37

Constructing an all-positive-weight portfolio
When the technique described above is implemented on real stock market data, some of theresulting weights wi are typically negative (so that the corresponding stocks are shorted). Thisis fine, but entails some risk of catastrophic decline (see examples below).
So as an alternative option, one can also ask for an optimal set of weights W subject to theconstraint that each weight wi ≥ 0.
To that end, we have employed a logarithmic barrier scheme, which is to append a logarithmic
term to the minimization problem, as follows:
R(w1 , w2 , · · · , wm ) =

n
m
X
X
j=1

i=1

!2
wi zi (tj ) − vj

+ 2C

m
X

log wi .

i=1

The presence of this logarithmic term penalizes very small weights and thus serves as a barrier,keeping the weights away from zero or negative values. This is not the same as solving theconstrained problem, but by successively reducing the constant C , the desired limiting solution
can be obtained.
23 / 37

Constructing an all-positive-weight portfolio (continued)
In this case, the equivalent minimizing condition is
m
X
i=1
m
X

wi

wi

n
X
j=1
n
X

i=1

j=1

m
X

n
X

zi (tj )z1 (tj ) =zi (tj )z2 (tj ) =

n
X
j=1
n
X

vj z1 (tj ) + C /w1 ,vj z2 (tj ) + C /w2 ,

j=1

···

i=1

wi

j=1

zi (tj )zm (tj ) =nX

vj zm (tj ) + C /wm .

j=1

This system can be efficiently solved by Newton iterations, where one takes, as starting
estimates of the weights W , the solution to the unconstrained problem above, replacing zero or
negative weights with some very small positive value.
24 / 37

Constructing an all-positive-weight portfolio (continued)
In summary, the algorithm for the constrained problem is the following:
1. Perform the unconstrained matrix calculation to obtain an initial set of weights W .
2. Replace zero or negative weights with a small positive value (we use 10−8 ).
3. Select C = 1, then perform the Newton iteration until convergence (typically inten or fewer iterations).
4. Reduce C by a factor of ten and repeat step (3), continuing until overall
convergence (typically when C = 10−6 or so).

25 / 37

Constructing portfolios from real stock data
Our computer program constructed stock portfolios based exclusively on real S&P 500
historical stock data. Data for S&P 500 stocks are easy to obtain online. For example,
Apple Computer’s daily stock closings going back to 1980 can be downloaded from:https://finance.yahoo.com/q/hp?s=AAPL
http://www.google.com/finance?q=AAPLThe in-sample period was 1991–2005; the out-of-sample period was 2006–2015.
Our program found 277 valid stocks from the S&P 500 database for which dataspanning this time period was available. All stock data used here include reinvested
dividends.

26 / 37

Three types of performance profiles
Using our program, one can generate any of three target profiles (here p is an annual
percentage rate):
1. Steady capital growth: A steady increase by the fraction (1 + p/(100r )) per time
period (i.e., growing by p/r percent each time period, where r is the number oftime periods per year; e.g., r = 12).
2. Stair-step growth: A stair-step function that is constant, except that at the end ofeach q-year period it increases by the fraction (1 + p/(100r ))qr (i.e., at the end ofeach q-year period, it increases by a full q years’ growth of Profile 1 above). We
took q = 1 in the examples below.
3. Sinusoidal growth: A sinusoidal function that increases by the fraction(1 + p/(100r )) per time period, as in profile #1, but is multiplied by a sine wavethat varies from 1/2 to 3/2, with period q years. We took q = 5.
The second and third profiles are included mainly to illustrate that any reasonable
function whatsoever may be specified for the profile.
27 / 37

Results: Steady growth profile, APR = 6%

50 00050 00040 00040 00030 00030 000

20 000

20 000

10 000

10 000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.
28 / 37

Steady growth profile, APR = 8%50 00050 00040 00040 00030 00030 000

20 000

20 000

10 000

10 000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

29 / 37

Steady growth profile, APR = 10%

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

30 / 37

Steady growth profile, APR = 12%30 00030 00025 00025 000

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

31 / 37

Sinusoidal profile, APR = 10%

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

32 / 37

Summary of 20 runs

ProfileSteadygrowthStairstep
Sinusoidal

Fig.
1
2
3
4
5
6
7
8
9
10

APR
6%
8%
10%
12%
15%
18%
8%
10%
8%
10%

Standard weights
RMS dev.
Sharpe ratio
IS
OOS
IS
OOS
0.000 7.658 -0.120 0.168
0.000 2.534 -0.079
FAIL
0.000 0.996 -0.038
FAIL
0.000 1.178
0.003
FAIL
0.000 5.953
0.065 0.178
0.000 0.996
0.126
FAIL
0.000 9.395 -0.066 0.167
0.000 0.996 -0.024
FAIL
0.000 4.518 -0.064
FAIL
0.000 0.996 -0.029
FAIL

All-positive weights
RMS dev.
Sharpe ratio
IS
OOS
IS
OOS
1.426 1.910 0.163 -0.025
1.016 0.970 0.162 -0.025
0.695 0.391 0.161 -0.026
0.452 0.276 0.157 -0.027
0.223 0.557 0.145 -0.016
0.218 0.711 0.177 -0.021
1.086 1.039 0.162 -0.025
0.768 0.442 0.161 -0.025
1.584 1.528 0.162 -0.024
1.267 0.867 0.158 -0.024“APR”: annual percentage rate; “IS”: in-sample period, 1991–2005 (15 years); “OOS”: out-of-sample
period, 2006–2015 (10 years); “RMS dev.”: root-mean-square deviation from target profile; “Sharpe
ratio”: Sharpe ratio relative to S&P 500 with reinvested dividends; “FAIL”: 100% loss of capital.
33 / 37

Analysis
I In every case, the standard portfolio performance achieved zero deviation over thein-sample period. Only beginning with 2006 (the out-of-sample period) do theblue curves depart from the orange curves in the plots.
I In some cases the standard portfolios did remarkably well, but in other cases theyfailed catastrophically.
I The positive-weight portfolios are significantly less erratic and often outperformed

both the target profile and the S&P 500 benchmark. But these portfolios failed tomatch the target profiles either in-sample or out-of-sample.
The central objective here, namely to achieve, by means of a weighted portfolio of
S&P 500 stocks, a desired performance profile that also holds on out-of-sample data,is certainly not met in either case.

34 / 37

“Beating the market” and backtest overfitting
I Overfitting and erratic performance are unavoidable in this or any scheme thatamounts to searching over a large set of strategies or fund weightings, and only
implementing or reporting the final optimal scheme.
I The same difficulty afflicts many other attempts to construct an investment

strategy based solely on daily, weekly, monthly or yearly historical market data,such as with charts (as is often done by technical analysts) or tracking a particularrisk profile, as many smart beta ETFs attempt.
I By and large, any underlying actionable information that might exist in such data

has long been mined by highly sophisticated computerized algorithms operated bylarge quantitative funds and other organizations.
I Any lesser efforts, such as those described here, are doomed to be statistically

overfit, and if followed may well have disastrous consequences.

35 / 37

Why the silence in the mathematical finance community?
Historically scientists have exposed those who utilize pseudoscience for commercial
gain. Yet financial mathematicians in the 21st century have remained disappointingly
silent with the regards to those in the community who, knowingly or not:1. Fail to disclose the number of models or variations that were used to develop an
investment strategy or fund.
2. Make vague predictions that do not permit rigorous testing and falsification.
3. Misuse probability theory, statistics and stochastic calculus.
4. Use pseudomathematical charts and jargon: “Fibonacci ratios,” “cycles,” “Elliottwaves,” “golden ratios,” “parabolic SARs,” “technical analysis,” “pivot points,”“symmetrical triangles,” “rising wedges,” etc.
As we wrote in a recent paper:“Our silence is consent, making us accomplices in these abuses.”

36 / 37

Mathematicians against fraudulent financial and investment advice(MAFFIA)
DHB, Jon Borwein, Marcos Lopez de Prado and Jim Zhu started a website and blog
devoted to financial mathematics and abuses of mathematics in the field. Samples:
I “Charts and technical analysis: Do they work?”: http://mathinvestor.org/charts-and-technical-analysis-do-they-work
I “Which hedge funds actually beat the market?”: http://mathinvestor.org/which-hedge-funds-actually-beat-the-market
I “How accurate are market forecasters?”:
http://mathinvestor.org/how-accurate-are-market-forecasters
I “Backtest overfitting in smart beta investments”: http://mathinvestor.org/backtest-overfitting-in-smart-beta-investments
Thanks! Visit our website at:
http://www.mathinvestor.org

This talk is available at: http://www.davidhbailey.com/dhbtalks/dhb-jbcc-fin.pdf

37 / 37