Backtest overfitting in stock fund design and market prediction
David H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and University of California, Davis
Collaborators: Jonathan M. Borwein (Univ. of Newcastle, Australia, deceased 2016),
Marcos Lopez de Prado (Guggenheim Partners and LBNL), Amir Salehipour (Univ. ofNewcastle, Australia) and Jim Zhu (Western Michigan Univ.)

October 9, 2017

1 / 45

Reproducibility crises in biomedicine, psychology, economics
I In 2011, Bayer researchers reported that they wereable to reproduce only 17 of 67 pharma studies.
I In 2012, Amgen researchers reported that they wereable to reproduce only 6 of 53 cancer studies.
I In August 2015, the Reproduciblity Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.

Reproducibility Project staff
Credit: NY Times

2 / 45

A credibility crisis in finance?
I Many individual investors believe that the financial

system (high-frequency trading, “dark pools,” etc.) isrigged against them.
I Many are skeptical of the claims of numerous financial

gurus and forecasters.
I Many are skeptical of the hundreds of new investment

funds and strategies that are marketed each year.
I Financial news is replete with pseudomathematicalcharts and jargon: “Fibonacci ratios,” “cycles,” “Elliottwaves,” “golden ratios,” “parabolic SARs,” “technicalanalysis,” “pivot points,” “symmetrical triangles,”“rising wedges,” etc.
What should the mathematical finance community do? First and foremost, ensure that
our own published research and strategies are mathematically and statistically sound.
3 / 45

Email from a mathematical colleague to DHB, 5 May 2015
You have written about economics and risk assessment and so I’d like to know if you have any
ideas about protecting personal wealth.
I thought of you while reading Janet Tavakoli’s Decisions: Life and Death on Wall Street. Have
you read it? I turned to the book after noting it was [promoted] by Nomi Prins, another WallStreet ex-exec like Tavakoli who’s been spilling the beans about Wall Street shenanigans.
Economists like Simon Johnson, Anat Admati and Joseph Stiglitz have been writing similar
stories from a broader theoretical perspective, but all-in-all, all five (and they are hardly alone)
describe a rigged game.
So what to do about it at the personal level? This comes down to wondering about specificthings like savings accounts, CDs, stocks, bonds and annuities, life insurance andhome-ownership vs renting.

4 / 45

Email from DHB to a financial colleague, 10 June 2013
One thing that has always puzzled me about the financial world is the following sort of thing:[examples cited]. Excuse me for being “dumb,” but this sort of thing seems to me to beoutright nonsense. ...
After all, the stock market, by definition, contains the consensus of all available information,
including the tens of thousands of stock market analysts and economists worldwide who scour
every morsel of information in the business world, and then advise the leading mutual funds
and pension funds. ...
In addition, ... there are thousands more very bright mathematicians using program-trading
schemes, plying every trick of time series analysis, machine learning, stealth and anti-stealththat money can buy, to wriggle every conceivable angle out of the market and beat theircompetitors to the punch with trades. ...
So when people like those above say that they “know” where the stock market is heading, ...
or that by following their strategies, John Q Public can enjoy reliable, above-market returns,
this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?
5 / 45

Response from colleague to DHB, 17 June 2013
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. ...
I completely agree with your assessment. The amount of nonsense ... is incredible.
The good news is, the quants are silently taking over Wall Street, thanks to high frequency andbig data. For the same reason that alchemists and astrologers fought the chemists andastronomers, the market wizards are fighting the quants. So all this ... nonsense is in part the
tug of that war. An attempt of the wizards to squeeze out a few more dimes.

6 / 45Mathematicians against fraudulent financial and investment advice(MAFFIA)In 2013, myself, Jonathan Borwein, Marcos Lopez de Prado and Jim Zhu formed“MAFFIA,” with the goal of doing research in financial mathematics, and, inparticular, to highlight the abuses of mathematics in the financial field:
I We wrote the paper Pseudo-mathematics and financial charlatanism: The effectsof backtest overfitting on out-of-sample performance.
I We wrote numerous additional papers and studies, continuing to the present day.
I We started the Mathematical Investor blog, with new articles posted roughly every
3–4 weeks: http://www.mathinvestor.org.
I With help of some students, we developed online demonstrations of backtest
overfitting, one of the chief abuses in the field.
The remainder of the talk discusses some of our results.
I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and J. Zhu, “Pseudo-mathematics and financial

charlatanism: The effects of backtest overfitting on out-of-sample performance,” Notices of theAmerican Mathematical Society, May 2014, 458–471.

7 / 45

What is backtest overfitting?
4

I Proposing a model for a dataset that inherently possesses a3higher level of complexity than the historical data.
2

I Using a computer to try millions or billions of variations of amodel or strategy on the historical data, and then only
presenting results from the variation that works best.
I Constructing an exchange-traded fund by exploring millions

or billions of weighting factors, then only marketing the one
with the highest backtest score.

1

1

2

3

4

5

Fitting six data points(almost perfectly!) with afourth-degree function.

When a computer can analyze millions or billions of variations of a fund or strategy on
a fixed backtest dataset, it is almost certain that the optimal fund or strategy will bebadly overfit and thus of dubious value.

8 / 45

How easy is it to overfit a backtest? Very!
I If only 2 years of daily backtest data are available, then no more than 7 strategy

variations should be tried.
I If only 5 years of daily backtest data are available, then no more than 45 strategy

variations should be tried.
A backtest that does not report the number of trials N makes it impossible to assess
the risk of overfitting.

MinBTL ≈




 !2
(1 − γ)Z −1 1 − N1 + γZ −1 1 − N1 e −1
E [maxN ]I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and

J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest overfitting on
out-of-sample performance,” Notices of the American
Mathematical Society, May 2014, 458–471.
9 / 45

Letters to clients: An absurd investment scheme
I A financial advisor sends letters to 5, 120 = 5 × 210

prospective clients, with 2560 predicting a certain stock will
go up, and 2560 predicting it will go down.
I One month later, the advisor sends letters only to the 2560

investors who were previously sent the correct prediction,with 1280 letters predicting a certain stock will go up, and1280 predicting it will go down.
I After ten months, the final five investors will have been sent

ten consecutive spot-on predictions!
This strategy is absurd, even fraudulent, because the final five investors are not told of
the 10,235 other letters with different predictions.
But why is promoting a statistically overfit strategy, where potential investors are notinformed of the millions of failed computer trials behind the strategy, any different?
10 / 45

A not-so-absurd investment strategy
Suppose an investor believes that there are daily, weekly ormonthly patterns in stock market data, and she seeks to
exploit them. Sample strategies:
I Basic strategy: Buy a set of stocks each Monday, thensell on Wednesday; buy on the 6th of the month, thensell on the 19th; sell in May and go away, etc.
I Refinements: Sell the portfolio if it drops more thanApple stock price

10% from start; purchase shares only when they
increase in value more than 10% from start; etc.

31 Aug 2014 – 31 Aug 2015

Even with these very simple strategies, there are literally millions of variations (bychanging various parameters), which can be quickly explored by computer.
Selecting only the best combination of parameters (and not mentioning the manyothers that were tried) is a classic selection bias statistical error.
11 / 45

Backtest overfitting: An interactive example
An online demonstration is backtest overfitting is now available:
I The user can select either pseudorandom data or real S&P500 historical data.
I The program then runs a simple monthly-cycle strategy with parameters (day in,holding period, stop-loss percentage, side, etc.), adjusting the parameters to findan optimal strategy.
I The final optimal strategy is then tried on a new (out-of-sample) dataset.
I This software is now available in an online demo (try it yourself!):

http://www.mathinvestor.org

I Credits: Stephanie Ger, Marcos Lopez de Prado, Amir Salehipour, Alex Sim and Kesheng Wu.

12 / 45

Initial strategy on input data (S&P500, 1960–1980): Sharpe ratio = -0.23

13 / 45

Improved strategy on input data: Sharpe ratio = 0.73

14 / 45

Final (optimal) strategy on input data: Sharpe ratio = 1.04

15 / 45

Final strategy on new data (S&P500, 1980–2013): Sharpe ratio = 0.07

16 / 45

Analysis
I After exhaustively exploring the space of strategy variations, the computer

program found a strategy that achieved a Sharpe ratio of 1.04 on the input(backtest) pseudorandom time series (i.e., 1.04 standard deviations above zero).
I However, this optimal strategy, when applied to a new (pseudorandom) timeseries, failed miserably — the Sharpe ratio was 0.07 (i.e., no significant gain).
I In other words, the “optimal” strategy found by the computer search only fit

idiosyncrasies of the input (backtest) dataset — it has no fundamental“intelligence” whatsoever.
For additional analysis (aimed at a fairly elementary audience), see:
I D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting and backtest

performance,” manuscript, 07 Oct 2014, http://ssrn.com/abstract=2507040.

The software demo program is NOW AVAILABLE online:
http://www.mathinvestor.org.
17 / 45

Additional details on backtest overfitting
I Presents formulas relating size of dataset to likelihood of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “Pseudo-mathematics
and financial charlatanism: The effects of backtest over fitting on out-of-sample
performance,” Notices of the American Mathematical Society, May 2014, pg. 458–471.
I Presents formulas for calculating the probability of backtest overfitting:

D. H. Bailey, J. M. Borwein, M. Lopez de Prado and Q. J. Zhu, “The probability ofbacktest overfitting,” Journal of Computational Finance, vol. 20, no. 4 (Apr 2017), pg.
39-69.
I Introduces backtest overfitting for a general audience:

D. H. Bailey, S. Ger, M. Lopez de Prado, A. Sim and K. Wu, “Statistical overfitting andbacktest performance,” manuscript, 2014.
I Defines a “deflated Sharpe ratio,” correcting for some forms of distortion:

D. H. Bailey and M. Lopez de Prado, “The deflated Sharpe ratio: Correcting for selection
bias, backtest overfitting and non-normality,” Journal of Portfolio Management, vol. 40(2014), pg. 94-107.

Preprint copies are available at: http://www.davidhbailey.com/dhbpapers
18 / 45

Proliferation of new stock funds
I Roughly USD$2.1 trillion is held in U.S.-listed exchange-traded funds (ETFs),

with hundreds of new ETFs minted each year.
I In a 2012 study, researchers found that the median time between the definition ofa new index and the inception of a new ETF based on the index dropped fromalmost three years in 2000 to only 77 days in 2011.
I As a result, the report concludes, “most indexes have little live performance
history for investors to assess in the context of a new ETF investment.”
I Out of 370 new indexes, 87% of the indexes outperformed the broad U.S. stock
market over the time period used for the backtest, but only 51% outperformed
the broad market after inception of the index.
I The study found an average 12.25% annualized excess return above the broad
U.S. stock market for a five-year backtest, but -0.26% excess return in the five
years following the inception of the index.

I Vanguard Research, “Joined at the hip: ETF and index development,” July 2012,

http://www.vanguard.com/pdf/s319.pdf.

19 / 45

How difficult is it to design a stock portfolio to achieve a desired
performance profile?
I Given some desired performance profile (a time series), we construct a weighted

subset of S&P500 stocks whose performance matches, as closely as possible, thatof the profile over the specified backtest time period.
I The design minimizes the sum of squares deviation of the weighted portfolio timeseries from the given profile time series.
I In a typical run, some of the resulting weights are negative, corresponding toshorted positions in certain stocks. This potentially exposes the portfolio to losses.
I As an alternate option, the weights are calculated subject to the constraint that
all weights must be greater than or equal to zero.

I D. H. Bailey, J. M. Borwein and M. Lopez de Prado, “Stock portfolio design and backtest

overfitting,” Journal of Investment Management, vol. 17 (2017), no. 1, preprint available athttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739335.
20 / 45

Constructing a weighted portfolio to achieve a desired performance profile
Given a target time series (vj ) and a collection of m stocks (zi ), each with a time series(zi (tj )), we wish to find m weights (wi ) that minimize the objective function
!2
n
m
X
X
R(w1 , w2 , · · · , wm ) =wi zi (tj ) − vj .
j=1

i=1

Pm

Since i=1 wi zi (tj ) is the weighted portfolio time series, this expression is the sum-of-squares
deviation of the weighted portfolio from the target time series. The function R is minimized
when the following are satisfied:
!
n
m
X
X
∂R
=2
wi zi (tj ) − vj z1 (tj ) = 0,
∂w1
j=1
i=1
!
n
m
X
X
∂R
=2
wi zi (tj ) − vj z2 (tj ) = 0,
∂w2
j=1

i=1

···
n
m
X
X
∂R
wi zi (tj ) − vj
=2
∂wm
j=1

i=1

!
zm (tj ) = 0.
21 / 45

Constructing a weighted portfolio (continued)
This can be rewritten as
m
X
i=1
m
X

wi

wi

n
X
j=1
n
X

i=1

j=1

m
X

n
X

zi (tj )z1 (tj ) =zi (tj )z2 (tj ) =

n
X
j=1
n
X

vj z1 (tj ),vj z2 (tj ),

j=1

···

i=1

wi

j=1

zi (tj )zm (tj ) =nX

vj zm (tj ),j=1

which can be solved for the W vector by using conventional linear system solver software.
Note that it is not essential that n > m; if n < m this scheme produces a best least-squares fit
to the target profile, although the quality of this fit degrades when the ratio n/m falls much
below one.
22 / 45

Constructing an all-positive-weight portfolio
When the technique described above is implemented on real stock market data, some of theresulting weights wi are typically negative (so that the corresponding stocks are shorted). Thisis fine, but entails some risk of catastrophic decline (see examples below).
So as an alternative option, one can also ask for an optimal set of weights W subject to theconstraint that each weight wi ≥ 0.
To that end, we have employed a logarithmic barrier scheme, which is to append a logarithmic
term to the minimization problem, as follows:
R(w1 , w2 , · · · , wm ) =

n
m
X
X
j=1

i=1

!2
wi zi (tj ) − vj

+ 2C

m
X

log wi .

i=1

The presence of this logarithmic term penalizes very small weights and thus serves as a barrier,keeping the weights away from zero or negative values. This is not the same as solving theconstrained problem, but by successively reducing the constant C , the desired limiting solution
can be obtained.
23 / 45

Constructing an all-positive-weight portfolio (continued)
In this case, the equivalent minimizing condition is
m
X
i=1
m
X

wi

wi

n
X
j=1
n
X

i=1

j=1

m
X

n
X

zi (tj )z1 (tj ) =zi (tj )z2 (tj ) =

n
X
j=1
n
X

vj z1 (tj ) + C /w1 ,vj z2 (tj ) + C /w2 ,

j=1

···

i=1

wi

j=1

zi (tj )zm (tj ) =nX

vj zm (tj ) + C /wm .

j=1

This system can be efficiently solved by Newton iterations, where one takes, as starting
estimates of the weights W , the solution to the unconstrained problem above, replacing zero or
negative weights with some very small positive value.
24 / 45

Constructing an all-positive-weight portfolio (continued)
In summary, the algorithm for the constrained problem is the following:
1. Perform the unconstrained matrix calculation to obtain an initial set of weights W .
2. Replace zero or negative weights with a small positive value (we use 10−8 ).
3. Select C = 1, then perform the Newton iteration until convergence (typically inten or fewer iterations).
4. Reduce C by a factor of ten and repeat step (3), continuing until overall
convergence (typically when C = 10−6 or so).

25 / 45

Constructing portfolios from real stock data
Our computer program constructed stock portfolios based exclusively on real S&P 500
historical stock data. Data for S&P 500 stocks are easy to obtain online. For example,
Apple Computer’s daily stock closings going back to 1980 can be downloaded from:https://finance.yahoo.com/q/hp?s=AAPL
http://www.google.com/finance?q=AAPLThe in-sample period was 1991–2005; the out-of-sample period was 2006–2015.
Our program found 277 valid stocks from the S&P 500 database for which dataspanning this time period was available. All stock data used here include reinvested
dividends.

26 / 45

Three types of performance profiles
Using our program, one can generate any of three target profiles (here p is an annual
percentage rate):
1. Steady capital growth: A steady increase by the fraction (1 + p/(100r )) per time
period (i.e., growing by p/r percent each time period, where r is the number oftime periods per year; e.g., r = 12).
2. Stair-step growth: A stair-step function that is constant, except that at the end ofeach q-year period it increases by the fraction (1 + p/(100r ))qr (i.e., at the end ofeach q-year period, it increases by a full q years’ growth of Profile 1 above). We
took q = 1 in the examples below.
3. Sinusoidal growth: A sinusoidal function that increases by the fraction(1 + p/(100r )) per time period, as in profile #1, but is multiplied by a sine wavethat varies from 1/2 to 3/2, with period q years. We took q = 5.
The second and third profiles are included mainly to illustrate that any reasonable
function whatsoever may be specified for the profile.
27 / 45

Results: Steady growth profile, APR = 6%

50 00050 00040 00040 00030 00030 000

20 000

20 000

10 000

10 000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.
28 / 45

Steady growth profile, APR = 8%50 00050 00040 00040 00030 00030 000

20 000

20 000

10 000

10 000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

29 / 45

Steady growth profile, APR = 10%

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

30 / 45

Steady growth profile, APR = 12%30 00030 00025 00025 000

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

31 / 45

Sinusoidal profile, APR = 10%

20 000

20 000

15 000

15 000

10 000

10 000

5000

5000

1995

2000

2005

2010

2015

1995

2000

2005

2010

2015

Standard portfolio (L) and all-positive portfolio (R). Blue: portfolio; orange: target
profile; green: S&P500.

32 / 45

Summary of 20 runs

ProfileSteadygrowthStairstep
Sinusoidal

Fig.
1
2
3
4
5
6
7
8
9
10

APR
6%
8%
10%
12%
15%
18%
8%
10%
8%
10%

Standard weights
RMS dev.
Sharpe ratio
IS
OOS
IS
OOS
0.000 7.658 -0.120 0.168
0.000 2.534 -0.079
FAIL
0.000 0.996 -0.038
FAIL
0.000 1.178
0.003
FAIL
0.000 5.953
0.065 0.178
0.000 0.996
0.126
FAIL
0.000 9.395 -0.066 0.167
0.000 0.996 -0.024
FAIL
0.000 4.518 -0.064
FAIL
0.000 0.996 -0.029
FAIL

All-positive weights
RMS dev.
Sharpe ratio
IS
OOS
IS
OOS
1.426 1.910 0.163 -0.025
1.016 0.970 0.162 -0.025
0.695 0.391 0.161 -0.026
0.452 0.276 0.157 -0.027
0.223 0.557 0.145 -0.016
0.218 0.711 0.177 -0.021
1.086 1.039 0.162 -0.025
0.768 0.442 0.161 -0.025
1.584 1.528 0.162 -0.024
1.267 0.867 0.158 -0.024“APR”: annual percentage rate; “IS”: in-sample period, 1991–2005 (15 years); “OOS”: out-of-sample
period, 2006–2015 (10 years); “RMS dev.”: root-mean-square deviation from target profile; “Sharpe
ratio”: Sharpe ratio relative to S&P 500 with reinvested dividends; “FAIL”: 100% loss of capital.
33 / 45

Analysis of stock fund study
I In every case, the standard portfolio performance achieved zero deviation over thein-sample period. Only beginning with 2006 (the out-of-sample period) do theblue curves depart from the orange curves in the plots.
I In some cases the standard portfolios did remarkably well, but in other cases theyfailed catastrophically.
I The positive-weight portfolios are significantly less erratic and often outperformed

both the target profile and the S&P 500 benchmark. But these portfolios failed tomatch the target profiles either in-sample or out-of-sample.
The central objective here, namely to achieve, by means of a weighted portfolio of
S&P 500 stocks, a desired performance profile that also holds on out-of-sample data,is certainly not met in either case.

34 / 45

“Beating the market” and backtest overfitting
I Overfitting and erratic performance are unavoidable in this or any scheme thatamounts to searching over a large set of strategies or fund weightings, and only
implementing or reporting the final optimal scheme.
I The same difficulty afflicts many other attempts to construct an investment

strategy based solely on daily, weekly, monthly or yearly historical market data,such as with charts (as is often done by technical analysts) or tracking a particularrisk profile, as many smart beta ETFs attempt.
I By and large, any underlying actionable information that might exist in such data

has long been mined by highly sophisticated computerized algorithms operated bylarge quantitative funds and other organizations.
I Any lesser efforts, such as those described here, are doomed to be statistically

overfit, and if followed may well have disastrous consequences.

35 / 45

Can “gurus” forecast the market?
Motivation:
I Many investors rely on market experts and forecasters when making investment

decisions.
I Ranking and grading market forecasters provides investors with metrics on which

they may choose forecasters with the best record of accuracy.
Aim of our study: to a rank prominent market forecasters using a novel methodology:
I We distinguish forecasts by their time frame, and specificity, rather thanconsidering all forecasts equally important.
I We analyze the impact of the number of forecasts made by a particular forecaster.

36 / 45

Kaissar’s analysis of forecasters
When analyzing a set of strategists’ predictions from 1999 to 2016, Kaissar found that
forecasts were surprisingly unreliable during major inflection points:
I The strategists overestimated the S&P 500’s year-end price by 26.2% on averageduring the three recession years 2000 to 2002.
I They underestimated the index’s level by 10.6% for the initial recovery year 2003.
I They overestimated the S&P 500’s year-end level by a whopping 64.3% in 2008,

but then underestimated the index by 10.9% for the first half of 2009.
Kaissar’s conclusion: The forecasts were least useful when they mattered most.
I N. Kaissar, “S&P 500 forecasts: Crystal ball or magic 8?,” Bloomberg News, 23 December 2016,

https://www.bloomberg.com/gadfly/articles/2016-12-23/
s-p-500-forecasts-mostly-hit-mark-until-they-matter-most.

37 / 45

Our analysis of market forecasters
In 2012, the CXO Advisory Group ranked 68 forecasters based on their 6,582 forecasts(forecasts made for the S&P 500 index). We extended and advanced this study as
follows:
I Each individual forecast is treated according to two factors: (a) time frame and(b) specificity.
I Long-term forecasts are treated as more significant than the short-term forecasts(because in the long-term underlying trends, if any, tend to overcome short-term
noise; wt ∈ {0.25, 0.50, 0.75, 1.00} (for no time frame, forecasts up to one month,
up to three months, beyond nine months, respectively).
I Specific forecasts are treated more important than non-specific ones
(ws ∈ {0.50, 1.00}).
For details:
I item D. H. Bailey, J. M. Borwein, A. Salehipour, and M. Lopez de Prado, “Evaluation andranking of market forecasters”, Journal of Investment Management, to appear,https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2944853.

The following few viewgraphs are adapted from Amir Salehipour.
38 / 45

Forecaster score
Combined weight for a forecast:wi+ = wt × ws
wi− = wt × ws

if forecast i is correct
if forecast i is not correctwhere wi+ denotes a true forecast is true, and wi− denotes a false one.
Then the score (accuracy) of a forecaster is:
n

j
Σi=1
wi+
j = njnj
Σi=1 wi+ + Σi=1
wi−

where j is the forecaster’s index, and nj is the total number of forecasts made byforecaster j.
39 / 45

Abby Joseph Cohen
Aden Sisters
Ben Zacks
Bernie Schaeffer
Bill Cara
Bill Fleckenstein
Bob Brinker
Bob Doll
Bob Hoye
Cabot Market Letter
Carl Futia
Carl Swenlin
Charles Biderman
Clif Droke
Comstock Partners
Curt Hesler
Dan Sullivan
David Dreman
David Nassar
Dennis Slothower
Don Hays
Don Luskin
Donald RoweDoug Kass
Gary D. Halbert
Gary Kaltbaum
Gary Savage
Gary Shilling
Igor Greenwald
Jack Schannep
James Dines
James Oberweis
James Stewart
Jason Kelly
Jeremy Grantham
Jim Cramer
Jim Jubak
Jim Puplava
John Buckingham
John Mauldin
Jon Markman
Ken Fisher
Laszlo Birinyi
Linda Schurman
Louis Navellier
Marc Faber
Mark Arbeter
Martin Goldberg
Mike Paulenoff
Nadeem Walayat
Paul Tracy
Peter Eliades
Price Headley
Richard Band
Richard Moroney
Richard Rhodes
Richard Russell
Robert Drach
Robert McHughRobert Prechter
S&P Outlook
Stephen Leeb
Steve Saville
Steve Sjuggerud
Steven Jon Kaplan
Tim Wood
Tobin Smith
Trading Wire

Results: forecaster accuracy

80

70
Total accuracy versus accuracy per forecast and forecast share

60
6

50
5

% 40
30
4

20
3

10

0
2

1

0

Accuracy (This study)
Accuracy per forecast
Forecast share (no. of forecasts/total forecasts)

40 / 45

% of specific forecasts

Richard Moroney

Richard Band

Price HeadleyPeter Eliades

Paul Tracy

Nadeem Walayat

Mike Paulenoff

Martin Goldberg

Mark Arbeter

Marc Faber

Louis Navellier

Linda Schurman

Laszlo Birinyi

Ken Fisher

Jon Markman

John Mauldin

John Buckingham

Jim Puplava

Jim Jubak

Jim Cramer

Jeremy Grantham

Jason Kelly

James Stewart

James Oberweis

James Dines

Jack Schannep

Igor Greenwald

Gary Shilling

Gary Savage

Gary Kaltbaum

Gary D. Halbert

Doug Kass

Donald Rowe

Don Luskin

Don Hays

Dennis Slothower

David Nassar

David Dreman

Dan Sullivan

Curt Hesler

Comstock Partners

Clif Droke

Charles Biderman

Carl Swenlin

Carl Futia

Cabot Market Letter

Bob Hoye

Bob Doll

Bob Brinker

Bill FleckensteinBill Cara

Bernie Schaeffer

Ben Zacks

Aden Sisters

Abby Joseph Cohen

Tim Wood
Tobin Smith
Trading Wire

Tobin Smith

Tim Wood
Trading Wire

Steve Sjuggerud
Steven Jon Kaplan

Steve Sjuggerud

Steve Saville
Steven Jon Kaplan

S&P Outlook
Stephen Leeb

S&P Outlook
Steve SavilleRobert Prechter
Stephen Leeb

Robert Drach
Robert McHughRobert Prechter

Robert Drach
Robert McHugh

Richard Russell

Percentage of specific forecasts versus non-specific forecasts

Richard Russell

100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%

Richard Rhodes% of forecasts with weight 0.25

Richard Rhodes

Richard Moroney

Richard Band

Price HeadleyPeter Eliades

Paul Tracy

Nadeem Walayat

Mike Paulenoff

Martin Goldberg

Mark Arbeter% of forecasts with weight 0.50

Marc Faber

Louis Navellier

Linda Schurman

Laszlo Birinyi

Ken Fisher

Jon Markman

John Mauldin

John Buckingham

Jim Puplava

Jim Jubak

Jim Cramer% of forecasts with weight 0.75

Jeremy Grantham

Jason Kelly

James Stewart

James Oberweis

James Dines

Jack Schannep

Igor Greenwald

Gary Shilling

Gary Savage

Gary Kaltbaum

Gary D. Halbert% of forecasts with weight 1.00

Doug Kass

Donald Rowe

Don Luskin

Don Hays

Dennis Slothower

David Nassar

David Dreman

Dan Sullivan

Curt Hesler

Comstock Partners

Clif Droke

Charles Biderman

Carl Swenlin

Carl Futia

Cabot Market Letter

Bob Hoye

Bob Doll

Bob Brinker

Bill FleckensteinBill Cara

Bernie Schaeffer

Ben Zacks

Aden Sisters

Abby Joseph Cohen

Results: forecaster time frame and specificity
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Percentage of forecasts time frames% of non-specific forecasts

41 / 45

Accuracy (This study)

80
70
60
50
40
30
20
10
0
100

80

60

40

20

0

Percentage of forecasts

Robert Prechter

Mike Paulenoff

Robert McHugh

Bob Hoye

Comstock Partners

Bill Fleckenstein

Jim Puplava

Gary Shilling

Nadeem Walayat

John Mauldin

Richard Russell

Jim Jubak

Percentage of forecasts

60
50
40
30
20
10
0

Marc Faber

Steve Saville

Steven Jon Kaplan

Curt Hesler

Abby Joseph Cohen

Jim Cramer

Jeremy Grantham

Gary D. Halbert

Clif Droke

Stephen Leeb

Paul Tracy

James Oberweis

Ken Fisher

Accuracy
60
50
40
30
20
10
0

Gary Savage

Doug Kass

Price Headley

Donald Rowe

Richard Band

Linda Schurman

Bernie Schaeffer

Tim WoodBill Cara

Dennis Slothower

Charles Biderman

James Dines

Martin Goldberg

Jon Markman

Peter Eliades

Accuracy

Accuracy (This study)

Bob Brinker

S&P Outlook

Trading Wire

James Stewart

Don Hays

Carl Futia

Richard Rhodes

Mark ArbeterCarl Swenlin

Tobin Smith

Igor Greenwald

Richard Moroney

Gary Kaltbaum

Ben Zacks

Don Luskin

Aden Sisters

Dan Sullivan

Bob Doll

Jason Kelly

Robert Drach

Steve Sjuggerud

Laszlo Birinyi

Louis Navellier

Cabot Market LetterDavid Dreman

David Nassar

Jack Schannep

John Buckingham

Results: traders versus investors
Accuracy of forecasters in the investor group

Percentage of forecasts with weights > 0.5

Accuracy of forecasters in the trader group

Percentage of forecasts with weights <= 0.5

42 / 45

Summary of forecaster study
I Across all forecasts, the accuracy is around 48%.
I Two-thirds of forecasts predict as far as only a month.
I Only one-third of forecasts predict periods over one month.
I Two-thirds of forecasters have an accuracy level below 50%.
I Only about 6% of forecasters have their accuracy values between 70% and 79%

the highest accuracy value is still below 80%.
The majority of forecasters perform at levels not significantly different than chance,
which makes it very difficult to tell if there is any skill present.
For details:
I item D. H. Bailey, J. M. Borwein, A. Salehipour, and M. Lopez de Prado, “Evaluation andranking of market forecasters”, Journal of Investment Management, to appear,https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2944853.
43 / 45

Why the silence in the mathematical finance community?
Historically scientists have exposed those who utilize pseudoscience for commercial
gain. Yet financial mathematicians in the 21st century have remained disappointingly
silent with the regards to those in the community who, knowingly or not:I Fail to disclose the number of models or variations that were used to develop an

investment strategy or fund.
I Promote products or strategies based on statistically overfit backtests.
I Make vague predictions that do not permit rigorous testing and falsification.
I Misuse probability theory, statistics and stochastic calculus.
I Use pseudomathematical charts and jargon: “Fibonacci ratios,” “cycles,” “Elliottwaves,” “golden ratios,” “parabolic SARs,” “technical analysis,” “pivot points,”“symmetrical triangles,” “rising wedges,” etc.
As we wrote in a recent paper:“Our silence is consent, making us accomplices in these abuses.”
44 / 45

Recent MAFFIA blogs
I “Charts and technical analysis: Do they work?”:

http://mathinvestor.org/charts-and-technical-analysis-do-they-work
I “How difficult is it to design a stock fund based on backtests?”: http://mathinvestor.

org/how-difficult-is-it-to-design-a-stock-fund-based-on-backtests
I “How accurate are market forecasters?”:

http://mathinvestor.org/how-accurate-are-market-forecasters
I “Backtest overfitting in smart beta investments”:

http://mathinvestor.org/backtest-overfitting-in-smart-beta-investments

Thanks! Visit our website at:
http://www.mathinvestor.org

This talk is available at: http://www.davidhbailey.com/dhbtalks/dhb-risk-2017.pdf
45 / 45