Why financial research is prone to false statistical discoveriesDavid H. Bailey
https://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and University of California, Davis
Marcos López de Prado
https://www.quantresearch.org
Cornell University and Abu Dhabi Investment Authority
This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-risk-2022.pdf

January 25, 2022
1 / 24

Why financial research is prone to false statistical discoveries
I The chances of finding a truly profitable investment
design or strategy is very low, due to intensecompetition.
I True findings are mostly short-lived, as a result of thenon-stationary nature of most financial systems.
I Since it is rarely possible to rigorously test statisticalfindings with controlled experiments, it is often difficult
to debunk a false claim.
One would hope that financial researchers would be particularly careful when
conducting statistical inference. Sadly, the opposite appears more accurate.
I D. H. Bailey and M. Lopez de Prado, “How backtest overfitting in finance leads to false discoveries,”
Significance, Royal Statistical Society, Jan 2022, condensed fromhttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=3895330.
2 / 24

Reproducibility crises in biomedicine, psychology and economics
I In 2011, Bayer researchers reported that they wereable to reproduce only 17 of 67 pharma studies.
I In 2012, Amgen researchers reported that they wereable to reproduce only 6 of 53 cancer studies.
I In August 2015, the Reproduciblity Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.
I In an updated 2018 study, the Reproducibility
Project was able to replicate only 14 out of 28
psychology experimental studies.

Reproducibility Project staff
Credit: NY Times

3 / 24

Reproducibility crisis in high-performance computing
I In 1990, highly parallel computer arrays emerged forhigh-performance scientific computing.
I Some in the research community hyped the newtechnology with inflated claims of performance.
I I personally published two articles warning of such
distortions, but such warnings were mostly ignored.
I The field’s credibility declined; dozens of parallel
Japan’s Fugaku supercomputer
computer firms went bankrupt. At least one CEO
personally blamed me for his firm’s demise.
I After improved benchmarks and performance standards,credibility was restored. Now virtually all scientificsupercomputers feature a massively parallel design.
Lesson: If a field has sloppy standards, with a public perception that does not match
reality, it usually ends badly.
4 / 24

Is there a similar crisis in finance?
I Only a small fraction of investment funds beat thecorresponding market averages over a 10-year window.
I Only a small fraction of financial forecasters do better
than chance when their records are checked.
I Financial news is replete with pseudomathematicaljargon: “Fibonacci ratios,” “cycles,” “waves,” “goldenratios,” “parabolic SARs,” “pivot points,” “risingwedges,” “shoulders,” etc.
I Many analysts (and brokerages) recommend discredited
statistical methods (e.g., “technical analysis”).
I Even in academic literature, few authors disclose the full
extent of computer searches used in developing ortuning a model, and few if any journals require authors
to disclose such information.

What should responsible
finance researchers do?
Ensure that their own
published research ismathematically andstatistically sound.
5 / 24

Email exchange between DHB and a finance colleague
Email from DHB to finance colleague, 10 June 2013:
One thing that has always puzzled me about the financial world is the following sort of thing:[several specific examples cited]. Excuse me for being “dumb,” but this sort of thing seems to
me to be outright nonsense. ... [For example,] when people like those above say that they“know” where the stock market is heading, this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?
Response from finance colleague to DHB, 17 June 2013:
It is not a dumb question at all. ... I completely agree with your assessment. The amount ofnonsense ... is incredible.

6 / 24

Backtest overfitting: Statistical overfitting of historical market data
Backtest overfitting is arguably the financial field’s version of“p-hacking,” namely the regrettable practice of only citing
results from a subset of data or tests. Examples:
I Employing a theoretical model that inherently possesses ahigher level of complexity than the backtest data.
I Using a computer to try millions or billions of variations of amodel or strategy on historical market data, and then only
presenting results from the variation that works best.
I Using a computer to explore millions or billions of weighting
factors for a mutual fund, then only marketing the one thatscores best on a backtest.

43

2

1

1

2

3

4

5

Fitting six data points
almost perfectly with afourth-degree function.

When a computer can analyze many variations of a model, fund or strategy on a fixed
dataset, it is virtually certain that the “optimal” selection will be statistically overfit.
7 / 24

How easy is it to overfit a backtest? Very!
I If only 2 years of daily market data are available,
then no more than 7 variations should be tried.
I If only 5 years of daily market data are available,
then no more than 45 variations should be tried.

I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and J. Zhu, “Pseudo-mathematics and financialcharlatanism: The effects of backtest overfitting on out-of-sample performance,” Notices of the American
Mathematical Society, May 2014, 458–471, https://www.ams.org/notices/201405/rnoti-p458.pdf.

8 / 24

Parameter variations in a simple investment strategy
I Consider a very simple
strategy, wherein one
buys a set of stocks ofstocks on one day of themonth then sells them onanother day.
I Even with this simple
example, there arethousands of choices andparameter variations.
I There are 435 choices
just for the start and end
dates.
Searching by computer over all of the possible parameters for an “optimal” strategy isvirtually certain to produce an overfit result.
9 / 24

Proliferation of new exchange-traded stock funds (ETFs)
I A 2015 study computed the performance of all
ETFs that were launched in the U.S. market from1993 to 2014.
I Researchers found that the investment strategies
underlying those ETFs delivered average annual
excess returns of approx. 5% prior to their launch(i.e., by using backtests).
I After the launch, annual excess returns dropped toapprox. 0%.

Such disappointing behavior is entirely consistent with a design process that involves
extensive computer exploration of parameters and weights, but selecting only the“optimal” set for an index fund subsequently fielded in the market.
I C. Brightman, F. Li and X. Liu, “Chasing performance with ETFs,” Fundamentals, November 2015.
10 / 24

How difficult is it to design a stock portfolio to achieve a desired
performance profile?
I Given some desired performance time series profile (e.g., a steady 1% per monthgrowth), we devised a computer program that constructs a weighted subset ofS&P500 stocks whose performance exactly matches the profile over the specified
backtest time period (1991–2005).
I But most of these portfolios fail miserably when presented with new data
(2006–2015).
These erratic and often catastrophic results on new (out-of-sample) data aresymptomatic of statistical overfitting.

Technical details are given here:
I D. H. Bailey, J. M. Borwein and M. Lopez de Prado, “Stock portfolio design and backtest overfitting,”
Journal of Investment Management, vol. 17 (2017), no. 1,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739335.
11 / 24

Steady growth profile, APR = 8%50 00040 00030 000

20 000

10 000

1995

2000

2005

2010

2015

Blue: constructed portfolio; orange: target profile; green: S&P500.
12 / 24

Steady growth profile, APR = 12%30 00025 000
20 000
15 000
10 000
5000

1995

2000

2005

2010

2015

Blue: constructed portfolio; orange: target profile; green: S&P500.
13 / 24

How many actively managed funds have beaten the market averages?
Despite the proliferation of mutual funds, exchange-traded funds and target date funds
in recent years, the sad fact is that very few beat the corresponding market averages
over a 10-year window:
I Among actively managed “U.S. large value” stock mutual funds, only 8.3% beatthe corresponding passive index fund over the 10-year period 2010–2019.
I Among actively managed “U.S. large growth” stock mutual funds, only 8.3% beatthe corresponding passive index fund over the 10-year period.
I Among actively managed “world stock” mutual funds, only 26.3% beat thecorresponding passive index fund over the 10-year period.

I D. H. Bailey “Mutual fund report card: February 2019,”
https://mathinvestor.org/2019/02/mutual-fund-report-card-february-2019/.
I Morningstar, “Morningstar active/passive barometer,” Feb 2019,https://www.morningstar.com/lp/active-passive-barometer?con=15949&cid=CON_RES0054.
14 / 24

How well have stock market gurus forecasted the market?
Kaissar’s analysis of prominent market forecasters (covering 1999 to 2016):
I The forecasters overestimated the S&P 500’s year-end price by 26.2% on averageduring the three recession years 2000 to 2002.
I They underestimated the index’s level by 10.6% for the initial recovery year 2003.
I They overestimated the S&P 500’s year-end level by a whopping 64.3% in 2008.
I They underestimated the index by 10.9% for the first half of 2009.
Kaissar’s conclusion: “The forecasts were least useful when they mattered most.”

I N. Kaissar, “S&P 500 forecasts: Crystal ball or magic 8?,” Bloomberg News, 23 December 2016,
https://www.bloomberg.com/gadfly/articles/2016-12-23/
s-p-500-forecasts-mostly-hit-mark-until-they-matter-most.

15 / 24

Our analysis of market forecasters
In 2012, the CXO Advisory Group ranked 68 forecasters based on their 6,582 forecasts
for the S&P 500 index.
In 2018, we extended and advanced this study by classifying forecasts according totime frame and specificity.
Results:
I Across all forecasts, the average accuracy was 48%.
I Two-thirds of forecasters had an accuracy level below 50%.
I Only about 6% of forecasters had accuracy values between 70% and 79%; thehighest accuracy value was still below 80%.

For details:
I D. H. Bailey, J. M. Borwein, A. Salehipour, and M. Lopez de Prado, “Evaluation and ranking of marketforecasters”, Journal of Investment Management, vol. 16, no. 2 (Apr 2018), 47–64,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2944853.

16 / 24

Abby Joseph Cohen
Aden Sisters
Ben Zacks
Bernie Schaeffer
Bill Cara
Bill Fleckenstein
Bob Brinker
Bob Doll
Bob Hoye
Cabot Market Letter
Carl Futia
Carl Swenlin
Charles Biderman
Clif Droke
Comstock Partners
Curt Hesler
Dan Sullivan
David Dreman
David Nassar
Dennis Slothower
Don Hays
Don Luskin
Donald RoweDoug Kass
Gary D. Halbert
Gary Kaltbaum
Gary Savage
Gary Shilling
Igor Greenwald
Jack Schannep
James Dines
James Oberweis
James Stewart
Jason Kelly
Jeremy Grantham
Jim Cramer
Jim Jubak
Jim Puplava
John Buckingham
John Mauldin
Jon Markman
Ken Fisher
Laszlo Birinyi
Linda Schurman
Louis Navellier
Marc Faber
Mark Arbeter
Martin Goldberg
Mike Paulenoff
Nadeem Walayat
Paul Tracy
Peter Eliades
Price Headley
Richard Band
Richard Moroney
Richard Rhodes
Richard Russell
Robert Drach
Robert McHughRobert Prechter
S&P Outlook
Stephen Leeb
Steve Saville
Steve Sjuggerud
Steven Jon Kaplan
Tim Wood
Tobin Smith
Trading WireForecaster accuracy

8070
Total accuracy versus accuracy per forecast and forecast share

60
650% 405430203102010

Accuracy (This study)
Accuracy per forecast
Forecast share (no. of forecasts/total forecasts)

17 / 24

Study of anomaly indicators
A 2020 study analyzed the statistical reliability of a large set of anomaly indicators(signals in financial market data that may indicate an investment opportunity) fromrecently published literature.
I The authors concluded that they were not able to statistically replicate most of
the findings that had been reported.
I Of the 452 anomaly indicators studied, 65% did not even clear the single test
threshold of t = 1.96 or greater, when correctly analyzed. With a more stringent
criteria that partially compensates for multiple testing, namely t = 2.78 at the 5%
significance level, the failure rate increases to 82%.

I K. Hou, C. Xue and L. Zhang, “Replicating anomalies,” The Review of Financial Studies, vol. 33 (May2020), 2019–2133, https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3275496.
18 / 24

The false strategy theorem
I An investment analyst may carry out a large number of simulation trials on
historical data, and report only the model, fund or strategy with the maximum
Sharpe ratio.
I However, the distribution of the maximum Sharpe ratio is clearly not the same as
the distribution of a Sharpe ratio randomly chosen among the trials.
I Instead, the expected value of the maximum Sharpe ratio is greater than theexpected value of the Sharpe ratio from a random trial.
I In particular, given an investment strategy with expected Sharpe ratio zero andnon-zero variance, the expected value of the maximum Sharpe ratio steadily
increases, up from zero, as a function of the number of trials.
Conclusion: No Sharpe ratio criterion is sufficient to rule out a false result. Givenenough trials, any preset Sharpe level will eventually be exceeded by random noise.

I M. Lopez de Prado and D. H. Bailey, “The false strategy theorem: A financial application of experimentalmathematics,” American Mathematical Monthly, vol. 128 (2021), no. 9, 825–831,
https://papers.ssrn.com/sol3/papers.cfm?abstract_id=3221798.
19 / 24

The false strategy theorem
Given a sample of K independent trials, one can deduce an expected maximum Sharperatio, namely the hurdle or threshold that the reported Sharpe ratio must exceed
before it can be considered a significant finding:
Let N denote the Gaussian normal distribution; let Z −1 [·] denote the inverse of thestandard Gaussian cumulative distribution function (CDF); let E [·] and V [·] denote
expected value and variance; and let γ is the Euler–Mascheroni constant= 0.5772156649 . . . .
False strategy theorem
noc k , k = 1, · · · , K , with
Given a sample of estimated performance statistics SR
independentanddistributed Gaussian distribution, i.e.,
oh identicallyhnoiinccSR k ∼ N 0, V SR k, thenno  hnoi−1/2
1
1−1−1ckck
E max SRV SR≈ (1 − γ)Z1−+ γZ
1−kK
Ke20 / 24

The false strategy theorem: Monte Carlo analysis
1. We generate a random array of size (S × K ), where S is the number of Monte
Carlo experiments, and K is number of trials among which the highest Sharpe
ratio will be selected. The values in this random array are drawn from a standard
normal distribution centered at zero.
2. Thein this array are centered and scaled to match zero mean andhnrows oicvariance.
V SR knoc k , is computed, resulting in a
3. The maximum value across each row, maxk SR
number S of such maxima.
4. Wehncomputeoi the empirical average value across the S maxima, namelybc
E SR k .
5. We then compute the estimation error, in hn
relativeoiterms,
hnto theoianalyticalc k /E SR
c k − 1.
prediction made by the theorem as  = Eb SR
6. We repeat the previous steps R times, resulting in a set of estimation errors{r }, r = 1, 2, . . . , R, and allowing us to compute the mean and standard
deviation of the estimation errors associated with K trials.

21 / 24

The false strategy theorem: Monte Carlo plot

22 / 24

Why the silence?
Historically scientists have exposed pseudoscience: astrology in the 1800s; young-earth
creationism in the 1900s; global warming denial and anti-vaxx propaganda in the 2000s.
Yet present-day financial researchers have remained disappointingly silent with regards
to those in the finance community who, knowingly or not:I Fail to disclose the number of models that were tried in their study or fund design.
I Withhold key details, such as the extent of computer searches used in the analysis.
I Use pseudomathematical jargon (e.g., “Fibonacci ratios,” “cycles,” “waves,”“golden ratios,” “parabolic SARs,” “pivot points,” “rising wedges”, “shoulders”).
I Use or recommend discredited statistical methods (e.g., “technical analysis”).
I Use or recommend models or strategies that are no longer provably effective.
I Make public claims and forecasts not backed up by solid scientific facts.
I Resist rigorous analysis and assessments of long-term performance.
As we wrote in a recent paper:
Our silence is consent, making us accomplices in these abuses.
23 / 24

Mathematicians Against Fraudulent Financial and Investment Advice(MAFFIA)
Visit our MAFFIA websites at:https://www.maffia.org
https://www.mathinvestor.org

This talk is available at: http://www.davidhbailey.com/dhbtalks/dhb-risk-2022.pdf

24 / 24