Getting financial predictions rightDavid H. Bailey
http://www.davidhbailey.com
Lawrence Berkeley National Lab (retired) and University of California, Davis

1 / 22

DeepMind’s AlphaGo Zero teaches itself to play Go
I In October 2017, DeepMind researchers programmed an AI

computer with the rules of Go, then had the program playitself — literally teaching itself with no human input.
I After just three days of training, the resulting program“AlphaGo Zero” defeated their earlier program (which haddefeated Ke Jie, the highest rated human) 100 games to 0.
I The Elo rating of Ke Jie, the world’s highest rated Goplayer, is 3661. After 40 days of training, AlphaGo Zero’s
Elo rating was over 5000.
I Thus AlphaGo Zero is as far ahead of Ke as Ke is ahead ofa good amateur.
I “The latest AI can work things out without being taught,” The Economist, 21 Oct 2017,https://www.economist.com/news/science-and-technology/
21730391-learning-play-go-only-start-latest-ai-can-work-things-out-without.
2 / 22

Artificial intelligence and big data in financeAccording to a Bloomberg report, Wall Street is entering a new era:
The fraternity of bond jockeys, derivatives mavens and stock pickers who’ve
long personified the industry are giving way to algorithms, and soon, artificialintelligence.
Banks and investment funds have been tinkering for years, prompting anxiety
for employees. Now, firms are rolling out machine-learning software tosuggest bets, set prices and craft hedges. The tools will relieve staff ofroutine tasks and offer an edge to those who stay. But one day, machines
may not need much help.

I S. Kishan, H. Son and M. Rojanasakul, “Robots are coming for these Wall Street jobs,”
https://www.bloomberg.com/graphics/2017-wall-street-robots/.
3 / 22

Artificial intelligence and big data in finance, continued
I Sell side credit markets: Automating subjective human decisions.
I Sell side foreign exchange: Anticipating variations in demand and prices.
I Sell side equities: Optimizing order execution.
I Buy side equities: Timing stock purchases and assessing risk.
I Buy side credit: Understanding bond deals, legal documents and court rulings.
I Buy side macroeconomics: Analyzing satellite images of Chinese industrial sites,

Walmart parking lots, etc., to spot trends in the economy.
The message to the finance world is clear: Prepare to adapt, or face irrelevance.

4 / 22

Challenging times for hedge funds
I As of 1 July 2017, the HFRI Fund Weighted Composite Index is up 3.28%

year-to-date, and 4.79% annualized gain for the previous 5 years.
I The corresponding figures for the S&P500 Index (including dividends) are 9.34%

and 13.6%.
I In 2014, the California Public Employees’ Retirement System (CalPERS)

announced that it would liquidate its USD$4 billion investment in hedge funds.
I In 2016, the New York City public pension fund announced that it will also

liquidate all hedge fund investments.
Yet some hedge funds do consistently beat the market! According to Institutional
Investor’s Alpha:Five of the six largest firms in this year’s ranking rely all or mostly on
computers and algorithms to make their investment decisions, a theme that
has increasingly played a role in the top-100 ranking over the past few years.
And all five posted asset increases last year.
5 / 22DANGER AHEADArtificial intelligence, big data and high-performancecomputing can generate nonsense faster than ever before!

Details to consider any computer-based research and development activity:
I Are the theoretical models realistic?
I Is the computer implementation debugged?
I Are the statistical methods and data analysis techniques appropriate?
I Is there sufficient data to draw statistically reliable conclusions?
I Can other researchers reproduce the findings?

I V. Stodden, M. McNutt, D. H. Bailey, E. Deelman, Y. Gil, B. Hanson, M. A. Heroux, J. P. A. Ioannidisand M. Taufer, “Enhancing reproducibility for computational methods,” Science, vol. 356, no. 3617 (3Dec 2016), 1240–1241.
6 / 22

Reproducibility crises in biomedicine, psychology and economics
I In 2011, Bayer researchers reported that they wereable to reproduce only 17 of 67 pharma studies.
I In 2012, Amgen researchers reported that they wereable to reproduce only 6 of 53 cancer studies.
I In August 2015, the Reproduciblity Project inVirginia reported that they were able to reproduce
only 39 of 100 psychology studies.
I In September 2015, the U.S. Federal Reserve wasable to reproduce only 29 of 67 economics studies.

Reproducibility Project staff
Credit: NY Times

7 / 22

A crisis in finance?
I Many individual investors believe that the financial

system is rigged against them.
I Many are confused about the thousands of investment

funds and services available.
I Many are skeptical of the claims of financial forecasters.
I Financial news is replete with questionable charts,

graphs and pseudomathematical jargon: “Fibonacci
ratios,” “cycles,” “waves,” “golden ratios,” “parabolicSARs,” “pivot points,” “symmetrical triangles,” “risingwedges,” etc.
What should responsible finance researchers do?
First and foremost, ensure that their own published research is mathematically andstatistically sound.
8 / 22

Email exchange between DHB and a finance colleague
Email from DHB to finance colleague, 10 June 2013:
One thing that has always puzzled me about the financial world is the following sort of thing:[press examples cited]. Excuse me for being “dumb,” but this sort of thing seems to me to beoutright nonsense. ... When people like those above say that they “know” where the stock
market is heading, this cannot have any scientific basis. ...
So why doesn’t somebody blow this whistle on this sort of thing? Am I missing something?

Response from finance colleague to DHB, 17 June 2013:
It is not a dumb question at all. It is a question I have struggled with and which answer makes
me an unhappy man. The truth is, most people in this industry are charlatans. They do nothave any particular model or theory to understand the world. They are not scientists. ...
I completely agree with your assessment. The amount of nonsense ... is incredible.
9 / 22

What is backtest overfitting?
Backtest: Testing a model based on historical market data.
Backtest overfitting: Statistical overfitting of historical market data.
Examples:I Proposing a theoretical model that inherently possesses ahigher level of complexity than the backtest data.
I Using a computer to try millions or billions of variations of amodel or strategy on historical market data, and then only
presenting results from the variation that works best.
I Constructing an investment fund by using a computer toexplore millions or billions of weighting factors, then only
marketing the one that works best.

432

1

1

23

4

5

Fitting six data points(almost perfectly!) with afourth-degree function.

When a computer can analyze many variations of a fund or strategy on a fixed dataset,
it is almost certain that the “optimal” selection will be statistically overfit.
10 / 22

How easy is it to overfit a backtest? Very!
I If only 2 years of daily market data are available,

then no more than 7 variations should be tried.
I If only 5 years of daily market data are available,

then no more than 45 variations should be tried.

Backtest overfitting is now thought to be the principal reason why financial funds andstrategies that look great on paper often disappoint in practice.

I D. H. Bailey, J. M. Borwein, M. Lopez de Prado and J. Zhu, “Pseudo-mathematics and financial
charlatanism: The effects of backtest overfitting on out-of-sample performance,” Notices of the American
Mathematical Society, May 2014, 458–471.
11 / 22

Letters to clients: An absurd investment scheme
I A financial advisor sends letters to 5, 120 = 5 × 210

prospective clients, with 2560 predicting a certain stock willgo up, and 2560 predicting it will go down.
I One month later, the advisor sends letters only to the 2560

investors who were previously sent the correct prediction,with 1280 letters predicting a certain stock will go up, and1280 predicting it will go down.
I After ten months, the final five investors will have been sent

ten consecutive spot-on predictions!
This strategy is absurd, even fraudulent, because the final five investors are not told of
the 10,235 other letters with different predictions.
But why is promoting a statistically overfit fund or strategy, where investors are notinformed of the millions of failed computer trials, any different?
12 / 22

Proliferation of new stock funds
I Roughly USD$2.1 trillion is held in U.S.-listed exchange-traded funds (ETFs), withhundreds of new ETFs minted each year.
I In a 2012 study, researchers found that the median time between the definition of a newindex and the inception of a new ETF based on the index dropped from almost threeyears in 2000 to only 77 days in 2011.
I As a result, the report concludes, “most indexes have little live performance history forinvestors to assess in the context of a new ETF investment.”
Performance of the new funds:
Out of 370 new indexes, 87% of the indexes outperformed the broad U.S. market, based oncomputer-based tests before inception, but only 51% outperformed after inception.
The funds had an average 12.25% return above the broad market, based on computer-based
tests before inception, but -0.26% return after inception.
I Vanguard Research, “Joined at the hip: ETF and index development,” July 2012,
http://www.vanguard.com/pdf/s319.pdf.
13 / 22

How difficult is it to design a stock portfolio to achieve a desired
performance profile?
I Given some desired performance profile (a time series), we devised a computer

program that constructs a weighted subset of S&P500 stocks whose performance
matches the profile over the specified backtest time period (1991–2005).
I But most of these portfolios fail miserably when presented with new data

(2006–2015).
These erratic and often catastrophic results on new (out-of-sample) data aresymptomatic of statistical overfitting.

Technical details are given here:
I D. H. Bailey, J. M. Borwein and M. Lopez de Prado, “Stock portfolio design and backtest overfitting,”
Journal of Investment Management, vol. 17 (2017), no. 1, preprint available athttps://papers.ssrn.com/sol3/papers.cfm?abstract_id=2739335.
14 / 22

Steady growth profile, APR = 8%50 00040 00030 000

20 000

10 000

1995

2000

2005

2010

2015

Blue: constructed portfolio; orange: target profile; green: S&P500.
15 / 22

Steady growth profile, APR = 12%30 000
25 000
20 000
15 000
10 000
5000

1995

2000

2005

2010

2015

Blue: constructed portfolio; orange: target profile; green: S&P500.
16 / 22

Can stock market “gurus” forecast the market?
Kaissar’s analysis of prominent market forecasters (covering 1999 to 2016):
I The forecasters overestimated the S&P 500’s year-end price by 26.2% on averageduring the three recession years 2000 to 2002.
I They underestimated the index’s level by 10.6% for the initial recovery year 2003.
I They overestimated the S&P 500’s year-end level by a whopping 64.3% in 2008.
I They underestimated the index by 10.9% for the first half of 2009.

Kaissar’s conclusion: The forecasts were least useful when they mattered most.

I N. Kaissar, “S&P 500 forecasts: Crystal ball or magic 8?,” Bloomberg News, 23 December 2016,
https://www.bloomberg.com/gadfly/articles/2016-12-23/
s-p-500-forecasts-mostly-hit-mark-until-they-matter-most.
17 / 22

Our analysis of market forecasters
In 2012, the CXO Advisory Group ranked 68 forecasters based on their 6,582 forecasts(forecasts made for the S&P 500 index). We extended and advanced this study:I Each individual forecast is treated according to two factors: (a) time frame and(b) specificity.
I Long-term forecasts are considered more significant than the short-term forecasts,

since long-term underlying trends tend to overcome short-term noise.
I Specific forecasts are treated more important than non-specific ones.

18 / 22

Summary of forecaster study
I Across all forecasts, the average accuracy was 48%.
I Two-thirds of forecasters had an accuracy level below 50%.
I Only about 6% of forecasters had accuracy values between 70% and 79%; thehighest accuracy value was still below 80%.

For details:
I D. H. Bailey, J. M. Borwein, A. Salehipour, and M. Lopez de Prado, “Evaluation and ranking of marketforecasters”, Journal of Investment Management, to appear,https://papers.ssrn.com/sol3/papers.cfm?abstract_id=2944853.

19 / 22More than just a finance problem: Pitfalls in big data science
Artificial intelligence and big data have great promise to accelerate progress in science,engineering, finance and commerce.
But there are numerous potential pitfalls:
I Using unrealistic theoretical models.
I Using questionable statistical tests and techniques.
I Using insufficient data to draw statistically reliable conclusions.
I Failing to test thoroughly for software errors.
I Failing to facilitate reproduciblity.
I Becoming so enamored with the science that we fail to see errors and weaknesses.

Let’s not be guilty of generating scientific nonsense!

20 / 22

Why the silence?
Historically scientists have exposed pseudoscience — astrology in the 19th century;
young-earth creationism and health fads in the 20th century, etc.
Yet 21st century scientific researchers have remained disappointingly silent withregards to those in the community who, knowingly or not:I Engage in “p-hacking” to inflate the statistical significance of their results.
I Fail to disclose the number of models or variations that were tried in their study.
I Misuse probability theory and statistical methods.
I Present misleading charts and graphs.
I Use questionable pseudomathematical jargon.
I Make vague claims that do not permit rigorous testing and falsification.
I Withold key details, so other researchers cannot test or reproduce their results.
As we wrote in a recent paper:
Our silence is consent, making us accomplices in these abuses.
21 / 22

Thanks!
Visit our website at:
http://www.mathinvestor.org

This talk is available at:
http://www.davidhbailey.com/dhbtalks/dhb-mediax-2017.pdf

Collaborators:
I Jonathan M. Borwein (University of Newcastle, Australia, deceased 2016).
I Marcos Lopez de Prado (Guggenheim Partners and LBNL).
I Amir Salehipour (University of Newcastle, Australia).
I Jim Zhu (Western Michigan University).
22 / 22