Anomalies and Multiple Hypothesis Testing: Evidence from Two Million Trading Strategies

We construct a large laboratory of over two million trading strategies, which we obtain by data-mining the two most commonly used datasets in finance (i.e., CRSP and COMPUSTAT). We use this very large sample for three purposes. First, we evaluate the properties of multiple hypothesis testing methods when applied to financial data. We find that only adaptive methods should be employed in finance applications (i.e., FDP-SetpM). Second, we provide an optimal thresholding for applications that evaluate trading strategies. Third, we quantify the proportion of false discoveries due to the failure to take into account testing a multitude of hypotheses. Our estimates for the proportion of lucky discoveries is over 90%, which is considerably larger than previously reported. ∗Tarun Chordia is from Emory University, Amit Goyal is from Swisss Finance Institute at the University of Lausanne, and Alessio Saretto is from University of Texas at Dallas. We thank Hank Bessembinder, Ing-Haw Cheng, Campbell Harvey, Ohad Kadan, Yan Liu, Kalle Rinne, Olivier Scaillet, Peter Westfall, Michael Wolf, Lu Zhang and seminar participants at the 6th Luxembourg Asset Management Summit, TAU Finance Conference 2017, 2017 FRA conference, Inquire Europe Autumn Seminar 2017, Lone Star Conference, Caltech, Case Western, CUHK, Texas Tech, Tinbergen Institute Amsterdam, University of San Diego, UNSW, and the University of Texas at Dallas for helpful discussions. All errors are our own. An increasingly large body of literature studies the profitability of trading strategies based on signals obtained from publicly available information. Researchers are currently tracking a number of strategies well in excess of 300 and new papers keep adding to that list. A recent paper by Yan and Zheng (2017) finds that a very large number of trading signals (more than two thousand) “exhibit genuine predictive power. ” In his presidential address, Harvey (2017) questions the performance of these strategies due to a number of possible problems with the way in which these strategies are discovered and evaluated. One particular problem is related to the fact that test procedures are often evaluated in isolation. Harvey and Liu (2014, 2015) and Harvey, Liu, and Zhu (2015) advocate the use of multiple hypothesis testing (MHT) as one tool that finance researcher should employ to limit the number of false discoveries. The essential idea is that, when studying the entire distribution of trading strategies, one has to account for the fact that some strategies’ performance will appear exceptional by luck, thus leading to some false rejections of the null hypothesis of no outperformance. Applying MHT, Harvey, Liu, and Zhu (2015) advocate the use of a threshold of three for t-statistics (as opposed to the traditional 1.96) and find that the abnormal performance of as many as 50% of the 316 strategies they consider could be due to luck. Applying a threshold of three, Hou, Xue, and Zhang (2017) also find that as many as 58% of the strategies that survive their replication effort could be due to failure to account for MHT. Harvey, Liu, and Zhu (2015) note that optimal thresholding for MHT requires the researcher to have a good sense of the entire distribution of tests, not only those that end up in the right tail (i.e., the ones for which the null was rejected and made it to a circulated paper). Yet, virtually all papers on this subject look at the sample of only strategies contained in published or working papers. In contrast, we construct a very large sample of approximately two and half million trading strategies obtained by data-mining the information contained in the two most commonly used datasets in finance, viz. CRSP and COMPUSTAT. For example, Harvey, Liu, and Zhu (2015) examine 316 strategies, Green, Hand, and Zhang (2013) study over 300, and Hou, Xue, and Zhang (2017) study 447.