Advancing meta‐analysis beyond simple parameter estimation

In evolutionary biology, questions of general importance are rarely amenable to answers from single experiments or studies. Because we deal with living organisms and because evolution is fundamentally about change and diversity, answers come from aggregating information from many experiments or studies. Meta-analysis is a formal way to do such aggregating. Over the past 25 years, meta-analysis methodology in general and its use in ecology and evolution have made great strides (Koricheva et al., 2013) resulting in a substantial increase in the rigour of synthetic studies. That work, however, has focused on a narrow class of parameters (e.g. means and regression slopes) and a specific methodology. Morrissey (2016) shows that for some types of parameters, the standard methodology produces biased estimates. These other parameters include those that are central to evolutionary theory, such as the magnitude of selection coefficients and the shape of reaction norms. The standard methodology was developed with a goal of improving the precision of parameter estimates; Morrissey broadens that to considerations of estimation bias. His results in no way invalidate the previous work on meta-analysis methods; they merely show that such methods are not appropriate for all questions. The outcome should be a renewed focus on meta-analysis methods that expands our conception of what is considered part of metaanalysis and its toolbox of available techniques. Meta-analysis is a way to formalize information aggregation. It can be defined broadly, as by Glass (1976) who coined the term: ‘the statistical analysis of a large collection of analysis results from individual studies for the purpose of integrating the findings’. Or it can be defined more narrowly, as by Koricheva & Gurevich (2013) in a recent handbook on meta-analysis: ‘a set of statistical methods for combining the magnitudes of the outcomes (effect sizes) across different data sets addressing the same research question’. The problem with the latter definition is that all questions are not about effect sizes, nor are all data aggregated from data sets addressing the same research question. I agree with Morrissey that meta-analysis should be considered to be broader than just a specific set of statistical methods. In aggregating data, we can ask a variety of types of questions. The meta-analysis tradition represented by Koricheva and Gurevitch is primarily about hypothesis testing: Does the mean effect size of some sort of treatment differ from zero? The meta-analysis of Morrissey is more about parameter estimation: What is the mean magnitude of the directional selection gradient in natural populations? A third type of question is about the frequency of occurrence of some phenomenon like the one we (Palacio-L opez et al., 2015) asked in a meta-analysis about adaptive plasticity: How often do plant traits show adaptive plasticity? That question is not about mean effect sizes. We do not expect reaction norm slopes to centre around some particular parameter values; just the opposite, we expect them to vary widely. The question is about the percentage of times that reaction norms are greater than a (admittedly arbitrary) threshold and show a particular pattern when compared among populations. Such questions are sometimes derided as ‘vote counting’ but doing so is a mischaracterization. Meta-analysis was developed, in part, as a way of advancing research syntheses beyond simply counting up the number of studies that reached statistical significance. For example, if one was interested in the importance of competition as a structuring process in plant communities, one could survey the literature for all studies that experimentally manipulated competition (e.g. growth alone vs. growth with another individual) and counted the ones that found a statistically significant effect. If, say, 25% were statistically significant, one would then declare that competition was an important process approximately a quarter of the time. That would be wrong and Koricheva and Gurevitch correctly condemn such an approach because it fails to account for variation in statistical power among the studies. Instead, they properly advocate that for each study an effect size be calculated that accounts for such variation and then those corrected effect sizes be aggregated and tested for a departure from zero. But standardized effect sizes can be used in other ways. In our analysis (Palacio-L opez et al., 2015), we determined the mean effect size that represented a one standard deviation difference among treatments and used that to determine the frequency of traits that fell into various categories (e.g. not plastic, adaptively plastic, nonadaptively plastic). Importantly, we also looked at the effect of changing that threshold and provided the bootstrapped cumulative distribution function so that readers could choose their own threshold. This procedure is a different, but equally valid, use of effect sizes. I emphasize that our study followed the other advice in Koricheva et al. (2013) about gathering and appraising data. All syntheses, whether testing Correspondence: Samuel M. Scheiner, Division of Environmental Biology, National Science Foundation, 4201 Wilson Blvd., Arlington, VA 22230, USA. Tel. : +1 703 292 7175; fax: +1 703 292 9064; e-mail: sscheine@nsf.gov