Meta-analysis: formulation and interpretation.

Meta-analysis is a research synthesis that uses a quantitative measure, effect size, to indicate the strength of relationship between the treatments and dependent measures of studies making up that synthesis. For the healthcare professions, the internationally known Cochrane Collaboration publishes systematic reviews of the effects of healthcare interventions (see Antes and Oxman, 2001). Not all research syntheses are metaanalyses. Often, the purpose of a research synthesis is to provide a description of a subject area, illustrating the studies that have been undertaken. In other cases, the studies are too varied in nature to provide a meaningful effect size index. The focus of this column, however, is on research syntheses that result in a meta-analysis. One advantage of performing a meta-analysis includes the computation of a summary statistic for a large number of studies. This summary statistic provides an overall estimate of the strength of relationship between independent and dependent variables. Previously, research syntheses were divided into those studies that supported a particular hypothesis and those that did not support this hypothesis, making it difficult to form a conclusion. A second advantage of metaanalysis is that it provides evidence of the reliability of a research finding. Researchers have more confidence in the findings of multiple studies than the results of a single study. A third advantage is that it takes into account studies that failed to find statistical significance and may not have been published because of a lack of statistical power (reduced sample size). Not all researchers see this as an advantage. Kraemer and colleagues (1998) demonstrated that the effect sizes generated from underpowered studies were likely to be poor estimates of the population effect sizes. Introducing a large proportion of studies with inadequate statistical power into a meta-analysis could introduce bias into the overall effect size. A fourth advantage of metaanalysis is increased external validity. Many studies, strong in internal validity (design characteristics), do not use a representative sample of subjects. This limits the generalization of results. However, including many studies increases the variation of the sample and strengthens external validity. The most frequent criticism of meta-analysis is that it combines “apples and oranges.” Synthesizing studies that might differ on both independent and dependent variables brings into question the usefulness of the end product. Further, many studies have similar independent and dependent variables but differ in the strength of design. Should these studies be combined? Last, even though the statistics used in meta-analysis are quite sophisticated, the end product will never be better than the individual studies that make up the meta-analysis. In this introduction to meta-analysis we use, as an example, a meta-analysis published in this journal by Connor and colleagues (2002) that demonstrated that pharmacologic interventions were effective for aggression-related behaviors in attention deficit hyperactivity disorder (ADHD). We discuss the criteria for inclusion and exclusion of certain studies, the selection and computation of an effect size for an individual study, and the computation of an overall effect size and other statistical indices. We recognize that this column is brief and recommend the text by Lipsey and Wilson (2001) as an introduction to meta-analysis; we also recommend the edited text by Cooper and Hedges (1994) for the more sophisticated reader. Connor and colleagues’ meta-analysis included 28 studies investigating the effects of stimulants on overt Accepted June 9, 2003. Dr. Gliner is Professor of Occupational Therapy and Education, Colorado State University, Fort Collins. Dr. Morgan is Professor of Education and Human Development, Colorado State University, Fort Collins, and Clinical Professor of Psychiatry, University of Colorado School of Medicine, Denver. Dr. Harmon is Professor of Psychiatry and Pediatrics and Director of the Irving Harris Program in Child Development and Infant Mental Health, University of Colorado School of Medicine, Denver. The authors thank Nancy Plummer for manuscript preparation. Parts of the column are adapted, with permission from the publisher and the authors, from Gliner JA, Morgan GA (2000), Research Methods in Applied Settings: An Integrated Approach to Design and Analysis. Mahwah, NJ: Erlbaum. Permission to reprint or adapt any part of this column must be obtained from Erlbaum. Correspondence to Dr. Harmon, CPH, Room 2K04, UCHSC Box C268-52, 4200 East Ninth Avenue, Denver, CO 80262; e-mail: robert.harmon@ uchsc.edu. DOI: 10.1097/01.chi.0000085750.71002.01