Letters: Meta-Analysis of Research: A Rejoinder to Glass

Glass (November, 1976) has called attention to an increasingly significant problem: how to interpret the findings from many individual studies, when the studies are diverse with respect to populations sampled, instruments used, variables measured, methodological designs employed, and findings obtained. Our attempts to summarize the research on creativity (Mansfield & Busse, in press) encounter the same problem but lead us to take exception to two of Glass's points: (1) that well designed and poorly designed studies probably do not give very different findings, and (2) that it is meaningful to compute the average effect size across studies varying in methodological sophistication. Glass suggests that well designed and poorly designed studies do not give very different results. Our experience reviewing the effectiveness of creativity training programs convinces us that large, well designed studies are less likely to yield significant training effects than smaller, poorly designed ones. Logical considerations also suggest that Glass's position is probably mistaken. Poorly designed studies are likely to yield spurious findings precisely because these studies are poorly designed. If, as is often the case in educational studies, an experimental program is confounded with the expertise and commitment of those implementing it, spurious findings are likely to be obtained. A related difficulty, which Glass does not mention, is a bias in the publication rates of well designed and poorly designed studies. Large, well designed studies are likely to be published whatever the statistical significance of their findings. But small, poorly designed studies are likely to be published only if significant treatment effects are obtained. Thus, summaries of published studies can be expected to yield over-optimistic estimates of the effectiveness of treatments. Turning to the second point, we find fault with computing the average effect size across studies varying greatly in methodological sophistication. The effect sizes obtained in small, poorly designed studies are likely to be misleading for two reasons. First, the use of small samples may produce spuriously large treatment effects by chance alone. Second, the effect size for any treatment or program is probably related to the expertise and commitment of the persons implementing it. In studies using small samples, these personal characteristics are often atypically favorable. A new curriculum, for example, is likely to be tried out with teachers selected, at least in part, for their enthusiasm (i.e., they volunteered). In large-sample studies such effects are less likely, since many people must be used to implement the program, often including persons with low levels of expertise and commitment. For these reasons we believe that large samples produce more accurate estimates of the effects that are likely to be obtained when a treatment is implemented on a large scale. Glass's proposal for the use of average effect size is related to his view that 44 the proper integration of research requires the same statistical methods that are applied in primary data analysis (1976, p. 6). We regard this position as simplistic, although it may be applicable to selected problems. As a general strategy, the use of statistical methods appropriate for primary data analysis assumes too much comparability among studies. It ignores differences in sample size, sample characteristics, instruments used, quality of data collection (e.g., mailed vs. in-person data collection, group vs. individual testing), and appropriateness of statistical analyses. Our own recommendations for meta-analysis require more judgment by the researcher: 1. Locate all available studies on a topic, including dissertations, research reports, reports to foundations, etc. 2. Eliminate from consideration studies with severe methodological inadequacies. Results from these studies are likely to be more misleading than helpful. 3. Divide the remaining studies into two categories: those that are reasonably well designed and those that have significant methodological limitations. 4. Integrate the results of the studies as follows: (a) If all or almost all of the studies point in one direction, tentatively accept that conclusion. The effect size in the large, well designed studies serves as a rough estimate of the effects likely to be obtained under mass implementation. (b) If the well designed studies yield results different from the mediocre studies, tilt toward the findings of the well designed studies. (c) If the well designed studies yield inconsistent results, look for differences among the well designed and mediocre studies that might explain the discrepancy. Both measured and unmeasured variables should be considered. If differences can be found which might explain the inconsistent findings, tentatively attribute the findings to these differences. (d) If the inconsistencies cannot be explained easily by such variables they may be due either to chance or to undetected, unmeasured differences among the studies. No conclusions are possible.