Book Reviews: Summarizing Evidence: Crusaders for Simplicity

A fundamental task in all research is summarizing evidence. Whether the raw data consist of thousands of rows of numbers or thousands of pages of field notes, these data, undigested, are not informative. Given a set of research questions and a set of data, infinitely many summaries are possible. Choosing among alternative summaries is therefore inescapable, and criteria for evaluating alternative summaries are essential. Many researchers would agree that the adequacy of a summary depends on both completeness and parsimony. A summary is incomplete if it omits an important feature of the data. Among the summaries which capture the important features of the data, the best summary is the simplest. A tension exists between these two considerations. On the one hand, it can be argued that each addition to the complexity of a summary adds, at least incrementally, to its completeness. On the other hand, adding detail to a summary must, at some point, be harmful; otherwise, the best“summary" would be the raw data itself! Miles and Huberman (1984, p. 24) have argued that qualitative researchers need to develop shared, explicit procedures for summarizing data as "a part of analysis that sharpens, sorts, focuses, discards, and organizes data in such a way that final conclusions can be drawn and verified. " They do not imply that data must be quantified as a prelude to summary. However, when data are indeed numerical, explicit procedures for summarizing and explicit criteria for evaluating the resulting summaries do exist and are quite highly codified. In fact, summarizing may be regarded as the essential subject matter of statistical science. Quantitative researchers use descriptive statistics to organize and display graphical and tabular summaries of evidence, and they use inferential statistics to decide how elaborate the summary ought to be. Consider, for example, Englert, Raphael, Fear, and Anderson's (1988) comparison between experimental instruction and a control method on children's writing skill. The simplest summary in this case would contain two numbers: an effect size—typically a standardized mean difference between the two treatment groups—and a standard error. The effect size conveys the experimenters' belief about the size of the experimental effect on writing skill, and the standard error quantifies the degree of the experimenters' uncertainty about that belief. Such a simple "two-number summary" would be unjustified if the effect of the experimental approach were greater for special education children than for children in regular classrooms, as the experimenters hypothesized. The two-number summary would then be incomplete because it would ignore an important feature of the data. On the other hand, suppose that the experimental effect were really the same for the two groups, so that the only discrepancy between the estimated effects reflected the nuances of sampling error. Then a more elaborate summary reporting different experimental effects for special education and regular classrooms would be unjustified and might even be harmful. Such a summary would tend to convey the false impression that the two groups are differentially responsive to the new approach, an impression which could mislead educational policy-makers. When there is little empirical evidence supporting the existence of different treatment effects for different groups, such an elaborate summary is unparsimonious, and therefore inadequate: It is more detailed than the evidence warrants. In many educational studies, the logic of summarizing quantitative evidence works as follows. A researcher postulates a more or less complex model of the phenomenon under investigation. Analysis of variance and regression models are familiar examples in education. Here complexity is indexed by p, the number of parameters in the model. Given a particular model, p sufficient statistics often exist. The principle of sufficiency tells us that these statistics contain all of the information in the original data about the parameters of the model, and for that reason, such statistics are sometimes called "summary statistics. " Thus, given a particular model, the sufficient statistics unquestionably provide a complete summary. The question of evaluating the adequacy of the summary of evidence then boils down to evaluating the adequacy of the model. A variety of graphical displays and statistical tests are available for this purpose. they help the researcher decide whether the model is incomplete (too few parameters) or unparsimonious (too many parameters).