Experimental Tests in Education: Recommendations from the Holtzman Report

In the fifth century B.C., the philosopher Democritus maintained that he would rather uncover a single causal connection than sit on the throne of Persia. It is not clear that policy makers can or should have the same preference, especially if the choice of thrones is enlarged a bit. The two aspirations are compatible, however, and this article is addressed to their limited union. In particular, this paper deals with a report submitted in 1980 by a group at Northwestern University to Congress and the U.S. Department of Education (Boruch and Cordray 1982; Boruch, Cordray, Pion, and Leviton 1981). The study on which it is based was mandated under the 1978 Educational Amendments: it reviewed evaluations of federally supported education programs at the national, state, and local levels of government. Congresswoman Elizabeth Holtzman of New York introduced the bill that required the study. Her interest lay in understanding the product of the government's $30-40 million annual investment in educational evaluation. The enterprise was designed to answer questions about why evaluations are undertaken by the government, who does them, how well they are done, and how the results of evaluation are used. A parallel report, commissioned later under the same law, was issued by the National Academy of Sciences Committee on Program Evaluation (Raizen and Rossi 1981). Two other reports, developed independently by Pincus (1980) and others at Rand and Cronbach and others (1980) at Stanford, are pertinent but discussed only briefly here. Northwestern's report considers a number of problems in this area, not the least being lexical ambiguity. A federal director of research, for instance, announced that he did no evaluations, and 15 minutes later his supervisor, a deputy secretary, said that everything in his shop was evaluation. Analogous problems of course occur in popular and scientific reports about statistics as well (see Kruskal and Mosteller 1979 on "representative sampling"). Here, for clarity's sake, evaluation is defined as attempts to answer one or more of the following questions: Who needs services and who receives them? What are the nature and cost of services? What are the effects of service? What are the costs and benefits of alternatives? This paper focuses on one aspect of the report, the recommendations on evalution designs, especially randomized experiments, in estimating the effects of education programs. Experiments here include settings in which children, classrooms, or entire school districts are randomly assigned to one of two or more methods of improving education, in order to estimate their relative costs or effects fairly and to make probabilistic statements about the result. The design is sturdy in the sense that the logic is plain (albeit deceptively so), robust, and well thought out. But as many readers know, it can be difficult to use outside the laboratory. The concern about randomized field tests is not unique-Rivlin (1971), Riecken and others (1974), Gilbert, Light, and Mosteller (1977), and others have considered its role in public policy. Fisher, too, seems to have shipped in these waters, if we judge correctly from Cochran's (1976) review of the Lanarkshire trial. The concern is, however, distinctive-we are aware of few reports to Congress that deal with the topic at all, much less break down its applications to federal, state, and local levels of government. Still fewer deal with experiments in the broad context of evaluation policy.