OF 
3i^istiaiiiia 

rnRevised 

March 1992 

The Evaluation Synthesis^ 

GAO/PEMD-10.1.2 
Preface 

GAO assists congressional decisionmakers in thei

deliberative process by furnishing analytical inform

tion on issues and options under consideration. M

diverse methodolo^es are needed to develop soun

and timely answers to the questions that are posed 

the Congress. To provide GAO evaluators with ba

infomiation about the more commonly used 

methodologies, GAO's policy guidance includes do

ments such as methodology transfer papers and te

lucal guidelines. 

The Evaluation Synthesis presents techniques by 

'Which questions about a federal program are deve

oped collaboratively with congressional committe

staff, existing studies addressing those questions 

identified and collected, and the studies are asses

in terms of their quality and, based on the strengt

the evidence supporting the findings, used as a da

base for answering the questions. Tlie end-produc

information about the state of knowledge in relati

to the particular questions at a particular point in 

time. 

The evaluation synthesis seeks to address the nee

a client for the rapid production of infonnation re

vant to a pecific program and the analysis of larg

amounts of sometimes conflicting infomiation on 

topic. Conflicts cannot always be readily resolved

course, but sometimes they can be when it turns o

for example, that one study has been soundly 

designed, implemented, and reported, ^ereas 

another has been inappropriately designed for the 

questions it seeks to answer, bi addition to meetin

these needs, the evaluation synthesis develops an 

agenda showing clearly vrheie the gaps in needed 

information are that call for new agency research

it also lays the groundwork for fiirther evaluation 

audit work. This reissued version supersedes tlie 

1983 edition. 

Pkgel GAO/PBIID-10.i.a Evalnatlui^^iil
Preface 

We look forward to receiving comments from the 

readers of this paper. They should be addressed to 

Eleanor ChelimslQr at 202-275-1854. 

Wemer Grosshans 

Assistant Comptroller General 

Office of Policy 

Eleanor Chelimsky 

Assistant Comptroller General 

for Program Evaluation and 

Methodology 

GAO/PEMD.IO.1.2 Evaluation Oynt
BLANK PABE 

nwe « 6AO/PEIID-10.1.a EvaliiBtlao Sy
Preface 

Chapter 1 

Defining 

Evaluation 

Synthesis 

Chapter 2 

Why Do 

Evaluation 

Synthesis? 

Chapter 3 

Developing the 

Synthesis 

Chapter 4 

Performing the 

Synthesis 

Chapter 5 

Evaluation 

Synthesis Can 

Answer Questions 

a Single Study 

Cannot 

What Exactly Is Evaluation Synthesis? 

Steps in an Evaluation Synthesis 

Why Evaluation Synthesis Is Important 

The Sliengths and Limitations of Synthesis 

Evaluation Synthesis Can Guide Fldure Research 

Quantitative .Approaches for Evaluation Synthesis 

Special Problems of Quantitative Synthrais 

Nonquantitative i^proaches in Evaluation i 

Presenting the Findings 

Specifying the Questions 

Gathering the Studies 

Developing Criteria for Ghoo^ng Studies 

Organiang and Implementing a Reviewing Strategy 

Redetermining the ^propriateness ofthe 

Syntheas Method 

An Example 

Why Interaction Effects Are bnportant 

Why Synthesis Is Useful in Identifying brteracttons 

Summary 

Fage 4 GAO/FEBID-10.1.2 Evalnattan Byn

.,.;;.;..^..:..:,:;...-
Contents 

Chapter 6 

Special Topics in Comparing and Contrasting Studies and Their 

Evaluation Findings 

r" **"*~**«^"*» Merging tiie Quantitative and Nonquantitative 

byntneSlS Approaches 

Exploiting Differences in Study Findings 

Anticipating Problems That Mijght Emerge 

Bibliography 

Glossary 

Contributors 

Papers in This 

Series 

Tables 

Figures 

Table 4.1: Mean Birthweight Quantitative Summary 

Figure 1.1: Sequence of Steps in Evaluation 

Synthesis 

Abbreviations 

CETA Comprehensive Employment and Itainin

GAO General Accounting OfiBce 

OEO Office of Economic Opportunity 

VVIC Spedal Supplemental Food Program for 

Women, Infonts, and Children 

Pages QAO/PElID-tO.1.8 Bvalaatton Synl
Chapter 1 

Defining Evaluation Synthesis 

To provide timely yet comprehensive and integra

information to a client on how a program is work

one approach that the General Accounting Offic

(GAO) applies is a cluster of techniques known c

tively as the evaluation synthesis. This approach 

addresses the problem of timeliness by making u

existing evaluations. The evaluation synthesis is 

methodology for addressing questions that can b

isfactorily answered without conducting priinary 

collection; it is not a replacement for original da

collection. 

The evaluation synthesis has two msyor benefits

the ability to draw on a number of soundly desig

and executed studies adds great strength to the 

knowledge base when findings are consistent ac

different studies conducted by different analysts 

different methods. No single study, no matter ho

good, can have this kind of power. Second, whe

studies are not well designed and executed, the 

kno^edge that there exists no firm basis for act

also an important benefit: the size ofthe risk bei

taken is clarified, necessaiy caution is introduce

the debate, and over the long term, the number 

failed shots in the dark is likely to be diminished

What Exactly Is 

Evaluation 

Synthesis? 

An evaluation synthesis is a systematic procedu

organizing findings fi'om several disparate evaiu

studies. It enables the evaluator to gather result

different evaluation reporte, performed by differ

people at different places and at different times, 

to ask several questions about this group of rep

Some of the questions are broad; otiiers are qui

cific and narrow. 

An evaluation synthesis can answer several diffe

kinds of questions—about overall program 

effectiveness, about specific versions ofthe prog

that are woridng especiaUy well or especiaUy po

and about how to organize future evaluation stu

Fage 6 GAO/FEMD-IO.1.2 Evaloatloii Sy
Chapter 1 

Defining Evaluation Synthesis 

provide even more useful information about a 

program. 

GAO has used the evaluation synthesis to answer 

congressional questions about both how programs 

operating and what their effecte are. For example, 

evaluation synthesis can provide an estimate of how 

many people are actually receiving program servic

The report entitied Disparities Still Exist in Who G

Special Education on the Education for aU Handicapped Children Act used 14 existing studies and t

date bases to describe the handicapped children 

receiving special education services (GAO, Septem

1981). This report was able to use different source

not only to provide an estimate of how many childr

were recei'ving services but also to describe their 

racial and ethnic background and the severity of th

handicaps. No study provided estimates on each 

description, nor did multiple estimates necessarily 

agree. 

Similarly, we have used the evaluation synthesis to 

detennine how many people need a program servi

The special education report again serves as an 

example. The studies aUowed for an examination o

this issue, including estimates of particular handic

ping conditions underrepresented and grade and a

levels with particular underrepresentation. 

As for program effecte, GAO's 1982 report on the 

Comprehensive Employment and Training Act 

(CETA), for example, examined tlie effecte of CETA 

programs on disadvantaged adult enroUees (GAO, 

June 1982). Entitied CETA Programs for Disadvan

tage! Adulte-What Do We Know About Their 

EnroUees, Services, and Effectiveness? the report w

able to provide estimates of CETA participante' ex

riences before and after program participation wi

respect to wages eamed and time employed, pubU

benefite received, and private sector employment. 

AdditionaUy, estimates were provided for participa

Fage 7 6AO/FBBID-10.1^ Evalaation Synlfa
Chapter 1 

Defining Evalaation Synthesis 

experiences by type of CETA service received. 

FoUow-up reporte fi'om the Continuous Longitud

Manpower Survey provided the date base. Anoth

report used the evaluation synthesis method to st

the effectiveness of expanded home health care s

vices to the elderly (GAO, December 1982). Estim

of effect were provided for dient outeomes and c

Twelve msyor studies were used in determining t

estimates. 

We have also used the evaluation synthesis to com

pare the perfonnance of two or more programs. 

example, Lessons Leamed FVom Past Block Gra

ImpUcations for Congressional Oversight examin

the question of whether the poor and other 

disadvantaged groups have been served equaUy 

block grante and categorical prograins (GAO, Se

tember 1982). Eight basic evaluation studies, so

comprising a series of reporto, were used. 

As these examples show, the evaluation synthesi

brings together existing studies, assesses them, a

uses them as a date base for answering specific c

gressional questions. It enables evaluators to det

mine ix^iat is actuaUy known about a particular t

estimate the confidence (based on study method

and execution) that can be placed in the various 

studies used in the date base and their findings, a

identify gaps that remain in evaluative research w

regard to the congressional questions. 

Designed to be performed in a short time period, 

evaluation synthesis has the important advantag

low cost. One or two persons with sufficient expe

typicaUy can provide an evaluative summary of t

state of kno^edge in a particular area. The prec

amount of time necessary depends on the narrow

of the topic area, the size ofthe date base avsilab

and the familiarity of the evaluators with the top

the date base. 

Fage 8 GAO^FBMD-10.1.2 Evaloattai Qyi

:••
Chapter 1 

Defining Evaluation Synthesis 

An additional advantage ofthe evaluation synt

method is that by integrating evaluation findin

establishes an easUy accessible base of knowle

identifies knowledge gaps or needs with respec

specific topic upon which future evaluations ca

biuld. It can integrate adimnistrative date and 

from studies with either qualitative or quantiti

emphasis. It improves the use made of evaluati

infonnation since, in and of itself, it helps ensu

systematic legislative use of evaluations that ha

already been completed. 

What differentiates the evaluation synthesis fro

many other efforte involving the review and an

of evaluative Uterature is that, as part of an ove

strategy, it is designed backwisutl from the end

That is, the evaluation ^nthesis is driven not b

quest to increase knowledge but by a specific 

need—requested or anticipated—for certain 

infomiation. This means that the work must al

begin with a firamework of questions that impa

ical cohesion to the effort. Some ofthe questio

be answerable by the available infonnation bu

may not be. Those left unanswered serve to ide

gaps in the desired array of information. 

Steps in an 

Evaluation 

Synthesis 

Throughout this document, we will ^ve detaile

gestions on how to organize and cany out an e

tion synthesis. We also gjnre several illustratio

clarify how to unplement each suggestion. But 

helpful to begin by summarizing the seven step

all evaluation syntheses require. They are show

figure 1.1. 

PkgeQ OAO/FEBID-10.1.2 Evalaatioa B
Chapter 1 

Deflning Evaiuation Synthesis 

Rgure 1.1: Sequence of Steps in Evaluation Synthesis 

Selecting the Questions to 

Be Addressed 

Developing and 

Implementing Critical 

Review of Studies 

Fage 10 GAO/FEMD-IO.1.2 Evaluation Synt
Chapter 1 

Defining Evalnation Synthesis 

The seven steps fall under the two broad cate

chapters 3 and 4 of developing the synthesis a

forming the synthesis. In the seven steps, the 

evaluator should 

specify the questions: how questions are state

determine how an evaluation synthesis is orga

gather the documentation: coUect journal art

bibliographies firom computerized dato bases

unpiibUshed evaluations and research reporte 

evaluators in the field to identify key studies; 

develop criteria for choosing studies: justifyin

initial decision of viddch studies to include is c

organize and implement a reviewing strategy

studies a^inst basic standards for research d

conduct, analysis, and reporting; 

redetermine the appropriateness ofthe synth

method: this takes place after a preliminary r

the available evidence; 

implement the evaluation synthesis and check 

problems: tiie qnnthesis can be done using qu

tive and qualitative evidence, and it is particu

helpful to anticipate problems that may occur 

take steps at the outset to minimize them; 

present the findings: be sure to state the ol^e

describe the scope and methodology, and of c

reispond clearly and condsely to the question

were asked. 

Fage 11 GAO/FSlID>ilO.U Evalaation 
Chapter 2 

Why Do Evaluation Synthesis? 

Evaluation synthesis is a fonnal technical procedu

for combining the resulte from several empirical 

studies. We use the word "formal" to indicate tha

execution of the synthesis is not specific to a part

ular evaluator or a particular set of studies. In fac

^stematic nature is ite primaiy strength. Two eva

tors using the same synthesizing procedure shou

arrive at the same statistical output, although the

interpretations ofthe output may d^er. 

Why Evaluation 

Synthesis Is 

Important 

;?-'•;•• . • • . 

MM. 

Faced with tens or even hundreds of studies on a 

single topic, an evaluator unarmed witii systemat

procedures is forced to use suliyective criteria for 

deciding how to ^nthesize. The evaluator may ch

several favorite studies, relatively well done finm 

classical experimental deengn standpoint. Or 

evaluators may favor studies carried out by inves

tors they respect. In either case, their impresslo

conclusions will often differ from those of other 

weU-intentioned evaluators. A good example of tw

evaluators' differing dramaticidly in their 

interpretations of the same set of studies is provi

by the delMite between Munsinger (1974,1978) a

Kamin (1978) conceming studies of adopted chU

dren's IQ's. 

Several researchers in the 1970's (Glass, 1977; K

KuUk, and Cohen, 1979; Ught ahd Smith, 1971; 

Rosenthal, 1978) commented on the unsystemati

ways that social science research findings were b

synthesized, lliey argued that the typical literatu

review was highly subtjeothre and feUfsur short of

orous scientific standards for the accumulidion o

evidence. In response, they tried to develop proc

dures for combining the resulte of independent 

stodies. The goal was to draw, in a systematic ma

as much infonnation as possible firom existing ev

dence. (Hedges, 1988; Wachter and Straf, 1990) 

12 OAO/FE1ID>10.1.2 Evaloatlonl^t

'.'•'-''i
Chapte

r 2 

Why Do Evaluation Syntheaia? 

So one clear importance of synthesis relates t

research. Narrative overviews of prior finding

offering a certain contextual richness that a si

technical index cannot, generaUy do not provi

systematic information a researcher needs to d

more powerful future investigations. (Rubin, 1

A second importance involves rendering scien

research useful to pubUc pohcy. PoUcy decisio

to be made. If research findings are to inform 

they must be put into an understandable form 

provide answers or partial answers. The answ

occasionally clear-cut, but more often they ar

to be more complex, reflecting the real-worid 

relationships between poVicy variables and ou

For example, the question 'What are the effec

titie I legistation?" does not have a dmple an

Certain programs under that legislationm^y '

while others may seem to fail. Hie effecte of a 

ular program depend upon a variety of factor

who participates, the size ofthe community, a

the mon^ is distributed. 

Even wh&n a policy question is complex, ther

strong need for sunuiuuy informatioii. A nair

description of 100 s^cUes is firequentfy not en

there is not a stegle "matai effecte" answer, an

program's success depends largely on 

setting-by-tieatment inteiadions, synthesis m

ceed in identifying and summariztaig these iht

tions concisely. This c$n lead to guideliii)^ ab

^^ere and how to implement a paiticuW prog

improvtaig the chances for ite success. (Cbrdr

1990) 

Wu'.- ". 

l-i'v-:-' : 

I'V;'- • Fag

e 18 

GAO/FBlID-10.1.2 Evalaation 
Chapter 2 

Why Do Evaluation Synthesis? 

The Strengths 

and Liihitations of 

Synthesis 

Strengths 

iii>. 

The nuuor advantage ofthe evaluation synthesis 

abUity to provide relatively inexpensive, compreh

sive, and timely infonnation. It is designed to be 

ducted by one or two persons, with methodologi

expertise, and can be performed usuaUy in 6 to 9 

months. By integrating findings firom already 

completed stodies, the evaluation synthesis can 

tially serve a cUent's needs for relatively short-te

evaluative infonnation. The focus ofthe evaluati

synthesis is tailored to specific concems. 

Another strength is that the evaluation synthesis 

increase the power ofthe individual stody findin

Con^dence in a number of weU-done stodies wit

same finding is greater than in the finding of a si

weU-done stody. 

By drawing together information about a specifi

question ftom a disparate number of completede

ation stodies, the evaluation synthesis also creat

common knowledge base about a particular topi

cleariy sete out what is known-and with what le

confidence—and what is not known about the to

thus enabling program managers and evaliuition 

to determine where they might best commit futu

evaluation resources. Thus, a particularly valuab

feature of the synthesis is the identification of 

remaining unanswered questions. 

FinaUy, the evaluation synthesis can serve, to a l

extent, as a check on the quality ofthe evaluatio

being performed conceming a particular progrtt

The technical review of each stody identifies 

methodological strengths and weaknesses that 

Fage 14 GAOyFElfD-10.1.2 Evaluation Syn
I Chapter 2 

I Why Do Evaluation Synthesis? 

influence a sponsor's posture with regard to f

stodies. GAO's synthesis of special education 

for example, found that many stody reporto d

adequately describe the methodology they em

The U.S. Department of Education indicated i

commente on the report that it had reviewed t

stodies the report used and agreed that the cr

was valid. Since most ofthe stodies were cond

under contract, the department indicated that 

approval from ite Office of Procurement and M

ment, a requirement to include a methods des

in final reporte could be written into future re

for proposals. (Bomstein, 1989; Bowers and 

1988; Chalmers etal., 1981; Cordray, 1990; D

Hirt, and Schroeder, 1989) 

Limitations The main limitations of the evaluation synthe

odology stem firom ite reliance on extant date

methodology is best appUed to areas in vriiich 

a base of evaluation infonnation. PoUcy conc

which there is Uttie or no existing stody infor

cannot be satisfactorUy investigated. Thus, th

odology will not be appropriate for new progr

where evaluation stodies have not been comp

perhiq[>s even initiated) and no existing infon

base has appUcabiUty. 

Even '<^en a substantial infonnation base is a

the evaluation synthesis is limited in that it ca

questions only to the extent that the existing 

have addressed them. Thus, for example, fin

response to a particular question may or may 

generalizable to the nation, depending on tlie 

ofthe relevant studies conducted on this top

Poor reporting also limite the evaluation synt

Procedures may have been described iii so br

manner that judgmente cannot be made abou

stody's technical adeqUaqr. AdditionaUy, in 

experimental or quasi-experimental stodies. 

Fage IB 6AO/FEBID-10.1.2 Evalaation 
Chapter 2 

Why Do Evaluation Synthesis? 

Evaluation 

Synthesis Can 

Guide FHiture 

Research 

treatmente may have been so minimally describ

judgmente cannot be made about the similaritie

differences across stodies or variables of interes

not have been reported consistently across stod

Some stodies may report demographic date suc

sex, age, and education, for example, whUe othe

stodies focusing on the same questions do not. 

evaluation synthesis is limited by the form and 

of the reporte it uses. 

Finally, the evaluation synthesis is only as curre

the stodies it analyzes. If stodies are several yea

they may have Identified findings that program 

agers have already taken steps to address and t

no longer characteristic ofthe program. The m

ology is no substitote for priinary data collectio

it is useful iK^en questions can be answered usi

infonnation firom existing stodies and when tim

short. (Feingold, 1988; Hazelrigg et al., 1987; 

Johnson and Eagly, 1989; Parker et al., 1988; Y

and Wortinan, 1989) 

We have emphasized looking carefiiUy at existi

to see ii^ere things stand now. But some synth

are undertaken priinarily to help guide future 

research. Their goal is to suggest to the designe

the eleventh stody what can be leamed from th

10. The evaluation synthesis can provide such 

guidance in at least two ways. 

First, the synthesis can help by identifying the m

promising experimental manipulations and com

sons. With finite resources, it is not possible to 

aU variables formaUy into each effort. A review 

examine a large number of possible variables th

might be important and eliminate many of them 

serious candidates for new research. If hospital 

not related to surgical success in 10 weU^one 

stodies, it is unUkely to emerge as cracial in the 

eleventh. Using a review to reduce tiie number 

16 OAO/FElID-10.1.2 Evaluation Sy
Chapter 2 

Why Do Evaluation Synthesis? 

experimental variables should unprove the stati

power and guide the allocation of resources in 

stody. 

Second, the synthesis can help researchers cho

between organizing one big new effort at a sing

and organizing a series of small efforte at many 

Suppose funds are available to evaluate a new t

ment for breast cancer involving 1,000 women

better to conduct one stody with aU 1,000 wom

one hospital or to commission five smaller stod

five different hospitals with 200 patiente each? 

Existing research can guide this decbion. On t

hand, suppose past evsduations show Uttie varia

the success of cancer treatment across differen

pitals. Then the wisest decision probably is to f

the entire new effort at one site. The large sam

will help identify subtie ways in wiach the new 

treatment differs from current practice. On the 

hand, suppose a review of earUer findings show

value of cancer treatment to vary widely across 

Tlien it could be a rnistake to focus on one part

setting or hospital. Setting-by-treatment intera

should be expected. This expectation can only 

assessed by trying the new cancer treatmeiit in 

places. 

The particular guidance a research review prov

wiU differ firom one substantive area to another

examples illustrate the benefite of designing in

research the messages of the old. The implicati

evaluators is that simply condudbig with the u

"more research is needed" is not enou|^. E^i

must make a conscious effort to idehtifyiR^iat s

directions new initiati'ves should take. This linl

past and present is cracial if research is to achi

filll potential for enhancing both science and p

Fage 17 OAO/FEllD-10.1.2 Evalaatioa ^
Chapter 3 

Developing the Synthesis 

The process of developing the synthesis is itera

Tlurough a series of steps shown in figure 1.1, t

thesis topic and information base are defined a

reexamined. The first five steps of the evaluatio

thesis covered in this clisv>ter are specifying th

tion, gathering the stodies, developing criteria 

choosing stodies, organizing a reviewing strate

redetermining the appropriatenras ofthe synth

method. 

Specifying the 

Questions 

The three most common questions that may be 

answered with an evaluation ^nthesis are 

1. For any program or treatment, what is ito ef

the average? 

2. Where and with "whom is the program or tre

particularly effective or ineffective? 

3. Will it work here? In other words, what are 

tical guidelines for implementing the progrun 

treatment in a particular place? 

Different reviewers can approach the same gro

stodies with quite different goals. PoU<7inaker

face decisions requiring an estimate of average 

gram perfprmance. For example, Blue Cross a

Shield must decide whether to offer third party 

mente for psychological counseling to persons 

have just had cancer surgeiy. The goal is to eh

recovery rate and reduoe morbidity. Here, an 

trative regulator may simi>fy want an answer t

question, 'On the average, does psychological 

seling after cancer surgery help people?" Rese

may think this is far too broad a question. But 

poUcymaker's main concem is not with 9ii»n$i

feet m^ches between psychologist and cli ^ 

need for a decision about whether postamcer

services should or should not be reimbursed. 

F ^ U 6AO/FB1ID-I0.i.2 Evalaation ^
Chapters 

Developing the Synthesis 

Researchers might come at the problem differ

The researcher might decide that averaging ac

several mental health protocols, with different 

of cUente, misses the main objective. The mor

tant questions here might be, "What kind of co

seling is usuaUy best and what sort of cUent be

most from that type of counseling?" Research

not be surprised if a particular treatment does 

work for everyone. Indeed, it is sometimes exh

rating to discover that a new treatment works 

anyone. So a researcher will usually organize a 

to go beyond an "on the average" question to 

examine what works, how well, and for whom

Local program managers may have yet anoth

pose in mind. WhUe interested in the question 

treatment is best for whom, their main focus i

biUty. Can an innovative treatment for breast 

be implemented successfiilly in specific, real-w

locales? It is one thing to leam that streptokin

administrated at a certain time can help. It is 

thing to build this finding into practice with g

resulte. A local hospital director or physician 

to know what treatment works best in genera

concrete evidence about what it takes to Unpl

Isreatment successfuUy in a specific environme

as a smaU, rural hospital rather than an urba

teaching center) will be particularly valuable

for this purpose wiU emp^hasize any available 

about implementation efforte at similar histitu

To summarize, aii evaluation i^nthesis design

answer the "on the average" question emphas

search for main effiecto. A syntheds that asks 

benefite most from what" wUl focus on a sear

interaction effecte. The eynthesis that asks ^h

wUl work here" sAiould emphasbse quaUti^Jve 

ofthe setting, the locale, and the context for a 

ment. (Glass et al., 1981; Green andHaU, 19I

Hedges, 1986; Hedges and OUdn, 1985; Rose

1984) 

Fage 19 OAO/FEBID-10.1.2 EvalaatioB 
Chapters 

Developing the Synthesis 

Evaluation ^nthesis can be used to answer a w

variety of questions, including descriptive, norm

and impact (cause and effect) questions. For ex

a report on tiie handio^ped focused on "who w

receiving" questions and vi^ether these groups 

over- or underrepresented vrith respect to the r

of special education services. The specific ques

were: Who does this program serve? (a descrip

question) To what extent are the intended bene

ries being served? (a nonnative question). An 

tion that attempted to assess the effect of race 

death peiudty sentencing answered the foUowin

impact question: Does the race of either the vic

the defendant influence the likelihood that defe

wiU be sentenced to death? 

The Idnds of questions for widch the evaluation 

thesis may be appropriate, at least for service d

types of prograins, axe, however, likely to fall in

distinct categories. These are program operati

program effecte, both themselves componente 

broad question of v^ether the program is work

While the spedfic wording of tiie questions wil

examples are as foUows. "Die first three are prb

operations questions. 

Who does the program serve and to what exte

the intended beneficiaries being serwtd? The r

on the handioqpped, for example, asked not on

was receiving services but also vriiat groups w

over- and underrepres^ted with respect to the 

receipt of special education services. 

What are the program's services, vdiat service

delivered to whom, what is the service dfellvery 

cess, and are these QoiisiBtent with program 

objectives? In a report on CETA, for ejahiplei 

examined fitiifto in ttie inix of services oveir ti^ 

CETA programs. Services included classroom 

training, on-the-job training, work experience, 

pubUc service employment We also investigat

Fage 20 OAO/FEIID-IO.1.2 Evalaation S

^^_.,.. ,;;,./ • ••..;;•'•... •;-„..;; , •.•.,... • •':•:' ' ..•/. • -. •."•'•'. . ^ ./-

iHi !;*:• • 
Chapters 

Developing the Syntheaia 

differences in the characteristics of persons re

these services—in other words, how were the s

targeted? 

What administrative processes and procedure

implemented? How is the program administer

stody of lessons leamed fiom past block grant

investigated stodies of the coste of administeri

block grante and the effecte of fixed percentag

on administration. 

Here are four typical program effecte question

What are the general outcomes for program 

recipiente? A stocfy on home health caie, for e

investigated stodies of the effecte of expanded 

health care on cUent longevity, satisfaction, ph

fimctioning, and mental health. 

Do program outeomes vary by type of recipie

types of service? The CETA study examined^

differences across service types (classroom tr

on-the-job training, work experience, and pub

vice employment), hi the characteristics of jpa

pante, and in their occupational areas pf empl

and training were refleded in dato on their 

experiences before and after CETA. 

What is the prognun effect on other than prog

redpiente? A nu^or question in a staidy ofexp

home health care services was the effect of e^

home health care on niirshig home and hospi

How effective is the prpgram in terms of opste

native programs, or ^Ufferentverisiohs^ 

program? The CETA stody, for example. irtVJ

the fectiveness pf CETA in terms of pi ^ 

eamings that could be attributed diredly to^^^^^ 

partidpation in adult>K>riented services^it 8 ^ 

examined gains b>y service type to determine 

one type of service was more effective than an

Fage 21 GAO/FEIID-IO.1.2 Evaluation 
Chapters 

Developing the Synthesis 

type (for example, on-the-job trauning versus 

classroom trauiing). 

Any one or more of these general questions may 

as the basis for a limited yet comprehensive subs

questions that can be used to respond to the con

sional need for program information. These ques

provide a framework not only for conducting the 

uation synthesis but also for reporting the findin

The process of selecting the precise topic and ide

fying the actual stody questions drives the evalu

synthesis method. Tliis is particularly hnportant 

because the evaluation synthesis can answer onlj

questions for which there already existe stody in

mation. Even then, it can aiwwer questions only t

depth or extent that the evaluation stodies have 

addressed them, and it can be only as current as 

stodies themselves. 

It is important during question specification to c

duct a preliminary review of the kinds of dato av

able. Before settling on the stody topic and quest

the evaluator must have some familiarity with th

nature and extent ofthe evaluative information a

able on the proposed topic. The actual questions 

investigation must be carefiilly formulated so tha

they are neither so broad that addressing aU tiie 

pertinent evaluation information is not possible i

short time nor so narrow that Uttie evaluation in

mation is available for responding to them. 

There is Uttie Umitation on the type of topical are

suitable for evaluation synthesis. Ihe method b 

appropriate to drfense topics, for examjple, as to 

social service delivery topids. Given the heed, ho

ever, for a base of completed evaluation studies, 

method is.generaUy less appUcable to new ppUd

prograins. Conversely, for a program with a long 

it may be deskable to set a cutoff point for the t

Fage 22 GAO/FBBID-IO.1.2 Evalaation Syn
Chapter 3 

Developing the Synthesis 

Gathering the 

Studies 

period ofprogram operations to be covered in 

synthesis. 

An important consideration in the early fonnu

stody questions is the degree of precision nee

the answers to be found. For instance, the cUe

wish to know how many people need a service 

many are receiving a service. An exact answe

impossible. The answer wiU either be a forma

confidence mterval or, if the analysis is based 

stodies or less rigorous methods, have the fla

confidence interval. (For this and other statis

cepte, see UUman, 1978.) We mean by this th

^nthesis should specify a range of possible v

with some confidence that the trae value is in

that range. How narrow that range of possibl

must be to make the synthesis practically use

how high the confidence level must be that th

fied range includes the trae value will vitally 

each of the next steps of evaluation synthesis

The need to defme questions, to determine th

of precision needed in the answers, to assess 

appropriateness of evaluation synthesis versu

possible methods, perhaps to renegotiate the 

questions after having looked at the available 

evidence—these steps suggest an iterative, 

coUaborative approach between the informat

and the evaluator. 

Once the spedfic questions have been 

developed—remembering that the questions c

developed soundly only if ttiey are guided hyi

some prior knovidedge ofthe topic^ area and 

existing evaluation Uterature^relevmteyaliU

infDnxuition shouM be cominled. Vniiteth^ 

agency admbiistering a policy or pippgram is 

place to beghi, the evaluatipn qmi^esAs metii

requires that the investigation go b^n d tlds 

information base and indude nonagency-spo

Fage 28 GAO/FEIID-IO.1.2 Evaluation 
Chapter 8 

Developing the Synthesis 

Uterature. Without including nonagency-sponsor

Uterature, only a part of the universe of relevant 

stodies is likely to be obtained, and it wUl not be 

how large a part of the universe has been obtain

how biased or representative it is. 

In going to the agency, the objective is a thoroug

comprehensive search for information. Badcgrou

infonnation such as legislative and fimding Idsto

and regulations should be obtained as well as re

administrative or management information syste

dato and evaluation stodUies. Summaries of dato 

(or the actual computer ti^[>es) may additionaUy 

collected as part of the data base. Secondaiy dat

analysis, while not a necessary part of the appro

may be impropriate in oi^es where existing date 

have not been fiiUy explpited. 

While the short time period and the focus on sec

ondary dato coUection implied by the method cl

that interviews of agency officials and others be 

to a mmimum, mterviews may be needed to com

an understanding of the program and ite evaluat

andto identify ongoing iaitd planned evaluation 

stodies for which reports are not yet available. A

visite to project sites are not routinely ihidicated

they may also be informative. 

Nonagency-sponsored literature covers all evalu

stodies other than those initiated by the federsd 

agency administering the poUcy or prpgram. lid

indudes stodies conducted l^ other federal age

m the executive brancli; studies conducted by 

legislative agencies siic^ iss the General Acxouii

Office, Congressional R^earch Service^,and;Gpi

sional Budget ()ffic«; stodies undertaken itidepe

dentfy by stiUe and looal agencies, national 

associations, and membejrs of the academic com

nity; or stodies focusing c>n the same topic done 

other countries. (An evalijifidion synthesis of the 

"guestworker" program ekperience, for example

Faga 24 OAO/FBBiD-10.U Evalaation Sy
Chi^terS 

Developing the Synthesis 

might need to consider the European Uteratur

experience.) While it may be time-consuming 

otherwise problematic to attempt to explore a

infonnation sources, such efforte underUe and 

enhance the credibility and worth ofthe evalu

synthesis and, at a minimum, must be conside

One pitfiiU hi coUecting the literature for the s

thesis, as documoitedl^ White (1982), is tha

focusing onfy on publiidied reporte can lead to 

neous ccmdiisicmsi Whtte ifound th^ 

rraearch report tencleNi to more signific

positive findings than Uji^ 

with less significant fibndiid^ 

and, therefore^usUailfy 1 ^ Thus^ju

examining piitdidhed i^p ^ might teaul to sm

viewc>faprogjram's effe^^ al., 1|9$

Rosnow iemciltosenthal^^^ 

Being sure that no iJni|Jpr|!iublishe(ilc>r^^u^^ 

stUffy has been onutteiiis^]^^ 

lenge in an cnrali^M^iohi^th^ Qne fltppr«

in preventing an cmnissioni^ 

outside experts to luedpridentafy th^l literature 

review the Uterature c^olbeded In this vray. 

There are at least three SE^ific steps thatwefe

mend for c>rgaiiifdiig a sys^ ^ searcher 

artteleS'for an- eyEduadxm'i^^ 1nat!it:&C!

use a cx>mputerized^dl4;i| acexis^isi^^il$^ 

by chcmsingkey worc!s,!i^ exaih ^ ii rt^cia

tion synthesis;^I4p8ey^Ci990) examine^tiiei|

lowing ciato.basc»'M'|ii^^ 

research:'AIM/AItM Arte;|md:Himiahi6^ 

Index, EkM>Iaiih^ Print, B ^ 

Ecfaication Index, ChUd Abuse aiuiNi^icki;i,;

Justice Perio(Ucalihdek,JGRISP: Natton^lhM

Mental Health, Di{»eitation Abstrads OpOine^ 

Famify'Resources^ FVeHtiin^'Reseai^ 

PubUcations, Ubruyc>|Cbngress|k^^ 

Mental HeaItihAbstnu:te;!Nad^^ 

Reference Service, Nidonal Ttehhicalihfo^^

.•.,..:

Fage 25 GA(»FBBa>-10.1.2 Evaluation 
Chapters 

Developing the Synthesis 

Service, PAIS Intemational, Psychological Abstra

Social Science Citation Index, Sodological Abstra

SSIE Current Research, and U.S. PoUtical Scienc

Documento. 

WhUe a search of computerized dato bases is cra

and wiU generally identify most key articles and 

research reporte, a good second step is to examin

the listo of references at the end of key research 

reports. Such cxoss checking wiU often tum up 

adcUtional dtes, often dtes that you did not iiutia

locate simply because you used a k^ word that t

authors of the ori^nal article did not use as their 

word. If you identify several aitides m this way, 

constmctive to see if the laarge computerized dat

base actuaUy has these additional articles in ito U

and if yes, viiiat key word these artides use. Flnd

new key word may lead you to additional relevan

articles. 

FinaUy, a third step is to ask knovdedgeable colleagues and feUow scholars around the countiy. 

ever there were a good use for expert advice, thi

the time. 

The purpose that drives these approaches to ide

fying and gathering original stocties is that of be

all-inclusive. There is some risk, in the real world

the computerized dato base searches will tum up 

many artides that the synthesis can become imw

So an evaluator must be prepared to deal with th

potentiaUy enormous size of a fiill-fledged seaurd

must be willing to tolerate, for some topic areas, 

enormous set of potential stodies to indude in an 

evaluation synthesis. (Cooper, 1988,1989) 

Fage26 GAO/FEBfD-lO.1.2 EvaluationSyn
Chapters 

Developing the Synthesis 

Developing 

Criteria for 

Choosing Studies 

Once the relevant Uterature has been identified 

coUected, the question becomes: What types o

stocUes should the synthesis indude? A goal of 

tion ^nthesis is the identification and control 

potential sources of bias in the technical sense

stodies used in the evaluation synthesis share 

common, usuaUy unknown, sources of bias, th

thesis as a whole wiU take on that bias. 

When determining i^ch stodies to include in 

uation synthesis, the evaluator must also fl4>pl

standards of evidence. Was the stody sufficient

is, did it provide enough factual and c»nvinc±

dence to support ite fincUngs and condusions? 

would indude an assessment of whether statis

methods were appropriate. Was the evidence 

the stocfy relevant? Evidence is relevant if it ha

ical relationship to the asragnment issues. The 

evaluator must also detennine if the evidence u

the stocfy is competent—valid and reliable. Usi

criterion, the evaluator should independentiy a

the stodies to be included. If such an assessme

not made, it must be stated in the bocfy of the 

This identification and control of bias require

part, an understanding of how variations in sto

methodology may influence resulte. For instan

Wortinan and Yeaton (1983) were carefiil in t

thens of stodies on coronary bypass surgery 

indude both randomized and quasi-experimen

stocties. The two seto of stodies produced mai

different estimates of the effect of the surgery

investigation set out to account for the gtv in 

ings of the two sete of stodies. They conduded 

although the randomized experimente led to a 

different estunate than the quasi-experiments

part of the gap between the two estimates wa

utable to biases in the randomized stodira. So

patiente were randomfy assigned to have med

rather than surreal treatment, and the evalua

were able to recount for a source of bias. 

Fage 27 GAO/FBBID-IO.1.2 Evaluation 
Chapter 8 

Developing the Syntheaia 

As the example above shows, n^enever possible t

evaluator should seek stocUes that use a variety of 

methods. Variations in stocfy types may control bi

and prove helpfiil in accounting for discrepandes 

stocfy findings, leading to more accurate answers 

the cUent's questions. To illustrate <a^Bahi, suppos

congressional committee wanted td?find out, first

how many people have been victimized by violent 

crime in each of the past 5 years and, second, how 

many of these victims have recreived services from 

programs providing aid to victims of violent crim

answer the first ciuestion, stocties might have used 

variety of methods. For instancse, spme stocUes m

be based on poUce reporto, vi^ch Uatiii to undere

mate the number of crimes because many crimes 

unreported. Other stocties might have used surv

a sample of peopte selecited at random &om a def

populaticm. But, among a nuinber oi^problems su

stocties may have, the popidations mijght have be

defined locaUy (so that all the people In a ^ven d

were equaUy Ukefy to be siurveyed) and since loca

crime rates vary, variations in estimates may refl

variations in local crime rates. This^i^xanq;>te imu

lines the importance of enlisting a reipresentative 

sanqile of stocties and stody types sp^that the eva

tion synthesis as a whole does not take on the bi

single stocfy type (Cordray, 1990). 

If the congressional comnUttee were interested in 

finding out how many people are receiving aid to 

tims of violent crime, there are agaMlliwcj^en

ctifferent ways individhial stucties nuy be de^^ ^ 

provide an answer. One method, forexample, is t

identify aU govemment programs providii^ aid t

victims of violent crime, to retrieve crKaluative inf

mation on these prograins, and to dcihre from th

reconis a coimt of pec>pie recehongjiid. A second 

method is to consult victim surv^cioncemiiig v

lent crime as to whether people received govenu

aid. Again, the k^ pohit is that hi conducting a 

synthesis, one should indude both Idnds of stocti

Fnge28 6AO/FE1ID>10.1.2 Evalnation Syn
Chapters 

Developing the Synthesis 

they are avaUable. The two methods may hav

biases, and unless both are included, the synt

takes on the bias ofthe incUvidual stocties pro

dato for it (Cooper, 1986,1988). 

A common criticism of narrative research sum

is that they are not objective or that th^ are t

impressionistic. More-quantitative efforte, by 

trast, should more "ob|ectivefy" synthesize th

able evidence. But if many stocties of a particu

treatment, such as a certain method for treati

breast cancer, are available, a reviewer must 

vi^ch to include. Several options are avaikdil

decision must be made earfy in the review pr

The sunplest option is to hidude every availa

stody: published and impublished reporte, do

theses, academic stodies, and contract resear

stocties. When a reviewer has no prior hypot

wante to explore broacUy what is known abou

ment, including such ctiversity may help. Sde

precision is less important than identifying in

trends, pattems, or outUers. 

But an evaluator faces ctifflcult trade-offs in a

to track down and hidude everything. For ex

it is dear that a certain stocfy is fimdamentaU

say with obvious numerical errors, it is hard t

for ite indusion. Wrong information isnot be

no information. Another example is that the d

about a treatment may have changed over ti

biducUng very old stocties, even if they wiere w

done, when the question driving a review is h

the treatment currentiy works, is foolish. A co

illustration comes firom Wortinan anci Yesfon

pooUng of data from randomized trials of c»r

artery heart bypass surgeiy. The survival rat

risen dramaticaUy as supngjic^l techmque lti» 

If this improvement is quite obvious, do we w

include very old stocties? Probabfy not, but th

Fage 29 6AO/FE1ID-10.1.2 Evalnation 
Chapters 

Developing the Synthesis 

decision wiU depend upon the specific poUcy 

problem. 

A second option is to use a panel of experte to 

erate a list of stocties for inclusion. Hauser-Cra

Shonkoff (1986) used this approach to choose 

for their review of the effectiveness of earfy int

tion programs for young chUdren with cerebr

developmental delay, and Down's syndrome. A 

of pubUshed Uterature yielded hundreds of sto

that had some potential for inclusion in their s

maiy. They used experts to sharpfy narrow the 

candidates. A quick caution here is that somet

experts pay more attention to large stodies of m

quaUty than to weU-designed, smaUer stocUes. 

bias should be controUed for. 

Organizing and 

Implementing a 

Reviewing 

Strategy 

Given the substantud number of evaluation sto

that concem a topic of interest, some wiU prob

have focused exclusivefy on the topic, while fo

others, addressing the topic may have been on

ondaiy stody purpose. Some stucties, as discus

previous section, are Ukefy to have similar typ

designs wbSle others wiU have differed on desi

and therefore alsp on the types and sources of 

As a group, it is likefy that the stocties wiU hav

in the soundness or rigor of procedures and ex

and perhi^s even the appropriateness ofthe d

Wlule it is important to include different types 

stocties in the evaluation synthei^, what does 

evaluator do with stocties that vaiy hi quality? 

question that has provoked heated debate. A c

issue in this debate is what constitotes a "good 

stocfy." It seems reasonable that all stocties ind

a synthesis should be assessed against basic st

for research design, conduct, analysis, and rep

Thus, the evaluation synthesis requires an ass

of the overaU soundness of each incUvidual sto

Fage 80 6AO/FBMD<10.1.2 EvaluationS
Chapters 

Developing the Synthesis 

Miyor weaknesses of stody design, conduct, a

or reporting that affect the reliabiUty or vaUct

each stody's findings must be identified and c

ered in using the stocfy and plachig confidenc

stucfy findings. Whether experiment, case stoc

survey, or content anafyas, each stody should 

questioned as to ite reUabUity and vatictity. Qu

such as the foUowing wiU detemune the overa

ness of the uictividual stocfy to the evaluation s

thesis: 

Are the stocfy's objectives stated? Were the ob

appropriate with respect to the developmenta

of the program? 

Is the stocfy design dear? Was the design app

given the stocfy objectives? Was the inctic:ated 

in fact executed? 

Did the variables measured relate to and adeq

translate to the stucfy objectives and are they 

priate for answering the cUent's questions? 

Are sampling procedures and the stocfy samp

cientiy described? Were th^ adequate? 

Are sampUng procedures such that poUcyma

generalize to other persons, settings, and tim

interest to them? 

Is an analysis plan presented and is it approp

Were datapcoUector selection and training ad

Were there procedures to ensure reliabiUfy ac

date coUectors? 

Were there any inadequacies hi dato coUectio

dures? 

Were problems enc»untered during dato coU

that affect dato quatity? 

Are the statistical procedures weU specified a

i4>propriate to the task? 

Are the conclusions supported by the date an

analysis? 

Are stody limitations identified? What possib

founds the interpretation ofthe stody fhiding

81 OAO/FEMD-IO.1.2 Evaluation 
Chapter 3 

Developing the Synthesis 

This Ust shows some of the issues that should be 

raised m reviewing the stocties. The infonnation 

derived by answering these questions should lea

an overaU judgment of the usefiUness of each sto

does not mean, however, that stocties with desig

other weaknesses are automaticaUy exduded fh

synthesis. Instead, if such stocties are mcluded, 

ment should be made about the confidence that 

be placed in the stocfy findings in relation to oth

stocfy findings. 

Of particular concem, however, is the consisten

reliabiUty of judgmente of stocfy quaUty. In a syn

thesis, for example, Stoc^ et al. (1982) had cod

judge a random sample of 30 primary research 

mente. Among the items requiring a coding dec

was one global item caUed quaUty of the stocfy. 

(^rrelation coeffidente among the cKKlers were 

acceptable with a mean level of .52. The stody s

geste strategies for improvuig reUabiUfy, induct

summing ratings across methodologicid variabl

superior to a shigle global item rating), coder tr

and retraining, and group rather than incUvidua

mente of quaUty. 

At a minimum, the issue of coder reUability sho

raised in the evaluation synthesis. It seems reas

to describe steps taken to address the reliabiUfy 

or as several GAO evaluation syntheses have do

describe the strengths and weaknesses ofthe st

that led to a summaiy judjgment of quaUty and u

A report synthesizing stodies on specnal educat

example, included the actual review of each stoc

technical appencUx, making the basis for the ju

available for each reader to assess. 

Fage82 6AO/PEMD-10.1.2 EvaluationSy
Chapters 

Developing the Synthesis 

Redetermining 

the 

Appropriateness 

ofthel^thesis 

Method 

Is the available research sufficient to answer 

cUent's questions? In developing the evaluat

thesis, it is useful to classify each stody or da

that is to be induded te the synthesis accordin

both the questions in the stocfy firamework tha

addresses and the stocfy design. This procedu

ensures that aU stocties to be hiduded in the 

are relevant, and it quickfy shows commona

weU as information gaps. 

Sometimes, although preliminary evidence ap

suffident, it may simpfy not be possible to ans

dient's question usiiig evaluation synthesis. F

exampte, a GAO report coiiected a number of 

attempting to estimate the size of the iUegal a

ulation in the United States. However, the ran

estimates was enonnous. It was pc»ssible to id

biasuig factors in some cases. One household 

conducted in Mexico, for instance, quite dear

underestimated the number pf Mexican citize

had iUegaUy emigrated to the United States. W

stody put a lower bound on the trae value, th

ofthe remaining stodies was so questionable, 

resulte so discrepant, and potential explanato

tors so numerous ui relatipn to the number o

available that the evaluators conduded that a 

new research effort rather than evaluation sy

was required to answer the question. In this i

the main use of ^nthesis was to help identify 

and what research waaneeded to uncover Un

features requisite for the design of such resea

There is a danger that a methodology that sol

tain thorny problems of emptied research wiU 

more than it can deUver. Evaluation synthesi

exception. The purposes of redetermining the 

priateness of the synthesis method are the fol

1. To clarify information-user expectations bd

evaluator becomes hivolved in the detaUs oft

synthesis itself. 

Faga 38 OAO/FEBID-IO.1.2 Evalnation 
Chapters 

Developing the Syntheaia 

2. To enlist the coUaboration ofthe dient hi 

addressing likefy difficulties in the work. 

3. To prevent months of labor being wasted vr

synthesis is unlikefy to meet the cUent's mform

needs. 

4. When qmthesis is found in84>propriate, to fo

and systematize the process whereby new rese

recommended on the basis of gc ^ in past kncy

5. If syntheds is found appropriate, to sharpen 

understanding of research questions just prior 

immersion in the details of the woric 

A n EiXail^le '^^ Ulustrate the steps hi the evaluatic>n synthe

cess discussed in this cduipter, let us consicter 

was done hi a GAO stocfy of the Special Supple

Fcxxl Pragram for Wonieit, Infaiite, and ChUdr

WIC ((JAO, Januaiy 198^. The U.S. Senate C

mittee on Agriddture, Nu^tion, and Forestiy 

GAO to synthesize aU avaUable evidence aboa

This program, fimded at pver $1 biUion ayear

vides nutrition supptemeiito to i^proxhnatefy 

Uon people each yirar. These people are pregn

women firom Ibw-income famiUes and childbraa

birth to age 5 in Icyw-lncpme temiUes who are c

ered at high nulritioiud i^Mk. The Senate c»ini

request was motivated ly the sharpfy confUdin

mony that it received about WIC's effectiveh<»

witaesses argued that it was a highfy effective 

gram and that it had dear positWe effecte in 

increasing chUchien's birthweight, reducing fet

neonatal mortaUty, improving nutrition In mp

and chUdren, and reduchig mental retardation 

dren. Other witaessro arigued that there was n

crete evidence for these positive assertions. T

testified that ¥Me it seemed hud-hearted to o

the distribution of fopd vouchers to Ipwrhicpm

mothers, the facte did not support assertions t

Fage 84 OAO/FEIID*10.1JI Evaluathm S
Chapters 

Developing the Synthesis 

women or tiieir chUdren benefited m any conc

way. 

The questions that were finaUy agreed to as b

most relevant to the committee were as foUow

1. Does participating in WIC affect birthweigh

2. Does participating m WIC prevent miscarr

StiUbirths, and the mortalify ofthe newbom? 

3. Does participating in WIC affect the health 

nutrition of pregnant women? 

4. Does partidpating hi WIC affect the incide

anemia in infante and chUdren? 

5. Does partidpating in WIC affect the incide

mental retardation ui uifante and chUdren? 

bl proceeduig to identify and coUect the relev

verse of documente that possibfy provided ins

the aiBwers of aU or any of these questions, t

evaluators cast as broad a net as possible, md

agency bibUogr^hies; joiimals; discussions w

many professionals in ttte fieki, among them 

nutritionisto, health professioiuds, and resear

and an iterative iiiaUout pjf a Ust of document

experte requesting additipns as appropriate. 

documente were identffied, some containing m

than one evaluation stucfy report. FYom their 

reading ofthis set of documente, the evaluato

54 to be relevant because they contained hifo

pertaining to one or more ofthe evaluation q

posed above. The evaluators then identifed, 

these 54 dcM:umente, 61 stocties to be hiclude

synthesis to be perfprmed. 

The evaluators then developed a reviewing st

that mduded establishfaig a idne-point scale t

used by expert reviewers to rate the crectibUif

Fage85 GAO/FEBID-IO.1.2 Evaluation
Chapters 

Developing the Syntheaia 

Stocfy for each question. The reviewers used a s

criteria regarding the soundness and appropri

of the methcxlology underlying each stocfy's fi

and then assigned a numerical rating, bi this w

each stody was judged to be somevriiere on a s

between high and low crecUbUify. Each stocfy w

by more than one evaluator. This review resul

the matching of stocties to questions and ted to 

like the foUowing. 

Question 1: effect on birthweighte, 39 relevant 

stocties, 6 of hi ^ or mecUum crecttbUtty and 33 

crectibility; 

Question 2: effecte on mortalify, 12 relevant st

of hi ^ or mecUum credibUily; 

Question 3: effecto on matemal nutrition, 24 r

stocties, 6 of high or mectium credibiUty and 1

credttiilUy. 

These steps tiien led to the pohit where the syn

of infonnation avulable for eacdi question cou

ceed, if at least some stocties of high or medium 

bUify had been identifiecL These procedures w

discussed in the next du^iter. 

V 

: ; • 

•.. Fage 86 

. . .-. 

OAO/FEllD.10.1Ji Evaluation S
Chapter 4 

Performing the Synthesis 

Given a set of stodies that have been hictividuaU

assessed and deemed usable for the synthesis, th

next steps are to implement the evaluation synth

and check for problems and to present the find

These are discussed in this chapiter. The questio

How are the different stodies compared? Tiiere 

standard approach, but two nuyor factors wiU in

ence how the stodies are compared. First, differ

evalmdive questions are Ukefy to reqiure differe

approaches for synthesizing the infonnation and

second, the nature ofthe stocfy ctesigns wiU limi

possible analyses. 

As mentioned previousfy, the question that mot

the synthesis m large part drives the spedfic pr

dure used to synthe^ze. For example, in examin

how weU a program is woikmg, the targeted qu

might be, Who does the prpgram serve under id

circumstances? Altemativefy, Who does the pro

serve on the average? In the first instance, the 

evaluator might want to hxvestigate a number o

stocties and provide a narrative description of th

ings, bl the second instaiice, the evaluator migh

the arithmetic average ofthe answers given by t

uictividual stodies available or might express th

answer as the range between the highest ahd lo

estimates. A problem here is that, sbice the eval

synthesis is employed to answer quMtionsi^^ 

existing infonnation. the evaluator wUl notoffcen 

the ideal quantitative anafyas possible. 

As with the discussion on what stodies to indud

the synthesis, this is an area where considerable 

ature existe. The Uterature assumes for the mos

however, that the stody designs are experiment

least quasi-experimental hi nature, which may, 

course, not be the case, ((^per, 1989; Hedges

1986,1988; Hedges and OUdn, 1982,1985, and 

1986) 

Fage87 6AO/FEMD-10.1.2 EvaluationSyn
Chapter 4 

Performing the Synthesis 

This chapter discusses both quantitetive and 

titative approaches to evaluation synthesis. 

Quantitative approaches are ideal for certain 

questions but nonquantitetive approaches ar

most evaluators wiU have to wrestle with wh

responding to questions driven by poUcy. 

Quantitative 

Approaches for 

Evaluation 

Synthesis 

The Uterature describes two basic quantitati

approaches for synthesizing the findings of e

mental or quasi-experimental stocties. These 

approaches, detaUed in the foUowing section

computing an average effect size and (2) con

combined significance test. It may be relativ

uncommon to use these specific techniques i

work because of the character of the questio

as weU as the disparate, fragmented nature 

evaluations. Quantitative approaches are, ho

powerfiil tools when the basic assumptions 

met. (Bryk and Raudenbush, 1988; Green a

1984) 

Computing an 

Average Effect Size 

The key descriptive statistic that Glass (198

employed in his pioneering synthesis is the 

When one compares a treatment to a contro

common definition of effect size is simpfy th

ence between the two group averages, expr

terms of the control group's standard devia

To iUustrate, suppose we were stodying two 

teenagers, one group receivmg a certain ty

training and the other receiving none. After 

the job market, each person in both groups 

about his or her mcome. If the average annu

for the group that received training is $10,5

the average for the group receiving no train

$10,000, with a standard deviation of $ 1,00

effect size for this program is simpfy 0.5, or 

standard deviation. There are several elabor

Fage 88 6AO/PEMD-10.1.2 Evaluatio
Chapter 4 

Perfonning the Synthesis 

this basic idea, some of which incorporate the 

treatment group's standard deviation and other

are based on the idea of change over time. Our 

example provides a woridng defimtion that is 

congraent with Glass's extensive work. (Coldite 

1988) 

Assuming that an effect size is reported (or can 

computed) for each of several stocties, the avera

effect size for the entire set is easUy calculated. 

important aspect of computing an average effec

is that it provides a single siimmary value for an 

area of stucfy: "Most c>f cnir work is aiined at sam

and sweeping generaUzations that sticdc in the r

memory. If what ah integrsdive anal^is slipws c

be stated in one uncjompliicated sentence, then i

message wiU be lokphaUi^iitafewspecJUdiste" 

(Glass, 1978, p. 3). Fpr escampte. Glass andi Sm

(1976) cpmputed the average effect size for psy

tiierapy across 400 sepaiaite stocties to be .68. 

ccmcluded that, pn the average, psychc>iherai^ 

efidal, smce "the average person recieiving som

of psychotherapy was abdiit two-thirds standitfd 

ation more unproved on ah outcome measure tl

average control group member" (Glass, 1977, p

363). 

Fage 89 6AO/FEJMD-10.1J Evaluation S

Effect size averaghig requires that we know the 

means and the control group staiuiard cteviatkH

mating an avenge effect libte is inc>st d^^ 

iiriien a gnmp c>f stucfy c>utcomes seem heatfyir p

normaUy, distributed arpiind their mean, bi t ^ 

an average gives a useful idiigle summaiy Of 

But-wiien stocfy outcomes appear to confUd,p

an unusual distribution, a single average Is less 

(Feldman, 1971; Guzzo et al., 1987; Hedges, 19

1984; Hyde and Unn, 1988) 
Chapter 4 

Perfonning the Synthesis 

Conductinga 

Combined 

Significance Test 

The relationship between sample size and the 

of a statistical test is weU known: the larger t

sample size, the more likefy that a certain eff

be detected as statisticaUy significant. For ex

an observed difference of 10 IQ pointe betwe

Start and non-Head Start chUdren may not b

caUy significant with 10 children per group; 

this same 10-point difference can be highfy s

with group sizes of 100. 

When there exist a number of stodies on the 

topic, the various smaUer date sete often can 

pc»oled into a single oveiaU anafy^. This inc

effedive sample size and witt dramaticalfy un

the power of statistical teste. Ihis iq>proach 

ciaUy appealing ^e n rampte sizes of individu

stocties are smaU. Suppose we have several S

investigating the effeciiveiiess of highfy strac

versus less-stractured cuniculums. AU the st

may tum up concordant resuhs, wtihout any 

mctividual fmcUngs reaching statistical signif

Yet an overaU test on the pooled date may sh

significant resulto. 

When multiple independent stodies aU comp

treatmento that are simUar across stocties an

group differences are tested statisticaUy in e

instance, one strategy for dra'wing a smgle "

conduslon from these resulte involves comb

separate significance testfi into an overaUtes

commcm nuU hypothesis. This is generaUy th

treatment groups have the same population 

A number of procedures using this idea have 

suggested. Rosenthal (1978) summarized m

them and provided guidelines as to when th

likefy to be most usefid. To iUustrate one tec

we take the method of adding Z scores (stand

normal deviates). If two groups are compare

stocfy, there is a Z score associiated with eac

p value. The Z's are added across stocties, an

Fage 40 OAO/FEliD-lO.1.2 Evaluath

?.' 
CJhapter4 

Ferfomdng the Syntheaia 

sum is divided by the square root of the numb

stodies that are combined. The probabUtiy va

assodated with the resulting overatt Z score p

the level of significance for the combined stat

test (See Rosenthal, 1978, for a detaUed expl

and computational examples of other, concep

^mUar techniques.) 

A strength of the combmed significance teste 

conditions for their use can be met is that th

aUy accomplish the goal of increaring power

thal added the caveat that the stodies should 

tested the same directional hypothesis.) We c

trate this approach by assumhig that curridd

more effective than cuiriciihim B but that the 

ference for large poptiilatiqiiis is smatt. If A an

repeatedfy compart using smaU sanq>les, o

eqpect to find, on the avdiage, small differen

favoring A. But many ofthe ctiifferences woul

statisticalfy significant. Ah informal review of

research might condude that the effect is not 

calfy reliable or that the pjuralily of stodies ifi

difference at aU. Howinrer, if the stodies are c

(for example, by adding Z scores) the overall 

tical test is much more likefy to be significan

bl general, technique for conducting a comb

significance test seem most usefiil when the s

StocUes can be considered independent and e

random samples, estiiMUting a staigle "true" d

between populations, so that variation amoi^

outcomes is attributabteip chance. In this Ca

the treatmente are infacst cUfferentiBlfy effecti

overatt comparison witt often detect this diffie

because it hicreases the efifective sampte ^ 

the test When the variatipn among outcomes 

ferent stucties cannot be attributed simpfy to 

variation, however, the csombined dgnificanc

less usefiil. The overaU test wUl stiU provide a

"answer" as to whether or not the common nu

Faga 41 OAO/FEIIIVIO.1.2 Evalaatlcm 

..w-.-^fi'-i^rii,-
Chapter 4 

Perfonning the Synthesis 

hypothesis should be rejected, but a single answ

may not be a useful representation of reaUty. 

A key pomt is that since many separate stocties a

combined mto one "big test," ite use should be p

ceded by efforte to detennine if the variation m o

comes can be viewed as random. This is a cracia

In c:ases where conflicrte exist, an analyst may ch

to use other techniques that are more sensitive t

ation among stody outeomes. (Biyant and Worti

1985; BuUock and Bvyantek, 1985; KuUk and K

1986,1988; McGaw, 1988; White et aL, 1986) 

Special Problems 

of Quantitative 

Synthesis 

This discussion has focused primarUy on statist

procedures: computing effect sizes and conduct

significance teste. The foUowing three issues als

come up in most quantitati've evaluation synthe

Different Outcome 

Measures Across 

Studies 

(Tombuung stodies is easiest when they aU use t

same outcome measure. But given the diverse p

ties and resources of different researchers, sucd

formity fis extremefy rare. Take day care as an 

example. Investigators have used various cogni

physical, health, scxnal, and emotional hidexes 

assess ite effect on participating chUdren (Belsk

Steinberg, 1978). 

When outeome measures differ, the reviewer fa

dUemma. Is it reasonable to combine across see

ingfy different measures? The problem is not pr

marUy a technical one. Whenever means and sta

deviations are available, effect sizes can be com

and averaged. Whether or not to do so is a subs

question. The answer is ultunatefy ctictated by g

sense rather than any rote formula. The key issu

conceptual clarify. Suppose a review of day-car

ings hidudes coffiitive measures for 3-year-old

some stocties and emotional measures for 6-yea

Page 42 GAO/FBUD-10.1.2 EvaluationSyn
Chapter 4 

Performing the Synthesis 

Ul others. Then the reviewer must decide wh

overaU quantitative summary wiU be usefid a

substantivefy sound. Just throwing together 

measures because the title of each stucfy ccm

teim "day care" can be fppUsh, no matter ho

caUy elegant or precise the review. (Anderso

1983; Bayarri and Degroot, 1987; Bredderm

Fienberg etal, 1985; Hall etal., 1986; Hhit

1986; Stemkamp and Maehr, 1984; WUlson

Multiple Measures A sec»nd issue is how to tireat studies that r

Within Studies more than one outcome. Take clay c»re agam

pose some stodieis c^ompare day-care ahd ho

children on berth cognitive and socud cle^ne

with several measures ^i»ch, while other i

on onfy a smgte iniiex. Hc^ should we bala

respective c:cmtribikibhs in a review? 

One way is to ccmipute a separate effiectsli

measure witldheadhis^ ^ 

dren m da y care t o hbme;reared c^ ^ 

ferent c>utc(mies thenccK^^ 

the review. This apE»rcKeu^ 

analysis to each comparison ratherthaii t^^ 

It use s all availtdpte infonnaticm. But pe^ 

teiuted con»eiviei^]is thiat stucties^vnth^^ 

sures wiU be weighted mc^e heavify thimth

onfy one or tviro. AliK>r8eyeral ccrni^poris^^ 

stucfy are not ihcl^nci^ . Th ^ were dbheb

hxvestigater pn bite cpra^ partidpaiite^^

lead to .repeiriied'bias;>''C'.';'.':' 

One solution is to categprize outeomes by y

measure—such as eihptidiial, sociid; or i^ ^ 

abUities--and titen;COiH^^ separate ianaly

subgroup. Hcyvi^vi^/stocemany iB(^ ^ 

than cme cogpnitive measure, or eihotipil^

this might not ahvaQi« be sufftelent 

FBge48 GMfl/ViU^ 

•''• • ^ -• '.•'.•'•'•''•'•.• '•...\i-..'-V...S.-.-;. .!'-.|.;'.i.-.•;v-vi'' : ^.'-^^f.^:^.^^;;•^:^:••^^:'^:-:r:
Caiapter4 

Perfonning the Synthesis 

A second solution treate each stody as the unit o

ysis and gives each stocfy equal weight. It mvolv

computing a "grand" effect for each stocfy by av

aging across the several measures (for example

and KuUk, 1982). This way, each stocfy rather th

each comparison gete one "vote" in the review. 

trade-off here is loss of information within stocti

We recommend foUowhig and reporting both pr

dures. This wiU expand a final report. But since 

aging within stodies requires computing effecte 

individual comparisons anyway, presenting both 

yses raises costo minimalfy. Doing both aUows a 

reader to explore any differences between analy

For instance, suppcMe a large average effect siz

emerges firom a summary using each cpmpariso

imit of analysis. Then we can ask ^iiether such 

ings depend undufy on one or two stocties with 

tiple measures. (BangertpDrovms, 1986; Becke

Hedges, 1984; Eysenck, 1984; PUiemer, 1984) 

Missing Numbers A quantitative review is impossible unless stodi

report the necessary statistical hiformation. Da

requiremento for computing effect sizes are min

AU we need are means and standard deviations 

exact test statistics such as t and sample sizes. Y

surprising how often this hiformation is unavail

For example, one analyst recentiy looked at 24 

stocties of day care's effect on chUdren's inteUe

development. Over half did not report suffident 

mation for computing simple effec:t sizes. 

What are a reviewer's options ^e n confionted 

missing or insufiBcient date? One fis to tiy to obt

missing infonnation directfy ftom authors if tim

resources pennit. Since the statistics needed ar

basic—means and standard deviations—one wo

exped such efforte to be successfiU. The chance 

success probabfy depends quite idiosyncraticaU

upon the field, the investigators, and other facto

Fage 44 OAO/FEIID-IO.1.2 Evalnadon Sy

K-
Cliapter 4 

Performing the Synthesis 

such as how dated the stocties are. (Becker, 

Hedges, 1986; Tanur, 1985; Wolf, 1986) 

A second strategy is to fiU in conservative es

effect sizes vriien stocties have missing date. 

this means assigning effed sizes of zero. We 

know the treatment effect when statistics ar

So if we plug in a zero, we are assuming min

treatment effect. If, despite this poUcy, the r

^ows the treatment to be effective, we can b

dent that this overaU ccmclusion would not c

even if missing statistics were available. 

This seemingfy conservative strategy, howev

always conservative. It ctepen^ upon your p

view, bl some cases, such as research on the 

day care or reduced cost reimbursement for 

ization, finding no effecat of the new program 

ha4>py outcome. We may not expect day car

IQ's or to make chUdreh tisqppier; we are sa

sunpfy does no harm. We rarefy expect redu

to unprove healtii; the goal is to not do signi

harm. In such cases, plug^ng in conservati

tics may bolster such an optimistic c»nclusi

fiabfy. 

When effect sizes cannot be extracted firom 

stocties, and ^e n efforte to get this hiforma

dhrectfy firom authors fedls, it makes sense to 

quantitative analyses on the subgroup of stii

good information. Basmg analyses on datet

firm increases confidence in the review as a 

Table 4.1 iUustrates the concepte so far. Frt>

WIG report, tt shows how the resulte of the W

stocties, finaUy used in the synthesis, can be 

to faicrease confidence hi the fincUngs. 

page 45 OA0/FBBID>10.U Evahiatio
Chapter 4 

Perfonning the Synthesis 

Table 4.1: Mean 

Birthweight Quamttathre 

Summary 

Stody 

Kotelchuck 

Metcoff 

Stockt)auer 

Silverman 

Bailey 

Kennedy 

Year and location 

1978. Mass. 

1980-82. Oklahoma City 

1979-81. Mo. 

1971-77, Allegheny Cou

Pa. 

1980,2 Fla. counties 

1973-78. Mass. 

Summary 

Average 

Weighted average" 

Range 

Lowest 

Highest 

Fnge4e OAO/PBIID-IO.1.2 EvaluationSyn
Chapter 4 

Performing the Synthesis 

Reported birthv 

grama' 

WIC 

3.281 

(4.126) 

3.254 

(238) 

3.254 

(6.657) 

3.189 

(1.047) 

3.229 

(37) 

3.261.4 

(897) 

velghtin 

Non-Wic 

3.260 

(4.126) 

3.263 

(172) 

3,238 

(6.657) 

3.095 

(1.361) 

3.276 

(42) 

3.138.9 

(400) 

Quamitatlve Indicators 

Raw difference 

21.0 

di.o*= 

16.0 

94.0 

-47.0 

122.5 

% difference" 

0.6 

2.9 

0.5 

3.0 

-1.4 

3.9 

Stet

sign

Marg

Yes 

Yes 

Yes 

No 

Yes 

3.244.7 

3.257.8 

3.195.1 

3.225.9 

49.6 

31.3 

1.55" 

0.9/* 

3.189.0 

3.281.0 

3,095.1 

3,276.0 

47.0 

122.5 

-1.4 

3.9 

^ e numbers in parentheses are sample sizes. 

"Raw difference divided by non-WIC birthweight. 

'^Adjusted. 

''Average raw difference divided by average non-WIC 

"Each mean is weighted by the number of participant

its group and an overall average is obtained by divid

number of participants or controls in the six studies. T

ence is based on the total of participants or controls. 

Source: U.S. General Accounting Office. WIC Evaluat

Some Favorable but No Conclusive Evidence on the 

Expected for the Special Suppiernental Program for W

and Children. GAO/PEMD-84-4 (Washinoton. D.C: Ja

1984). p. 16. 

Fage 47 OAO/PEMD-10.1.2 Evaluatio
Chi4tter4 

Perfonning the Synthesis 

Nonquantitative 

Approaches in 

Evaluation 

Synthesis 

Many evaluation stocties do not meet the assum

or contain sufficient uiformation to aUow the us

the stetistical approaches desc:ribed above. Cas

stodies and other kinds of infonnation have ofte

been avaUable for synthesis. There are at least 

types of information valuable to evaluation synt

for t)^ch the statistical approaches described a

are not appUcable. The discussion that foUows 

these five types of information, describes gener

situations in induch this information should be s

sized, and outlines some guidelines for incorpo

such hiformation. 

Five Types of 

Infonnation 

The five types of mformation potentiaUy valuab

the evaluation synthesis that are not suitable fo

tistical analysis are (1) single case designs, (2) 

quantitative aggregate stocties, (3) nonquantita

information in quantitative stodies, (4) expert j

mente, and (5) narrative reviews of coUections 

research stodies. We wiU review each type of in

tion in tum. 

Shigle Case Design DetaUed stocties of shigle cases are common, a

techniques for analyzing such infonnation hav

developed (Herson and Barlow, 1976; Kratoch

1977,1978). Observations of smgle individual

contributed heavify to the theories of Freud, Pi

and Skinner—among the most influential psycd

gisto of modem times. Dukes (1965) and Herso

Barlow (1976) presented many examples of "N 

research in psychology. Case stocties are also f

quentfy used in pubUc poticy analysis to exami

effecte of nonexperimental evente such as poUt

decisions by cities and towns (Yin and Heald, 1

The term "case stody" can refer to the stody of 

single event or desegregated stodies of multiple 

evente (Kennedy, 1979). Even if a case stody u

quantitative outcome, it is not possible to comp

Fage 48 GAO/FBMD-10.1.2 Evaluation Sy
Chapter 4 

Peif orming the Synthesis 

effect size m the traditional manner. If each incU

is viewed as a separate stody, there is no direct 

measure of within-group variation and no contro

group. Many of the stocties used in the GAO syn

on specnal education were case stocties of local s

districte. (Cuitis and Shaver, 1987; Sabsbeig et a

1987; Sampson et al., 1987; Scruggs et al., 1987

Slavui, 1986; Strobe et aL, 1985) 

Nonquantitative Some research areas have important outeomes t

Aggregate Stocties are difficult to measure objectivefy or numericaU

cUnical psychologist may report that obese peop

usuaUy show general life unprovemente after we

loss or that hypnosis is effective hi helphig canc

patiente ai|iiist to chemothen^y. WhUe an impU

baseUne must exist, the benefite may not have b

assessed with objective teste. In fact, an investig

may beUeve that the psychological effecte of wei

loss or hypnosis cannot be accuratefy assessed w

shnple numerical measurement. A reviewer of su

stocties may stiU want to hiclude these nonquan

msi^te. 

As ZimUes pohited out, this problem is particula

c»inmon in evaluations of csomplex prograins: 

"Most programs for clilldren, espedally educational progra

aimed at produdgg a mulUiriicWy of outcomes. As already n

many of the psycholpgical characteristics they are conceme

fostering-whettier it be ego strength, or resouroefubiess, or 

problem sohing abiUly-aie difficult or tanpossible to measu

espedally within the time and cost constraints of an evaiuat

8tu4y. The usual response to this ditemma is to sift through 

rostor of multiple outcomes and single out for assessmott

most important ones, but those that are capalde of being 

measured" (Ztaniles, 1979, p. 7). 

Here an evaluator is faced with a tradeK>ff betwe

precision and meaning. Organizing a synthesis f

evaluators to confiront a similar dUemma. Which 

outcomes appearing m the stodies should be hic

Fage 49 OAO/FE1ID-I0.1.a Evalnatlmi Syn
Chapter 4 

Performing the Synthesis 

in a synthesis? If they decide not to refy exclusiv

quantitative measures, they must figure out how 

incorporate nonquantitative evidence. 

A related situation occurs when quantitative sto

do not contain sufficient infonnation for statist

synthesis. For example, weak experimental des

may include a quantitative assessment. The rea

performance of a group of chUdren may be ass

with a standardized test foUowing a special tot

session. But without a comparison group, an e

size cannot be computed. Other stocties compa

treatment group to a control but do not report 

cient information for producing a statistical su

Many of the stocties included in various GAO s

tiieses faU mto this categoiy. For example, in 

blocdr grant report, admmistrative coste were c

lated before and after program consoUdation, b

calculation of comprehensive and reliable estim

effect was hindered by differing definitions of 

administrative activities and other accounting 

dures, uiadequacy hi date coUection procedure

weakness m sampling. These characteristics o

stocties led to a choice of either omitting them 

treating them hi some nonquantitative manner

(Becker and Hedges, 1984; Cariberg and Wal

1984; Cariberg et al., 1984; Center et al., 1986

Skivhi, 1987) 

Nonquantitative 

Information in 

Quantitative Stodies 

In preparing a stocfy report, researcdiers and e

tors do not sunpfy list numerical resulte. llie t

ment and participante are carefiiUy described; 

or limitations are painstakuigfy laid out. Often 

effort put mto these nonquantitative descriptio

surpasses that involving the numerical inform

is not always either appropriate or desirable to 

a stocfy to one or several numerical indexes. N

may not accuratefy be interpreted without taki

account factors such as subjec;t attrition, chaun

Fage 50 GAO/FEBID-10.1.2 Evaluation S
Chapter 4 

Perfonrdng the Synthesis 

Stucfy procedure, and a variety of unexpected or 

otherwise noteble happenings that become major 

stocfy Umitations. Most evaluators wiU need to m

infonnation in the evaluation synthesis that goes 

beyond numerical outcomes. (Chipman, 1988; C

1988; MuUen and Rosenthal, 1985; Stanley, 198

Expert Judgment An evaluator may choose to mclude expert ophu

earfy stages ofthe synthesis, such as m evaluatin

incUvidual stocties. Instead, tiie evaluator may wa

systematicaUy compare stocties relying on expert 

judgmente about program effectiveness. Syntiies

should be able to incorporate these inpute. 

Narrative Reviews of 

CoUections of Research 

Stodies 

As Cook and Leviton (1980) have pohited out, a 

careful narrative review, expUcit about ite analyt

procedures, can be extremefy valuable. Narrative 

reviews of coUections of research stocties may fr

quentiy, for exunple, identify methodological 

weaknesses of certain broad types or groups of 

stocties hi a particular topic area. The evaluator w

need to consider these pointe in deciding whethe

not to include these stocties in the synthesis and, 

they are included, in hiteipreting findings from th

stodies. (NobUt and Hare, 1988; Wachter, 1988; 

White, 1987) 

Indications of the 

Need for 

Nonquantitative 

Approaches 

Under special circumstances, nonquantitative 

sqpproaches to the evaluation synthesis are partic

larfy appropriate. Four are n^en (1) treatmente m

be inctividual or more concemed with process th

outeomes, (2) program effecte are assessed acro

multiple levels of effect, (3) uncontroUed treatme

groups are compared witii the treated control gr

and (4) the 'Vrong" treatment is stodied. 

1. Treatmente may be hictividualized and focused 

process objectives. Some educational and social 

Page 61 6AO/PE1ID-10.1.2 Evaluation Synt
Chapter 4 

Peiformlug the Synthesis 

programs are taUored icUosyncraticalfy to the p

or communify receiving services (Yhi and Heald

1975). Such treatment variations do not residt 

haphazard implementation. Ratiier, there is an u

tional effort to individualize. 

An example is the Education for AU Handiciq;)p

ChUdren Act (PubUc Law 94-142), passed by th

gress in the inid-1970*s. Hie act requues that e

handicsq;>ped chUd receive an i4;>propriate, or m

alized, program of special education and relate

vices. It c!Overs many handici4>s, including phy

cK>giiitive, and emotional hancticaps, and so the 

vices piovided are extremefy diverse and specia

The desued outeomes vary as much as tlie trea

both withhi and across hancticapping conctition

Is, the desired outcomes and treatmente might 

mucdi for two paitialfy deaf chUdren as they wo

a paitialfy deaf chUd anci an emotionaUy distur

chUd. AdctitionaUy, treatment lengths are hictiv

detennined. 

Nonquantitative information is important in tha

act stresses the process aspecte of each treatm

rather than the outcomes. The handicapped ch

parente, for example, are to receive notice of a 

posed change in theur chUd's educational prog

t h^ are to be provided the opportunify to part

in the program, and the cMd's treatment and t

ment outcomes are to be reviewed at least onc

year. 

Thus, aggregated and h|ter synthesized child o

data would be of Uttie use to a poUcymaker wh

to know if PubUc Law 94-142 is workhig weU o

whole and hcnv it should be c:hanged. A variety 

desc^ptive date from various sources would b

useful. For example, descriptions of tlie quaUty 

parent and school interaction might be helpfiil

(Guzzo and KatzeU, 1987; Jackson, 1980; Levh

1987; WaUieig, 1986; Ward etal., 1987) 

FageSa GAO/FEMD-10.U Evalaatioa S
Ch^ter4 

Performing the Synthesis 

2. Assessing program effecte across multiple 

effect. Quantitative approaches can be emplo

¥^en aU the stodies have assessed program e

the same "level" or unit of effect. This level is 

the inctividual participant. For example, most 

stodies examhie the behavior of participating 

dren. But prograins can have an effect at oth

as weU (Yhi and Heald, 1975). With day care

example, ite availabiUty can mfluence famiUe

labor market as weU as chUdren (Belsky and 

berg, 1978). 

If a program's influence is felt at several level

overaU decision about it may fon» the aggre

resulte across the different levels as weU as a

outcomes measured at the same leveL WhUe 

at any particular level can profit from quantit

methods (when the aaninqitions for using su

methods are met and it is feasible to use them

aggregation across levels usuaUy demands m

itative decisions about trade-ofiEs. 3. Uncontrolled treatment groups and treated 

groups. Salter (1980) has pohited but that vr

eral stocties compare people who receive a tr

to others ¥^o do not, subtie differences betw

larfy labeled treatments are common. Nonqu

infonnation can offer valuable guidance in hd

reviewer decide how SimUar the treatmente a

An example of this comes firom a stodfy by Fo

al. (1981). They revieWed a soles Of stodira 

dren's nutrition program sponsored by the U

Department of Agriculture. The simplest qua

anafysis w<)uld have iiivc>lved ccHiqnitiiig an e

for each stocfy compiuring the h^dth of chttd

received food supplemente with those ^b d

and then avera^i^ fhidhigs across the stocti

nonquantitative infonnation hicduded in man

inctividual stodies convinced them this would 

fruitiess. WhUe for admhdstrative purposes t

Page 53 OAO/FSIID-10.U Evalaation 

'ii-
Chapter 4 

Performing the Synthesis 

treatment was the same in each stody, information 

about "plate waste" (food not eaten) ofthe 

supplementary food suggested important differen

among sites. In some cases, the plate waste was h

other stocties reported almost none. In every case, 

these date were informal and descriptive. But the 

reviewers decided they were crucial. Combining tr

mente that had the same achnmistrative name, m t

setting, would have amounted in fact to combuiin

groups receiving vastfy different treatmente. They 

were "uncontroUed." 

The same dUemma arose for the control groups. T

were not aU "pure" control groups, in textbook 

fashion. Many stocties reported that chUdren at si

not receiving assistance from the U.S. Departmen

Agriculture, rather than receiving nothing at aU, w

getthig some food assistance under titie XX ofthe 

Social Security Act. This titie provides various for

of aid to low-income famiUes. So control groups h

some of the stocties hi the review were actuaUy qu

heavify "treated," whUe others were in fact "pure" 

control groups, receiving no food assistance at aU

In this case, the quaUtative descriptions of \iiiat a

aUy happened to chUdren in treatment and contr

groups hi each stody led the anaiyste to reorganiz

their synthesis into subgroups. These subgroups 

ognized differences between treated versus untre

controls. A shnple effect size averaging over aU a

able stocties would have missed this step. 

(Bangert-Drowns, 1986; Becker, 1986; Begg, 198

Cooper, 1982,1988) 

4. Stodyfaig the "wrong" treatment. OccasionaUy, 

^ e n i^thesizing outeomes, hi cases m Miiich qu

tative approaches have proved feasible, one finds 

a relationship between a program and an outcom

not as strong as was the ori^naUy planned treatm

that might explain the differential success. Here, 

Page 54 GAO/FEBID-10.1Ji Evaluation Syn
cliapter 4 

Performing the Synthesis 

descriptive or nonquantitative date can play a

important role. 

A quantitative analysis can systematicaUy exa

across many research stocUes, the relationsh

between planned program and outeome varia

descriptive infonnation in one or several stodi

give a clue to an evaluator that there existe a d

feature of the treatment, one not formaUy buU

stocfy's expeiimental design, that may be mor

tant than the original planned treatment. 

How Nonqtiantitative 

Infonnation Can 

Influence Policy 

A nuyor hnpetus for developing quantitative ^

methods was a wish to make research finding

useful for poUcy. When presented with a simp

numerical summaiy ofthe average effed of ps

therapy (Smith and Glass, 1977) or personali

mstmction (Kulik, KuUk, and Cohen, 1979) o

size (Glass and Smith, 1979), a poUcymaker c

uate program effecte without wading through 

of research reporte or vague rhetoric. 

The "best" fonnat for presenting research fin

remains an open and compUcated question. 

are cases in which quaUtative findings have h

clear effect on poUcy. One exunple of how qu

mfonnation led to actual admhiistrative chan

comes from stocties of the comparative effect

of professional versus panvrofessional "help

Durlak conducted a systematic review of 42 c

tive stocties. He reported consistent findings 

different patient populations that for crertaui 

services "paraprofessionals achieve cUnical o

equal to or significantiy better than those obt

professionals" (1979, p. 80). 

This is not the sort of finding that many phys

expect when th^ review the Uterature on the 

effectiveness of nurse practitioners, yet it led 

practical outeome. Lewis et al. (1974) and 

PageSS 6AO/FEMD-10.1.a Evaluation
Chapter 4 

Performing the Synthesis 

Merenstein, Wolfe, and Barker (1974), looked ear

for quaUtative infonnation about uriiy the nurse 

practitioners seemed to be so effective. A key obs

vation was that nurses aUocated their time among 

patiente differentfy from physicians. The two gro

also gave different weighting to the importance o

ious symptoms and incidente. The result of these 

itative findings was that physicians made acyustm

in their time aUocations. It is interesting to trace t

sequence of evente here. Because of the quantitat

infonnation underlying the original comparative 

stocties, the physicians viewed them as surprising 

took them seriousfy as scientific evidence. This w

mgness to accept suiprising findmgs led to a quaU

tive searcdi for an explanation and, ultimatefy, to 

acUustmente hi the way some physicians aUocate 

time and resources. (Cooper, 1989; Hazelrigg et 

1987; Ught and PUiemer, 1982,1984; Slavin, 19

Smith, 1980; Strube and Hartman, 1983) 

Presenting the 

Findings 

The infonnation generated throug the evaluatio

synthesis process is brought together hi a report 

must be carefuUy formatted to respond to tbe que

tions that were formulated hi cxn\)unction with t

stocfy's requester. The hitroductoiy chapter ^ou

briefly describe the histoiy of both the stocfy and 

particnilar program under discussion and should 

present the stody objectives, scope, and methodo

The latter section might include a firameworic sho

the evaluation stocties and date bases, a table sho

the relationship between tiie evaluation question

the avaUable stocties, and a description ofthe ana

steps undertaken. At a mhiimum, however, this s

tion should describe the search to identify the ev

tion stocties, including any limite that were put o

search (such as a requirement that aU stocties ha

experimental designs). The section idiould answe

foUowhig types of question: How was the Uiform

obtained? Firom what source? What Umite, if an

56 OAO/PBIID»10.1.a Evahiatloa Synt
Chapter 4 

Performing the Synthesis 

were put on the effort? How confident are the 

uivestigators that aU relevant mformation, or a 

sentative sample of that infonnation, was obtah

If possible, other report crhapters should cones

to the cUent's questions. The body of the repor

course includes discussion ofthe adequacy oft

avaUable for response to a particular question. 

nical appendix might systematicaUy describe e

stody across such dunensions as titie, report 

reference, stocfy purpose, date coUection peri

sample selection, date collection, and date ana

Date bases should also be described, although 

the same dimensions wiU be appropriate. 

For several reasons, caution should be exercis

drawhig conclusions firom the synthesized data 

foimulating recommendations. The evaluation 

thesis cannot substitote for a carefuUy designe

with primaiy date coUection for investigating t

qu^rtion of uiterest. Sources for the evaluation 

thesis may be ciated; additionaUy, idl aspecte o

particular issues may not have been thoroughf

explored. Confinnation firom the agency admin

tering the particular program under review nu

needed to detennine that the cx>ncdumoiis draw

past stocties are stiU appUcable. 

One ofthe most common concluding sentence

research reporte is, "More research is needed." 

is this statement based on a systematic assessm

available evidence, and when is it a casual rem

simpfy concludes an evaluation irtucfy? A ^the

help answer this questicm. For example, the G

synthesis of fhidhigs about the Special Supplem

Food Program for Women, Infante, and GhiUr

cussed at the end of chapter 3) concluded thait 

the evidence that the program resulted hi fewie

low-birthweight babies was strong, there was n

comparabfy convhicing evidence as to ito effec

chUdren's mental retardation. Conclusions suc

Fage 57 OAQ/FEIID.10.1.2 Evaluation S

„ 
Chapter 4 

Performing the Synthesis 

this can help poUcymakers understand exactfy wh

known, and exactfy what is not known, about the 

problems they pose. Such conclusions also point 

where good, new infonnation would be particular

valuable. 

Fage 58 GAOyFBBID-10.1.2 Evalnation Syn
Chapter 5 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

What can a synthesis of evaluation stocUes do th

single stocfy cannot? In ttus chapter, we discuss 

issues that synthesis helps resolve. The most fr

quentfy cited virtue of synthesis is that the uicr

sample size can increase statistical power. Tlus 

has been discussed widefy. However, the six pro

ties of synthesis emphasized here have Uttie to 

sample size. What they have in common is that 

help us say when a social, mecUcal, educational

some other type of program works, not just wh

or not it worics on the average. 

One way we c:an identify when a program work

focusing on uiteraction. Statisticians often use t

word to mcticate nonlinearity. That is how we in

the word here. In a program evaluation context

can ask two questions. First, does the program 

weU for certain kinds of people and less weU fo

others? Second, does the program work weU h

certaui settings and less weU m others? Both th

questions are about interactions. A shigle stody 

find certain kinds of interactions, but synthesi

eral stocties can tum up much riclier, more use

infoimation. (Wachter and Straf, 1990; Yeaton

Why Interaction 

Effects Are 

Important 

UsuaUy, social, educational, and health progra

evaluated to see how weU they work. Good eva

tions also examine how changes in program fo

could incrementaUy help improve them. One w

askhig indiether a progriun works is to ask Aet

works on the average. Another way is to ask w

it worics for a subgroup of people or in special 

setthigs. 

For poUcy purposes, the hiteraction question c

as unportant as the main-effecte question. For 

example, when a physician considers what ane

to give a patient prior to suigeiy and has a cho

between two drugs, it is useful to leam which o

two is better on the average. However, it is eve

Fkge50 GAO/FEMD-10.1.2 EvaluationS
Chapters 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

valuable to leam which of the two is preferable fo

precise surgeiy the patient wiU have. Or v^ch o

two has a better track record for the particular k

patient. It would not be surprismg to find, for 

example, that the anesthesia best suited for a 

20-year-old hi exceUent general health is differen

from the anesthesia best suited for a 70-year-old 

poor health. 

Finding such mteractions is important not onfy w

maldng decisions for individuals but also when 

assessmg the effectiveness of laige-scale prograi

Suppose that Head Start works generalfy weU fo

dren under 4 but far Iras weU for duldren 5 year

or older. That would be worth knowmg. If resou

for the program were limited, such knovriedge c

teU us whete to concentrate them. Or, if substan

resources were available, this finding of interact

would suggest that the Head Start curriculum sh

be modified for older children. So, whether the m

purpose of an evaluation is to target resources o

change a program incramentalfy, fincting an Uite

tion can guide decisions. (Raudenbush and Biyk

1985; Rosenthal and Rubhi, 1986; Shapho, 198

Whyl^thesisls 

Usd^iilin 

Identifying 

Interactions 

Let us recaU how a shi^e research stody can ide

an interaction effect. Basicalfy, there are two wa

One way is to buUd a search for the interaction 

directfy hito the stocfy design. For example, let u

hypothesize that Job trahUng program A woiks 

for high school chopoute than it does for hi^ sc

graduates and that the reverse is trae for traiidn

program B. Uien, if we have control over treatm

assignmente, we can test this hypothesLs 1^ maU

sure tbat aU four combinations of people and pr

type are represented. Id(»Uy, randomization wU

used to develop the four groups-dropoute ghnm 

dropoute ghren B, graduates given A, and giadu

given B. Then, comparisons ofthe four effect siz

wiU give a clear hicUcation of vdiat program type 

F*ge60 OAO/FBIID-10.U EvalnatloB Syn
Chapters 

Evaluation Synthesis Can Answer 

Questlona a Single Study Cannot 

works best, on the average, for what type of per

These finctings wiU either refute or strengthen t

tial hypothesis. 

The other way of identifymg uiteraction effecte 

shigle stocfy involves the use of post hoc proced

Suppose that a search for interaction has not be

maUy designed hito a stody. In that case, such p

dures as regression analysis and other appUcat

the general Unear model can be a^pUed 

retrospectivefy. The dUemmas and caveate invo

this process are weU known. If people were fire

choose their own treatment, there might be 

self-selection. There may be a confounding of b

ground variables. For example, most of the high 

school dropoute may come from midcUe-hicom

Ues hi rural areas. Suppose that a single stocfy o

type ctid not assign people to training prograin

random. Then, because the stocfy was not desig

examhie uiteraction, ite findmgs could weU be c

founded by graduation status, setting, and famU

Against this background, we c:an now address t

tral theme of this chi^ter-that research synthe

be far more effective m identifying mteractions 

any single stody. Any one stocfy is conducted in 

ticular context, under a particular set of constr

Unless the stocfy is extraordhiarify large in scop

has a limited group of participante who are ass

to treatmente hi a certain way. Each of these fa

good for a single stocfy. It is important to know 

exactfy what population is in and what populat

out. It is important to know how people chose, 

were assigned to, a treatment. 

The advantage of lookhig at a group of evaluati

stocties is that the hidividual stocties often take 

in different contexte. And we can leam much ab

hiteractions from noticing how flbidings relate t

context. To iUustrate this idea concretefy, we ca

Fage61 OAO/FEllD-10.1.2 EvaluationSy
Chapter 6 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

address the six evaluation issues that synthesis 

resolve better than any single stocfy. 

Issue 1: Matching 

Treatment l^pe With 

Recipient TVpe 

The Head Start program was created in the earf

1960's hi response to a growing beUef that som

had to be done to help poor chUdren start schoo

stronger footing. In 1964, Sergeant Shriver, dir

ofthe Office of Economic Opportunity (OEO), f

a committee chaired by the pediatrician Robert 

Cooke. Ite charge was to develop a program for 

reducing the effecte of poverty on chUdren. Th

efforte led to the creation of Head Start, which 

seven concrete goals, including improving the c

mental proc:esses anci skills, with particular atte

to conceptual and verbal sldlls. 

The program was formaUy authorized to begin 

summer 1965. Between 50,000 and 100,000 ch

were expected to participate in the first summe

gram, bl fact, 560,000 did. By 1967, Head Star

fimduig had grown to $349 mUUon. OEO decid

evaluate ite performance and contracted in 196

Westinghouse Leaming Corporation and Ohio 

University to conduct a formal evaluation. The 

ings were released in 1969, and they stunned th

cation community. 

The key sentence in the Westinghouse final rep

says: "Although this stocfy uicticates that fiiU-ye

Head Start appears to be a more effective comp

toiy program than summer Head Start, ite bene

cannot be described as satisfactoiy" (CicireUi, 

p. 43). According to Datte, 

"children who participated in Head Start sununer program

score higher atthe beginning of firrt, second, and third gra

such programs on all measures of academic achievement, l

development, and personal/social development tiian cliildre

had not participated. Children vho had attended the i^>ye

programs and were tested In the first grade achieved lUghe

on the MetropoUtan Reading Test and some subtests of the 

Page 62 6AO/PEBID-10.1.2 EvaluationSy
Chapter S 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

Test of I*^cholinguistic Abilities. Scores of ctiildren vho lia

attended full-year programs and were tested in the second a

grade were not different from tlie scores of comparison ctiil

(1976, p. 134). 

The disappomting findings of this evaluation ge

ated great controversy. Smith and BisseU (1970

CambeU and Erlebacher (1970), and others crit

the methodology severefy. Supporters of presch

education found many problems with the stocfy

design and unplementation. Yet, despite the cri

the stocfy had a great effect on poUcy. Supporte

Head Start were placed on the defensive. For 

example, both AUce Rivlin and Christopher Jen

uriio supported such remedial prograins as Hea

in the late 1960's, became more cautious afier t

Westinghouse-Ohio stocfy. Ri^din noted that "Je

and his associates dismiss the ixdiole preschool 

development movement in a fev^ skeptical para

graphs, cithig the Westinghouse-Ohio stocfy's f

that, on the average. Head Start chUdren showe

long-term cognitive gains over non-Head Start 

ChUdren" (1971, p. 32). 

How should we inteipret the findings ofthis shi

large stocfy, which had such a great effect? A sy

of earfy education prograins conducted by Biss

(1970) throws much light on Head Start and r

preschool prograins. Her review emphasized a 

for interactions. BisseU reanalyzed date coUecrt

three researchera: Kames in Urbana, Illinois; 

DiLorenzo hi New York state; and WeUcart hi Y

lanti, Michigan. She chose these three date sete 

because each author compared two or more sp

curriculums, each project had weU-formulated 

and each project was conducted and document

carefuUy. 

Taken together, these three date sete compare 

types of curriculum, each of which has support

the preschool communify: the Kames AmeUora

PageSS 6AO/FEMD-10.1.2 EvaluationSy
Chiq^terS 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

curriculum, a highfy stractured cognitive curricul

the Bereiter-Engehnann curriculum, a highfy 

stractured informational program; a traditional 

enrichment program emphasizing language deve

ment, with a relativefy permissive low-stracture e

ronment; a tractitional enrichment program 

emphasizhig psychosocial development, with a r

tivefy permissive low-stracture envuonment; and 

Montessori program with a stractured environme

BisseU found smaU main effecte. For example, pr

grams with strong quaUty control, weU-trained s

high degree of staff superviaion, and a low 

pupU-to-teacher ratio produce bigger cognitive g

than other programs. Her big finding invoh/ed ui

tion. To quote her: "Directive, highfy stractured 

scdiool prograins tend to be more effective with t

more disadvantaged of poor chUdren.... bi cont

noncUrective, less-stractured programs tend to b

more effective with the 1 ^ disadvantaged of poo

ChUdren" (BisseU, 1970, p. 62). 

BisseU's date make her point sharpfy. The reana

of scores on three standardized teste—the Bhiet, 

PeabcHfy Picture Vocabulaiy Test, and the QUnol

ofPaychc>lhiguisticAbUiti|»—shciwthatviiienacd

is weU matched with the optimal program (for 

example, exceptionaUy down-and-out children an

highfy stractured programs), the average differe

between experimental and control groups is betw

two thirds and three quarters ofa standard devia

If tiie match is poor (as when down-and-out c:hild

from pcx>r backgrounds are exposed to a relativ

open curriculum), the comparative gains are mih

A few of the comparisons even find a marginalfy 

tive program effect 

A synthe^ such as BisseU's has at least three vh

First, since the inctividual evaluations examined 

protJecte organized to serve ctifferent chUdren in 

different places with ctifferent programs, we get a 

Fage 64 OAO/FEMD-IO.1.2 Evalnation Syn
Chapters 

Evaluation Synthesis Can Answer 

Questioaa a Single Study Cannot 

broad panorama of finctings. Second, since the 

coUected by several independent investigators d

similar interaction pattems—that highfy stractu

prograins are best for the poorest chUdren—the 

bUitj' of this overaU finding is enhanced. Third, 

synthesis of several evaluations pute the resulte 

single, big Westinghouse stocfy in a new light. M

the earfy Head Start ^tes, such as those examin

Westinghouse, had dearly open and permissive 

They offered relativefy Uttle fonnal cognitive w

quote BisseU again, "dhectors favor supportive

unstractured, sociaUzation programs ratiier tlu

stractured infomiational programs for poor chU

(1970, p. 81). Knowmg this about the earfy He

Start centera that Westinghouse and Ohio Stat

versity examined and combiiung this fact with 

seU's review findings, we can see iRdiy the stocfy 

Uttie success. There is also reason for optimism 

stodent perfonnance should improve as more s

ture is introduced at local Head Start sites. (Ka

al., 1985; Levm et al., 1984; National bistitote 

cation, 1984; Proleau et al., 1983) 

Issue 2: Explaining 

Inq;x>itant'nreatznent 

In 1968, Rosenthal and Jacobson wrote: 

'As teacMng training institutions begbi to teacii the possU

teachen* ezpectattons of their piqiUa' perfonnance may s

self-fulfilling prophecies, there may be new eatpectancy cr

The new expectanor mav be that children can leant more 

been believed possible, an etpectation hdd by many educ

theorists, thoii^ for quite dlEferait reasons" (p. 141). 

Three years later, Baker and Crist asserted the 

site: 

Teacher expectancy probably does not affect pupil IQ. T

conclusion is supported by a background of decades of re

suggesting the lability of human intelligence and its resiQ

alterations by environmental manipulation, by the reanal

Rosenthal and Jacobson (1968) 8tu(|y..., and Iv die fail

replication studies to demonstrate etTects on IQ" (1071, p

Fkge68 6AO/FEJIIIV16.1.2 EvaluationS
Chapter 6 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

So, here we find argumente from distinguished 

scholars that disagree sharpfy. The expectancy 

hypothesis is central to classroom conduct in edu

tion, because it has both substantive and ideolog

componente. Suppose that teachers' expectations 

a particular stodent's perfonnance actuaUy play a 

in determining the stodent's perfonnance. Some 

people see schools as exacerbating or even perpe

ating inequaUty among chUdren's Ufe achievemen

For these people, the expectancy argument offers 

strong explanation for why poor chUdren do less 

in school than other chUdren. Educators have vig

ousfy debated the hnportance of teachers' expect

tions. Ryan (1971) and Kohl (1971) both argued 

teachers expect le^ from poor chUdren and the

receive less. Elashoff arid Snow (1971) argued th

reverse—that methodological flaws in the stocfy b

Rosenthal and Jacobson (1968) undercut theur f

mgs. 

To assess the importance of teacher expectancy o

stodent IQ's, Raudenbush (1983) synthesized 18 

experimental stodies. Seventeen of these stocUes 

strong research design, in which children were 

assigned at random to treatmente. WhUe the 18 

stocties included chUdren of different ages and in

groups, they aU used IQ as an outeome measure

denbush used several ctifTerent methods for com

biiung stocties in a quantitative synthesis (Edghig

1972; Fisher, 1973; MosteUer and Bush, 1954; W

1971). He emphasused the effort to explam varia

among outeomes (PiUemer and Light, 1980). His 

conclusion was not at aU obvious for someone si

lookmg at the finctings of 18 stocties: 

"The eifect sizes of the studies, in standard deviation units, ra

firom .56 down to -.13. Five ofthe eighteen achieved statist

nificance, three at the .06 level and two at the .01 level. Fpr t

teen other studies, in five the experimental children scored I

than the controls, while tai the other eight the conbnl chUdre

scored higher" (Raudenbush, 1983). 

Fage66 GAO/PEBID-10.1.2 EvaluationSynt
Chapte

r 5 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

Raudenbush's findings are clear and important

found a smaU average effect size across the i8;

c>f. 11. But, as he leppitid; this main effi^ 

"certahify ccmcejeds imu^ than it clarifies." 'niia

because the stuciids craii jteidivicled hito t#6 brb

gr(>ups. bl cme group, teisu^^ 

mformationalnnitdej^ 

after a few week^|>f ihitial^^ro^ 

teachers got ihfomud^^ 

This difference betnn^ subgtoups^^ 

be the.kcy JBndiii^ 

before :they:ineJ^ sti^ ^ 

dSiect^;T^iacheis 1 ^ 

kmiwingstoci^nlB;^ 

noexpe6^Biu^^^^ 

cc>rrd[aticn'.bet«i^ee^ ciKf the toea^^ 

ancl^c>iJrtxxHneS'isT-;^l!^6^ 

"Whehno;te{uJ^ 

experiiii^||tbe;8n^r^^ 

Afterthesec^ 

cmfy cme recited a i^rbt^if^^ of lesst ithiBm 

So, thiS;^^iit]iei3ii9.sh^ 

ragecl:fbr'msungr^ye^ 

focused :Onlma^^:i^^ 

effect':c>rnot?;^ni^ 

very 'shiaU^liie'syi(^^ iiB;J^tiit'theli^

treatmeiit .comik)^^ 

i^ven:'It^¥raUl4:)^ to;le^^1li^ 

cme -of'title: li3jitod|eisiU^ 

ofithissyii^^ 

in^stud^esviiun^'Jii^^^ 

ccmtactJ|:iShxiiisu4^'i^^ 

tiEm(^iiiduetion:t^^ 2^wey|$:0i':i^^ 

sc±icM>lyew:8hc>w 

fouiMl^thstfi'tiieibil^i^^ 

tiie treathit^ inritppl^^ 

ttiis^ enablisii its to'uhd^niihd 

wnrits ••"'' •• •'•'••-•••'X:::<^•-^y^•:i••^ -'-''• •i'-^:['i''':>yr:;:):'i'':i'i--^^

• • :•••;•• ••••••:••.•..• •••s-i'-.. - • ^ ;v:'-^^:.:.Vi V •: ^•;^:

'••''' --r^W'Mk •'m

F4e67 .:::;;XGJtt|fiW>'MKl'S!::By^^ 

itii: 

' ./•-,••••.:-..• •••.•,••-.,•'-.•.•.'•.•- ..•• •^ •,....•.-..... •••••, ,••••.•••:•;''••,• 
Chapter 6 

Evaluation Synthesis Can Answer 

<)neations a Single Study Cannot 

Issue 3: Eliqplaining Most researchers and poUcymakers have at som

Conflicting Results P^"^^ reviewed a group of stocties to come up wi

"overaU fincUngs." A natural inclination in domg 

a review is to hope that aU or nearfy aU the outc

wUl agree and that we can feel reasonabfy comfo

with these resulte. But it would be a shame if our 

ural hope to find agreement among stocfy outcom

led us to view contractictoiy outcomes with firu

tion. Indeed, a major strength of date syntbeas 

it helps us view conflicting outcomes in a constr

way. The conflicte may be offering valuable 

information. (Yeaton and Wortmaii, 1984) 

c' •:'.: 

i-S'f'-',':.?.•• 

bl the late 1970's, discussions of job trauung 

enqihasized the importance of "integrated servic

Evaluations ofthe Ck>mprehenshre Employment 

Training Act, the broad umbreUa jobs program b

geted at several bUUon dbUara a year, were find

margiiial success at trast Some of tiiese evaluati

(National Academy of Sciences, 1978,1979) sug

gested that Job trahdng atone, vi^en narrowfy de

could not break a famify's cycle of poverty aiid u

ployment lliese assessmente found that integra

services, which included madching a famify's nee

education, heidtii services, and job training wtth 

wett-ooordlnated group of "helpers," offered far 

promise than a stand-^done Jobs program, 

To assess this idea, the U.S. Depaitment of Labo

hdtiated several stocties of integrated services pr

grams. The key idea was to coordinate a series o

vices for poor famUies hi nMch Job traiidng was 

important component but not the onfy compone

Several demonstration programs took place at s

ctifferent sites. But the resulte were conflicting. W

these conflicting findhigs about the value of inte

grated services were discoiuraging, the faivestig

ultimatefy capitalized on the vai^ng outcomes t

leam a great deal about the contexte hi which 

hitegFBted services worked weU and worked bad

Fiige68 OAO/FSIIP-X0.1J Bvafautloa Syn
Chapter 6 

Evaluation Syntheais Can Answer 

Questlona a Single Study Cannot 

about how to organize a gcx>d matching plan 

servicres and redpiente. 

The broad question, then, is. How can a synth

ness ctifferent findings firom several stodies to 

enhance our understancting about a program

tiveness? ft is iic>t rare ih ahi^r t to puU tc)ge

infomiation about a program's effecrte across 

to find that the stucties provide sevnefyconf

mformation. llieseconflic3tsow be fi^ 

is precasefy such c»iifllidte tlifi^ may give eiialii

scmie insighte into thib iiud>d^ problem. "Th

training esaihptestuyvnsihiS;^^!^ us look a£sqx

rounciedoffforilhistiai^ 

employmientisthebiitcx)^^ 

Take the case c>ftw(» stucties inducted in cUf

states by cfiffernit inviec^^ Eacliconny^N

integrated sendees j^it]^^ a shiglersfei^

trahdng pvogrom:^ 1 ^ one stete too

men. Forty receiye one i»og^^ and 40 recc^

other This stikfy fiGntind 

group had.ah;aVeiage';c^'^ cri^emiijf

while the:shigle-seryicxr^gto^ 70:Wedffl^:

grated siMyices;se^'to;'be;ni0re eflfectiye.^;*

in the cither statofomidp^^ 

examined 80 -ihehi^ Aii^dii^:40;receiw^ onelp

ami 40 received the bth^ 

that the integra|>Ml:seii»^^ wcnikeid cndy 

weeks, white lliiedngteii^^ men |ya^tiie^r

grcMip worlred:ama*^lW?jO^ 

What can be ctone ydtlicxki^lk^^ resulte?A

can be made to 9e«if|vt^^ 

fllct somettihig^iivt mat^ ^ 

services that ieavoifei:^ 

how synthesis ep ^ 

the 80 meh:iiieaic^'itu^J^^ 

ccmsteUatton. iSku^i <cate|^^ cimltMe <dU

when thepebple ser ^ 

let us Amplify hereind^as^nei^ 

Fage 66 - (^ayraiqj^io.1.21^^^ 

i^SUiS-::- lii^iiM^MHiiiiAiilii
Chapters 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

broad types of problem sets, problem set A a

problem set B. First, look at the aUocation o

with each type of problem to each program 

stocties: tiien separate that aUocation firom th

effect found for the people in each program

A simple table might ^ow^hat although the 

means of the two studies caused the resulte t

flict, the two stocUeshad jclentical effects for 

groups of men receivir^ each treatment ^ 

found that men with problem set A iNiio m»

grated servicres were;|eani|d^^ an averagiei o

weeks, vriiile men witii jpni^^ 

the sin^ scnidce wtnited^^a^ average <^pi||i

weeks. Bc>thstodiclsi^ilui that the revei^ 

for men with iKroblm^ Why, tten, wi

ccmflict? llie <»nflicti^ 

aUocaticms c>f pniblnltVtgi^ across the ti^^

types in tiie t^) 8tu)dtii»!; In the first stiulty^

men than B men reoei<i^iiUe^^ 

seccmd stucfyy tite ieve^ true. This di

aUocations, oc^ineci vnth the ccmBisl^|l

both studies that inteJSirlEi^ 

effecth^ for ineh vrith iiro^^^ set Aaiui li

for men witii problmseitB created tite co

What pc>Uc7 inqpUcsatto^ be diawnfram 

synthesis If these clataare in fact a goQ^d

of reality, we leaimhbwlntegrated t^^ 

targeted to a siibj^^p <^^ idiose in^ie

match those aeiyic»^ 

the c»se emergeclfircim 

stucties comiMurhig j^^ wittis^i^ i

reached c>pppsite ftnctii^ ft woiiiid lurtilu

fromeither of the^dk» atone. Vife Icnilm H 

exaihinhig praj^rai^ ami aJUbo^scitic

across stucUof^ Vtexaii hsvprove tJ^ihat^^ 

and target integnuted^t^^ to thbsie iwt

efit the hio^.;(Cheliii^lqi^ 1983; Ccx>k,198

and .(>owIqr,a986)-f; "'^ ^•'•"J; '

70 OAO/FBIID-10.1J 

^^ri;;^Cfy^% 
Chapter 6 

Evaluation Synthesis Can Answer 

(Questions a Single Study Cannot 

Issue 4: Detennining 

Whether Relative or 

Absolute Perfonnance 

Is the Critical 

Outcome 

Most programs can be looked at m at least two 

ferent ways. One way is to see whether an hite

has taken hold as intended. For example, doe

chUd reaUy know how to count better? Does t

for hypertension actuaUy lower the patient's b

pressure? Is the prisoner who is about to be r

actuaUy a competent carpentei? The other way 

assessdng a program is to see ^v^iat happens in 

end. Does the chUd who now counte better ge

marks in school? Do the patiente vdio now ha

btood pressure also have is lower incidence of 

attacks? Does the ne ^ released carpenter ea

sonable hicome with the new skUls? 

A synthesis of evaluation fiiictings that is weU 

usualfy looks at the studies ti^ it examines hi 

ways. But it is worth noticing that synthesis h

cial ccmiparative advaiitese;<ir«wr any singte ^l

answering the second questioh. This is becaiu

some programs can confer benefite or inculca

that indeed take hold; it is licit alwaysthe caiie 

these skUls or benefits can be traiudated into 

positive outcomes in the eiui. In paiticulai; »o

efite are valuable only in a comparative sensie

ofthe Umtted nuinber of ^p[K)rtunities in wlW

tain skills are usefiil. If too hiliny otiiers have 

skUls, they beccmte less valiitebte to any p 

person who has them. 

Job training can Ulustrate both pointe-the po

about the comparative beniefUs and the point 

the special value c^ sjinthe^^ Svvpose that aJ

proj^am undertaken in phct'city is evaluated. 

research design is exc^slji)^; Qne huiuiredjtqn

are divided at rahcfom into tWb groups. One g

receives trahdng to be cflpienters, ^i^iUe the O

group does iu)t. If this evi^^ fiiuJM^t^t 2 

afierthe job prc}gram tltetra^^ cleauly cau

than the contn>l group, Vt e we ocmcpdc

would probabfy ccmcdikhe tiiat;the jc>b tral^^ 

wcnks—and weU vre fiiiPUlil,^ i^ ^ 

71 OAO/FE1I0^1O.U EvaluationS
Chapter 6 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

Stocfy has positive findmgs, and they come fr

randomized experimental design aUowhig ca

uiferences. 

Let us assume that this positive fincting is no

that the same program is offered at 10 other 

around the countiy. Leammg fix)m the exceU

example set in the hdtial evaluation, each of 

new sites organizes ite own randomized tria

resulte become avaUable 2 years later, and 

difUcult to inteipret. At three rates, the trahd

clear success; the trainees have good jobs. A

ates, it is at best marghial; onfy some traine

jobs. At the five other sites, tt ctidn't work at 

Efforte to organize these 10 findings into an 

tion synthesis can move forward hi two quit

ways. One way is to emphasize the skiUs que

the trauiees at aU 10 sites become reasonab

caipentera? The other way is to empharaze t

come question: Why ctid the findings differ s

across the 10 sites? By tackling both questio

thesis can generate valuable insighte. For ex

fincting that sites varied enormousfy in the

knoMdedge of carpentiy provides inanageme

mation. Clearfy, the substantive training com

needs to be improved across sites, and tt ne

strengthened in c:eitain weak plac:es. Howev

fincting that trainees leamed carpentiy qu

10 sites would be even mpie informative, be

would forcre us to ask why trainees differed 

stantiaUy across the 10 sites m thehr abUity t

One possible explanation is that the benefit 

carpentiy training for any one recipient dep

the number of other people who receive the 

Sfynthesis could support this explanation by 

inhig the cx>irelation ac:ross sites between th

of trainees who got jobs and the opportunity 

success as measured by, say, the totid popu

each site. If the correlation is clearfy positiv

Fage 72 OAO/FEIID-10.1.2 Evalaatio
Chapter 5 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

leam three things. First, we leam that traini

in a prectictable way. Trainees in bigger citie

better prospecte than trainees in smaUer to

Second, we leam two unportant things abou

training itself, that it indeed confera skills on 

participante and that when relative perform

a constramt on any one trainee—that is, whe

trainee lives hi a big city-^the program succ

Tlurd, we leam how to organize and manage 

trauung programs better in the fiiture: th^ 

targeted to settings m which there are oppo

for tramees to put their training to use. 

To summarize, then, ^nthesis can identify p

^os e value dependls not onfy on their subst

features but also on the number of people w

pate in them. That is, synthesis can pohit ou

prograins are constramed by Umited oppor

success. A suigle stocfy cannot answer quest

this nature. 

IssueSlAsses^ngthe UsuaUy, any single stocfy is organized by a s

Stability of Treatment investi^^r or a smaU group of investigator

Eflfectiveness takes place hi one or a very few sites. A shig

asingle site aUows us to assess vidietber a tr

worked overaU. We can even examine the v

among outcomes at several sites. But we can

how robust tbe treatment is when tt is provic

several different mvestigatora or organizatio

synthesis of resulto across several stodies a

answer this question. When each of several 

tions hnplemente the same program in diffe

places, the variation in outcomes offera a go

of the program's robustaess. If it works extr

narify weU in a few places and pooriy hi oth

must tiy to expbdn why. But, whatever the 

tion, we wUl have ctiscovered that the prog

robust. We leam that it is sometimes effect

ite strength is easy to undercut At some site

poor perfonnance may be explidned by wea

Fage 78 OAO/FBIID-10.1JB Evalnatio

M& 
Chapter 5 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

unplementation or by a poor mateh between 

recipiente and program. The onfy way to asses

StabiUty* of a program in different settings is to 

how it fimctions hi different settings. 

Issue 6: Assessuig the Some scholara spend a large fimction of their t

Importance of arguing that research design matters. Gilbert e

Res^ffch Desien (1975), Chahners (1981), and HoagUn et al. (

^^ worked hard to convince the evaluation comm

that randomization is a cracial ingrectient for e

tion, shice it underUes our ability to make cau

ences. The efforte of these invmtigatora made 

headway. However, randomization is sometim

cult or impossible, so we must tum to altemati

do the best that we can despite thehr imperfec

Altematives include case stodies, quasi-experi

observational designs, stocties of management 

records, and computer simulations vdien iqipr

(HoagUn etal, 1985). 

None of this is news. But when a researcher or 

poticymaker faces a concrete problem, such a

Aether a certaui nutrition program is effectiv

Aether a new surgical procedure is worth ush

single stocfy is ahnost certain to have a single 

design. There are a few exceptions, but they a

The reader of tiUs one stocfy must Iben ask two 

tions. Fhst, does the stocfy stand weU on ite o

merit Is it weU done? Sec»nd, does the resear

design introduce any constrainte, limitations, 

biases? It is difUcutt to answer this second qu

with evidence from onfy cme stocfy, even one t

been weU designed and executed. But a synth

helps us a lot. It aUows us to compare fincUn

the research designs that lead to thc>se fihclh

across a group of stocUes. If there are correlat

leam two thhijgs. Fbst, we see what specific de

to vdiat specific outeomes. Second, we can oig

fiiture research knowing more about the 

consequences of specific designs. 

FBge74 OAO/PBIIIK10.U Evalnation S

fe?. 
1-

I: 

Chapter 5 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

Several concrete iUustrations show how res

design can matter. One example comes firo

Chalmera (1982) reviewed the findings from 

stocties of portacaval shunt surgery. These 

were published over a period of many year

ferent investigatore who woriced at ctifferen

Chahnera asked two questions about each 

First, did ite research design have adequate 

poor controls, or no controls at aU? Second

the uivestigator say about the surgery: Wa

marked enthusiasm, moderate enthusiasm, 

enthusiasm? The conclusion is that poorfy 

stocties of this suigeiy are far more Ukefy t

weU-controUed stodies to lead to positive r

haps iUustrating Hugo Muench's law of cU

stocties: resulte can always be improved by 

controls (Bearman, Loewenson, and GuUe

(Something to be guarded agahist.) 

A second example comes from the dUemm

best to control spiraUng health care coste. 

the Committee on Labor and Huinan Rrao

U.S. Senate asked GAO to examine aU ava

dence about the effecte on mecUcal coste o

the amount of health care provided at hom

elderfy citizens. It was proposed in Senate 

if more health care were provided at home

vice coste would drop, because the chronic

would make less use of hospitals. GAO's fin

strildng. The case stocties, mostiy smaU-sam

tive reporte, almost unaiumousfy suggested 

would decUne. But the quantitative stocUes 

opposite: total coste would not decline and

they might even increase sUghtfy. 

The quantitative stocties tumed up a clear 

this surprising result. Rather than leading 

elderfy recipiente of service to change the s

hospital to home, the new opportunity for 

considerabfy expanded the total number of 

requesting care. People not receiving servi

Fige 78 OAO/FEMD-10.1.8 Evalnat

^f'i-...-.;',. . 
Chapter 5 

Evaluation Synthesis Can Answer 

Questions a Shigle Study Cannot 

to request them. So, whUe the offering of 

reimbursable home care as an altemative to 

hospitalization can reduce the cost per recipie

those vidio accept the altemative, it seems also 

create a substantial new group of service recip

and total service coste do not drop. 

This example is not here to argue for or again

care. Some people aigue that the home care o

a good idea even if coste are higher. Othera di

But vidiatever one's values about the trade-ofE

between hospital and home care, the pomt is t

stocfy's design is closefy related to ite outeome

evaluator could examhie every published case 

and conclude that the evidence for lower cost

home care was overwhelming. MeanwhUe, an

evaluator who examined onfy stocties with com

or control groups would find overwhelmmg ev

in the other ctirection. Knowing that different 

studies generalfy lead to different sorte of fin

offera guidance for the fiiture. Whoever design

next stody to examine the coste of a home car

gram can tiy to mcorporate the strengths of b

types of design. 

Summar y Research synthesis is not a remecfy for aUUls. 

effort faces dUemmas. Perhiqps because certa

judgmente must be made, such as the weight 

must be placred on finctings from different rese

designs, some investigatore may be tempted to 

back on tractitional nanative reviews. Ibis wo

mistake. Just because a synthesis turns up co

requires a judgment caU is not good reason to 

the messenger. The messenger gives us inform

that is vital hi two ways. First, synthesis poin

features of a treatment or program that seem 

matter. Is there a cracial background variable

researeh design matter much? How stable are 

findingB across a group of stocties? Second, sy

helps us design the next stocfy. Examining the 

Page 76 OAf|/FEMD-10.U Evaluation 
Once the evaluator has grasped the tools and 

niques of evaluation synthesis, he or she is pr

for some of the finer pointe of the methodolog

this chapter, we discuss some of these, includ

comparing and contrasting the stocties and th

ings, (2) mergmg the quantitative and nonqu

approaches, (3) exploiting differences in stod

findings, and (4) anticipating problems. 

Comparing and 

Contrasting 

Studies and Their 

Findings 

The general GAO synthesis approach has bee

compare and contrast the stocties and their fi

In comparing the stocties, we look for the natu

extent of similar fmdings or trends across the

tiy to rule out alternative explanations for the

mgs. The key questions asked are: What rule

placing support in similar finctings across sto

What factora, if any, might mcrease our confi

findings across the stocties? To what extent 

place confidence in the findings? 

In contrasting the stocties, we focus on the ex

and confUcte. We tiy to identify the stocfy cha

tics that might result in outeome variations. T

may suggest tentative hypotheses for fuither 

gation. 

Begin with a review of the inctividual stocfy, o

type, to identify the strengths and weaknesse

WiU affect confidence m the findings. If there 

weakness, low confidence in the inctividual st

ings wiU, of course, be inctic»ted. For examp

^nthesis on home health care referred to ear

found that project evaluations using compari

groups experienced problems such as the pre

special populations, noncomparabitity of site

selection bias but tbat more confidence coul

placed hi stocties with random assignment to 

In evaluating the effectiveness of CETA, stocU

considered onfy the postprogram experience

CETA trahiees without regard to participante

Page 78 6AO/FEMD-10.1.2 Evalnation 
Chapter S 

Evaluation Synthesis Can Answer 

Questions a Single Study Cannot 

stodies and leaming which program featores 

important and v^ch are not help us develop 

tive research plan for the eleventh stocfy. Find

from a ^nthesis help make a stucfy as powerf

possible in answering a specific poticy questi

resolvmg a poUcy dUemma. In a world of sc

resources, such targeting is valuable. WhUe 

stocfy is unportant, a great vutue of synthesis 

makes systematic use of existing date and he

answer poticy questions that single stocties c

answer. 

Faga 77 OAO/FEBII>-10.1.a Evaluation 
Chapters 

Special Topics in Evaluation 

Synthesis 

preprogram experiences or without compa

groups were omitted firom the ^nthesis. 

Weak stocties are not always omitted, howe

example, the synthesis on block grante exam

administrative coste. AU stocties identified h

methodological problems. Rather than eith

weight on any smgle estimate or take the po

no date were available, the evaluatora exam

stodies to see if any general pattems were d

across the entire set of estimates. Given the 

nesses of the date, pattems were considere

tive rather than definitive. 

Even when stodies are sound, issues such a

generalizabiUty may limit confidence in the 

bUity of the fhidings. Information avaUable 

a particular question might come, for exam

several sound but smaU case stodies. WhU

mation is readify synthesized, confidence in 

izing from the findmgs would remain a pro

Differences m fiiiduigs across stocties can s

be explained through the nonquantitative a

For example, the special education synthe

large differences in two date sete in counte 

capped chUdren. Narrative analysis of the s

crepancies hi the efforte—mcluding date co

methods, tuning, and repoiting content 

procedures—were shown as reasonable exp

for the differences in estimates. 

WhUe the fincUngs across stocUes may be c

toiy, they can also be complementaiy. In fa

firom a stody with a comparativefy weak d

be reconsidered if they are consistent with 

other stodies. For example, confidence in f

firom a smaU case stucfy may increase wh

similar to those of a more powerful stody. L

series of independentiy conducted case stu

consistent in their finctings may yield a stro

F»ge79 6AO/FEMD-10.1.2 Evaluati
Chapters 

Special Topics in Evaluation 

Synthesis 

of confidence than would any stocfy taken indi

Process evaluations are always helpful m mter

the resulte of evaluations of effect. 

In brief, the nonquantitative approach has gen

required that we describe the characteristics, 

strengths, and weaknesses ofthe available sou

information. This requires analysis of hidividu

stocties and of stocties taken as groups. It then 

fiirther analysis of similarities and difference

finctings of the stocties. 

Merging the 

Quantitative and 

Nonquantitative 

Approaches 

IdeaUy, the nonquantitative approaches to eva

^nthesis should CK>mplement the quantitative 

approaches. Several ofthe types of informatio

quantitative stocties iUustrate how nonquantit

infonnation can supplement the quantitative, 

is hi fact feasible to implement quantitative 

approaches. In some situations, such as the un

troUed treatment groups and treated control 

the quantitative analysis would be, at best, m

without the msighte provided by the nonquan

mfonnation. 

S":''t'iiv''. 

Nonquantitative approac:hes to evaluation syn

are especiaUy helpful in dealing with coidUcti

higs among stodies that have surfaced m a qu

tive i4>proach such as the blockmg techniqu

cluster approach. Investigating confUct can s

times reveal important infonnation about pro

that would not otherwise be avaUable. The co

act as waming flags, suggesting that it may b

to look at stodies that show how a shnilar pro

was hnplemented at different sites or to exam

ation across stocties in relation to design 

characteristics and anafysis strategies. FVom 

specrtive, variation among stocfy findings unco

through one approach to synthesis and invest

through another can be a useful, constractive

information-laden ocx:urrence. 

Page 80 GAO/FEHD-IO.1.2 Evaluation
Chapters 

Special Toplca in Evaluation 

l^theaia 

Three Strategies for 

Combkung 

Quantitative and 

Nonquantitative 

Evidence 

There are three broad strategies for usmg dif

kinds of information hi the same synthesis: (1

nonquantitative infonnation into a quantitativ

fonnat, (2) discussing quantitative indexes na

tivefy, and (3) using the two types of infonn

combination whUe maintaining the integrity o

one. 

Quantifying 

Nonquantitative 

Infonnation 

One way to tiy to integrate quaUtative and qu

infoimation is to translate the former into a n

fonnat. Here are three suggestions. 

1. Case stodies and nonquantitative aggregat

A first strategy here is to somehow simmuur

case or aggregate witii a numerical index aiR

across stocties. For example, outeomes of ind

cases could be assignecl yalues of +1 (succes

(neutral), or -1 (unsuccessful), dependirig oh 

reviewer's overall evaluation of treatment SH

This quantification call be done at a more di^

level by assigiung numbera to several Individ

ponente of a case stu<ify and summing the ra

developmg weights for ctifferent indicators p

(Laxarafeld and Robinson, 1940). This prod

single numerical index fpr each stocfy, -viHhic^ 

be averaged or shown in a distribution. 

The "case surv^ methoid'' developed by Yin 

Heald (1975) offers ainbre sophisticated wa

quantify case stocties: Each stocfy is rated pn 

dimensions, such as research quatity, progra

acteristics, and outcomes. These multiple rat

cumulated across stodies, providing an oven

ical summaiy. Scorera aiiso incticate theur lev

fidence for each judgment, aUowing reUid

comparisons for "sure" and "unsure" ratings

weakness of this "numbering" is that much 

descriptive detaU is lost. 

Fage 81 GAO/PEMD-IO.1.2 Evaluation 
Chapters 

Special Toplca In Evaluation 

(^thesis 

2. QuaUtative hiformation in quantitative stoc

Quantitative research stocties usualfy report m

information beyond statistical summaries. Mo

nals require authora to carefuUy describe the 

ment, give information about participante and 

research settings, and discuss Umitations and 

features. A key insight often appeara in the 

"ctiscusaon" rather than the "resulte" section 

research report. 

Glass et al. (1981) suggest that aU this "other

infonnation be coded wben possible and brou

the formal qiuntitidive analysis. Walbeig and 

(1980) present many specnific reviews where 

ground features are coded and statisticaUy rel

program effectiveness. 

An advantage of this s^proach is that it helps 

identify quaUtative feajbates of stocties that ar

maUy related to the quantitative oidcomes we 

testing. A reviewer often faces too many stoct

conduct an efficient search for important qua

information without statistical tools. This is e

trae m the evaluation of educational innovati

^er e such relationships are often modest 

However, there is a fvnUiar drawback. Qy qu

stocfy characteristics to facititate statistical cw

scms, we lose information and obscure impor

real-world differences. $ilnUarfy, tt is hard to 

quantify icUosyncratic features that cdiaracter

ticular stody, such as a n^port that the testing 

place cn a particularfy hot day or that tiie cbi

certain class had more opportunities for info

practice than children hi another class. 

3. Expert judgment. A third way to quantify n

information is perhaps the most controveraii

hiteresting. It hivolves an effort to incoiporit

review the wisdom of researchera and practi

Some people wlU have Invested yeara of stocf

FsgeSa 6AO/FEIID-10.1.8 Evaluation 
Chapter 6 

Special Topics in Evaluation 

Synthesis 

thought and have had intimate experience wi

program or curriculum. Scientiste considt fre

with coUeagues, and both researchera and pra

tionera aie caUed upon routinefy to provide e

testimony in poticy mattera. Wldle no individu

opinion c^an encompass aU the detaUed evide

pubUshed literature, sometimes wisdom and 

insight may transcend the "sterile" date of res

reporte. 

We can suggest two ways of translating exper

ment into quantitative formate for use hi synt

First, a reviewer can incorporate expert evalu

stodies prior to statistical integration. One wa

doing this is to weight each stocfy according t

expert's judgment of ite overaU value. Techni

alreacfy exist for weighting mdividual stody ou

by their sample size that can be adapted to ex

ratings (Rosenthal, 1978). Ibis may serve to 

"formalize" what experte do when sufctjectivef

"weighing" the resulte of different stodies to r

overaU conclusion. If an eixpert beUeves that 

provides especiaUy strong evidence, the resut

that stocfy wiU receive extra weight 

Incorporating experts' judgmente could enric

review. For example, one can compare synth

using weightings of different experte and also 

pare the various resulte incorporating weight

simple unweighted anafysis. This wcrald mak

where experte disagree. If certain stodies are 

positivefy by some experte and negativefy by 

the discrepancies should be explored. Lacdc o

ment may phipomt methodological, substanti

ideological issues that tie at the core of contro

about an issue. When eiipert evaluations are 

tent, we can be more confident about the hm

in question. 

Hie use of weighte ties expert evaluations to q

research stodies. Our second suggestion invo

Faga 88 6AO/FEHD-10.1.a Evaloatlan 
Chapters 

Special Topics in Evaluation 

Syntheaia 

obtaiiung an expert's overall judgment about 

specific issue, based on a global integration o

her knowledge. Experte are often asked ques

"How big a risk does day care pose to an infa

emotional development?" or "Is reading prog

reaify better than program B?" WhUe tt is pos

give a precise numerical answer to such ques

experte may prefer to suppfy judgmente or as

mente verbaUy (for example, tt is "unlikely" t

tional development will be iiiq>eded, or it is 

posirible" the new curriculum is better). 

PresentuigQuantitathre A second broad strategy worics to cto the rev

Stodies Narrativefy quantitative evidence anci present tt naixativ

Rather than summarizing aiseiies of residte w

mimical hidexes, evaluKtora discuss stodie

ualfy. Stirengths and weaknesses are identifie

weighed, and overall cohcdumons are offered 

precise quantitative dcMnimentation. 

(Mtics of narrative reviews have clescribed t

characleristics as drawback. If stocties are r

precdston Is lost vriien a reviewer ghres an ap

mate or hnpressionistic summaiy. However, 

purposes may be seived t^ the cliscurtive fo

example, narrative reviews nu^ be more acc

practttionera and poUcymakera who are unfo

with formtd techniques and unwilling to refy 

numerical hidexes. When writing for a broad 

ence, an evaluator may choose to supplemen

sizes and significance teste with discussion o

stocties. 

Narrative presentation may be especiaUy usd

the purpose ofthe synthesis is not to summa

rather, to stimulate research or program imp

mente. Reviews often expllore questions such 

are stodies designed? What are their major s

and weaknesses? How easy or cUfBcutt was tt 

Unplement the treatment? Are there hnporta

Fage 84 6AO/FBBID-10.1.8 Bvaluatioa 

llj;:''..;. 

ii-ilASJ"i-'i.V •';••..>.. 
AUymg Statistical and 

Descriptive Evidence 

Chapters 

Special Toplca In Evaluation 

l^theaia 

"overlooked" program characteristics? Answ

suc:h questions gives newc»mera to a field a

nonspedaliste a broad picture of "what the 

are." It gives poUcTmakera some ideas abou

strengths and weaknesses of "overaU" findi

how confident one can be in adopting some 

suggestions. It may offer researchera impor

insij^te not onfy about how to interpret finc

existing stocties but also about how to impr

efforte. 

Hie two strategies above treat the synthesis 

as one of translation. Words and numbera a

ferent "languagesTSo, for consistency, wor

transformed into numbera or vice versa. Ho

both strategies bave a crracaal weakness. Wh

perfective is transformed into tbe crtber, it

benefite are weakened. Statistical summaari

their precision and the advantage of date re

viiien transformed hito narratives. SimUariy

rizhig case descriptions with a shnple num

loses much richness. 

We think it is woith wliUe for evaluatora to 

toward buUcting an "alUancx^ of evidence": 

both quantification and clcscription within t

synthesis, whUe maihtatii^ the integrtty o

Each type of information offera unique ben

larfy, ratber than chcKx^ngbetween number

rative when combining riiidte across sever

we need instead to detenhine vidiere each is 

useful and use them In syhc^irony. 

A review can be not only primarify quantita

descriptive but also strong or weak on botii 

slons. Cook and Levtton (1980) put tt weU: 

^ntbesls makes the most out of both types 

infomiaaon. 

FhgeSS GAO/PBMD-10.1.a Evalaatio

iliiaii:'^^^' 
Three Benefits of 

Combining 

Information 

Chapter 6 

Special Topics In Evaluation 

Synthesis 

There are three ways numerical and descriptiv

information hiteract They iUustrate the benefi

working toward an alliance of evidence. We be

they show how the benefite of combining quan

and descriptive stocties outweigh the simpUcity 

offered by an exclusive choice between them. 

1. Using stetistics to identify relationships not 

a^pparent from visual htspection. One view of f

quantitative methods isadi^rsarial. Statistical 

cance Is a dreaded hurctie that must be overco

before a stocfy is considered "legitimate" and w

of discusion. But some comparisons of statist

veraus visual criteria for assi^smg change sug

that statistics are more ofbeai alfy than adversa

relying solefy on visual in^section and subject

ment, we are often likefy to overlook small but 

effecte. 

This view can be generalized to methods of co

stocties. Cooper and Itosentiial (1980) had uni

faculty and graduate stodente summarize the r

of seven investigations of sex difference in tas

sistence. Half of the reviewera were asked to "

whatever criteria you would use if this exercis

being undertaken for a class term paper or a 

manuscript for pubUcatioh" (p. 445), n^iUe th

half were uistracted hi how to use statistical co

torial procedures. WhUe several of the indivi

StocUes ctid not show significant sex difference

statistical procedure demonstrated an overaU 

cant effect favoring females (p = .016). 

Descriptive reviewera were significantiy more 

than statistical reviewera to find littie or no su

for the hypothesis of a sex difference hi persis

"Tractitional reviewera either neglect probabiU

combhie tbem hituitivefy in an overfy conserv

fashion" (Cooper and Rosenthal, 1980, p. 448

However, the statistical reviewera ctid not 

unquestioidn^ accept the hypothesis as "pro

FsgeSS GAO/FBMI>>10.1.2 Evaluatton S
Chapters 

Special Topics In Evaluation 

Synthesis 

No one in either group of reviewera conclud

there "definitefy" was support ofthe hypoth

FXuthermore, the type of reviewhig procedu

strongfy related to recommendations for futu

research or to judgmente about the method

adequacy of stodies. Statistical reviewera ca

interpreted rather than bUndfy accepted num

mdexes. 

These findings suggest that statistical proce

help an evaluator identify relationships that 

be large enough to detect mformaUy. Tbeir 

should mcrease as the number of stodies gr

or vdien a program effect Is smaU. One mig

wixy an evaluator should be excnted about t

positive but smaU effecte. We can suggest tw

sons. First, the limite on the degree of contr

be exerted over program participante in edu

or medical innovattons are likefy to lead to 

uicremental gains rather than "slam-bang" 

(GUbert, Ught, and Mosteller, 1975; Gottm

Glass, 1978). Second, when a smaU effect is 

it sometimes can be enhanced by program 

This requires a judgment about whether a m

finding is wortii purauihg. In such mstanc

analysis and expert judlgment become parti

important. TMs brings us to suggesting ano

in which descriptbe evidence can be aUied 

titative finctings. 

2. Using nonquantitative evidence after det

programi! effect. Statistical procedures cam 

toidentif ientifi y smaU effecte and to formaUze the 

for unusualfy successful or unsuccessful prp

comes or outUera. But such finctings, stand

are not veiy informative. Suppose a reviewe

at a dozen Head Start evaluations findls that

average, curriculum A sUghtiy outperforms 

a review of 10 studies of urban hi]^ sdiool

to be unusuaUy effective. What is one to ma

resulte? Fonnal procedures can detect subt

FBge87 OAO/FEMD-10.1.2 Evaluatio
Chapters 

Special Topics in Evaluation 

^thesis 

differences but they cannot explam them. The

starting point, not a final answer. 

After an effect Is identified statisticalfy, the re

must tiy to explain ^riiy this finding existe. Is 

cable? What program characteristics are resp

Can it be enlarged or improved? Answering th

questions requires fiirther efforte that often r

heavUy on case stodies and descriptive eviden

QuaUtative mfonnation may be necessaiy to e

the quantitative finctings. 

A more general point is that quatitative case 

descriptions are paiticidtufy valuable hi helpi

gram managera interpret statistical findings. 

managen are conscientious and want to stren

their prograins as much as possible. For them

espedalfy usefid to have descriptive date such 

y/hat are the characteristics of successful 

hnplementations? How were the teachera trah

How were parente involved? What were detaU

educational program? this hiformation heli» 

agera incrementaUy hnprove programs, u^teg 

parative findings from a review that ^veS insi

about why certain versions of a program wor

than othera. (Tase stocfy and other discursive 

tion can help a manager at a "micro" prog|ra

and at the same time it can inform "macro" c

about program effecrtiveness, sometimes acro

dreds of local sites. 

3. Usfaig the alliance to capitaUze on conflic 

outcomes. We have emphasized the valUe of 

quantitative and descriptive stocties as alUes 

than adversaries for dato syntheds. For exam

review some yeara ago, the two different sort

luetics led to sharpfy csontrasting findings bu

theless Ulustrate our argument, bi the 1946's

of educatora and psychologlste working with 

retarded hicUviduals came to beUeve that ^ut

acid would hnprove a person's capacity to lea

VttSB OAOyPEMD-lCU Evalaation 

3a'i ,:,
Chapte

r 6 

Special Toplca In Evaluation 

syntheaia 

ii.'' 

'.'pC 

that tbis would be reflected by higher IQ sc

late 1940's and earfy 1950's, a series of unc

stocties and case reporte appeared in the m

psychological Uteratures, most of them fmd

modest improvement m IQ's of retarded pe

receivmg this drug (Kane, 1953; Levine, 19

merman and Burgmeister, 1950). 

Ibese findings ctid not go unchaUenged. Sk

pointed out many threats to the vaUctity of t

and questioned how this drug could work p

calfy to improve IQ. A series of controUed c

trials were carried out to examhie the effecr

tamic acid more systematicaUy (McCuUoch

Quinn and Duriing, 1950; Zabrenko and Ch

1953). For example, McCSdlocb used match

mental and control groups, with the contro

a placebo. C^aretakers apd examiners were 

informed of subjects' group membership. S

these experimente showed quickfy and conv

that ^utamic acid ctid not outperform the p

although both groups showed an improvem

people receiving no treatment at aU (the us

ctial care common in the 1940's). In 1960, A

Ross mimmarized the discrepant finctings b

case reporte and experimental stocties and 

that the experimental evidence was far mor

vinchig: glutamic add was ineffective. 

It is tempting to conclude from ttds exampl

controUed, experimental, quantitative stud

"right" vidiile the uncontrolled stocUes were 

and that the latter served no useful scientifi

We come to a ctifferent conclusion: the coid

resulte cany valuable information about ho

improve the Uves of retarded people. The c

experimente are indeed convincing that glu

does not raise IQ. But something was worid

patiente' behalf, since most of the eariier ca

documented IQ gains. Sdentiste were press

account for the improvement. 

Page 88 GAO/FBIID-IO.1.8 Evaluati
Chatters 

Special Toplca In Evaluation 

^theals 

Contrasting the controUed and uncontroUed s

prompte us to examine the context hi which t

was administered. Including the uncontroUed 

m a synthesis reveals an example of "stodying 

wrong treatment" People receivmg glutamic 

got environmental stimulation far b^ond wha

"usual." Increased attention and expectations 

seemed to improve the perfonnance ofthe "p

group" in the experimental trials. One stocfy (

brenko and Chambera, 1953) focused on the 

"environmental stimulation" hypothesis direc

confirmed ite posttxve effect on IQ. 

This example IUustrates how different f oims o

dence, taken together, can lead to msi^te wi

hnportant poticy impUcations. Tbe seemingfy 

sistent findhigs end up displaying information 

both glutamic acnd and supportive environme

flicte in outcomes have not hindered us. The

enriched educational practice. 

The glutamic acid controversy occurred over 

ago, but the lesson stiU api^Ues today. Tlie th

that different types of evidence may be compl

taiy and that singlemindedhess about etther q

tive or quatitative ^proaclies to synthesis im

unnecessary Umite on what we can leam from 

work of othera. The pursuit of good science s

transcend personal preferences for numbera 

narrative. 

Exploiting 

Differences in 

Study Fuuitings 

To benefit from discrepancies among stocties

uncovered through a quantitative or nonquan

approach, we must repeatfedfy ask the questio

may explain the different findings? TVyhig to 

tbis forces a systematic inquiiy that may or m

be quantitative. There are at least five specifi

seek out and conflim explanations for confUc

findings. 

mFsgeSO OAO/FEIIIKIO.1.2 Evalnation 
Chapter 6 

Special Topics In Evaluation 

Synthesis 

Four were described in chapter 2: detennme 

larfy labeled treatmente anci programs differ 

important ways, look for setting-by-treatmen

tions, investigate different research designs 

across stodies, and examine different analys

strategies used in ctifferent stocties. A fifth w

relate background variables to findings. 

One strategy for doing the latter involves co

information about participante' background 

teristics and the design c:haracteristics of th

(for example, the method of assigning subje

groups) and relating this information to stocf

ings. The woik of HaU (1979) Ulustrates thi

strategy. She related several features of each 

the size of the effect of sex differences hi dec

nonveibal cues. These features include both 

ground characteristics ofthe participante an

research design descriptions. For example, s

no relationship between participante' age an

size, wtale the year in iHddch the stocfy was c

tumed out to be important (more recent stoc

tended to show the larg^t effecte). 

A second strategy foUows KUtgaard's (1975

tion "to use the unusual as a guide to the us

"the unusualfy successful (or unsuccessful) 

vide a clearer picture of processes operating 

lesser extent else^ere" (p. 531). Comparin

extremefy successfiU programs to particular

unsuccessful ones may produce a Ust of oth

differences between them. For example, com

successfiU titie I program to one that faUed 

may point out differences hi staffing, expenc

curriculums. 

With a few key explanatoiy factora identifie

anafyst can form specific hypotheses about 

may influence finctings. For histance, one m

expect staff-to-chUd ratio to hifluence Head 

effectiveness, but there may also be comple

Page91 GAO/FEBID-10.1.8 Evaluatio
Chapters 

Special Toplca hi Evaluation 

Synthesis 

interactions between this variable and othera, 

the amount of money spent per chUd or total n

of chUdren in the program. The hypotheses ca

examined ushig date firom less extreme studies

example, if staff-to-chUd ratio in Head Start is 

salfy impoitant, there should be some evidence 

across the entire range of sto^ outeomes. In f

since pubUc poUcies or regulations wUl often m

ence the "usual" more than the "unusual," this 

can be critical. 

A third strategy looks at what is "typical." Foc

on atypical programs should not deter an anal

examining tiie msyor bulk of the stocties for ba

ground features related to outcome difference

just because a stody outcome falls in the midd

distribution, this does not hidicate that the pro

typical. It Is possible that a highfy succsessful p

or curriculum Is paired with unusuaUy needy 

participante, or poor resources, resulting in a m

ocre final perfonnance leveL In these mstance

analyst would ideaUy want to acliust for some 

ground factora before searching for effective o

fective prograins (see KUtgaard, 1975, for fiu

discussion). A "typical" program may ^pear 

"atypical" after adljustmente are made for back

characteristics related to outoomra. 

The examination of stocties that have roughf

"average" outcomes can be valuable In anoth

Focusing on extremes pute our emphasis on I

fying program or participant differences in o

explain ctivergent ftndmgs. But in large ^nthe

invohing many potential background variable

other side of the coin Is hnportant as weU. Ex

StocUes with shnilar outcomes may be usefid h

fying inoperative variables. For example, sup

10 Head Start prograins produce relativefy co

resulte. Suppose also that iriiUe the program 

curriculums and participante are quite shnUa

fonnal educational level of the teachera varies 

Page 92 OAO/FEBa>-10.1.a Evaluation 

^'Sfe 
Chapter 6 

Special Toplca tn Evaluation 

Synthesis 

dramaticaUy across centera. This fact by itse

not prove that teacher education is unhnpor

it may interact with other measured or unm

variables. But it would strongfy suggest that 

education should not be our number one can

a variable that wiU explain outeome differen

there are usuaUy enormous numbera of vari

we think might be hnportant, tius process of 

at "typical" outcomes can help Umit the field 

first-cut analysis and stucfy designs. 

Anticipatuig 

Problems That 

Might Emerge 

Publication Bias PubUcation bias resulte when not aU stocties 

drug or therapy are equaUy likefy to appear 

ereed joumal. A fmding may be more likefy 

Ushed in a joumal If it turns up as statisticaU

significant. So if a synthesis mcludes onfy pu

stocties, one might suspect a bias toward lar

tisticaUy significant, effecte. 

Reviewera hi education and psychology hav

empirical support for this view. For exampl

(1980) gathered groups of stocties assessing 

tions hi education. She found empiricaUy th

effect sizes were noticeabfy smaUer for unpu

stocties than for pubUshed stocties. Greenwa

also suspecting pubUcation bias in psycholo

research, coUected date from a group of re

the Joumal of PersonaUty and Social Psycho

surveyed authora who had recentiy pubUshe

joumal. He asked two key questions. First, "

hiltial fiiU-scale test ofthe focal hypothesis t

rejection of the nuU hypothesis, what is the 

that you wiU submit the resulte for pubUcati

Page 98 OAO/FEMD-10.1.2 Evaluatio
Chapters 

Special Topics In Evaluation 

Synthesis 

immectiatefy, before more date coUection?" Th

response for reviewera was more than 40 perc

for authora it was 58 percent. The second que

was, "After an initial fiiU-scale test of the focal 

esis that does not aUow rejection of the nuU h

esis, what is the probabiUty that you wiU subm

resulte before further date coUection?" The m

response for reviewera was 5 percent, and fo

it was 6 percent. If these resulte are even roug

the baUpark of reaUty, we see that a statisticaU

icant fincting is nearfy 10 times more Ukefy th

significant finding to be submitted for pubUca

refereedjoumal. 

This Idea led Rosenthal (1979) to cohi the ter

"file drawer problem." His thou^t is that for 

pubUshed stucfy there may be several sitting i

researcher's file drawer, unsubmitted or unp

because the researcher ctid not tum up statist

significant resulte. Ignoring this problem and 

onfy at published stodies can lead an evaluat

overestimate a treatment effect, perhaps dram

(Lane and Dunlap, 1978; Orwm, 1983; Rose

1979; Shadish et al., 1987; Sunes, 1987; Som

1987) 

Combining Results 

Across Different 

Treatments 

Are the treatmente given hi different stocties 

enough so that resulte can be combined In a 

fashion? Answering this question is probabfy 

social science reseuchthan hi mecticuie, but 

be asked in drug trials nonetheless. In the fie

hunting, the National Academy of Sciences is

report aggregating findings from many ana

broad job training program caUed YEDPA th

unemployed youth. Tlieir biggest finding is th

specific protocol for this training program va

enormousfy from site to site, despite the com

"template" over the training workshop's doo

site. Borach (1980) raises, m this academy r

fundamental question of wtiether the resulte f

Fage 94 OAO/nBHD-lClJi Eyalnattoa
Chapter 6 

Special Topics In Evaluation 

Synthesis 

these several sites are combuiable at aU. (Cha

al., 1987; Cordray, 1990; HoagUn et al., 1985

et al., 1985; Sacks et al., 1985) 

Examining Control 

Groups Ul Different 

Studies 

Have control groups ui different stodies been 

ined for similarities and differences? This qu

appUes specificaUy to comparative stocties. W

some stocties show a treatment group outper

the controls whUe othera show no difference

reviewer asks why. One possible explanation 

control groups hi various stocties are fimda

ctifferent. 

Control groups might differ simpfy by how d

researchera define them. Some studies com

treatment to a "control" that is no treatment 

Othera compare a new treatment to a "contr

an old or standard or existing treatment. StiU 

compare a new treatment to "controls" that 

altemative new treatmente. In each of these 

circumstances, there Is a clear comparison g

the group's fundamental purpose varies. 

An example of these ctifferent definitions fro

day-care Uterature comes from the work of R

(1979). Ruopp examhied many stocties of a 

caUed "developmental day care" for young c

as part of a project for the U.S. Department 

and Human Services. By examining control 

depth, tius researcher found at least four dif

khids: chUdren cared for fuU thne by a pare

home, chUdren hi nurseiy school, chUdren h

costiy care caUed "custoctial day care," and 

cared for in a private home by adulte other t

parente. Simpfy aggregating finctings across 

four kinds of comparative stocUes ctid not m

The resutts tum out to depend heavify upon 

kind of comparison group Is used. 

PageSS GAO/PEIIID-IO.1.2 Evaluatio
Chapters 

Special Topics In Evaluation 

Synthesis 

The fundamental pouit here, of course, is that a

gating across aU available stocties regarcUess of 

form ofthe control group can dUute rather than 

strengthen the Inferences from a research revi

(BaUar et al., 1986; Begg and BerUn, 1988; Ug

PUiemer, 1984; Wortinan, 1984,1985) 

The Evaluator's 

Attitude Toward 

Conflicting and 

Discrepant Outcomes 

We have left this question for last because we b

it is the most important and yet the hardest to d

with concretefy. It is astonishing how often eva

are surprised that different stocUes of the same 

or treatment produc:e discrepant resulte. What 

expect? It would be remarkable If each of 30 in

dent stodies evaluating a new drug for high blo

pressure found that it brought pressure down b

exactfy 10 systoUc unite. Indeed, it would be m

than remarkable—It would be suspect. Some ch

variation among fbidings is expected. 

Usualfy reviewera have the opposite problem. M

summaries flounder because inctividual stocties 

highfy discrepant resulte. So a productive hUtla

in quantitative analysis is searching for orderfy 

tems of resulte. Probabfy the easiest way to do t

with a shnple gjraph. Plotting stocfy outcomes o

axis and their firequency on the Y axis c:an offe

prisuigfy rich msighte. Light and KUemer (198

describe a number of simple graphic proc:edur

examining variation among findings. He>.-e we 

mention onfy the briefest summaiy of inferenc

a simple graph of stocfy outcomes. 

Furst, if treatmente in several stocUes are reaif

the graph should be weU-behaved. It should lo

i^iproximatefy like a nonnal distribution, sugg

that differences among finctings are basicalfy th

result of sampUng error. If outcomes look gros

irregular, a reviewer must question whether aU 

come from the same population. For example, 

bhnodal distribution would be a fust inctication 

Page 96 OAO/FEBilD-lO.1.2 Evaluation l^
Chapter 6 

Spedal Topics In Evaluation 

Synthesis 

group of studies should not be combined in t

a way—there might be two underlying popula

The chaUenge for a reviewer is then to identif

tora that ctivide stocties into two groups. 

Second, a graph should make outUera more n

able. These extreme observations may or ma

bother a reviewer, dependmg upon the purp

review. If ite purpose is to identify a typical o

value, a few scattered outUera cany no speci

mation. But if the reviewer's purpose is to sp

rare faUure of a new drug, or an exceptionaU

cessfiU curcumstance for that drug, identifym

can be the most hnportant part ofthe entire 

To tie this back to the earUer ctiscnission of w

tion drives the evaluation synthesis, outUera 

ciaUy important when a researcher is lookm

subjec^by-treatment hiteractions—say, certa

of iUness in which specific drugs work espec

or especiaUy poorfy. 

After finding outUera that seem inq[)ortant, th

reviewer must look for explanations. Why m

have happened? Is it just a chance fincting? S

group of heart bypass suigeiy stocties have a 

cluster of particuburfy successful reporte. Th

c:haUengiiig question is vidiether they share a

feature. Perhaps the exceptionaUy successfu

aU involve younger patiente. Perhaps th^ w

done at large urban ho£Q>itals with exception

ties. There are usualfy many possible explan

similarfy labeled treatmente or prograins ma

impoitant ways, there may be setting-by-trea

interactions (that Is, a program or treatment 

more or less effective depending on ^ o par

hi it, where it is administered, or some other 

tional fscrtor), different stocties may have be

designed differentfy, and analysis strategies 

ctifferent stodi^ may vary. 

Page 97 GAO/PEMD-IO.1.2 Evaluation 
Chapter 6 

Special Topics In Evaluation 

Synthesis 

Determining a convincing reason or reasons i

chaUenge to the evaluator. This brings home t

enormous value of successfidfy comblnhig sub

and technical knowledge in syntheses. It is eas

enough to graph outeomes and spot outiiera. I

much harder to identify what features consiste

tinguish the exceptional stocties from the oth

(BerUn et al., 1989; Light and PUiemer, 1984

1990; Toth and Horwite, 1983; Yeaton and W

1984,1985). 

PageSS 6AO/FE1ID-10.1.2 Evaluation 
MH. 

BLANK RA8E 

Page 99 GAO/PEIID-IO.1.2 Evaluation S
Bibliography 

Abrami, P. C, P. A. Cohen, and S. D'ApoUonia. 

"Implementation Problems in Mete-analysis." Re

of Educational Research, 68 (1988), 151-79. 

Anderson, R. D., et al. "Science Education: A 

Mete-anafysis of Major Questions." Joumalof 

Research m Science Teaching, 20 (1983), 379-8

BaUar, J. C, m, and F. MosteUer (eds.). Mectica

of Statistics. Waltham, Mass.: New England Jou

Medicme Books, 1986. 

Baker, P. J., and U. L. Crist. "Teacher Expec:tan

Review of the Literature." In R. E. Snow and J. 

Elashoff (eds.), Pygmation Reconsidered. Worthington, OMo: Jones Publishmg Company, 197

Bangert-Drowns, R. L. "Review of Developmente 

Mete-anafytic Method." Psychological BuUetin, 

(1986), 388-99. 

Bayarri, M. J., and M. DeGroot. "Bayesian Anal

Selection Models." The Statistician. 7 (1987), 1

Bearman, J. E., R. B. Loewenson, and W. H. Gu

"Muench's Postulates, Laws, and CoroUaries, or 

metricians' Views on CUnical Stocties." Biometr

Notes, number 4. Bethesda, Md.; Office of Biom

and Epidemiology, National Eye Institote, Natio

Institotes of Health, 1974. 

Becker, B. J. "Influence Again: An Examhiation 

Reviews and Stocties of Gender Differences in S

Influence." The Psychology of Gender; Advanc

Through Mete-Analysis, 178-209. Baltimore. M

Johns Hopkms University Press, 1986. 

Becker, B. J., and L. V. Hedges. "Mete-anafysls 

Cognitive Gender Differences: A Comment on a

Analysis by Rosenthal and Rubin." Journal of 

Educational Psycdiology. 76 (1984)^683^7. 

Page 160 GAO/FI^fflD-10.1.2 Ewdnathmfl
BlhUography 

Begg, C. "A Measure to Aid m the Interpretation o

PubUshed CUnical Trials." Statistics hi Medicme, 

(1985), 1-9. 

Begg, C, and J. A. BerUn. "PubUcation Bias: A 

Problem hi Inteipreting Mectical Date (with 

Discussion)." Joumal of the Royal Statistical Soci

series A, 151 (1988), 419-63. 

Belsky, J., and L. D. Sternberg. "The Effecte of Da

Care: A Critical Review." ChUd Development, 49 

(1978), 929-49. 

BerUn, J. A., C. B. Begg, and T. A Louis. "An Ass

ment of PubUcation Bias Using a Sample of Publi

Clinical Trials." Journal of tiie American Statistic

Association, 84 (1989), 381-92. 

BisseU, J. W. "The Effecte of Preschool Prograins 

Disacbrantaged OUldren." Ph.D. dissertation. Ha

Graduate School ofEducation, Cambridge, Mass

June 1970. 

Bomstehi, R. F. "Exposure and Affect; Overview 

Mete-anafysis of Research 1968-1987." Psycholo

BuUetin, 106 (1989), 265-89. 

Borach, R. F. An ^praisal of Education Program 

Evaluations: Federal, State, and Local Agencies. 

Washington, D.C: U.S. Departmentof Education

June 30,1980. 

Bowero, T. G., and G. A. Clum. "Relative Contr

of Specific and NonspecUic Treatment Effecte: 

Meta-analysis of Placebo-cx>ntroUed Behavior 

Research." Psychological BuUetin, 103 (1988)

315-23. ^~ 

Bredderman, T. "The Influence of Activity Based 

mentaxy Science Programs on Classroom Practie

Page 101 GAOyFEllb-lO.1.8 Evatoatton Syn

^Vfi^^Mi^ir^^^/^^^^i^ . ^..-.^•••-vl.-^V:J-". .•.:^: •^.U..^:...^Ul^\^^::.^.•:^•J.^.^Xi^<:'•^ilZ^^^^ 
Bibliography 

Quantitetive Synthesis." Joumal of Research i

Science Teachmg, 21 (1984), 289-303. 

Brown, R. "The Issue of Independence of Effe

hi Mete-Analyses." Presented at the annual me

the American Educational Research Associatio

Francisco, CaUf., AprU 16-20,1986. 

Biyant, F. B., and P. M. Wortinan. "Issues in D

Synthesis." New Directions for Program Evalu

No. 24. San FVancisco, CaUf.: Jossey-Bass, 19

Biyant, F. B., and P. M. Wortmann. "Methodo

Issues hi the Mete-anafysis of Quasl-experime

Evaluation Studies Review Annual. 10 (1985)

Biyk, A. S., and S. W. Raudenbush. "Heteroge

Variance hi Experimental StocUes: A ChaUeng

Conventional Inteipretations." Psychological 

104 (1988), 396-404. 

BuUock, R. J., and D. J. Svyantek. "Anafyzhig 

Mete-anafysis: Potential Problems, an Unsucc

RepUcation, and Evaluation Criteria." Jouma

AppUed Psychology, 70:1 (1985), lOS^IS: 

CampbeU, D. T., and R. Borach. "Making the 

Randomired Assignment to Treatmente by Co

ering the Altematives: Six Ways in Which 

Quasi-Experimental Evaluations in Compens

Education Tend to Underestimate Effecte." f

Evaluation and Experiment, C. A. Bennett an

Lumsdaine (eds.). NewYork: Academic Pres

Cariberg, C. G., et al. "Mete-anafysls In Educa

Repfy to Slavin." Educational Researoher. 13

(1984), 16-27. 

Cariberg, C. G., and H. J. Walbeig. "Techniq

Research Synthesis." The Joumal of Spec^ 

Education, 18 (1984), 12-49. 

Page 102 OAO/PEBID-IO.1.2 Evaluation 

^"^^""'^"""^^^"^•""^^•^^~^^~*^"^**^^^""i"TMiiM*iiiTMiWiiiii^^ 
BibUography 

Center, B. A., R. J. Skiba, and A. Casey. "A Metho

ology for the Quantitative Synthesis of Intra-subje

Design Research." The Journal of Special Educat

19 (1985-86), 388-400. 

Chahnera, T. C, et al. "A Methodology for Assess

the QuaUty of Randomized Control Trials." Contr

CUnical Trials. 2 (1981), 31-49. 

Chalmera, T. C, et al. "Meta-analysis of CUnical T

as a Scientific DiscipUne, II: RepUcate VariabUity 

Comparison of Stocties That Agree and Disagree." 

Statistics hi Medlchie. 6 (1987), 733-44. 

CheUmsky, E. "The Definition and Measurement 

Evaluation QuaUty as a Management Tool" New 

Directions for Program Evaluatton, No. 18. San F

dsco, CaUf.: Jossey-Bass, 1983. 

Chipman, S. F. "Far Too Sexy a Topic." Review o

The Psychology of Gender; Advances Through 

Meta-AnafystsTEducational Researcher, 17:3(198

46-49. 

Chow, S. L. "Significance Test or Effect Sire?" 

Psychological BuUetin, 103(1988), 105-10. 

CichreUi, V. "The Impact of Head Start; An Evalu

of the Effecte of Head Start on ChUdren's Cogniti

and Affective Development." Clearinghouse for F

eral Scientific and Technical Information, 

Washington, D.C, 1969. 

Coldttz, G., J. MUler, and F. MosteUer. "The Effec

Stody Design on Gate in Evaluation of New Tteat

mente hi Medicine and Surgeiy." Drug Informati

Journal, 22 (1988), 343-52. 

Cook, T. D. "What Have Bbbck ChUdren Gained A

demicalfy firom School Integiration? Examhiation 

the Mete-analytic Evidence." fai "School 

Fage 108 OAO/FBIID-IO.1.2 Evafautlon Synt
BibUography 

Desegregation and Black Achievement," T. Co

(eds.). Unpublished report. National Institote 

Education, Washhigton, D.C, 1984. 

Cook, T. D., and L. C. Leviton. "Reviewing the 

ture: A Comparison of Tractitional Methods w

Mete-Anafysis." Joumal of PersonaUty. 48 (19

449-72. 

Cooper, H. M. "Scientific GuideUnes for (indu

Integrative Research Reviews." Review of Ed

Research. 52:2 (1982), 291-302. 

Cooper, H. M. "Literature Searchhig Strategi

Integrative Research Reviewera: A Fust Surv

Knowledge: Creation, Diffusion, Utilization, 8

(1986a), 372-83. 

Cooper, H. M. "On the Social Psychology of U

Research Reviews: The Case of Desegregatio

Black Achievement." In The Social l^diolog

Education, pp. 341-63, R. Feldman (ed.). Cau

England: Cambridge Univeralty Press, 1986b

Cooper, H. M. "Organizing Knowledge S^tbe

Taxonoiny of Literature Reviews." Knowledg

Society, 1 (1988), 104-26. 

Cooper, H. M. Integrating Research: A Guide 

erature Reviews. 2iid ed. Newbury Park. Cali

1989. 

Cooper, H. M., and R. Rosenthal. "Statistical 

Traditional Procedures for Summarizing Res

Fbidmgs." Psychological BuUetin. 87:3 (1980

442-49. 

Cordray, D. S. (ed.). "UtUlzhig Prior Research 

Evaluation Planning." New Directions for Pro

Evaluation, No. 27. San Ftanclsco, CaUf.; Jos

1985. 

Fage 104 OAO/FEIID-10.1.2 Evahiation 
BibUography 

Cordray, D. S. "Quasi-experimental Anafysis: A 

ture of Methods and Judgment." New Direction

Program Evaluation, 31 (1986), 9-27. 

Cordray, D. S. "Meta-anafyslB: An Assessnient 

the Poticy Perspective." fai The Future of 

Meta-Anafysis; K. WacAiter and M. Straf (eds.)

York; RusseU Sage Foundation, 1990. 

(Xurtis, G. K., and J. I^ Saut^ 

Toward Persons with biaabilttJes: A Reviw^ 

Reviews." International Jouraial of Sh^ ^ 

2:2 (1987), 103-29. 

Datta, L. "Ilie bnpact of the Westin^pi^ 

uaticm on the PevekiipNn^^ 

C.C. Abt(ecil.),TheEvaaii^^ 

Beveriy HUls, Catif^:r 

DiJtes^W:F.^= i.r p^ydibi^^ 

(l965),;74-79.''. ''yjm^^^^^ 

Dickeii^,K;, et td. !T\d)lic^Edi^ 

Ttials." (JcwtrbUedCaihiMTriato.^ 

Duilak/J. A; "Cknnpfl^^ 

professtonal and PrbfEaaio^ 

BuUetin;86(197^^Bb-92:; • '0}^ '-h •••'"^^

Dush,D;M.,M.::L;;iiW;/anid.lLS^^dhitbe^ 

' "^Self^tatementlliodifi^^ 

ChUd'Beh8vi(>r Diwrdim:;A^IIeia;in^^ 

PaycbPtoidcalBuUetln; 106 Q969)^ 9^ 

Eagfy,'A;>H.vand'MvCro^!d#^''"Gl^^ 

Behavibn A BlelahAhafytiCjRev^ 

chc)toglcal'Ltterature.'^.'l%chblci^i^'^iii^^ 

(1986);':283-308.-. •;- •• ^1^"'''''^''^^^^ 

•••••'.,••••:-;•• • '•.••.•.••••••. • ,.• ' •'••••.:.• ' .••• • ;,Fy'.'•

FhgelOS . •••^••QJliiDmi^^ 

<i yymMmms
Edghigton, E. S. "An AdcUtive Model for Combinin

ProbabUity Values from Independent Stocties." 

Joumal of Psychology, 80 (1972), 351-63. 

Eysenck, H. J. "An Exercise in Mega-SUUness." 

American Psychologist. 33 (1978), 517. 

Eysenck, H. J. "Mete-anafysus: An Abuse of Resear

Integration." The Joumal of Special Education, 18 

(1984), 97-106. 

Femgold, A. "Matching fpr Attractiveness hi Roma

Partoera and Same-Sex Friends: A Mete-anafysls a

Theoretical Critique." Pgycbologlcal BuUetin, 104 

(1988), 226-35. 

FeUhnan, K. A. "Using the Woric of Othera: Some 

Observations on Reviewing and Integrating." 

Sociology of Education, 44 (Wmter 1971), 86-102

Fuiebeig, S. E., M. E. Martin, and M. L. Straf (eds

Sharing Research Data. Washington, D.C; Nation

Academy Press, 19857 

Fisher, R. A. Statistical Methods for Research 

Workere. London; OUver and Boyd PubUshhig Co

1973. 

Fosbuig, S., and F. Giante. "Analysis Plan for the 

ChUd Care Food Program." Submttted to the Foo

Nutrition Service, U.S. Department of Agriculture, 

Abt Associate, Inc., Cambridge, Mass., i^ril 198

GaUo, P. S., Jr. "Mete-Analysis-A Mbied Meta-ph

American PsychologlBt, 33 (1978), 615-17. 

GUbert, J. P., R. J. Light, and F. MosteUer. "Asses

Social Innovations: An Empirical Base for F^Uqr.

Evaluation and Experiment, C A. Bennett and A. 

Limsdahie (eds.). New York: Academic Press, 197

Fage 106 GAO/FElID-18.1.8 Evnlnation SynO
BibUography 

is 

>!•]• . 

'if^'.y^y 

Glass, G. V. "Integrating Fuidhigs: The Mete-

of Research." Review of Research m Educatio

(1977), 351-79. 

Glass, G. V. "BibUography of Writings on the 

tion of Research Fhictings." Manuscript, Labo

Educational Research, Univeralty of Colorado

Denver, Colo., 1978. 

Glass, G. v., B. McGaw, and M. L. Smith. 

Mete-analyris In Research. Beverfy Hills, Cali

PubUcations, 1981. 

Glass, G. v., and M. L. Smith. "Primaiy, Seco

and Meta-Anafysls of Research." Educational 

Researcher, 5 (November 1976), 3-8. 

Glass, G. v., and M. L. Smith. "Repfy to Eysen

American Psychologist, 33 (1978), 517. 

(xottman, J. J., and G. V. Glass. "Analysts of f

rapted Time Series Experimente." In T. R. K

(ed.), Shigle Subject Research. NewYork: Ac

Press, 1978. 

Green, B. F., and J. A. HaU. "Quantitative Me

Literature Reviews." Annual Review of Psych

(1984), 37-53. 

Greenwald, A. G. "Ccmsequences of PnjiuUce 

the NuU HnN>thesis;" Psycdtiotoglcal Bulktln, 

(1976), 845-67. 

Guzzo, R., S. Jac:kson, and R. KatzeU. "Mete-

%••'• -

Jj:;. - • . . .. . 

§y\. Analysis." fai Reseturcltin OrgaidgatipnalBeha

vol. 9, B. Stew and L. Cummings (eds.): Gree

Conn.: JAI, 1987. 

HaU, B. W., A. W. Wanl, and C B. Comer. "P

Educational Research: An Emphrical Stody of 

QuaUty." Presented at the animal meeting cif 

Page 107 GAQ/FBBID-10.1JI Evalnation 

'• 
BibUography 

American Educational Research Association, San 

FYancisco, CaUf., April 16-20,1986. 

HaU, J. A. "Gender Effecte m Decoding Nonveibal 

Cues." Psychological BuUetin. 85 (1978), 845-57. 

HaU, J. A. Statistical Methods for Mete-Anafysis. N

York: Academic Press, 1985. 

HaU, J. A., and L OUdn. "Vote Counting Methods 

Research Synthesus." Psychological BuUetin. 88:2 

(1980), 359-69. 

Haney, W. "Unite of Analysds Issues in the Evaluat

of Project FoUow Throug." Prepared for U.S. Ofi

ofEducation, Contract No. OECO-74-03-94. The 

Huron faistitote, Cambridge, Mass., 1974. 

Hauser-Cram> P.> and J. Shonkoff. "Report ofa 

Meta-Anafysls firom the Earfy Intervention ChUdho

Project." Prepared for the National Institotes of 

Heatth, Bethesda, Md., 1986. 

Hazehrigg, M. D., H. M. Cooper, and C M. Borduh

"Evaluating the Effectiveness of FamUy Therapies

Integrative Review and Analysis." P$ychologlcal 

BuUetin. 101 (1987), 428-42. 

Hedges, L. V. "Distiribution Theoiy for Glass's Est

mator of Effect Size and Related Estimatora." Jou

of Educational Statistics. 6 (1981). 107-28. 

Hedges, L. V. "Estilmating Effect Size from a Serie

Independent Experimente." Psychological BuUetin

92 (1982a), 490-99. 

Hedges, L. V. "Fitting Categorical Models to Effec

Sizes from a Series of Experimente." Joumalof 

Education Statistics. 7 (1982b), 119-37. 

Faga 108 6AO/FEMD-10.1.8 Evaluation Syntl
BibUography 

Hedges, L. V. "Fitting Contmuous Models to Eff

Size Date." Joumal of Educational Statistics, 7 

(1982c), 245-70. 

Hedges, L. V. "A Random Effecte Model for Effe

Sizes." Psychological BuUetui, 93 (1983), 388-9

Hedges, L. V. "Advances in Statistical Analysis." 

Directions for Program Evaluation, 24 (1984a), 

25-42. 

Hedges, L. V. "Achrances hi Statistical Methods f

Mete-Anafysis." In Issues in Date Synthesis, pp. 

25-44, W. H. Yeaton and P. M. Wortinan (eds.). 

Directions for Program Evaluation, No. 24. San 

dsco, CaUf.; Jossey-Bass, 1984b. 

Hed^s, L. V. "Estimation of Effect Size Under N

random SampUng: The Effecte of Censoring Sto

Yieldmg Statistic^alfy Insignificant Mean Differe

Journal of Educational Statistics. 9 (1984c), 61-

Hedges, L. V. "Estimating Effect Size from Vote 

Counte or Box-score Data." Presented at the an

meeting of the American Educational Research 

ciation, San FYandsco, CaUf., April 1986a. 

Hedges, L. V. "Issues in Mete-anafysis." Iteyiew

Research hi Education. 13 (1986b), 353-98. 

Hedges, L. V. "Mete-Anafysis." Review of Resea

Education, 17 (1986c). 1-55. 

Hedges, L. V. "fanprovhig Statistical Procedures 

VaUdity (SeneraUzation." hi Test Vrilctity forthe 

1990's and Beyond, pp. 191-212. H. Brtunand

Wamer (eds.). HUlsdale, N J.; Lawrence Erlbau

I 1988. 

Hedges, L. V. "Tlie Meta-analysis of Test VaUdtt

I Stocties: Some New /^proaches." fai Test VaUdi

Faga 109 OAO/PEIID-10.1.8 Evalnation Sy
BibUography 

Warner and H. Braun (eds.). HiUsdale, N J.: Law

Erlbaum, 1988b. 

Hedges, L. V., and I. OUdn. "Analyses, Reanafyse

and Mete-aiudysls." Review of G. V. Glass, B. M

and Maiy L. Smith, Mete-anafysis in Social Resea

Contemporaiy Education Review, 1 (1982), 157

Hedges, L. V., and I. OUdn. Statistical Methods f

Mete-anafysis. New Yoric: Academic Press, 1985

Hedges, L. V., and I. Olkhi. "Mete-anafysls: A Re

and a View." Educational Researcher. 15:8 (198

14-21. 

Herson, M., and D. H. Barlow. Slngle^ase Expe

mental Designs: Strategies for Stodyhig Behavio

Change. NewYork: Pergamon Pre^, 1976. 

Himel, H. N., et al. "Adjuvant Cbemothert^iy for 

Breast Cancer: A Pooled Estimate Based on Pub

Randomized Control Trials." Journal of the Am

Medical Association, 256 (1986), 1148-59. 

Hine, L. K., N. Laird, and T. C Charhners. 

"Mete-Anafysis of Randomized Control Trials of 

tine Antiarriiythmic Therapy of Post Acute BlyO

Infarcticm Patiente Incticates Uigent Need for M

Trials." Manuscript, Harvard School of PubUc H

Boston, Mass., n.d. 

HoagUn, D. C, F. MosteUer, and J. W. Tukey (e

Exploring Date Tables. Trends, and Shapes. New 

York: Wdey, 1985. 

Howard, K. I., et al. "The Dose-response Relatio

in Psychotherapy." American Psychologist, 41:2 

(1980), 159-64. 

Fage 110 OAO/FEMD-IO.1.2 EvaluationSy

ili^!:vfe!4.:;v:}i.:;!c;:;,:^^:^..,^t;;^^^ . • ;.. '. ^ ^ 
BibUography 

Himter, J. E., F. L. Schmidt, and G. B. Jackson

Analysis: Cumulating Research Finctings Acro

Stodies. Beverfy HUls, CaM.: Sage, 1982. 

Hyde, J. S., and M. C Unn. "Gender Differenc

Veibal Ability: A Mete-anafysls." Psycboloidca

BuUetin, 104 (1988), 53-69. 

Iyengar, S., and J. B. Greenhouse. "Selection M

and the FUe Drawer Problem." Statistical Scie

(1988), 109-35. 

Jackson, G. B. "Methods for Integrative Revie

Reviews of Educational Research, 50 (1980), 

Johnson, B. T., and A. H. Eagley. "Effecte of In

ment on Persuasion: A Mete-anafysls." P$ycfa

BuUetin, 106 (1989), 290-314. 

Kamin, L. Comment on Munsinger's review 

tion stodies. ^diolo^ralBuIletin, 85 (1978

194-201. • 

Kane, E. D. "Differential IncUcations for the U

Glutamic Acid." American Jouroal of Psychia

(1953), 699-700^: «-— 

Kate, B. M., L. A. MarascuUo, and M. McSween

"Nonparametric Alternatives for Testing Mfdn 

Ifypotheses: A Moclel ifoir Combhiing Data Acr

Independent Stocties." Rsychcrfoglcal BiiUetin. 

(1985),200-8. ••• ••••• •• •••; 

Kennedy, M. M. "Develophig an Evaluation Pl

PubUc Law 94-142." New Directions for Prog

Evaluation. 2 (1978). 19^6. •• •• 

Kennecfy, M. M. "GeneraUzfaig FVom Single Ca

Stodies." Evaluation Quarterfy. 3 (1979). 661

Pagelll GAfMFBIID-lO.1.2 BvalnatlooS

: 

•.-',.•• . • • •: • 
BibUography 

KUtgaard, R. "Going Beyond the Mean in Educati

Evaluation." PubUc PoUcy. 23 (1975), 59-79. 

Kohl, H. "Great Expectations." In R. E. Snow and 

D. Elashoff (eds)., PygmaUon Reconsidered. Wor

thmgton, Ohio; Jon^ Publishing Company, 1971

KratochwUl, T. R. "N = 1: An Attemative Researc

Strategy for School PsychoIogiBte." JoumalofSch

Psychology, 15 (1977), 239-49. 

KratochwUl, T. R. Shigle Subject Research. New Y

Academic Press, 1978. 

KuUk, J. A., and C C KuUk, "Operative and Inter

able Effect Sizes in Meta-analysus." Presented at t

annual meeting of the American Educational Res

Association, San Francisco, CaUf., April 16-20,1

Kulik, J. A., and C. C KuUk. "Meta-analysis in 

Education." Intemational Journal of Educational 

ReseartA, 13:3 (1989). 221-340. 

KuUk, J. A., and C C KuUk. "Mete-anafysls: Hist

Orighis and Contemporaiy Practice." Presented a

annual meeting of the American Educiational Res

Association, New Orleans, La., ^ril 1988. 

KuUk, J. A., and C-L. C KuUk. "Operative and 

Interpretable Effect Sizes hi Mete-Analysis." Pres

ented at the annual meeting of the American Edu

tional Research Association, San Fkanclsco, Cal

April 16-20,1986. 

KuUk, J. A., C C KuUk, and P. A. Cohen. "A 

Mete-Anafysis of Outcome Stodies of KeUer's Per

alized System of Instmction." American Psycbolo

34 (1979), 307-18. "

Lane, D. M., and W. P. Dunlap. "Estimating Effec

Size: Bias Resulting from the Significance Criteri

Fage 112 QAO/FEMD-IO.1.2 Evaluation Syn
Editorial Decision." British Joumal of Matiiemati

and Statistical Paychology. 31 (1978). 107-12. 

Leifer, A., N. Gordon, and S. Graves. "(ChUdren's 

Television More Than Mere Entertahunent." Harv

Educational Review, 44 (1974), 153-71. 

Levin, H. "Cost-Benefit and Ciost-Effecllveness 

Analysis." New Durections for Program Evaluation

(1987), 83-99. 

Levhi, H. M., G. V. Glass, and G. R. Meister. Cost

Effectiveness of Four Educational Interventions. 

ford, Calif.: Institote for Research on Edticationa

Finance and (governance, JSchool ofEducation, S

ford UnhrersUy, 1984. 

Levine, E. S. "( ^ We Speed Up the Slow ChUd?" 

Votta Review. 51 (1949), 169-70. 

Lewis, C E., et aL "An Evaluation ofthe Impacit 

School Nurse Practitiohera." Journal of School 

Heatth, 44 (1974). 331-35. " " 

Li^t, R. J. "Cq)ttaUzhig on Variation: How Conflicting Research Fhictings Can Be Helpful for Po

Educational Researcher. 8:9 (1979). 7-11. 

Light, R. J., and D. B. PUiemer. "Numbera and Na

tive: Combhdng Thehr Strengths in Research 

Reviews." Harvard Educiatibnal Review, 52 (1982

1-26. ' ""^ 

Light, R. J., and D. B. PUiemer. Sumndng Up; The

ence of Reviewing Research. (Cambridge. Btoss.: H

vard Uidverstty ftess, 1984. 

Ught, R. J., and P. V. Smith. "Accumubiting Evid

Procectures for Resolving (Contradictions Among 

I Dififerent Studies." Harvard Educational Review. 

I (November 1971), 4^9-tl. ~ ~ 

::• • 

iy- • • 

}y 

i':••'•• 

<•.':' • • 

Page 118 OAO/FEBID-10.1.8 Evalnation Syn
BibUography 

Unn, M. C, and A. C Peterson. "Emergence an

Characterization of Sex Differences m Spatial A

A Mete-anafysis." ChUd Development. 66 (198

1479-98. 

Upsey, M. W. "Juventte DeUnquency Treatmen

Mete-aiudlytic faiquhy into the VariabUity of Eff

Paper for the Research Synthesis Committee o

RusseU Sage Foundation, February 1990. 

Loids, T. A., H. V. Fineberg, and F. MosteUer. 

"Fhidings for PUbtic Health from Metapanafytn

Annual Review of Pubtie Heatth, 6 (1985), 1-2

McKDaU, R. "ChaUenges to a Science of Develop

Psychology." ChUd Development, 48 (1977), 3

McCuUogh, T. L. "The Effect of Glutamic Acid 

Feeding on Cognitive Abilities of faistitutipna

Mental Defectives." American Journal of JMent

Defideney. 65(1950), 117-22. 

McGaw, B. "Meta-analyals;" In Educational R

Methodology imclMeasuregnCTt; An 1^ 

Handbook, pp. 678-85. J. P. KeeVes (ed.). New 

Pergamon Press/1988^ 

Matt, 6. E. "Decision Rules for Selecting Effec

in Meta-analyi^: A Review and Reanalysis pf P

therapy Outcome Stodies." Psycbologjcai BuU

105 (1989), 10645. 

Merenstehi, J. H.,H. Wolfe, and K. M. Baiter. 

Use of Nurse Practitionera in a General Practi

Medical Care, 12 (ld74), 445-62. 

MosteUer, F., and R. Bush. "Selecting Quantita

Techniques." In G. iAndaasy (ed.), l^ancJli^k o

Psychology. Vol. 1. 'lliebiyandMettibd.Ci^ 

Mass.; Acidson Wesley PiibUshihgComjptaiiy, 

Page 114 OAO/FBIID-IO.1.2 Evalnation S
BibUography 

MuUen, B., and R. Rosenthal. Basic Meta-anafysis

Procedures and Programs. HUlsdale, N J.: Lawren

Erlbaum, 1985. 

Munshiger, H. "The Adopted ChUd's IQ: A Critic

Review." Psychological BuUetin. 82 (1974), 623-

Munsinger, H. Repfy to Kamin. P^chological BuU

85 (1978), 202-6. 

National Institute of Education. "School Desegreg

tion and Black Achievement." UnpubUshed repo

U.S. Department ofEducation, Washington, D.C

1984. 

NobUt, G. W., and R. D. Hare. Mete-ethnography

thesizhig QuaUtative Stocties. QuaUtative Researc

Methods, voL 11. Beverfy HUls, CaUf.: Sage, ISig

OUdn, I. "History and Goals." In The FUtare of M

anafysis, K. W. Wachter and M. L. Straf (eds.). N

York; RusseU Sage Foundation, 1990. 

Orwin, R. "A FaU-Safe N for Effect Size hi 

Mete-Anafy^." Joumal of Education Statistics, 8 

(1983), 167-59. 

Oiwin, R., and D. S. Cordray. "Effecte of Defiden

Reporting on MetapAnafysis; A Conceptiud FVam

and Reanalysis." Psychological BuUetin. 97:1 (19

134-47. 

Parker, K. C H., R. K. Hanson, and J. Hunsley. 

"MMPI, Rorachach, and WAIS; A Mete-anafytic C

parison of ReUabUity, StabUity, and VaUdtfy." 

Psychological BuUetin, 103 (1988), 367-73. 

PiUemer, D. B. "Conceptual Issues In Research 

S^thesls." The Joumal of Spedal Education. 18 

(1984), 27-40. 

Fage 118 OAO/railD-10.1 Jl Evaluation l^
BibUognvhy 

PUiemer, D. B., and R. J. Light. "Using the Resu

ControUed Experimente to Constract Social 

Programs; Three Caveate." Evaluation Stocties 

Annual. Beveriy HUls, CaUf.; Sage PubUshhig C

pany, 1979. 

PUiemer, D. B., and R. J. Ught. "Syntheslzhig O

comes: How to Use Research Evidence from M

Stodies." Harvard Educational Review, 50 (198

176-95. 

Prioleau, L., M. Murdock, and N. Brocfy. "An A

of Psychotherapy vs. Placebo Stodies." The Be

ioral and Brafai SdenceS, 6:2 (1983), 276-85. 

Quhm, K. v., and D. Durling. "New Esperimeh

Glutamic Add Therapy: 24 Cases Classified as 

Defidency, Undifferentiated, TYeated with Glu

Add for Sbc Months." American Jounud of Me

Defidency, 66 (1950); 227-34. — 

Raudenbush, S. W. "Magnttude of Teacher Exp

tance Effecte on PupUIQ as a F^mdicm of the C

bUtty of Espectance InidUction: A Synthesis of 

Finctings from 18 JExperimente." Journal of Ed

tional Psychology, 76 (1984). 86^97. 

Raudenbush, S. W,, and A. S. Bi;^. "Emphrical 

Mete-Anafysis." Journal of Educational Statist

(1985), 76-98. 

RwUn. A. Systemtatic Thinking for Sodal Acti

Washhigton, D.C: Tlie BrooUiigs Institution, 

Rosenthal, R. "Combhdng Residte of Independ

Stodies." Psychological BuUetin. 85 (1978), 1

Rosenthal, R. "The 'FUe Drawer Problem' and 

ance for NuU Resulte." PSycAiotoglcad Bulldin, 

(1979), 638-41. 

116 

IxkfM 

OAOmaSD-lOAJi BvainatloaS

•-. ' • ' 

• :• ;> ..,, .:.-.; ..•...-;:.... > .•,•'':•'.
BUiUography 

Rosenthal, R. "Comparing Effect Sizes of Inde

Stodies." Psycholo^cal BuUetin. 92 (1982), 6

Rosenthal, R. Meta-Analytic; Procedures for S

Research. Beverfy HUb. CaUf.: Sage. 1984. 

Rosenthal, R. "Experimenter Expectancy, Cov

Communication, and Mete-Anafytic Methods." 

ented at the annual meeting of the American P

logical Association, New Orieans, La., 1986. 

Rosenthal, R., and L. Jacobson. IfygmaUon in 

Classroom. New Yoric; Hott, Rinehart and Wi

1968. 

Rosenthal, R., and D. B. Rubin, "faiterpersona

tancy Effecte: The First 345 Stodies." The Be

and Brain Sdences. 3 (1978), 377-416. 

Rosenthal, R., and D. B. Rubin. "Cdnqiaring S

cance Levels of Independent Stocties." Psycho

BuUetin, 86 (1979a), 1166-68. 

Rosenthal, R., and D. B. Rubhi. "A Note on P

Variance Explahied as a Measure of the Itepo

Effecte." Joumalof AppUed Social Psycholog

(1979b), 396-96; 

Rosenthal, R., and D. B. Rubin. "A Simple, Ge

Purpose Display of Ma^jpnltode of Eicperiment

Effect." Joumal Of Ediica^onal ftydioloiy. 7

(1982), imW'—^~"^—— ^ 

Rosenthal, R., and D. B: Rubhi. MetarAnafytlc 

dures for Combining Stodlra with Muttlple E

Sizes." PaychologlcalBuUetfai. 99:3 (19Si6)r 4

Rosnow, R. L., and R. Rosenthal. "Statistical 

dures and the JustUlcation of KnowlcNlge te 

Psychological Science.'^ American PaiydiolOg

(1989), 1276-84. ~ 

Fage 117 GAO/FEBID>10.1.2 Evaluation 

fe 
Bibliography 

Rubm, D. B. "Using Empirical Bayes Techniques h

the Law School VaUctity Stocties." Joumal of the 

American Statistical Assodation. 75 (1980), 801-1

Rubm, D. B. "Estimation m ParaUel Randonuzed 

Experimente." Joumal of Educational Statistics, 6

(1981), 377-401. 

Rubm, D. B. "A New Perapective." In The Future 

Mete-Anafysis, K. Wachter and M. Straf (eds.). Ne

York: RusseU Sage Foundation, 1990. 

Ruopp, R. R. ChUdren at tbe Center: Report of the 

National Day Care Stocfy. Cambridge, Mass.: Abt 

Books, 1979. 

Ryan, W. Blamhig the Victim. New York; Pantheo

Books, 1971. 

Sacks, H. S., et al. "Should MUd Hypertension Be 

Treated? An Attempted Mete-anafysis of the CUni

Trials." Mount Shiai Journal of Medidne. 52 (198

265-70. 

Sacks, H. S., et al. "Mete-anafysls of Randomized C

troUed Trials." New England Journal of Medicine, 

(1987), 450-66. 

Setter, W. J. "Conducting Social Program 

Evaluations." Prepared for Bolt, Beranek, and 

Newman, Inc., Cambridge, Mass., August 1980. 

Salzberg, C L., P. S. Strain, and D. M. Baer. 

"Mete-anafysis for Shigle-Subjed Research: When 

Does It Churify, When Does It Obscure?" Remedi

and Special Education. 8 (1987), 43-48. 

Sampson, G. E., et al. "Tbe Effecte of Teacher Qu

tioning Levels on Stodent Achievement: A Quant

Synthesis." Tbe Joumal of Educational Research. 

(1987), 290=95: " 

Fage 118 6AO/FEMD-10.1.2 Bvalnation Synt
BibUography 

Scruggs, T. E., M. A. Mastropieri, and G. Ca

Quantitative Synthesis of Single-Subject Res

Methodology and Vahdation." Remectial and 

Education, 8 (1987), 24-33. 

Shadish, W. R., L. M. Montgomery, and M. 

"How Many Stocties Are in the FUe Drawer? 

ical Esthnate." Presented at the meeting of t

ican Evaluation Association, Boston, Mass., 

1987. 

Shadish, W. R., et al. "Marital/FamUy Thera

Effectiveness; Meta-Anafysls of 163 Random

Trials." Presented at the American Psycholo

Association, New Orleans, La., August 1989

Shapiro, D. A. "Recent AppUcations of Me

hi CUnical Research." CUnical Psycholog

(1986), 13-34. 

Shnes, R. J. "Confronting PubUcation Bias: 

Design for Mete-anafysis." Statistics hi Med

(1987), 66-29. 

Slavin, R. E. "Mete-anafysls In Education; H

Been Used?" Educational Researcher. 13 (1

6-15. 

Slavhi, R. E. "Best-evidence Synthesis: Why 

More." Educational Researcher. 16:4 (1987

Smith, M. L. "PubUcation Bias hi Meta-Ana

Evaluation in Education: An Intemational R

Series, 4 (1980), 22-24. 

Smtth, M. L., and G. V. Glass. "Metaranafys

Psychotherapy Outeome Stocties." America

Psychologist, 32 (1977), 752-60. 

page 119 GAO/PBIID-10.1 J! Evaluati
BibUography 

Sommer, B. "The FUe Drawer Effect and PubUc

Rates hi Menstrual (fycle Research." Psychology 

Women Quarterfy. 11 (1987), 233-42. 

Stanley, J. C "Note About Possible Bias Resultm

When Under-Statisticized Stodies Are Excluded 

Mete-analyses." Journal of Educational Measure

24:1 (1987), 72-76. 

Steinkamp, M. W., and M. L. Maehr. "Grender D

ences in Motivational Orientations Toward Achi

ment in School Sdence; A Quantitative Syntbrai

Joumal of the American Educational Research 

Association, 21 (1984), 39-59. 

Stock, W., et al. "Rigor hi Date Synthesis: A Cas

Stody of ReUabiUty in Mete-anafysls." Education

Researcher, 11 (1982), 10-14. 

Strabe, M. J., W. Gardner, and D. P. Hartmann. 

"Umitations, UabiUties, and Obstades m Revie

the Uterature: The Current Status of Mete-anaf

CUnical Paydiology Review, 5 (1984), 63-78. 

Strabe, M. J., and D. P. Hartman. "Mete-anafysl

Techniques, AppUcations, and Fimctions." Joum

Consulting and CUnical Psychology. 61 (1983), 

Tamir, P. "Mete-anafysls of Cognitive Preferenc

Learning." Journal of Research m Sdence Teac

22 (1985), 1-17. 

Toth, P. J., and R. L Horwite. "ConfUcting CUn

Trials and the Uncertamty of Treating MUd 

Ifypertension." The American Joumal of Mectid

(1983), 482-88. 

Tracz, S. M., and P. B. Elmore. "The Effect of th

>^olation ofthe Assumption of Independence W

Combining Conelation Coeffidente in a 

Page 120 GAO/FEMD-10.1.2 Evalnation Sy
BibUography 

Mete-anafysis." Multiple Unear Regression 

Viewpohite, 14 (1986), 61-80. 

UUman, NeU R. Elementary Stetistics; An Ap

ApproaJch. New York: John WUi^ and Sons, 

Urkowite, A. G., and R. E. Laessig. "Assessin

BeUevabUity of Research Resutts Reported h

Envhoiimental Healiii Matrix." In Prbgrayi 

Evaiuation: Pattemaand Directions. E. CheU

(ed.). Washington, D.C: American Sodety fo

Admhdstration, 1986. 

U.S. General Accounting Office. DisparitieaS

hi Who (»ete Spedal Education. IPE-81-1 .W

ington, D.C.; September 30,1981. 

U.S. General Accounting Office. GETA Progr

Dlsadyantaged Adults; What Do WKnog^

EnroUees. Services/ahd Effectiveness? inB-8

Washington, D.C: June 14,1982. 

U.S. (general Accounting OfBce. Lessoiis Iga

FYom Part; Block Grante: fanpUcations forjG

sional Oversight. iPE-82-8. Washhigton. D;C

tember23,1982. 

U.S. (Jeneral Acoounting Office. Problems an

Options hi Estimating the Size of theIUei^S

Population. IPE-82T9Twasbtagton. D.C> Sep

24,1982. 

U.S. General Acoouitfing Office. Tbe Elderfy 

Benefit FVcro Expanded Hcmtt HeaUb Catei

faicreasihgThgs^ Servicea WUNO^^^ 

Reduction, lPE-83-1; Waddiigton.D.C.: Dec

1982. 

U.S. General Accounting Office. WIC Eyalui

vide Scmie Fetvorable but No Ccmclusive l^>

the Effecte Eacpected for the Spedal Sut 

Fage 121 OAO/FEMD-10.1J1 Evaluation 
BibUography 

Program for Women, Infante, and ChUdren. 

GAO/PEMD-84-4. Washuigton, D.C: Januaiy 3

1984. 

Wachter, K. W. "Distiubed by Meta-Analysis." 

Sdence, November 1988, pp. 1407-08. 

Wachter, K. W., and M. L. Stiraf. "Introduction

The Future of Mete-anafyris. New York; RusseU 

Foundation, 1990. 

Walberg, H. "I^tbesls of Research on Teachin

Handbook of Research on Teaching. 3rd ed., M

trock (ed.). NewYork: MacndUan, 1986. 

Ward, M. M, G. E. Swan, and M. A. Chesney. 

"Aroiuud-Reduction T\reatmente for MUd Hyper

sion: A Metaranalysis of Recent Stocties." fai 

Handbook of Hypertenaon, VoL 9.. Behavioral 

tora In Hypertension, pp. 1-19, S. JuUusandD

Bassett (^.). New Yoric: Elsevier Sdence Pub

1987. 

White, K. R. "The Rekition Between SocUd Eco

Status and Academic Achievement." Psycholog

BuUetin, 91 (1982), 461-81. 

White, K. R., D. W. Bush, and G. C Casto. "Le

from Reviews of Eariy Intervention." The Joum

Specud Education, 19 (1985-86), 401-16. 

White, O. R. "Some Commente Concerning 'Th

Quantitative Syntiiesis of Shigle-Subject Resear

Remectial and Spedal Education, 8 (1987), 34-

WUlson, V. L. "A Metapanafysls ofthe Relations

Between Sciencre Achievement and Sdence Att

Joumal of Research in Science Teaching. 21 (1

289-303. 

Fage 122 GAO/FEIID-IO.1.2 Evaluation S
BibUography 

Wmer, B. J. Statistical Prindples in Experimen

Design. New York: McGraw HIU PubUshmg C

1971. 

Wolf, F. M. Mete-anafysis; Quantitative Metho

Research Synthesis. Sage Univeralty Paper ser

Quantitative AppUcations in the Social Sdenc

erfy HUls, CaUf.: Sage, 1986. 

Wortman, P. M. "School Desegregation and B

Achievement: An Integrative View." In "Schoo

regation and Black Achievement," T. Cook e

(eds.). Unpublished report. National faistitote 

cation, Washhigton, D.C, 1984. 

Wortinan, P. M. "School Desegregation and B

Achievement." Sodological Methods and Rese

13 (1986), 289-324. 

Wortinan, P. M., and W. H. Yeaton. "Syntheaj

Resutts in (3ontroUed Trials of Coronaiy Arte

Bypass Graft Surgeiy." Evaluation Stocties Re

Annual. 8. Beverfy HUls, Catif.: Sage PubUcat

Inc., 1983. 

Yeaton, W. H., and P. M. Wortinan. "Evaluati

in Mectical Research Synthesis." In Issues in D

Synthesis, pp. 43-66, W. H. Yeaton and P. M. 

Woitman (eds.). New Directions for Program 

tion Series. San FYancnsco, Calif.: Jossey-Bas

Yeaton, W. H., and P. M. Wortman (eds.). "Is

Date Synthesis." New Directions for Program 

Evaluation, no. 24. San FYandsco; Jossey-Ba

I9g3b: 

Yeaton, W. H., and P. M. Wortman. "Medical 

nology Assessment: The Evaluation of Corona

Arteiy ^pass Graft Surgery Using Date [fynt

Techniques." Intemational Joumal of Tedmo

Assessment m Health Care, 1 (1985). 126-46

Fage 128 OAO/FEIIIKIO.1.2 Evaluation 
BibUography 

Yeaton, W. H., and P. M. Wortinan. 

"Reconceptualizing ReliabUity in Mete-anafytic 

Reviews." Manuscript, State Univeralty of New Y

Stonybrook, NewYork, 1989. 

Yhi, R. K., and K. A. Heald. "Using tbe Case Sur

Method to Analyze PoUcy Stodies." AdndiUstrat

Science Quarterfy, 20 (1975), 371-81. 

Zabrenko, R. N., and G. S. Chambera. "An Eval

of Glutamic Add hi Mental Defidency." Americ

Joumal of Psychiatry, 108 (1953), 881-87. 

ZhnUes, H. M. "GeneraUzing From Shigle Case 

Stodies." Evaluation Quarterfy. 3 (1979), 661-7

Zhnmerman, F. T., and B. B. Burgmeister. "The 

of Glutamic Acid on Borderline and Hi^-Grade 

Defective faitdligence." New York State Joumal 

Medidne. 50 (1950), 693=97: 

Page 124 GAO/FEBID-IO.1.2 Evaluation l^
Glossary 

Bias The extent to uridch a measurement, sampUng, 

lytic method systematicaUy underesthnates or 

timates the trae value of an attribute. 

Case Stucfy A method of learning about a complex histanc

on a comprehensive understanding of that inst

obtained by extensive decxiption and analysis 

instance, taken as a uriiole and in ite context. 

Construct Validity The extent to which a measurement method ac

ratefy represente a constract and produces an 

vation distinct firom that produced by a measu

another constract. 

Extemal Validity The extent to which a fincting appUes (or can b

generalized) to persons, objecte, settings, or ti

other than tiiose that were the subject of stody

Generalizability Used Interchangeabfy with "extemal vaUctity." 

Intemal Validity The extent to which the causes of an effect are 

Ushed by an inquiry. 

NuU Hypothesis In hypothesis testing, we should state the assu

hypothesized value of tbe population figure be

beghi sampling. Tbe assumption that we want 

is caUed the nuU hypothesis. The term bad ite 

eariier agricultural and mectical appUcations o

statisticas. 

Page 126 GAO/FBllD-10.1.2 EvaluationS
Gioaaaiy 

Outlieis Instances that are abenant or do not fit with o

mstances; instances that, compared to other m

of a population, are at the extremes on relevan

dimensions. 

ProgFam Evaluation The appUcation of scientific research method

assess program concepte, implementation, an

effecrtiveness. 

Qualitative Data Infonnation based on judgmente (such as the 

mated speed of a UFO) that may be expressed 

numerical or nonnumerical ways and date tha

not be based on judgmente (such as state of b

are not meaningfiilfy expressed numericaUy. T

sources are often textual and observational an

expressed in words. 

Quantitative Data Information based on measures that do not re

judgmente and that are meaningfiiUy measure

are usualfy expressed numericaUy and oflen u

tinuous rather than discrete or categorical le

measurement and scales with intenral or ratio 

properties. 

Reliability The extent to which a measurement process p

shnilar resulte on repeated observations ofth

conctition or event. 

Representative 

Sample 

A sample tbat has approximatefy the same di

tion of characteristics as the population firom 

wasdravm. 

page 126 GAO/FEliD-lO.1.2 Evahutlcni 
Gloaaary 

Simple Random A method for drawing a sample from a popula

Sample <^"^^ ^^^^ ^ samples of a given size have equ

bUity of being drawn. 

page 127 GAO/PEMD-IO.1.2 Evaluation S
Contributore 

Linda G. Morra 

RicbaidJ.Light 

Richard T. Bames 

Christuie A. Fossett 

Penny Pickett 

Page 128 GAO/FEBID-IO.1.2 Evaluation l
Papers in This Series 

This is a flexible series continuaUy being add

updated. The interested reader should inquur

the possibUity of additional papera in the ser

The Evaluation Synthesis. Transfer paper 10

merfy methods paper I. 

Content Analysis: A Method^ogy for Stractu

Analyzmg Written Material. Transfer paper 1

formerfy methodology transfer ps^er 3. 

Designing Evaluations. Tranter paper 10.1.

merfy methodology transfer pi^ier 4. 

Ushig Stiructured Interviewing Techniques. T

paper 10.1.5, formeriy methodology transfer 

Usmg Statistical SampUng. Transfer paper 1

merfy methodology traiurfer paper 6. 

Developing and Ushig QuestionnalTM. Tran

10.1.7, formeriy metiiodology transfer paper 

Case Stocfy Evaluations. Transfer paper 10.1

merfy methodology transfer paper 9. 

Prospective Evaluation Methods; The Prosp

Evaluation Synthe^. Transfer paper 10.1.1

merfy methodology transfer paper 10. 

Page 120 GAO/FEMD-IO.1.2 Evalnatio
;t?:'#|-iBLre-;dii^couhted '''y'^y^WJWy''^^"-Mi0

U.S. General Accountmg Office • ! 

P.O. Box 6015 

Gaithersburg, MD 20877 

Orders maj^also be placed l^y calling (202) 275
/ 

|«MiMiiiis^idi»igBii»jii
M ii '"''""" ''"•••'•"'• 

::[:iy'iiy.yyy: 

''iMlii^'i^}M'i^^!^ir^^i^'\y^