Division

March 1991 Designing

Evaluations

GAO/PEMD-10.1.4

GAO assists congressional decisionmakers 

deliberative process by furnishing analytic

information on issues and options under

consideration. Many diverse methodologie

needed to develop sound and timely answ

questions that are posed by the Congress. 

GAO evaluators with basic information ab

more commonly used methodologies, GAO

guidance includes documents such as met

transfer papers and technical guidelines.

This methodology transfer paper addresse

of program evaluation designs. It provides 

systematic approach to designing evaluati

takes into account the questions guiding a 

constraints evaluators face in conducting i

information needs of its intended user. Tak

time to design evaluations carefully is a cr

toward ensuring overall job quality. Indeed

important outcome of a careful, sound des

be an evaluation whose quality is high in q

ways.

Evaluation designs are characterized by th

which the evaluators have

• defined and posed the evaluation question

• developed a methodological approach for 

those questions,

• formulated a data collection plan that anti

problems, and

• detailed an analysis plan for answering the 

questions with appropriate data.

Designing Evaluations is a guide to the suc

completion of these design tasks. It also pr

discussion of three kinds of evaluation

questions—descriptive, normative, and ca

various methodological approaches appro

Page 1 GAO/PEMD-10.1.4 Desigin
each one. For illustration, the paper conta

narration of a design undertaken by the Pr

Evaluation and Methodology Division (PE

response to a congressional request. The o

paper was authored by Ray Rist and Carl W

July 1984. This reissued (1991) version, pr

Carl Wisler, supersedes the earlier edition

Designing Evaluations is one of a series of 

issued by the Program Evaluation and Met

Division. The purpose of the series is to pr

evaluators with guides to various aspects o

evaluation methodology, to illustrate appli

and to indicate where more detailed inform

available.

We look forward to receiving comments fr

readers of this paper. They should be addr

Eleanor Chelimsky at 202-275-1854.

Werner Grosshans

Assistant Comptroller General

Office of Policy

Eleanor Chelimsky

Assistant Comptroller General

for Program Evaluation and Methodology

Page 2 GAO/PEMD-10.1.4 Desigin
Page 3 GAO/PEMD-10.1.4 Desigin
Preface 

Chapter 1 

Why Spend Time

on Design?

Chapter 2 

The Design

Process

Asking the Right Question 

Considering the Evaluation’s Constraints 

Assessing the Design 

Chapter 3 

Types of Design

The Sample Survey 

The Case Study 

The Field Experiment 

The Use of Available Data 

Linking a Design to the Evaluation Questio

Chapter 4 

Developing a

Design: an

Example

The Context 

The Request 

Design Phase 1: Finding an Approach 

Design Phase 2: Assessing Alternatives 

Design Phase 3: Settling on a Strategy 

Appendixes Bibliography 

Glossary 

Papers in This Series 

Tables Table 3.1: Evaluation Strategies and Types 

Design

Table 3.2: Characteristics of Four Evaluati

Strategies

Table 3.3: Some Basic Contrasts Between

Three Field Experiment Designs

Page 4 GAO/PEMD-10.1.4 Desigin
Figure Figure 3.1: Linking a Design to the

Evaluation Questions

Abbreviations

AFDC Aid to Families with Dependent C

GAO U.S. General Accounting Office

JSARP Job Search Assistance Research 

PEMD Program Evaluation and Method

Division

WIN Work Incentive

Page 5 GAO/PEMD-10.1.4 Desigin
According to a Chinese adage, even a thou

journey must begin with the first step. The 

of reaching one’s destination is much enha

first step and the subsequent steps take th

the correct direction. Wandering about her

without a clear sense of purpose or directi

consumes time, energy, and resources. It a

diminishes the possibility that one will eve

One can be much more prepared for a jour

collecting the necessary maps, studying al

routes, and making informed estimates of 

costs, and hazards one is likely to confron

It is no less true that front-end planning is 

to designing and implementing an evaluati

successfully. Systematic attention to evalu

design is a safeguard against using time an

ineffectively. It is also a safeguard against 

an evaluation of poor quality and limited u

The goal of the evaluation design process 

course, to produce a design for a particula

evaluation. But what exactly is an evaluati

Because there may be different views abou

answer to this question, it is well to state w

understood in this paper. Evaluation perta

systematic examination of events or condi

have (or are presumed to have) occurred a

time or that are unfolding as the evaluation 

place. But to be examined, these events or 

must exist, must be describable, must hav

or be occurring. Evaluation is, thus, retros

that the emphasis is on what has been or i

observed, not on what is likely to happen (

forecasting).1 The designs and the design p

outlined in this paper are focused on the o

performance of completed or ongoing pro

1

Despite the retrospective character of evaluation, pr

evaluation findings can often be used as a sound bas

calculating future costs or projecting the likely effect

Page 6 GAO/PEMD-10.1.4 Desigin
To further characterize evaluation design, 

to look closely at the questions we pose an

answers we seek. Evaluation questions can 

into three kinds: descriptive questions, nor

questions, and impact (cause-and-effect) q

The answers to descriptive questions prov

name implies, descriptive information abo

conditions or events—the number of peop

received Medicaid benefits in 1990, the con

cost of a nuclear power plant, and so on. T

to normative questions (which, unlike des

questions, focus on what should be rather 

is) compare an observed outcome to an ex

level of performance. An example is the co

between airline safety violations and the s

has been set for safety. The answers to im

(cause-and-effect) questions help reveal w

observed conditions or events can be attri

program operations. For example, if we ob

changes in the weight of newborns, what p

changes is the effect of a federal nutrition 

In sum, the design ideas presented here ar

producing answers to descriptive, normati

impact (cause-and-effect) questions.

Given these questions, what elements of a 

should be specified before information is c

The most important elements can be listed 

• kind of information to be acquired,

• sources of information (for example, types 

respondents),

• methods to be used for sampling sources (

example, random sampling),

• methods of collecting information (for exa

structured interviews and self-administere

questionnaires),

• timing and frequency of information collec

Page 7 GAO/PEMD-10.1.4 Desigin
• basis for comparing outcomes with and wi

program (for impact or cause-and-effect qu

and

• analysis plan.

They form the basis on which a design is c

As will be seen, the choices that are made 

element are major determinants of the qua

information that can be acquired, the stren

conclusion that can be drawn, and the eva

cost, timeliness, and usefulness.

Before each component in this design proc

identified and discussed, it would be well t

systematically why it is important to take t

be concerned with evaluation design. First

probably most importantly, careful, sound 

enhances quality. But it is also likely to co

and ensure the timeliness of the findings, e

when the evaluation questions are difficult 

complex. Further, good design increases t

and specificity of findings and recommend

decreases vulnerability to methodological 

and improves customer satisfaction.

In thinking about these reasons for taking 

design an evaluation carefully, one may we

guaranteeing evaluation quality is the pree

concern, the critical dimension of the desi

Stated differently, the most important outc

careful, sound design should be that the ov

quality of the evaluation is enhanced in a n

specific ways.

An evaluation design can usually be recogn

way it has

1. defined and posed the evaluation questi

study,

Page 8 GAO/PEMD-10.1.4 Desigin
2. developed the methodological strategies 

answering these questions,

3. formulated a data collection plan that an

and addresses the problems and obstacles 

likely to be encountered, and

4. detailed an analysis plan that will ensure 

questions that are posed are answered wit

appropriate data in the best possible fashi

A well-designed evaluation will be more po

germane than one in which attention has n

paid to laying out the methodological strat

planning the data collection and analysis c

will also develop a stronger foundation an

convincing in its conclusions and recomm

Implementation also will be strengthened, 

once the design has been established, less 

lost in having to make ad hoc decisions ab

do next. Good front-end planning can subs

reduce the many uncertainties of an evalu

helps provide a clear sense of direction an

to the effort.

Similarly, good front-end planning contain

costs by preventing (1) “down time” from 

sporadic and episodic decisions on what t

(2) waste of staff time on the collection an

of data that are irrelevant to the question,

(3) duplication of data collection, and (4) u

data analysis in a search for relevant findin

be recognized that careful attention to des

take time and does necessitate front-end c

However, the investment can save time an

later in the evaluation, and this is especiall

big, complex projects. There is, of course, 

assurance that careful work will require le

expenditure of resources than ill-defined s

Page 9 GAO/PEMD-10.1.4 Desigin
Attention to the design process also make

quality by focusing on the usefulness of th

the intended recipient. If attention is paid 

of the user in terms of information or

recommendations, the design process can

systematically address these needs and ma

that they are integrated into the project. In 

the relevance of an evaluation can be stren

tying it specifically to the concerns of its u

addition, a concern with relevance is likely 

increase the user’s satisfaction with the pr

A sound design can help ensure timeliness

and logical design can reduce the time tha

accumulates on a project because of exces

unnecessary data collection, the lack of a 

analysis plan, or the constant “cooking” of 

when the omission of a sound methodolog

strategy has made it impossible to answer 

evaluation questions directly. The timeline

findings with respect to the needs of the cu

make or break a technically adequate appr

not enough that a study be conducted with 

degree of technical precision to argue for i

the study must also be conducted in time t

findings to be of service to the user.

In summary, to spend the time to develop 

design is to invest time in building high qu

the effort. Devoting attention to evaluation 

means that factors that will affect the qual

results can be addressed. Not allowing the 

necessary for this vital stage of the project 

end, self-defeating. It can be a crippling, if 

blow to any evaluation that skips quickly t

step. The pressure of wanting to get into th

soon as possible has to be held in check w

systematic planning takes place. The desig

guides the data collection and analysis.

Page 10 GAO/PEMD-10.1.4 Desigin
Having looked at why it is important to de

evaluations well, we can turn our attention 

various components and processes that ar

in evaluation design. Our discussion is in f

parts: asking the right question, adequately

considering the constraints, assessing the 

settling on a strategy that considers streng

weaknesses, and rigorously monitoring the 

incorporating it into the management strat

persons who are responsible for the evalu

Page 11 GAO/PEMD-10.1.4 Desigin
Asking the Right

Question

The first and surely the most fundamental 

every design effort is to ensure that the qu

are posed for the evaluation are the correc

Posing a question incorrectly is an excelle

lead a study in the wrong direction. It is ob

one must ask the right question, but decidi

exactly the “right question” is not necessar

fact, reaching agreement with the sponsor

program operators, and others on the cont

implications of a question can be difficult 

challenging. Among the several reasons fo

strenuousness of the task is that the formu

problem has preeminent importance in the 

phases of the evaluation. How a problem i

implications for the kinds of data to be col

sources of data, the analyses that will be n

trying to answer the question, and the con

that will be drawn.

Consider a brief example: juvenile delinqu

the question of what motivates young peop

commit delinquent acts. The question abou

motivation could be posed in a variety of w

could ask about the personality traits of yo

persons and whether particular traits are a

with differences in who does or does not c

crimes. Asking the question this way entai

sources, and program initiatives that are d

from those that are required in examining, 

example, the social conditions of young pe

here, the focus might be on family life, sch

groups, employment opportunities, or the 

stretch the example further, each of these 

of posing the question about what motivat

to commit crime would lead to evaluations 

different from either asking whether juven

crimes because of a temporary hormonal i

1

Often studies have more than one key question or a 

questions. Every question has to be given the same se

attention.

Page 12 GAO/PEMD-10.1.4 Desigin
or asking whether a youth culture uses cri

“rite of passage” into adulthood.

Posing a question in four quite different w

clearly how the way in which a problem is 

implications for an evaluation design. How 

defined influences directly how variables o

dimensions are to be selected and examin

the analysis will test the strength of the rel

between a cause and its expected consequ

Question formulation is important also in t

concerns of the customer must be attende

question is framed has to take the informa

and spheres of influence of the intended a

into consideration. Does the customer nee

the general effectiveness of a nationwide p

is the concern limited, for example, to indi

problem sites and public attitudes to the p

those sites? The difference of type in these 

questions is extremely important for evalu

design, and attention to the difference allo

evaluator to help make the job useful to its 

Clarifying the Issue Working toward the formulation of the rig

has two phases (Cronbach, 1982, pp. 210-4

first phase, the largest number and widest 

potential questions (and methods by which 

these questions) ought to be considered, e

do not seem especially plausible or defens

example, congressional staff often begin w

broad concern, so that it is necessary to tr

number of less sweeping questions in orde

determine the priorities of the staff and to 

researchable questions. Thus, it is often us

evaluator and requester to work through in 

which questions can be answered easily, w

more difficult, expensive, and time-consum

Page 13 GAO/PEMD-10.1.4 Desigin
which cannot be answered at all and why. 

evaluator is in a much stronger position to 

final phrasing of a question if it is apparen

number of alternatives have been systema

considered and rejected.

During this phase, the evaluator has severa

aids for developing a range of questions. O

imagine the various stages of the program—

objectives, start-up procedures, implemen

processes, anticipated outcomes—and to a

questions that could be asked about each s

example, in considering program objective

evaluator could ask questions about the cl

precision of those objectives, the criteria t

been developed for testing whether the ob

have been met, the relationship between th

objectives and program goals, and whethe

objectives have been clearly transmitted to 

understood by the persons who are respon

the program’s implementation.

Another aid is to focus on the nature of the 

objectives—on whether they are short term 

term, intense or weak, continuous or spor

behavioral or attitudinal, and so on. Yet an

to think of questions that would describe t

as it exists or that would judge the program 

existing norm or that would point out the 

that are a direct result of the program.

Each of these three kinds of question—de

normative, and impact

(cause-and-effect)—necessitates a differen

consideration. What is important for the ev

to separate a potential question into one o

types and then to consider the implication

type of question for the development of a d

choose a set of evaluation questions is to c

Page 14 GAO/PEMD-10.1.4 Desigin
certain cluster of design options for answe

Design options are discussed in chapter 3.

The second phase of formulating the right 

to match possible questions against the re

will be available for the project. We discus

following section.

Deciding Which

Questions Are

Feasible to Answer

It is one thing to agree on which questions 

important and have highest priority. It is q

to know whether the questions are answer

so, at what costs in money, staff, and time

second phase of formulating the right ques

evaluator ought not to assume that a desig

developed to answer questions of highest p

be implemented within the given constrain

For example, the evaluator might determin

would be very informative to collect data o

years, but the requirements of money, staf

might necessitate a less comprehensive or 

complex design that could answer fewer q

less conclusively, within given constraints

alternative design that might be appropria

focus on what a particular group of people

remembers about a program or service du

years in which they were involved with it. 

place of the long-term, objective monitorin

during years to come, the evaluator would 

a look backward that is dependent on the 

and attitudes of the people involved with t

in the past.

Another less comprehensive alternative, o

quality, would be to inquire of the group at 

future points in time rather than to make n

inquiries over several points in time. In oth

the design option can influence the techni

Page 15 GAO/PEMD-10.1.4 Desigin
of the evidence and, hence, the expectatio

what the evaluation can accomplish.

Meeting an

Information Need

Reasonably

A large-scale and expensive evaluation is n

seem reasonable for a program that is sma

and short in duration. Similarly, a study th

allow national generalizability will probab

effort and resources quite different from th

narrower study. To make national generali

from a single case study, for example, is di

not impossible. That is, whether or not an 

need can be reasonably met has to do with 

conclusive the answer to the question bein

investigated has to be. Questions that call 

degree of conclusiveness in the answers w

necessity, require stronger designs than qu

which brief descriptions or quick assessm

adequate answers. For example, to ask for 

description of the children who receive se

an education program for migrants is quite 

from asking whether those services are aff

attendance in school, academic achieveme

proficiency in English. The first question c

answered descriptively with the collection 

tabulation of demographic data, but the se

impact (cause-and-effect) question that de

knowledge about, first, what is happening 

children who are not in the program; secon

children who are in the program were perf

before they joined it; and third, whether ot

possible causes for how the children are p

that have nothing to do with the program c

justifiably excluded.

The “Strength Versus

Weakness” Issue

Strong evaluations employ methods of ana

are appropriate to the question, support th

with evidence, document the assumptions

Page 16 GAO/PEMD-10.1.4 Desigin
procedures, and modes of analysis, and ru

competing evidence. Strong studies pose q

clearly, address them appropriately, and d

inferences commensurate with the power 

design and the availability, validity, and re

the data. Strength should not be equated w

complexity. Nor should strength be equate

degree of statistical manipulation of data.

Neither infatuation with complexity nor st

incantation makes an evaluation stronger.

The strength of an evaluation is not define

particular method. Longitudinal, experime

quasi-experimental, before-and-after, and c

evaluations can be either strong or weak. A 

study design will always be weaker than a 

survey design in terms of its external valid

simple before-and-after design without con

always present problems of internal validi

sample surveys and control groups can be 

for a variety of reasons. That is, the streng

evaluation has to be judged within the con

question, the time and cost constraints, th

the technical adequacy of the data collecti

analysis, and the presentation of the findin

strong study is technically adequate and u

short, it is high in quality (Chelimsky, 1983

Evaluators have considered the concept o

some length. Some argue that strong evalu

employ methods that allow the evaluator t

causal, as opposed to correlational, statem

a policy or program. It is argued that sayin

program intervention X caused outcome Y 

program’s participants is a stronger statem

saying that X and Y are associated but it is 

that X caused Y. In this argument, the noti

strength is related to the judgment that cau

Page 17 GAO/PEMD-10.1.4 Desigin
statements are more powerful than correla

statements. Another argument is that the s

study or a method can be determined by c

what was done with what was possible.

Pilot Versus Full

Study

Formulating the right question is a necessa

a sufficient condition for success. There is 

matter of translating the design and analyt

assumptions into practice—into pragmatic 

and patterns of implementation that will a

evaluator to find the stipulated data and an

In short, the evaluator must ask whether th

matches the area of inquiry. Answering thi

is a “reality check” on whether the assump

the kinds and availability of data hold true

whether the legislation and regulations be

resemblance to what has been implemente

whether the proposed analysis strategies w

the question conclusively.

At this stage of an evaluation, the entire en

still quite vulnerable and tentative. What if 

are not available? What if the program is n

its description in its documents or the gran

application? What if the methodology will 

for sufficiently conclusive answers to the e

questions? Any one of these situations cou

entire study into question.

That the condition of an evaluation can be 

in these ways argues for a limited explorat

question before a full-scale, perhaps expen

evaluation is undertaken. This limited exp

referred to as a “pilot phase,” when the ini

assumptions about the program, data, and 

methodology can be tested in the field. Te

work at one or more sites allows the evalu

Page 18 GAO/PEMD-10.1.4 Desigin
confirm that data are available, what their 

be, and by what means they can be gathere

Site selection for the pilot phase is importa

than choosing a site where the pilot could 

conducted, it is critical to choose a site tha

represents an average, if not the worst, cas

Choosing a noncontroversial site may hide 

resistance an evaluator is likely to experie

sites.

The pilot phase allows for a check on prog

operations and delivery of services in orde

ascertain whether what is assumed to exis

Finding that it does not may suggest a nee

the question to ask why the program that h

implemented is so different from what wa

This phase allows also for limited data col

which provides an opportunity to assess w

analysis methodology will be appropriate a

alternative interpretations of the data may 

possible.

The study’s pilot phase is very useful. It is 

important opportunity to correct aspects o

that can determine the success or the failu

overall effort. To undertake a large-scale, f

study without this phase is a high-risk prop

allocate staff and financial resources and e

time and cooperation of the persons in the 

to be studied without making as certain as 

that what is proposed will work is to court 

problems. It may well be that conducting a 

confirm what was originally designed, but 

ahead with this confirmation is preferable 

assuming that everything will fall successf

place.

Page 19 GAO/PEMD-10.1.4 Desigin
To be sure, there are instances when a pilo

possible: time pressures may not allow it, 

may be so scarce that there is but one opp

field work, or the availability of staff may b

constrained. Yet the evaluator ought to rec

not performing a pilot test increases the lik

problems and difficulties, even to the degr

study cannot be completed successfully. T

evaluator must give high priority to the pil

when considering time, resources, and sta

A frequently posed question is how much p

is necessary before the large-scale evaluat

undertaken. There is no “cookbook” answ

is an evaluation tool that increases the odd

effort will be high in quality. By itself, the p

provide a fail-safe guarantee. It can sugges

alternative data collection and analysis str

can also stimulate further thinking about a

clarification of the evaluation. The pilot is 

for reducing uncertainty. That uncertainty 

reduced to zero does not detract from the 

utility.

Perhaps the best answer to how extensive 

ought to be is a second question: How muc

uncertainty is the evaluator willing to toler

evaluation begins? Only the evaluator can 

trade-off between the scope and resources 

and problems on the project.

Considering the

Evaluation’s

Constraints

Time is a constraint. It shapes the scope o

evaluation question and the range of activi

can be undertaken to answer it. It demand

and establishes boundaries to what can be

accomplished. It continually forces the eva

think in terms of what can be done versus 

be desirable. Because time is finite (and th

Page 20 GAO/PEMD-10.1.4 Desigin
enough of it), the evaluator has to plan the 

“real time” with its inevitable constraints o

question can be posed, what data can be c

and what analysis can be undertaken.

A rule of thumb is that the time for a study 

scope of the question being addressed oug

directly related. Tightly structured and nar

investigations are more appropriate when 

short. Any increase in the scope of a study 

accompanied by a commensurate increase 

amount of time that is available for it. The 

recognize and plan for this link between ti

scope is the Achilles heel of evaluation.

Linking scope and time in the study design 

important because the scope is determined 

difficulty of the evaluation, the importance 

subject, and the needs of the user, and the

determinants of time. Though it may be se

say so, difficult evaluations, important eva

and evaluations in which there is a great d

interest have different demands with respe

than other evaluations. No project is “too l

“too short” within this context.

The need of the study’s audience as a time 

merits additional comment. Evaluations ar

and conducted because someone perceive

information. Producing that information w

sensitivity to the user’s timetable diminish

usefulness. For example, a report to the C

may answer the questions correctly but wi

little or no use if it is delivered after the le

hearings for which it is needed or after the

preparation of a new budget for the progra

Cost is a constraint. The financial resource

for conducting a study partly determine th

Page 21 GAO/PEMD-10.1.4 Desigin
the study. Having very few resources mean

evaluator will have to consider tight limita

questions, the modes of data collection, th

of sites and respondents, and the extent an

of the analysis. As the resources expand, t

constraints on the study become less conf

Having more funds might mean, for examp

longer time in the field or the opportunity 

multiple interviews with respondents or to 

sites or choose larger samples for sites. Ea

items has a price tag. What the evaluator is 

purchase depends on what funds are avail

It should be stressed that regardless of wh

available, design alternatives should be co

Cost is simply an important constraint wit

the design work has to proceed. If only a s

sum is available, the evaluator has to deter

can be done with that sum in order to prov

information that is relevant to the question

same resources might allow three or four q

distinct approaches to an evaluation. The c

to consider the strengths and weaknesses 

various approaches. Like the constraint of 

does not determine the design. It helps est

range of options that can be realistically ex

Even when resources can be expanded, co

constraint. However, the design problem t

becomes one of cost-effectiveness, or gett

for the dollar, rather than one of what can 

within a stipulated sum.

One other point: the quality of an evaluatio

depend on its cost. A $500,000 evaluation i

necessarily five times more worthy than a 

evaluation. An expensive study poorly des

executed is, in the end, worth less than on

less but addresses a significant question, is 

reasoned, and is carefully executed. A stud

Page 22 GAO/PEMD-10.1.4 Desigin
be costly only when the questions and the 

answering them necessitate a large expend

with the constraint of time, there is a direc

correlation between the scope of a study a

money available for conducting it.

Staff expertise is a constraint. The design f

evaluation ought not to be more intricate o

than what the staff can successfully execu

Developing highly sophisticated computer

simulations or econometric models as par

evaluation when the skills for using them a

available to the evaluation team is simply a 

mismatch of resources. The skills of the st

be taken into account when the design is d

It is perhaps too negative to consider staff 

as only a constraint. In the alternative view

design accounts for the range of available 

expertise and plans a study that uses that 

the maximum. It is just as much a mismatc

design that is pedantic, low in power, and 

unsophisticated when the staff are capable 

more and the questions demand more as it 

a design that is too complex for the expert

available. In either instance, of course, a d

determined not by expertise but by the nat

questions.

A realistic understanding of the skills of th

play an important role in the kinds of desig

that can be considered. An option that req

that the staff do not have will fail, no matt

appropriate the option may be to the evalu

questions. A staff with a high degree of tec

training in a variety of evaluation strategie

tremendous asset and greatly expands the 

Page 23 GAO/PEMD-10.1.4 Desigin
Some designs demand a level of expertise 

available. When this happens, consultants 

brought into the study or the staff can be g

intensive courses or complex and difficult 

the design can be isolated and performed u

contract by evaluators specializing in the a

type of study. In other words, the stress is 

considering the options available. Preferen

be given to building the capability of curre

When this cannot be done, or time and cos

allow it, expertise can be procured from o

order to fulfill the demands of the design.

Location and facilities are secondary cons

comparison to the others we have discuss

do impinge on the design process and influ

options. Location has to be considered fro

aspects. One is the location of the evaluato

where the evaluation is to be conducted. L

less critical for a national study, since mos

be reached by air within a few hours, but i

in importance if the study examines only a 

individual projects. The accessibility and c

data collection may be jeopardized if the e

on the east coast and the sites are in the S

Midwest, and on the west coast. A situatio

this may have to incorporate local persons 

members of the evaluation team and may i

utility of a mail questionnaire or telephone 

compared to face-to-face interviews.

Another aspect of location has to do with t

and cultural mores of the area where the e

to be conducted. For example, to gain vali

insightful data on attitudes toward rural m

clinics, it may be wise not to send intervie

urban areas. Good interviewing necessitat

between the persons involved, and it may 

Page 24 GAO/PEMD-10.1.4 Desigin
generate between an interviewer and a res

whose backgrounds are very different.

A third aspect of location is the stability of 

population being studied. A neighborhood 

residence is transient may necessitate a di

strategy from a neighborhood where most 

have lived in the same house for 40 years a

intention of moving.

Finally, the evaluator must consider wheth

a site is justified at all. For example, if it co

to travel to a remote town to ascertain wh

school there is using a $1,500-computer pr

U.S. Department of Education grant, the c

going is defensible.

The constraint of facilities on the design o

has more than one aspect. One has to do w

collection and data processing. For examp

study involves entering large aggregates of 

computer, the equipment to do so must be 

or the money must be available for contrac

work. Similarly, if the design calls for data 

computer terminals with phone connectio

main computer, the equipment is a must. T

of such facilities limits both the kind and t

the data one can collect.

Another aspect is the need for periodic acc

facilities that are not under the auspices o

or program being studied. For example, to 

welfare clients in a welfare office about th

and service they are receiving there may b

highly biased answers. How candid can a c

knowing that the caseworker who has mad

on food, clothing, and rental allowances fo

client’s family is in the next room? “Neutra

cannot guarantee candid answers, but it m

Page 25 GAO/PEMD-10.1.4 Desigin
anxiety and it can contribute to the authen

evaluator’s promise of anonymity and con

The example applies equally to interviews 

persons who hold positions of power and 

Assessing the

Design

Once a design has been selected, the impe

move full steam ahead into the execution o

However, the evaluator must fight this imp

take time to look back on what has been

accomplished, on the design that has final

selected, and on what the implications are 

subsequent phases of the study.

The end of the design phase is an importan

milestone. It is here that the evaluator mus

clear understanding of what has been chos

has been omitted, what strengths and wea

have been embedded in the design, what th

the customer are, how usefully the design 

meet those needs, and whether the constra

time, cost, staff, location, and facilities hav

fully and adequately addressed.

GAO’s Program Evaluation and Methodolo

has developed and uses a job review syste

includes a detailed and systematic assessm

design phase. This system helps establish 

moving forward into implementation. It m

to other evaluators in judging their own de

key questions figure prominently in the rev

system.

1. How appropriate is the design for answe

questions posed for the study? The evaluat

be able to match the design components

systematically to the study questions in or

demonstrate that all key questions are bein

addressed and that methods are available 

Page 26 GAO/PEMD-10.1.4 Desigin
so. Even though this entails a judgment, th

should assess the match between the stren

design and the information necessary to an

study questions. If the design is either too 

strong for the questions, serious considera

be given to whether the design ought to be

implemented or whether the questions oug

modified. This judgment about the approp

the design is critical, because if the study b

an inappropriate design, it is difficult to co

later for the basic incongruity.

2. How adequate is the design for answerin

questions posed for the study? The empha

on the completeness of the design, the exp

precision of the answers, the tightness of t

the thought given to the limitations of the 

the implications for the analysis of the dat

evaluator should have reviewed the literat

should give evidence of knowing what was

undertaken previously in the area from bo

substantive and methodological viewpoint

the evaluator should be aware of not only 

of questions have been asked and answere

past but also what designs, measures, and 

analysis strategies have been used. A caref

the literature prevents “rediscovering” or d

existing work. Thus, in judging the adequa

design, the evaluator must link it to previo

evaluations.

Second, the design should explicitly state 

evaluation questions that determined the s

the design. Knowing the evaluation questio

were thought germane and those that were 

the reader a basis for assessing the strengt

design. Since every evaluation design is co

by a number of factors, recognizing them a

candidly describing their effect provides im

Page 27 GAO/PEMD-10.1.4 Desigin
clues to whether the design can adequately 

the study questions.

Third, there is a need to be explicit about t

limitations of the study. How conclusive is 

likely to be, given the design? How detaile

data collection and data analysis plans? W

trade-offs were made in developing these p

answers to these questions provide data on 

design’s adequacy.

3. How feasible is the execution of the des

the required time and proposed resources? 

and appropriate designs may not be feasib

ignore time and cost—that is, if they are n

The completeness and elegance of a design 

quickly relegated to secondary importance 

design presents major obstacles in the exe

Further, asking about feasibility puts an im

check on studies that simply cannot be do

example, discovering that a particular eval

a true experimental design cannot be exec

prevent proceeding with a project that wil

4. How appropriate is the design with rega

user’s needs for information, conclusivene

timeliness? What kind of information is ne

conclusive does it have to be? When does 

delivered? Being able to determine how w

design responds to the user’s needs requir

evaluator and the user to be in close agree

continuous consultation. In the absence of

cooperation, the evaluator is left to presum

be of relevance—and presumption is a poo

for knowledge. Since evaluations are unde

because of a need for information, the deg

which they provide useful information is a

inescapable and critical design considerati

Page 28 GAO/PEMD-10.1.4 Desigin
5. How adequate were the negotiations wit

regarding the relationship between the inf

need and the study design? It is one thing t

what the user needs and when it is needed

another to agree on how the questions oug

framed so that the information can be gath

user has causal questions in mind while th

believes that only a descriptive study is fea

if the gap between these two perspectives 

resolved, the user’s satisfaction with the fi

likely to be quite low and the ensuing repo

be used.

Further, the consideration of time is releva

size, complexity, and completeness of the 

that is finally undertaken. If the user is inte

involved in determining the project’s timet

products, the evaluator will know how to d

whether what is proposed can be accompl

ignore, or only guess at, rather than negoti

agree on a timetable would be to risk the r

the whole effort. The negotiations with the 

should be carefully scrutinized at the end o

design phase to make sure that there is co

understanding and agreement on what is b

proposed for the remaining phases of the e

Page 29 GAO/PEMD-10.1.4 Desigin
In chapter 2, we examined the factors to c

arriving at an evaluation design. Here we t

systematic look at four major evaluation s

and several types of design that derive from 

(See table 3.1.) The discussion is brief and

nontechnical. More details can be found in 

references given under the heading “Wher

for More Information” for each design type

Table 3.1: Evaluation

Strategies and Types of

Design

Strategy Design

Sample survey Cross-sectional

Panel

Criteria-referenced

Case study Single

Multiple

Criteria-referenced

Field experiment True experiment

Nonequivalent comparis

Before-and-after (includi

series)

Use of available

data

Secondary data analysis

Evaluation synthesis

Evaluation strategies and designs can be c

a variety of ways, each with some advanta

disadvantages in communicating a logical 

the different forms of evaluation inquiry. W

word “strategy,” as the broader of the two 

to connote a general approach to finding a

evaluative questions. A strategy embraces 

types of design that have certain features i

Our classification scheme is similar to sch

by Runkel and McGrath (1972), Judd and K

(1986), and Black and Champion (1976), b

adapted to the work of the U.S. General Ac

Office. Sample surveys, case studies, field

experiments, and the use of available data 

Page 30 GAO/PEMD-10.1.4 Desigin
strategies because they can be readily link

types of evaluation questions that GAO is a

answer, and they explicitly accommodate 

strategies that are prominent in GAO’s hist

For simplicity, we speak only of program e

but we imply the evaluation of policies als

Some of the design elements we identified 

1—in particular, kinds of information, sam

methods, and the comparison base—help 

the evaluation strategies. Table 3.2 shows 

relationship between these three design el

the four evaluation strategies, the types of 

and the availability of data. In the rest of th

we discuss this relationship in detail. Othe

elements—information sources, informati

collection methods, the timing and frequen

information collection, and information an

plans—are essential in specifying a design 

useful in making distinctions among the m

evaluation strategies.

Page 31 GAO/PEMD-10.1.4 Desigin
Table 3.2: Characteristics of Four Evaluation Strategies

Design element

Evaluation

strategy

Type of

evaluation

question

most

commonly

addressed

Availability

of data

Kind of

information

Sampling

method

Sample survey Descriptive

and

normative

New data

collection

Tends to be

quantitative

Probability

sampling

Case study Descriptive

and

normative

New data

collection

Tends to be

qualitative;

can be

quantitative

Nonprobability

sampling

Field

experiment

Impact

(cause and

effect)

New data

collection

Quantitative

or qualitative

Probability or

nonprobability

sampling

Use of

available data

Descriptive,

A normative,

and impact

(cause and

effect)

vailable

Tdata

ends to be

quantitative;

can be

qualitative

Probability or

nonprobability

sampling

aIn this classification, sample surveys and case stud

have an explicit comparison base by definition. This 

is not universal.

Two points about the use of the classificat

should be stressed. First, as we indicated i

a program evaluation design emerges not o

the evaluation questions but also from con

such as time, cost, and staff. Therefore, th

cannot be used independently as a “cookb

evaluation. Second, and related to the first 

every evaluation design is likely to be a ble

several types. Often, two or more design ty

combined with advantage.

Page 32 GAO/PEMD-10.1.4 Desigin
Each of this chapter’s sections on the four 

strategies is broken down into subsections 

design types that may be applicable in GAO

type of design, we give several kinds of inf

description of the design, appropriate appl

planning and implementation consideratio

sources of more information. The last sect

chapter makes further connections betwee

evaluation questions and the design types.

The Sample

Survey

In a sample survey, data are collected from 

of a population to determine the incidence

distribution, and interrelation of naturally 

events and conditions.1 The overriding con

sample survey strategy is to collect inform

such a way that conclusions can be drawn 

elements of the population that are not in 

as well as about elements that are in the sa

characteristic of the strategy is the use of p

sampling, which permits a generalization f

findings about the sample to the populatio

interest. In probability sampling, each unit 

population has a known, nonzero probabil

selected for the sample by chance. The con

from this kind of sample can be projected 

population, within statistical limits of erro

Because of the aim to aggregate and gener

the survey results, great importance is atta

collecting uniform data from every unit in 

Consequently, survey information is usual

through structured interviews or self-admi

questionnaires. Most of the information is 

close-ended form, which means that the re

1

The special case in which the sample equals the pop

called a “census.” The word “survey” is sometimes us

a structured method of data collection without the go

conclusions about what has not been observed. We d

term in this narrow sense.

Page 33 GAO/PEMD-10.1.4 Desigin
chooses from among responses offered in 

questionnaire or by the interviewer. Design

consistent set of responses into the data c

process helps establish the uniformity of d

units in the sample.2 The three main ways 

the data are by mail, phone, and face-to-fa

interviews.

The sample’s units are frequently persons 

organizations such as schools, businesses, 

government agencies. A crucial matter in s

is the quality of the “sampling frame” or lis

from which the sample will be drawn. Sinc

is the operational manifestation of the pop

does much to determine the generalizabili

precision of the survey results.

Sample surveys have been traditionally us

describe events or conditions under invest

example, national opinion surveys report t

of various segments of the population abo

candidates or current issues. A survey may 

conditions such as the extent to which per

support one side of an issue also tend to b

candidates who advocate that side of the i

interpretation of such relationships, there 

no attempt to impute causality.

However, some analysts attempt to go bey

purely descriptive or normative interpreta

sample surveys and draw causal inference

relationships between the events or condit

reported. The conclusions are frequently d

but there are circumstances in which caus

inferences from sample survey data are wa

Special data analysis methods are used to 

qualified causal interpretations but even th

2

Open-ended questions may be used in sample survey

results are to be aggregated across the sample, respo

coded—placed into categories—after the data are co

Page 34 GAO/PEMD-10.1.4 Desigin
procedures may not silence methodologic

In the rest of this section, we describe the 

cross-sectional, panel, and critera-referenc

surveys.

The Cross-Sectional

Survey

A cross-sectional design, in which measur

made at a single point in time, is the simpl

sample survey.

EXAMPLE: In 1971, a survey was made of 

families (11,619 persons) to provide descri

information on the use of and expenditure

services. A probability sample was drawn 

total U. S. population not residing in institu

Because of special interest in low-income, 

residents, rural residents, and the elderly, 

groups were sampled in numbers beyond t

proportion in the population so that suffic

precise projections could be made for thes

Data were collected by holding interviews 

and some of this information was verified 

other records such as those maintained by 

and insurance companies. Information pro

the survey, which was projected to the nat

population, included the kind of health ser

people receive, where and why they receiv

how the services are paid for, and how mu

cost.

Applications When the need for information is for a des

a large population, a cross-sectional sampl

may be the best approach. It can be used t

factual information—such as the living con

the elderly or the costs of operating govern

programs. It can also be used to determine 

and opinions—such as the degree of satisf

among the beneficiaries of a government p

Page 35 GAO/PEMD-10.1.4 Desigin
Because the design requires rigorous samp

procedures, the population must be well-d

kind of information that is sought must be 

enough that structured forms of data colle

work. A sample survey design cannot be u

is not possible to settle on a particular sam

before the data are collected. It is hard to u

the information that is sought must be acq

unstructured, probing questions and when 

understanding of events and conditions m

pieced together by asking different questio

different respondents.3

A cross-sectional design can sometimes be 

imputing causal relationships between con

in inferring that educational attainment ha

on income. Other evaluation designs, such 

experiment or nonequivalent comparison 

designs, are ordinarily more appropriate, w

are feasible. However, practical considera

rule out these and other designs, and the

cross-sectional design may be chosen for l

better alternative. When the cross-sectiona

used for causal inferences, the data must b

by structural equation models and related 

although the data collection procedures ar

as for descriptive applications (see, for ex

Hayduk, 1987).

Planning and

Implementation

Sampling. Having a sampling frame that cl

approximates the population of interest an

the sample in accordance with statistical

requirements are crucial to the success of 

cross-sectional sample survey. The size of 

determined by how statistically precise the 

must be when the sample results are used 

3

A procedure that is suitable for this situation, called 

matrix sampling,” applies to each respondent a subse

number of questions.

Page 36 GAO/PEMD-10.1.4 Desigin
population parameters such as the mean a

variance.

Pretesting the Instruments. To ensure the 

of the data, the data collection instrument

unambiguous and likely to elicit complete, 

answers from the respondents. Pretesting 

instruments a number of times before usin

the survey is an essential preparatory step

Nonrespondent Follow-up. The failure of a 

unit to respond to a data collection instrum

failure to respond to certain questions may 

results when the data are aggregated. Furt

attempts must be made to acquire missing

information from the respondents, and the 

analysis must adjust, as well as possible, fo

information that cannot be obtained.

Causal Inference. The procedures for mak

inferences from sample survey data requir

hypotheses about how two or more factor

related to one another. Causal analysis me

the hypotheses to test the consistency of t

That is, the credibility of causal inferences 

sample survey data rests heavily on the pla

the hypotheses. For plausible hypotheses, 

is placed on broad literature reviews and a 

understanding of the events and condition

question.

Where to Look for More

Information

Babbie (1990), Bainbridge (1989), Fowler 

Warwick and Lininger (1975) are general r

on the sample survey strategy. Kish (1987) 

design issues for sample surveys as well as 

designs. Kish (1965) provides a comprehen

treatment of sampling procedures, while K

(1983), Henry (1990), Scheaffer, Mendenha

(1990), Sudman (1976), and U.S. General A

Page 37 GAO/PEMD-10.1.4 Desigin
Office (1986a) give introductory treatment

collection methods are the subject of Brad

Sudman (1979), Converse and Presser (198

and Mangione (1990), Payne (1951), and U

Accounting Office (1985 and 1986b). Grov

(1989) offers a broad look at survey errors 

Routine data analysis methods are covered 

numerous texts on descriptive and inferen

statistics, and the U.S. General Accounting 

plans to issue an elementary introduction 

methods in 1991. Examples of advanced te

may be found in Lee, Forthofer, and Lorim

(1989) and Hayduk (1987).

The Panel Survey A panel survey is similar to a cross-section

but has the added feature that information 

from a given sample unit at two or more p

time.

EXAMPLE: The “panel study of income dy

carried out by the Institute for Survey Res

University of Michigan, is based on annual 

with a nationally representative sample of 

families. The extensive economic and soci

are collected can be used to answer many 

questions about occupation, education, inc

family characteristics. Because follow-up i

are made with the same families, question

be asked about changes in their occupatio

education, income, and activities.

Applications The panel design adds the important elem

to the sample survey strategy. When the su

used to provide descriptive information, th

design makes it possible to measure chang

Page 38 GAO/PEMD-10.1.4 Desigin
attitudes, and opinions.4 For making decis

government programs and policies, dynam

information—that is, information about ch

frequently more useful than static informa

The panel survey’s use of time is also impo

the survey data are used for causal inferen

application, the panel design may help sett

question of whether, of two factors that ap

causally related, one is the cause and the o

effect.

Planning and

Implementation

Sampling, Pretesting the Instruments, Non

Follow-Up, and Causal Inference. Panel su

designs are similar to cross-sectional desig

need for attention to these activities.

Panel Maintenance. To the extent that sam

leave the sample, changes in the sample m

mistaken for changes in the conditions bei

assessed. Therefore, keeping the panel int

important priority. When sample units are

unavoidably lost, it is necessary to attemp

adjustments to minimize distortion in the r

Where to Look for More

Information

The references in the discussion on cross-

survey designs are applicable.

The

Criteria-Referenced

Survey

Sometimes the evaluation question is, How 

outcomes associated with participation in 

compare to the program’s objectives? Ofte

normative question like this is best answer

sample survey design (although a criteria-r

case study design may sometimes be used

4

Change can also be measured by two or more cross-

time-separated surveys if the samples and data collec

procedures are consistent (see Babbie, 1990, for deta

it is possible to associate change on a measure not w

individual but with populations, so that the kinds of q

can be answered are more limited than with the pane

Page 39 GAO/PEMD-10.1.4 Desigin
EXAMPLE: A soil conservation program h

objective of reducing soil loss by 2 tons pe

year in selected counties. A panel survey c

designed in which actual soil loss on the la

subject to the program could be compared 

criterion. That is, two measurements of so

year apart could be recorded for a probabi

of locations in the targeted counties. Subje

limitations of measurement and sampling 

amount of soil loss in the counties could b

and then compared to the program objecti

This criteria-referenced survey design emp

probability sample to acquire information 

program’s outcome because a conclusion 

about a representative sample of the progr

population.

A normative evaluation question may also 

does actual program implementation matc

intended, or how well does it match a stan

operating performance? The attention is n

outcomes but on processes and procedure

EXAMPLE: Federal policies require that co

airlines observe certain safety procedures

criteria-referenced design could produce i

on the extent to which actual procedures c

these criteria. A population of maintenanc

activities—engine overhauls, for example—

sampled to see if required steps were follo

infraction rate, projected to the population

then be compared to the standard rate, wh

be zero.

In this example, the passengers’ safety—b

evaluation is focused not on the result but 

implementation of the program’s policy on 

Page 40 GAO/PEMD-10.1.4 Desigin
Applications Whether dealing with outcomes or process

evaluators can use criteria-referenced desi

answer normative questions, which always 

actual performance to an external standar

performance. However, criteria-referenced 

not generally permit inferences about whe

program has caused the outcomes that hav

observed. Causal inference is not possible

the criteria-referenced model does not pro

estimate of what the outcomes would have 

the absence of the program.

An audit model—the “criterion, condition, 

effect” model—is a special case of the

criteria-referenced design that is widely us

 Outcomes, the condition, are often compa

objective, or a criterion, and the difference 

an indication of the extent to which the ob

been missed, achieved, or exceeded. Howe

not ordinarily possible to link the achievem

objective to the program, because other fa

accounted for may enter into failure or suc

meeting the objective.

A variety of evaluation questions lead to th

the criteria-referenced design. For service 

examples are questions about whether the 

participants are being served, the intended 

are being provided, the program is operati

compliance with legal requirements, and th

providers are properly qualified. Regulator

give rise to similar questions:

whether activities are being regulated in c

with the statutory requirements, inspectio

carried out, and due process is being follow

5

The word “cause” in the audit model has a different 

the usual notion of causation. “Purported cause” wou

accurate term, because the criteria-referenced design 

permit inference about causation.

Page 41 GAO/PEMD-10.1.4 Desigin
Sometimes outcome questions are framed 

criteria. Did the missile hit the right target

participants of the training program get job

sale of timber yield the expected return? D

of strategic minerals meet the quotas?

Whenever the evaluation questions are nor

criteria-referenced designs are called for. F

but not always, a sample survey is embedd

criteria-referenced design so that the conc

be regarded as representative of the popul

Planning and

Implementation

Consensus About the Criteria. It is often d

gain consensus about the objectives of fed

programs. It follows that in those cases it i

to decide which criterion to use in an evalu

best way is usually to use not one criterion 

several criteria, to allow for the objectives 

several interests in the program—legislato

participants, taxpayers, and so on. The pro

consensus is usually of less concern with

implementation criteria, because statutes 

regulations are more likely to be specific a

implementation requirements.

Measuring Performance Against the Criter

may be difficult to reach consensus on the 

of a program, so there is likely to be debat

procedures for measuring performance ag

criteria. For example, Are tests of military 

against simulated enemy targets a satisfac

estimating the probability that the weapon

real enemy targets? Similarly, views may d

the appropriate way to measure performan

implementation criteria.

Where to Look for More

Information

Herbert (1979) outlines the normative desi

by auditors. Provus (1971) covers the “disc

model,” an early treatment of the normativ

Page 42 GAO/PEMD-10.1.4 Desigin
in evaluation, and Steinmetz (1983) is a lat

reference for the same model. Popham (19

educational evaluation focuses on criteria-

evaluation. The performance-monitoring a

Wholey (1979) includes the comparison of 

program performance to that which is exp

The Case Study The case study strategy is less well defined 

other evaluation strategies we have identif

indeed, different practitioners may use the 

mean quite different things. For GAO’s pur

case study is an analytic description of an 

process, an institution, or a program (Hoa

1982).

One of the most commonly given reasons 

choosing a case study design is that the th

described is so complex that the data colle

to probe deeply beyond the boundaries of 

survey, for example. The information to be 

will be similarly complex, especially when 

comprehensive understanding is wanted a

process works or when an explanation is s

large pattern of events.

Case studies are frequently used successfu

address both descriptive and normative qu

when there is no requirement to generalize 

findings. Impact (cause-and-effect) questio

Page 43 GAO/PEMD-10.1.4 Desigin
sometimes considered, but reasoning abou

from case study evidence is much more de

We present three types of case study desig

case, multiple case, and criteria-referenced 

Even in a study with multiple cases, the sa

usually small. However, if the sample size 

large and data collection is at least partiall

structured, the case study strategy may be 

the sample survey strategy, except that the 

requires a probability sample.

The Single Case In single case designs, information is acqu

single individual, entity, or process.

EXAMPLE: The Agency for International

Development fostered the introduction of 

maize into Kenya. An evaluation using a si

design acquired detailed information abou

processes of introducing the maize, cultiva

making it known to the populace, and usin

evaluation report is a minihistory construc

interviews and archival documents.

Single case evaluations are valued especia

utility in answering certain kinds of descri

questions. Ordinarily, much attention is giv

6

The use of case studies to draw inferences about cau

been approached from diverse points of view. The sc

paper permits only two examples. One approach is c

induction” and involves establishing a hypothesis abo

of an effect and then searching among cases for an in

refutes the hypothesis. When one is found, a new hyp

a new cause is established, and the cycle continues u

hypothesis cannot be refuted. The cause, or pattern o

associated with that hypothesis is then taken as a lik

for the effect. Another is in “single case experimenta

originated largely in the area of psychology and relat

experiments. With substantial control over and mani

hypothesized cause in a single case, inferences can b

cause-and-effect relationships.

Page 44 GAO/PEMD-10.1.4 Desigin
acquiring qualitative information that desc

events and conditions from many points o

Interviewing and observing are the commo

collection techniques. The amount of struc

imposed on the data collection may range 

flexibility of ethnography or investigative r

the highly structured interviews of sample 

There is some tendency to use case studie

conjunction with another strategy. For exa

studies providing qualitative data might be 

with a sample survey to provide quantitati

However, case studies are also frequently 

Applications Three applications of single case studies a

illustrative, exploratory, and critical instan

and other applications are described in de

Study Evaluations (U.S. General Accountin

1990), another paper in this series.

An illustrative case study describes an eve

condition. A common application is to des

federal program, which may be unfamiliar 

abstract, in concrete terms and with exam

aim is to provide information to readers w

personal experience of what the program i

it works.

An exploratory case study can serve one o

at least two purposes. One is as a precurso

possibly larger evaluation. The case study 

whether a program can be evaluated on a l

and how the evaluation might be designed 

out. For example, a single case study migh

feasibility of measuring program outcome

evaluation questions, or help in choosing a 

collecting data for the larger study. The ot

of an exploratory case study is to provide 

information, with no further study necessa

intended.

Page 45 GAO/PEMD-10.1.4 Desigin
A single case study may also be used to ex

critical instance closely. Most common is t

investigation of one problem or event, suc

overrun on a nuclear reactor. In this exam

question is normative and the issue is prob

complex, requiring an in-depth study.

Planning and

Implementation

Selecting a Case. The choice of a case clea

a problem, except for the critical instance 

in which the instance itself prompts the st

other applications, the results depend to s

on the case that is chosen. If it is expected 

will differ greatly from case to case, it may 

necessary to use a multiple case design.

Information Collection. Because the goal i

in-depth information about a complex case

collection may be challenging. Although ca

typically require a mix of quantitative and 

data, particular care is required with the la

because there is a tendency to be less rigo

obtaining qualitative information. For exam

data collection is unstructured, the reliabil

data may be doubted. The question is whe

data collection teams examining the same 

end up with quite different findings. Steps 

taken in the planning stages to avoid this f

unreliability. Yin (1989) suggests three prin

help establish construct validity and reliab

case study: (1) use multiple sources of evi

(2) create a case study data base, and (3) m

chain of evidence.

Data Analysis and Reporting. Because ana

reporting qualitative data can be difficult, 

for the single case study must have explici

these tasks. Miles and Huberman (1984) of

suggestions for manipulating and displayin

qualitative information. Tesch (1990) desc

Page 46 GAO/PEMD-10.1.4 Desigin
computer programs that can be used to an

qualitative data.

Where to Look for More

Information

Yin (1989) and U.S. General Accounting Of

(1990) set forth general approaches to the 

strategy. Hoaglin et al. (1982) devote a cha

studies. Patton (1990) gives an overall app

“qualitative” evaluation. In a somewhat bro

science context, Strauss and Corbin (1990

prescriptions for qualitative research and M

and Rossman (1989) cover the design of qu

research. With respect to analysis of qualit

Miles and Huberman (1984), Strauss (1987

Tesch (1990) offer many suggestions.

Multiple Cases Single case designs are weak when the eva

question requires drawing an inference fro

to a larger group. A multiple case study de

produce stronger conclusions. In our class

an important distinction between the mult

study design and sample survey designs is 

latter require probability samples while th

does not.

EXAMPLE: A program known popularly as 

“general revenue sharing act” appropriated 

funds for nearly 38,000 state and local juri

An evaluation intended to answer both des

and impact (cause-and-effect) questions u

multiple case study design. Sixty-five juris

were chosen judgmentally for in-depth dat

collection, including questionnaires, interv

public records, and less formal observatio

selecting the sample, the evaluators consid

of the nation’s most populous states, coun

cities but also considered diversity in the t

jurisdiction. Budget constraints required a

geographically clustered sample.

Page 47 GAO/PEMD-10.1.4 Desigin
In this example, the evaluators weighed th

in-depth information and the need to make

generalizations, and they chose in-depth in

over a probability sample. They tried to mi

limitations of their data by using a relative

diverse sample.

The multiple case study design may be app

evaluating either program operations or pr

results (and it can be useful for explorator

applications as described for single case d

aim is usually to draw conclusions about a 

from a study of cases within the program, 

sometimes the conclusions must be limite

statements about the cases. When the aim 

inferences about a program, the best appli

probably to base a description of the progr

operations on cases from a very homogene

program. The least defensible application 

determine a program’s results from cases t

a heterogeneous program.

Planning and

Implementation

Selecting Cases. Since the evaluation strat

not involve probability sampling, the goal 

shifts from one of getting a statistically de

sample to something else, frequently one t

getting variety among the cases.7 The hope 

ensuring variation in the cases will avoid b

picture that is constructed of the program

Information Collection. Even though the in

evaluation may not be to literally aggregat

information from multiple cases, the frequ

make statements about a program as a wh

compare across cases suggests the need fo

7

Variety is not the only criterion. For other possibiliti

General Accounting Office (1990) and Patton (1990). 

the evaluation question is about cause and effect, see 

Hersen and Barlow (1976) for a discussion of how th

problem is analogous to the problem of replicating ex

Page 48 GAO/PEMD-10.1.4 Desigin
uniformity in the information collection. T

conflict with the in-depth, unstructured m

inquiry that produces the rich, detailed inf

that may be sought with case studies. If th

multiple data collection teams across case

attention must be given to data reliability.

Data Analysis and Reporting. Multiple site

analysis more complicated and reporting m

voluminous. The analysis techniques sugge

Miles and Huberman (1984), Strauss (1987

and Corbin (1990) and the software descri

Tesch (1990) may be useful.

Where to Look for More

Information

The references in the section on single cas

apply.

The

Criteria-Referenced

Case

Case studies can be adapted for answering 

questions about how well program operati

outcomes meet their criteria.

EXAMPLE: Social workers must be able to 

fraudulent claims under the Social Securit

Insurance Program. To make sure of the u

application of the law, program administra

developed standard procedures for substa

claims for benefits under the program. A c

could compare procedures actually used b

workers to those prescribed by the progra

administrators.

The examination of a number of cases mig

violations of prescribed claims-verification

procedures. Unlike the criteria-referenced 

design, the criteria-referenced case study w

permit an estimate of the frequency with w

violations occur. It could show only that v

or do not occur and, if they do, it might giv

Page 49 GAO/PEMD-10.1.4 Desigin
to why. Of course, if the number of cases i

violations are rare, the fact that there are v

may go undetected with the case study app

Applications The applications of the criteria-referenced 

design are similar to those of the counterp

under the sample survey strategy. The maj

difference stems from the fact that data fro

studies cannot be statistically projected to 

population. However, for a fixed expenditu

resources, the case study may allow deepe

understanding of a program’s operations o

and how these compare to the criteria that 

set for the program. Since case studies can 

expensive, care must be taken to ensure th

of cost estimates before choosing case stu

other designs. Two applications are likely: 

exploration that looks forward to a more

comprehensive project and a determinatio

possibility, if not the probability, that a cri

not been met.

Planning and

Implementation

How to reach consensus on the criteria an

measure performance against a criterion—

are important in criteria-referenced sampl

surveys—are considerations in criteria-ref

case studies. In addition, the question of h

choose cases for study is crucial because t

conclusions may differ, depending on the s

cases.

Where to Look for More

Information

The references cited above for case studie

criteria-referenced survey designs are app

The Field

Experiment

The main use of field experiment designs i

causal inferences about programs—that is

impact (cause-and-effect) questions. These 

allow the evaluator to compare, for examp

Page 50 GAO/PEMD-10.1.4 Desigin
of persons who are possibly affected by a 

others who have not been exposed to the p

The evaluation question might be, Does th

School Lunch Program improve children’s 

answer the question, the evaluator could c

measurement of the health of children par

the program to a measurement of the heal

children who are not participating.

Field experiments are distinguishable from 

experiments and experimental simulations 

field experiments take place in much less 

settings. Conducting an inquiry in the field 

reality to the evaluation, but it is often at t

of some accuracy in the results. From a pr

point of view, GAO’s only plausible choice 

three is usually experiments in the field. T

experiments, nonequivalent comparison gr

before-and-after studies—the field experim

designs we outline below—have in commo

measurements are made after a program h

implemented. Their major difference is in 

which program participants’ outcomes are 

as can be seen in the first row of table 3.3. 

important differences—the persuasivenes

arguments derived from the designs and th

administration—are shown in rows two an

Page 51 GAO/PEMD-10.1.4 Desigin
Table 3.3: Some Basic Contrasts Between Three Field Experiment Desig

Design

Basis for contrast True experiment

Nonequivalent

comparison group Before 

Measurements of

program participants

are compared to

measurements of

others in a randomly

assigned

comparison group

others in a

nonequivalent

comparison group

same p

before p

impleme

Persuasiveness of

argument about the

casual effect of

program on

participant is

generally strong quite variable usually 

for inter

subtype

Administering the

design is

usually difficult often difficult relativel

The True

Experiment

The characteristic of a true experimental d

that some units of study are randomly assi

treatment group and some are assigned to 

more comparison groups.8 Random assign

means that every unit available to the expe

a known probability of being assigned to e

and that the assignment is made by chance

flip of a coin. The program’s effects are es

comparing outcomes for the treatment gro

outcomes for each comparison group.

EXAMPLE: The Emergency School Aid Ac

grants to school districts to ease the probl

school desegregation. An evaluation quest

children in schools participating in the pro

attitudes about desegregation that are diff

those of children in schools that are deseg

not participating in the program? For each 

receiving a grant, a list was formed of all s

8

In some experiments, units are assigned randomly to 

of treatment—for example, different guaranteed inco

Page 52 GAO/PEMD-10.1.4 Desigin
eligible to participate in the program. The 

available to the evaluation consisted of the 

eligible to participate in the program. With

school district, some schools were random

to receive program funds in the treatment 

the remainder became the comparison gro

Although the true experimental design is u

be applied much by GAO evaluators, it is a

design in other evaluation settings in that i

the strongest design for causal inference a

a useful yardstick by which to assess weak

potential weaknesses in a cause-and-effect 

The great strength of the true experimenta

that it ordinarily permits very persuasive s

about the cause of observed outcomes.

An outcome may have several causes. In e

government program to find out whether i

particular outcome, the simplest true expe

design establishes one group that is expos

program and another that is not. The differ

their outcomes is attributed, with some qu

to the program. The causal conclusion is ju

because, under random assignment, most 

factors that determine outcomes other tha

program itself are evenly distributed betw

groups; their effects tend to cancel one an

a comparison of the two groups. Thus, onl

program’s effect, if any, accounts for the d

Applications When the evaluation question is about cau

effect and there is no ethical or administra

obstacle to random assignment, the true e

is usually the design of choice. The basic d

used frequently in many different forms in 

and agricultural evaluations but less often 

fields.

Page 53 GAO/PEMD-10.1.4 Desigin
The true experiment is seldom, if ever, fea

GAO because evaluators must have contro

process by which participants in a program 

assigned to it, and this control generally re

executive branch. Being able to make rand

assignments is essential: the true experime

is not possible without it. The obstacles m

overcome in a joint initiative between the 

branch and the evaluators, making a true e

possible. Also, GAO occasionally reviews t

experiments carried out by evaluators in th

executive branch.

Planning and

Implementation

Generalization. If the ability to generalize i

true experimental design may be unwarran

Generalization requires that the units in th

experiment be a probability sample drawn 

population of interest, but with a probabili

more than a few units are likely to refuse t

participate.9 Sometimes, as in the school e

above, this may not be a problem because

participation in the experiment may be a c

program participation. In other true exper

limitations in the available units may not b

because either generalization from the res

broad population is not a goal or the effect

treatment are expected to be reasonably u

within the population. In the latter case, an 

can be made to generalize even without a p

sample from the population of interest. Su

likely in some fields like medicine, where r

constant treatment effects may be expecte

9

It is important to bear in mind that a random sample 

population and random assignment to a treatment or 

group are two quite different things. The first is for th

generalizing from a sample to a population; random s

ensure external validity. The second is for inferring

cause-and-effect relationships; random assignment h

internal validity.

Page 54 GAO/PEMD-10.1.4 Desigin
less likely in evaluating government progra

policies.

Maintenance of Experimental Conditions. 

apply the logic of random assignment to re

about cause and effect, the evaluator must 

the composition of the groups, and thus th

of the experiment, is maintained. One of th

threats to causal reasoning from a true exp

that the members of the treatment and com

groups may drop out at different rates. If p

out more from one group than from anoth

might if they find the treatment disagreeab

example—then the evaluator’s estimate of 

effects may be distorted. Likewise, if the tr

allowed to weaken or to vary from particip

participant or to spill over to a comparison 

findings from the evaluation will be compr

Where to Look for More

Information

Judd and Kenny (1981) and Rossi and Free

(1989) are general evaluation texts that giv

considerable attention to the true experim

intensive treatments may be found in Boru

Wothke (1985), Hausman and Wise (1985)

(1982), Keppel and Zedeck (1989), and Spe

Many references listed under the nonequiv

comparison group design apply here as we

The Nonequivalent

Comparison Group

Design

As with the true experiment, the main purp

nonequivalent comparison group design is 

impact (cause-and-effect) questions. A fur

is that both designs consist of a treatment 

one or more comparison groups. Unlike th

the true experiment, however, membershi

nonequivalent comparison groups is not ra

assigned. This difference is important beca

implies that, since the groups will not be e

Page 55 GAO/PEMD-10.1.4 Desigin
causal statements about treatment effects 

substantially weakened.

EXAMPLE: Occupational training program

provide people with skills to help them ob

keep good jobs. An evaluation question mi

the average weekly earnings of program gr

higher than would have been expected had 

participated in the training? Participants h

ordinarily selected themselves for enrollm

programs, which rules out random assignm

be possible to compare the participants wi

members of another group, but the membe

participant group and the comparison grou

almost certainly not be equivalent in age, g

race, and work motivation. Therefore, the 

difference in their earnings would probabl

appropriate indicator of the effect of the tr

program, but other comparisons might be 

drawing cause-and-effect inferences.

This example is intended to show that whe

treatment and comparison groups are not 

assigned, it is usually not possible to infer 

“raw” difference between the groups has b

by the treatment. In other words, the two g

probably differ with regard to other factor

the difference in outcome, so that the raw 

should be adjusted to compensate for the 

equivalence between the groups. Using adj

procedures, including such statistical tech

the analysis of covariance, may strengthen 

evaluation conclusions.

Applications Nonequivalent comparison group designs 

used to answer impact (cause-and-effect) 

because they are administratively easier to 

than true experiments and, in appropriate

circumstances, they permit relatively stron

Page 56 GAO/PEMD-10.1.4 Desigin
statements. Evaluations of health, educati

criminal justice programs can generally co

from untreated comparison groups but can

noted above, easily assign subjects random

groups in a true experimental design. For 

evaluation designed to look at the effects o

correctional treatment on the recidivism o

criminals through a true experiment would 

not be feasible, because judges base their 

on the severity of a crime, number of prior 

and similar factors, and they would not or

willing to randomize the correctional treat

they declare.

Planning and

Implementation

Formation of Comparison Groups. The aim 

nonequivalent comparison group design is 

causal inferences about a program’s effect

evaluator’s two most important considerat

doing this are the choice of the compariso

and the nature of the comparisons. In the a

random assignment, treatment groups and

comparison groups may differ substantiall

dissimilarity usually weakens the conclusi

because it is not possible to rule out factor

than the program as plausible causes for th

For example, to evaluate a nutritional prog

pregnant women, it might be administrativ

convenient to compare program participan

urban area with nonparticipants in a rural 

would be unwise, however, because dietar

such differences between the two groups c

account for differences in the status of the

and thereby exaggerate or conceal the effe

program. Therefore, in most circumstance

advisable to form treatment and comparis

that are as alike as possible.10

10The evaluator who has precise control over assignm

group may prefer instead the “regression discontinui

assignment, design, in which the groups are distinctly 

known ways that can be adjusted for by statistical pr

Page 57 GAO/PEMD-10.1.4 Desigin
Naturally Occurring Comparison Groups. F

evaluations, the evaluator is not the one w

the treatment and comparison groups. Rat

evaluator is often presented with a situatio

some people have been exposed to the pro

others have not. Although the presence of 

constituted comparison groups somewhat 

evaluator’s options, the general logic of th

the same.

Nature of the Comparisons. The way in wh

treatment groups are compared to compar

involves statistical techniques beyond the 

this paper. We can point out, however, tha

important that plans for the comparison b

early, because it will be necessary to colle

precisely how the groups are not equivalen

Design Sensitivity. It is crucially important 

experimental designs be sensitive enough 

effects if they exist. A number of factors, s

sampling error, measurement error, subjec

variability, and the type of statistical analy

determine the likelihood that a given evalu

reveal a true effect. Lipsey (1990) provides 

overview of the most important considera

developing a design.

Where to Look for More

Information

Many design issues are covered by Cook a

Campbell (1979), Cronbach (1982), Judd a

(1981), Kish (1987), Mohr (1988), Posavac 

(1989), Rossi and Freeman (1989), and Sax

(1981). Design sensitivity and statistical po

treated by Cohen (1988), Kraemer and Thi

(1987), and Lipsey (1990). Achen (1986), A

al. (1980), Keppel and Zedeck (1989), and 

(1982), as well as many of the preceding au

address data analysis issues.

Page 58 GAO/PEMD-10.1.4 Desigin
The

Before-And-After

Design

The distinguishing feature of before-and-af

is that they compare outcomes for the unit

before the units were exposed to a program 

outcomes measured one or more times aft

began to participate in it. There is no comp

group as it exists in the other designs.

EXAMPLE: A training program was establi

help increase the earnings of workers who 

job skills. For a random sample of trainees

evaluation reported their average weekly i

before and after their participation in the p

Although this simple version of a before-an

design can be used to answer questions ab

amount of change that has been observed, 

allow the attribution of that change to exp

program. This is because it is not possible 

the effects of the training program from ot

influences on the workers such as the avai

jobs in the labor market, which would also 

earnings. The absence of a comparison gro

weakens the kinds of conclusions that can 

because comparison groups help rule out 

explanations for the observed outcomes.

Before-and-after designs can be strengthen

addition of more observations on outcome

instead of looking at a given outcome at tw

time, the evaluator can take a look at man

time; with a sufficient number of points, an

interrupted time series analysis can be app

before-and-after design to help draw causa

inferences. (Such longitudinal data can als

to advantage with the nonequivalent comp

group design: comparisons can be made b

or more time series.)

Page 59 GAO/PEMD-10.1.4 Desigin
EXAMPLE: After the development of a me

vaccine early in the 1960’s, the Centers for 

Control instituted a nationwide measles er

program. Grants were made to state and lo

authorities to pay for immunization. By 19

series of data was available that reported c

measles by 4-week periods. The evaluation 

was, What was the effect of the federal me

eradication program on the number of me

The answer, provided by a before-and-afte

using interrupted time series analysis, requ

distinguishing the effects of the federal pro

the effects of private physicians acting in c

state and local health authorities.

Before-and-after designs with a number of

observations over time may provide defen

answers to impact (cause-and-effect) ques

Multiple observations before and after an 

rule out alternative explanations, just as co

groups do in other designs.

Applications GAO evaluators are most likely to apply

before-and-after designs that employ inter

series analysis to data either collected by G

made available from other public sources. 

of the Census, the National Center for Hea

Statistics, the National Center for Educatio

Statistics, the Bureau of Labor Statistics, a

other such agencies may provide data for

investigating the effects of introducing, wi

or modifying national programs. Evaluator

that the best application is for studies in w

series of observations has been interrupte

change in the operation of a federal progra

Planning and

Implementation

Alternative Causal Explanations. The gene

weakness of before-and-after designs arise

absence of comparison groups that could h

Page 60 GAO/PEMD-10.1.4 Desigin
out alternative causal explanations. Howev

an interrupted time series can often help m

arguments relatively strong.

Number of Observations. The simple befor

design is seldom satisfactory for cause-and

arguments, although it may suffice for mea

change. The traditional rule of thumb for i

time series analyses says that at least 50 ob

are required, but some analysis methods u

(Forehand, 1982).

Data Consistency. When measurements ar

repeatedly, definitions and procedures ma

Care must be taken to see that time series 

definitional and measurement changes, be

can be mistaken for program effects.

Where to Look for More

Information

Many of the references for design and ana

nonequivalent comparison group design co

before-and-after design as well. In addition

(1982) and McCleary and Hay (1980) speci

address the time-series design.

The Use of

Available Data

The evaluation strategies discussed above 

involve the need to collect new data in ord

answer an evaluation question. Because da

collection is costly, it is always wise to see 

information will suffice. Even if the conclu

new data should be acquired, the analysis 

are already available may be warranted for 

tentative, answers to questions that will be 

completely addressed with new data at a l

Available data may be used to address eva

questions not intended when the data wer

collected. We discuss two approaches to t

of using available data: secondary data ana

evaluation synthesis.

Page 61 GAO/PEMD-10.1.4 Desigin
In the first approach, the evaluator may ha

access to data and the need to analyze the

others have done so. For example, second

analysis might answer an evaluation quest

looking at decennial census data published 

Bureau of the Census and widely used by o

In an evaluation synthesis, the evaluator c

number of previous evaluations that more 

address the current question. For example

possible to synthesize several evaluation f

how behavior-modification programs affec

delinquents in such a way that the synthes

is more credible than the finding of any of 

evaluations taken individually.

Secondary Data

Analysis

We refer to secondary data analysis as an 

rather than a design because the data that 

involved have already been acquired unde

design for data collection, using some tech

as self-administered questionnaires. If the 

was a sample survey, for example, the ana

have produced descriptive statistics. The s

data analysis might produce causal inferen

another method.

EXAMPLE: Data from 11 sample surveys w

a major secondary analysis that sought to 

the effects of family background, cognitive 

personality traits, and years of schooling o

economic success. The data that were ava

varied from survey to survey, but overall th

investigation focused on American men 25 

old, and economic success was expressed 

annual earnings or an index of occupation

Multivariate statistical methods were used 

inferences about cause-and-effect relation

among the variables.

Page 62 GAO/PEMD-10.1.4 Desigin
Applications Probably the most common application of 

data analysis in GAO is in answering quest

were not posed when the data were collec

large data sets produced by sample survey

of a program’s administrative procedures a

available for secondary analysis. The most 

answers in secondary data analysis are de

but normative and impact (cause-and-effec

questions can be considered.

Planning and

Implementation

Access to Data. Some data bases, such as t

produced by the Bureau of the Census, are 

easy to obtain. Others, such as those produ

private research firms, may be much more 

even impossible to acquire. Confidentiality 

privacy restrictions may prevent access to 

data.

Documentation of Data Bases. There are g

two kinds of documentation problems. Au

data may be difficult to read if the informa

been recorded idiosyncratically. The secon

arises when it is hard to understand how t

were collected. How were the variables de

What was the sample? How were the data 

How were the data processed and tabulate

were composite variables, such as indexes

from the raw data? Misunderstanding such 

lead to a misuse of the data.

Data Mismatched to Questions. When the 

wants to answer an evaluation question wi

collected for another purpose, it is very lik

data will not exactly meet the need. For ex

population may be a little different from th

evaluator has in mind, or variables may ha

defined in a different way. The solution is 

the question or to state proper caveats abo

conclusions.

Page 63 GAO/PEMD-10.1.4 Desigin
Where to Look for More

Information

Boruch (1978), Boruch et al. (1981), Bowe

Cook (1974), Hoaglin et al. (1982), Hyman 

Jacob (1984), Kiecolt and Nathan (1985), a

(1984) address a variety of issues pertainin

secondary analysis.

The Evaluation

Synthesis

Some evaluation questions may have been 

already with substantial research. The eva

synthesis aggregates the findings from ind

studies in order to provide a conclusion m

than that of any one study.

EXAMPLE: Many studies have been made 

effects of school desegregation. An evalua

synthesis statistically aggregated the resul

studies of students who had been reassign

segregated to desegregated schools in ord

answer the question of how the achieveme

students is affected when desegregation o

government action. The evaluation combin

samples of black students from 67 cities. E

original studies used some type of field ex

design.

An evaluation synthesis may take any one 

forms. At the opposite extreme of this exa

synthesis may be qualitative but beyond th

typical literature review. The evidence is w

qualitatively combined, but there is no atte

statistically aggregate the results of individ

A variety of synthesis procedures have bee

for statistically cumulating the results of s

studies. Probably the most widely used pro

answering questions about program effect

“meta-analysis,” which is a way of averagin

sizes” from several studies. Effect size is p

Page 64 GAO/PEMD-10.1.4 Desigin
to the difference in outcome between a tre

group and a comparison group.

Applications Some form of synthesis is appropriate whe

evidence can answer or partially answer a

question. When there is much information 

quality, a synthesis alone may satisfactoril

the question. If the information falls consid

short, it may be useful to perform an evalu

synthesis for a tentative, relatively quick a

to follow some other strategy for a more d

answer.

When an issue is highly controversial, the 

synthesis may help resolve it, because the 

takes account of the variable quality of con

evidence. The evaluations being reviewed 

synthesis may be graded for quality. Judgm

be made about what to include from them 

synthesis, or all usable information may be 

as in some forms of meta-analysis. For the 

relationship between quality and effect is s

analyzed.

Syntheses almost always identify gaps in a

information. Finding gaps is not the aim of 

evaluation synthesis, but when a dedicated 

information reveals them, they can be usef

clarifying a debate. Of course, knowing ab

information gaps may usefully trigger the g

new evidence.

Planning and

Implementation

Choice of Form. The nature of the evidenc

determines the appropriate form. Quantita

techniques such as meta-analysis are prob

most stringent, but all syntheses require in

about how the evaluations being examined 

conducted. This means that the evaluator 

Page 65 GAO/PEMD-10.1.4 Desigin
become familiar with the literature before 

a form to use.

Selection of Studies. In synthesizing evalu

evaluator must make important decisions 

to define the population of applicable stud

how to ensure that the population or an ap

sample of it will be examined. Typically, th

systematically screens the population, sele

specific studies for consideration.

Reliability of Procedures. A synthesis typic

involves the detailed review of many studi

may be undertaken by several staff membe

the work is divided among evaluators, atte

be given to the reliability of the synthesis p

that the staff members use. Although cons

procedure does not alone ensure sound co

it is necessary. Uniform procedures, such 

of codebooks, must be established and che

be made to verify their effectiveness.

Where to Look for More

Information

U.S. General Accounting Office (1983), Lig

Pillemer (1984), and Yeaton and Wortman 

relatively broad treatments of the evaluati

synthesis. Glass, McGaw, and Smith (1981

and Olkin (1985), Hunter and Schmidt (198

Schmidt, and Jackson (1982), Rosenthal (1

Wachter and Straf (1990), and Wolf (1986) 

on meta-analysis. Cooper (1984) and Jacks

(1980) discuss integrative research review

Heald (1975) discuss a method for aggrega

case studies, and Noblit and Hare (1988) tr

considerations involved in synthesizing qu

studies.

Page 66 GAO/PEMD-10.1.4 Desigin
Linking a Design

to the Evaluation

Questions

With particular strategies, designs, and app

mind, the evaluator should consider the ty

evaluation question being asked and a num

design-screening questions in order to nar

choices. The point of departure is the eval

question. Is it descriptive (about how a hig

training program was implemented)? Is it n

(about whether the job-placement goals of 

high-tech training program were met)? Is i

(about whether the high-tech training prog

effect on job-placement rates)? The answe

determine the design or approach to choo

The choice of what design or approach to 

further narrowed with the help of several

design-screening questions about the defin

needed in the conclusions and the kind of 

that are expected. An example of the form

we be able to generalize from what we exa

evaluation to some larger class of things? E

of the latter are, Can a comparison group b

Do we have 6 months or 18 months in whi

perform the evaluation?

Figure 3.1 is a decision tree that illustrates 

process of choosing an evaluation design. 

branches at the top of the figure point the 

answer about the type of evaluation questi

(descriptive, normative, or causal). Branch

down in the figure point out the place at w

design-screening questions (Do we want to 

the findings? Can a comparison group be f

formed? Can subjects be randomly assigne

groups? Can outcomes be measured over t

Page 67 GAO/PEMD-10.1.4 Desigin
Figure 3.1: Linking a Design to the Evaluation Questions

Page 68 GAO/PEMD-10.1.4 Desigin
Page 69 GAO/PEMD-10.1.4 Desigin
It must be stressed that the design-screeni

questions in the figure are illustrative and 

figure presents only selected technical ma

example, approaches using available data 

omitted. Other, equally important factors i

a design have also been omitted. They incl

availability of resources, the intended use 

evaluation, and the date when the evaluati

expected. When these factors represent co

they put boundaries around what can be d

As a design evolves, and as the evaluation 

become more specific and research possib

narrow, the evaluator must balance the tec

considerations against the constraints. For 

it might be necessary to choose between c

new data, which might answer the evaluat

questions comprehensively, and using ava

which is usually the least expensive cours

quickest but may leave some avenues unex

The decision tree almost always ends with 

instruction to consider a particular type of 

However, we emphasize the tentativeness 

“consider,” because we do not want to sug

there is only one way of designing evaluati

Answers to design-screening questions are 

as clear-cut as the decision tree suggests, a

relative importance of even these question

debated. Furthermore, most evaluations m

several questions, and where there are sev

questions, there may be several design typ

with only one question, it may be advisabl

more than one design. The strengths and t

weaknesses of several designs may offset 

another. Thus, the decision tree is not a rig

procedure but a conceptual guide for a sys

consideration of design alternatives (McGr

and Kulka, 1982).

Page 70 GAO/PEMD-10.1.4 Desigin
We have been stressing a consistent theme

development of an evaluation design is a s

process that takes time, thought, and craft

evaluator must pay careful attention to the

formulation of questions and the means of 

them. This painstaking work can be length

start of a job, but postponing or eliminatin

invitation to costly delays, incomplete or m

data collection, and uncertain analysis. To 

design is to think strategically; it is to see t

between the questions being asked and the 

which to collect and analyze the data for a

them. Our theme is exemplified in the narr

follows about the development of a design 

congressionally requested evaluation of th

1981 changes to the Aid to Families with D

Children (AFDC) program.

The Context The Omnibus Budget Reconciliation Act o

mandated important changes to AFDC, a m

welfare program at the center of debate ab

and work. On the one hand were people w

suggested that providing welfare income r

recipient’s motivation to work and creates

dependence on welfare and a permanent u

of nonworkers; these people favored strict 

criteria for the program and work requirem

welfare recipients. On the other hand were 

suggested that work incentives and work

requirements are irrelevant to a welfare po

composed largely of households headed b

with small children, who either cannot find 

cannot find work that pays enough to mee

daycare, transportation, or medical expen

The AFDC program had grown during the 

3.0 million to 7.3 million in recipients and f

billion to $3.5 billion in costs. By 1980, the 

Page 71 GAO/PEMD-10.1.4 Desigin
was 11.1 million persons and the yearly co

$12.5 billion. Throughout the period, attem

made to slow the growth.

For example, AFDC’s expansion during th

both in the level of benefits and in the cate

eligibility, had been accompanied by a mov

encourage mothers who were receiving be

work. In 1962, a community work and train

program had emphasized voluntary trainin

social services as an alternative to prolong

participation in AFDC.

Another strategy had been to reduce the 1

federal tax on the earnings of AFDC famili

that was seen as a “disincentive” to work b

each dollar earned was a welfare dollar los

Modifying this strategy in 1967, the Congre

incorporated an “earned-income disregard

into the AFDC program, allowing recipient

$30 each month with no reduction in bene

rate of 0 percent—and disregarding one th

additional earnings.

Along with this change, the Congress enac

Work Incentive (WIN) program, in which A

recipients could volunteer to receive train

During the 1970’s, however, as the caseloa

to grow, registration in WIN was made ma

some AFDC households.

The changes in the AFDC regulations that 

specified in the 1981 Omnibus Budget Rec

Act focused again on work requirements b

the states to operate mandatory “workfare

Other amendments to the legislation chang

policy of allowing working welfare familie

accumulate more income than that availab

nonworking welfare families. One of the k

Page 72 GAO/PEMD-10.1.4 Desigin
provisions limited the earned-income disre

months and the total income of an AFDC h

to 150 percent of the AFDC need standard

established by each of the states.

The Request In June 1982, the House Committee on Wa

Means asked GAO to study the 1981 modif

the AFDC program. The changes were exp

remove many working AFDC families from 

program’s rolls, causing many of them to l

eligibility for Medicaid. Other families wou

to remain on the rolls but with significantl

benefits. One concern of the committee w

faced with the prospect of losing benefits 

them greatly diminished, the families woul

choose to work less or quit working entire

cutting back on work, they could retain th

eligibility for AFDC and Medicaid. Howeve

with the loss of benefits, families might ins

increase their work effort in order to comp

the loss.

The committee specifically asked GAO to 

(1) the economic well-being, 6 to 12 month

act’s effective date, of the AFDC families t

been removed from the rolls and that had 

benefits reduced and (2) whether families 

benefits had returned to the rolls or compe

their welfare losses by cutting back on wo

If working families who would lose AFDC 

their grants reduced were to lessen their w

in order to stay on the rolls, projected bud

from the legislated changes would be nega

diminished. Therefore, GAO was asked to 

the budgetary effect of the program chang

request also required GAO to find out whe

changes had affected family or household

Page 73 GAO/PEMD-10.1.4 Desigin
composition and to provide information ab

demographic, income, and resource chara

the AFDC families both before and after th

and the frequency with which they moved 

the rolls. The committee asked GAO to ma

report early in 1984, which it did with the A

report entitled An Evaluation of the 1981 A

Changes: Initial Analyses (PEMD-84-6), iss

Program Evaluation and Methodology Div

final report entitled An Evaluation of the 1

Changes: Final Report (PEMD-85-4) was is

July 2, 1985.

Design Phase 1:

Finding an

Approach

The evaluators began by exploring ways o

key questions and strategies for answering 

They reviewed the substantive and the

methodological literature and acquired inf

on the program’s operations. They explore

relevance of available data, and they consu

the committee’s staff and other experts.

The literature review centered on welfare

dependence, the effects of earlier changes 

program, and the methods other researche

to address questions of similar scope and c

A systematic reading of the voluminous lit

these topics generated a number of import

that guided further thinking and refinemen

study. For example, the reading on welfare

dependence led to three hypotheses on the 

growth of the AFDC caseload. Similarly, th

pointed out areas where information is lac

as on the rate at which people leave welfar

and do not return within a specified time.

The evaluators found that the literature on 

effects stressed the need for a longitudinal

perspective. They found that the reports re

Page 74 GAO/PEMD-10.1.4 Desigin
to changes in the AFDC tax rates were info

design approaches as well as on findings. I

the earlier research methods, the evaluato

interested in identifying both designs and m

that fell short or were especially vulnerabl

that were successful. Thus, the review ind

not to do and suggested strategies that we

promising and worth further consideration

The evaluators also explored the relevance 

available data. The ability to make use of e

sets has the advantage of cutting the cost o

collecting, organizing, verifying, and autom

information. Five data sets were identified 

carefully scrutinized.

The consultation with experts included co

committee staff, economists, political scie

social welfare analysts, policy analysts, ev

specialists, and statisticians. Discussions r

a wide number of substantive and method

issues, and they were held frequently to al

ongoing critique of the design as it was bei

formulated. The consultation continued th

the study, suggesting valuable leads to pur

dead ends to avoid.

In acquiring information on the operation 

AFDC program, the evaluators paid attenti

operational procedures but also concentra

three areas. The first was how the states d

AFDC benefits before and after the 1981 a

and how the changes were implemented. T

was how the program was related to other 

from state to state. The third was the relat

the states between the participation of AF

and local economic conditions. Clearly ger

the questions posed by the committee, the

were stated as questions in language suffic

Page 75 GAO/PEMD-10.1.4 Desigin
general to allow the exploration of multipl

sources of information. The goal was not t

prematurely on potentially useful material 

lead to a thorough understanding of the pr

history, how it changed when federal polic

translated to the local level, and whatever 

increase the possibility of making cause-an

statements.

After about 6 weeks, this group of evaluato

design team, began to feel confident about 

several possible designs. Then they began 

alternative designs to evaluation questions

Design Phase 2:

Assessing

Alternatives

The constraints that came to light in phase 

subsequent thinking about the job and sha

assessment of various alternatives. This al

evaluators to refine the evaluation questio

they did in phase 2, so that they could sett

strategy and a final design.

The first of the constraints began to influe

design when the discussions with experts 

numerous visits to the states made it readi

that the “national” AFDC program is actua

different AFDC programs, one for each sta

heterogeneity was evident in the fact that 

develops its own payment levels and proce

setting work and child-care expense deduc

within the framework of the federal regula

For example, the evaluators found conside

variation with respect to two-parent famili

requirements about the presence of an une

parent, “need” standards, the percentage o

standard being paid to recipients, and ded

allowable for child-care and work expense

variations meant that quite dissimilar gran

Page 76 GAO/PEMD-10.1.4 Desigin
were being made to families whose compo

financial circumstances were identical. Th

circumstance placed pronounced limitatio

evaluators’ ability to generalize from indiv

to the nation.

A second constraint was that the states ha

their implementation of the changes unifo

states began to implement most of the cha

October 1981, but some states did not imp

some provisions until 6 months later, in sp

The variation meant that an aggregation of 

all states would be problematic and that

generalizations would be limited. Consequ

baseline for making comparisons would ha

from state to state.

Another constraint was that the study cou

predicated on the simple assumption that 

recipients would make choices between w

funds and employment funds. AFDC provi

income support but also enables the recipi

draw on a number of services, most notab

care under the Medicaid program. Any stu

people choose to stay in or leave the AFDC 

has to account for the other benefits. They 

an important, if not decisive, role in influe

financial decisions.

A constraint of a different type had to do w

of the population of working AFDC recipie

changes in the legislation were of immedia

relevance to working families, but their pr

small in relation to the total caseload. Nati

1979 figure was about 14 percent, but in so

was as low as 6 or 7 percent. The small pe

meant that data would have to be collected 

such that the numbers of earners would be 

enough to make statistical projections mea

Page 77 GAO/PEMD-10.1.4 Desigin
These and other constraints told the evalu

to refine the evaluation questions, they wo

pose a study within, rather than between, 

Similarly, the evaluators began to see the d

which the study would be able to isolate th

the legislative changes from other causal f

particularly when addressing AFDC recipi

decisions to stop working and stay on the 

remain off the rolls and seek to support th

through their own earnings. That is, the 19

to the program were initiated at a time wh

economies varied widely, so that the econ

not be “held constant,” or presumed to be 

among the states. Thus, it had to be consid

possible cause in earners’ decisions. The e

also found that their questions would have 

for reductions in other social welfare prog

As the design team refined the questions, g

constraints on answering them, it was able 

examine data collection and analysis strat

is, what the evaluators had learned about t

questions, and the considerations of time, 

availability, and user needs, enabled the de

to pull together and assess methods for ga

analyzing data. The evaluators saw two br

strategies, one that would primarily analyz

data and one that would require the collec

original data.

It was thought that using one of the five av

sets would be an economical and quick wa

early findings to the Congress. A data set c

“Job Search Assistance Research Project” 

was the most promising for a study of the 

the changes in the legislation. JSARP was 

the U.S. Department of Labor late in 1978 

large-scale effort to measure the effects of 

assistance, public-service employment, an

Page 78 GAO/PEMD-10.1.4 Desigin
training on the employment, earnings, and 

dependence of low-income persons (not a

were AFDC participants). Ten jurisdiction

Comprehensive Employment and Training 

were chosen as “treatment” sites, where sp

demonstration programs were established 

the employment opportunities of the targe

population. Each site was matched with a 

site as similar as possible in racial and eth

composition, unemployment rate, primary 

and occupations, size, and location. The re

interviewed 30,000 respondents in spring 1

the demonstration programs were being in

Slightly fewer than 3,000 of the responden

AFDC recipients for at least part of the yea

the interview. In 1980, a follow-up intervie

5,700 of the original respondents used sub

the same interviewing instrument; among 

respondents were all who had indicated ea

they had AFDC support, and a large propo

incomes below 225 percent of the poverty 

JSARP provides a lengthy record of earnin

income, work behavior, job search, job tra

family composition for a large sample prio

institution of the 1981 changes to AFDC.

The evaluators therefore thought that usin

before-and-after design and the JSARP dat

could interview the same respondents (or 

selected for their similarity to the JSARP

respondents) with the same or nearly the s

collection instrument to find out their exp

the 1981 changes. This would provide for a

comparison of work and welfare patterns 

after the program change, although it wou

establish with certainty whether the 1981 a

sole cause of any difference between the t

interview periods. Nevertheless, statistical 

might lead to defensible conclusions abou

Page 79 GAO/PEMD-10.1.4 Desigin
The alternative strategy, the one that was 

selected, involved collecting before-and-af

five sites across the country, making interv

five sites with members of working AFDC 

who were terminated from AFDC when th

was implemented, and analyzing national

before-and-after data on AFDC caseloads a

Of the designs we discussed in chapter 3, t

approach included three designs—a noneq

comparison group design, a one-group

before-and-after design, and a national inte

time series design.

The plan for the nonequivalent comparison 

design was to identify at each site two sam

AFDC recipients, one from a year and a m

the changes and one from the month imme

preceding them. The earlier group would p

baseline from which to look at the dynami

and welfare both immediately before and a

implementation of the act. Both samples w

for separate subsamples of working and n

AFDC recipients. Depending on the compl

case records at the sites, the following info

could be compared: length of participation 

percentage of AFDC households with earn

different times, percentage of households 

then returning to the rolls, average dollar a

AFDC benefits and earned income, percen

households drawing on various other welf

benefits, and reasons for the termination o

payments. Thus, the comparisons could be 

within and between groups and of several 

across three points in time (the baseline an

and after implementation). The evaluators 

compare the static characteristics of earne

nonearners, the employment status of the 

groups, and the relationship between chan

administrative practices and the behavior 

Page 80 GAO/PEMD-10.1.4 Desigin
respondents in terms of the time they spen

AFDC’s rolls, their average net earnings, a

they did because of changes in AFDC bene

Having decided on this approach, the evalu

constructed interviews within the case stu

component that were intended to collect d

assess the economic well-being of the pers

were removed from the rolls, how they cop

the loss of benefits, and whether they wor

keep up an income. Here, the comparisons 

within groups of households before and af

program changes. For example, the evalua

compare household composition, employm

earnings, and total disposable income. Of p

interest would be data on whether people 

their work effort or shifted their reliance f

to other programs such as General Assista

Unemployment Insurance.

The national analysis component, with its 

time series analysis, would rely on data pr

the U.S. Department of Health and Human 

and by state welfare departments on the o

AFDC programs, including the implementa

1981 provisions, and on caseloads and out

AFDC and related programs. The objective

planned were to document the degree to w

1981 AFDC provisions represented change 

practices, to explore their effects on natio

caseloads and costs, and to determine whe

states tried to negate or reduce the effects 

provisions. The design team planned for a 

all the states to provide GAO with the resu

own independent evaluations.

Two smaller and complementary compone

also posited. One would use archival data 

other would require conducting interviews 

Page 81 GAO/PEMD-10.1.4 Desigin
and local program officials and staff. The a

data would include information on AFDC 

fluctuations and local economic condition

Collecting these data would explore the de

which different patterns of dependence on 

three periods might be the product of even

than the AFDC changes, such as a deterior

market.

Design Phase 3:

Settling on a

Strategy

In the end, a choice has to be made betwe

competing design options. The difficulty fo

evaluator making this choice is in assessin

alternatives. Each one will have strengths 

weaknesses, so that the decision comes to 

be both most feasible and most defensible

AFDC study, the choice was made in favor 

multistrategy approach. The JSARP appro

available data and interviewing a sample o

original respondents was dropped.

To be sure, both approaches had strengths

strong arguments were made for both. The 

tipped against the simpler approach when 

weaknesses. There were several reservatio

using the JSARP data. There were problem

accuracy, precision, and completeness (lar

because the respondents’ report of AFDC

participation were retrospective to as far a

months).

There was a possibility of bias, since 23 pe

original respondents did not turn up for th

set of interviews, and the difficulty of findi

respondents for the new study could be ev

There were not enough earners in the sam

finally, practical problems included the fac

JSARP data were not for public use and m

obtainable.

Page 82 GAO/PEMD-10.1.4 Desigin
In light of all this, the multistrategy approa

adopted. Even with it, there was concern a

availability of case records, finding respon

had left the AFDC program, the extensive 

required to code case records at sites that 

have automated data, the ability to control 

disparate economic conditions site by site

sheer volume of data that would have to b

coded, analyzed, and reported. However, c

the concern about JSARP, which tended to 

analytical, these problems were more simp

procedural. In the end, it was concluded th

analytical problems were a greater threat t

ability to answer the study questions than 

procedural ones.

Page 83 GAO/PEMD-10.1.4 Desigin
Achen, C. H. The Statistical Analysis of

Quasi-Experiments. Berkeley, Calif.: Unive

California Press, 1986.

Anderson, S., et al. Statistical Methods for

Comparative Studies. New York: John Wile

Sons, 1980.

Babbie, E. R. The Practice of Social Resea

Belmont, Calif.: Wadsworth, 1989.

Babbie, E. R. Survey Research Methods, 2n

Belmont, Calif.: Wadsworth, 1990.

Bainbridge, W. S. Survey Research: A

Computer-Assisted Introduction. Belmont

Wadsworth, 1989.

Black, J. A., and D. J. Champion. Methods 

in Social Research. New York: John Wiley 

1976.

Boruch, R. F. (ed.) Secondary Analysis. Sa

Francisco: Jossey-Bass, 1978.

Boruch, R. F., et al. Reanalyzing Program E

Policies and Practices for Secondary Analy

Social and Educational Programs. San Fra

Jossey-Bass, 1981.

Boruch, R. F., and W. Wothke (eds.). Rand

and Field Experimentation. San Francisco

Jossey-Bass, 1985.

Bowering, D. J. (ed.) Secondary Analysis o

Data Bases. San Francisco: Jossey-Bass, 1

Page 84 GAO/PEMD-10.1.4 Desigin
Bradburn, N. M., and S. Sudman and Assoc

Improving Interview Method and Question

Design. San Francisco: Jossey-Bass, 1979.

Chelimsky, E. “The Definition and Measur

Evaluation Quality as a Management Tool.

St. Pierre (ed.). Management and Organiza

Program Evaluation, pp. 113-26. San Franc

Jossey-Bass, 1983.

Cohen, J. Statistical Power Analysis for th

Behavioral Sciences, 2nd ed. Hillsdale, N.J

Erlbaum, 1988

Converse, J. M. and S. Presser. Survey Que

Handcrafting the Standardized Questionna

Newbury Park, Calif.: Sage, 1986.

Cook, T. D. “The Potential and Limitations 

Secondary Data Analysis.” In M. W. Apple, 

Subkoviak, and H. S. Lufler, Jr. (eds.). Edu

Evaluation: Analysis and Responsibility, pp

Berkeley, Calif.: McCutchan, 1974.

Cook, T. D., and D. T. Campbell.

Quasi-Experimentation: Design and Analy

for Field Settings. Chicago: Rand McNally, 

Cooper, H. M. The Integrative Research Re

Systematic Approach, 2nd ed. Newbury Pa

Sage 1989.

Cronbach, L. J. Designing Evaluations of E

and Social Programs. San Francisco: Josse

1982.

Forehand, G. A. (ed.). Applications of Tim

Analysis to Evaluation. San Francisco: Jos

1982.

Page 85 GAO/PEMD-10.1.4 Desigin
Fowler, F. J., Jr. Survey Research Methods

Newbury Park, Calif.: Sage, 1988.

Fowler, F. J., Jr., and T. W. Mangione. Stan

Survey Interviewing: Minimizing Interview

Error. Newbury Park, Calif.: Sage, 1990.

Glass, G. V, B. McGaw, and M. L. Smith. M

in Social Research. Newbury Park, Calif.: S

Groves, R. M. Survey Errors and Survey Co

York: John Wiley and Sons, 1989.

Hausman, J. A., and D. A. Wise (eds.). Soci

Experimentation. Chicago: University of C

Press, 1985.

Hayduk, L. A. Structural Equation Modelin

LISREL. Baltimore: Johns Hopkins Univer

1987.

Hedges, L. V., and I. Olkin. Statistical Meth

Meta- Analysis. New York: Academic Press

Henry, G. T. Practical Sampling. Newbury 

Sage, 1990.

Herbert, L. Auditing the Performance of M

Belmont, Calif.: Lifetime Learning Publicat

Hersen, M., and D. H. Barlow. Single Case

Experimental Designs. New York: Pergam

Hoaglin, D. C., et al. Data for Decisions. Ca

Mass.: Abt Books, 1982.

Hunter, J. E., and F. L. Schmidt. Methods o

Meta-Analysis: Correcting Error and Bias i

Findings. Newbury Park, Calif.: Sage, 1989

Page 86 GAO/PEMD-10.1.4 Desigin
Hunter, J. E., F. L. Schmidt, and G. B. Jack

Analysis: Cumulating Research Findings A

Studies. Newbury Park, Calif.: Sage, 1982.

Hyman, H. H. Secondary Analysis of Samp

Middletown, Conn.: Wesleyan University P

Jackson, G. B. “Methods for Integrative Re

Review of Educational Research, 50 (1980

Jacob, H. Using Published Data: Errors an

Newbury Park, Calif.: Sage, 1984.

Judd, C. M., and D. A. Kenny. Estimating th

of Social Interventions. Cambridge, Eng.: C

University Press, 1981.

Judd, C. M., and L. H. Kidder. Research Me

Social Relations, 5th ed. New York: Holt, R

Winston, 1986.

Kalton, G. Introduction to Survey Samplin

Park, Calif.: Sage, 1983.

Keppel, G. Design and Analysis: A Researc

Handbook, 2nd ed. Englewood Cliffs, N.J.:

Prentice-Hall, 1982.

Keppel, G., and S. Zedeck. Data Analysis fo

Designs. New York: W. H. Freeman, 1989.

Kiecolt, K. J., and L. E. Nathan. Secondary 

Survey Data. Newbury Park, Calif.: Sage, 1

Kish, L. Survey Sampling. New York: John 

Sons, 1965.

Kish, L. Statistical Design for Research. Ne

John Wiley and Sons, 1987.

Page 87 GAO/PEMD-10.1.4 Desigin
Kraemer, H. C., and S. Thiemann. How Ma

Subjects? Statistical Power Analysis in Res

Newbury Park, Calif.: Sage, 1987.

Lee, E. S., R. N. Forthofer, and R. J. Lorimo

Analyzing Complex Survey Data. Newbury 

Calif.: Sage, 1989.

Light, R. J., and D. B. Pillemer. Summing U

Science of Reviewing Research. Cambridg

Harvard University Press, 1984.

Lipsey, M. Design Sensitivity: Statistical Po

Experimental Research. Newbury Park, Ca

1990.

Marshall, C., and G. G. Rossman. Designin

Qualitative Research. Newbury Park, Calif

1989.

McCleary, R., and R. A. Hay. Applied Time 

Analysis for the Social Sciences. Newbury 

Sage, 1980.

McGrath, J. E., J. Martin, and R. A. Kulka. J

Calls in Research. Newbury Park, Calif.: Sa

Miles, M. B., and M. Huberman. Qualitative 

Analysis. Newbury Park, Calif.: Sage, 1984

Mohr, L. B. Impact Analysis for Program E

Chicago: Dorsey, 1988.

Noblit, G. W., and R. D. Hare. Meta-Ethnog

Synthesizing Qualitative Studies. Newbury 

Calif.: Sage, 1988.

Patton, M. Q. Qualitative Evaluation and R

Methods, 2nd ed. Newbury Park, Calif.: Sa

Page 88 GAO/PEMD-10.1.4 Desigin
Payne, S. L. The Art of Asking Questions. P

N.J.: Princeton University Press, 1951.

Pedhazur, E. L. Mulitple Regression in Beh

Research, 2nd ed. New York: Holt, Rinehar

Winston, 1982.

Popham, W. J. Educational Evaluation. En

Cliffs, N.J.: Prentice-Hall, 1975.

Posavac, E. J., and R. G. Carey. Program E

Methods and Case Studies, 3rd ed. Englew

N.J.: Prentice-Hall, 1989.

Provus, M. M. Discrepancy Evaluation. Be

Calif.: McCutchan, 1971.

Rosenthal, R. Meta-Analytic Procedures fo

Research. Newbury Park, Calif.: Sage, 1984

Rossi, P. H., and H. E. Freeman. Evaluation

Systematic Approach, 4th ed. Newbury Pa

Sage, 1989.

Runkel, P. J., and J. E. McGrath. Research 

Behavior: A Systematic Guide to Method. N

Holt, Rinehart, and Winston, 1972.

Saxe, L., and M. Fine. Social Experiments: 

for Design and Evaluation. Newbury Park, 

Sage, 1981.

Scheaffer, R. L., W. Mendenhall, and L. Ott

Elementary Survey Sampling, 4th ed. Bost

PWS-Kent, 1990.

Spector, P. E. Research Designs. Newbury 

Calif.: Sage, 1981.

Page 89 GAO/PEMD-10.1.4 Desigin
Steinmetz, A. “The Discrepancy Evaluation 

In: G. F. Madaus, M. Scriven, and D. L. Stu

(eds.) Evaluation Models: Viewpoints on E

and Human Services Evaluation. Boston:

Kluwer-Nijhoff, 1983.

Stewart, D. W. Secondary Research: Inform

Sources and Methods. Newbury Park, Cali

1984.

Strauss, A. L. Qualitative Analysis for Soci

Scientists. Cambridge, Eng.: Cambridge Un

Press, 1987.

Strauss, A. L., and J. Corbin. Basics of Qua

Research. Newbury Park, Calif.: Sage, 1990

Sudman, S. Applied Sampling. New York: A

Press, 1976.

Tesch, R. Qualitative Research: Analysis T

Software Tools. New York: Taylor and Fra

U.S. General Accounting Office. The Evalu

Synthesis, methods paper I. Washington, D

April 1983.

U.S. General Accounting Office. Using Stru

Interviewing Techniques, methodology tra

5. Washington, D.C.: July 1985.

U.S. General Accounting Office. Using Stat

Sampling, transfer paper 6. Washington, D

April 1986a.

U.S. General Accounting Office. Developin

Questionnaires, transfer paper 7. Washingt

July 1986b.

Page 90 GAO/PEMD-10.1.4 Desigin
U.S. General Accounting Office. Case Stud

Evaluations, transfer paper 10.1.9. Washin

November 1990.

Wachter, K. W., and M. L. Straf (eds.). The 

Meta- Analysis. New York: The Russell Sag

Foundation, 1990.

Warwick, D. P., and C. A. Lininger. The Sam

Survey: Theory and Practice. New York: M

1975.

Wholey, J. S. Evaluation: Promise and Perf

Washington, D.C.: Urban Institute, 1979.

Wolf, F. M. Meta-Analysis: Quantitative Me

Research Synthesis. Newbury Park, Calif.: 

Yeaton, W. H., and P. M. Wortman (eds.). I

Data Synthesis. San Francisco: Jossey-Bas

Yin, R. K. Case Study Research: Design and 

rev. ed. Newbury Park, Calif.: Sage, 1989.

Yin, R. K., and K. A. Heald. “Using the Case 

Method to Analyze Policy Studies.” Admin

Science Quarterly, 20 (September 1975), 3

Page 91 GAO/PEMD-10.1.4 Desigin
Bias The extent to which a measurement or a s

analytic method systematically underestim

overestimates a value.

Construct An attribute, usually unobservable, such a

educational attainment or socioeconomic 

is represented by an observable measure.

Construct Validity The extent to which a measurement metho

accurately represents a construct and prod

observation distinct from that produced by 

of another construct.

Covariation The degree to which two measures vary to

Cross-Sectional Data Observations collected on subjects or even

single point in time.

External Validity The extent to which a finding applies (or c

generalized) to persons, objects, settings, 

other than those that were the subject of s

Generalizability Used interchangeably with “external validi

Internal Validity The extent to which the causes of an effec

established by an inquiry.

Longitudinal Data Sometimes called “time series data,” obser

collected over a period of time; the sample 

or cases) may or may not be the same each 

the population remains constant.

Page 92 GAO/PEMD-10.1.4 Desigin
Measurement A procedure for assigning a number to an 

object or event.

Panel Data A special form of longitudinal data in whic

observations are collected on the same sam

respondents over a period of time.

Probability Sampling A method for drawing a sample from a pop

such that all possible samples have a know

specified probability of being drawn.

Program

Effectiveness

Evaluation

The application of scientific research meth

estimate how much observed results, inten

are caused by program activities. Effect is 

cause by design and analysis that compare 

results with estimates of what might have 

observed in the absence of the program.

Program Evaluation The application of scientific research meth

assess program concepts, implementation

effectiveness.

Qualitative Data Information expressed in the form of word

that in some of the references cited, qualit

means numerical information in which the 

the difference between two numbers is no

meaningful.)

Quantitative Data Information expressed in the form of num

Measurement gives a procedure for assign

numbers to observations. See Measuremen

Page 93 GAO/PEMD-10.1.4 Desigin
Random Assignment A method for assigning subjects to two or 

groups by chance.

Reliability The quality of a measurement process that 

produce similar results on (1) repeated ob

of the same condition or event or (2) multi

observations of the same condition or even

different observers.

Representative

Sample

A sample that has approximately the same

distribution of characteristics as the popul

which it was drawn.

Simple Random

Sample

A method for drawing a sample from a pop

such that all samples of a given size have e

probability of being drawn.

Statistical

Conclusion Validity

The extent to which the observed statistic

significance (or the lack of statistical signi

the covariation between two or more varia

based on a valid statistical test of that cov

Structured Interview An interview in which questions to be aske

sequence, and the detailed information to 

are all predetermined; used where maximu

consistency across interviews and intervie

needed.

Treatment Group The subjects of the intervention being stud

Page 94 GAO/PEMD-10.1.4 Desigin
This is a flexible series continually being a

updated. The interested reader should inqu

the possibility of additional papers in the s

The Evaluation Synthesis. Transfer paper 

formerly methods paper I.

Content Analysis: A Methodology for Struc

Analyzing Written Material. Transfer paper 

formerly methodology transfer paper 3.

Designing Evaluations. Transfer paper 10.

formerly methodology transfer paper 4.

Using Structured Interviewing Techniques

paper 10.1.5, formerly methodology transf

Using Statistical Sampling. Transfer paper 

formerly methodology transfer paper 6.

Developing and Using Questionnaires. Tra

10.1.7, formerly methodology transfer pap

Case Study Evaluations. Transfer paper 10

formerly methodology transfer paper 9.

Prospective Evaluation Methods: The Pros

Evaluation Synthesis. Transfer paper 10.1.

methodology transfer paper 10.

(973317) Page 95 GAO/PEMD-10.1.4 Desigin
Ordering Information

The first copy of each GAO report and 

is free. Additional copies are $2 each. 

should be sent to the following addres

accompanied by a check or money orde

out to the Superintendent of Documen

necessary. VISA and MasterCard credi

are accepted, also. Orders for 100 or m

copies to be mailed to a single address 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax numbe

(301) 258-4066, or TDD (301) 413-000

Each day, GAO issues a list of newly a

reports and testimony. To receive fac

copies of the daily list or any list from 

30 days, please call (202) 512-6000 us

touchtone phone. A recorded menu w

information on how to obtain these lis

For information on how to access GAO 

on the INTERNET, send an e-mail mes

"info" in the body to:

info@www.gao.gov
Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Postage & F

GAO

Permit No