Division

November 1990 Case Study

Evaluations

GAO/PEMD-91-10.1.9

GAO assists congressional decisionmakers 

deliberative process by furnishing analytic

information on issues and options under

consideration. Many diverse methodologie

needed to develop sound and timely answ

questions that are posed by the Congress. 

GAO evaluators with basic information ab

more commonly used methodologies, GAO

guidance includes documents such as met

transfer papers and technical guidelines.

This methodology transfer paper on case s

evaluations describes how GAO evaluator

case study methods in performing our wor

describes six applications of case study m

including the purposes and pitfalls of each

explains similarities and differences amon

This paper presents an evaluation perspec

studies, defines them, and determines thei

appropriateness in terms of the type of eva

question posed. The original report was au

Lois-ellin Datta in April 1987. This reissued

(1990) version supersedes the earlier editi

Case Study Evaluations is one of a series o

issued

by the Program Evaluation and Methodolo

(PEMD). The purpose of the series is to pr

evaluators with guides to various aspects o

evaluation methodology, to illustrate appli

and to indicate where more detailed inform

available.

Page 1 GAO/PEMD-91-10.1.9 Case Stud
We look forward to receiving comments fr

readers of this paper. They should be addr

Eleanor Chelimsky at 202-275-1854.

Werner Grosshans

Assistant Comptroller General

Office of Policy

Eleanor Chelimsky

Assistant Comptroller General

for Program Evaluation and

 Methodology

Page 2 GAO/PEMD-91-10.1.9 Case Stud
Page 3 GAO/PEMD-91-10.1.9 Case Stud
Preface 

Chapter 1 

Introduction

Chapter 2 

What Are Case

Studies?

What Is Meant by “a Case Study”? 

Some Common Benefits Expected From

Case Study Evaluations

Instance Selection in Case Studies 

Chapter 3 

Case Study

Applications

Illustrative 

Exploratory 

Critical Instance 

Program Implementation 

Program Effects 

Cumulative 

Design Decisions and Case Study

Applications

Chapter 4 

Data Collection

and Analysis

Data Collection 

Data Analysis 

Handling Multisite Data Sets 

Basic Models for Data Analysis 

Pitfalls and Booby Traps 

Where to Go for More Information 

Chapter 5 

Summary

What Are Case Studies? 

When Are Case Studies Appropriately Use

in Evaluation?

What Distinguishes a Good From a

Not-Good Case Study?

Impartiality and Generalizability 

Page 4 GAO/PEMD-91-10.1.9 Case Stud
Appendixes Appendix I: Theory and History 

Appendix II: Site Selection Example 

Appendix III: Guidelines for Reviewing Ca

Study Reports

Bibliography 

Glossary 

Papers in This Series 

Tables Table 2.1: What Is a Case Study? Exercise. 

Table 2.2: Complexity of Questions 

Table 2.3: Methods of Obtaining Descriptio

and Analysis in Case Studies

Table 2.4: Some Common Benefits Expect

From Case Study Evaluations

Table 2.5: Instance Selection in Case Studi

Table 2.6: Hypothetical Data on Instance

Selection

Table 3.1: Illustrative Case Studies 

Table 3.2: Exploratory Case Studies 

Table 3.3: Critical Instance Case Studies 

Table 3.4: Program Implementation Case

Studies

Table 3.5: Illustration of Differences in

Note-Taking

Table 3.6: Program Effects Case Studies 

Table 3.7: Cumulative Case Studies 

Table 3.8: Some Design Decisions in Case

Study Methods

Table 4.1: Ways of Analyzing Case Study

Data

Table 5.1: Some Common Pitfalls in Case

Study Evaluation

Table I.1: Criteria of Good Research 

Table I.2: Evaluation Adaptations of the

Research Case Study

Table II.I: Hypothetical Data on Unfiled

Corporate Income Tax Returns for 1986

State Income Tax Returns

Page 5 GAO/PEMD-91-10.1.9 Case Stud
Table III.1: Checklist for Reviewing Case

Study Reports

Abbreviations

GAO General Accounting Office

OTTR Observe, think, test, and revise

PEMD Program Evaluation and Methodo

Division

SSA Social Security Administration

Page 6 GAO/PEMD-91-10.1.9 Case Stud
Page 7 GAO/PEMD-91-10.1.9 Case Stud
At his government-required anti-terrorist t

session recently, a captain for a major airli

“The bits of information were so few and far between 

weren’t even paying attention. My instructor for the e

course entered the room only to change videotapes. 

talking; they were doing other things, including readi

(Philadelphia Inquirer, 1986)

This is a case instance. It is an effective wa

drawing attention to a problem such as tra

quality. Such anecdotes are remembered a

convincing. What they are not, however, is

generalizable: that is, an anecdote doesn’t 

it is the only such instance or whether the 

wide-spread. And anecdotes usually don’t 

reasons for a situation, and thus are of lim

in suggesting solutions.

The challenge for evaluators is how to use 

aspects of an anecdote that are effective fo

work—the immediacy, the convincingness

attention-getting quality—and, at the same 

other informational requirements for our j

generalizability and reliability. Case study 

while not without their limitations in this r

help us answer this challenge.

GAO already does a lot of case studies—o

what we ourselves call case studies in des

methods. There are GAO case studies in m

areas—urban housing, weapon systems te

community development, military procure

contracts, influences on the Brazilian expo

balances, how programs aimed at improvin

quality are working, and the implementatio

grants—to name only a few.

Most of these case studies are either “illus

“critical” instance applications. The first ty

Page 8 GAO/PEMD-91-10.1.9 Case Stud
application illustrates findings established 

techniques, supplementing, for example, n

findings on clean air from administrative r

other sources, with in-depth description on 

have been used and with what results in se

cities. The second type of application is in

analysis of a case of unique interest, such 

funds have been awarded and managed pr

specific community health center or if a ce

former government official had done anyth

improper before or after leaving the gover

There are, however, four other application

studies that are less often used at present 

could be appropriate for our jobs. In brief, 

types of case study, which we examine in 

are as follows:

1. Illustrative. This case study is descriptiv

character

and intended to add realism and in-depth e

other information about a program or poli

2. Exploratory. This is also a descriptive ca

but is aimed at generating hypotheses for l

investigation rather than illustrating.

3. Critical instance. This examines a single 

unique interest or serves as a critical test o

assertion about a program, problem, or str

4. Program implementation. This case stud

investigates operations, often at several sit

often normatively.

5. Program effects. This application uses th

study to examine causality and usually inv

multisite, multimethod assessments.

Page 9 GAO/PEMD-91-10.1.9 Case Stud
6. Cumulative. This brings together finding

many case studies to answer an evaluation 

whether descriptive, normative, or cause-a

Case Study Evaluations is a review of meth

issues involved in using case study evaluat

not a detailed guide to case study design. I

however, explain the similarities and diffe

among the six kinds of case study and disc

for successfully designing them. It also giv

to the manager who, in reviewing complet

studies, wants to assess their strengths. Fi

presents an evaluation perspective on case 

defining them and determining their appro

in terms of the type of evaluation question 

The methods and types of case studies out

are not definitive. The case study as a rese

method has evolved over many years of ex

but evaluative use of the method has been 

limited. Indeed, the history of the case stu

evaluation method is little older than a dec

Therefore, discussion of some of the appli

described here is based on relatively exten

experience (with questions in such domain

justice, education, welfare, environment, h

foreign aid), while the discussion of some 

applications is based on more constrained

experience.

We have paid particular attention to the co

wisdom that case studies are always subje

nongeneralizable. In many uses of case stu

is no need to generalize. Nonetheless, we f

there are steps that can be taken to genera

case studies when this is desired. Howeve

not devote any particular emphasis to the 

idea that case studies are inexpensive to c

(issues of research management common 

Page 10 GAO/PEMD-91-10.1.9 Case Stud
designs were outside the scope of our wor

However, one thing that should emerge qu

from the discussion of design features intr

case study is that it can be a rather costly 

given the time required, the rich in-depth n

information sought, and the need to achiev

credibility. This reinforces the importance 

weighing carefully the decisions to employ 

study method in program evaluation.

In this paper, we have taken positions on m

expecting to revise these as experience ac

and as we receive reactions from evaluato

researchers. This paper is intended to tran

we believe to be good practice in case stud

help establish the principles of applying ca

to evaluation. Thus, while the document o

preliminary guidance, it is also a point of d

For example, we are developing the variat

call the “cumulative” case study. It can ent

prospective and retrospective designs and 

synthesis of many individual case studies u

at different times and in different sites.

The quality of case studies can be variable

score high on reasonable tests of quality; o

lower scores. Three problems often encou

to do with matching the question the evalu

to answer and the method for selecting the 

examined, reporting the basis for selecting 

instances, and integrating findings across 

instances when the findings in one were in

with those in another.

The next sections of this paper will first pr

new ways of thinking about a familiar met

case study, and then introduce the six app

describing what is required, in terms of me

to get the benefits case studies can offer. I

Page 11 GAO/PEMD-91-10.1.9 Case Stud
chapter, we turn to two basic questions: W

need to take into account with regard to th

objectivity of case studies and their genera

Page 12 GAO/PEMD-91-10.1.9 Case Stud
Almost everyone in GAO probably has wor

case study at one time or another yet may 

unfamiliar with what is meant, methodolo

case study. The methodological meaning i

in understanding what differentiates a cas

from a noncase study and a good case stud

not-so-good case study.

What is a case study? The exercise in table 

describes a job we might be asked to do an

for it and asks you to decide whether or no

case study. Take about 10 minutes to think 

this example and write out your answer. It 

important that you try this out yourself, so 

it before continuing.

Table 2.1: What Is a Case

Study? Exercise. Item Writing assignment

Exercise Suppose GAO has b

whether the informed 

requirements for exp

with human subjects 

properly implemente

further that we visit t

where humans are u

subjects for research

hospital, a university

clinic— and that we 

informed consent pr

each site.

Question 1 Is this an application 

study method? Why?

Question 2 If not, would case st

appropriate for answ

question we were as

Question 3 What is your definitio

study”?

Page 13 GAO/PEMD-91-10.1.9 Case Stud
The answers some GAO evaluators gave m

illustrate the range of definitions surround

study methods.

To some GAO evaluators, the instance was 

application of the case study method, beca

were looking at only a few sites or becaus

not generalize or because “actual subjects 

used for analysis of a specific question.” T

instance was clearly not an application of 

study method, because “we do not know if 

instances are representative of the univers

“there doesn’t appear to be enough done a

To still others, it was not possible to tell w

was a case study because looking at instan

what we do in all our methods, and there w

differentiation between this job and a com

audit.

The definitions given also varied greatly. T

person, a case study involves looking at in

people. To another, a case study examines 

defined site and reports on that one site, s

multiple site studies would not be case stu

another, case studies involve getting a grea

information about a single site or circumst

generalizability isn’t important. To others, 

sample is necessary for a case study,” “cas

are nonnormative research that investigate 

without prejudice,” “where we could look 

number of cases that would represent the 

overall,” and “a review of relevant conditio

specific environment with no attempt to p

larger universe.” There were almost as ma

definitions as people, and few of them had 

in common. While exact uniformity isn’t ex

perhaps even possible when people are as

recall a definition, the extreme variability 

that we could be talking about very differe

Page 14 GAO/PEMD-91-10.1.9 Case Stud
a proposal or report when we discuss case 

methods. Thus a decision to “do case stud

lead to the collection of irreconcilably diss

information from groups working on the s

What Is Meant by

“a Case Study”?

We have developed a definition of case stu

leads to appropriate uses and says someth

how a good case study is conducted. It is s

technical, so we turn next to giving this de

to discussing each of its elements.

“A case study is a method for learning about a compl

based on a comprehensive understanding of that inst

by extensive description and analysis of that instance 

whole and in its context.”

For example, if we were asked to study wh

the Three Mile Island disaster and scoped 

describe whether required safeguards wer

with, this would not be a case study. If, ho

scoped the job to examine in depth events 

to the disaster, what went wrong, and why 

wrong, this would be a case study. For a se

example, if we were asked to study the saf

nuclear plants in general, we might select 

method a survey of self-reported complian

safeguards in all existing plants. This woul

case study. If, however, we scoped the job 

in depth recent problems in appropriately 

nuclear plants including among others Thr

Island, seeking to understand why the safe

either were not complied with or were not 

then we would have selected the case stud

to answer the question.

As we will discuss later, several methods c

in one job; these examples are only intend

highlight what is not, and what is, a case s

Page 15 GAO/PEMD-91-10.1.9 Case Stud
Examining the elements of the definition a

help make this distinction clear.

“A complex instance” means that input an

cannot be readily or very accurately relate

several reasons why such a relationship m

difficult. There could be many influences o

happening and these influences could inte

nonlinear ways such that a unit of change 

can be associated with quite different chan

output, sometimes increasing it, sometime

decreasing it, and sometimes having no dis

effect.

Table 2.2 gives an example of a less and a 

complex instance. “Are U.S. airports follow

required U.S. and international security pr

for passengers?” is a less complex question 

the criterion is fairly clear, the focus is nar

influences on compliance are likely to be r

few, and the relation of input and output is 

be fairly direct. Staff knowledge of proced

to play some role in following these proce

instance.

Some questions are more complex, howev

the question: “Are security procedures in U

sufficient to protect the safety of passenge

equipment?” This is more complex becaus

criterion of “sufficient protection” is much 

certain; the focus is broader; the influence

achievement of sufficient procedures are l

many; and the relation of input and output 

likely to be both direct and indirect but als

to measure.

The second key element in our definition i

comprehensive understanding.” Here the s

more straightforward. This means that the 

Page 16 GAO/PEMD-91-10.1.9 Case Stud
case study is to obtain as complete a pictu

possible of what is going on in an instance

The third key element, “obtained by exten

description and analysis,” has three compo

These are summarized in table 2.3. Case st

involve what methodologists call “thick” d

rich, full information that should come fro

data sources, particularly from firsthand

observations. The analysis also is extensiv

method compares information from differ

data sources through a technique called

“triangulation.” That is, reliability of the fin

developed through the multiple data sourc

each type. This is akin to corroboration as 

in the General Policy Manual, chapter 8.0. 

of the findings, particularly validity with re

Page 17 GAO/PEMD-91-10.1.9 Case Stud
Table 2.2: Complexity of

Questions Example Characteristic

A less complex question

Are U.S. airports following

required U.S. and

international security

procedures for

passengers?

Criterion is fairl

“required U.S. 

international se

procedures”

Focus is narrow

“passengers”

Influences on c

are likely to be 

few: staff know

procedures, sta

their implemen

functioning equ

number of staff 

to workflow, de

supervision, sta

and selection 

Relation of inpu

on compliance

(that required s

procedures are 

fairly direct

Page 18 GAO/PEMD-91-10.1.9 Case Stud
Example Characteristic

A more complex question

Are security procedures in

U.S. airports sufficient to

protect the safety of

passengers and

equipment?

Criterion is less 

would be suffic

present conditi

existing and po

technologies? 

broader: passe

equipment (alth

fairly well spec

Influences on a

of sufficient pro

likely to be man

the state of the 

detection techn

number and m

potential threat

and the willingn

passengers, ai

personnel, and 

personnel to ac

different costs 

protection

Relation of inpu

on security) an

(safety) likely to 

to measure and 

indirect and dir

Page 19 GAO/PEMD-91-10.1.9 Case Stud
Table 2.3: Methods of

Obtaining Description

and Analysis in Case

Studiesa

Technique Methodology

Extensive or “thick” analysis Analysis of mu

of data sources 

—Interviews w

relevant person

—Observations 

—Participant o

—Documents 

—Archives 

—Physical info

Analysis via triangulation of

data

Analysis throug

—Pattern matc

—Explanation 

—Thematic rev

Comparison of evidence for

consistency

Analysis throug

techniques suc

—Matrix of cate

—Graphic data 

—Tabulation of 

frequencies 

—Chronologica

series ordering

aDifferent types of evidence and standards for them 

discussed in General Policy Manual, chapter 8.0.

Page 20 GAO/PEMD-91-10.1.9 Case Stud
cause and effect, is derived from agreemen

the types of data sources, together with th

systematic ruling out of alternative explan

the explanation of “outlier” results. Exami

consistency of evidence across different ty

sources is akin to verification. There are sp

strategies for making these comparisons—

pattern matching, explanation building, an

review. The technical how-tos for these th

strategies will be summarized later in this 

involve techniques such as graphic data di

tabulations of event frequencies, and chro

time series orderings. Generally, data colle

analysis are concurrent and interactive—t

“yoked” in case study methods.

The next element of the definition is “take

whole.” As this list indicates, the size of th

can be as small as one individual or as larg

nation. The instance as a whole can be

• An individual (Ferdinand Marcos).

• A site (Three Mile Island).

• A function (joint test and evaluation progr

• An office (program evaluation groups in

departments).

• A department or agency (IRS, Census).

• An event (Cuban missile crisis1; Challenge

• A region, nation, or organization (Chesape

water cleanup efforts, democracy in Philip

UNESCO).

• “Nested” units in a large or complex case s

that the instance or unit must be specified 

appropriate to it collected).

One example of a GAO case study that exa

individual is our examination of whether o

1

These instances have been the subject of case studie

General Accounting Office, February 22, 1984, and A

Others are general illustrations.

Page 21 GAO/PEMD-91-10.1.9 Case Stud
senior official behaved improperly with re

influence and accepting money before and 

leaving the White House (U.S. General Acc

Office, July 11, 1986). Another example wo

request to examine in detail ex-President M

of funds intended by the United States for 

civilian purposes for his personal benefit. A

extreme, an instance may be as large as an 

such as the Cuban missile crisis (Allison, 1

the swine flu vaccine (Neustadt and Fineb

which have been the subjects of two well-k

studies, or the Challenger tragedy. It can b

(Chesapeake Bay water cleanup programs

(democracy in the Philippines), or an orga

(UNESCO). Moreover, it is possible to hav

that require nested case studies. For exam

answer a question about how programs to 

handicapped children are working, we mig

the cases of preschool and elementary pro

might further select within preschool prog

for the hearing impaired and those for the

orthopedically impaired. Each of these nes

is treated, in terms of specification of the u

and collection of data appropriate to it, as 

case study would be.

The last key element of the definition is “a

context.” Context means all factors that co

what is happening in an instance. As an ex

the Challenger tragedy, inquiry began with 

locate the technology that failed as the rea

explosion. The righthand booster rocket w

identified as the source of the explosion an

the rocket, technological attention focused 

O-rings. The inquiry expanded very quickly

from asking what technology failed to an e

of contextual influences, such as

Page 22 GAO/PEMD-91-10.1.9 Case Stud
• decisionmaking on whether or not to go, in 

the O-rings;

• decisionmaking on whether or not to go, in 

other components, such as tiles;

• decisionmaking more generally in NASA w

to NASA-contractor-astronaut relations an

responsibilities;

• influences on NASA, such as alleged press

cancel flights;

• quality control tradeoffs in NASA generally 

management.

That is, the Challenger inquiry could be se

similar to a case study in some ways. The r

spread of inquiry from an examination of t

technology to an investigation of decisionm

that flight, to inquiry about NASA managem

affected the Challenger disaster generally, 

“taking the context into account” means. I

study methods, to understand what happe

why, context always is considered, and it i

consideration that gives the case study its 

a way of understanding cause and effect.

Some Common

Benefits

Expected From

Case Study

Evaluations

Doing a good case study is more than just 

what is happening in a few instances. It is 

systematic way of looking at what is happe

selecting the instances, collecting the data

the information, and reporting the results.

There are nine features of case study evalu

merit special discussion. Each of these fea

carried out—confers certain benefits in te

product. Two of the features relate to desi

data collection, three to analysis, and one 

reporting. These features and their benefit

in table 2.4. For example, with regard to de

information over time—the longitudinal fe

Page 23 GAO/PEMD-91-10.1.9 Case Stud
design—provides assurance that the final 

represents what is happening and is not re

atypical situation.

Table 2.4: Some Common Benefits Expected From Case Study Evaluatio

Study feature Benefits expected

Design

Longitudinal Assurance that a short-term si

may be unrepresentative of wh

happening isn’t inflated in imp

Triangulation Assurance that reasons given 

properly reflect influences from 

different sources

Purposive instance Ability to match questions ask

generalization of findings at le

appropriate to the questions

Data collection

Comprehensive Assurance that important cond

consequences, and reasons fo

have not been overlooked

Flexible Broader perspectives, increas

assurance that what is importa

scene rather than centrally wil

examined

Multiple data sources Assurance that a full picture w

obtained and that bias associa

self-protection or self-interests 

reduced

Analysis

“Yoked” or concurrent with data

collections

Assurance of the ability to coll

needed to test alternative inter

and to make rapid adjustment

Search for disproving-proving evidence Assurance that alternative inte

have been thoroughly searche

checked; thorough identificatio

instances that don’t fit the gen

and, often, understanding of th

for the outliers

Page 24 GAO/PEMD-91-10.1.9 Case Stud
Study feature Benefits expected

Chain-of-evidence and pattern matching

techniques

Permit fairly direct assessmen

convincingly the evidence of c

are related

Reporting

Actual instances persuasiveness Assurance of authenticity thro

ease of recall; use of the tende

generalize from personal expe

via the substitution of more ob

experience for anecdotes of u

credibility

These features are the price of admission t

expected benefits. One frequent question a

study methods is how rigorously these fea

to be followed. Obviously, the more closel

requirements are followed, the more benef

expected. It is a judgment call as to how m

features can be compromised before the “

becomes a site visit or turns into a survey. 

the most critical features are appropriate i

selection, triangulation, and the search for 

evidence. And of these three, probably the 

critical is appropriate instance selection.

Instance

Selection in Case

Studies

There are three general bases for selecting 

convenience, purpose, and probability. Ea

function and can be used to answer certain 

A good case study will use a basis for insta

selection that is appropriate for the questi

answered. Using the wrong basis for selec

instance is a fatal error in case study desig

designs. Such a case study is a not-good ca

and it is irredeemably flawed despite any

methodological virtues it may have in term

collection, analysis, and reporting.

Page 25 GAO/PEMD-91-10.1.9 Case Stud
Table 2.5 summarizes the three general ba

selecting instances and the questions each 

answer. Of particular interest may be the s

varieties of purposive site selection: brack

cases, worst cases, cluster, representative

and special interest.

Instance selection is crucial to generalizab

answering the evaluation questions approp

Only rarely will convenience be a sound ba

instance selection; only rarely will probabi

sampling be feasible. Thus, instance select

basis of the purpose of the study is the mo

appropriate method in many designs.

Page 26 GAO/PEMD-91-10.1.9 Case Stud
Table 2.5: Instance

Selection in Case Studies

Selection basis

When to use and w

questions it can an

Convenience “In this site, selected 

was expedient for da

purposes, what is ha

and why”

Purpose

Bracketing “What is happening 

What explains such 

Best cases “What accounts for a

program?”

Worst cases “Why isn’t the progra

Cluster “How do diffe

of programs compar

other?”

Representative “In in

chosen to represent 

variations, what is th

like and why?”

Typical “In a typical 

happening and why?

Special interest “In t

circumstance, what 

and why?”

Probability “What is happening 

program as a whole, 

The match between the question asked an

method of purposive sampling chosen can 

For example, studies that attain “represen

by conducting a few case studies in a rural 

few in a suburban setting, and a few in an 

setting will produce a report in which the t

settings receive more or less equal weight. 

however, 90 percent of the clients or sites 

program are rural, such “representativenes

appropriately capture the range of site exp

Page 27 GAO/PEMD-91-10.1.9 Case Stud
but be rather unrepresentative of the prog

whole, and care will be needed to generali

the range of settings and not to the progra

whole.

Page 28 GAO/PEMD-91-10.1.9 Case Stud
Page 29 GAO/PEMD-91-10.1.9 Case Stud
Table 2.6: Hypothetical Data on Instance Selection

Location Operated by Number of beds Cliente

1. San Diego, Calif. CAIM, Inc. 800 Men an

2. Amarillo, Tex. CAIM, Inc. 130 Men an

3. El Paso, Tex. PIC 75 Families

4. El Paso, Tex. CAIM, Inc. 350 Men an

5. Miami, Fla. Security 100 Men an

6. Clearwater, Fla. CAIM, Inc. 300 Men an

7. Pensacola, Fla. Security 100 Families

8. Denver, Colo. PIC 100 Families

9. Salida, Colo. Security 200 Men an

10. Salinas, Calif. CAIM, Inc. 100 Men an

11. Los Angeles,

Calif.

Security 300 Men an

12. San Francisco,

Calif.

Security 250 Men an

13. San Francisco,

Calif.

PIC 100 Men an

14. New York, N.Y. ARIVA, Inc. 100 Men an

15. Washington, D.C. ARIVA, Inc. 300 Families

16. Seattle, Wash. Security 100 Men an

Page 30 GAO/PEMD-91-10.1.9 Case Stud
Years in operation Funded by Costsa 

2 INS 25 

1 INS 30 

3 INS 15 

1 BOP/INS 60 

1 BOP/INS 150 

5 BOP/INS 100 

5 INS/State 70 

3 INS/State 20 

4 INS 70 

2 INS 30 

3 INS 75 

3 INS/State 70 

3 INS 25 

2 INS 55 

2 INS 85 

3 INS/State 60 

aCosts per person per day, charged by contractor t

(hypothetical data).

bProblem rates include all problems considered und

as serious, such as escape, acts of violence by or t

individuals, vandalism requiring more than $1,000 t

and suicides. Rates are number of such instances p

days per year (hypothetical data).

To illustrate what each variety means, and 

might be operationalized, consider the info

table 2.6. This gives hypothetical data abou

situation in designing a study—selecting in

Page 31 GAO/PEMD-91-10.1.9 Case Stud
this study, sites or locations) for an assess

costs and operations of federal detention f

managed by private contractors under OM

A-76. There are not many such facilities—s

hypothetical facilities represent what we m

actually find in such a study. The following

paragraphs describe what a sample would 

it were chosen according to the bases in ta

Convenience

Samples

If our location were the Denver Regional O

convenience sample would be sites 8 (Den

(Salida). That is, ease of collecting data an

minimizing resources required would have 

choice.

Purposive Sample

Bracketing If our interests were extreme costs, numbe

Paso, at $15 per person day) and 5 (Miami

person day) would bracket the cost extrem

wanted the three least expensive and the t

expensive, we could select 3 (El Paso), 8 (

$20), and 13 (San Francisco, at $25) in com

5 (Miami, at $150), 6 (Clearwater, at $100)

(Washington, D.C., at $85). Such an additio

also give us a better basis for analysis beca

includes not only high-cost and low-cost si

services to men and boys and to families, a 

that in itself might be expected to lead to c

variations.

Best Cases If our interests were in operating centers w

least problems, we might examine number

(Denver, 3 percent) and 10 (Salinas, 3 perc

both are in Colorado (although operated b

firms and serving different groups), we mi

add sites. Such an addition could show wh

Page 32 GAO/PEMD-91-10.1.9 Case Stud
were looking at something about Colorado 

than about low-problem centers. We could 

selecting 1 (San Diego, 4 percent), 2 (Ama

4 percent), and 13 (San Francisco, 14 perc

Worst Cases Sites 5 (Miami, 15 percent problems) and 6

(Clearwater, 10 percent) stand out as wors

Selecting an out-of-state comparison, if we 

is harder here. The next highest problem r

Salida, at 9 percent) is run by a different c

costs much less. Security has a site in San 

for men and boys, which costs $70 daily w

7-percent problem rate. The costs of site 1

(Washington, D.C.) are higher, but this site 

families and has a low problem rate. The b

probably is 12 (San Francisco): it serves th

group (men and boys) and is run by the sa

company (Security).

Cluster We might be interested in administrative

arrangements—in, for example, how admi

works out when INS alone is the contracto

responsibility is shared with another feder

(Bureau of Prisons), and when responsibil

shared with the state. One cluster of sites 

10, 11, 13, 14, and 15) is administered by IN

Another cluster (4, 5, and 6) is shared betw

and INS, and the last cluster (7, 8, 12, and 

INS and the state. We could pick one or tw

from each cluster to get a sense of how ag

auspices may affect program operations.

Representative One issue we might need to examine could 

efficiencies of operation—particularly in t

facility size. Here we might select numbers 

Diego, 800 beds), 6 (Clearwater, 300 beds)

(Salinas, 100 beds). All are run by CAIM, a

men and boys. We would have to limit our

generalizations to facilities for men and bo

Page 33 GAO/PEMD-91-10.1.9 Case Stud
these three sites should give a good sense 

and operations issue.

Typical This would be a challenge. In terms of size

“typical” bed size (100 beds); in terms of p

served, there is a “typical” population (me

and in terms of years of operation, 3 years 

with 2 years a close runner-up. In terms of 

however, the distribution is trimodal—tha

values appear about equally often—and fo

problems, it is almost flat with two outlier

there is not a single site that matches all th

“typical” characteristics well. Miami, for ex

100 beds and serves men and boys, but it h

operation only 1 year, costs $150 per perso

and has a 15-percent problem rate. The be

would be to indicate that it is not possible 

site that is “typical” of such distributions.

Special Interest Any one of the 16 sites might be examined 

of special congressional interest. Such inte

would be based on information extraneou

in the table: a complaint might be received

example, about conditions in the San Dieg

allegations might be made that the high co

Miami site were due to mismanagement.

Probability Samples Probabilistic sampling is the method of ch

answering questions about “how much,” o

extensive a problem is in a population. Pro

carried out, it provides strong generalizabi

assurance of representativeness. A probab

is one in which all members of the populat

known and equal chance of being selected

a table of random numbers, and selected a

two sites those corresponding to the first t

numbers between 1 and 16 in the table, we 

have selected a probability sample. Each s

Page 34 GAO/PEMD-91-10.1.9 Case Stud
have a 1-in-16 chance of selection, and tha

would be equal among sites. A fair objectio

statement is that the laws of probability op

large numbers, and selecting fewer than 30 

does not always provide the generalizabili

population as a whole that probability sam

promise. However, in terms of actual oper

which we want to illustrate here, the meth

sketched is a probabilistic one, and some c

have involved 30 or more sites selected on 

probabilistic basis. (See PEMD’s transfer p

entitled Using Statistical Sampling (U.S. G

Accounting Office, May 15, 1986) for more

information.)

For readers who want to check out their s

applying different types of purposive selec

appendix II gives information for a job inv

50 states (a fairly common situation for GA

for indicating which you would select for e

seven kinds of purposive selection, and ou

for comparison against yours.

In many jobs, what is a “case” and what di

are important to consider in selection will 

For example, the population of detention f

supported by INS contracts can be defined 

the contract awarded), and the relevant di

(length of time in operation, facility size, d

mix) are straightforward. There are, howe

problematic circumstances. An example w

study of the extent to which voluntary org

have taken up any slack in welfare suppor

voluntary organization can be defined broa

“any nonprofit organization,” or narrowly, 

service-oriented group whose members do 

receive payment for their work.”

Page 35 GAO/PEMD-91-10.1.9 Case Stud
Dimensions of potential relevance for the 

interest are many, and the empirical basis 

selecting any one dimension over others fe

situations, the evaluator can turn to past e

a search of the appropriate theoretical as w

empirical literature, the advice of knowled

persons, an examination of key issues in p

pending legislation, customer guidance, an

techniques. That is, while it is important to 

the difficulties, there are ways of dealing w

case definition.

Page 36 GAO/PEMD-91-10.1.9 Case Stud
As noted earlier, there are six types of app

for case study methods—illustrative, explo

critical instance, implementation, program 

and cumulative. But case study reports co

only two of the six applications: illustrativ

critical instance. Greater use could be mad

four others in selecting alternative ways of 

questions, because these may be able to gi

information that is more valuable to custo

other techniques. Also, improvements can 

made in how even the two approaches alre

frequently are carried out, especially in the 

selecting instances for study. The next sec

summarize, for each of the six types, the e

questions they can answer, the functions t

perform, their design features, and their pi

last section shows what basis for selecting 

appropriate for each of the six application

Illustrative As table 3.1 indicates, illustrative case stud

primarily describe what is happening and w

or two instances, to show what a situation 

can help in the interpretation of other data

particularly if we have reason to believe m

know too little about a program or situatio

understand fully the information from surv

other methods.

Page 37 GAO/PEMD-91-10.1.9 Case Stud
Table 3.1: Illustrative

Case Studies Aspect examined Characteristics

Evaluation

questions

Help interpret other data 

is reason to believe that 

know too little about a pr

descriptive, often used in 

with other methods

Functions Make the unfamiliar fami

surrogate experience; av

over-simplification of rea

reader a common langua

the topic

Design features Site selected as typical o

representative of importa

small number of cases to 

reader’s interest; data of

visual evidence; analysis 

with data quality and me

reports use self-containe

narratives or descriptions

Pitfalls May be difficult to hold re

interest while presenting 

information on each illust

not adequately represen

where considerable dive

(in such situations, it may 

impossible to represent v

enough to use illustrative 

studies); and may not ha

on-site for in- depth exam

GAO has many examples of such illustrativ

1982, for instance, CED examined housing 

grants through a survey supplemented by 

studies. The results of the survey were pub

the main report (U.S. General Accounting 

December 13, 1982). For three of the sites 

Seattle, and Dallas), individual reports des

each city was like with regard to housing a

housing-related activities and how the mon

Page 38 GAO/PEMD-91-10.1.9 Case Stud
used in that city and included before-and-a

pictures of what rehabilitation meant for in

neighborhoods and houses (U.S. General A

Office, March 24, 1982; March 30, 1982; Ap

1982). In a similar application, HRD descri

projects funded under the Emergency Job

Appropriations Act of 1983 in communitie

Alabama, California, Georgia, and Massach

(U.S. General Accounting Office, March 26

August 27, 1985; September 25, 1985; Dece

1985).

Illustrative case studies are used by evalua

other agencies. When the Department of H

Human Services was trying out delivery of 

services to parents and children in their ow

called Home Start, the Department supple

formal assessment of the development of t

before and after the program with case stu

(High/Scope Educational Research Found

These case studies described what service

delivered, the conditions in rural as well a

areas, and what the Home Start teachers d

the home visits and generally provided a s

vicarious experience for readers who migh

have visited a Head Start or a Home Start c

case studies told, too, of the development 

program over time and helped give a realis

problems in start-up and implementation, 

changes in staffing were accommodated, a

impact of shifting federal guidance on effo

out the program in the field.

Case studies such as these are well accept

valid way of amplifying a more systematic

presentation via the realism and vividness 

anecdotal information. There are, however

presenting illustrative case studies. The m

is selecting the instances. The case or case

Page 39 GAO/PEMD-91-10.1.9 Case Stud
adequately represent the situation or prog

relatively easy if the program is small and

homogeneous. Where considerable diversi

may not be possible to select a “typical” si

diversity may be so great that to represent 

adequately would require more case studie

most people would want to read for illustr

purposes. In the example of privately oper

detention facilities, an illustrative case stu

run the risk of oversimplifying a more com

situation. The example was contrived to il

exactly this point: that sometimes we cann

site that fits our needs and thus the metho

appropriate.

However, in many real-world situations, it 

to represent diversity adequately for illustr

purposes and to obtain the benefits of this

application: helping readers feel, hear, see

when this kind of surrogate site experienc

necessary to undo stereotypes or explain a 

otherwise inaccessible for most people.

Such a situation might be a bilingual educa

about which stereotypes can abound, or li

nuclear-weapon-equipped submarine, a sit

readers will ever experience themselves b

to get a feel for in order to understand staf

training, and management on modern subm

Exploratory The exploratory case study is a shortened 

undertaken before launching into a large-s

investigation. Its function is to develop the 

questions, measures, designs, and analytic 

the bigger study. As table 3.2 indicates, it i

helpful where considerable uncertainty ex

program operations, goals, and results. Als

than initiate a job requiring 1,000 staff day

Page 40 GAO/PEMD-91-10.1.9 Case Stud
when we do not have an adequate on-the-s

designs and measures, an exploratory case 

save time and money in implementation as 

improving the confidence we have in our r

can aim more precisely and hit the target m

Page 41 GAO/PEMD-91-10.1.9 Case Stud
Table 3.2: Exploratory

Case Studies Aspect examined Characteristic

Evaluation

questions

Usually cause and effect

Functions Where considerable unc

exists about program op

goals, and results, explo

studies help identify que

important measurement 

develop actual measures 

which can be used later 

larger-scale tests; formu

expectations; safeguard 

in larger studies (for prob

programs that are not we

Design features Site selected: needs at le

that represents each imp

variation to make a conv

sample acceptable; num

sufficient to cover divers

focus on program operat

on-site observation, are n

longitudinal but need eno

find out what is going on

closely concurrent with f

does not require strong c

evidence or audit trail; re

usually internal or parts o

longer reports

Pitfalls Temptation to prolong th

phase; site selection only 

convenience, inadequate 

of diversity; prematurity —

findings released as con

over-involvement in evalu

hunches so that initial fin

confirmed rather than tes

Some of our scoping work already may inv

exploratory case studies. For example, in 

design study was done as a separate job, c

Page 42 GAO/PEMD-91-10.1.9 Case Stud
in a briefing, prior to an in-depth study of t

implementation of the Bail Reform Act of 

methodology included 90 interviews, obse

and data analysis from the population of 9

districts selected purposively for their cha

on significant variables. Researchers and e

the field were also interviewed. An expert 

used to give feedback at various points to 

we had a comprehensive picture of the situ

product of this exploratory case study was 

with the study design choices described, in

detailed research questions, outlines of da

significant variables, extant data bases, an

selection criteria. From this, a larger study 

designed to meet the needs of the requeste

jobs may involve similar efforts that are no

reported as separate jobs and thus are less 

exploratory case studies.

Also reports that include some features of 

case studies have been issued by GAO. In 

example, NSIAD examined emerging issue

competition through a case study of the Br

market (U.S. General Accounting Office,

September 26, 1985). Combining site visits 

Japan, West Germany, and France, intervie

many officials of appropriate agencies and 

private sector, examination of official gove

files, and a questionnaire survey of high te

firms active in the Brazilian market, the ev

amassed a rich array of contextual and foc

information and identified four trade pract

considered to be key factors in export

competitiveness in Brazilian markets. The

bilateral trade accords, countertrade, expo

financing, and compliance with trade-relat

industrial policy. Although to meet the req

of the job, NSIAD did not need to test thes

for generalizability to other countries thro

Page 43 GAO/PEMD-91-10.1.9 Case Stud
study, the product would permit such testi

is using the findings in this way, as part of 

work on bilateral initiatives. Of particular

methodological note in this report is the d

explanation of why export competitivenes

Brazilian markets (the instance) was selec

case study.

The exploratory case study has been used 

outside GAO. The Department of Justice, f

supported an exploratory case study of the 

criminal program (Chelimsky and Dahman

The career criminal program aimed at “sw

certain” justice by trying to expedite and s

processing of individuals who had long cri

histories at the time of apprehension. The 

study looked in depth at four of the nine

demonstration sites prior to conducting a 

effects evaluation. The evaluators identifie

elements of the programs as implemented 

measurable changes were likely to occur a

developed measures of the outcomes, as w

designs for testing cause and effect in the 

larger study (Chelimsky and Sasfy, 1976).

The greatest pitfall in the exploratory stud

prematurity: that is, the findings may seem 

convincing that it can be difficult to resist 

to report on these as if they had the streng

larger study. Also, care must be taken to s

sequence the exploratory study so that it y

enough information to be worthwhile and 

use in the larger study but does not unduly 

answering the questions through the large

addition, it is inappropriate to use the scop

as an ad hoc exploratory case study accom

an urge to issue the product at the end of s

when the necessary procedures for an exp

Page 44 GAO/PEMD-91-10.1.9 Case Stud
case study with regard to such issues as in

selection have not been followed.

Critical Instance The critical instance is the most frequent a

of the case study method in GAO, so much 

may be seen as a “usual GAO review” rath

recognized as what it can be—a case study 

General Accounting Office, January 22, 19

1982; October 30, 1985). The advantage of 

the approach as an application of case stu

is that some aspects of the method—such 

yoking of data collection and analysis—th

be widely used now could be applied in a w

increases timeliness without reducing qua

technique, discussed in more detail in the 

analysis, can increase efficiency by reduci

collection of data and large-scale analyses 

data that subsequently do not prove useful

The critical instance case study examines 

few, sites for one of two purposes. First, a 

frequent application is the examination of 

of unique interest, such as Three Mile Islan

Challenger disaster, or allegations concern

for a specific presidential campaign. There 

no interest in generalizability. The instanc

“selected” by us; rather, we are called to it

GAO conducts many critical instance stud

example, already mentioned, was our revie

representation of foreign interests by form

high government officials (U.S. General Ac

Office, July 11, 1986). Another is PEMD’s r

the readiness of the Big Eye Bomb for pro

(U.S. General Accounting Office, May 23, 1

another is RCED’s review of a constructio

award at Jean Lafitte National Historical P

General Accounting Office, September 26, 

Page 45 GAO/PEMD-91-10.1.9 Case Stud
their examination in a separate report of th

service actions at Delaware Water Gap Na

Recreation area in awarding a lease, closin

ground, and raising a house rent (U.S. Gen

Accounting Office, October 28, 1987).

A second, rare, application is where a high

generalized or universal assertion is being 

question, and we are able to test it through 

one instance.

In one such study, GGD examined whethe

policies, procedures, and practices with re

cargo imports were causing problems in p

operations (U.S. General Accounting Offic

December 1986). The Port of New York of

critical test because, given the diversity of 

and the volume of work, if problems were 

they would be likely to show up clearly in 

no problems were observed, problems in o

were unlikely. GGD used observations, int

and document analysis at three sites in the 

New York and supplemented these with a 

number of less intensive observations at o

The method, in this instance, was sufficien

recommendations that were systemwide a

generalizable with the single case.

Table 3.3 summarizes the features of the c

instance case study. As noted, the method 

particularly suited for answering cause-an

questions about the instance of concern. It 

assurance that we have not prematurely ov

important factors, that we have not been s

information from limited or perhaps biase

and that we have taken context into accou

giving a fair and balanced picture of the sit

Page 46 GAO/PEMD-91-10.1.9 Case Stud
Perhaps the biggest pitfall in this applicati

insufficient specification of the customer’s 

That is, the job may be presented to us as 

situation is of concern, but the underlying 

may call for a broader look at the issue. A 

investigate the reasons for the bank failure

for example, may reflect an interest only in 

it could be a “tip of the iceberg” question. 

customer may really want to know is whet

states are likely to have similar problems. 

situation, Ohio might be selected as a site 

but we would also need to look at other st

other approaches to achieve the generaliza

needed. This then rules out the critical ins

method as appropriate for this job. The im

probing the underlying questions in a requ

achieve good specification of the evaluatio

is not unique, of course, to the critical inst

study but it is crucial in its appropriate app

Page 47 GAO/PEMD-91-10.1.9 Case Stud
Table 3.3: Critical

Instance Case Studies Aspect examined Characteristic

Evaluation

questions

Cause and effect, usually 

Functions Investigation of specific 

(frequently encountered 

decisive testing of univer

assertion; cause-and-effe

Design features Site selects itself in spec

problem—for decisive te

to assume uniform syste

regard to issue and so c

sample acceptable; num

is usually one instance;

comprehensive data for 

problem—for decisive te

more modeling, hypothe

targeting to know what to 

analysis and collection c

and interactive: data fee

collection, and emphasis 

out alternative causes; re

describes instances, pre

conclusions about cause

evidence

Pitfalls Inappropriate selection o

technique as real issue m

specific problem (e.g., O

failure) but more general 

premature closure may n

causal search too early;

overgeneralization from e

Program

Implementation

We frequently are asked whether a program 

implemented and, often, whether impleme

compliance with congressional intent. The 

implementation case study is helpful wher

legislation offers considerable flexibility. I

cases, a wide variety of expenditures or ac

be consistent with legislation and complia

Page 48 GAO/PEMD-91-10.1.9 Case Stud
intent may be a matter of understanding th

by which decisions were made, who was in

and whether the actions are meeting local 

example is the 1981 legislation consolidati

small categorical grants into larger block g

funds for which could be spent very flexib

Another situation where program impleme

case studies may be called for is when con

about implementation problems. In-depth,

longitudinal reports of what has happened 

and why can set a context for interpreting 

implementation variability: that is, whethe

seem to be basic structural problems or if 

program understandably requires time for 

adaptations, and building an infrastructure

In some instances, GAO has been able to f

intensively the implementation of program

activities. One example is GGD’s series of 

how the 1980 census was conducted. GAO 

in addition to being “on the scene” due to t

location at the major audit site accompani

enumerators into the field and examined, i

Census procedures at field offices. In othe

we have spent somewhat less elapsed time 

field, with less direct observation, and with 

reliance on interview and documentary ev

1985, for example, RCED was asked how t

Department of Interior was implementing 

of Management and Budget’s Circular A-76

with privatization of all appropriate servic

request overlapped with another similar re

request reflected a senator’s special intere

Glacier National Park in Montana. The eva

were able to combine the jobs in a review 

eventually involved information from 8 of 

Park Service regional offices and 19 of 402 

offices. The report aggregates findings acr

Page 49 GAO/PEMD-91-10.1.9 Case Stud
sites and concludes that agencies have bee

implement the circular, although progress 

made since 1982 (U.S. General Accounting 

March 15, 1985).

Another example is GAO’s review of 23 fed

agencies’ efforts to implement the Federal 

Financial Integrity Act of 1982. A series of 

studies, together with an overview report, 

produced. Among these, RCED’s review of 

Department of Commerce implementation

one report, examined the actions Commer

were intended to improve internal control

training senior financial analysts in evalua

applicants and borrowers in the troubled E

business loan program and overhauling th

which computer resources were used for t

Weather Service. RCED also examined the 

these efforts and highlighted priority areas 

improvement, such as better information o

for internal management purposes.

Table 3.4 summarizes the design, data coll

analysis, and reporting features of program

implementation case studies. Usually, in su

generalization is wanted and care is requir

negotiate the question with the customer (

situations? worst? typical?) and to match i

selection carefully with the questions. Unl

program is small and homogeneous, the ev

faces two possibilities. The first possibility 

number of instances will need to be fairly 

order to achieve the generalizability wante

consequence, skill will be needed to mana

collection with sufficient flexibility to obta

insights case studies offer and sufficient st

permit cross-site aggregation of findings. T

possibility is that the diversity will be so gr

would be impossible to have enough instan

Page 50 GAO/PEMD-91-10.1.9 Case Stud
meet needs for generalizability and still ma

data collection and analysis.

Table 3.4: Program

Implementation Case

Studies

Aspect examined Characteristic

Evaluation

questions

Descriptive, normative

Functions Learn what implementati

achieved, understand un

aspects; understand rea

implementation looks the 

useful when enabling leg

given flexibility

Design features Site selection cannot be 

because usually general

wanted, and purposive s

be typical and represent

diversity and best and w

number of cases depend

program diversity since

generalization usually wa

rely on common instrume

published documents, a

observation; reports are 

theme, site, chronology, 

Pitfalls Bias detection methods 

inadequate; may fail to ta

account diverse views ab

program goals and purp

competence of all on-site 

may not be sufficiently h

costly due to study size; 

demands of data manag

quality control, validation 

and analytic model (with

site, etc.) may lead to cu

many corners to maintain 

An important requirement for good progra

implementation case studies is investment 

time on site to get longitudinal data and to 

Page 51 GAO/PEMD-91-10.1.9 Case Stud
breadth of information. If the purpose is to 

what is happening in a descriptive sense o

site visits together with administrative rec

provide adequate bases for findings. If, how

evaluation question requires GAO to repor

satisfactory progress is or the reasons for 

implementation, the more staff who can be 

over time, with the richest or “thickest” ba

examining the situation as the many peopl

see it, the sounder our causal conclusions 

subsequent recommendations will be.

The multiple sites usually required for pro

implementation questions impose demand

training and supervision needed for quality 

Because of tight resources, lack of travel f

the need to use staff with uneven experien

skills, this becomes critical in situations in

many evaluators working in different regio

time is needed to train staff adequately in 

study techniques as the note-taking require

descriptions, which is in turn required for 

analysis of themes in the instance. It is pos

example, for two persons to interview the 

informant and find that one has used a one

summary for a detailed, rich, 5-minute disc

while the other captured much more of the

complexity and essence of what was said a

was happening. Table 3.5 illustrates such a 

Page 52 GAO/PEMD-91-10.1.9 Case Stud
Table 3.5: Illustration of Differences in Note-Taking

Situation Technique Characteristic

In an interview with the Director of

the National Science Foundation

program for grants to small

colleges, the following question is

asked: “How does your program

inform the eligible colleges of the

opportunity to apply for grants?”

Rich notes “The Director indicated t

procedures had change

times since the inception 

program. In the first 4 ye

announcements were m

individual named as pre

listing, for the same yea

American Association of 

Colleges. Because appl

very sparse, with about 

eligible colleges applyin

procedure was changed 

two-stage mailing, first to 

president to find out the 

official in charge of fede

and then to the official. T

well for a 5-year period, 

receipt of applications fr

80% of the eligible colle

when overall federal fun

research was reduced, t

of federal program coor

were abolished and app

to about 40% of eligible 

responding. Two years a

decision was made to m

the persons listed as ch

relevant science college 

appropriate professiona

listings. This has increas

of outreach by about $1

about 25% more than th

system. To date, returns 

80% rate again.”

Thin notes “The current system is to 

of the announcements to 

of relevant science depa

such as chemistry, biolo

and computer science.”

Page 53 GAO/PEMD-91-10.1.9 Case Stud
Program Effects Case studies can determine the effects of p

and reasons for success (or failures). In 19

example, RCED examined the progress ma

the 1970’s in cleaning up the nation’s air, w

land, finding that while strides had been m

meeting the established goals (cleaner air, 

treated wastewater, more drinkable water

had been extended and unresolved issues 

meeting even these deadlines difficult (U.S

Accounting Office, July 21, 1982). We poin

of flexibility as a source of cascading prob

delays. The bases for these conclusions we

case studies of three sites (Cleveland, Dall

New York City) together with information 

reports prepared by six federal agencies an

environmental organizations and public in

groups and interviews with Environmenta

Agency officials. Particularly notable

methodologically in this report is the integ

case study findings with other sources of i

throughout the first volume.

A PEMD report has focused on water qual

effectiveness of efforts to improve water q

the reasons for successes and failures. In-d

extensive case studies of several water cat

areas were conducted, and the final report 

a synthesis of the findings from the case

studies—another example of integration o

across diverse sites (U.S. General Account

December 17, 1986a, b; September 19, 198

series of reports also is useful for illustrati

in which causality is established in case st

through development of internally consist

explanations of what led to what and the

conscientious use of information from wit

and from contrasting sites to rule out alter

explanations.

Page 54 GAO/PEMD-91-10.1.9 Case Stud
For another example, to determine whethe

taken by the states since the mid-1970’s to 

medical malpractice insurance reduced in

costs, the number of claims filed, and the a

amount paid per claim, HRD conducted ca

in six selected states (Arkansas, California

Indiana, New York, and California). Work 

obtaining views of organizations represent

physicians, hospitals, insurers, and lawyer

perceived problems, actions taken to deal 

results of these actions, and the need for f

involvement. Other information came from 

nonfederal hospitals about the sources, co

limits, and costs and claims from leading i

each state and, for comparison, the same t

information from a nationwide company. T

are presented separately in six case study 

aggregated in the overall report (U.S. Gene

Accounting Office, December 31, 1986).

Other federal agencies have used the case 

method successfully in answering program 

questions. The National Science Foundatio

example, assessed the effectiveness of a c

science program aimed at increasing innov

knowledge transfer between university an

researchers. Ten case studies were undert

carefully selected group of projects that ra

computer language systems through nucle

to fisheries biology and chemical engineer

is the methodological detail given on proje

selection, data collection, analysis, and ca

In a companion report, results from a surv

recipients are analyzed, giving both a quan

a qualitative sense of how the program wa

Results from the two methods were not in

both suggested, however, that the program 

generally working well (National Science F

1984).

Page 55 GAO/PEMD-91-10.1.9 Case Stud
Table 3.6 summarizes key features of prog

case studies. Like the program implementa

study, the evaluative question often requir

generalizability and, for a highly diverse pr

may not be possible to answer the questio

adequately and still have a manageable nu

sites.

Table 3.6: Program Effects Case Studies

Aspect examined Characteristic

Evaluation questions Cause and effect, can be stand alone or mult

and can be conducted before, during, or afte

methods

Functions Determine impact and give strong inference a

reasons for effects

Design features Site selection depends on program diversity, 

used with highly diverse programs; best, wor

representative, typical, or cluster bases appr

keep number of cases manageable or risk be

minisurvey, can use survey before or after to 

generalizability or mix survey with concurrent 

studies selected for special purposes; data re

observation and structured materials, often c

qualitative and quantitative data; analysis use

degrees of formalization around emergent or

predetermined themes; reports are usually th

describe site differences and explain these; v

degree of integration of data across sites and 

from different methods

Pitfalls Not collecting the right amount of data; not ex

right number of sites; insufficient supply of we

evaluators; difficulties in giving evaluators en

collection latitude to obtaining insight without 

There are some methodological solutions 

problem. One solution would be to conduc

studies first in a set of sites chosen for

representativeness and to verify the findin

case study through targeted examination o

Page 56 GAO/PEMD-91-10.1.9 Case Stud
administrative data, prior reports, or a sur

second solution would be to use these oth

first. After identifying the findings of partic

interest, case studies would be conducted 

selected to maximize the ability to get the 

understanding required. Both of these app

have been used with good effect in progra

evaluation.

Cumulative This relatively new and not as yet widely u

application of case study methods brings t

findings from case studies done at differen

The applications previously discussed that 

multisite case studies are cross-sectional: 

information from several sites is collected 

time. In contrast, the cumulative case stud

aggregates information from several sites 

different and even quite extended times.

The cumulative case study can be retrospe

aggregating information across studies don

past, or prospective, structuring a series o

investigations for different times in the fut

techniques for ensuring sufficient compara

quality and for aggregating the information 

constitute the “cumulative” part of the met

That is, the cumulative case study is simila

evaluation synthesis, in that it is a method 

aggregating the findings of several studies

from an evaluation synthesis in that specia

techniques are required to aggregate the qu

information that often is a feature of case 

to maintain the sense of the “instance as a 

its complexity that distinguishes case stud

surveys of several sites. For some jobs, bo

study and noncase study reports can be ag

Page 57 GAO/PEMD-91-10.1.9 Case Stud
each using the appropriate techniques, in o

produce capping reports or similar produc

GAO does not appear to have done a cumu

study using our own case study reports or 

studies. GAO reports have been used with 

results, however, in cumulative case studie

published by others outside GAO. One exa

book on bureaucratic failures, which is ba

on GAO reports of management problems 

agencies over a considerable period of tim

1981). The author began with a set of hunc

hypotheses about what can go wrong in ag

management, and what would be evidence

supporting—or contradicting—these hypo

reviewed the GAO reports in detail, analyz

from each one in terms of his framework, 

aggregated the results in his final chapter.

Other examples of cumulative case studies 

two international agencies. A retrospective

cumulative case study was conducted by t

Bank in its examination of four in-depth ca

of the effectiveness of educational program

case studies were intended initially as stan

assessments of the programs but were bro

together to learn about the effectiveness o

evaluations themselves in the context of e

programs (Searle, 1985). A prospective cum

case study was commissioned by the U.S. 

International Development. The purpose w

identify input and process components of 

assistance that could be quantitatively ass

with differences in outcome measures. Th

was the specification of a common set of d

qualitative and quantitative) to be collecte

5-year period as projects were initiated, to

a means of coding the data across the 47 s

eventually completed. The coded results w

Page 58 GAO/PEMD-91-10.1.9 Case Stud
analyzed quantitatively in the final report

(Finsterbush, 1984).

Table 3.7: Cumulative

Case Studies Aspect examined Characteristic

Evaluation questions Cause and effe

Functions Retrospective 

allows general

without cost an

conducting nu

case studies; p

cumulation als

generalization 

unmanageably 

numbers of ca

process at any 

strengthens in

new studies by 

with results fro

studies

Design features Uses site selec

usually a large 

cases; data as 

(retrospective)

on-site observ

(prospective); 

techniques; an

case survey m

cumulate findi

to examine inte

directly since n

instances is la

may resemble 

syntheses

Page 59 GAO/PEMD-91-10.1.9 Case Stud
Aspect examined Characteristic

Pitfalls Publication ba

severely limit

generalization

or uncertain qu

original data; q

data-reduction 

may be very d

determine; the 

changes in ma

contextual fact

time may be d

separate from 

programs

Two features of the cumulative case study

table 3.7, are the case survey method just d

a means of aggregating findings (Lucas, 19

Heald, 1975; Yin et al., 1976) and backfill te

(Berger, 1983). The latter are helpful in ret

cumulation as a means of obtaining inform

the authors that permits an otherwise unu

study to be included in the aggregation. Kn

basis on which the case instances were se

example, is crucial in cumulation; otherwi

possible to know whether best case, worst 

typical, or the like instances are being aggr

Some published case studies do not provid

detail on this. In backfilling, the evaluator 

the author, visit the author to review the o

or contact others who were knowledgeabl

design decisions in order to get adequate i

on instance selection.

Opinion varies as to the credibility of cum

studies for answering program implement

effects questions. One authority notes that 

biases may favor programs that seem to w

could lead to a misleadingly positive view 

1983). Other experts are concerned about 

Page 60 GAO/PEMD-91-10.1.9 Case Stud
of the original data and analyses and probl

verifying their quality (Hoaglin et al., 1982; 

For the cumulative use of GAO reports, th

concerns are less important, since we alre

“audit trail” procedures recommended in t

and other manuals for verification of data 

and analysis quality. We do, however, have 

opposite concern: that is, we would need t

there was not “bad news” selectivity in a p

area, associated with killing jobs that did n

problems during scoping.

Table 3.8: Some Design Decisions in Case Study Methods

Type of question

Design decision

Illustrative,

exploratory Critical instance

Implem

program 

cumula

Basis for site selection Typical,

representative,

cluster

Convenience,

unique interest

unique

Best-wo

bracket

represe

cluster, 

If multimethod Concurrent Concurrent Before, 

after

Prestructuring Low, moderate Low, moderate Modera

Type of data Qualitative only,

qualitativequantitative

Qualitative only,

qualitativequantitative,

Qualitat

qualitat

quantita

quantita

Sequence of analysis Within sites, then

across

Within sites, then

across

Within s

across; 

then wit

concurr

Reporting Narrative, thematic Narrative, thematic Themat

Page 61 GAO/PEMD-91-10.1.9 Case Stud
Design Decisions

and Case Study

Applications

In earlier sections, we discussed seven bas

purposive selection of instances and six ap

of the case study method, each of which w

associated with a different evaluation purp

question. Bringing this information togethe

shows the relations among case study app

and design decisions. For example, if the p

the study is illustrative, an appropriate bas

selection could be typical, representative, 

the case studies would be conducted conc

with other methods used in the main study

prestructuring or guidance to the evaluato

field would be low to moderate to permit t

thickness and richness of insights needed; 

be qualitative only or both qualitative and

quantitative; the case studies probably wo

analyzed within sites only; and the reportin

probably be narrative.

Page 62 GAO/PEMD-91-10.1.9 Case Stud
We have said that the features distinguishi

studies from other methods are how sites 

selected, how the data are collected, and h

analyzed. In the last chapter, we covered i

selection. We turn now to other elements t

distinguish a case study from a not-case st

good case study from a not-good case stud

discussion is an introduction to the approa

Data Collection In other transfer papers on program evalu

have emphasized the importance of validit

involves measurement and also design. A v

measure—that is, one with construct

validity—reflects what it claims to reflect 

something else. For example, whether or n

are active opposition parties may be a mor

measure of whether a country is a democr

how many people vote in an election. A va

cause-and-effect design—that is, one with 

validity—rules out alternative explanation

by comparing what happened with an inte

what happened in the absence of the interv

example, in a study of the effects of an em

training program, greater employment of p

after the training than before must be show

due to the training and not simply to bette

conditions, which also could increase emp

Measurement

Validity

Case study methods can use two tactics fo

measurement validity: multiple sources of 

and using the chain-of-evidence technique 

reduction.

Multiple Sources of

Evidence

Turning first to multiple data sources: case 

require “thick” description in order to get e

information to check for trends, to rule ou

Page 63 GAO/PEMD-91-10.1.9 Case Stud
explanations, and to corroborate findings. 

techniques are used—sometimes all of the

same study—to collect information (Neust

Fineberg, 1978; Yin, 1989).

1. Collect physical articles.

2. Collect documents such as contracts, m

reports.

3. Examine archives such as lists of person

computerized order records.

4. Conduct open-ended interviews.

5. Conduct focused interviews.

6. Conduct structured interviews and surv

7. Undertake direct observations.

8. Carry out participant observations.

Many of the eight techniques are discussed 

General Policy Manual, chapter 8.0. Of the

the approaches that most differentiate cas

from other techniques are direct observati

participant observation.

GAO has used both approaches in its jobs. 

example, in NSIAD’s study of conditions o

submarines, auditors spent time aboard su

a variety of situations, getting firsthand kn

life in these vessels. Their direct observati

the primary data source for our report. We 

sea in this instance, however, in our GAO r

auditors and evaluators and so—it could b

argued—might have seen what special gue

not what life would be like for the average 

Page 64 GAO/PEMD-91-10.1.9 Case Stud
To get more authentic information, evalua

sometimes become participants in situatio

identified to the other persons involved as 

One example of how we have adapted this

participant-observer approach was in GGD

the services available to taxpayers from IR

reduced the number of public information 

(U.S. General Accounting Office, April 5, 1

developed a set of standard income tax qu

about which citizens typically would call I

obtained IRS agreement on the correct ans

these questions, and then, on a probabilist

basis, called IRS offices around the countr

help. We used names such as Gerald A. Of

conversations but did not say we were fro

were able to report how long it took to get 

answered, how long it took to get informat

consistency of information, and general he

the responding agent. Such an approach ga

authentic information than relying only on 

records of calls received, or a survey of tax

the first instance, IRS would have no recor

before the person could get through to an 

of “discouraged callers.” In the second, a s

taxpayers would have to be very large to g

“hit” rate of individuals who sought assista

the diversity of individual questions would 

blurred ability to interpret variation in IRS

responsiveness. HRD used a similar appro

reviewing the Social Security Administrati

telephone inquiry program; over 4,000 call

made, with GAO personnel taking the role 

citizens in asking the randomly selected, p

questions (U.S. General Accounting Office

1986).

One element of data collection that disting

studies from other techniques is that

comprehensiveness of interviewing is very 

Page 65 GAO/PEMD-91-10.1.9 Case Stud
In order to learn the meaning of events to 

involved in them, a key element of case stu

views of more senior officials are not given 

weight than views of less highly placed pe

fact, a case study where the only people in

were senior officials would be seen as a no

study, in contrast to one where the views o

individuals at all levels affected was obtain

For example, if we wanted to learn about h

noncompetitive awards were reviewed in a

good case study would obtain information 

agency head, the head of the procurement 

the inspector general’s office, the contract

responsible for selected awards, staff invo

reviews for these awards, counterpart per

the contractors’ procurement and program 

staff, and the legal divisions within the age

contractors. We might shadow several

noncompetitive procurements, following t

history from initiation through actual awar

in on meetings, and studying, over time, ho

awards were handled.

Chain of Evidence A chain of evidence is the sequence from o

to conclusions. In a strong chain of eviden

independent second evaluator could follow 

evaluator from original observations, the “

unreduced data, through all the steps of da

aggregation and analysis, and conclude tha

evaluator’s findings were justified by the e

fairly represented it. This requires careful

organization of the files of original observa

complete documentation of the conditions 

collection that are relevant to the trustwor

credibility of the information, and making 

and reproducible the manner in which the 

moved from phase to phase of the analysis

Page 66 GAO/PEMD-91-10.1.9 Case Stud
evaluators call such a procedure “building 

trail” and use procedures similar to indexi

referencing to establish both the construct 

the measures reported and the convincing

causal explanations developed in the case 

(Halpern, 1983). That is, they have an inde

evaluator review the equivalent of their wo

rather than providing so much detail in the 

itself that a reader can come to the same c

Some information in a case study is likely 

judgmental, particularly when observer an

participant-observer modes of data collect

used. And the collection process involves j

calls of promising leads and the meaning o

information. While documenting the basis 

judgments can be more difficult than docu

nonjudgmental information, overall the ch

evidence or audit trail techniques should n

greater difficulty for GAO evaluators than 

documentation procedures for other evalu

methods.

Data Analysis Case studies, obviously, can generate a gre

data, data that need to be analyzed sufficie

with appropriate techniques in order to be 

Much is qualitative. As table 4.1 indicates, 

six general features of data analysis. Four 

essential to case study methods: iteration, 

triangulation, and ruling out rival explanat

A unique feature of case studies is that dat

and analysis are concurrent. In most meth

plan for data collection, then we collect th

information, then we analyze it, and then w

report. In case studies, the data coming in 

analyzed as they become available, and the 

results are used to shape the next set of ob

Page 67 GAO/PEMD-91-10.1.9 Case Stud
The sequence in which this takes place is t

which stands for “observe, think, test, revi

observations have been made in the first p

during the observations, because that is a 

for our minds to work), the evaluators thin

meaning of the information: what does it s

about what is happening and why? What e

explain what is going on? The

Page 68 GAO/PEMD-91-10.1.9 Case Stud
Table 4.1: Ways of

Analyzing Case Study

Data

Feature Methodology

Iterative Data collection 

concurrent ana

OTTR Observe, think

revise

Triangulation Comparison of 

independent s

evidence befo

there is a findi

Rival explanations Developing alt

interpretations 

and testing thr

for confirming 

disconfirming 

until one hypot

confirmed and 

out

Reproducibility of findings Establish throu

of multiple site

over time

Plausible and complete Data analysis e

plausible expla

been develope

considering co

the evidence

Specific techniques for

handling multisite data sets

Matrix of categ

graphic data d

tabulating freq

different event

developing co

tabulations to c

relationships, a

information ch

for time series 

second, or “think,” phase ends with specifi

what new information would be needed to 

alternative explanations or confirm interpr

This triggers the third phase: test. In this p

Page 69 GAO/PEMD-91-10.1.9 Case Stud
evaluator collects more information, as re

the specifications from the “think” cycle. T

collected in the third phase are not specifi

the first phase: they emerge, often with su

from the initial observations. The fourth p

examination of the second round of data c

and a revision of initial interpretations and

expectations—the “revise” phase. The revi

may lead to another test phase, if informat

the second round of data collection was in

to rule out alternatives, or if, during revisio

interpretations emerged. This iterative pro

when a plausible explanation has been dev

and, at the end of a “revise” phase, there a

or unexplained data, no further interpretat

possible, or it is clear that despite the mos

search for information, more is not availab

further refine description and explanation

In case study methods, causality is establis

through the internal consistency and plaus

explanation, derived additively through th

sequence. This is in considerable contrast 

evaluation methods, where control and co

groups are used subtractively to rule out o

reasons for a finding and establish firm att

Handling

Multisite Data

Sets

Several techniques have been developed re

handling multisite case study data sets. Th

setting up a matrix of categories, graphic d

displays, tabulating frequencies, developin

cross-tabulations, and time series analysis

Matrix of Categories In this technique, a coding scheme is deve

to data collection. It is modified during dat

collection and the OTTR process and final

the evaluation team has read through all th

Page 70 GAO/PEMD-91-10.1.9 Case Stud
materials. The categories are related to the 

subquestions; for example, if a subquestio

does the Immigration and Naturalization S

monitor the conditions of confinement in p

contracted detention facilities,” coding cat

might include who is responsible, how the

get information, what they do with inform

received, evidence that minimum standard

evidence of shortfalls, changes over time i

monitoring, and conflicting guidance or

responsibilities. These categories might be 

matrix by facility size or groups served. Th

is similar to content analysis, and the PEM

paper on content analysis gives further ho

information (U.S. General Accounting Offi

June 1982).

Graphic Data

Displays

This is a family of techniques, some of whi

been adapted for computers and some of w

wall-space. The evaluators immerse thems

information on a site, following OTTR. The

story of what is happening and why is disp

flowchart with a series of critical paths for 

Evidence supporting the story is arrayed in 

display. The materials then are searched fo

counterevidence and subsidiary or branch

are laid out. As a satisfactory graphic is de

one site, the evaluators turn to the next sit

evaluators could at this point either modif

graphic, based on information from the se

or prepare an independent flowchart. In th

approach, aggregation would come after a

had been charted, and the charts would be 

data base for aggregation.

The graphic techniques can be applied to a

as a whole or to subcomponents. For exam

analysis of life-threatening or fatal inciden

Page 71 GAO/PEMD-91-10.1.9 Case Stud
national parks were needed, the evaluator

develop separate graphics for events leadi

incidents, the incidents themselves, and po

actions. More complex case studies might 

several “layers” or graphics; less complex, 

Tabulating Event

Frequencies

Another technique for analyzing multisite 

identifying events within each case study (

between Jones and Smith”; “Smith staff pr

recommendations”) and tabulating their fr

occurrence. Such a simple tabulation can d

evaluator’s attention to events that may be 

or to informal networks and give a sense o

contrasted to on-paper) organizational rela

Divergences between observed and expec

can be examined further to see what happ

result of these meetings and identify poten

problem nodes: for example, when an exp

high-communication node turns out to be, 

speaking, a low-communication spot.

Complex

Tabulations

Cross-tabulations of events can identify in

and check the developing story more form

example, service coordination is a popular 

limited funds. An evaluator in the field ma

that coordination among local agencies fu

through the same federal agency is more f

than coordination among local agencies fu

different federal departments. Tabulations 

meetings and of consequent actions for sa

funded and different-agency funded servic

check out whether this impression is reliab

Time Series Analysis Organization of information within each si

of occurrence, coupled with a systematic a

contextual influences on events, permits a

Page 72 GAO/PEMD-91-10.1.9 Case Stud
nonquantitative time series analysis for ca

data. The flow of events over time for each 

actor and for significant points in the serie

forms the organizing framework for data a

within each site. Such comparisons of whe

actions occurred, how well (or poorly) the

carried out, and what influenced both timi

quality of performance can be particularly 

case studies of program implementation.

In some instances, only one component of 

study may be analyzed in this way. For exa

case study of the effectiveness of a job tra

program might need to take into account g

economic trends, such as unemployment r

community. A time series comparing local

unemployment rates with placement rates 

training program participants could be com

quantitatively and changes interpreted thr

more qualitative time series data about the 

Basic Models for

Data Analysis

Two basic models of data analysis are patt

matching and explanation building. Patter

requires using past experience, logic, or th

the job begins to specify what we expect t

analysis then compares actual findings to

expectations. When the findings fit, the pa

confirmed. When the findings don’t fit, the 

adjusts the expectations or elaborates them

a subroutine that can explain the unexpec

findings. Explanation building is the invers

procedure: starting with the observations, 

evaluator develops a picture of what is hap

why. Data are used to fill in the initial hun

change them, to elaborate on them. The fir

matches findings to hypotheses or assump

second uses the data to structure the hypo

assumptions.

Page 73 GAO/PEMD-91-10.1.9 Case Stud
In either strategy, the evaluator needs to s

full data base thoroughly for disconfirming 

in order to avoid the pitfall of premature c

and data analysis ends when the best fit po

been reached between the observations an

statement about what they mean.

In either strategy, expectations and explan

be expressed as themes: a job dealing with 

failures, for example, might have as theme

about credit risks, procedures for reviewin

decisions, or controls over the accuracy an

of information on bank solvency. A job de

employee training might have as themes d

about training needs, how employees are s

training, how course quality is monitored, 

employees and supervisors view the purpo

training.

Themes, in turn, can be analyzed within in

sites first, then findings on each theme agg

across sites. Alternatively, all themes with

can be analyzed first; then data from the se

subsequent) sites can be examined. Theme 

also can proceed in matrix fashion. On the 

AFDC study, for example, evaluators were 

as site managers, responsible for understa

across themes all there was to know abou

for their site. They also were assigned to in

themes, such as health and employment, r

concurrently for looking across all sites fo

information on their topic. This organizati

helpful in ensuring that reasons why a site 

as an outlier for a given theme could be di

someone who knew the site as a whole.

Pitfalls and

Booby Traps

Case study methods, like any other metho

plenty of opportunity to go awry. Two freq

Page 74 GAO/PEMD-91-10.1.9 Case Stud
concerns are the risks in using other peop

and in generalizability.

Impartiality The biggest risk when we use other people

studies is that GAO standards of impartiali

have been met. There are three meanings o

impartiality, one of which does not create 

Case studies use as data the impressions a

judgments of the evaluator, which are inhe

subjective. For a case study methodologist 

GAO, if proper care is taken, this should n

problem. If we want to illustrate, for exam

working conditions for immigrant laborers

report what the thermometers registered a

also report, firsthand, how people were sw

what it felt like to be out in the fields. Such

observation is part of the richness, immed

“thick” description of a case study. Howev

studies, like any other method GAO uses, h

meet two other criteria of impartiality: acc

lack of bias, in the sense that the evaluator

preconceived opinions about a situation d

distort reporting and that the evaluator is

scrupulously evenhanded in examining all 

situation.

Some authorities on evaluation methods b

case studies reflect the author’s values in w

can be difficult to detect. Other experts co

three actions, taken together, are sufficien

safeguards for lack of bias and adequate a

These are (1) submitting reports to people 

whom data were collected and printing the

with the report, (2) use of multiple data co

methods within case studies, and (3) adop

audit trail or chain-of-evidence technique. 

supervisory controls also are recommende

Complying with these safeguards should g

Page 75 GAO/PEMD-91-10.1.9 Case Stud
major problems in our own jobs. The guid

mainly expand the range of reviewers. We 

conduct exit conferences and, following th

Book” and Communications Manual, subm

reports for agency comments. We often us

methods, and the audit trail technique now

recommended for case study use was itsel

from such auditing procedures as workpap

referencing, which are standard practice w

We also require adequate supervisory cont

such means as prompt review of workpap

would need to assure ourselves, however, 

studies whose results we are going to use 

adopted the same procedures for ensuring

impartiality. (Appendix III gives a checklis

reviewing proposed or completed case stu

quality.)

Generalizability We often are asked questions where the cu

wants in-depth information that is nationa

generalizable, but frequently the issue may 

ripe for a national study or we do not have 

resources to collect in-depth data from nat

representative samples. Using 4, 10, or 15 

studies might be feasible, but we would sti

be concerned about the risks in generaliza

main point of this paper is that generalizab

depends less on the number of sites and m

right match between the purpose of the stu

how the instances were selected, taking in

the diversity of the programs.

An example of an efficient combination of 

specification of the purpose of the study m

with appropriate site selection is the GGD 

the productivity of the Social Security

Administration’s (SSA’s) regional operatio

review examined in depth only one SSA re

Page 76 GAO/PEMD-91-10.1.9 Case Stud
General Accounting Office, September 11, 

Atlanta was selected because it had the be

productivity among the 10 regions; if GAO 

demonstrate opportunities for improveme

most productive SSA region, then similar

improvements might be possible in the les

productive regions. Following the case stu

inexpensive (25 staff day) check was made 

productivity data and trends from other SS

and similarities were noted. While other pr

might be affecting these less productive re

findings from the single site plus the trend

convincing that SSA concluded the single 

examination had national implications. Su

analysis of regional office plans for produc

improvement led to the conclusion that th

implementation could save about $60 milli

annually.

Sometimes, however, it is not possible to a

evaluation question using case studies, if t

is diverse and the user needs national gene

The user may prefer to sacrifice in-depth i

for generalizability and we will have to use 

methods, such as surveys or secondary an

existing data. However, it often is possible

appropriate instance selection—to obtain 

generalizability with a manageable numbe

instances. In addition, the evaluator can ap

case survey method to increase the genera

findings and can combine case studies wit

methods. Taken together, these strategies 

the use of the case study technique with en

generalizability for many users’ purposes. 

the first three types of case studies (illustr

exploratory, and critical instances) genera

needed, cannot be achieved unless they ar

with other methods. Generalizability can b

for the three other types, however, even w

Page 77 GAO/PEMD-91-10.1.9 Case Stud
are used independently, as long as they are 

designed in terms of case selection and an

strategies.

Where to Go for

More Information

More detail on data collection and analysis 

found in two books on case study methods

Study Research by Yin (1989) and Analyzin

Qualitative Data by Miles and Huberman (

detail on applicable GAO guidance can be 

the Communications Manual.

Page 78 GAO/PEMD-91-10.1.9 Case Stud
We can summarize this paper in the answe

questions: What are case studies? When ar

appropriately used in evaluation? What dis

a good case study from a not-good case stu

What Are Case

Studies?

The case study is a method of learning abo

complex instance, based on a comprehens

understanding of that instance obtained by 

description and analysis of the instance ta

whole and in its context. Applying this def

means learning virtually everything about 

being studied, including how it operates an

does, in relation to the extrinsic or contex

it is part of.

Case studies often use one or only a few in

because collecting and analyzing compreh

are prohibitively difficult for large number

However, not all studies of a small number 

instances are case studies. Some studies c

from a small number of sites but have no o

features in common with case studies and 

of their advantages. Thus, the fact that a st

involves only one or a few sites does not

automatically make it a case study. For ex

evaluators may not have selected the sites

appropriately for the generalizability need

may have collected minimal information w

depth of inquiry.

When Are Case

Studies

Appropriately

Used in

Evaluation?

We discussed six types of case study that d

considerably in their requirements for site 

data collection, and analysis, among other 

six types are illustrative, critical instance,

exploratory, program implementation, pro

effects, and cumulative. Together, they cov

range of evaluation questions, although cle

Page 79 GAO/PEMD-91-10.1.9 Case Stud
evaluation questions. For example, case st

not well suited for answering the question

does something happen?

Some applications of the case study to eva

purposes have been tried fairly extensively

example, program implementation case st

Others are relatively untried—for example

cumulative case studies. The latter is a par

promising method for GAO, because it can 

on the large number of case-study-like rep

are available, on the quality of the docume

supports their findings, and on the general

methodological framework that GAO’s sta

provide. We have not stressed, in our analy

costs, feasibility, and timeliness of case stu

these are management criteria that are con

all designs rather than issues of particular 

case studies. However, the implications of 

features discussed here are that, contrary 

many people think, the case study is not n

inexpensive, easy to conduct, or quick. It m

in-depth data collection dependent on sen

the setting that takes time to acquire and i

extended periods for data analysis, interpr

reporting.

What

Distinguishes a

Good From a

Not-Good Case

Study?

We have addressed quality in two ways. On

prospective and intended to help those wh

evaluation to know the minimum features 

various case study applications. The other 

retrospective and intended to help those w

case study reports to assess the quality of 

case studies. Table 5.1 summarizes commo

that we have mentioned throughout this p

Page 80 GAO/PEMD-91-10.1.9 Case Stud
Table 5.1: Some Common

Pitfalls in Case Study

Evaluation

Study stage Common pitfa

Design Mismatch betw

for the specific 

what the case 

application ca

insufficient atte

contrasts and 

needed for pu

study

Site selection More sites sele

needed; fewer 

selected than 

inappropriate 

selection, for t

job and evalua

Data collection Reliability jeop

lack of commo

in data collect

noncomparab

quality control 

collector roles 

responsibilities

threatened; ov

relationship be

collected and 

evaluation que

inadequacy of 

Data analysis Insufficient atte

requirements o

plan chosen: lo

plausibility of r

insufficient atte

management a

reduction: inef

lateness, incom

data; inadequa

of relating find

sites; inadequa

for relating qua

quantitative da

sites

Page 81 GAO/PEMD-91-10.1.9 Case Stud
Study stage Common pitfa

Reporting Overgeneraliza

compared to a

for site selectio

of sites studied

requirements f

in the design; 

interpretation, 

narrative, resu

adequately rel

questions; inad

attention to thr

impartiality and 

to which these 

avoided

Quality and

Evaluation Design:

Planning

We have presented six types of case study 

and for each one described features such a

of sites, site selection, data collection, data 

and reporting. Our descriptions represent 

quality for each evaluation application. Th

of the six types of case study are not interc

That is, the features of a case study that ar

appropriate for answering one kind of eva

question are not necessarily appropriate fo

answering another kind of evaluation ques

Evaluators considering the case study as a 

evaluation must first decide what type of e

question their specific question is and then 

the strengths and limitations of each type 

study for answering it. The crucial next ste

at the features of each type and decide wh

be possible to meet these methodological

requirements in the specific situation.

For example, the basis on which instances 

selected differs for the different case study

applications. Usually, an illustrative case s

should be typical of the program being exa

Page 82 GAO/PEMD-91-10.1.9 Case Stud
while exploratory case study sites should b

diversity that is likely to be encountered in 

program, population, and setting of a large

Usually, sites for program effects case stud

be selected with great care for criteria suc

whether there is evidence that the program 

implemented at the site, whether the site h

subjected to changes that could have the s

as the program or that could mask its effec

how the addition of this site to the group o

being studied supports the generalizability 

findings.

Quality of

Evaluation Design:

Reviewing

Turning to ways of assessing the quality of 

case studies, we have provided guidelines 

reviewing case study reports in appendix I

guidelines are intended to apply to all type

study applications.

On matters of design, the guidelines discus

clarity of issues, the relationship of the eva

question to the case study application sele

basis for case study selection, and the time 

the study. The data-collection guidelines e

appropriateness of data-collection method

training, and information sources. Guidelin

data-base formation and analysis deal with

explicitness of procedures and techniques

interpretation differences, and the relation

findings to those of similar studies. With re

reporting, the guidelines emphasize constr

study, arguments for and against various r

of the issues, and the role of judgment in r

conclusions. With regard to impartiality an

generalizability, the guidelines emphasize 

case study report (or, for GAO purposes, j

documentation) describes both the evalua

training and work on related studies, prese

Page 83 GAO/PEMD-91-10.1.9 Case Stud
comments on the draft report, and supplie

information for judging generalizability. Re

will need to refer in addition to the feature

special type of case study application for

supplementary guidance on what to look f

individual case studies.

Impartiality and

Generalizability

Partiality and—in some instances—the ina

generalize from the findings can limit the u

case study methods for evaluation. There a

main threats to impartiality: subjectivity, in

and bias. The case study method inherentl

subjective and judgmental elements. When 

procedural safeguards are used, these elem

do not diminish the value of case study me

However, inaccuracy and bias are unaccep

any case study. Some ways of detecting an

preventing bias, such as the audit trail, hav

developed. Their applicability to case stud

evaluations outside of settings such as GA

explored.

Many evaluation questions do not require 

degree of generalizability. Certain case stu

applications provide high degrees of gener

with small numbers of instances. When bo

generalizations and in-depth understandin

required, designs that cumulate case studi

wide number of sites and settings, or that 

case study methods and other methods in 

concurrent effort may meet this dual need 

General Accounting Office, April 2, 1984). 

the diversity of the population to which ge

is required is a limiting factor in case study

applications. It is also true that without ca

attention to standards, case studies are pro

kind of overgeneralization that comes from 

a few instances, assuming without evidenc

Page 84 GAO/PEMD-91-10.1.9 Case Stud
are typical or representative of the popula

then offering national projections. The GA

Manual cautions against overgeneralizatio

method. For case studies in particular, the

an empirical basis for instance selection an

assurance of adequate population homoge

Page 85 GAO/PEMD-91-10.1.9 Case Stud
As a research method, the case study origi

social sciences, particularly in the fieldwo

anthropology and sociology. Within these 

researchers have defined the case study an

discussed its critical elements in a variety 

general, six elements appear frequently: pu

of data collected, method of data collectio

method of data analysis, and reporting.

Purpose The research case study has been defined 

for learning the “right” questions to ask (H

al., 1982). That is, the purpose of case stud

by some researchers to be to generate hyp

rather than to test or confirm them. The m

involves an in-depth, longitudinal examina

single instance. The product is a sharpene

understanding of what might be important 

further in similar situations and what expl

the instance happened as it did. Because s

explores only one situation, it is argued th

contribute directly to the testing of genera

propositions, although it can contribute po

the invention of hypotheses.

Some other methods have a similar purpos

Exploratory analyses of large data bases a

used to select a smaller number of variable

additional testing, on the basis of interesti

that emerged from various combinations o

elements of the large data base. Promising

relationships are singled out and those tha

uninteresting are set aside. Like findings fr

studies, the result is considered as contrib

answers but a better understanding of wha

to ask and how to ask them.

An analogy might be drawn also to “evalua

assessment.” Such assessment may provid

Page 86 GAO/PEMD-91-10.1.9 Case Stud
information valuable in itself about how co

program has been implemented. It is unde

to testing the effects of the program, chief

as an indicator of appropriate evaluation d

Other researchers regard case studies as n

different way of knowing but as a better w

1981a; 1982a). More specifically, they emp

method’s unique value in many complex si

importance, such as studies of school dese

or economically distressed communities, o

Challenger tragedy. One reason they give i

skilled observers and interviewers can ma

judgments and valuations about factors th

otherwise very difficult to assess, such as 

effort a manager made to get information b

decision was made or how much that pers

about what was going on. Also, these resea

believe, in complex situations the many pe

are significantly involved have different “r

their explanation of events and even in the

perceptions of what happened, and this is 

matched with a method that gradually repr

reconstructs these multiple realities, rathe

method that assumes a single “truth” exist

As seen by such scholars,

“there is no single reality on which inquiry may conv

there are multiple realities that are socially construct

when known more fully, tend to produce diverging re

multiple and constructed realities can not be studied 

variables, for example), but only holistically, since th

interrelated in such a way as to influence all other pi

Moreover, the pieces themselves are sharply influenc

nature of the immediate context.”

Methodologists who focus on case studies 

their criteria of good research in different 

although they may deal with underlying co

Page 87 GAO/PEMD-91-10.1.9 Case Stud
similar to those of researchers from more

experiment-oriented traditions. Some crite

however, are seen as unique to case studie

qualitative approaches. One example is fai

assurance that a study has presented a bal

of the many constructions of reality and th

underlying these. Another example is auth

another, realism. Each of these criteria is a

in the literature on case study methods, wi

performance standards such as triangulati

useful in ensuring—if they are carried out—

the study will be a good one.

Table I.1: Criteria of Good

Research Case study Other approaches

Truth value;

trustworthiness;

credibility

Internal validity

Applicability;

transferrability

External validity; general

Confirmability of

data

Objectivity of observer

Consistency,

dependability of of

data; explainable

instabilities

Replicability; stability reli

Type of Data In some textbooks on evaluation, case stu

synonymous with qualitative data—that is

are subjective or judgmental. Such data in

narratives of events written by participant 

accounts of what the participants understo

an event, reports of what was said at a me

interview, observational records of how an 

place, and statements of impressions abou

going on, why it was happening, and how p

about it, themselves, or each other.

Page 88 GAO/PEMD-91-10.1.9 Case Stud
To illustrate differences among types of in

we might base the conclusion that “the day 

on data from an instrument that records th

temperature (numerical and objective), a r

the atmospheric temperature as written do

observer checking a thermometer (numeri

relatively nonsubjective), a survey asking p

hot they felt (nonnumerical and subjective

“thick” description of what clothes people 

wearing, how much they perspired or shiv

whether they turned up the furnace or the 

conditioner, and how much energy they se

have for work (nonnumerical and judgmen

researchers describe case studies as using 

data, they usually mean the “thick” descrip

evaluation question involved an understan

working conditions for migrant laborers o

heavy industry, a thick description, even in

information on how exhausted the evaluat

the heat, would be more appropriate—acc

some case study methodologists—than on

that the thermometer registered 95 degree

Suppose we needed to know about the ava

housing for low-income people. If official r

were adequate, good quantitative measure

availability might be the number of low-inc

persons applying for housing relative to th

of units that met minimum standards and c

30 percent of household income or the num

persons on waiting lists for such housing a

long they had to wait. We might also be ab

the number of applications for housing con

permits and how many units suitable for lo

housing were coming on the market within 

months. Often, however, the records are n

adequate. Here we might rely on qualitativ

information, such as the estimates of know

officials of demand and supply (judgmenta

Page 89 GAO/PEMD-91-10.1.9 Case Stud
numerical) or of severity of the problem (j

nonnumerical). We might also interview se

low-income families with regard to their ex

in seeking housing or we might, as

participant-observers, pose as low-income 

and report our own experiences in finding 

families of different sizes and within differ

payment ranges (judgmental, numerical, a

nonnumerical).

Many researchers who write case studies u

qualitative data because they believe them 

richer, more insightful, and more flexible t

quantitative data. They believe that the me

event is more likely to be caught in the qua

than on the quantitative hook. For exampl

qualitative data permit dealing fairly direct

values, politics, and factors that may be an 

part of many situations. A frequency distri

events—such as a table showing the numb

decision points in a community economic

development program and a decrease in th

probability of action as the number of dec

increases—are about as numerical as qual

are likely to be in a research case study, ac

some experts.

Method of Data

Collection

To some researchers, case studies are syn

with methods of data collection deriving fr

anthropology, psychology, and sociology. 

techniques include fieldwork, ethnography

observation, and participant observation a

common that an observer is physically pre

site, stays at the site for a fairly long time, 

flexibility in deciding what data to collect 

and under what circumstances, and can or

inquiry according to the meaning of events 

participants rather than having to decide b

Page 90 GAO/PEMD-91-10.1.9 Case Stud
on a closed set of constructs or data eleme

most instances, the observer is the senior 

and the only researcher: Margaret Mead in 

Oscar Lewis in Puerto Rico are famous ex

The greatest difference, to some experts, b

other methods and case studies is the disti

between the researcher’s (1) beginning by 

a priori, to know the relevant constructs a

variables, measuring their incidence, and f

how changes in them may be influenced b

events and (2) entering into an event to lea

significant in it to the participants. As this 

researchers, the case study is an intensely 

method, dependent on the investigator’s se

insights, and skill in noticing many things, 

them, and producing a narrative that sugge

pattern of the elements—or that recognize

pattern that is there in the culture in its ow

is a demanding method, requiring specific 

as fluency in the language of the participan

general self-awareness to maintain the fine 

between seeing things as others see them 

identifying their perspective wholly with o

The researcher must weigh the value of ex

what it is like to be part of the culture agai

hazard of internalizing the experience too 

can jeopardize the capacity to see the cult

many perspectives. Nonetheless, some of t

reports have come from observers who en

fully as possible as participants in the even

investigated.

The case study method is further distingui

researcher’s self-conscious effort to under

the observed events mean to the participan

observer can enter a scene without precon

ideas, but they can be set aside. Thus, a stu

a group is organized economically might b

Page 91 GAO/PEMD-91-10.1.9 Case Stud
finding out what is valued in that group an

items of value are exchanged. They might 

goods or services, and exchanges might no

For example, in a basic research study of 4

low-income women, Belle and her colleagu

many months among them as observers, c

and friends, listening to what they said and 

what they did. The researchers found that 

someone for even modest help (like mindi

for an hour) had the cost of later demands 

of the favor and that this cost was nearly i

The researchers found expected stresses l

of a check in the mail and the illness of lov

They found also, unexpectedly, that any ch

was stressful: being promoted to a higher-

the graduation of a child from school, falli

even the restoration to health of a loved on

been ill. In terms of the purposes of the

study—finding out what was stressful to th

and why the incidence of mental health pr

among them was so high—the case study m

disclosed the importance of any change in 

circumstances as a source of stress rather 

merely confirming change that the observe

have thought stressful a priori.

Design Case study methods have been defined by 

researchers as designs that focus on a sing

or a few instances. They also are identified 

designs that are nonexperimental in the se

investigator is not deliberately manipulatin

variable to see its possible effects on the s

studied. Two classic aims of inquiry are to 

the nature of events and to understand the

Since case study designs center on one or 

and lack the controls usually thought nece

understanding of causal relationships, kno

Page 92 GAO/PEMD-91-10.1.9 Case Stud
results from case studies is controversial w

to generalizability and causality.

With regard to generalizability, some meth

see case studies as above all particular, se

describe and understand the aspects of an 

without much concern for knowing wheth

arise in or are characteristic of a larger po

The focus is on this school, this emergency 

military base, or this nuclear power plant.

Researchers can choose relatively freely w

instance to study on any one of several ba

depending on the questions to be examine

Thus, in a case study design, an instance m

selected because it is a unique event of nat

interest, such as the Cuban missile crisis o

distribution of the swine flu vaccine, both 

research case studies. The instance may b

because it has been affected by events of i

such as the desegregation of schools. It ma

selected as an unusually effective or unusu

ineffective instance. However, whenever t

is an understanding of the particular, the r

of the instance to the various populations 

part of is less important than the assuranc

selected instance can be fully examined.

With regard to causality, researchers using 

methods cannot rely on familiar ways of ru

alternative explanations. Case studies do n

individuals or groups to others randomly a

different treatments. Case studies do not u

statistical adjustments to facilitate compar

studies do not estimate statistically the inf

the many variables on the instance being e

To understand a single case, the researche

develop hunches about what is happening 

instance under study and systematically se

Page 93 GAO/PEMD-91-10.1.9 Case Stud
evidence consistent or inconsistent with th

As evidence accumulates, a second tier of 

looked for that would be consistent or inc

with alternative explanations for why the h

or did not take the shape of a coherent pat

is, a very high standard of inferential logic 

When this method produces a coherent, pl

story, the researcher can assert a relations

between cause and effect. When conflictin

cannot be resolved, the careful investigato

that causality cannot be established. The s

making this judgment requires the diligenc

investigator in formulating alternative exp

what is happening, in specifying the kind o

that would be supportive or nonsupportive

searching intensively for evidence that wo

initial hypotheses, and in thoroughly consi

reasons for inconsistent patterns of eviden

second tier. These techniques, of course, h

parallels in other research traditions.

The ability of the case study to capitalize o

to shift focus as the data demand, and to le

pieces of evidence fall into place in ways t

always easy to describe or command is be

yield a richer, fuller, and truer explanation 

things look the way they do than the more 

number of tests of a priori hypotheses that 

methods use. In case studies, the criterion 

deciding whether casuality has been estab

coherence of the evidence, its consistency 

patterns ascribed to it, and its inconsistenc

other explanations. In research designs ba

statistical inference, the criterion for estab

casuality is whether the findings are likely 

occurred by chance following appropriate

comparisons to eliminate alternative interp

Page 94 GAO/PEMD-91-10.1.9 Case Stud
In both instances, comparisons must be ap

alternative explanations are to be ruled ou

Method of Data

Analysis

Still another distinguishing feature of case 

according to some researchers, is a nonsta

approach to data analysis. The elements o

are familiar: the identification of regulariti

and relationships and the assessment of th

importance of meaning. In quantitative me

regularities are identified by manipulating 

produce indicators agreed on as sensible d

of the patterns. For example, an average is 

convention that creates a single number to 

the collection of all the numbers in a set. I

or meaning is assessed in part by estimatin

variability within the set of numbers to ob

probability that the regularity represents t

characteristics of the population of instanc

The logic of analysis in case studies is the 

techniques for identifying regularities and 

their meaning are different. Consider first 

regularities. The case study analyst is tryin

an accurate description and explanation o

both the observer and participants frame t

would be little point in trying to identify a 

number as an accurate representation of s

this complex.

The analyst searches for clusters or paths 

using verbal notes and graphic aids, review

data and other records of observations, un

is evident. Then the data base is searched 

evidence that confirms or conflicts with th

When the evidence is more inconsistent th

consistent, the pattern is rejected. When th

is more consistent than not, the analyst co

pattern and looks for others related to it. W

Page 95 GAO/PEMD-91-10.1.9 Case Stud
evidence is consistent, no further examina

needed. An array of techniques such as gra

displays has been developed to help stand

study analysis.

A key element of case study analysis is the 

and organization of material to account fo

complexities and interactions of the event

are judgmental, not probabilistic. Have all 

of pertinent information been considered? 

adequate attention been given to the outlie

the pattern seem plausible or forced? Have

inconsistencies been sensibly resolved?

Using an analogy, we can say that the case 

analyst seeks to explain 100 percent of the 

by relying on a data base that includes mo

than most quantitative studies can accomm

over more points in time, and on a method 

on the integrative powers of the mind, whi

computers do not have. The method inher

sociology and anthropology entails early im

the setting, recorded observations, reflecti

spot, and analysis that occur throughout fi

data are being collected. Analysis is infuse

throughout the research process in case st

not a step after the completion of data coll

Reporting Case studies are usually reported as narrat

read like chronologies of what led up to an 

what happened during and after it. They h

called “then-they-did-this” studies. In orde

comprehensive and cohesive, the research

a great deal of detail and description and q

directly from the participants’ own words 

vignettes in the observers’ field records.

Page 96 GAO/PEMD-91-10.1.9 Case Stud
To some proponents of case studies, the c

the method depends on what they call “na

generalizability.” By this they mean that re

compare their own observations, experien

belief to the narrative and regard the parts 

investigation that are consistent with these 

confirmed. What is inconsistent tends to b

more closely and may be rejected as less c

unless—so the argument runs—there is en

that readers can “see it with their own eye

readers who know a handicapped child or 

aging parent, for example, a case study rep

conditions in residential care for the sever

handicapped or the aged may compare dra

with vivid personal experiences. The case 

report can provide an organizing framewo

thinking about these and other experience

The usefulness of case study reports, there

depends to some degree on how well the i

has portrayed the participants’ ways of thi

what happened and on how divergent the

investigator’s analysis is from the reader’s 

thinking about the subject. The credibility 

authenticity of the case study report may d

the writer’s having provided extensive det

description, making unexpected conclusio

difficult to deny as if the reader had been p

event. In this respect, the narrative mode i

stylistic choice; it is inherent in the purpos

studies and the nature of their inquiry. It b

obstacle only when authors seek publicati

outlets whose customers generally ask for 

details.

The Case Study

Adapted for

Evaluation

The traditional case study belongs to resea

evaluation. To apply case study methods t

evaluation, evaluators have had to adapt w

largely from sociological and anthropologi

Page 97 GAO/PEMD-91-10.1.9 Case Stud
fieldwork (Patton, 1980). Before 1970, how

evaluation case studies were similar to res

studies. They were longitudinal, were mad

observers who sought participant-observe

constituted an inquiry structured from an 

understanding of events and their meaning 

persons involved in them. There was usua

research investigator, and the data consist

descriptions, observations, impressions, u

interviews, and existing materials gathered 

that were organized intuitively and inform

case study report was a narrative whose p

to illustrate or portray what a program wa

it was being implemented, and how those 

part of it both affected it and were affected 

In these early uses of the case study metho

evaluators wrote their reports to stand alo

effort was made to integrate two or more s

data, even when the evaluation design incl

although simple references might be made 

number of times a feature of other sites w

characteristic of the site reported in the st

case study was useful for readers intereste

particular program was like or what happe

typical beneficiary.

Early in the 1970’s when evaluators wante

studies that would capture some implemen

outcome features that were different or ex

measure reliably on a large scale—for exa

way a large organization handled a comple

innovation or the effect of an education pr

motivation to learn or growth in

self-confidence—they felt caught between 

considerable effort in trying to quantify qu

variables and risking the criticism that the

ignoring really important things because th

not be measured. The case study seemed a 

Page 98 GAO/PEMD-91-10.1.9 Case Stud
offered a relatively inexpensive, low-risk s

to the large-scale collection of other inform

could be measured more cheaply and relia

with large-scale studies. By and large, inve

commissioned to carry out the early case s

evaluations had been trained in the academ

disciplines with the strongest fieldwork tra

and they had to struggle with the extent to 

their research method could be adapted to 

retain its integrity, and yield positive bene

struggle is not yet fully over.

The Study

Questions

The first adaptation was a shift of the spec

study question from the principal investiga

the period of study performance to the per

commissioned the study in advance of dat

For example, those who supported an eva

training program might want the researche

out whether the development of the partic

self-concepts, self-esteem, task orientation

habits, and personal and social traits seem

associated with the program or with some

Researchers from fieldwork traditions wou

argued that they could provide the most us

information by spending some time at the 

training program, trying to understand wh

program meant to those who were involve

reporting on what was happening from the

perspective of those who were making it h

Since this might have everything to do wha

participants’ chance to socialize with frien

nonthreatening environment and nothing t

self-esteem or work habits, it seemed to th

researchers that it was therefore logical no

on the evaluation questions until their

appropriateness could be determined.

Page 99 GAO/PEMD-91-10.1.9 Case Stud
The compromise that has developed is to i

the evaluation the questions of interest to 

customer and to permit the researchers to 

what data elements are relevant to these q

and from whom and how they should be c

allows the evaluator to remain alert to oth

that might prove more salient if allowed to 

How satisfactory is the compromise? The 

of some non-GAO case studies show little 

resemblance between the final questions a

the approved study proposal, and a numbe

about this have not been resolved. We do n

whether the discrepancy is more frequent 

studies than in other methods. It may be th

questions are the ones the investigators w

look at all along, so that the methodology 

vulnerable to subterfuge. Possibly the eme

questions were those that should reasonab

been expected to come into focus—and w

emergence may be why case studies rathe

surveys are used. From the perspective of 

authenticity and integrity of results, the lar

interest may have been served. If the meth

susceptible to this kind of internal change

appropriate scope for case studies should 

examined. When the case study involves o

modest expense, the price for identifying b

questions early may seem affordable.

This is not to say that all case study evalua

divergence between the questions that we

and those that were answered or that an a

balance between the evaluator’s and the cu

needs is never reached. However, applying 

study methods of research to evaluation re

dealing with matters of control, power, an

responsibility that were less visible in the 

Page 100 GAO/PEMD-91-10.1.9 Case Stud
academic researchers before their method

adapted to evaluation.

The Number of

Sites

The demands of evaluation led to other ad

in ethnographic methods. One such deman

a method developed for understanding the 

had to be modified for learning about the g

Another was the need for something more 

than “naturalistic generalization” for evalu

purposes. A third was the problem of site v

which in the mid-1970’s was identified in q

studies as an ill-understood source of grea

differences in a program’s outcomes than 

itself.

The case study method seemed born to he

forces of time and cost associated with ma

multisite evaluations led to considerable a

First, since evaluators often needed simult

study at several sites, they needed several 

which created issues of coordination and

interpretation. Second, the cost of maintai

trained full-time field worker at a site runs 

that evaluators had to settle for shorter ob

or untrained field workers or both.

All these changes—to multiple observers,

professionally supervised but not professi

trained observers, and shorter observation 

to others. The across-sites data base got m

as the number of sites in a study rose. The 

data bases became less extensive as obser

times were shortened. It became a challen

integrate the work of different observers if 

focused their attention on different topics 

site. And this much larger, much less exten

probably less reliable data base had to be 

and reported in a much shorter time than t

Page 101 GAO/PEMD-91-10.1.9 Case Stud
leisurely academic research. Not surprisin

has become a major methodological conce

more structured and perhaps more efficien

approaches to analysis have been develop

Quantitative

Methods

To these adaptations, another was added. 

study was given a purpose—program

evaluation—beyond that of illustration, ex

or generation of hypotheses. As the exami

program implementation and program effe

became more central to the case study, so 

ability to generalize findings. In turn, quan

methods in case studies expanded.

Quantitative methods were incorporated in 

study in two ways. The first was in triangu

use of several forms of data within a single 

in order to give many reference points for 

patterns and ruling out alternative explana

order to achieve what evaluators call “inte

validity.” The second was in the combinati

study methods with other methods, particu

surveys, in order to achieve the generaliza

evaluators called “external validity.” These

adaptations created the need for a better

understanding of the relationship between 

techniques and other techniques and betw

quantitative and qualitative approaches wi

studies.

Summary Table I.2 summarizes the changes that hav

made to adapt the research case study to e

needs. Adapting the research case study to 

evaluator’s needs has entailed a number of 

Less time is spent at sites. Information is c

junior staff working under the supervision 

investigator trained in case study methods

Page 102 GAO/PEMD-91-10.1.9 Case Stud
is allowed for training and monitoring qua

are combined from several sites to allow

generalization; and data collection has bee

greater structure. Methods of assessing th

of observations, techniques for transformi

Table I.2: Evaluation Adaptations of the Research Case Study

Case study element Research Evaluation

Design specifications

Study questions Researcher

asks

Sponsor asks

Variables Emerge from

observation

Sponsor specifies

Site selection Of specific

interest

Representative

Instances One Many

Data Researcher

specifies

Sponsor or sponsor and res

specify

Design Trends at one

site

Comparison of many sites

Methods One Several

Costs Usually

inexpensive,

timeconsuming

May be very expensive and

time-consuming, particula

studies with many sites

Data collection

Type of data Quantitative Quantitative and qualitative

Time span studied Long Short; may be cross-sectio

Time at sites Long Short

Sources Informants,

observation

Informants, documents, ad

data

Collection method Researcher

specifies

Sponsor or sponsor and res

specify

Role of insight Central Supplementary

Collector Researcher Staff

Page 103 GAO/PEMD-91-10.1.9 Case Stud
Case study element Research Evaluation

Analysis

Analyst Researcher Staff

Researcher’s role Comprehensive Supervisory

Data reduction Minimal,

original data

Considerable; codification, 

analysis

Multiple data Triangulate

within site

Triangulate across and with

Analysis techniques Nonformalistic,

pattern

recognition

Formalistic; graphic and co

analysis techniques

Procedure Intuitive,

thematic

Formal, comparative, thema

Establishing casuality Coherent,

plausible story

Greater emphasis on desig

in addition to internal coh

Reporting Narrative,

descriptive,

detailed

building of

coherent story

Conclusion-oriented, use o

for examples

large amounts of qualitative data, and met

aggregating qualitative data or findings fro

sites have been developed. The ability to g

has become a matter of design and analysi

Reporting methods have changed.

Case studies in evaluation today have mad

adaptations in different degrees. Some stu

not only generalized but also tested hypoth

case studies rely wholly on quantitative da

rely wholly on information collected by ot

trained as sociologists or anthropologists, 

on firsthand observation. Some aim for un

comparability of data both within a site wi

observers and across several sites. Some u

inferential statistics as well as descriptive 

Page 104 GAO/PEMD-91-10.1.9 Case Stud
Some present findings and conclusions in 

closely resembling those of other methods

These adaptations are not uniformly value

case study methodologists work with struc

evaluation questions, structured data colle

observers untrained as anthropologists or

sociologists, but they believe that case stu

qualitative way of knowing that should not 

with quantitative results. Others believe th

studies cannot be used for making the kind 

generalizations that probabilistic models a

so that little is to be gained and so much is 

from increasing the number of sites. Still o

believe in using many sites in case studies 

evaluations and see the next step as establ

explicit procedures for analyzing data and 

quality.

“Case study” means different things to diff

methodologists, who reach different concl

about how to do case studies, how to repo

and their overall appro-priateness for answ

specific question. If case studies can vary 

how can we assess their usefulness for eva

One way is to develop a working definition 

study that embodies its essential methodo

features and then to examine the strengths 

limitations of case studies for different eva

questions. This is the approach taken in th

developing our initial definition.

Page 105 GAO/PEMD-91-10.1.9 Case Stud
Imagine that in 1987, within an effort to es

extent of tax revenues lost or delayed from 

of businesses to file returns, the General A

Office examined revenue shortfalls to indi

states. Imagine we found 170,076 such inst

national projection based on a sample) an

that, cumulatively, over $500 million was l

states. Our report attracted much congres

interest. Variation among states in the rate 

“missing returns” was of particular concer

we now have been asked to examine in mo

what explains differences among states in 

returns,” since cumulatively the effect is to 

states look poorer than they actually woul

collected revenues authorized by their ow

legislatures. (Hypothetical data for this ex

given in table 2.2.)

Question 1:

Instance

Selection

Using the hypothetical data in table II.I, id

for each type of purposive selection that w

consider.

Bracketing

Best case

Worst case

Cluster

Representative

Typical

Special interest

Page 106 GAO/PEMD-91-10.1.9 Case Stud
Table II.I: Hypothetical

Data on Unfiled

Corporate Income Tax

Returns for 1986 State

Income Tax Returns

State Number unfiled 

Alabama 6,100 

Alaska 610 

Arizona 3,475 

Arkansas 4,391 

California 28,841 

Colorado 3,012 

Connecticut 2,738 

Delaware 995 

District of Columbia 1,562 

Florida 13,372 

Georgia 8,887 

Hawaii 1,197 

Idaho 732 

Illinois 16,103 

Indiana 6,077 

Iowa 2,096 

Kansas 2,125 

Kentucky 3,724 

Louisiana 8,462 

Maine 1,032 

Maryland 6,292 

Massachusetts 4,427 

Michigan 8,849 

Minnesota 3,074 

Mississippi 6,002 

Missouri 5,886 

Montana 770 

Nebraska 1,324 

Nevada 781 

New Jersey 7,985 

Page 107 GAO/PEMD-91-10.1.9 Case Stud
State Number unfiled 

New Mexico 2,394 

New Yorka 19,349 

North Carolina 7,460 

North Dakota 539 

Ohio 12,088 

Oklahoma 3,593 

Oregon 2,246 

Pennsylvania 11,774 

Rhode Island 856 

South Carolina 5,529 

South Dakota 736 

Tennessee 5,734 

Texas 18,061 

Utah 1,152 

Vermont 463 

Virginia 8,032 

Washington 3,806 

West Virginia 1,760 

Wisconson 4,559 

Wyoming 442 

aIn 1984, New York implemented a “corporate respo

law that made CEOs personally liable for timely filin

corporate tax returns.

Answer to Question

1

Bracketing Given the size differences between states, 

bracket might be considered. New York an

Page 108 GAO/PEMD-91-10.1.9 Case Stud
might form one pair; Kansas and Arizona a 

pair.

Best Case Three states have missing returns (unfiled

less than 1 percent. These are Hawaii, Kan

New York. Hawaii and Kansas are relative

states and New York has implemented a sp

initiative. Adding states with 1 percent unf

the pool would not add larger states, howe

these are Iowa, Maine, Montana, North Da

South Dakota. It may be that the correlatio

“smaller” states and very low rates of unfil

is a “real” phenomenon that should be exa

the initial cut of less than 1 percent should 

Worst Case Texas is an outlier, with a 15-percent unfil

North Carolina has a 10-percent rate, Arka

9 percent. The next closest states are Ohio 

Oklahoma, with 6 percent each. Selecting 

North Carolina, and Arkansas would be a r

worst-case choice.

Cluster Except for New York, no information is giv

programs or state initiatives. Using only th

the table, several bases for clustering coul

considered. One frequently used basis is “s

problem”: that is, 7 states account for abou

45 percent of all unfiled returns (California

Illinois, New York, Oklahoma, Pennsylvan

Texas). This basis for selection should be 

for this job, however, because there is no m

cluster from the group, except that the sta

among the larger states. The rationale for t

bolstering each individual state’s revenue, 

national pooled aggregrate. Since there are 

smaller, semirural states than big states, th

well-being of individual states would not n

be best served by examining what happen

larger states. Another basis might be a

Page 109 GAO/PEMD-91-10.1.9 Case Stud
crosstabulation of state size and rate of un

returns; here selection of six states could g

reasonable fix on reasons for the problem 

essentially reproduce the strategy used in 

representative sample. We would conclude 

data in the table are not sufficient for draw

cluster sample.

Representative The distribution of unfiled rates is positive

which means that instances are piled up at 

end and scattered out over the high end. W

distribution, “representative” in terms of u

would sensibly mean at the low (1 and less 

1 percent), lower middle (2 and 3 percent)

middle (4, 5, and 6 percent), and high (9, 1

15 percent) points. Assuming state size wo

“second cut” variable, New York (1 percen

California (3 percent), Ohio (6 percent), an

(15 percent) could be one group to study, 

Kansas (less than 1 percent), Massachuset

(2 percent), Oklahoma (6 percent), and Ar

(9 percent) could form a second group of s

states. Together, the eight states also woul

reasonable geographic representativeness

industrialized versus more rural spreads.

Typical A frequency distribution of unfiled rates sh

14 states had rates of 3, which turns out to 

mode and the median for this distribution. 

this category include California, Connectic

Indiana, Kentucky, Maryland, Michigan, M

New Jersey, New Mexico, Oregon, Rhode 

Virginia, and Wyoming. With no other info

(for the purpose of this exercise), if fewer 

case studies were to be made, selecting sta

in size such as Maryland, Michigan, New J

Indiana would make sense.

Page 110 GAO/PEMD-91-10.1.9 Case Stud
Special Interest New York would be of special interest as a 

with a very low rate of unfiled returns. New 

is unique in implementing relevant legislat

might have some national potential.

Question 2 While it might be possible, given the data i

to select states on six of the seven purposi

would the evaluation question itself presen

situation in which we would want to consi

studies at all?

Answer Yes, but not as a stand-alone method. We h

asked to examine the reasons for state var

unfiled returns. One plausible reason is tha

differences are the result of how states sol

monitor compliance, and penalize failure t

could obtain tax codes and procedures for 

examine these, interview selected officials

generate some plausible patterns. Howeve

understanding reasons for behavior as com

filing is well suited for case studies. Expla

could range from (for example) failures of 

returns actually filed, which are quite susc

improvement, to economic cycles that affe

circumstances and that may be less suscep

change. Since the underlying concern is th

states may be asking for federal assistance 

would have resources to handle more of th

needs if they collected revenues owing to 

studies of a representative sample of state

with examination of the special interest st

an efficient strategy for ensuring that we h

comprehensive understanding of what wa

and why. To provide the generalization de

case studies could be followed by a nation

state officials, checking out the findings fr

in-depth studies. Such a sequence could be 

Page 111 GAO/PEMD-91-10.1.9 Case Stud
efficient, since the national survey would n

fishing expedition but targeted to verify in

findings. It also would offer considerable a

that we had accurately determined reason

most states.

Page 112 GAO/PEMD-91-10.1.9 Case Stud
Reports

There are at least six different types of cas

application in evaluation, and their strengt

limitations are different. Choosing an appr

method depends on understanding the eva

question. What is technically right for one 

not necessarily right for another. However

standards that can be applied to all case st

evaluation. Studies that fail to meet them h

questionable merit. These quidelines prese

minimum standard of quality in case study 

taken in conjunction with the guidance in 

Book,” Policy Manual, and Communication

Design 1. Are the evaluation questions stated clea

explicitly? A good study informs the reade

the report about the questions that were a

and the issues that were investigated.

2. Is the case study application clearly des

appropriate? A good case study describes 

study application that was used. It explain

application is appropriate for the kind of e

questions that were answered (descriptive

cause-and-effect). Where several methods 

the relationship of the case study to the ot

methods is clear and appropriate.

3. Was the time span of the study long eno

address the core issues fairly? A good case 

reports how much time the investigation c

relation to the history of the instance or pr

Case studies aiming at a comprehensive an

event as a whole begin as early as possible 

1

These guidelines have been adapted from “Guideline

Reporting Large Case Studies” by John R. Gilbert in D

Hoaglin et al., Data for Decisions: Information Strate

Decisionmakers (Cambridge, Mass.: Abt Books, 1982

and Robert K. Yin, Case Study Research: Design and 

(Beverly Hills, Calif.: Sage, 1984), pp. 140-45.

Page 113 GAO/PEMD-91-10.1.9 Case Stud
Repots

history and continue through its completio

stabilization. Evaluation case studies have 

shorter periods and involved less on-site in

than research case studies characteristical

Readers should recognize, however, that a

shortens, so may the value of the method a

presenting a comprehensive understandin

event as a whole.

4. Is the basis for case selection presented

appropriate for the purpose of the case stu

case study presents the reasons for selecti

instances that were examined. The reason

appropriate for the case study application, 

particular concern if a generalization of th

intended. For assessing the study’s adequa

of site selected is as important as the num

selected. Attention should be paid to the p

setting, to the people who are served by th

and to variations in treatment.

Data Collection 1. Are the methods of data collection prese

they appropriate for the purpose of the cas

Unstructured methods may be appropriate 

illustrative and exploratory applications.

Semistructured approaches may be approp

critical instance case studies involving mu

particularly if more than one investigator w

responsible for collecting data for several 

2. If more than one investigator collected t

how were the other evaluators selected, tr

supervised? There is considerable agreem

consequence of the many variants in data 

for multiple sites is uncertain, but providin

information on the procedures that are use

explanation of the reasons for the approac

essential to a good case study.

Page 114 GAO/PEMD-91-10.1.9 Case Stud
Repots

3. Are information sources described clear

fully? Are they appropriate? A good case s

presents in detail the sources of evidence. 

is greater than that required in other meth

case study report gives the numbers and p

the persons interviewed and the evidence 

were appropriate for the evaluation. The r

should be able to judge from the informati

given in the case study report how credibl

conclusions are in terms of the appropriat

completeness of information sources.

Data Base

Formation and

Data Analysis

Techniques

1. Are the procedures for the formation of 

base described? A good case study describ

data bases were formed and presents a jus

for decisions that were made about the qu

precision, and detail of information in the 

each site.

2. Are the techniques of data-gathering and

data-processing explicitly described? Read

good case study should know how the dat

collected and, step by step, how they were 

If semistructured packets of directions we

guide field workers through the issues, a g

study describes them or includes them in t

appendixes. All the steps of data reduction 

coding are described, along with the basis 

transformations in these steps. The analyti

techniques are explicitly described. What d

sources were used in triangulation? In wha

How were discrepant findings resolved? T

of case study methods partly depends on t

resolution process. At each step, safeguard

have been taken for completeness and the 

of the threat of bias.

Page 115 GAO/PEMD-91-10.1.9 Case Stud
Repots

3. Were there interpretation differences, an

were they resolved? A good case study is e

about differences in the interpretation of e

and events between members of the invest

team and the reviewers of the draft report

study method often uses data that are mor

judgmental, interpretive, and subjective th

methods. The data are often less accessibl

secondary analysis. Thus, a good case stud

argument and evidence more plainly than 

reports have to.

4. If other studies, investigations, or exper

relevant to the issue are available, have th

been presented and reconciled with the ca

findings? A good case study presents the f

conclusions for other studies on the same 

the findings do not converge, the case stud

reconciles or explains the differences as fa

possible. Completeness of information req

step.

Reporting 1. Are methodological strengths and limita

identified clearly? A good case study repor

methodological strengths and limitations f

answering the evaluation questions and ex

tradeoffs that were considered and who in

the decisions. When several decisionmake

involved, a good case study describes the t

decisions each one made and the constrain

decisions.

2. Are the arguments for various resolution

evaluation question presented? Most case 

on topics about which some kind of opinio

formed. In a good case study, the conceptu

framework for organizing the inquiry is qu

about expectations. A good case study ide

Page 116 GAO/PEMD-91-10.1.9 Case Stud
Repots

elements of the issue that was examined a

the initial arguments in favor of the variou

resolutions and the findings of the study th

these resolutions.

3. Are the arguments against various resol

the issue presented? A good case study pre

initial arguments against the various resolu

the issue that was considered. Case study

investigators are supposed to seek evidenc

confirms and evidence that contradicts the

observations and conclusions. Explicitly s

initial arguments for and against various re

helps readers know how thoroughly the in

considered the issues and how thoroughly 

sought evidence on both sides.

4. Does the case study identify the factors 

the phenomena that were observed and st

whether the identification of these factors 

on insight and recognition or on quantitati

techniques? Case studies are undertaken f

explanatory power and their superior abili

identify the reasons for problems and the n

events. A good case study explicitly identi

alternative explanations, lays out the chain 

reasoning, and makes clear which conclus

primarily on the investigators’ insightful re

of patterns of evidence and which have be

recognized in other ways.

Impartiality and

Generalizability

1. What is known about the competence an

impartiality of the investigators? A good ca

provides information about the experience 

investigators with case study methods and 

have written previously about the question

answered. The more evidence there is that 

investigators have had appropriate trainin

Page 117 GAO/PEMD-91-10.1.9 Case Stud
Repots

study methods, and that they have address

issues in ways that seem impartial and are 

reduce bias, the greater confidence the rea

have in the quality of the work. For GAO r

job documentation should contain evidenc

evaluation team as a group possessed the 

required and assurance that there were no

impediments to impartiality among individ

members. For others’ reports we plan to u

studies, we should seek similar assurance 

itself or from knowledgable persons.

2. Are comments on the draft report availa

Perhaps because case studies require mor

other methods, case study reports are som

criticized for failing to be convincing abou

impartiality. One way that a good case stu

this criticism is by the inclusion of a techn

appendix that gives the full comments of t

informants who reviewed the draft.

3. Is there adequate information for judgin

generalizability? The basis for claiming

generalizability is explicit in a good case s

provides the evidence, of whatever type an

that is needed for assessing this claim. In a 

study, generalizations do not exceed the b

these, considering program diversity and h

cases studies were selected.

We provide a checklist of the guidelines di

this appendix in table III.1

Page 118 GAO/PEMD-91-10.1.9 Case Stud
Repots

Table III.1: Checklist for Reviewing Case Study Reports

Yes 

Design

1. Are the evaluation questions stated clearly and

explicitly?

2. Is the case study application clearly described?

3. Was the time span of the study long enough to

address the core issues fairly?

4a. Is the basis for case selection presented?

b. Is it appropriate for the purpose of the case study?

Data collection

1a. Are the methods of data collection presented?

b. Are they appropriate for the purpose of the case study?

2. If more than one investigator collected the data, were

the other evaluators properly selected, trained, and

supervised?

3a. Are information sources described clearly and fully?

b. Are they appropriate?

Data base information and data analysis technique

1a. Are the procedures for the formation of the data base

described?

b. Are they appropriate?

2a. Are the techniques of data gathering and data

processing explictly described?

b. Are they appropriate?

3a. Were there interpretation differences?

b. If so, how were they resolved?

4. If other studies relevant to the issue are available, have

their results been presented and reconciled with the case

study findings?

Reporting

1. Are methodological strengths and limitations identified

clearly?

Page 119 GAO/PEMD-91-10.1.9 Case Stud
Repots

Yes 

2. Are the arguments for various resolutions of the

evaluation question presented?

3. Are the arguments against various resolutions of the

issue presented?

4a. Does the case study identify the factors explaining

the phenomena that were observed?

b. Does the study state clearly whether identification of

these factors was based on insight and recognition or on

quantitative techniques?

Impartiality and generalizability

1. Have proper safeguards to ensure the competence

and impartiality of the investigators been taken?

2. Are comments on the draft report available?

3a. Is there adequate information for judging

generalizability?

b. Have appropriate limitations to generalizations been

observed?

Page 120 GAO/PEMD-91-10.1.9 Case Stud
For readers with an interest in further info

but limited time, a few key references are 

Abert, James G., ed. Program Evaluation a

Research vs. Reality, parts 1-3. New York: 

Dekker, 1979.

Abt, Wendy P., T. Cerva, and T. J. Marx. W

Change? The Effects on Pupils of the Expe

Schools Program. Cambridge, Mass.: 1978.

Abt Associates. First Annual Substantive R

Study of Experimental Schools Projects in 

Schools Serving Rural Areas. Cambridge, M

Acland, Henry. “Are Randomized Experim

Cadillacs of Design?” Policy Analysis, 5 (S

pp. 223-41.

Allison, Graham T. The Essence of Decisio

Explaining the Cuban Missile Crisis. Bosto

Brown, 1971.

Anderson, Scarvia B., et al. “Case Study M

Encyclopedia of Educational Evaluation, p

San Francisco: Jossey-Bass, 1976.

Arrow, Kenneth J. Social Choice and Indiv

Values, 2nd ed. New York: John Wiley and 

Barzun, Jacques, and Henry F. Graff. The M

Researcher, 3rd ed. New York: Harcourt B

Jovanovich, 1977.

Becker, Howard S. “Problems of Inference 

in Participant Observation.” American Soc

Review, 23 (1958), 652-59.

Page 121 GAO/PEMD-91-10.1.9 Case Stud
Belle, Deborah. Lives in Stress: Women an

Depression. Beverly Hills, Calif.: Sage, 198

Berger, Michael A. “Studying Enrollment D

Other Timely Issues) via the Case Survey.”

Educational Evaluation and Policy Analys

(1983), 307-17.

Berman, P., et al. How Schools View and U

School

International, 1981.

Blalock, Hubert M., Jr. Causal Inferences i

Nonexperimental Research. Chapel Hill, N

University of North Carolina Press, 1964.

Bloor, M. “On the Analysis of Observationa

Discussion of the Worth and Use of Induct

Techniques and Respondent Validation.” S

The Journal of the British Sociological Ass

(1978), 545-52.

Bock, Edwin A., ed. Essays on the Case St

in Public

of Administrative Sciences, 1962.

Bock, Edwin A., ed. Essays on the Case St

Method. Syracuse, N.Y.: International Insti

Administration Sciences, The Inter-Univer

Program, November 1971.

Bogdan, Robert. Participant Observation in

Organizational Settings. Syracuse, N.Y.: Sy

University Press, 1972.

Brandt, R. Studying Behavior in Natural Se

York: Holt, Rinehart and Winston, 1972.

Page 122 GAO/PEMD-91-10.1.9 Case Stud
Broadhead, R., and Ray C. Rist. “Gatekeep

Social Control of Social Research.” Social 

23 (1976), 325-26.

Bulmer, M. “Concepts in the Analysis of Q

Data.” Sociological Review, 27 (1979), 651-

Burger, R., and M. Massaglia. RANN Utiliza

Experience: Case Studies 22 Through 31, v

Research Triangle Park, N.C.: Research Tr

Institute, August 1976.

Campbell, Donald T. “Degrees of Freedom 

Case Study.” Comparative Political Studies

178-93.*

Campbell, Donald T., and Julian C. Stanley

Experimental and Quasi-experimental Des

Research. Chicago: Rand McNally, 1963.

Chelimsky, Eleanor. “GAO’s Institute for P

Evaluation.” State Evaluation Network Ne

(1981), 2-5.

Chelimsky, Eleanor, and J. Dahmann. Fina

the Career Criminal Program National Eva

Case Studies of Four Jurisdictions, 1976-79

Va.: MITRE Corp., June 1980.

Chelimsky, Eleanor, and J. Sasfy. The Nati

Evaluation of the Career Criminal Program

and Plan. McLean, Va.: MITRE Corp., May 

Christoph, James B., ed. Cases in Compara

Politics. Boston: Little, Brown, 1965.

Connelly, W. L. Continuity and Change in R

Schooling. Cambridge, Mass.: Abt Associat

Page 123 GAO/PEMD-91-10.1.9 Case Stud
Cook, Thomas D., and Charles S. Reichard

Qualitative and Quantitative Methods in Ev

Research. Beverly Hills, Calif.: Sage, 1979.

Cook, Thomas D., and Donald T. Campbel

Quasi-experimental Design and Analysis Is

Field Settings. Chicago: Rand McNally, 197

Cronbach, Lee. “Remarks to the New Soci

Evaluation Research Society Newsletter, 1 

Cronbach, Lee, et al. Toward Reform of Pr

Evaluation. San Francisco: Jossey-Bass, 19

Datta, Lois-ellin. “Strange Bedfellows.” Am

Behavioral Scientist, 26:1 (1982), 133-44.

David, J. L., and Greene, D. A. Research De

Generalizing from Multiple Case Studies. P

Calif.: Bay Area Research Group, 1981.

Dawson, Judith. “The Validity of Qualitativ

Research.” Paper presented at the America

Educational Research Association meeting

Francisco, Calif., April 1979.

Dawson, Judith A. “Qualitative Research F

What Do We Do to Improve and Estimate T

Validity?” Paper presented at the Annual A

Educational Research Association meeting

York, March 1982.

Denzin, Norman K., ed. Sociological Metho

Sourcebook. New York: McGraw-Hill, 1978

Denzin, Norman K. “The Logic of Naturalis

Inquiry.” Social Forces, 50 (1971), 166-82.

Page 124 GAO/PEMD-91-10.1.9 Case Stud
Denzin, Norman K. The Research Act, 2nd 

York: McGraw-Hill, 1978b.

Derthick, Martha A. New Towns In-Town: 

Federal Program Failed. Washington, D.C.

Institute, 1972.

Dittman, Laura, et al. “Study of Selected C

Head Start Planned Variation, 1969-70. Firs

Report. Case Studies of Children.” Univers

Maryland, College of Education, College P

1971.

Dobbert, Marion Lundy. Ethnographic Res

Theory and Applications for Modern Scho

Societies. New York: Praeger, 1982.

Downey, H. Kirk, and Duane R. Ireland. “Q

Versus Qualitative: The Case of Environme

Assessment in Organizational Studies.” Ad

Science Quarterly, 24 (1979), 630-37.

Farley, Joanne. “Combining Quantitative a

Qualitative Methods in Evaluation Researc

Evaluation Network Newsletter, 1 (1981), 

Farrar, Eleanor, John DeSanctis, and Davi

“Views from Below: Implementation Resea

Education.” Teachers College Record, 82:1 

77-100.

Fiedler, Judith. Field Research: A Manual f

and Management of Scientific Studies in N

Settings. San Francisco: Jossey-Bass, 1978

Fienberg, S. E. “The Collection and Analys

Ethnographic Data in Educational Researc

Anthropology and Education Quarterly, 8 

50-57.

Page 125 GAO/PEMD-91-10.1.9 Case Stud
Filstead, William J., ed. Qualitative Method

Chicago: Markham, 1970.

Finsterbush, Kurt. “Statistical Summary of 

Projects: Lessons on Project Effectiveness

University of Maryland, College Park, Md., 

Glaser, Barney G. Theoretical Sensitivity: A

the Methodology of Grounded Theory. Mil

Calif.: Sociology Press, 1978.

Glaser, Barney G., and Anselm L. Strauss. 

Discovery of Grounded Theory: Strategies 

Qualitative Research. Chicago: Aldine, 196

Glazer, Myron. The Research Adventure: P

Problems of Fieldwork. New York: Random 

1972.

Goetz, J. P., and M. D. LeCompte. “Ethnog

Research and the Problem of Data Reduct

Anthropology and Education Quarterly, 12 

51-70.

Guba, Egon G. Toward a Methodology of N

Inquiry in Educational Evaluation. Los Ang

University of California, Center for the Stu

Evaluation, 1978.

Guba, Egon G. “Criteria for Assessing the

Trustworthiness of Naturalistic Inquiries.”

Educational Communications and Techno

Journal, 8 (1981), 42-54.*

Guba, Egon G., and Yvonna S. Lincoln. Eff

Evaluation: Improving the Usefulness of R

Through Responsive and Naturalistic Appr

San Francisco: Jossey-Bass, 1981.

Page 126 GAO/PEMD-91-10.1.9 Case Stud
Halpern, Edward S. “Auditing Naturalistic 

Some Preliminary Applications.” Paper pre

the American Educational Research Assoc

meeting, Toronto, Canada, April 1983.

Hamilton, D., et al. Beyond the Numbers G

Berkeley, Calif: McCutchan, 1978.

Hargrove, Erwin C. “The Bureaucratic Pol

Evaluation: A Case Study of the Departme

Labor.” Evaluation Studies Review Annual

Howard E. Freeman and Marian A. Solomo

179-288. Beverly Hills, Calif.: Sage, 1981.

Hedrick, Terry E., Robert F. Boruch, and K

“On Ensuring the Availability of Evaluation 

Secondary Analysis.” Evaluation Studies A

4, L. Sechrest et al., eds. Beverly Hills, Cali

1979.

Herriott, Robert E. “Ethnographic Case Stu

Federally Funded Multi-disciplinary Policy 

Some Design and Implementation Issues.”

Anthropology and Education Quarterly, 9 

106-15.

Herriott, Robert E. Federal Initiatives and 

School Improvement. Cambridge, Mass.: A

Associates, 1980.

Herriott, Robert E. Case Study Methods in 

Evaluation and Research: A Synthesis of E

Part I. Final Report. Washington, D.C.: Nat

Institute of Education, June 15, 1982.

Hersen, Michel, and David H. Barlow. Sing

Experimental Designs: Strategies for Study

Behavior Change. New York: Pergamon Pr

Page 127 GAO/PEMD-91-10.1.9 Case Stud
High/Scope Educational Research Founda

National Home Start Evaluation Study, Int

Reports IA and IB, Case Studies. Ypsilanti, 

1972.

Hoaglin, David C., et al. Data for Decisions

Information Strategies for Policy Makers. 

Mass.: Abt Books, 1982.

Holt, Robert T., and John E. Turner, eds. “T

Methodology of Comparative Research.” T

Methodology of Comparative Research, pp

York: Free Press, 1970.

House, Ernest R. The Logic of Evaluative A

Los Angeles: University of California, Cent

Study of Evaluation, 1977.

Huberman, A. M., and M. B. Miles. “Drawin

Meaning from Qualitative Data: Some Tech

Data Reduction and Display.” Quality and 

(1983), 283-339.

Jauch, L., R. Osborn, and T. Martin. “Struc

Content Analysis of Cases.” Academy of M

Review, 5 (1980), 517-25.

Jerome, Chris H. National Home Start Eva

Study, Interim Report III, Case Study Summ

Ypsilanti, Mich.: High/Scope Educational R

Foundation, 1973.

Jerome, Chris H. National Home Start Eva

Study, Interim Report V, Case Studies. Yps

Mich.: High/Scope Educational Research F

1974.

Page 128 GAO/PEMD-91-10.1.9 Case Stud
Jick, Todd D. “Mixing Qualitative and Qua

Methods: Triangulation in Action.” Admini

Science Quarterly, 24 (1979), 602-11.*

Johnson, John M. Doing Field Research. N

Free Press, 1975.

Johnson, Steven D. “On the Use of Qualita

Methods in Policy Research: A Review of T

Multi-site Studies.” Mimeograph, Cornell U

Ithaca, N.Y., February 1980.

Kendall, Patricia L., and Katherine M. Wolf

Analysis of Deviant Cases in Communicati

Research.” Communications Research: 194

eds. Paul Lazarsfeld and Frank Stanton, pp

New York: Harper & Row, 1949.

Kennedy, Mary M. “Generalizing from Sing

Studies.” Evaluation Quarterly, 3 (1979), 6

Khadduri, Jill, and Raymond J. Struyk. “Im

Section 8 Rental Assistance: Translating E

into Policy.” Evaluation Review, 5 (1981), 

Kidder, Louise H. Research Methods in So

Relations, 4th ed. New York: Holt, Rinehar

Winston, 1981.

Kirschen, Etienne S., and Lucien Morissen

Objectives and Instruments of Economic P

Qualitative Planning of Economic Policy, e

Hickman, pp. 111-13. Washington, D.C.: Th

Brookings Institution, 1965.

Kirschen, Etienne S., et al. Economic Polic

Time. Amsterdam, Netherlands: North Hol

Co., 1964.

Page 129 GAO/PEMD-91-10.1.9 Case Stud
Kraft, Richard H. P., et al. Four Evaluation 

Anthropological, Economic, Narrative and 

Chicago: Rand McNally, 1974.

Kratochwill, Thomas R. Single Subject Res

York: Academic Press, 1978.

Kyle, Diane Wells, and Dorene D. Ross. “E

Qualitative Research: Criteria and Their Ap

Paper presented at the Evaluation Networ

meeting, Chicago, Ill., October 1983.

Lazarsfeld, Paul F., and Allen H. Barton. “Q

Measurement in the Social Sciences: Class

Typologies and Indices.” The Policy Scienc

Developments in Scope and Methods, eds. 

Lerner and Harold D. Lasswell, pp. 155-92. 

Calif.: Stanford University Press, 1951.

Lecompte, Margaret D., and Judith P. Goet

“Problems of Reliability and Validity in Eth

Research.” Review of Educational Researc

(Spring 1982), 31-60.

Levine, Harold G. “Principles of Data Stora

Retrieval for Use in Qualitative Evaluation

presented at the American Educational Re

Association meeting, Montreal, Canada, Ap

Lewy, Arieh, and Marvin, Alkin, The Impac

National Evaluation Study: Israel’s Van Le

Los Angeles: University of California,

Center for the Study of Evaluation, Interna

Monograph Series in Evaluation, April 198

Light, Richard J. “Six Evaluation Issues Th

Synthesis Can Resolve Better Than Single 

Issues in Data Synthesis, eds. William H. Y

Page 130 GAO/PEMD-91-10.1.9 Case Stud
Paul M. Wortman, pp. 57-74. San Francisco

Jossey-Bass, 1984.

Light, Richard J., and David B. Pillemer. Su

The Science of Reviewing Research. Camb

Mass.: Harvard University Press, 1984.*

Lijphart, Arend. “Comparative Politics and 

Comparative Method.” American Political 

Review, 65 (1971), 682-93.

Lincoln, Yvonna S. “Strategies for Inquirin

Dependability (Reliability) of Naturalistic 

Paper presented at the Evaluation Researc

meeting, Austin, Texas, September 30-Octo

1981.

Lipset, Seymour Martin, Martin A. Trow, a

Coleman. Union Democracy. Glencoe, Ill.: 

1956.

Lofland, J. Analyzing Social Settings. Belm

Wadsworth, 1971.

Lucas, W. The Case Survey Method of Agg

Case Experience. Santa Monica, Calif.: Ran

McCall, George J., and J. L. Simmons, eds. 

Participant Observation: A Text and Reade

Mass.: Addison-Wesley, 1969.

McClintock, Charles C., Diane Brannon, an

Maynard-Moody. “Applying the Logic of Sa

Surveys to Qualitative Case Studies: The C

Method.” Administrative Science Quarterly

612-29.

McDaniels, Garry, et al. “Case Studies of C

Head Start Planned Variation, 1970-71.” Un

Page 131 GAO/PEMD-91-10.1.9 Case Stud
Maryland, College of Education, College P

1972.

McGowan, Eleanor, and David Cohen. “Ra

Fantasies.” Policy Sciences Journal, 1 (197

Marsh, Robert M. “The Bearing of Compar

Analysis on Sociological Theory.” Social F

(1964), 191-96.

Miles, Matthew B. “Qualitative Data as an A

Nuisance.” Administrative Science Quarter

(1979), 590-601.

Miles, Matthew B. “A Mini-Cross-Site Anal

American Behavioral Scientist, 26:1 (1982)

Miles, Matthew B., and A. M. Huberman. Q

Data Analysis: A Sourcebook of New Meth

Beverly Hills, Calif.: Sage, 1984.*

Mill, John Stuart. A System of Logic, 8th ed

England: Longmans Green, 1972.

Mintzberg, Henry. “The Emerging Strategy 

Research.” Administrative Science Quarter

(1979), 582-89.

Moberg, P. “The Collection and Analysis o

Data in Evaluation Research.” Paper prese

National Conference on Evaluation in Alco

Abuse, and Mental Health Programs, Wash

D.C., 1974.

Mosteller, F., and D. L. Wallace. Inference 

Disputed Authorship: The Federalist. Read

Addison-Wesley, 1964.

Page 132 GAO/PEMD-91-10.1.9 Case Stud
Mosteller, F., et al. “The Pre-Election Polls 

Social Science Research Council Bulletin, 

Mulhauser, Frederick, “Ethnography and P

Making: The Case of Education.” Human

Organization, 3 (1975), 311-15.

National Science Foundation. Case Studie

Education, Vols. 1 and 2. Washington, D.C

Government Printing Office, 1978.

National Science Foundation. Cooperative 

National Study of University and Industry

Researchers. Assessment of the Industry/U

Cooperative Research Projects Program, v

Washington, D.C.: November 1984.

Neustadt, R. E., and H. V. Fineberg. The Sw

Affair: Decision Making on a Slippery Dise

Washington, D.C.: U.S. Government Printin

1978.

Office of Technology Assessment. Assessi

Efficacy and Safety of Medical Technologi

Washington, D.C.: U.S. Government Printin

1978.

Paige, Glenn. “Problems and Use of the Sin

Political Research.” Ph.D. diss., Northwest

University, Evanston, Ill., 1959.

Patton, Michael Quinn. Qualitative Evaluat

Methods. Beverly Hills, Calif.: Sage, 1980.*

Pelto, Pertti J., and Gretel H. Pelto. Anthro

Research: The Structure of Inquiry, 2nd ed

Cambridge, England: Cambridge Universit

1978.

Page 133 GAO/PEMD-91-10.1.9 Case Stud
Philadelphia Inquirer. “Anti-Terrorism Vid

Personnel Say FAA-Ordered Training Prog

Boring, Ineffective.” April 29, 1986.

Pierce, William Spangar. Bureaucratic Fail

Public Expenditure. New York: Academic 

Piore, Michael J. “Qualitative Research Te

Economics.” Administrative Science Quart

(1979), 560-69.

Platt, J. R. “Strong Inference.” Science, 146 

347-53.

Plog, Michael. “The Use of Case Study Met

State Evaluation Network Newsletter, 1:2 

Popkewitz, Thomas S., and Robert B. Taba

The Study of Schooling: Field-Based Meth

Educational Research and Evaluation. New 

Praeger, 1981.

Pressman, Jeffrey L., and Aaron Wildavsky

Implementation. Berkeley, Calif.: Universi

California Press, 1973.*

Przeworski, Adam, and Henry Teune. The 

Comparative Social Inquiry. New York: Joh

and Sons, 1970.

Rist, Ray C. “On the Relations Between Ed

Research Paradigms: From Disdain to Det

Anthropology and Education Quarterly, 8 

42-49.

Rist, Ray C. Earning and Learning: Youth E

Policies and Programs. Beverly Hills, Calif

1981a.

Page 134 GAO/PEMD-91-10.1.9 Case Stud
Rist, Ray C. “On the Utility of Ethnographi

for the Policy Process.” Urban Education, 

485-94.

Rist, Ray C. “Mandating Collaboration Thr

Federal Legislation: YEDPA and the CETA

Linkage.” Research in Sociology of Educat

Socialization, vol. 3, ed. R. Corwin, pp. 187

York: JAI Press, 1982a.

Rist, Ray C. “On the Application of Ethnog

Inquiry to Education: Procedures and Poss

Journal of Research in Science Teaching, 1

(1982b), 439-50.

Rist, Ray C. “Beyond the Quantitative Cul-

Qualitative Perspective on Youth Employm

Programs.” Applied Poverty Research, eds

Goldstein and Stephen M. Sachs, pp. 123-3

N.J.: Rowman and Allanheld, 1984.

Roncek, Dennis W., and Gail Weinberger.

“Neighborhoods of Leased Public Housing

Evaluation Review, 5 (1981), 231-44.

Sanday, Peggy R. “The Ethnographic Parad

Administrative Science Quarterly, 24 (1979

Scheirer, Mary Ann, and Eva L. Resmovic. 

the Degree of Program Implementation.” E

Review, 7 (1983), 599-633.

Scriven, Michael. “Objectivity and Subjecti

Educational Research.” Philosophical Red

Educational Research, ed. L. G. Thomas. C

University of Chicago Press, 1972.

Searle, Barbara (ed.). Evaluation in World 

Education Projects: Lessons from Three C

Page 135 GAO/PEMD-91-10.1.9 Case Stud
Report EDT5. Washington, D.C.: World Ba

July 1985.

Sechrest, Lee, ed. Unobtrusive Measureme

San Francisco, Calif.: Jossey-Bass, 1979.

Shapiro, E. “Educational Evaluation: Rethi

Criteria of Competence.” School Review, 8

523-49.

Sieber, Sam D. “The Integration of Fieldwo

Survey Methods.” American Journal of Soc

(1973), 1335-59.

Sjoberg, Gideon. “The Comparative Metho

Social Sciences.” Philosophy of Science, 2

106-17.

Smith, Allen G., and Karen S. Louis, eds.

“Multimethod Policy Research: Issues and

Applications.” American Behavioral Scient

(1982), 1-144.

Smith, Louis M. “An Aesthetic Education W

for Administrators: Some Implications for 

Case Studies.” Paper presented at the Ame

Educational Research Association meeting

Ill., 1974.

Smith, Louis M. “An Evolving Logic of Part

Observation, Educational Ethnography an

Case Studies.” Review of Research in Educ

(1979), 316-77.

Smith, Louis M., and S. Schumacher. Exten

Trials of the Aesthetic Education Program

Qualitative Description. St. Louis: CEMRE

Page 136 GAO/PEMD-91-10.1.9 Case Stud
Snow, Richard E. “Representative and

Quasi-representative Designs for Research 

Teaching.” Review of Educational Researc

(1974), 265-91.

Spirer, Janet E. The Case Study Method: G

Practices and Applications for Vocational 

Columbus, Ohio: National Center for Rese

Vocational Education, 1980.

Stake, Robert E. “The Case Study Method 

Inquiry.” Educational Researcher, 7 (1978)

Stake, Robert E., and J. Easley, eds. Case S

Science Education. Urbana, Ill.: Center for

Instructional Research and Curriculum Ev

1978.

Stein, Harold. Public Administration and P

Development: A Case Book. New York: Ha

Brace Jovanovich, 1952.

Stenhouse, Lawrence. “Case Study in Com

Education: Particularity and Generalizabil

Comparative Education, 15:1 (1979), 5-10.

Trend, M. G. “On the Reconciliation of Qua

Quantitative Analyses: A Case Study.” Hum

Organization, 37 (1978), 345-54.

Trow, Martin. “Comment on Participant O

and Interviewing: A Comparison.” Human

Organization, 16 (1957), 33-35.

Turner, B. A. “Some Practical Aspects of Q

Data Analysis: One Way of Organizing the 

Processes Associated with the Generation 

Grounded Theory.” Quality and Quantity, 1

225-47.

Page 137 GAO/PEMD-91-10.1.9 Case Stud
University of Sussex, Social Science Resea

Unit. Success and Failure in Industrial Inn

London, England: Center for the Study of I

Innovation, 1972.

U.S. General Accounting Office. Lands in t

Chelan National Recreation Area Should B

to Private Ownership, GAO/CED-81-10. Wa

D.C.: January 22, 1981.

U.S. General Accounting Office. Housing B

Activity in Pittsburgh: A Case Study, CED-

Washington, D.C.: March 24, 1982.

U.S. General Accounting Office. Housing B

Activity in Seattle: A Case Study, CED-82-6

Washington, D.C.: March 30, 1982.

U.S. General Accounting Office. Review of 

Operations of the Sea Island Comprehensi

Care Corporation and the Franklin C. Fett

Health Center, HRD-82-69. Washington, D.

1982.

U.S. General Accounting Office. Housing B

Activity in Dallas: A Case Study, CED-82-7

Washington, D.C.: April 30, 1982.

U.S. General Accounting Office. Content A

Methodology for Structuring and Analyzin

Material. Methodology transfer paper 3. W

D.C.: June 1982.

U.S. General Accounting Office. Cleaning U

Environment: Progress Achieved but Majo

Unresolved Issues Remain, vols. 1 and 2, C

Washington, D.C.: July 21, 1982.

Page 138 GAO/PEMD-91-10.1.9 Case Stud
U.S. General Accounting Office. Block Gra

Housing: A Study of Local Experiences an

GAO/RCED-83-21. Washington, D.C.: Dece

1982.

U.S. General Accounting Office. How Well 

Military Services Perform Jointly in Comb

Joint Test-and- Evaluation Program Provid

Credible Answers, GAO/PEMD-84-3. Wash

D.C.: February 22, 1984.

U.S. General Accounting Office. An Evalua

1981 AFDC Changes: Initial Analyses,

GAO/PEMD-84-6. Washington, D.C.: April 2

U.S. General Accounting Office. Need to B

Assess Consequences Before Reducing Ta

Assistance, GAO/GGD-84-13. Washington, 

April 5, 1984.

U.S. General Accounting Office. Implemen

Status of the Office of Management and Bu

Circular A-76 Program at the Department o

Interior’s National Park Service and Burea

Reclamation, GAO/RCED-85-56. Washingto

March 15, 1985.

U.S. General Accounting Office. Projects F

Northeast Texas by the Emergency Jobs

Appropriations Act of 1983, GAO/HRD-85-

Washington, D.C.: March 26, 1985.

U.S. General Accounting Office. Projects F

the Montgomery, Alabama, Metropolitan A

Emergency Jobs Appropriations Act of 198

GAO/HRD-85-59. Washington, D.C.: May 7, 

U.S. General Accounting Office. Projects F

Fresno County, California by the Emergen

Page 139 GAO/PEMD-91-10.1.9 Case Stud
Appropriations Act of 1983, GAO/HRD-85-

Washington, D.C.: August 27, 1985.

U.S. General Accounting Office. Projects F

South Central Georgia by the Emergency J

Appropriations Act of 1983, GAO/HRD-85-

Washington, D.C.: September 25, 1985.

U.S. General Accounting Office. Improving 

and Staffing Practice Can Increase Produc

Reduce Costs in SSA’s Atlanta Region,

GAO/GGD-85-85. Washington, D.C.: Septem

1985.

U.S. General Accounting Office. Emerging 

Export Competition: A Case Study of the B

Market, GAO/NSIAD-85-121. Washington, D

September 26, 1985.

U.S. General Accounting Office. Informatio

Forest Service’s Efforts to Control the Spr

Western Spruce Budworm in the Carson N

Forest, GAO/RCED-86-6. Washington, D.C

October 30, 1985.

U.S. General Accounting Office. Departme

Commerce’s Second-Year Efforts to Imple

Federal Manager’s Financial Integrity Act,

GAO/RCED-86-21. Washington, D.C.: Nove

1985.

U.S. General Accounting Office. Emergenc

of 1983: Projects Funded in the Lawrence-

Massachusetts, Area, GAO/HRD-86-30. Wa

D.C.: December 6, 1985.

U.S. General Accounting Office. Using Stat

Sampling. Methodology transfer paper 6. W

D.C.: May 15, 1986.

Page 140 GAO/PEMD-91-10.1.9 Case Stud
U.S. General Accounting Office. Bigeye Bo

Evaluation of DOD Chemical and Develop

GAO/PEMD-86-12BR. Washington, D.C.: M

U.S. General Accounting Office. Foreign

Representation: Former High-Level Federa

Representing Foreign Interests,

GAO/NSIAD-86-175BR. Washington, D.C.: 

1986.

U.S. General Accounting Office. Social Sec

Improved Telephone Accessibility Would B

the Public. GAO/HRD-85-86. Washington, D

August 29, 1986.

U.S. General Accounting Office. The Natio

Quality: Key Unanswered Questions Abou

of Rivers and Streams, GAO/PEMD-86-6. W

D.C.: September 19, 1986.

U.S. General Accounting Office. Cargo Imp

Customs Need to Better Assure Complianc

Trade Laws and Regulations, GAO/GGD-86

Washington, D.C.: December 1986.

U.S. General Accounting Office. Water Qu

Evaluation Method for the Construction G

Program—Methodology, GAO/PEMD-87-4A

Washington, D.C.: December 17, 1986a.

U.S. General Accounting Office. Water Qu

Evaluation Method for the Construction G

Program—Case Studies. GAO/PEMD-87-4B

Washington, D.C.: December 17, 1986b.

U.S. General Accounting Office. Medical M

Six Case Studies Show Claims and Insuran

Still Rise Despite Reforms, GAO/HRD-87-2

Washington, D.C.: December 31, 1986.

Page 141 GAO/PEMD-91-10.1.9 Case Stud
U.S. General Accounting Office. Parks and

Recreation: Construction Contract at Jean 

National Historical Park, GAO/RCED 86-23

Washington, D.C.: September 26, 1987.

U.S. General Accounting Office. Parks and

Recreation: Concerns Raised About Nation

Service Actions at Delaware Water Gap,

GAO/RCED-87-24BR. Washington, D.C. Oc

1987.

VanderPutten, Elizabeth. “Toward a Theor

Loosely Coupled Systems.” Ph.D. diss., Ge

Washington University, Washington, D.C., 

Van Maanen, John. “The Fact of Fiction in

Organizational Ethnography.” Administrat

Quarterly, 24 (1979), 539-50.

Van Maanen, John. Qualitative Methodolog

Hills, Calif.: Sage, 1983.

Vidich, Arthur J., and Gilbert Shapiro. “A C

of Participant Observation and Survey Dat

American Sociological Review, 20 (1955), 

Voss, Harwin L. “Pitfalls in Social Research

Study.” American Sociologist, 1 (1966), 136

Wax, Rosalie. Doing Field Work: Warnings 

Advice. Chicago: University of Chicago Pre

Weatherly, R., and M. Lipsky. “Street Level

Bureaucrats and Institutional Innovation:

Implementing Special-Education Reforms.

Educational Review, 47 (May 1977), 171-97

Page 142 GAO/PEMD-91-10.1.9 Case Stud
Webb, Eugene, and Karl E. Weick. “Unobtr

Measures in Organizational Theory: A Rem

Administrative Science Quarterly, 24 (1979

Weiss, Robert S., and M. Rein. “The Evalua

Broad Aim Programs: Experimental Desig

Difficulties and an Alternative.” Administr

Science Quarterly, 15 (1980), 97-109.

Wilson, Steve. “The Use of Ethnographic M

Educational Evaluation.” Human Organiza

(1977), 2.

Wilson, Steve. “Explorations of the Useful

Case Study Evaluations.” Evaluation Quar

(1979), 446-59.

Yeaton, William H., “The Case Study Crisis

Answers.” Administrative Science Quarter

(1981b), 58-66.

Yeaton, William H., Eveleen Bingham, and 

Heald. “The Difference That Quality Make

of Literature Reviews.” Sociological Metho

Research, 5 (1976), 139-56.

Yeaton, William H., and Paul M. Wortman. 

Data Synthesis. San Francisco: Jossey-Bas

Yin, Robert K. Case Study Research: Desig

Methods, rev. ed. Beverly Hills, Calif.: Sage

Yin, Robert K., et al. A Review of Case Stu

Technological Innovation in State and Loc

Santa Monica, Calif.: Rand, 1976.

Yin, Robert K., and Karen A. Heald. “Using 

Survey Method to Analyze Policy Studies.”

Administrative Science Quarterly, 20 (1975

Page 143 GAO/PEMD-91-10.1.9 Case Stud
Yin, Robert K., and Ingrid Heinsohn. Using 

Research Sponsored by the AOA. Case Stu

Transportation Service for the Elderly. Wa

D.C.: American Institute for Research, 198

Zelditch, Morris J. “Some Methodological P

Field Studies.” American Journal of Sociol

(1962), 566-76.

Page 144 GAO/PEMD-91-10.1.9 Case Stud
Backfill Techniques Techniques used in cumulative case studie

information needed if the study is to be us

aggregation; these techniques include, for 

obtaining missing information from the au

how instances studied were identified and 

bases for instance selection.

Bias The extent to which a measurement, samp

analytic method systematically underestim

overestimates the true value of an attribut

Case Study A method for learning about a complex ins

based on a comprehensive understanding 

instance, obtained by extensive descriptio

analysis of the instance, taken as a whole a

context.

Convenience Sample Instances selected where the only basis is 

or ease of data collection. Rarely useful in 

and is usually hazardous.

Construct An attribute, usually unobservable, such a

educational attainment or socioeconomic 

is represented by an observable measure.

Construct Validity The extent to which a measurement metho

accurately represents a construct and prod

observation distinct from that produced by 

of another construct.

External Validity The extent to which a finding applies (or c

generalized) to persons, objects, settings, 

other than those that were the subject of s

Page 145 GAO/PEMD-91-10.1.9 Case Stud
Focused Interview An interview organized around several pre

questions or topics but providing some fle

the sequencing of the questions and witho

predetermined set of response categories 

data elements to be obtained.

Generalizability Used interchangeably with “external validi

Internal Validity The extent to which the causes of an effec

established by an inquiry.

Longitudinal Data Sometimes called “time series data,” obser

collected over a period of time; the sample 

or cases) may or may not be the same each 

Matrix of Categories A method of displaying relationships amon

analyzing case study data that shows whet

changes in categories or degrees along one 

are associated with changes in the categor

another dimension.

Normative Question A type of evaluation question requiring com

between what is happening (the condition

and expectations or standards for what sh

happening (the criterion).

Open-Ended

Interview

An interview in which, after an initial or le

question, subsequent questions are determ

topics brought up by the person being inte

the concerns discussed, their sequence, an

information obtained are not predetermine

discussion is unconstrained, able to move 

unexpected directions.

Page 146 GAO/PEMD-91-10.1.9 Case Stud
Outliers Instances that are aberrant or do not fit wi

instances; instances that, compared to oth

of a population, are at the extremes on rel

dimensions.

Program

Effectiveness

Evaluation

The application of scientific research meth

estimate how much observed results, inten

are caused by program activities. Effect is 

cause by design and analyses that compar

results with estimates of what might have 

observed in the absence of the program.

Program Evaluation The application of scientific research meth

assess program concepts, implementation

effectiveness.

Purposive Sample Instances appropriately selected to answe

evaluation questions, on various systemati

such as best or worst practices; a judgmen

If conducted systematically, can be widely 

evaluation.

Qualitative Data Information based on judgments (such as 

estimated speed of a UFO) which may be e

in numerical or nonnumerical ways and da

not be based on judgments (such as state o

are not meaningfully expressed numerical

sources are often textual and observationa

expressed in words.

Quantitative Data Information based on measures that do no

judgments and that are meaningfully meas

are usually expressed numerically and ofte

continuous rather than discrete or categor

Page 147 GAO/PEMD-91-10.1.9 Case Stud
of measurement and scales with interval o

properties.

Reliability The extent to which a measurement proce

similar results on repeated observations o

condition or event.

Representative

Sample

A sample that has approximately the same

distribution of characteristics as the popul

which it was drawn.

Simple Random

Sample

A method for drawing a sample from a pop

such that all samples of a given size have e

probability of being drawn.

Structured Interview An interview in which questions to be aske

sequence, and the detailed information to 

are all predetermined; used where maximu

consistency across interviews and intervie

needed.

Triangulation The combination of methodologies in the s

same phenomenon or construct; a method 

establishing the accuracy of information b

comparing three or more types of indepen

of view on data sources (for example, inte

observation, and documentation; different

investigations; different times) bearing on 

findings. Akin to corroboration and an ess

methodological feature of case studies.

Page 148 GAO/PEMD-91-10.1.9 Case Stud
Yoked Concurrent with. For example, data collec

analyses in case studies are iterative and

concurrent—that is, are yoked.

Page 149 GAO/PEMD-91-10.1.9 Case Stud
This is a flexible series continually being a

updated. The interested reader should inqu

the possibility of additional papers in the s

The Evaluation Synthesis. Transfer paper 

formerly methods paper I.

Content Analysis: A Methodology for Struc

Analyzing Written Material. Transfer paper 

formerly methodology transfer paper 3.

Designing Evaluations. Transfer paper 10.

formerly methodology transfer paper 4.

Using Structured Interviewing Techniques

paper 10.1.5, formerly methodology transf

Using Statistical Sampling. Transfer paper 

formerly methodology transfer paper 6.

Developing and Using Questionnaires. Tra

10.1.7, formerly methodology transfer pap

Case Study Evaluations. Transfer paper 10

formerly methodology transfer paper 9.

Prospective Evaluation Methods: The Pros

Evaluation Synthesis. Transfer paper 10.1.

methodology transfer paper 10.

(973317) Page 150 GAO/PEMD-91-10.1.9 Case Stud
Ordering Information

The first copy of each GAO report and 

is free. Additional copies are $2 each. 

should be sent to the following addres

accompanied by a check or money orde

out to the Superintendent of Documen

necessary. VISA and MasterCard credi

are accepted, also. Orders for 100 or m

copies to be mailed to a single address 

discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax numbe

(301) 258-4066, or TDD (301) 413-000

Each day, GAO issues a list of newly a

reports and testimony. To receive fac

copies of the daily list or any list from 

30 days, please call (202) 512-6000 us

touchtone phone. A recorded menu w

information on how to obtain these lis

For information on how to access GAO 

on the INTERNET, send an e-mail mes

"info" in the body to:

info@www.gao.gov
Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Postage & F

GAO

Permit No