Division

November 1990 Prospective

Evaluation Meth

The Prospective

Evaluation Synthe

GAO/PEMD-10.1.10

GAO assists congressional decisionmakers 

deliberative process by furnishing analytic

information on issues and options under

consideration. Many diverse methodologie

needed to develop sound and timely answ

questions that are posed by the Congress. 

GAO evaluators with basic information ab

more commonly used methodologies, GAO

guidance includes documents such as met

transfer papers and technical guidelines.

This methodology transfer paper on prosp

evaluation synthesis focuses on a systema

for providing the best possible information 

other things, the likely outcomes of propo

programs, proposed legislation, the adequ

proposed regulations, or top-priority probl

paper uses a combination of techniques th

answer prospective questions involving th

of alternative proposals and projections of 

kinds. As GAO receives more requests for

assessments about the implications of futu

occurrences, evaluators should find this sy

approach a beneficial tool to facilitate them 

work.

The present transfer paper is one of a serie

issued by the Program Evaluation and Met

Division (PEMD). The purpose of the serie

provide GAO evaluators with guides to var

aspects of audit and evaluation methodolo

illustrate applications, and to indicate whe

detailed information is available. This pap

originally authored by Lois-ellin Datta. Thi

version supersedes the July 1989 edition.

Page 1 GAO/PEMD-10.1.10 Prospecti
We look forward to receiving comments fr

readers of this paper. They should be addr

Eleanor Chelimsky at 202-275-1854.

Werner Grosshans

Assistant Comptroller General

Office of Policy

Eleanor Chelimsky

Assistant Comptroller General

for Program Evaluation and

 Methodology

Page 2 GAO/PEMD-10.1.10 Prospecti
Page 3 GAO/PEMD-10.1.10 Prospecti
Preface 

Chapter 1 

What Is a

Prospective

Question?

Chapter 2 

The Need for

Systematic

Methods for

Answering

Forward-Looking

Questions

Chapter 3 

Prospective

Methods and the

Prospective

Evaluation

Synthesis Broadly

Defined

When the PES Is and Is Not Appropriate 

The PES and the Recommendations GAO

Makes

Chapter 4 

The PES: Initial

Steps

Defining the Problem 

Selecting Alternatives to Evaluate 

Chapter 5 

The PES: Middle

and Final Steps

The Conceptual Analysis 

The Operational Analysis 

Testing the Model 

Page 4 GAO/PEMD-10.1.10 Prospecti
Presenting the Results 

Chapter 6 

Variants of the

PES

Targeted PES 

Variants Using Other Sources of Informati

Appendixes Appendix I: A Brief History of the PES and

Some Other Prospective Methods

Appendix II: Data Quality Judgment Mode

Appendix III: A Project Evaluation Profile 

References 

Papers in This Series 

Tables Table 1.1: Types of Forward-Looking

Questions and What We Are Asked to D

Table 1.2: Features of Retrospective and

Prospective Methods

Table 3.1: Some Prospective Methods 

Table 3.2: Illustrations of Where a PES Mig

Strengthen Our Recommendations

Table 3.3: Situations in Which a PES Shoul

and Should Not Be Considered

Table 4.1: Steps in the Basic PES Approach

and Persons Involved

Table 4.2: Step 1: Defining the Problem 

Table 4.3: Step 2: Selecting Alternatives to

Evaluate

Table 5.1: Step 3: Conceptual Analysis 

Table 5.2: Step 4: Operational Analysis 

Table 5.3: Step 5: Testing Key Assumptions

Against Existing Evidence

Table 5.4: Step 6: Presenting Results 

Table 6.1: Targeted PES and Related Critic

Issues

Table II.1: Advantages and Disadvantages 

Four Data Quality Judgment Models

Table II.2: Example of a Fatal Flaws Analy

Page 5 GAO/PEMD-10.1.10 Prospecti
Figures Figure 3.1: The Triad of Analysis 

Figure 5.1: Underlying Conceptual Model o

the First Bill

Figure 5.2: Underlying Conceptual Model o

Program A in the Second Bill

Figure 5.3: Underlying Conceptual Model o

Program B in the Second Bill

Figure 5.4: Underlying Operational Model 

Program B in the Second Bill

Table 5.5: Example of Presenting PES

Findings

Abbreviations

GAO U.S. General Accounting Office

PES Prospective evaluation synthesis

Page 6 GAO/PEMD-10.1.10 Prospecti
Page 7 GAO/PEMD-10.1.10 Prospecti
Why should a GAO evaluator read a paper 

prospective evaluation synthesis (PES)? G

evaluators must know about methods such 

because the changing nature of our work r

to be familiar with the strengths and limita

the applicability, of ways to answer questi

with the future. The PES is one these meth

GAO is increasingly being asked to answer 

about the future that involve analyses of a

proposals and projections of various kinds

support GAO’s capacity to answer these qu

well, our policy and project manuals have 

expanded to discuss, for example, differen

forecasting and formal modeling approach

standards for carrying these out. This is be

systematic methods for dealing with quest

the future can be more efficient and yield 

better-documented answers than more inf

methods do.

Many methods exist to deal with forward-l

future-oriented questions. Collectively, the

referred to as prospective methods to disti

them from approaches designed to answer 

about what is happening now or what has 

in the past—that is, retrospective methods

Among the prospective methods, we have 

focus here on the prospective evaluation s

GAO developed the PES as a systematic m

meeting congressional requests for analyzi

proposed legislation and helping identify t

problems. Other applications of the PES m

the analysis of recommendations in draft G

and in assessing the adequacy of proposed

regulations.

Page 8 GAO/PEMD-10.1.10 Prospecti
This paper shows how the tools of evaluat

methodology can be applied in order to pr

best possible information prospectively on 

outcomes of proposed programs. A PES m

conducted through the comparison of poli

program alternatives, although it is also us

focused on a single policy or program. It is 

perform when an adequate data base alrea

Fortunately, data bases concerning propos

programs frequently do exist, primarily be

problems are rarely new. Often they have b

addressed by past programs whose experi

be drawn upon for the PES.

In essence, a PES is a combination of the f

activities: (1) a careful, skilled textual anal

proposed program, designed to clarify the 

goals of that program and what is assumed 

results, (2) a review and synthesis of evalu

studies from similar programs, and (3) sum

judgments of likely success, given a future 

that is not too different from the past. In th

the PES resembles the evaluation synthesi

except that the focus of the PES is on how 

studies cast light on the potential for succe

proposed programs, as opposed to reachin

conclusions about the actual performance 

programs.

Three other points emerge from the exper

PES. First, the PES may call for a greater s

than the evaluation synthesis. The latter in

comprehensive review of all existing studi

can allow us to generalize quite broadly. T

time-driven nature of PES may restrict it to 

narrower focus and the use of strategies, s

sampling, to balance resources and the ne

external validity. Second, legislators and

congressional staff who have received a P

Page 9 GAO/PEMD-10.1.10 Prospecti
as a useful tool. From the congressional pe

a PES means that expert design assistance 

for a new program at the point when it is m

and when it can help convince others of th

logic and likely success of the program. Th

public policy perspective, providing under

ahead of time about how a program is like

renders an important service by validating 

soundness of what is to be undertaken and 

increasing its chances for success.

To understand prospective questions, it ca

to begin with some examples of GAO repo

reported that the passage of a proposed bi

would probably open some jobs to women 

currently closed and that might otherwise 

closed after the review required by the sec

the Department of Defense was finished. (

General Accounting Office, 1988h) GAO al

the Congress about difficulties with specif

Drug Administration forecasts. These fore

predicted the increase in the number of

medical-device problems that would be re

hospitals and the number of agency staff t

be necessary to analyze the reports of thos

under the proposed Medical Devices Impr

Act of 1988. We concluded that these forec

biased and not representative of what wou

generated from data obtained from U.S. ho

general. (U.S. General Accounting Office, 

GAO found in yet another study that the In

Revenue Service needed to review its entir

revenue-estimating process in order to val

assumptions used to better reflect actual h

trends. (U.S. General Accounting Office, 1

These reports illustrate the prospective, or

forward-looking, questions that GAO is oft

Page 10 GAO/PEMD-10.1.10 Prospecti
deal with.1 As table 1.1 shows, at least four 

forward-looking questions can be identifie

we have issued already, requests that have 

in ways other than through reports, and ou

policies regarding our recommendations.

Table 1.1: Types of

Forward-Looking

Questions and What We

Are Asked to Do

What we are ask

Question type

Critique

others’

analyses

Do 

ou

Anticipate the future 1. How well has

the

administration

projected

future needs,

costs, and

consequences?

3. W

futu

cos

con

Improve the future 2. What is the

potential

success of an

administration

or

congressional

proposal?

4. W

of a

the 

pot

suc

the 

ap

GA

rec

1

GAO does not normally make forecasts, although we 

on special request (for example, in response to our a

under requests related to Gramm-Rudman-Hollings). 

evaluate the forecasting process and the methodolog

past work has indicated, for example, that agencies c

forecast accuracy by using better techniques and val

predictions. The same points apply to modeling. It sh

noted that other agencies are frequently called upon 

forward-looking analysis. The Office of Management 

requires regulatory impact analysis before any major 

regulation is put into effect. And the Congressional B

required to “price out” all new legislation. Thus, there 

applications and methods in this prospective area.

Page 11 GAO/PEMD-10.1.10 Prospecti
The use of the PES described in this paper 

consistent with GAO’s policy on forward-l

questions and on the methodology to be us

developing recommendations. This policy 

in the General Policy Manual, chapter 10.0

chapters 12.10 and 12.18 of the Communic

Manual. These latter chapters specify, for 

the procedures that are to be followed wh

with programs and policies under legislativ

consideration or recommendations asserti

possibility of budgetary savings. Particular

in the General Policy Manual are the sectio

formal modeling, economic optimizing, an

forecasting.

1. How well has the administration project

estimated the future needs, costs, and con

In responding to such a forward-looking qu

GAO may need to address issues such as t

following:

• How well has it anticipated, for example, r

staff needs or emerging problems?

• Are the methods for projection sound?

• Are the data bases reliable and adequate?

• Are the assumptions explicit?

• Are they reasonable?

• Have the projections been overgeneralized

• Are there feasible improvements to the pro

the reporting?

• Are better estimates, or better-reported es

available?

In the case of repeated or regular forecast

have to examine whether the relevant agen

systematically evaluates their accuracy an

whether the error rates are acceptable and 

bias. Further, when the administration pub

claims about the likely consequences of its 

Page 12 GAO/PEMD-10.1.10 Prospecti
proposed activities, we may examine whet

are methodologically sound and properly p

And, when the administration has sought t

prevent action, using projections or estima

future costs or consequences, we may dete

whether these projections, too, are sound 

accurately reported.

2. What is the potential for the success of a

congressional or administration proposal? 

answering this type of inquiry, GAO could 

following questions:

• Given the characteristics of new or amend

legislation being considered by the Congre

likely is it that a bill will achieve its stated 

• What features might be modified to improv

chances of success?

• Are there side effects or pitfalls known fro

experience that could be remedied prospe

• When the administration initiates a new po

legislation by proposing a set of activities, 

is it that these will work?

• What changes that might be made before t

is put into effect would better achieve the 

results?

• What unidentified dangers should be cons

before action is taken?

3. What are future needs, costs, and conse

many areas, GAO is asked to anticipate the 

analyses such as the costs of future illegal

immigration, the flow of future legal immig

future costs of the AIDS epidemic, military 

needs, and the adequacy of stockpiles of m

critical to the national defense. According 

policies, we are expected to use state-of-th

methods for making any quantitatively bas

forecasts or projections and to use due pro

Page 13 GAO/PEMD-10.1.10 Prospecti
care in applying qualitative approaches, su

expert panels. We could check on whether 

used the technically most solid procedures

considered alternative methods, and appli

reported properly the ranges of uncertaint

in any prediction, using approaches such a

sensitivity analyses to test systematically t

of different assumptions.

4. What course of action should we recom

most likely to succeed in addressing the pr

identify? Our policies require us to careful

alternative actions resulting from our findi

weigh the costs of these alternatives and t

likelihood of success before we present th

matters for consideration or as recommen

This requirement distinguishes GAO from 

congressional support agencies. They follo

policy analysis approach of presenting opt

not make recommendations. GAO goes thr

analytic steps and makes its choice of the 

solution. Further,

GAO systematically follows up and reports 

acceptance of the recommendations it ma

reports. In this context, procedures for dev

alternatives and selecting recommendation

seen as the most crucial part of our work. 

used the most methodologically sound pro

identifying alternative actions and for mak

documenting the analyses required in our 

procedures manuals?

While these illustrations do not exhaust th

prospective questions, what they say is tha

effectively in the futures business, both th

implications of our own policies and becau

Congress is asking us to make or examine 

Page 14 GAO/PEMD-10.1.10 Prospecti
of and projections about the future.2 This m

expected to continue (1) as the effort requ

members of the Congress to push new legi

through the Congress and to amend existin

legislation becomes greater, (2) as evaluat

programs demonstrate problems that coul

prevented in existing programs, and (3) as 

methodology and the motivation to get sm

the future improve and increase. That is, w

important role in helping prevent future pr

in helping promote greater success before 

taken and before program actors and stak

become entrenched.

This role complements our mission to repo

objectively, but in retrospect, on what is h

now and on what has occurred in the past

different one, with distinctive methods of 

table 1.2 indicates, retrospective and prosp

methods differ on such features as the sou

evaluation questions, where we get our inf

and techniques for analyzing the evidence

method has its own requirements and its o

strengths and limitations for our work. Th

PES will be discussed in detail in this tran

The requirements of retrospective method

presented in other transfer papers.

2

The Kansas City Regional Office maintains a compre

review and bibliography of all GAO reports involving 

innovative methodologies, providing easy access to t

applications, for job planning purposes. This list incl

reports dealing with forward-looking questions, some 

included in our references to help illustrate further th

history of this aspect of our work.

Page 15 GAO/PEMD-10.1.10 Prospecti
Table 1.2: Features of

Retrospective and

Prospective Methods

Feature Retrospective Pros

Source of questions Criteria and

issues in existing

programs,

regulations, and

policies

Ideas 

assu

abou

prob

and p

solut

Primary sources of

information

Documents,

administrative

data, interviews,

observations,

opinion surveys

Prior 

theor

evalu

or ex

tests 

appr

opini

Primary types of

analysis

Qualitative

approaches to

empirical data,

quantitative

approaches to

empirical data,

information

synthesis in

relation to

program criteria

and issues

Simu

mode

inform

synth

relati

conc

opera

assu

prop

Delp

techn

analy

effec

We have already discussed the nature of

forward-looking questions, described the t

methodological issues they raise, and sum

when a PES would and would not be appr

Subsequent chapters present a definition o

prospective evaluation synthesis, a detaile

of how to carry it out, and some of its vari

Special attention is given to the crucial iss

judging the quality of the information bein

synthesized and models for aggregating re

many prior studies.

Page 16 GAO/PEMD-10.1.10 Prospecti
Answering Forward-Looking Quest

In doing our work, we should use the meth

appropriate to the complexity of the quest

the level of effort required by the situation

overkill or underkill in design would be a m

job management. The first wastes scarce r

the second fails to meet the need adequate

For some questions and some circumstanc

of highly systematic methods of dealing w

forward-looking questions would be overk

example, we may be asked about one prov

proposed legislation in an area in which w

many years of experience and in which we 

published reports whose recommendation

directly on the provision. Further, the idea 

one among several at early stages of consi

and it may be unclear that the legislation w

forward in the current session. Here, the e

might adequately satisfy methodological a

customer concerns by drawing on our cum

experience to discuss the issue as we have 

seen it and, subject to our usual reviews fo

comments, comment informally on it. Tha

use professional judgment and opinion.

Where the questions are controversial, far-

and sensitive, more systematic methods m

for. For example, our analyses of the savin

problems, and of various bailout proposals

more than informal methods, because of th

sensitivity and long-term consequences of 

issue is resolved.

Among the advantages of using systematic 

are the following.

1. The full range of existing information m

efficiently brought to bear on the question

than relying, in a somewhat happenstance 

Page 17 GAO/PEMD-10.1.10 Prospecti
seg oadoog Questos

individual’s memory, we identify, consider

the body of available knowledge to answer

question. Data that were costly to collect i

and are still relevant but that might otherw

neglected can be used. The risk of overloo

contradictory evidence may be notably red

2. The degree of confidence we have in ou

answers—whether analyses of other peop

forecasts, conclusions regarding the succe

proposed legislation, or our own

recommendations—can be stated more pr

less-formal methods permit. When we dea

future, uncertainty is part of any analysis, 

how sound, but the more precisely we stat

degree of uncertainty, the more complete, 

more useful, our prediction will be. Saying

95-percent confident that the number of co

awarded contracts will increase by betwee

and 15,000 for each of the next 4 years” pr

precise information to a decisionmaker ab

likelihood than does the statement “More 

will be awarded competitively in the futur

3. One method for promoting the quality o

prospective work is independent replicatio

we use systematic methods to review othe

projections or to make our own, we are be

replicate the analyses and thus promote qu

is, when independent analysts obtain the s

results, confidence in findings rises. In the 

sciences, such replication in independent l

is often required before a result is accepte

However, replication requires precision in 

and carrying out the analytic procedures. S

the social sciences, of which program eval

part, using systematic methods permits rep

and helps distinguish robust findings from 

differences in technique.

Page 18 GAO/PEMD-10.1.10 Prospecti
seg oadoog Questos

4. Systematic methods can help us follow h

standards of evidence and analysis in docu

the basis for answers about the future. Mu

work requires an element of judgment. Pro

jobs inherently involve a greater degree of 

than retrospective questions and, consequ

greater element of judgment. In all such jo

be scrupulous in identifying sources of un

and, consequently, the need for alternative

options. However, using systematic prospe

methods can reduce the qualifications we 

Fewer caveats may be necessary if we app

state-of-the-art methodology.

In short, systematic prospective methods h

promise for strengthening our ability to sp

emerging issues.

Page 19 GAO/PEMD-10.1.10 Prospecti
Prospective Evaluation Synthesis

Broadly Defined

Prospective questions deal primarily with 

happen in the future. However, most prosp

methods rely heavily on information about 

happened in the past, primarily empirical a

evaluative data. Judgments—that is, assum

interpretations—enter in, particularly whe

speculate on future conditions or alternati

scenarios. Methodologically, answers to th

questions require approaches that meet sp

challenges, compared with retrospective m

For example, almost all evaluations have t

context into account if the ability to gener

issue. In retrospective methods, one appro

permits generalization is simple random sa

from a properly defined population. Anoth

approach is stratified random sampling, in 

relevant subgroups are considered, such a

rural or rich and poor states. Where there 

expect that the results of a program will d

different circumstances—the economy, th

human resources—stratified random samp

typically used. For retrospective studies, w

relevant is usually clear, and how the char

of entities we could sample vary is usually 

Not so for prospective studies. What the re

characteristics of the future will be, and ho

will vary, encompasses a wide range of po

For example, whether participants in a pro

job-training program will be likely to find e

in a given period may be influenced more b

trends in the economy than by instruction

targeting nuances. But perhaps economic 

will be relatively unchanged, so that other

characteristics of the context will be more 

to consider.

Page 20 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

Putting this distinction somewhat more te

generalizations in retrospective studies are 

straightforward, empirically based stateme

which one moves logically from a sample t

population. Extrapolations in prospective 

contrast, require one to move logically and

conceptually, as well as empirically, by tak

account how a particular finding might op

varying conditions and situations. We thus 

make economic and other assumptions ex

otherwise, we are implicitly accepting the

continuation of the present unchanged int

(See Cronbach, 1982, for a more detailed d

Despite this and other challenges, a set of 

evaluation methods has been developed. A

illustrates, these include actual, empirical, 

judgmental, and mixed approaches.1

1

Economists have developed many quantitative meth

projecting the future, particularly those involving eco

forecasting, modeling, and simulations. These have in 

specification of a theory (conceptual model in PES te

is influencing relevant outcomes, the identification o

assumptions, quantification—on the bases of theory 

experience—of these assumptions, and running often 

quantitative analyses of most likely outcomes under 

assumptions about how the future will be similar to a

from the present and the past. For example, the Soci

Trustees Report is based on quantitative models who

assumptions include more and less optimistic estima

economic conditions. Our policy manuals describe so

techniques and suggest appropriate uses. The PES ca

results of these modeling and simulation studies but 

them in its greater reliance on prior empirical work o

programs in the past or on basic and applied researc

Page 21 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

Table 3.1: Some

Prospective Methods Type Illustrative technique

Actual Experimental tests;

Demonstration programs

Empirical Simulation;

Forecasting

Logical Front-end analysis;

Risk assessment;

Systems analysis;

Scenario building;

Anticipatory analysis

Judgmental Delphi techniques;

Expert opinion

Mixed Prospective evaluation s

The prospective evaluation synthesis, or P

new member of the class of prospective m

(Chelimsky, 1988). It was adapted by GAO 

evaluation synthesis in order to answer qu

about the future more systematically than 

methods and more rapidly than some othe

prospective methods such as experimenta

(U.S. General Accounting Office, 1983) (Ap

also gives a brief history of the PES.)

Conceptually, the PES provides a way in w

logic of evaluation methodology and its pr

can be appropriately used in assessing the 

consequences either of an individual propo

alternative and competing policy proposal

combines (1) the construction of underlyin

of proposed programs or actions as develo

Wholey for evaluability assessment with (2

systematic application of existing knowled

developed in the evaluation synthesis meth

(Wholey, 1977) That is, a PES is a prospec

anchored in evaluation concepts. It involv

conceptual, and empirical analyses, taken 

context of the future.

Page 22 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

As figure 3.1 illustrates, the conceptual an

results help focus the operational analyses 

answer the question, “Logically, should the 

work?” The operational analyses further sc

search for empirical findings and answer t

“Practically, could the proposal work?” Th

analyses can open both new conceptual an

operational possibilities and answer the qu

“Historically, have activities conceptually 

operationally similar to the proposal work

past?” Finally, the PES takes into account 

which the past is and is not likely to be sim

plausible future conditions.

Figure 3.1: the Triad of

Analysis

When the PES Is

and Is Not

Appropriate

As noted, the PES can be used either for e

an individual proposal or for comparing tw

policy alternatives. In examining an individ

proposal, the PES requires a criterion, or a 

good that needs to be made explicit. Devel

explicit criteria is a task familiar to GAO e

Nonetheless, it is often difficult, since legi

Page 23 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

proposals can result from greater agreeme

actions than on aims or goals. Assessing tw

proposals may be somewhat easier, becau

points of “common cause” can serve as a p

the hoped-for good. Further, it is generally 

make comparative judgments (“Which is b

than absolute ones (“Is it good at all? How 

The PES and

Timeliness

Additional conditions affect the use of the 

Although the PES has the promise of being 

most timely evaluation methods, obviously 

operate instantaneously. While times vary, 

of two or more bills might require about 3 

the part of at least two evaluators in order 

for adequate reviews of published and unp

literature, consultation with technical exp

the thorough assessment of the resulting in

However, a PES may take longer than 3 m

especially when the competing legislative 

are quite complex, when there is little prio

experience with issues, or when most of th

is unpublished.2

This time constraint indicates that a PES s

started as soon as possible after a custome

in order to ensure that the assigned evalua

the requisite time for their work. For less-c

issues, or situations such as analyses of po

recommendations, where a separate repor

have to be written, less time may be requir

noted earlier, a greater level of effort woul

2

The unpublished literature can include reports prep

contract to the government, work in progress that ha

presented as draft material or in speeches, and other 

material that may not have appeared yet in print. Sea

these materials usually involves reviewing federal co

grants, contacting project managers and principal inv

and canvassing other experts in the field.

Page 24 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

allocated to controversial, sensitive, and fa

questions.

The PES and Data

Availability

Another point affecting timeliness is that w

issue becomes extremely popular or extre

controversial in the legislature, it may hap

many different bills on the same subject ar

introduced within a short time. This can ca

logistical and other problems that a PES m

the appropriate method. But if this situatio

develop in the middle of the PES effort, th

evaluator would either have to resist expa

scope of the study or obtain an extension 

As indicated above, the PES relies heavily 

knowledge—basic and applied—already p

evaluators and researchers. The PES can b

effectively on topics for which a body of re

literature exists. For some mature issues t

long attracted the attention of evaluators a

researchers, the existing literature may be 

containing many studies and theories conc

basic mechanisms involved. For others tha

or have not yet stimulated much investigat

scholarship, PES evaluators may not be ab

great deal that is relevant.

As mentioned earlier, this outcome tells th

policymakers that there is little empirical b

their decisions. They can then judge the m

moving ahead, not moving forward, or lim

types of actions they take (targeting, demo

and so on). It may also be an important op

present to policymakers the research and 

that would have to be filled in order to ma

judgments. The case of the PES that includ

recommendations for demonstration, expe

or pilot projects may, therefore, be relative

Page 25 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

since such approaches can be useful altern

across-the-board changes in national polic

The PES and the

Recommendations

GAO Makes

In many situations, a full PES would be ov

prepare recommendations. For example, f

lack of accepted internal controls or findin

to report honestly information unfavorable 

weapon systems leads quite directly to we

recommendations. In other circumstances

our findings are more complex, our sense 

alternatives is broader, the results are mor

In some cases, these could be presented a

for consideration. In others, particularly th

involving controversial, sensitive, or far-re

conclusions, our recommendations—deriv

through other methods—could themselves 

be the subject of a PES.

Table 3.2 illustrates some of these circums

which include, for example, situations in w

federal role may be relatively complex, ou

recommendations would pose notable cos

burdens, and major structural or managem

changes might be involved. In such circum

investing some time in a PES might permit 

even more hard-hitting and convincing and 

solid effect, leading in turn to greater savin

nonmonetary benefits. These and other

considerations about when an evaluator sh

consider a PES are summarized in table 3.

Page 26 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

Table 3.2: Illustrations of

Where a PES Might

Strengthen Our

Recommendations

General circumstance Specific example

Involves complex

federal, state, and local

relationships

What is the best way 

federal government 

state and local gove

serve handicapped 

are older and young

regular school age?

What would be the b

to strengthen results 

funds in child abuse 

Nontrivial costs or

burdens

How many Internal R

Service agents shou

to current staff or red

current tasks to go a

unreported income n

computer matching?

Major structural or

management changes

How should the resp

and roles of the Offic

Management and Bu

other agencies be re

better identify low-qu

surveys?

Very high national

stakes are involved

What are the optimu

dealing with the

savings-and-loan cri

Page 27 GAO/PEMD-10.1.10 Prospecti
ospecte auatoSytess

Broadly Defined

Table 3.3: Situations in

Which a PES Should and

Should Not Be

Considereda

Consideration of PES a

Situation Probably should

Prob

not

Technical

Data base

quality

High, moderate Low

Proposal

complexity

relative to time

available

Complexity low or

moderate and

time short or

moderate; or,

complexity high

and time long

High 

little t

Proposal

stability

High, moderate Low

Contextual

Degree of

federal

leverage

(regulations,

funds)

Moderate, high Low

National stakes Moderate, high Low

Consequences

of our

recommendations

Far-reaching Restr

scop

aThese considerations apply to the PES. Other pros

methods could be useful when it would not be appr

do a PES.

Page 28 GAO/PEMD-10.1.10 Prospecti
As table 4.1 shows, there are six steps in th

PES approach, three of which closely invo

persons who request the job or are likely t

results to make decisions—the customer. 

steps are defining the problem, selecting th

or alternatives to evaluate, analyzing the c

underpinnings of the selected alternatives

the operational logic of the selected altern

testing the key conceptual and operationa

assumptions against existing evidence, and 

the results in relation to the key assumptio

Table 4.1: Steps in the

Basic PES Approach and

Persons Involved

Step Persons invol

Defining the problem Customer, eva

Selecting alternatives to

evaluate

Customer, eva

Conceptual analysis Evaluator

Operational analysis Evaluator

Testing key assumptions

Check on assumption

centrality

Customer, eva

Test against existing

evidence

Evaluator

Presenting results Evaluator

aFor GAO, the customer is the congressional reques

job. Other persons helpful at this step might include

stakeholders and experts in the field. In the catastro

insurance PES, for example, health provider and co

organizations provided useful input in defining the p

Input is, of course, received in the context of the us

guidance on ensuring our independence and objec

While these steps are essential in using the 

commenting on proposed congressional or

administration actions, they also apply to t

of possible recommendations, with two

Page 29 GAO/PEMD-10.1.10 Prospecti
modifications. First, generating alternative

recommendations involves either usual GA

procedures or the application of technique

forecasts, assessment of likely effects, and

scenario-building. Second, we need to use 

with regard to how extensively we can inv

customer in selecting options and in check

assumption centrality while maintaining o

independence at this stage of our work.

In this chapter, we discuss the first two ste

in table 4.1. The others are described in ch

each step, we first present what that step m

it is important, what its role is, and the kin

activities that would fulfill the requiremen

illustrate how to do the step through its ap

a GAO report. The applications in both cha

center on a specific example, a PES condu

competing legislative proposals dealing wi

problem of teenage pregnancies. (U.S. Gen

Accounting Office, 1986b)

Defining the

Problem

Detailed

Specification

Table 4.2 shows the key elements of this im

first step. Here the evaluator works with th

draw the target that the proposal is to hit, 

as clear as possible on the size and nature 

concerns that the proposal is intended to s

PES, the evaluator is trying to see if the pr

program will work to solve not a generic p

necessarily, but a specific one. Thus, a pro

may be well-aimed at one target may miss 

widely. For example, many programs can i

providing food supplements, nutrition edu

health screening. Some, however, may be 

Page 30 GAO/PEMD-10.1.10 Prospecti
solving the problem of low birth weight ba

low-income women and teenage mothers; 

be aimed at promoting age-appropriate pro

height and weight among preschoolers. He

pivotal question of this first step: What’s th

Page 31 GAO/PEMD-10.1.10 Prospecti
Table 4.2: Step 1:

Defining the Problem Aspect Definition

What “defining the problem”

means rules

Detailed specif

the concern tha

and out what w

considered pa

problem. This c

“target” to be “

successfully by 

proposal

Why this step is important Different peopl

an apparently “

problem broad

narrowly. Unles

and evaluator a

what is to be co

part of the prob

analyses aimed 

determining wh

proposals will w

themselves be 

The role of this step As the start of t

helps determin

of the work and 

foundation for t

the results

Activities that fulfill the

requirements for this step

(1) Discussions 

customer and r

hearings (if any

proposal with r

size and nature 

problem. (2) In

analysis of the 

regarding the s

nature of the p

Identification o

require agreem

decisions. (4) D

with the custom

others as nece

reach closure o

definition of the 

Page 32 GAO/PEMD-10.1.10 Prospecti
Illustration In 1984, there were about a million pregna

500,000 births to women under 20. In respo

bipartisan congressional efforts to increas

federal effort in this area. More than a sco

were introduced into the Congress in 1986

Concerned about the best way to assess th

legislation, a congressional requester aske

questions: (1) How effective had prior effo

address the problem? (2) What implication

structuring future legislation might be draw

existing knowledge about teenage pregnan

The first step of the PES was to clarify the 

order to focus the scope of the PES proper

example, the GAO staff determined that “t

pregnancy” per se was not the problem, be

policymakers were not concerned about b

married women under 20. Rather, two pro

posed in debates: (1) births to teenagers w

resources to support themselves or their c

(2) the negative health and social consequ

both mothers and infants associated with b

unwed and poor teenagers.

Faced with a subject that has been defined 

than one way, one can, of course, decide t

the focus of the PES to one definition or a

Following discussion with the customer, w

deal with both problems. In effect, this dec

meant enlarging the scope of the PES to a 

the literature addressing both the prevalen

teenage motherhood and the consequence

prevalence. Fortunately, the literature on t

pregnancies was not ordinarily restricted t

the other issue: most sources contained in

relevant to both. Certain topics that could 

included with the teenage pregnancy prob

received little or no attention. The exclude

Page 33 GAO/PEMD-10.1.10 Prospecti
also helped define the policy space.1 For e

congressional concern was expressed not 

pregnancies but only about those resulting 

births. Ignored in the discussion were the 

50 percent of the teenage pregnancies term

spontaneous or induced abortion.2 Further

interest centered largely on the pregnant w

not on the presumably teenage males who 

impregnated them.3 Whether correct or no

implicit legislative definition of teenage pr

1986 was as a problem primarily affecting 

women and their children.

Another aspect of defining the problem ce

who is to be considered a teenager. Clearly

18 or younger were included by everyone. 

discussions included all women under 25, 

others restricted the definition to persons 

By agreement with the customer, we focus

primarily on women 20 or younger.

Selecting

Alternatives to

Evaluate

The PES does not generate proposals at th

beginning: that is, a proposal has already b

and the issue is whether it is likely to hit th

we said earlier. Not all proposals are good 

good candidates for a PES, however. This 

two things. First, it screens out proposals 

PES is not the right evaluation tool. Perhap

1

“Policy space” is within the boundaries of politically 

policies. Thus, the set of policies enclosed within the 

of any given period consists of all the policies that ar

one or another of the principal political partisans.

2

It seemed obvious that a policy of promoting induce

a solution to adolescent pregnancies was clearly outs

policy space.

3

There was some concern in one proposal with teena

this was never an important center of attention, altho

problem could also be phrased as lack of family form

responsibility on the part of the young men. A PES co

stage, compare alternative target definitions in terms 

efficiency, and so on. Page 34 GAO/PEMD-10.1.10 Prospecti
example, the proposal seems to change da

perhaps we have already reviewed similar 

and can quickly draw on our corporate kn

provide comments on likely success.

Second, of the proposals for which the PE

right evaluation tool, this step selects the o

ideas for review. “Optimum” can include th

consideration of a variety of factors. One i

the specific interest of the customer. Othe

include variations among proposals in cos

groups, or the governmental means

proposed—regulatory, categorical, tax pol

grant. For example, proposals to provide l

nursing care to the elderly could vary nota

depending on such factors as the copayme

required, the conditions covered, and the d

care authorized. Some proposals could co

annually; others, billions. Selection on the 

variation among the proposals could in tur

such factors as maximum ranges, special i

and similarity to existing pilot work. The P

be explicit about the basis to be used, beca

choice made at the end of this step notably 

scope of the work and the utility of the res

4.3 describes this step.

Page 35 GAO/PEMD-10.1.10 Prospecti
Table 4.3: Step 2:

Selecting Alternatives to

Evaluate

Aspect Definition

What “selecting policy

alternatives to evaluate”

means

A PES usually begin

specific proposal wh

success is to be eva

is actually evaluated 

however, as a result 

conducted during th

“Selection” means th

of the step, the prop

assessed will have b

determined and alte

any, will have been s

Why this step is

important

Not all proposals are 

candidates for a PES

among the good can

all may be equal in o

of time: it may be mo

policy to analyze som

rather than others

The role of this step It helps ensure that t

will not be wasting ti

gives the analyses o

Activities that fulfill the

requirements for this

step

(1) Identification of th

viable alternatives. (2

to be sure there are 

such as rapidly mov

or an adequate body 

of similar prior propo

reject these as PES c

(3) Examination of th

that would be optimu

in depth through the 

according to criteria 

maximum difference

characteristics. (4) S

the PES proposals

Why the PES Begins

With Existing

Options

For any problem, a large number of potent

and programs may be relevant. However, a

the full range of possible alternative polici

Page 36 GAO/PEMD-10.1.10 Prospecti
concern of a PES. The PES task is constra

principles. (1) The task must be restricted 

can be examined by posing the evaluation 

“Is there evidence that a particular program 

will or will not be likely to meet its stated 

(2) The PES begins with the options that

policymakers are already considering in or

PES findings to be useful to them. Thus, th

process that starts with the alternatives un

consideration, then looks for any evidence

concerning the potential efficacy of those

alternatives, and, only if necessary, genera

options.

It is important to understand the implicati

centering the PES on existing alternative p

Another way to proceed would be to make 

comprehensive review of all the research a

evaluation literature relevant to the proble

question, attempting to infer the implicatio

for policy and designing alternatives ourse

However, this alternative is rejected in the 

method for two main reasons.

First, there may be only a loose fit betwee

findings and policy. It is possible for two r

draw different policy implications from th

research evidence.4

 Unless some obvious l

has been made, neither reviewer would be 

and neither would be incorrect in his or he

4

For example, given the existence of a large number 

pregnancies, one policy alternative would be to cond

to convince teenagers to have abortions. Another pol

the data is to conduct campaigns stressing sexual ab

among teenagers. Still a third would be to provide ca

and ongoing subsidies to men who would marry and 

pregnant teenage women, since the underlying probl

conceptualized as lack of family formation. None of t

is “incorrect” in the sense of misinterpreting the basi

existence of a widespread problem, but, also, none w

been relevant to the policy formation process in 1986

Page 37 GAO/PEMD-10.1.10 Prospecti
projection of policy implications. But cont

even equivocal recommendations are diffi

in decisionmaking.

Second, the PES approach allows the revie

make definite statements that are subject t

verification. The outcome of a PES review 

assessment of whether the policy or polici

consideration are supported or not suppor

existing evidence. If a PES concludes that 

is justified by the evidence and some other

commentator asserts that it is not, then it i

to compare the analytic procedures used b

the disagreeing parties to determine the po

is justified by the research evidence.

What about a situation in which none of th

already on the table is likely to work? To b

maximally helpful, the PES relies on prior 

and evidence as a way of refining the polic

If the prior research did not support the op

under consideration, then the PES would t

identify the policy options that were within 

realistic range of the research, when the qu

were considered at appropriate levels of c

For example, proposed legislation on hous

physically handicapped adults might focus 

increasing independence for single person

literature might consistently place greater 

on group homes or family units.5

Illustration As stated earlier, the PES is intended to w

closely the research and evaluation eviden

a proposed policy or one or another of sev

5

Care must be taken in using prior research to assess 

quality, including the independence and objectivity o

researcher. See our discussion on recognizing threat

in our transfer paper entitled Case Study Evaluations 

Accounting Office, 1987c).

Page 38 GAO/PEMD-10.1.10 Prospecti
alternative policies. In the case of the teen

pregnancy project in 1986, several alternat

be compared. Twenty-two separate bills re

teenage pregnancy had been introduced in 

Congress, twice the number proposed the 

For the PES, which had to be completed w

months, the selection of proposals to cons

on some importance. Clearly, full consider

22 proposals was out of the question.

To aid in the selection of proposals to asse

staff performed a content analysis of each 

proposal, listing its program requirements

such items as criteria for client eligibility, 

and required services, and any required ad

arrangements. (U.S. General Accounting O

1982) This information was presented in ta

to facilitate identifying the elements that w

and those that were different across propo

how each bill resembled or differed from t

With a few exceptions, most of the 22 cong

bills proposed national programs of assista

services exclusively for pregnant and pare

women. However, the bills differed on the 

the services to be provided, the types of cl

would be served, and the administrative an

arrangements that would be required. The

rather than attempt to assess the feasibilit

promise of all possible program options, th

was made, in consultation with the custom

the PES on those apparently key, congress

relevant dimensions of difference between 

proposals—that is, the choices presented t

Congress regarding scope of services, clien

administrative arrangements. Picking alter

differed widely also would help in the eval

other proposals that differed along the sam

dimensions.

Page 39 GAO/PEMD-10.1.10 Prospecti
In order to further narrow the focus of the 

staff, again in consultation with the custom

selected two proposals that embodied thes

by differing substantially on each of these 

dimensions.6 The first proposal was target

pregnant and parenting teenagers, flexible 

the services that should be provided, and

administratively straightforward. Grants w

provided directly to local agencies that wo

and deliver services. In contrast, the secon

was more broadly targeted to include econ

disadvantaged women up to age 25, was hi

prescriptive about services to provide, and 

administratively complex, requiring coord

five other federal programs. This bill also i

proposed program for preventing teenage 

permitting the PES to address both of the 

for policymakers that had been identified 

Page 40 GAO/PEMD-10.1.10 Prospecti
After narrowing the focus of the problem, 

remaining tasks of analyzing the chosen bi

of conceptual and operational models of th

programs; identifying from those models t

populations and the program features of in

selecting the appropriate evidence; arrayin

evidence against the models to assess whe

proposed programs are likely to meet their 

objectives; and reporting the results.

The Conceptual

Analysis

Underlying Logic The key elements of this step are presente

5.1. At this point, the evaluation aims at re

underlying logic of the proposal: why, in th

proposer thinks it will work. For example, 

aimed at reducing urban congestion by sub

satellite location of offices and businesses 

based on the assumption that a dispersion 

possible and desirable and that for a given

community, the primary centralization com

commercial or governmental requirements

proposal aimed at reducing urban congest

increasing mass transit and reducing indiv

parking facilities probably is based on the

assumptions that dispersion of businesses 

people centrally is not possible or desirabl

what will most motivate people to use mas

aversion to high parking-lot prices and hav

long distances from parking lots to busine

relative to cheaper, more readily accessibl

transit.

Page 41 GAO/PEMD-10.1.10 Prospecti
Table 5.1: Step 3:

Conceptual Analysis Aspect Definition

What “conceptual analysis”

means

Identification o

assumptions, b

values, and the

underlying the 

why, in princip

to work or not w

Why this step is important Two reasons. F

set up criteria f

out what prior r

program evalua

relevant: it is th

on the underlyi

or the program 

underlying ass

were similar. Se

step can identi

strengths) in lo

could lead to u

about program 

The role of this step In scoping, this 

increasingly ta

research that w

not have to be 

and it increase

efficiency of the 

Activities that fulfill the

requirements for this step

Content analys

proposed bill o

Graphic techni

helpful in efficie

displaying the 

models and ch

accuracy and

completeness 

interpretation. C

supplemented 

interviews with 

the proposals o

academicians 

worked on the 

Page 42 GAO/PEMD-10.1.10 Prospecti
Making the underlying assumptions or bel

explicit as possible helps identify gaps in t

helps focus the subsequent literature searc

relevant prior research or program evaluat

the urban congestion example, the literatu

first instance might focus on evidence rega

dispersion assumption and factors affectin

relocations. The second instance might foc

attention on research on individual incenti

disincentives involving money, convenienc

and so on in relation to using mass transit 

individual cars.

Illustration To assess both the promise and the feasibi

two teenage pregnancy bills, it was necess

break them down into components that co

addressed as subquestions. This required a

the texts of the two bills to develop two ty

model for each proposal: (1) a conceptual 

(2) an operational model. The strategy her

Page 43 GAO/PEMD-10.1.10 Prospecti
similar to that of developing an evaluation 

except that a PES reviews existing evidenc

of collecting new data.

The conceptual models would answer the 

questions: What was the problem to be add

What was the treatment? (Or what actions 

brought about by the program?) And what 

intended outcome of those actions? Figure

and 5.3, from GAO’s report, contain the re

disaggregation. (U.S. General Accounting O

1986b) These models helped determine the 

studied programs that should be considere

those proposed and the outcomes that sho

examined when judging their effectiveness

seen from figure 5.1, the first bill had the o

reducing the number of unintended repeat

pregnancies, while the second bill, whose 

shown in figures 5.2 and 5.3, articulated a 

detailed theoretical model. It proposed to 

mothers to avoid welfare dependence by a

Page 44 GAO/PEMD-10.1.10 Prospecti
them to complete school and gain employm

thus, the bill specified additional intermed

objectives.

Figure 5.1: Underlying

Conceptual Model of the

First Bill

Source: U.S. General Accounting Office, Teenage P

500,000 Births a Year but Few Tested Programs,

GAO/PEMD-86-16BR (Washington, D.C.: July 1986)

Page 45 GAO/PEMD-10.1.10 Prospecti
Figure 5.2: Underlying

Conceptual Model of

Program a in the Second

Bill

Source: U.S. General Accounting Office, Teenage P

500,000 Births a Year but Few Tested Programs,

GAO/PEMD-86-16BR (Washington, D.C.: July 1986)

Page 46 GAO/PEMD-10.1.10 Prospecti
Figure 5.3: Underlying

Conceptual Model of

Program B in the Second

Bill

Source: U.S. General Accounting Office, Teenage P

500,000 Births a Year but Few Tested Programs,

GAO/PEMD-86-16BR (Washington, D.C.: July 1986)

Page 47 GAO/PEMD-10.1.10 Prospecti
The Operational

Analysis

Underlying

Operations

The operational model of a proposed prog

how to accomplish the goals of the progra

conceptual model, it is constructed by a ca

textual analysis of the legislation, but it an

following question: Who is to be served, by 

and under what financial and operational

arrangements or constraints? An operation

defines the target populations, the intende

providers, the funding sources and amoun

administrative structures that should be th

the PES.

The details of the fourth step—operationa

analysis—are described in table 5.2. Here t

emphasis is not on the “why” of the propo

the “how” of the proposal: how the propos

would be carried out and how it would op

Page 48 GAO/PEMD-10.1.10 Prospecti
methods of operations research come into 

step. The proposals are analyzed to determ

doing what, when, and under what circum

whom in order for the proposal to be carri

step can identify the operational complexi

simplicities) in the proposal, the number o

decisionmakers, and how contingent the fi

will be on the agreement and coordination 

(or relatively few) actors.

Page 49 GAO/PEMD-10.1.10 Prospecti
Table 5.2: Step 4:

Operational Analysis Aspect Definition

What “operational analysis”

means

Identification o

mechanics of t

how it is suppo

carried out

Why this step is important Two reasons. F

up criteria for d

the relevant pri

or programs or 

experience wit

similar to those 

proposal. Seco

can also identif

strengths) in th

procedures tha

to more or less 

about program 

The role of this step It sets limits wit

the search for r

research or pro

evaluations tak

increasing job 

and completen

Activities that fulfill the

requirements for this step

Operations ana

proposal. The t

of operations

research—usin

content of the p

identify the des

elements- -are 

Graphic presen

operation helps 

accuracy and

completeness 

interpretation. I

with proposal s

developers pro

assurance of th

operational mo

Page 50 GAO/PEMD-10.1.10 Prospecti
The analysis in itself can reveal likely sour

success or failure for the proposal: gaps, fo

in authority for making decisions or assum

about the availability of resources other th

be provided directly through the proposed 

The operational analysis also serves anoth

it focuses the literature review on the relev

operational issues that could affect the suc

failure of the new program. Finding, for ex

the operation of one proposal would requi

establishing local stakeholder groups whil

competing proposal would involve using e

officials would turn attention to relevant p

experience of the efficiency and effectiven

contrasting modes of program managemen

control.2

Page 51 GAO/PEMD-10.1.10 Prospecti
Illustration Figure 5.4 shows the operational model co

for the second teenage pregnancy bill.

Page 52 GAO/PEMD-10.1.10 Prospecti
Figure 5.4: Underlying Operational Model of Program B in the Second B

(Figure notes 

Page 53 GAO/PEMD-10.1.10 Prospecti
Source: U.S. General Accounting Office, Teenage P

500,000 Births a Year but Few Tested Programs,

GAO/PEMD-86-16BR (Washington, D.C.: July 1986)

Testing the Model

Two Substeps Testing the model involves two substeps. T

substep—checking the centrality of the as

to be examined in depth—means reviewin

customer the assumptions selected as the 

review of prior evidence. The conceptual a

operational models usually involve many s

may not be valuable to delve into them all. 

Page 54 GAO/PEMD-10.1.10 Prospecti
evaluator selects those that seem to be mo

or to offer the most useful contrasts betwe

competing proposals. Discussion with the 

(or the developers of the idea or knowledg

academic sources) is a final check that the 

of entry into tests of key assumptions have 

selected.

The second substep—testing key assumpt

existing evidence—is summarized in table 

step uses the evaluation synthesis method

with two differences. The first difference i

is relevant has been determined through th

of specifying the conceptual and operation

and through checking the importance of th

assumption to the customer. A second diff

Page 55 GAO/PEMD-10.1.10 Prospecti
that the evaluations are synthesized with r

to the chosen assumptions.

Table 5.3: Step 5: Testing Key Assumptions Against Existing Evidence

Aspect Definition

What “testing key assumptions against

existing evidence” means

A complex body of evidence f

research and program evaluat

collected, and the key concep

operational assumptions are c

with the findings from prior stu

determine the likelihood of new 

success

Why this step is important The conceptual and operation

can reveal gaps in logic that a

affect program success. This 

against prior experience, howe

major criterion for deciding wh

idea will work. If relevant prior 

and experience indicate that t

assumptions have worked in t

then, if conditions are similar, 

likely to work in the future (sim

have not worked in the past an

are to work in the future)

The role of this step It completes the triad of analys

(conceptual, operational, emp

a conclusion on the proposal’s 

that is as solid as possible

Activities that fulfill the (1) Complete identification of r

research and program evaluat

assessment of the quality of th

(3) synthesis of credible findin

evaluation synthesis method is 

Systematic tabular or graphic 

of the evidence against each k

conceptual or operational ass

the efficiency and completene

analysis. Thus, techniques of m

analysis and multiple case stu

comparisons are applicable

Page 56 GAO/PEMD-10.1.10 Prospecti
In table 5.3, step 5 is described as complet

of analyses. As noted earlier, a central met

point in the PES is that the results of three 

types of analyses— conceptual, operationa

empirical—are all compared and otherwis

account in reaching conclusions, thus stre

what can be said with some confidence ab

future. When all three approaches give the 

answer, we can be more confident about it

soundness. When they differ, as seen from

conceptual, operational, or empirical persp

must qualify our results in terms of that la

reinforcing agreement. Finally, we need to 

ways in which the future may differ from t

identifying, for example, more or less optim

scenarios for relevant factors. Where the f

likely to be similar to the past on key dime

can have more confidence about the appro

of the PES to judge the likely success of pr

the scenarios differ from past or present e

our certainty necessarily decreases, althou

still specify conditions under which a prop

more or less likely to work.

Page 57 GAO/PEMD-10.1.10 Prospecti
Illustration The review of evidence in the teenage preg

example we are following started with a b

question: How many people would be eligi

programs in the proposed legislation? This 

relatively easy question to answer because 

excellent demographic data collected by th

of the Census and the National Center for 

Statistics concerning the number of teenag

at present, in the past, and in the near futu

as birth statistics. Less definitive data wer

on births by socioeconomic level, although 

surveys were the basis for our estimates. T

sections give further detail for the illustrat

Estimating Target

Population Size

Good estimates of the size of the target po

a proposed program are important for proj

Page 58 GAO/PEMD-10.1.10 Prospecti
program costs. However, the target popula

identical to client population, since few pr

ever able to reach all the eligible members 

population. In general, the more complex t

eligibility requirements are, the less precis

estimates of client participation can be. An 

data source can be experience with simila

programs. If the clients of an existing prog

identical (or nearly so) with the target pop

some proposed program, a good basis for 

estimates can be the existing program’s cu

number of participants. For example, data 

with catastrophic illness insurance progra

provided important insights for the PES on 

proposed national system. More usually, it 

necessary to synthesize population estima

combining numbers from census and adm

data, for example, with information from p

surveys and research data on the degree o

association between eligibility characteris

Page 59 GAO/PEMD-10.1.10 Prospecti
In this illustration, no existing program ser

pregnant and parenting teenagers. It was n

rely on published tables from the National 

Health Statistics on the characteristics and 

of women giving birth each year by age, m

status, years of school completed, and num

previous births. It was possible to add up t

of first births to women under age 18 over 

years to calculate the number of young un

mothers who constituted the target popula

However, this target population is too incl

some of the young mothers are not poor an

would be ineligible for program participati

the first proposal.

Page 60 GAO/PEMD-10.1.10 Prospecti
Unfortunately, the National Center for Hea

Statistics collects no information on the in

mothers. To estimate the number of poor y

mothers required using sample survey data 

applying survey findings to the vital statist

course, the potential client populations of 

programs are always problematic. Clients 

exceed in number the total target populati

participation rates can vary considerably, 

suggested earlier.3

Some information on participation rates ca

obtained by examining existing programs 

Page 61 GAO/PEMD-10.1.10 Prospecti
nature. The next task in the PES was to id

existing federal programs with related obj

target populations. This is important for se

additional reasons. First, there is always a

alternative to the proposals being consider

that is the status quo, consisting of all the 

programs already in place. Second, in this 

information on existing programs would a

the feasibility of both the proposed coordi

existing services and the proposed funding 

For example, if a proposed program relies 

coordinating services provided under anot

Page 62 GAO/PEMD-10.1.10 Prospecti
program or programs, whether those servi

fact available becomes crucial information

crucial because if the services funded by t

programs are not available, or if the provid

already operating at capacity and cannot t

clients, then the new program has to find a

of providing those services, and it will nee

funds to provide them. Further, if existing 

were apparently underutilized, a new prog

not be needed.

The review of existing programs provided 

information on what could be expected as

Page 63 GAO/PEMD-10.1.10 Prospecti
participation rates in either of the two pro

programs. The main reason for this disapp

outcome was that the existing programs w

one exception, not exclusively targeted at 

pregnancies but included other target grou

Finding the Studies The next task was to conduct a search for 

published in the recent past (5 years, in th

illustration) that evaluated pregnancy prev

programs and comprehensive service prog

Page 64 GAO/PEMD-10.1.10 Prospecti
pregnant and parenting young women.4 Th

included formal publications, such as prof

journals and monographs, as well as comp

data bases, usually containing bibliograph

abstracts, and informal (or so-called fugiti

publications, including reports of limited c

and monographs. It is especially important 

effort be made to obtain coverage of the la

as wide as possible, since informal publica

contain the latest studies, and to collect an

negative findings, since studies showing p

results are more likely to be published tha

do not.

4

Publication dates in journals can follow the time of 

by several years. The studies covered up to a decade 

previous to the time of the PES. This time restriction 

that applied social research has only recently been u

in the evaluation of programs and that the credibility 

data is slight for reason of age alone. For example, da

effectiveness of the Great Depression programs, such 

Civilian Conservation Corps, are not likely to be view

to similar contemporary programs. However, for som

time restrictions may be much looser. For example, i

training, studies that are a decade or two old may no

irrelevant, especially if studies over time are quite co

their findings.

Page 65 GAO/PEMD-10.1.10 Prospecti
To obtain information on all relevant evalu

studies, it is usually necessary to rely on p

contacts with knowledgeable persons. Thi

normally be accomplished by sending out 

publications already located and asking fo

be supplemented by other publications kn

experts.

In this case, all the studies—whether cont

outcome evaluations or not—were review

analysis of program costs, their sources of 

and implementation problems. The inform

where individual projects gained their fund

augmented the information on existing pro

services collected earlier on the federal lev

about program failures can provide invalu

information that gives balance and perspe

information gained from successes. For ex

may give clues as to the staff, public relati

recruitment, or support services required f

proposed programs to operate as intended

Special attention was paid to publications 

outcome evaluations. Each publication wa

carefully to ascertain how closely the prog

question resembled the proposed program

succinct summary of each program was pr

The outcome variables used in the evaluat

noted separately, particular attention bein

the quality of the “impact assessment” dat

result of this careful examination was a pr

each evaluated program, recorded in tabul

containing the crucial information on prog

description, outcomes, and ratings of data 

Appendix III gives an example of one such 

As mentioned earlier, it is important to bri

on the literature the same conceptual fram

in examining the proposed programs. For 

Page 66 GAO/PEMD-10.1.10 Prospecti
describing a program and its evaluation, a 

form was filled out, characterizing that pro

clients, services, and administrative arrang

The categories were the same as those dev

the analysis of the two legislative proposal

to ensure that the derived information was 

relevant to the consideration of the propos

programs.

Quality Assessment The most technically demanding aspect of 

of each evaluation was assessing the quali

information. Since this task is essentially i

that confronted in the evaluation synthesis

borrowed from criteria employed in previo

syntheses. Each evaluation outcome, as de

the conceptual models of the programs be

examined, was treated separately. The evi

each objective was rated separately. An ev

might provide evidence of adequate quality 

the outcomes of interest but not on anothe

for example, of the use of different data co

methods. Other outcomes that were not of 

concern in the conceptual models of the p

under scrutiny in the PES were also noted

assessments of the quality of the evaluatio

used.

Criteria The quality-rating criteria used in the asse

effectiveness evidence have to be tailored 

extent to the issues involved in the PES. N

the criteria are largely the same from PES 

this case, criteria centered primarily on th

validity of the research design used in arriv

Page 67 GAO/PEMD-10.1.10 Prospecti
effectiveness estimates.5 Most of the evalu

studies had used longitudinal comparison 

designs, making the composition of the co

group critical. (Appendix II presents more 

criteria.)

The criteria included (1) appropriateness o

comparison (or control) group; (2) sample 

adequacy, including attrition among client

comparison group; (3) standardization of d

collection, including measures of data reli

(4) validity of measures used to represent 

variables; and (5) appropriateness of statis

methods used, especially those used to en

internal validity of effectiveness estimates

for competing explanations of estimates.

The assessment of data quality requires so

in evaluation design, measurement, and st

well as some understanding of the substan

Several readings are often required. For ex

sometimes the fact that there is anything w

a particular measure of a variable is not ob

another study has been examined that is m

and accurate in its measurement strategy. 

5

Internal validity refers to the attribution of cause an

external validity, to the ability to generalize. An “idea

would offer strong evidence that effects, if any, stem

program (or event being studied) and would be obtai

groups and in situations as similar as possible to the 

circumstances in which the program was being appli

this ideal design would be appropriately sensitive, ab

effects of a size believed worth the costs of the progr

experts believe the controls necessary for internal va

limit external validity, and they argue that for policy 

external validity, with its implications for extrapolati

important in judging quality. Other experts are more 

about optimizing both or place heavier emphasis on i

validity. We thought the question with top priority fo

particular PES was evidence of any effects, and so w

that aspect of design. For some other PES, different 

be weighted more heavily, a point discussed in more 

appendix II.

Page 68 GAO/PEMD-10.1.10 Prospecti
often be necessary to read the set of evalu

studies several times before a final quality 

be arrived at.

Reliability As in other rating tasks, it is necessary to t

reliability of the ratings (that is, their repli

likelihood that our reviewers will reach th

rating conclusions) by ensuring that there 

least two readers for at least a subset of ev

studies. If a subset is used, the reliability c

ratings should be done early, midway, and 

coding process to avoid rater-drift and gen

fatigue. Discussion among raters concerni

disagreements on the subset often brings t

critical characteristics of studies that were 

immediately discerned.

Aggregation Although it is possible to arrive fairly easil

reliable and credible rating for each criteri

at an overall quality rating is usually more 

Much of the problem encountered in devel

overall assessments for the teenage pregna

arose because many reports did not provid

information with which to judge the adequ

evaluation on one or more of the criteria. I

absence of direct evidence, it is possible to 

evidence only questionable, unless some o

of information suggests that the absence o

information stems from some serious flaw

In addition, many evaluation studies provi

several evaluation outcomes, each outcom

the quality of evidence presented. It would 

mistake to discount entirely a study that c

acceptable evaluation of one outcome and 

evaluation of another. For these reasons, r

overall quality ratings for each evaluation 

outcome was presented separately along w

assessments of each outcome.

Page 69 GAO/PEMD-10.1.10 Prospecti
Presenting the

Results

Product Type Presenting the results of a PES differs from

presenting the results of an evaluation syn

PES, the underlying conceptual and opera

models have to be identified, the key assum

have to be highlighted, and the evidence h

summarized in relation to these assumptio

contrast, an evaluation synthesis arrays th

in relation to the questions to be answered

underlying models need not be explicated

summarizes the elements of step 6.

Page 70 GAO/PEMD-10.1.10 Prospecti
Table 5.4: Step 6:

Presenting Results Aspect Definition

What “presenting results”

means

Presentation of 

conceptual and 

models (usually 

form) and of th

the comparison 

assumptions a

concisely and c

Why this step is important The PES involv

uncommonly d

analysis of a pr

credibility of th

depends in pa

reader’s being 

follow the PES 

easily and to se

how the finding

developed

The role of this step Promoting cred

making our con

simple, clear, a

accessible as p

Activities that fulfill the

requirements of this step

Development o

appropriate gra

tables; prepara

necessary tech

appendixes (fo

details on proc

to rate the qua

evidence and t

findings)

Table 5.4 emphasizes the value of tabular a

techniques. The result of a PES might look 

briefing report than a chapter report. This 

of course, in terms of length, depth, wheth

recommendations are provided, and our o

criteria for deciding on product type.

Page 71 GAO/PEMD-10.1.10 Prospecti
Illustration The results of outcome evaluations are typ

presented in tabular form, as shown in tab

where some of the findings from the teena

pregnancy PES assessment are presented. 

was designed to draw the reader’s attentio

different things. Across the top are the exp

objectives of the legislation plus some oth

were found to be important in the field. Al

are program types generated by clustering 

according to similarity with regard to the s

they provided. In the body of the table are 

descriptions of the studies’ comparison gr

the results, expressed as whether the prog

“did better” than the comparison group at 

statistically significant level. The boxes rep

findings we considered to be most method

credible. All this information was transcrib

the rating sheets.

A summary table such as table 5.5 provide

information on how many studies address

particular outcome, how much of those da

credible, and the types of programs that ha

compared to other conditions or programs

information is presented in narrative rathe

numerical form. While this was an appropr

present the findings, an alternative would 

effect sizes. Where there are quite a few st

relevant results—and particularly where th

programs’ clients can be grouped by factor

age, race, education, and family income, w

be expected to influence the outcome vari

quantitative presentation can be efficient a

effective.

Note that in table 5.5, comparison groups a

described in detail. This is also critical info

because some evaluations compared the p

nothing more than ordinary prenatal healt

Page 72 GAO/PEMD-10.1.10 Prospecti
while other studies compared their progra

that was only slightly different from it. The 

and absence of effects under these types o

condition are thus difficult to assess. That 

high-quality test of a program includes the 

elements of a high treatment strength and 

basis for causal attribution. This point bec

conclusion of our illustrative PES: few pro

been adequately tested, a wide variety of p

appeared successful, and both comprehen

less-comprehensive programs appeared to 

successful.

More specifically, the findings of the illust

with regard to the requester’s questions w

summarized as follows.

1. The pattern of credible results showed n

preference between the two proposed pro

variety of past programs appeared success

there was little information on the compon

were responsible for their apparent succes

there was no convincing evidence that the 

comprehensive service packages were mo

than the least comprehensive.

2. Implementation analyses suggested that 

certain avoidable operational problems as

with the proposed administrative structure

example, program administrators as well a

evaluators frequently mentioned complex

coordination arrangements as a significant 

program success.

3. Therefore, if the Congress wanted to ini

nationwide program, then the administrati

simpler model might have a greater chanc

success. However, we concluded that the 

was most

Page 73 GAO/PEMD-10.1.10 Prospecti
Table 5.5: Example of Presenting PES Findingsa

Page 74 GAO/PEMD-10.1.10 Prospecti
Page 75 GAO/PEMD-10.1.10 Prospecti
aThis is one segment of a longer table. It illustrates a

intermediate summary of findings by offered service

table included verbal and graphic material. The “C7

other code numbers in the second column refer to f

bibliographic data for each comparison in the teena

pregnancy report from which we have taken the tab

Source: U.S. General Accounting Office, Teenage P

500,000 Births a Year but Few Tested Programs,

GAO/PEMD-86-16BR (Washington, D.C.: July 1986)

consistent with initiating a large-scale dem

program that would systematically test the 

costs, and benefits of different approaches 

reducing teenage pregnancy.6

In this particular instance, the conclusions 

clearly favor the legislative proposal that w

prescriptive (given the lack of strong evalu

knowledge) and relied on existing services 

experience with complex coordination pro

addition, the smaller, more flexible propos

take into account the need to develop info

about which strategies work with which te

Thus, no clear advantage adhered to the on

compared to the other. This is not always 

a PES and, in fact, did not occur in anothe

of the method dealing with catastrophic he

6

Two options were suggested as consistent with the a

expansion of available services is wanted, then it wo

sense to target services to the teenagers who are at h

risk—young and unmarried teenagers—to allow flex

type of services provided and to have a simple admin

structure. (2) In an alternative to a program of expan

the federal government could take the role of promo

and ensuring both sound comprehensive evaluations 

innovations and dissemination of the programs (or th

components) that have been shown to work.

Page 76 GAO/PEMD-10.1.10 Prospecti
insurance proposals.7 However, the impor

teenage pregnancy example is real in that 

taxpayer resources, since neither proposa

introduced in a form that was likely to suc

7

See U.S. General Accounting Office, 1987f. In this re

looked at six legislative proposals for protecting Med

from the financial hardships that often accompany ca

illness. Our review, and in-depth analysis of two of th

determined that while protection would increase, som

remain. We further identified issues requiring additio

consideration, such as coverage of prescription drug

conclusions played a significant role in both hearings 

subsequent configuration of the act.

Page 77 GAO/PEMD-10.1.10 Prospecti
Several variants of the PES are possible. T

two types. The first variant derives from ta

PES: customer interest in special aspects o

proposal. The second variant involves com

PES with sources of information other tha

written evaluations. Using multiple metho

course, notably expand the range of the PE

it is typical of designs for many of GAO’s i

controversial jobs that we use several met

that the limits of one are offset by the stre

another.

Targeted PES The basic model of the PES we described 

is appropriate when relatively well articula

proposals have been developed. However, 

can be helpful in other, more limited situat

when a problem is being defined or when 

particular interest. In essence, aspects of t

discussed earlier become the target of mor

work. Table 6.1 summarizes some of the v

the PES.

Table 6.1: Targeted PES

and Related Critical

Issues

Target Critical issue

Problem definition Determining th

the perceived p

legislative prop

Problem characteristics Assessing data 

narrowing or re

contradictory e

Relation of proposal to

prevailing scientific models

Clarifying unde

assumptions

Assessing projected costs Checking sens

projections aga

assumptions

The PES and

Problem Definition

For many issues that come before a legisla

some critical problem has been identified 

Page 78 GAO/PEMD-10.1.10 Prospecti
proposers of legislation, along with sugges

measures expected to resolve the problem

problem is a major one, it is rare that only 

of legislation will be proposed. Even in suc

already noted, every proposal has an impli

alternative—namely, not to enact any legis

all. In any case, before a judgment can be m

whether the proposed measure will resolv

problem, it is important to be clear about e

what the problem is.

Proposed legislation designed to address a 

problem is necessarily based on some defi

understanding of the issue involved. For e

two contending legislative proposals may 

addressed to the issue of homeless person

identifying the homeless as needy persons 

no kin upon whom to depend and the othe

homelessness as the lack of access to conv

shelter. The first definition centers attentio

on the social isolation of potential clients, 

second focuses on housing arrangements. 

that the ameliorative actions that follow w

different, as well. The first might emphasiz

program to reconcile estranged persons w

relatives, while the second might imply a s

housing program. Thus, the two definition

different proposals.

Especially critical in problem definition is 

between what is perceived to be the probl

who have pressed for attention to the issu

definition in the legislative proposals. In th

connection, the PES evaluator would ordin

to legislative proceedings, including comm

hearings and floor debates, journals, news

magazine editorials, and other sources in w

discussions of the problem may appear. Th

of this review of sources is to examine how 

Page 79 GAO/PEMD-10.1.10 Prospecti
problem has been formulated and to state 

as possible the range of politically accepta

alternatives.

Problem

Characteristics:

Density and

Distribution

To design a public program properly and t

costs reasonably well, good information is 

the density, distribution, and overall size o

problem. For example, in providing financ

for emergency shelters for homeless perso

make a significant difference if the total ho

population is 2.5 million or 250,000 (both e

have been advanced). It would also make 

whether the problem is located primarily i

cities or can be found in equal densities in 

larger places.

An identified problem is often a complex m

related conditions; for planning purposes, 

information is needed about that complex

example of homelessness, the proportions 

homeless suffering from chronic mental ill

chronic alcoholism, or physical disabilities 

known in order to appropriately design the 

mixture of programs.

It is much easier to identify and define a p

to develop valid estimates of its density an

distribution. For example, only a small han

battered children may be enough to establ

problem of child abuse exists. However, to 

great a problem is and where it is located

geographically and socially involves detail

knowledge about the population of abused 

and its distribution throughout the politica

jurisdiction in question. Such exact knowl

ordinarily much more difficult to obtain w

of precision that may be needed.

Page 80 GAO/PEMD-10.1.10 Prospecti
To collate and assess whatever informatio

the issues in question, evaluators need to u

they have learned from the literature (con

government reports, published and unpub

studies, and limited-distribution reports) a

understanding of the designs and methods 

conclusive results. Equal emphasis is given 

sentence to “collate” and “assess.” Unevalu

information can often be as worthless as n

information at all.

For some issues, existing data sources ma

sufficient quality to be used with confiden

example, an issue on which measurements 

routinely taken by either the Current Popu

Survey or the decennial census is typically 

about which accurate and trustworthy kno

ordinarily may be obtained from those sou

from some other statistical series, such as 

published by the Bureau of Labor Statistic

into the trustworthy category. But when w

data produced by other sources, it is neces

examine with care how the data were colle

A rule of thumb is that for any subject, exi

sources provide contradictory estimates. B

chaos can sometimes be reduced to some 

Seemingly contradictory data on the same 

collected by opposing stakeholders can be 

useful for assessment purposes. For exam

the Coalition Against Handguns and the N

Association have sponsored sample survey

U.S. population concerning their approval 

disapproval of gun-control legislation. Alth

two reports issued by the coalition and the

association differed widely in their conclu

one finding much popular support for mor

gun-control measures and the other the op

close inspection of the data showed that m

Page 81 GAO/PEMD-10.1.10 Prospecti
specific findings were nearly identical in th

surveys. The findings upon which both sur

substantially agreed can be regarded as ha

greater credibility.

Relating Proposal

Models and

Prevailing Scientific

Models

Whether explicitly intended or not, legislat

other proposals are based on some set of i

models of how the problem in question ma

arisen and how it is currently sustained. F

one welfare reform alternative suggests ex

all states the coverage of public welfare to 

families with unemployed parents in order 

the number of households headed by wom

proposal may be based on a model that se

welfare policies as penalizing marriage, sin

to a woman and her children would stop u

marriage.

An alternative welfare reform proposal mi

that benefits be continued upon marriage b

by some proportion to avoid subsidizing p

marriages. Both proposals involve extendi

to intact families, one to support such fam

both parents are unemployed and the othe

regard to the employment status of a new 

Each proposal is based on different model

payments might affect marriages of house

headed by women. In the first case, the pr

based on the idea that women will avoid m

unemployed men because they would lose 

benefits, and it ignores the effects that ma

employed man would have. The second pr

concerned with the possibility that the con

of benefits after the marriage of a woman 

household might render the woman suscep

marrying a man who was primarily interes

sharing her benefits.

Page 82 GAO/PEMD-10.1.10 Prospecti
Both proposals are based on models that s

role of economic incentives in marriage fo

topic that has received considerable attent

microeconomic theory, econometric resea

social psychology and sociology. An appro

tactic for the PES would be to review this 

seeking to establish two things: (1) the ext

which experts agree and (2) the existence 

empirical evidence concerning the intende

either proposal. A thorough review of the 

literature accompanied by consultation wi

subject-matter specialists and knowledgea

practitioners could determine that one of t

proposals has more support than the other

is as much evidence for one as for the othe

alternatively, that neither proposal has mu

backing in research and experience.

An important opportunity is presented wh

finds that there are very few or no previou

evaluations that are relevant because the p

program is a notable departure from progr

evaluated in the past. A clear message can 

decisionmakers that their proposals go far 

firm knowledge and are, hence, subject to 

more-than-ordinary risk of failure.1

 This ad

not be an admonition to stick to the progra

past. For example, the advice may be to fu

demonstration projects incorporating the n

proposals rather than to fund fully operati

programs. Pointing out areas on which exi

knowledge has nothing to say may be as im

the avoidance of public policy failures as g

rich harvest of firm knowledge.

1

We would need to take into account that not acting 

risks of failure. For example, while we may have littl

about effective AIDS prevention measures, not makin

efforts we can also incurs risks.

Page 83 GAO/PEMD-10.1.10 Prospecti
Assessing Projected

Costs

Legislative proposals are often accompani

projected costs. In fact, all the bills that ar

out of committee include a Congressional 

Office cost estimate. Although any project

easily upset by subsequent actual experien

usually possible to make a viable assessme

concerning whether projected costs are ba

reasonable and likely assumptions. For ex

projected cost of a proposed measure that 

subsidize flood insurance for structures bu

plains can be profoundly affected by assum

made about the number of structures that 

covered and the participation rate among p

covered households.

If the flood plains are defined as areas with

100-year flood zone—where a major flood 

at least once every century—coverage will 

but flood incidence will be lower than if th

the flood plain were defined as a 20-year fl

If all the applicable property owners partic

anticipated costs might be more than if the

participation rate were much lower. But th

other complications that affect cost. If onl

property owners who were close to the so

floods signed up, then the subsidy costs m

greater than if participation rates were mo

over the flood plain.

A PES can help assess cost projections by 

whether the appropriate assumptions have 

in their construction, as well as by proposi

alternative assumptions. Here the statistic

tests how responsive the projections are to 

in the assumptions. It raises questions like 

costs would be changed if participation ra

changed by a given amount or if unit price

services were changed. Sensitivity analyse

the assumptions concerning the costs that 

Page 84 GAO/PEMD-10.1.10 Prospecti
most critical to the overall cost estimate. F

part of the PES analysis, estimates of the m

and direction of the problems of under- or

overcosting that were identified could be a

existing information and synthesized into 

meaningful range.

Variants Using

Other Sources of

Information

The basic PES operational model uses prio

evaluations or research as the source of in

If, in reviewing this literature, tradeoffs sh

made between timeliness and comprehens

strategies such as sampling and time-limite

could be adopted. There may be situations

when available information must be suppl

with some original data collection and wh

more efficient to tap into existing knowled

panels or expert judgments. Further, there 

situations where the PES is combined with 

data collection and other audit work.

Combining the PES

With Some Original

Data Collection

The results of the PES may be supplement

some original data collection, such as exam

agency records or surveys. That is, where 

data are insufficient and where time and re

permit, evaluators may want to use PES pr

up to the point of matching evidence and k

assumptions. At this point, the PES could 

dual tracks with some highly targeted new 

collected while other, prior work is review

of the reports already mentioned, such as 

consequences of opening more combat su

positions and units to women, involved mu

methods of data collection in answering a 

question. (U.S. General Accounting Office, 

For example, we were asked by the Congr

determine what might be learned from sta

Page 85 GAO/PEMD-10.1.10 Prospecti
experience in addressing mandate burden

already in place since 1981 required the Co

Budget Office to estimate such costs for p

federal legislation. Similar requirements fo

the costs of proposed state legislation exis

states. New legislation proposed by the co

requesters would have required federal

reimbursement for additional costs. This a

was already in use in 14 states that reimbu

governments for burdens imposed by new 

(U.S. General Accounting Office, 1988n) T

for answering the prospective question inc

review of the literature, analysis of relevan

visits to 8 states selected by searching prio

plus a telephone survey. Data from the 8 s

supplemented by questionnaires for state o

state legislative leaders, and relevant inter

Using evidence from these 14 states, we fo

estimating and reimbursing costs have had 

limited effect on the burden of mandates, 

some special circumstances.

When may such original data collection be

particularly valuable? One might expect th

such as defense and tax policy, our unique 

data is likely to mean we would have bette

information than one could expect to find 

published literature. In other areas, howev

certain aspects of health that require confi

dealing with patients’ records, physicians w

also evaluators and researchers might hav

relative advantage and would find a richer 

in published reports than we ourselves mi

to collect. That is, combining the PES with 

forms of audit and evaluative work is cons

the multimethod approach we typically us

evaluators planning a PES can also anticip

certain extent, where we may find a relativ

data base and where our unique authorizat

Page 86 GAO/PEMD-10.1.10 Prospecti
suggest the need for new data collection to

supplement the PES.

Combining the PES

With Expert

Judgment

The evaluator supplementing other eviden

views of experts must be aware of the requ

of systematic methods such as Delphi tech

Properly applied, these systematic method

information that differs in some key ways 

anecdotal evidence on which congressiona

is often based. First, the effects of “charism

presenting testimony are ruled out. Second

same questions are usually asked of many 

informants, it is possible to determine wha

generally held. Third, the bases for opinion

brought out and can be compared objectiv

available evidence. Fourth, the experts or 

informants can be selected primarily or so

considerations such as knowledgeability a

appropriate diversity.

We have used expert judgment and panels 

of ways to answer prospective (and also

retrospective) questions. For example,

• to assess major welfare reform proposals 

case management, contracts between welf

recipients and agencies, coordination of se

target populations, HRD contracted for tw

experts. One panel consisted of experts at 

national level and was convened by the Na

Academy of Public Administration; the oth

consisted of experts at the local level and 

convened by the Federation for Communit

The findings of both panels were synthesiz

and the numerous concerns, observations, 

recommendations were presented to the C

the insights of expert panels. (U.S. Genera

Accounting Office, 1988b)

Page 87 GAO/PEMD-10.1.10 Prospecti
• to examine the probable effects of legislat

would change the conditions for legal imm

we identified (in consultation with the cus

issues and we brought together a panel of 

The experts identified the highest-quality d

relevant to these issues and presented thei

conclusions. We then independently asses

conclusions, relative to our own judgment 

quality of the evidence, in order to report t

soundest available statement on probable 

(Chelimsky, 1989)

The use of expert judgment to supplement 

prospective work requires (1) clarity in pr

when we are relying primarily on the opini

others and (2) careful planning when the e

significant source but our own, independe

is needed. In the instance of proposed imm

legislation, the experts helped sharpen the 

identified relevant empirical data, and exa

points of consensus and dispute in the inte

of the data. We then independently review

available information and reached our own

conclusions by the usual standards of audi

evaluation work. (Chelimsky, 1989)

In another instance, GAO had a problem-d

assignment—examining the nature and ex

sweatshops in the United States and identi

policy options that might help control the 

In this study, which was clearly entitled op

the extent of the problem and possible enf

options, we reviewed the relevant literatur

sweatshops, particularly with regard to the

and efforts at control; developed a workin

(since the term is not defined in federal sta

regulations) in agreement with the custom

2

See U.S. General Accounting Office, 1988j. This was 

PES but illustrates a multimethod approach to analyz

and possible action.

Page 88 GAO/PEMD-10.1.10 Prospecti
interviewed federal, state, and local officia

researchers, and union and management e

surveyed state labor departments and agen

officials; investigated possible sweatshops 

York and Los Angeles; and analyzed federa

inspection reports. While this required mo

than might usually be available for a PES, 

that for certain prospective questions, GAO 

negotiate with the congressional customer 

undertake quite extensive involvement of 

well as site visits, to supplement the literat

Page 89 GAO/PEMD-10.1.10 Prospecti
Other Prospective Methods

This appendix helps place the PES in relat

methods. Traditionally, the basic concepts 

evaluation have been used primarily in the

assessment of policies and programs that 

in place. This ex post application has beco

commonplace that it is the one most frequ

associated with evaluation. Less frequently

evaluation methodology has been used to 

ante the potential success of policies that 

consideration.

The conventional approaches to prospecti

evaluations have ranged widely from relat

freewheeling “demonstrations” to highly c

field experiments. However, most propose

are put into operation—often nationwide—

evaluative evidence attesting to their poten

success. (Some of the unevaluated program

have been put in place have to do with rec

laws, various regulatory programs targetin

health, “deinstitutionalization,” “the strate

initiative,” “pilot cities,” “impact cities,” “m

cities,” “operation push,” and “operation

breakthrough.”)

But even when small-scale pilot efforts of 

experimental sort are implemented—and m

evaluators would agree that highly control

experiments yield the most credible result

experiments have many practical drawbac

particular, three serious limitations must b

into account when they are considered for 

1

Pilot and experimental studies can provide crucial in

capital on which synthesis draws. They are among th

sources of information which the PES relies. That is, 

from having available a good fund of knowledge base

evaluations of other programs, research knowledge, 

Thus, the PES does not replace the new data collecti

program evaluation. Pointing out the limitations of p

experimental studies should not be misconstrued as 

this valuable prospective method.

Page 90 GAO/PEMD-10.1.10 Prospecti
Oteospecte etods

only application of evaluation methodolog

assessment of prospective public policies. 

for example, three randomized public poli

experiments: the five income-maintenance

experiments, the housing allowance exper

the several experiments on demand pricin

electricity. First, they were costly. On this 

alone, it would not be likely that more than 

handful of experiments could be set under 

any decade. That is, only a minute proport

public policies and programs that are in an

policy space could possibly be assessed th

experiments.

Second, these field experiments were limit

consideration of only a narrow band of alt

policies. Indeed, none of the income maint

experiments came close to testing the actu

welfare policies that were considered by th

and the executive branch in the years sinc

completion. Policy space tends to be occu

more contenders than can easily be accom

the design of the typical field experiment.2

Furthermore, with every new administrati

session of the Congress, the contending po

programs, as embodied in various versions 

proposed legislation, are never a static bod

in fact be constantly changing.

Third, public policy experiments take a lon

complete. Legislative proposals are often d

within the space of months and, at most, a 

Clearly, field experiments that take 5 years 

another 3 to analyze can rarely speak direc

set of specific, proposed laws for the many 

typically pass before results appear. To so

2

This does not mean that the field experiments were 

Almost all the proposed welfare reform measures inv

work-leisure tradeoff issues, a topic about which the 

maintenance experiments have much to contribute.

Page 91 GAO/PEMD-10.1.10 Prospecti
Oteospecte etods

these deficiencies are also characteristic o

prospective efforts.

Pilot demonstrations that call for the colle

original observations in the field may take 

long to carry through to completion as fiel

experiments. Even cross-sectional surveys 

significant periods of time. For example, a 

household sample survey ordinarily takes 

months to up to 2 years to complete (depe

the complexity of sampling and analysis). 

although “demonstrations” and quasi-expe

trials of prospective policies may take less 

conduct than the classical field experimen

may require more than several years to co

addition, they share the other drawbacks o

above, being expensive and subject to incr

irrelevance with changes in the policy spa

In sum, the traditional ways in which evalu

faced the problem of providing informatio

decisionmakers on the potential for succe

policies and programs that may be under

consideration at any time are not useful to 

decisionmaking process that may take no 

a year or two from proposal to definitive a

evaluations are to contribute to decisions 

proposed new programs, the contribution 

accomplished through procedures that are 

inexpensive, speak to each of the variety o

under consideration, and provide timely re

There is nothing especially new or startlin

idea, and many evaluators have given the p

some thought. A relevant example is an ap

evaluative techniques to proposed legislati

advocated the use of national health scree

identifying abused children. (Light, 1973) T

American Evaluation Association has iden

Page 92 GAO/PEMD-10.1.10 Prospecti
Oteospecte etods

front-end analysis as a major focus of atten

(“Evaluation . . . ,” 1982) Indeed, even the m

extended forms of evaluation, such as rand

field experiments, could benefit from a PE

conducted at the point of design. And ther

other efforts in recent years to come to gri

problems of timeliness that are inherent in 

front-end analysis. Many of the specific ele

PES have been advocated by others. In par

evaluability assessment as advocated by Jo

Wholey emphasizes the construction of un

models of proposed programs in order to a

whether a program or policy can be evalua

outcome effectiveness. (Wholey, 1977) In a

many others stress the importance of the t

underpinnings of prospective programs. (C

Rossi, 1983; Wang and Walberg, 1983; Gott

1984; Finney and Moos, 1984; Weick, 1980)

The main strength of the prospective evalu

synthesis is that because it draws upon ex

knowledge and research to assess the pote

success of a new proposal, it can be timely 

be used within the policy development pro

is, the PES will not necessarily provide the 

possible information that could be obtaine

optimal conditions, but it can provide in a 

manner the best possible information that 

available.

Page 93 GAO/PEMD-10.1.10 Prospecti
The PES relies primarily on the results of p

evaluations of previous or existing program

the results of a PES could be notably diffe

different rules were used for including a gi

Because the weighing of criteria used to ju

quality of prior studies is so critical to the 

PES, this appendix discusses in some deta

not elaborated upon in our paper on the ev

synthesis: how criteria are aggregated in re

decision of whether to use (or how much e

give) a specific study. There are at least fo

ways to assess the quality of prior evaluati

Table II.1 summarizes the advantages and

disadvantages of these four approaches.1

1

We also note the special case of where quantitative 

required as part of a PES. In this instance, careful att

be paid to the adequacy of our estimates of values th

PES analysis, including an examination of the quality 

and methods for checking their validity. If data are n

quality, provisions for boundary or sensitivity analys

made. Further, any time the functions we have to dea

likely to be multiplicative rather than additive, the ac

values entered into the analysis is critical, particularl

from local to national estimates. The PES could iden

which data must be aggregated and could identify the 

to multiplicative effects, where it is not possible as p

to make these better estimates ourselves.

Page 94 GAO/PEMD-10.1.10 Prospecti
Table II.1: Advantages

and Disadvantages of

Four Data Quality

Judgment Models

Model Advantages Disa

One criterion Maximum number

of prior reports

brought to bear

Large number of

reports permits

tests of

interactions

Analysis may be

quicker since

time for multiple

quality screens is

not taken

One 

may 

guide 

weak 

One 

unlike

adeq

intera

data 

quali

misle

Equally weighted If all criteria are in

fact equally

important, this

model may best

represent the

quality of the prior

evaluations

Permits direct test

of whether taking

quality into

account would

make a difference

in the findings

When 

criter

relev

have 

analy

thres

set, b

settin

thres

perm

stren

a ser

study

Rare 

criter

impo

Page 95 GAO/PEMD-10.1.10 Prospecti
Model Advantages Disa

Unequally weighted Better represents

relative

importance of

different criteria

Permits direct test

of whether taking

quality into

account would

make a difference

in the findings

A mo

stren

signi

criter

offse

flaw 

criter

are tw

heav

criter

Can 

cumb

assig

comp

for ea

for ea

well a

rating

criter

study

Threshold or fatal

flaw

Efficient in

focusing on most

crucial criteria

Ensures that a

study with high

scores on several

relatively minor

criteria but a

Ensures that a

study with high

scores on several

relatively fatal

weakness in one

or more crucial

criteria is not

included

Must 

fatal 

suffic

serio

scree

studi

other

poten

Page 96 GAO/PEMD-10.1.10 Prospecti
One Criterion

Only

In this method, the set of prior research an

evaluation studies on the general topic is

developed—say, on food-stamp participati

base closings, the effectiveness of federal 

aimed at disseminating knowledge, or the 

executive and managerial personnel. The s

examined against a single criterion.

For example, a decision might be made th

criterion such as measure validity should b

important for the job. This might be true if 

asked to assess the probable cost of a cert

child care. Prior evaluations of child care t

have information on costs that we conside

complete and properly measured would be 

Those with valid cost information would b

Except for the one selected criterion, othe

the quality of the relevant reports are not a

this method of synthesis. Rather, “strength 

numbers” is the intention, with the notion 

largest possible set of prior studies that m

selected criterion will offer the soundest g

answering the question. In a variant of this 

the information in the entire set of reports 

judged on the single criterion. The extent t

answer to the evaluation question would d

higher-quality and lower-quality studies (a

the single criterion) are used can be determ

Among advantages of this approach are th

on a large possible body of data. A prime

disadvantage is that it is quite rare that onl

criterion of study quality would be importa

evaluative question, as noted, would have 

limited in scope.

Equally Weighted

Criteria

In this approach, a set of criteria for select

prior research to be synthesized is develop

Page 97 GAO/PEMD-10.1.10 Prospecti
Typically, the set includes relevance, recen

similarity, and a variety of indicators of tec

adequacy including those appropriate to

measurement, design, analysis, and report

Each of these criteria is given equal weigh

deciding whether or not to include the rep

or book in the set of material to be synthes

is, a high score on relevance might offset a 

score on technical adequacy when a “total

score is derived and the cut-off established 

whether a study is included. Or, alternativ

threshold score in all criteria may be requi

report to be used.

GAO has many examples of this approach

the criteria described in the reviews of the 

illegal aliens on legal workers and the effe

drinking-age laws on highway safety. (U.S

Accounting Office, 1986a, 1987b) An advan

approach is that the effects of various aspe

quality can be tested empirically. A disadv

that particularly if a threshold is set for all 

almost no studies may pass the quality scr

Unequally

Weighted Criteria

In this approach, the criteria receive differ

weights. For example, technical quality ma

as more important than recency in decidin

or not to include the study. Among the

technical-quality criteria, for some questio

extent to which the design permits strong 

about causality may be seen as much more 

than, say, the extent to which documentat

measurement reliability exists. The weight

arbitrary but are guided by the theory und

methods. Again, there are examples of this 

(U.S. General Accounting Office, 1984a)

Page 98 GAO/PEMD-10.1.10 Prospecti
This approach has the advantage of better

representing the importance of different c

still possible, however, that modest streng

several relatively less important criteria ca

serious flaw on a significant criterion, if sc

each criterion are aggregated.

Threshold or

Fatal Flaw

In some situations, a report that does not p

on a specific criterion is not considered at 

other criteria come into play only after the 

test has been passed. For example, in a syn

studies on the homeless mentally ill, repor

not attempt to estimate the size of the loca

population of the homeless were excluded 

consideration. Further, within the useful s

fatal flaws criterion (sampling the range of 

set a cap on rated quality. That is, among t

that estimate population size, the quality o

was judged against seven other criteria an

direction and extent of bias were judged. T

technical-quality rating was the profile of w

errors were likely to lead to an overestima

an underestimate and the size of the bias. 

General Accounting Office, 1988i, 1988d)

This model is the most efficient way to ens

The fatal flaw must be carefully examined

to be sure that no offsetting features are p

since potentially informative studies that f

one criterion may be excluded from the re

Table II.2 provides a detailed example of t

used and how they were applied with rega

number of homeless mentally ill persons.

Page 99 GAO/PEMD-10.1.10 Prospecti
Table II.2: Example of a Fatal Flaws Analysis

What we did How we did it

Screening the studies In defining our universe of studies for the evaluation 

we purposefully kept our inclusion criteria broad. W

any study, regardless of methodological quality, tha

to estimate the size of the homeless or homeless m

population. We did, however, have some minimum 

criteria. Of our universe of 83 studies, 27 were sele

useful. Specifically, we included a study in our univ

each of the following three criteria:

1. The study was in written form. Telephone conver

speeches, or conference proceedings without a wr

were not included.

2. The study provided a count or estimate (by what

method) of the homeless or homeless mentally ill pe

assessed trends in a designated geographic area. 

exclude case studies of individuals or studies desc

service needs without a count or estimate.

3. The method used to make the estimate of the nu

homeless or homeless mentally ill was sufficiently d

permit us to evaluate its merits (or shortcomings). B

“sufficiently described,” we mean the study provide

information on

— the data used to make the estimate (for example

judgments or actual counts of persons in shelters);

— how those data were collected (for example, she

providers were interviewed over the telephone, stre

canvassed by car, and so on);

— how the estimate of the size of the homeless or h

mentally ill population was actually computed (for e

shelter and street counts were aggregated). That is

some kind of link between the data collected and th

population estimate.

Page 100 GAO/PEMD-10.1.10 Prospecti
What we did How we did it

Assessing the studies Next we rated the 27 relevant studies on two dimen

technical quality and soundness (that is, the extent 

chosen method would produce an underestimate o

overestimate of the size of the homeless population

discovered that many of the studies involved multip

for counting the homeless, reflecting the various se

(shelters, streets, institutions) in which the homeles

chronically mentally ill can be found. We considere

these “nested studies” for how well it met survey m

standards for soundness. Criteria for methodologic

soundness encompassed such issues as adequac

definition, coverage of sampling frame, implementa

procedures, and soundness of data analysis. We d

and applied a coding form to extract data relevant 

criteria. Finally, two staff members rated the full stu

criteria related to their overall sampling, measurem

implementation, and population estimation procedu

Sampling design Did the design cover the range of settings where ho

persons were likely to be found (shelters, streets, o

places, institutions)?

Was the sample of shelters and institutions represe

terms of the area’s shelter size (that is, number of b

type (public or private)?

Did the sample of streets and other public places (

census blocks) adequately cover the locations whe

homeless are known to congregate?

Did the sampling design account for seasonal varia

homelessness?

Was the unit of analysis (such as municipality) clea

Measurement Was the estimate of the number of homeless based 

actual count rather than expert judgment?

Was a respondent’s homeless status determined o

screening questions?

Page 101 GAO/PEMD-10.1.10 Prospecti
What we did How we did it

Implementation Were survey procedures explicitly stated in the rep

Were interviewers trained to engage with and admi

interviews to homeless persons?

Were instruments pretested?

If a street survey was conducted, were canvassing 

consistently applied in areas searched? Were area

enumerated before the actual street survey was co

If a shelter-and-institutions survey was conducted, 

count based upon administrative records rather tha

estimates? Were procedures developed to ensure a

unduplicated count of the homeless within shelters 

institutions?

Deriving the population

estimate

Was the estimate of the number of homeless based 

probability sample of areas (such as a national esti

upon a probability sample of cities)?

Were adjustments from the sample made to estima

population (for example, was the application of a

shelter-to-street ratio obtained from previous studie

appropriate and justified?

Fatal flaws analysis In applying these criteria, we gave a higher priority 

sampling dimension. That is, if a study did not adeq

sample the range of settings where homeless perso

there was a limit on how high the study could be ra

matter how strong the measurement, implementatio

estimation procedures. To illustrate, a study that ha

sampling design (for example, surveyed many sett

used simple estimation procedures was rated highe

study that had a weak sampling design (for examp

only shelters) and used sophisticated statistical adj

account for the fact that streets or institutions were 

surveyed. Accounting for sampling bias by using s

adjustments—in some cases the only option availa

based on assumptions about the size of the homele

population in the settings not included in the survey

actual count. Applying the criteria in this manner, w

study’s technical quality very high, high, moderate, 

low.

Page 102 GAO/PEMD-10.1.10 Prospecti
What we did How we did it

Our second rating helped us distinguish where on t

technical-quality scale (very high to very low) studie

considered sound enough to provide reliable estim

soundness of studies was determined by rating eac

the extent to which its methodology would produce

judgment, an underestimate or overestimate of the 

homeless persons. For example, a study that emplo

design that relied solely on the estimates of service 

would be rated as having the potential for overestim

size of the homeless population. Each study was as

rating on a 7-point scale that ranged from –3 (serio

underestimate) to +3 (serious overestimate). A writt

justification was given for each bias rating.

To determine a cutoff point for the methodological s

we selected studies that received a bias rating of –

In addition to providing a cutoff point, this second r

indicates the direction and likely magnitude of the b

study.

We used the information from these ratings to get a

of the current approaches and research designs th

used to count homeless and homeless chronically 

persons. This information formed the basis for a clo

examination of the patterns of strengths and weakn

were evident in the various studies and was applied 

developing our alternative approaches.

Page 103 GAO/PEMD-10.1.10 Prospecti
Page 104 GAO/PEMD-10.1.10 Prospecti
Page 105 GAO/PEMD-10.1.10 Prospecti
Page 106 GAO/PEMD-10.1.10 Prospecti
Page 107 GAO/PEMD-10.1.10 Prospecti
Page 108 GAO/PEMD-10.1.10 Prospecti
Chelimsky, Eleanor. “Federal Evaluation in 

Legislative Environment: Producing on a F

Track,” pp. 73-86. In C. G. Wye and H. P. H

Timely, Low-Cost Evaluation in the Public 

New Directions for Program Evaluation, N

Francisco: Jossey-Bass, Summer 1988.

Chelimsky, Eleanor. “Immigration: S. 358 W

Change the Distribution of Immigrant Clas

Statement before the Subcommittee on Im

and Refugee Affairs, Committee on the Jud

Senate, Washington, D.C. U.S. General Acc

Office, GAO/T-PEMD-89-1, March 3, 1989.

Chen, Huey-Tsyh, and Peter Rossi. “Evalua

Sense: The Theory-Driven Approach.” Eva

Review, 7:3 (June 1983), 282-302.

Cronbach, Lee. Designing Evaluations of E

and Social Programs. San Francisco: Josse

1982.

“Evaluation Research Society Standards fo

Evaluation.” Standards for Evaluation Prac

15. New Directions for Program Evaluation

Francisco: Jossey-Bass, September 1982.

Finney, John W., and Rudolf H. Moos. “Env

Assessment and Evaluation Research: Exa

Mental Health and Substance Abuse Progr

Evaluation and Program Planning, 7 (1984

Gottfredson, Gary D. “A Theory-Ridden Ap

Program Evaluation: A Method for Stimula

Researcher-Implementor Collaboration.” A

Psychologist, 39:10 (1984), 1101-12.

Hedges, L., and I. Olkin, Statistical Method

Meta-Analysis. New York: Academic Press

Page 109 GAO/PEMD-10.1.10 Prospecti
Light, Richard J. “Abused and Neglected C

America: A Study of Alternative Policies.” 

Educational Review, 43:4 (November 1973

Light, R., and D. Pillemer. Summing Up: Th

of Reviewing Research. Cambridge, Mass.

University Press, 1984.

U.S. General Accounting Office. What the 

of Agriculture Has Done and Needs to Do 

Agricultural Commodity Forecasting and R

GAO/RED-76-6. Washington, D.C.: August 

U.S. General Accounting Office. Models, D

War: A Critique of the Foundation for Defe

Analyses, GAO/PAD-80-21. Washington, D.

May 1980.

U.S. General Accounting Office. Minerals C

Developing Future Energy Technologies, T

Availability, and Projected Demand,

GAO/PEMD-81-104. Washington, D.C.: Jun

U.S. General Accounting Office. Content A

Methodology for Structuring and Analyzin

Material, methodology transfer paper 3. W

D.C.: June 1982.

U.S. General Accounting Office. The Evalu

Synthesis, methods paper I. Washington, D

April 1983.

U.S. General Accounting Office. WIC Evalu

Provide Some Favorable But No Conclusiv

on the Effects Expected for the Special Su

Program for Women, Infants, and Children

GAO/PEMD-84-4. Washington, D.C.: Janua

1984a.

Page 110 GAO/PEMD-10.1.10 Prospecti
U.S. General Accounting Office. Estimated

Employment Effects of Federal Economic

Development Programs, GAO/OCE-84-4.

Washington, D.C.: August 1984b.

U.S. General Accounting Office. Simulatio

Medicare Prospective Payment System for 

Health Care, GAO/HRD-85-110. Washingto

September 1985.

U.S. General Accounting Office. Illegal Ali

Research Suggests Illegal Aliens May Disp

Workers, GAO/PEMD-86-9BR. Washington

April 21, 1986.

U.S. General Accounting Office. Teenage P

500,000 Births a Year but Few Tested Prog

GAO/PEMD-86-16BR. Washington, D.C.: Ju

U.S. General Accounting Office. Hazardou

Uncertainties of Existing Data, GAO/PEMD

Washington, D.C.: February 18, 1987a.

U.S. General Accounting Office. Drinking A

An Evaluation Synthesis of Their Impact o

Safety, GAO/PEMD-87-10 Washington, D.C

1987b.

U.S. General Accounting Office. Case Stud

Evaluations, transfer paper 10.1.9. Washin

April 1987c.

U.S. General Accounting Office. Financing 

Education: Examples Comparing Existing 

Proposed Student Aid Programs, GAO/HR

Washington, D.C.: April 1987d.

Page 111 GAO/PEMD-10.1.10 Prospecti
U.S. General Accounting Office. Patent Po

Changes in Federal Law Considered Benef

GAO/RCED-87-44. Washington, D.C.: April 

U.S. General Accounting Office. Medicare:

Catastrophic Illness Insurance, GAO/PEM

Washington, D.C.: July 1987f.

U.S. General Accounting Office. Farm Pay

Analysis of Proposals to Amend the $50,00

Limit, GAO/RCED-88-42BR. Washington, D

October 1987g.

U.S. General Accounting Office. Immigrati

Future Flow of Legal Immigration of the U

States, GAO/PEMD-88-6.

Washington, D.C.: January 1988a.

U.S. General Accounting Office. Welfare: E

Panels’ Insights on Major Reform Proposa

GAO/HRD-88-59. Washington, D.C.: Februa

U.S. General Accounting Office. Milk Mark

Orders: Options for Change, GAO/RCED-8

Washington, D.C.: March 1988c.

U.S. General Accounting Office. Illegal Ali

Influence of Illegal Workers on Wages and 

Conditions of Legal Workers, GAO/PEMD-

Washington, D.C.: March 1988d.

U.S. General Accounting Office. USDA’s C

Program: The Accuracy of Budget Forecas

GAO/PEMD-88-8. Washington, D.C.: April 

U.S. General Accounting Office. Welfare R

Projected Effects of Requiring AFDC for U

Page 112 GAO/PEMD-10.1.10 Prospecti
Parents Nationwide, GAO/HRD-88-88BR. W

D.C.: May 1988f.

U.S. General Accounting Office. Medical D

FDA’s Forecast of Problem Reports and FT

H.R. 4640, GAO/PEMD-88-30. Washington, 

July 1988g.

U.S. General Accounting Office. Women in 

Military: Impact on Proposed Legislation t

More

Combat Support Positions and Units to Wo

GAO/NSIAD-88-197BR. Washington, D.C.: 

U.S. General Accounting Office. Homeless 

Ill: Problems and Options in Estimating Nu

Trends, GAO/PEMD-88-24, Washington, D.

August 1988i.

U.S. General Accounting Office. “Sweatsho

U.S.: Opinions on Their Extent and Possib

Enforcement Options, GAO/HRD-88-130BR

Washington, D.C.: August 1988j.

U.S. General Accounting Office. Tax Admi

Difficulties in Accurately Estimating Tax E

Yield, GAO/GGD-88-119. Washington, D.C.

August 1988k.

U.S. General Accounting Office. Federal L

Management: Consideration of Proposed A

Exchange Should Be Discontinued,

GAO/RCED-88-179. Washington, D.C.:

September 1988m.

U.S. General Accounting Office. Legislativ

State Experiences Offer Insights for Feder

GAO/HRD-88-75. Washington, D.C.: Septem

Page 113 GAO/PEMD-10.1.10 Prospecti
Wang, Margaret C., and H. J. Walberg. “Eva

Educational Programs: An Integrative,

Causal-Modeling Approach.” Educational E

and Policy Analysis, 5:3 (1983), 347-66.

Weick, Karl E. Social Psychology of Organ

ed. New York: Random House, 1980

Wholey, Joseph. “Evaluability Assessment

Evaluation Research Methods, ed. L. Rutm

Hills, Calif.: Sage Publications, 1977.

Page 114 GAO/PEMD-10.1.10 Prospecti
This is a flexible series continually being a

updated. The interested reader should inqu

the possibility of additional papers in the s

The Evaluation Synthesis. Transfer paper 

formerly methods paper I.

Content Analysis: A Methodology for Struc

Analyzing Written Material. Transfer paper 

formerly methodology transfer paper 3.

Designing Evaluations. Transfer paper 10.

formerly methodology transfer paper 4.

Using Structured Interviewing Techniques

paper 10.1.5, formerly methodology transf

Using Statistical Sampling. Transfer paper 

formerly methodology transfer paper 6.

Developing and Using Questionnaires. Tra

10.1.7, formerly methodology transfer pap

Case Study Evaluations. Transfer paper 10

formerly methodology transfer paper 9.

Prospective Evaluation Methods: The Pros

Evaluation Synthesis. Transfer paper 10.1.

methodology transfer paper 10.

(973317) Page 115 GAO/PEMD-10.1.10 Prospecti
Ordering Information

The first copy of each GAO report and 

is free. Additional copies are $2 each. 

should be sent to the following addres

accompanied by a check or money orde

out to the Superintendent of Documen

necessary. Orders for 100 or more cop

mailed to a single address are discoun

25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th & G Sts

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling

(202) 512-6000 or by using fax numbe

(301) 258-4066, or TDD (301) 413-000

Each day, GAO issues a list of newly a

reports and testimony. To receive fac

copies of the daily list or any list from 

30 days, please call (301) 258-4097 us

touchtone phone. A recorded menu w

information on how to obtain these lis
Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Postage & F

GAO

Permit No