FASID Evaluators Handbook

Impact Evaluation of
Development Assistance
Edition 2
A Practical Handbook for Designing Methodologically Sound Impact
Evaluations under Budget, Time, and Data Constraints

SHOPImpact Evaluation of
Development AssistanceDesigning
Methodologically Sound Impact Evaluations
Under Budget, Time and Data Constraints
A Practical Handbook
Edition 2
by Michael Bamberger& Nobuko Fujita

Foundation for Advanced Studies on International Development

Copyright © 2003,2008 by Foundation for Advanced Studies on International
Development
Foundation for Advanced Studies on International Development
International Development Research Institute1-16-17 Kudan-minami, Chiyoda-ku, Tokyo 102-0074, Japan
Telephone: 03-5226-0305
Fax: 03-5226-0023

Preface

This handbook is based on materials prepared for an ODA evaluators’ training workshop organized in Tokyo in November 2002 by
FASID (Foundation for Advanced Studies on International
Development).
Simple logical framework-based evaluations are sometimes inadequate to assess the extent to which observed changes in target populations are attributable to program interventions and not to factors unrelated to the program under study. However, most ODA evaluations are
conducted under budget and time constraints which make applicationof many rigorous and costly evaluation designs difficult. Consequently
there is a growing demand for uncomplicated, rapid, economical
designs which also can ensure acceptable standards of methodologicalrigor.
The “shoestring evaluation approach” (now known as “real-world
evaluation”) presented in this handbook draws on the experiences of
ODA agencies, national governments and NGOs who have conductedevaluations in many Asian, African, and Latin American countries
under budget, time, and data constraints. The approach has five steps
for improving the quality of evaluations conducted with such limitations: (1) reduce data collection cost and time by reviewing andassessing different quasi-experimental designs applicable under these
conditions; (2) reconstruct baseline data and control groups when

information on conditions was unavailable before the project
began; (3) compile a checklist of factors which can affect thevalidity of the evaluation’s findings; (4) strengthen evaluation
design and correct factors affecting validity; and (5) build evaluation data generation into the design of new programs. Thesesteps will assist ODA evaluators to produce persuasive andmethodologically valid findings even when working under budget, time or data constraints.
After the workshop, requests for the course materials werereceived from people who were unable to attend. To meet theserequests in a way that would facilitate independent study, thematerials were edited, exercises involving case studies wereadded and a handbook was produced. In response to continued
interest in the challenges of conducting evaluations under realworld constraints, this year we have updated the handbook, making some editorial changes with the assistance of Mrs. Suzanne
Akiyama. We hope this will help evaluators in constructing simple, economical and robust evaluation designs.
For more detailed discussion, see Bamberger, Rugh, Marby(2006) Real World Evaluation, Sage. Also, a recent summary of
the material discussed in this handbook has been compiled as a
35 pages color booklet, “Conducting quality impact evaluations
under budget, time and data constraints”(2006), which can be
downloaded free of charge from www.worldbank.org/ieg/ecd.
March 2008
Michael Bamberger
Nobuko Fujita

TABLE OF CONTENTS

Preface
Part I:

Introduction to Impact Evaluation

Chapter 1
Chapter 2

Part II:

Overview: Opportunities and Challenges ………………… 3
Introduction to the Theory and Practice of
Impact Evaluation Design ………………………………… 11

The Shoestring Evaluation Approach

Chapter 3
Chapter 4
Chapter 5

Reducing the Cost and Time Required for
Impact Evaluations ……………………………………… 35
Reconstructing Baseline Data and Control Groups ……… 40
Identifying and Addressing Threats to Validity ………… 44

Part III: Using the Shoestring Evaluation Approach at the Project,
Sector and National Policy Levels
Chapter 6
Chapter 7
Chapter 8
Chapter 9

Shoestring Evaluation at the Project Level ……………… 57
Shoestring Evaluation at the Sector and Program Level … 65
Shoestring Evaluation at the Country and Policy Level … 69
Building Evaluation into Project and Program Design …… 77

Annexes:
1. Threats to Validity in Interpreting Evaluation Findings ………… 85
2. Shoestring Project Evaluation Worksheet ………………………… 88
3. Shoestring Program Evaluation Worksheet ……………………… 91
4. Shoestring Policy and Country Evaluation Worksheet …………… 94
5. Example of a Completed Shoestring Project Evaluation
Worksheet: Using the worksheet to assess an alreadycompleted evaluation of alow-cost housing project ……………… 97

6. Case Study-Longitudinal Impact Evaluation of the Eritrean
Community Development Fund Feeder Road Component ……… 101
7. Case Studies - Three Approaches to Evaluating
the Gender Impacts of Micro-Credit Programs in Bangladesh:Different evaluation paradigms produce different findings ………116
Tables and Figures
Table 1:
Table 2:
Table 3:Figure 1:

Strengths and Weaknesses of the Five Most
Frequently Used Quasi-Experimental Designs ………… 30
Rapid and Economical Methods for
Impact Evaluations ……………………………………… 39
Strategies for Addressing Common Threats to
Validity in Shoestring Evaluations ……………………… 51
The Project Evaluation Framework ………………………31

Part I
Introduction to Impact Evaluation

Part I

3

Chapter 1
Overview: Opportunities and Challenges

1-1 Why impact evaluations?
In program theory-based evaluations, often applied to ODA projects,
observed changes in performance or output indicators are compared with thepre-project situation (“before/after” comparison). This approach assumes thatobserved outcomes are results of the project, “more or less.” Unfortunately,
this assumption is almost never valid because almost all projects operate in
dynamic environments where many social, economic, political, environmental
and demographic changes are occurring, any of which might have influenced
the observed outcomes. An assessment of impacts, changes actually
attributable to a particular project, requires some “with/without” comparison.
Impact evaluation is a way of dealing with this issue. For example, in theevaluation of the impact on poor households of the EI Salvador housing
improvement project discussed in Chapter 2, a 70.0 per cent increase wasobserved in the average household income of project families when thesituation “before” the project began was compared with the situation “after”
the project had been implemented and families had moved into their new
houses. Using the theory-based model described above, the evaluator would
assume that the project had had a significant impact on household income andmight have reported that “evaluation of this project suggests that investment in
low-cost housing is an effective way to increase the income of poor
households.”
However, the EI Salvador evaluation included a comparison with a control
group of similar families who were interviewed at the same time as the projectgroup, both before and after the project. The comparison revealed that over the

4same time period, average household income for the control group had
increased by 74.6 per cent, slightly more than for the project beneficiaries(although the difference between the two groups is not statistically significant).
Using the control group comparison, the evaluation report might now draw
exactly the opposite conclusion, that “evaluation of this project did not findany evidence that low-cost housing is an effective way to increase the incomeof poor households.“
Impact evaluation permits the evaluator to assess whether the observedchanges can be attributed to the project impact and the extent to which
projects, programs, and policies have produced their intended impact andbenefited the intended target populations.
“Impact” here refers to “net project impact,” which means “total observedchange” minus “change which should be attributed to other factors not related
to the project” (see the following figure). Impact can be positive or negativeand intended or unintended. Positive, intended net impact indicates that the
program theory was valid and accurately explains how and to what extent the
project contributed to the observed changes in the target population.
(net)
impact=total changes—

changes not
related to the project

Identification of causal attribution, while one of the most difficult issues in
evaluation, is necessary to be sure about the effectiveness of investment.
Whether or not the result or outcome of an intervention was due to the projector to other separate factors has serious implications when deciding whether a
project should be repeated elsewhere or implemented on a larger scale. An
understanding of the true magnitude and direction of project impact could
result in savings, more effective allocation of resources and avoidance ofinvestments in projects which do not really produce significant benefits.
Impact evaluation is useful also for determining which components of the
program theory worked the best. Suppose that a project with several different
components (activities) was considered to have been very successful. By
conducting impact evaluation, the relative contribution of each component tooverall success can be assessed; in addition, an understanding can be gained ofwhich elements of the theory proved to be correct in particular context. This
information can be very useful when planning whether and how to replicate aPart I

5

project. Impact evaluation can be useful also for assessing the validity of the
program theory and for testing some of the critical assumptions andhypotheses on which it is based.

1-2 Challenges and constraints in designing impact evaluations
Unfortunately, ODA projects/programs, and even most evaluations of socialand economic development programs in industrialized countries, face serious
limitations in conducting rigorous social experiments to identify causality
accurately. Evaluations are carried out with limited funds and time, andfrequently without baseline data. Evaluators develop innovative ways to
produce “acceptable” findings even while unable to follow all of the standard
evaluation design principles. As a result, a series of “quasi-experimental
designs”(QEDs) have been devised to provide the best possible estimates ofproject impacts under real-life situations which almost never approximate
laboratory research conditions. Chapter 2 describes the strongest quasiexperimental design (Model 1) which development evaluators normally seek
to follow. It also describes four methodologically weaker models which areused quite widely in situations where it is not possible to follow all of theconditions for Model 1.
Following are some of the limitations, constraints and challenges
encountered in designing impact evaluation for development
projects/programs:
a. The evaluator is not consulted until the project is already being
implemented. The evaluator frequently is not called in until the project isalready quite advanced. This creates numerous difficulties, including lackof baseline data (discussed below) and pressure to start the studies
immediately without taking the time to fully understand the project and plan
the evaluation. This sense of urgency also may create pressure to ignore
some of the basic principles of good evaluation design. A much morefavorable situation allows for initial consultations during the project design
stage so that appropriate measures can be taken to collect and organize
information for eventual evaluation.

6b. Limited baseline data. Evaluation studies which do not start until late in the
project cycle usually have little or no baseline data on the conditions of thebeneficiary group prior to the beginning of the project. Even if records areavailable, they tend not to be organized in the form that evaluators require
for before and after comparative analysis.
c. No control group. Even if some baseline data exists, most project managers
collect data on the target groups or affected populations only. The
possibility that the collection of information on non-beneficiaries might
create in these people an expectation of financial or other compensation
further discourages such data gathering. For these reasons, efforts to create
a control (or comparison) group at mid-project or after project completion
usually are hampered by insufficient data.
d. Time Constraints. Many evaluation studies must be conducted within aperiod of several months with only a few weeks of site visits. Nonetheless,the results of the analysis are expected to be presented in a timely-manner.
e. Budgetary constraints. Frequently funds for evaluation are not included in
the project budget. As a result, funds to apply data collection instruments(tracer studies or sample surveys, for example) are often not available,
making it difficult to use some of these methods to reconstruct baseline dataor create control groups.
◆ Additional complications for sector and national level/policy evaluations

While for the reasons given above it may be impossible to use the ideal
impact evaluation design, a project evaluator still has a good idea of theevaluation design she or he would like to approximate. But as the scope of theevaluation broadens to sector or national level, it becomes more difficult eventheoretically to define the appropriate design. Additional complications thismight raise include the following:f. Defining the appropriate counterfactual 1. Sector and national interventions
operate in a very complex environment where many economic, social,political and other changes are taking place simultaneously and many actors
are involved. This increases the challenge to develop an appropriate
1

A counterfactual is an estimate of what the situation would have been if the policy or program had not been
implemented. Chapter 8 discusses some of the methods for defining counterfactuals.

Part I

7

counterfactual to define what would have been the situation in the absenceof the particular program or policy intervention.
g. Data deficiency. Sector or country level evaluations frequently have to relyon secondary data collected for other purposes. Even when data is available,
it may not include all of the needed information for the right time periods.
Chapter 3 describes ways to reduce the cost and time required to conductimpact evaluations. Chapter 4 deals with reconstructing baseline data andcontrol groups.

1-3 Threats to Validity
Because of the constraints discussed above, there are a large number offactors which could lead the evaluator to make a wrong assessment about
project impacts. In some cases it might be inferred incorrectly that the project
has produced certain impacts; in other cases the evaluator might overlook
some impacts which the project has actually produced. These factors are called“threats to validity.” (See Chapter 5 for detailed discussion.)
Many of the approaches to reducing evaluation cost and time mean thatprinciples of technically sound impact evaluation design cannot easily beapplied. Less time and resources may be available for activities such as
developing sampling frames, instrument development and testing, ensuring
adequate sample size, and general issues of quality control. Often the demand
for rapid and low cost delivery of evaluation findings takes precedence overmethodological rigor.
An additional set of issues concerns the representativeness andgeneralizability of findings from qualitative/participatory studies. Frequently
very little documentation is presented on how interviewees were selected, who
actually attended a group discussion and the level of participation by people ofdifferent categories (for example, women or young adults) in the discussion.
There may be a similar lack of documentation with respect to the selection of
the communities or groups for whom case studies are prepared. These factors
all can affect representativeness and generalizability. The fact is that methods
do exist for addressing all of these issues in qualitative research (see, forexample, Patton 2002) but the necessary controls are often not used in ODA

8evaluations due to time and resource constraints.
Countering threats to validity requires particular care when assessing andinterpreting the findings of rapid and low cost impact evaluations. The
following chapters explain a framework which identifies and assesses four
categories of threats to the validity of interpretations about program impact.
Each category identifies factors which may yield misleading interpretations
about the impact or effects of program interventions. Discussed also are
actions which can be taken to address the different threats, thereby improving
validity. The four categories of threats to validity are the following:a. Threats to statistical conclusion validity. The statistical design and analysis
may assume incorrectly that program interventions have contributed to theobserved changes (impacts), or that some potential impacts may have been
overlooked.
b. Threats to internal validity. Reasons inferred why the observed relationshipbetween a project intervention and an output or impact is causal may beincorrect. Internal validity affirms the following questions “Does theevaluation design prove what it is supposed to prove about the effects of thetreatment on the subjects actually studied?” “Does it prove that thetreatment produced the claimed effect on the experimental subjects?”
Common threats to internal validity include poor instrumentation (datacollection methods), participant maturation (respondents learn over time
how they should respond to the questions), spontaneous change orassignment bias (people in the project group are different from those in thecontrol group).
c. Threats to construct validity. The underlying constructs (hypotheses/
concepts) on which the evaluation design (logic model) is based may notadequately describe the actual indicators of outputsÅCimpacts and settings
used in the study. For example, observed changes in income may notadequately measure the construct “changes in household welfare,”particularly in rural communities where there is a subsistence economy andmost farmers use very little money.
d. Threats to external validity. Reasons inferred how study findings might begeneralized to other settings (regions, social or economic groups, etc.) may
not be valid.

Part I

9

1-4 Realistic approaches and useful ways for impact evaluation
Despite the problems and challenges described above, the increasing
interest in rapid and low cost impact evaluation has produced a number ofpromising approaches, some of which will be discussed in the following
chapters.
a. Refinements to quasi-experimental designs. Recognition of the need toadapt QEDs to the real-life circumstances under which evaluations are
conducted has yielded many ways to simplify and streamline evaluation
designs. These are discussed in the following chapters.
b. Participatory methods. A wide range of participatory methods have beendeveloped to obtain the perspectives of project beneficiaries and otheraffected groups. Many of these methods can be used also to reduce the timeand cost of data collection. For example, groups rather than individual
households can be asked to estimate the impact of a project; i.e., a women’sgroup can be asked to estimate the average time household members spend
daily collecting water, or groups can assess improvements in the
management of community services such as water supply or health clinics.
Applicable participatory techniques include PRA (participatory rural
appraisal), described in publications such as Voices of the Poor (Narayan
and Petesch 2002), and The Poverty Reduction Sourcebook (World Bank
2002)2. Use of focus groups also can be considered a participatory method,
but an experienced facilitator, using explicitly participatory methods, isrequired to ensure that discussions are not directed or dominated by a few
people3.
c. Statistical methods to improve the use of cross-sectional studies.
Multivariate analysis is frequently used to statistically adjust for differences
between project and control groups; hence improving the utility of crosssectional studies is a tool for impact assessment. Statistical methods areused primarily to improve the analytical strength of sample survey designs,
but these methods can be used also to improve the way in which
participants in qualitative studies are selected and the results of the studies2
3

See also Chambers, Robert 1995. Poverty and Livelihoods: Whose Reality Counts?
It should also be mentioned that focus groups are frequently used in market research in a non-participatory
way with participants responding to a set of questions prepared by the researcher.

10

are analyzed4. Using various designs and approaches, rapid and low cost
impact evaluations have become possible not only at the project level(Chapter 6), but also at the sector and program level (Chapter 7), and at thecountry and policy level (Chapter 8). It is now understood that the mostefficient way to run impact evaluation is to build evaluation into project,program, and policy design, and Chapter 9 discusses some practicalmeasures for ensuring that the collection of data needed for possible future
impact evaluation is built into the project operating systems -- even in cases
where the possibility of conducting a future impact evaluation has not yet
been discussed with project management.

4

A counterfactual is an estimate of what the situation would have been if the policy or program had not been
implemented. Chapter 8 discusses some of the methods for defining counterfactuals.

Part I

11

Chapter 2
Introduction to the Theory and Practiceof Impact Evaluation Design

2-1 True experimental design and quasi-experimental design◆ True experimental design

True experimental design is used in such fields as medicine, animal
behavior and educational research studies conducted under carefully controlledlaboratory conditions. In the simplest design, subjects are randomly assigned
to the Experimental [E] Group, which will receive the treatment [X] (forexample, a new drug, or rewards/punishments used in animal research or
school programs), and the Control Group [C] which does not receive thetreatment. A test is applied to both groups in Time Period 1 [T1] before theexperiment begins to measure the behavior, physiological reaction or othervariables the treatment is intended to influence. The measurements arerepeated in T2 following the application of the experimental treatment. The
measurements in T1 and T2 are defined as E1 and E2 for the experimental group
and C1 and C2 for the control group. The research design is described below:
The simplest true experimental design

T1

Experimental group
Randomized control group

Experimental
T2
Treatment
E1___________X___________E2
C1_______________________C2

12

Assuming that the assignment of subjects to the two groups was truly
randomized and that the experiment was conducted under carefully controlledlaboratory conditions, the impact [I] produced by the experimental treatment[X] can be estimated by comparing the measured change in the treatmentgroup with that in the control group5.
Normally a number of different indicators will be used to assess the effects(impacts) of the treatment. In assessing the effects of a drug on learning ability
in first grade students, the indicators might include, for example, the numberof new words learned, the number remembered after one hour, after one day,after one week etc. These different impact indicators are normally referred toas I1, I2, . . . . In. If the value of impact indicator I1 differs significantly fromzero (either positively or negatively) then there is some preliminary evidence
that the treatment did have an impact. However, experiments have to berepeated many times under different conditions and usually with different
groups before it is possible to speak with confidence of the efficacy of thetreatment.
◆ Quasi-experimental designs

When evaluating the impact of development projects (water supply, road
construction, micro-credit, teacher training, provision of teaching materials,etc.), it is almost impossible to approximate the true experimental design levelof experimental control. For example, it is rarely possible to randomly assign
subjects to treatment groups and control groups, and treatments cannot beapplied in such a precise way. Consequently a series of quasi-experimental
designs (QEDs) have been developed to approximate as closely as possible thetrue experimental design, for the following purposes:• To make the best possible estimate of the extent to which a project,program or policy has produced its intended impact.
• To identify the factors which positively or negatively influence the5

If the observations E1, E2 and C1 and C2 refer to the mean scores for each group (for example, average
income, average educational test score or average anthropometric score), then a statistical test such as the tTest for the difference of means is used to determine whether the observed difference is statistically significant. If, on the other hand, the values refer to proportions (for example the proportion of children attending
school or the proportion correctly answering a test question), then the appropriate statistical test would be ameasure such as the Z score for difference of proportions (Herbert Blalock Social Statistics Chapter 13.
McGraw Hill 1972)

Part I13

magnitude and direction of the impact.
In the real world, evaluators using QEDs typically face the following
problems:• It is almost never possible to randomly assign subjects to experimentalgroups and control groups. For logistical reasons, most projects areaccessible to or affect everyone in a given community or area. For
example, a school or water supply system will be accessible to allfamilies and it is clearly not possible to tell some families they cannot
use the water or send their children to the school.
• Some projects use a self selection process, whereby, for example,
people decide if they wish to apply for micro-credits, enroll in a literacy
class, or plant new varieties of seed. In these cases it is likely that thepeople who do decide to participate will be different in important ways
from those who do not participate.
• Typically people who take the initiative to participate are economically
better off, better educated, and have more self-confidence.
Consequently, it is difficult to know whether observed changes inincome, reading skills, health, etc. are due to the effects of the projector to the differences in the initial conditions of participants and nonparticipants.
• It is very difficult to find a control group which closely matches theexperimental group on the key indicators. Project communities areoften selected because of special characteristics. In some cases projectplanners choose the poorest communities, in other cases they choose
communities which have the greatest likelihood of success. In eithercase it will be difficult to find a control group which closely matches
the project population.
• In many cases, for political or ethical reason it is difficult to use any
kind of control group at all. Frequently politicians and community
leaders in control group areas will exert pressure for their community to
be included in the project. It is often considered ethically inappropriate
to ask families to spend time responding to surveys if they will notreceive any benefit. The fact that families are being interviewed
sometimes creates false expectations that they will be eligible toparticipate in a later phase of the project.

14• It is also difficult to ensure that treatments (services) are administeredin exactly the same way to all project sites and families. Sometimes thedelivery of materials and equipment is delayed, in other cases there aremajor differences with respect to the organization of the project anddelivery of services in different sites. In one micro-credit program thelocal administrator may speak the local language and may create awelcoming atmosphere, encouraging families to visit the project todiscuss loans. In another site the administrator may not speak the local
language, the project may be seen as a hostile to the community andfewer people visit the center. For these reasons it is difficult todetermine whether differences in project performance are due todifferences in the responsiveness of different communities, or whether
the differences are due to the way the project was differently
administered.
• Finally, each project operates within a unique economic and politicalcontext. Each must interact with a number of government or nongovernment organizations which have their own particularcharacteristics. Also, the social, economic and cultural characteristics
of target population may vary significantly among project sites. All ofthese contextual factors can have an important influence on the project
outcome. As a result of these contextual factors, even when a project isadministered in exactly the same way at each site, there may besignificant differences in the outcomes.
Several lessons can be drawn from these evaluation design difficulties.
First, it is important to understand the problems facing a particular study and toselect the methodologically strongest design possible under the particular
circumstances. Second, the strengths and weaknesses of the evaluation design
should be carefully analyzed and the implications for the interpretation offindings and recommendations assessed. In some cases the methodologicalweaknesses may not seriously affect the kinds of recommendations to beprepared, whereas in other cases they may be very serious. For example:• The lack of a control group may not be very important if the purpose of
the evaluation is to assess whether indigenous communities participatingin pilot projects are able to manage and sustain community water supply

Part I

15

projects; or whether women will apply for small loans if a loan office
staffed by local language speakers is established in the community.
• On the other hand, if the purpose of the evaluation is to estimate whether
a pilot project could be replicated on a national scale; or whether it
would offer a more cost-effective way to deliver a particular service,
then the lack of a control group might be a serious problem.
Finally, if methodological problems are identified which seriously affect the
purposes of the evaluation, then the evaluator should consider measures torectify the problems (see later chapters for discussions of these measures). One
example of a serious problem is the common situation in which no baseline
study was conducted at the time the project was planned so there is no reliable
quantitative information available on, for example, school enrolment, distances
traveled in vehicles or on foot, or water consumption before the project began.
Consequently, even if precise information on enrolment, travel or water
consumption is collected after the project is implemented, it is difficult toassess the magnitude of changes which have occurred.
Under these circumstances, some of the possible tools which could be used
to estimate the baseline conditions are as follows:• focus groups in which community residents are asked to estimate theimpact of the project• a rapid sample survey in which families are asked to recall whichchildren went to school, how much water was consumed, etc., before theproject began.
• key informants such as community leaders, local health authorities,
school teachers etc. could also be asked to assess the impact of the
project
An important aspect of this approach is the use of triangulation (consistency
checks) to compare information obtained from different sources. If theinformation from all of the sources is more or less consistent, then the evaluator
can have confidence in the findings. If, on the other hand, the information fromdifferent sources is inconsistent or even contradictory, then further analysis isrequired to determine if the inconsistencies can be reconciled.

16

2-2 The evaluation framework for project impact evaluation
Figure 1 identifies seven stages in the project cycle which can beconsidered in impact evaluation design. Readers familiar with logicalframeworks will see many similarities to the LogFrame format and thisevaluation model can be coordinated with the LogFrame if that format isalready being used. The evaluator will often find that there is no written
documentation defining the project model and she or he will have to work withplanners and project managers to agree on a definition of project objects andcritical assumptions. The seven stages are as follows:
1. Project planning and design: This examines the following• the project's approach to planning (for example, central planning or
participatory consultations)• the information sources on which the project is based and theiradequacy . For example, how adequately do the surveys (and other data
sources) cover all sectors of the target population and how well do they
provide the information required for planning this project.
• whether the surveys (and other data) provide information on the
different needs and constraints of adults and children, men and women,
different ethnic groups and people engaged in different kinds of
economic activities• whether a systematic stakeholder analysis was conducted to ensure thatall sectors of the target group were consulted• the critical assumptions on which the project design was based. For
example, for a micro-credit program intended to benefit both male andfemale farmers some of the critical assumptions might include thefollowing:- Lack of credit is a major constraint to women's economic activities.
- If women receive credit they will be able to start up or expand
economic activities.
- If women start economic activities they will be able to control how
the profits are used.
- Increased income in the hands of women will improve their economic
and social welfare, giving them more influence in household andPart I

17

community decision-making.
This information can be used in at least two ways in the evaluation design:
first, to assess how well the project was planned and the quality of theinformation used; second, to develop indicators to monitor the validity of the
key assumptions as the project evolves. For example, if women's social andeconomic welfare did not improve, was this due to the following situations:• Even when women received loans they did not start up businesses.
• If they did start up businesses, a male household member often
controlled use of the profits.
• Even if women controlled the profits, this might not affect conventional
indicators of economic and social welfare (income, consumption,
expenditures, etc.). For example, a number of studies in India haveshown that women often save all of the profits from their business toprovide a dowry for their daughters.
2. Project inputs : This identifies the materials, money, staff, equipment,
extension workers, consultants, capacity building and other resources identified
in the project plan. The use of these inputs should be monitored because one of
the main reasons many projects do not achieve their intended impacts is that ahigh proportion of the resources never reach the schools, clinics or other service
centers through which project is implemented on ground6.
3. The project implementation process : Projects can be implemented indifferent ways. Some involve the community in planning and administrationand others are managed directly by the implementing agency (Ministry ofTransport, Agricultural Investment Bank, etc.). Projects also vary in terms of
the ease of community access to the services. For example, if a micro-credit
program is administered by the local branch of a large agriculturaldevelopment bank, it may be difficult for poor families without transport, or
women with small children, to reach the bank. Consequently this program may
6

A number of recent public expenditure tracking studies conducted by the World Bank in countries such as
Uganda, Tanzania and Ghana have found that in some cases as little as 10% of the project resources
reached the frontline services. There was some corruption, but often most of the education funds transferredfrom the Ministry of Finance to the General Fund of a local government agency were simply diverted toother purposes.

18

not reach the poorest farmers.
4. Project outputs or products: Projects are intended to achieve a set ofquantifiable outputs or products: i.e., the number of children attending schoolor continuing from primary on to secondary school, the number of families
with access to good drinking water, the number of micro-credit loans approved
and the number of small businesses started, the kilometers of roads orfootpaths constructed or maintained. There may also be outputs which areassessed qualitatively, such as the quality of leadership training or the strengthof community groups created.
5. Outcomes or short-term impacts: These are the impacts which are achieved
within a relatively short period of time, perhaps 6 to 12 months after projectcompletion. Poverty reduction programs frequently identify four types ofimpacts7:• Opportunity: Access to economic resources and improved economic
conditions• Capability: Access to public services (health, education etc.) and theaffect on human development indicators such as anthropometricmeasures, years of schooling, frequency of use of public transport• Security: Economic, environmental and personal security• Empowerment and voice: Participation in decisions affecting the social,economic and political life at the household, community and local
government level. This may also include access to information andcontrol of the means of communication.
6. Medium and long-term impacts: These are assessed on the same four
dimensions as the short term impacts, but given the longer time frame, broader
assessments are possible. For example, access to education can also include
access to labor markets after school completion.
7. Sustainability: The overall objective of a project is not simply to produce
impacts during the life of the project, but to ensure that the impacts are
7

These categories are based on the 2000/2001 World Development Report: Attacking Poverty; and theWorld Bank Poverty Reduction Strategy Sourcebook (see especially the chapter on gender).

Part I19

sustained over time. For example:• Schools and clinics continue to function after donor funding has ended• Communities are able to maintain minor irrigation works, rural roads
and bridges, and the bus company is able to maintain its fleet◆ Contextual factors affecting project outcomes

An important feature of this evaluation framework is its recognition thateach project is implemented in a particular economic, political, institutionaland socio-cultural context. Consequently, due to contextual factors evenprojects always implemented according to the same design may varysignificantly in outcomes, impacts and sustainability from one project site toanother. The model identifies three sets of contextual factors to be taken into
account in the evaluation:• Economic and political factors: A job training program is likely to havedifferent outcomes in areas where the economy is growing than in areas
with high unemployment and economic decline. Similarly, families
may be less inclined to invest in their children's education if the labormarket is very tight. The local political context is also relevant. A
project in a region where the local government is in the hands of an
opposition party may find it more difficult to obtain support fromnational authorities, or an impending local or national election might
affect project dynamics. There have been cases, for example, where alocal political candidate told farmers not to apply for small businessloans or pay service charges for water because if he were elected all ofthese services would become free to the poor.
• Institutional and organizational factors: Projects depend to aconsiderable extent on the efficiency and support of local government
agencies. For example, if the local office of the Ministry of Health,
Education or Transport is poorly managed, or if it has an acting director
and is short of staff, this will likely affect the efficiency and impact ofproject implementation. Non-government agencies also play animportant role in project implementation and an assessment of theefficiency of their operation also may be needed. Finally, among
government agencies or between government agencies and NGOs there
may be conflicts which affect project implementation.
• Social, economic and cultural characteristics of participating

20

communities: Often important differences in the social and culturalcharacteristics of participating communities can influence project
implementation and impact. This factor is particularly important where
there are ethnic differences but the analysis should be conducted for all
projects.
Contextual analysis can be used at any stage of the project cycle. For the
purposes of impact evaluation, it is particularly useful for explaining
differences in project impacts at different sites which are not explained by how
well or badly the project was implemented. Contextual analysis is usuallybased on qualitative methods: participant observation, meetings with
community leaders, focus groups and interviews with key informants(journalists, academics, NGOs, religious organizations, local government
agencies etc.). Secondary sources such as newspapers and university studies
can also be useful.

2-3 The most commonly used quasi-experimental designs
All of the following models can be strengthened if used in
combination with the evaluation framework described in section 2-2 andby using some of the shoestring evaluation methods of later chapters.
For the models which include a control group (Models 1, 2 and 4)
multivariate analysis may also be used to statistically control fordifferences in characteristics. See Table l for a summary of the strengths
and weaknesses of each model.
Model 1: The strongest general purpose quasi-experimental design
There are a number of changes in the symbols used to describe the QEDmodels reflecting differences from the true experimental design8. One of the8

The differences are: 1) [E] (experimental group) is replaced by [P] (project group) reflecting the fact that
this is an impact evaluation rather than an experiment; 2) [X] represents the project intervention’ rather thanthe experimental treatment; and 3) in most cases “non-equivalent control group” is used rather than “randomized control group” to reflect the fact that for most QEDs it is not possible to randomly assign subjects
to control and project groups. Italics are used to reflect the non-randomization of the control group.

Part I21

differences is that the post-project observation for the QEDs is defined as T3
rather than T2, as in the case of the true experimental design. This is because
during the project implementation process some QEDs include an observation
which is defined as T2. The evaluation of the Eritrean Feeder Roads Projectdescribed in Annex 6 is an example of a longitudinal design which includes
several observations during the period of project implementation.
For most purposes Model 1 is the strongest and preferred QED. This model is
described as follows:
T1

Project group
Non-equivalent control group

Project
T3
Intervention
P1___________X___________P3
C1_______________________C3

In this model a non-equivalent control group [C] is selected at the start ofthe project to approximate as closely as possible the project beneficiary group[P]. The term ''non-equivalent'' in italics reminds that it is rarely possible toassign subjects randomly to the project and control groups; thus, between thecharacteristics of the project and the control groups there may be differences
which could distort the interpretation of the findings. The project and control
groups are both interviewed in time period 1 [T1] before the project begins,
and information is obtained on a set of indicators [I1, I2 . . . In] measuring thechanges (impacts) the project is intended to produce (for example, increase inhousehold income, reduced daily travel time, number of children attending
school). Information is also collected on the social and economic
characteristics of the individuals or families [x1, x2, . . xn], called intervening
variables, which might affect project outcomes . Data collection is repeated intime period 3 [T3] after the project has been in operation long enough to haveproduced its intended impacts. Ideally the analysis will include the contextual
factors discussed in the previous section.
The analysis can be considerably strengthened if multiple regression
analysis is used to control statistically for differences in the social and22

economic characteristics of the project and control groups 9. Multiple
regression analysis matches subjects statistically on such characteristics as age,income and education to ensure that observed differences in the impact
indicator are not due to differences between the project and control groups onthese intervening variables. The analysis determines whether after controlling
for these household characteristics, differences remain between the two groups
with respect to the impact indicator (income, years of schooling, water
consumption etc.). The analysis does not guarantee that the differences are duenecessarily to the project, but the more that other factors are eliminated as
possible explanations, the more likely it is that the project contributed to theobserved changes.
Many refinements can be introduced into the basic QED design to assess
multiple treatments or to capture impacts which evolve gradually over time
(Shadish, Cook and Campbell 2002; Valadez and Bamberger 1994).
Example of Model 1:
Evaluating the impact of improved housing on households in El Salvador
A four year evaluation was conducted in 1976-80 in EI Salvador to assess
the impact of improved housing on poor households in San Salvador, thecapital. In 1976 a randomly selected sample of households was interviewed
shortly before they entered a self-help housing construction project. A control
group was selected by combining samples of randomly selected families fromthe three types of inner-city housing from which the project participants had
been selected. The samples were repeated in 1980.
It was found that between T1 and T3 the average household income for project
participants had increased by 70.0 per cent compared to an increase of 74.6 percent for the control group. This means that there was no evidence that improved
housing had a positive impact on income; in fact, the income of the control
group rose slightly faster. This illustrates the importance of a carefully selected
control group. If only project participants had been studied one might have
The simplest form of a multiple regression analysis of project impact is specified as follows:
I1 = f [D1, x1, x2, . . . xn] where: I1 = impact coefficient; Dl = dummy variable’ for project and control groups;
and x1, x2, . . . xn = attributes of the individual (age, sex, education), the household (income, family size, typeof housing etc.), or the community which might affect the estimated impact. The analysis is testing whether
there is a statistically significant difference in the impact coefficient for the project and control groups after
adjusting for individual, household and community characteristics.
9

Part I23

concluded that “improved housing has a significant impact on household income
because the income of participants in the low-cost housing project increased
by70 per cent in four years.’’ (Source: Valadez and Bamberger 1994. pp. 237-8)◆ Cheaper and faster -- but usually weaker-- QED

There are many situations in which it is not possible to use Model 1. In
some cases time and budget constraints do not permit the use of a control
group. In other cases the evaluator is not called in until the project is being
implemented so it is not possible to go back in time and collect baseline data.
A number of simpler and more economical QEDs can be used in thesesituations. However, each successive model sacrifices one or more essential
elements of a sound evaluation design, becoming vulnerable to a wider rangeof methodological problems.
Model 2: No pre-test control group
In Model 2 a baseline survey is conducted with the project’s intended
beneficiaries before the project begins, but no control group is used at thisstage. A control group is selected once the project is operational and an ex-post
survey is conducted in time 3 [T3]with both project and control groups. The
model is represented as follows:
T1

Project group
Non-equivalent control group

Project
T3
Intervention
P1___________X___________P3
C3This design works reasonably well for assessing how a project is being
implemented and whether it is able to produce the intended outputs. It also
allows comparison of the project and control group characteristics. For
example, with a rural road construction project, surveys and participatory
consultations with the community may have identified a number of factors
affecting the willingness of the community to participate in the project and thebenefits they obtain from it. These factors might include the following:whether local culture permits women to participate in road construction and to24

travel to market, the community’s distance from the local market and theagricultural surplus available to sell. A control group, if well selected, could
rate other local communities on these variables and hence determine thelikelihood that the project would be well received and might have an impact inother areas. The project and control groups could also be compared on
indicators such as amount of produce sold in the local markets, average
number of trips and distance traveled and kinds of consumer goods available
in community shops.
However, this design has some weaknesses. Most importantly the lack ofcontrol group baseline data means that it is not possible to determine whether
observed differences between the project and control groups in T3 are due tothe project or were pre-existing before the project began. Another weakness isthat we cannot control for local history which might have affected outcomes.
This is particularly significant for projects seeking to increase agricultural
output or sales. Sales of maize or wheat may have increased because of goodrains and not because of the project. The ex-post control group can provide
some information on this but the analysis will obviously be much stronger ifchanges in the project and control areas can be compared over time.
Example of Model 2:
Comparing the effects of resettlement on project beneficiaries and nonbeneficiaries in the second Maharashtra Irrigation Project, India
Sample surveys were conducted periodically between 1978-85 in areasfrom which families were to be resettled as a consequence of a large-scale
irrigation project. The study only covered families who were eligible to receive
land or housing plots in the relocation areas. The surveys were repeated in1990 after the relocations had taken place. An ex-post control group survey
was conducted in 1990 with a sample of families who had remained in theirrigation project command. This was not an ideal control as many of thesample households received families who had been forced to move as a resultof the dam, so their situation did not really represent families not affected by
the project. Recognizing that no information was available on theapproximately 45 per cent of families who were forced to relocate but who
were not eligible for compensation, a tracer study’ was conducted in 1990 totry to identify them. The study found that the economic conditions of mostPart I

25

families receiving compensation had improved. The situation concerning thefamilies who had not received compensation was more mixed, but in general,
forced resettlement appeared to have had less negative consequences than had
been expected. (Source: Valadez and Bamberger. 1994. pp. 264-266)
Model 3: No control group
In this model there is no control group and the analysis is based on acomparison of the project group before and after implementation. The model is
described as follows:
T1
Project group
Non-equivalent control group

Project

T3

Intervention
P1___________X___________P3

This model works reasonably well for projects having large and clearly
defined impacts; for example, the construction of a village school or clinic
where there had previously been no such facility within easy access. It can also
work well when the purpose of the evaluation is to understand the project
implementation process and where quantitative assessment of impacts is lessimportant.
This model does not work well, however, when precise estimates of themagnitude of project impacts are required. It also does not control for the
influence of local history. The lack of comparative data on the project groupand other communities also means that it is difficult to assess the potential for
replicating the project on a larger scale. For example, if the project was
successful because communities with higher than average levels of educationand income had been selected, the lack of control group would make it
difficult to assess how successful a larger project would be if extended to more
typical communities with lower education and income.

26

Example of Model 3:
Evaluating the impact of the feeder roads component of the Eritrea Social
Fund
In 1999 local consultants were commissioned to conduct a rapid evaluationof each of the eight components of the Eritrea Social Fund. Due to budget andtime constraints, it was decided that for six of the eight project components
only an ex-post survey of beneficiary communities would be possible.
However, for the feeder roads component (as well as one other component) it
was decided to conduct a longitudinal impact study in which surveys and otherforms of measurement (direct observation, key informants, counts of roadtraffic, etc.) would be taken at the start of the project, and again after 6 and 9months. No control group was used. The study found that the construction of
the road had a major impact on the production and marketing of agricultural
produce, mobility on foot and by vehicle, and use of schools and healthfacilities. Secondary data (such as market records and vehicle registration
records) were used to compare changes in surrounding regions (reconstructionof a control group) but it was difficult to get precise estimates due to thelimited availability of records and the considerable fluctuations in market
prices. (Source: Annex 6 case study)
Model 4: No baseline data
This model relies entirely on data collected after the project has been
implemented and no baseline data is collected on either project or control
areas. The model is represented as follows:
T1
Project group
Non-equivalent control group

Project

T3

Intervention
X________________________P3
C3This model can be used to obtain an approximate estimate of project
impacts. It works best in isolated communities where the project is the only
major outside intervention. Under these circumstances there is no need toPart I27

isolate the effects of other interventions which might be taking place at thesame time. It can also be used to compare the characteristics of project
participants with people from other similar communities. If project households
have characteristics similar to other communities, then it is more likely that theresults of the pilot project can be generalized. If, on the other hand, there are
significant differences between the groups it will be more difficult togeneralize.
This model does not control for historical events which may have affected
outcomes, and it has the same weaknesses as the earlier models which do notcollect baseline data. It is also not possible to evaluate the project
implementation process.
Example of Model 4:
Assessing the impact of micro-credit on the social and economic conditionsof women and families in Bangladesh
In 1991-92 a random sample of households were interviewed from a sample
of rural Bangladeshi villages in which village banks were operating. A control
group was interviewed in villages where no village bank programs wereoperating. The surveys were conducted ex-post, when the village banks had
been operating for several years, and no baseline information was collected on
the condition of the families prior to the banks’ operation. It was found thatborrowing from a village bank had much greater impact on women than onmen (although the latter also benefited). Per capita household expenditures
increased almost twice as fast for women, housing conditions improved andpersonal savings increased. Interestingly, it was found that contraceptive usage
declined for women borrowers and their fertility increased. The lack ofbaseline data made it difficult to determine the extent to which the observed
differences between the project and control groups were due to the effects ofthe project; or, at least in part, to differences existing before the project began.
(Sources: Khandker 1998; Pitt and Khandker 1998; Baker 2000 (Annex 1. 2);
and World Bank. 2001).
Model 5: Eliminating baseline data and control groups
This is the weakest QED. Only the project population is studied and only

28

surveyed after the project has been implemented. The model is represented asfollows:
T1
Project group
Non-equivalent control group

Project

T3

Intervention
X________________________P3

This model works reasonably well for exploratory studies, when the
purpose is to get a general idea of whether the model works. It can also be
used to get a very approximate estimate of impacts. This model works betterfor relatively isolated projects where the potential impact is expected to bequite large. Among its limitations are the following: It cannot be used to obtainreasonably precise estimates of impact. It cannot control for local history
events which might affect outcomes. And, since it does not provide any
comparative data on the characteristics of the project population, it cannot be
generalized to a wider population.
Example of Model 5:
Assessing the education impacts of the Eritrea Social Fund.
In the evaluation of the Eritrea Social Fund referred to earlier, an ex-post
survey was conducted in 48 communities representing the catchment areas for
10 newly constructed primary schools. No control group was used but baseline
data on school attendance prior to the construction of the schools wasestimated by asking families to recall the situation before the schools werebuilt. Recall data seems to have been very reliable because it was easy forfamilies to recall whether their children attended school before the village
school was built and because they had no incentive to give wrong information.
The analysis focused on the following topics:• Process evaluation: More than 90% confirmed that the school was ahigh priority but only 37% had attended meetings to participate inplanning the project.

Part I29

• Accessibility impact and gender: The schools were successful inreaching the poorest sectors of the community but it was more difficult
to involve recently returned refugees as they were still unsettled and not
motivated to send their children to school. Families are equally
motivated to send boys and girls to school, but if they were to choose for
economic reasons, they would normally give priority to the boy.
• Social impact: School construction reduced travel time for students by
one half to two thirds.
• Sustainability: Despite extreme poverty almost all households
contributed the required 10% of the cost of the school in cash, labor ormaterials.

Part II (Chapters 3-5) will present the main elements of the shoestring
evaluation approach, which is designed specifically for use in situations where
evaluators are working under budget and time constraints and where theyfrequently do not have access to baseline data on the conditions of the project
population before the project began. The approach can also be used moregenerally, to ensure that the highest possible standards of methodological rigor
are employed in all evaluations conducted under budget, time and data
constraints. The approach comprises the following main elements:• Reducing costs and time required to conduct impact evaluations(Chapter 3)• Reconstructing baseline data and control groups when these were notincluded in the original evaluation design (Chapter 4)• Identifying factors which may affect the interpretation of whether andhow the project contributed to producing the intended impacts, i.e.,threats to validity (Chapter 5)• Addressing potential problems affecting the validity of evaluation
conclusions once the problems have been identified (Chapter 5)
Part III (Chapters 6-9) will then discuss how the shoestring evaluation
approach can be applied at the project, sector and country levels.

30

Table 1: The Strengths and Weaknesses of the Five Most Frequently
Used Quasi-Experimental Designs
Model
1.Pre- and posttest survey of
project andcontrol groups

Works reasonably well toThis is the strongest QED. With awell-selected control group, it
provides good estimates of project
impacts.

Does not work well to

2.No pre-test
control group• Assess if the project model works
and produces the intended outputs.
• Assess similarities and differences
between project and control areas.
• Assess the extent to which the
project could potentially be
replicated• Assess whether observed expost differences between the
project and control groups are
due to the project or to preexisting differences between
the two groups• Control for local history whichmight affect outcomes

3.No control
group• Evaluate projects which have large
impacts or which operate inisolated areas where here is nointerference from other outside
interventions.
• Understand the project
implementation process• Estimate the exact magnitudeof project impacts.
• Control for local history.
• Assess potential for replication
on a larger scale

4.No baselinedata• Obtain an approximate estimate of • Estimate the exact magnitudeof project impacts.
probable project impacts,• Control for local history.
particularly in small or isolated
communities.
• Compare project with othercommunities.
• Control for the effect of intervening
variables through the use ofmultivariate analysis.

5.No control
groups orbaseline data

• Conduct exploratory studies to get • Obtain reasonably precise
estimates of project impact.
a general idea of how well the• Feel confident that the
project model works.
• Obtain a first, approximate estimate observed changes are due tothe project and not to otherof impacts, particularly for small or
factors or interventions.
isolated project.
• Control for external events.
• Obtain comparative data toestimate potential replicability.

Note that the strength of all of these models can be increased by combining them with theimpact evaluation framework and analysis of contextual factors discussed in section 2-2;
and with some of the shoestring evaluation techniques discussed in the following chapters.
For Models 1, 2 and 4, which use control groups, the analysis can be greatly strengthenedby using multiple regression analysis to statistically control for differences in thecharacteristics of the project and control groups.

Project inputs,
materials,
money, staff,
equipment,
consultants,
training,
technical
assistance
Implementation
process• community
involvement in
implementation• involvementof women andothervulnerable
groups• adequacy ofcommunication
Outputs or
products• access toschools, healthservices, etc.
• access to
productive
resources(credit, land,
vehicles, etc.)• access toextension
services andtraining
Outcomes or
short-term
impacts• economic• improved
health,education, etc.
• improved
economic,
environmental
and personal
security• empowerment
and voiceMedium andlong-term
impacts• economic• humanresource
development• security• empowerment
and voice

Institutional and organizational context within
which the project is implemented

Sustainability• sustained
improvementsin economic
and human
resource
conditions,security andempowerment• strengthening
capacity ofcommunities
andgovernmental
agencies tomanageprojects

Social, economic and cultural characteristics of the participating communities which may affect project implementation and sustainability

Project
planning anddesign• participatory
consultations or
centralplanning• information
sources andhow collected• stakeholder
analysis?
• criticalassumptions on
which theproject designis based

Economic and political context within which the
project is implemented

Figure 1: The Project Evaluation Framework

Part I
31

Part IIThe Shoestring Evaluation Approach

Part II

35

Chapter 3
Reducing the Cost and Time Required
to Conduct Impact Evaluations

3-1 Ways to reduce the cost and time required for data collection◆ Simplifying evaluation design

The strongest quasi-experimental design: It is rarely possible indevelopment evaluations to use a true experimental design in which subjects
are randomly assigned to treatment (project) and control groups. A large
number of quasi-experimental designs (QED) have been developed which seek
to approximate as closely as possible true experimental design. Model 1(described in Chapter 2 Section 2-3) describes the QED considered to be thestrongest for most purposes. As indicated in Chapter 2, the analytical power ofthis model can be strengthened if multiple regression analysis is used tostatistically control for differences between the project and control groups. A
number of additional refinements can be included to handle more complex
evaluation designs, including, for example, multiple treatments.
Cheaper and faster, but usually weaker, QEDs: Where cost or time is a major
concern, many evaluations eliminate one or more of the four observations. The
most common options are:
• Model 2 : the elimination of a pre-test control group• Model 3 : the elimination of a control group in both pre and post-test
data collection• Model 4 : the elimination of baseline studies for both project andcontrol groups• Model 5 : the elimination of both a control group and a pre-test36baseline study
Each successive model becomes progressively weaker as it fails to control
for a greater number of the threats to validity. However, many situations exist
in which the use of one of these less robust models is the only available option.
◆ Defining information needs to avoid collecting unnecessary information

The timing, focus and level of detail of an evaluation should be determinedby the information needs of key stakeholders and the types of decisions towhich the evaluation must contribute. Typical questions that decision-makers
must address do not require a high level of statistical precision, as thefollowing:• Is the project achieving its objectives? Which objectives are and are notbeing achieved?
• Are all sectors of the target population benefiting from the project? Are
any groups being excluded or benefiting significantly less?
• Is the project sustainable and are benefits likely to continue?
• What are the contextual factors determining the degree of success or
failure of a program or project?
Though many of the questions may not require a high level of statisticalprecision, some require reliable answers to such questions as the following:• Are there measurable and significant changes in the target population
with respect to the impacts the project is trying to produce?
• Is it reasonable to assume that the changes were due in a significant
measure to the project and not to other unrelated factors?
• Is the project reaching all sectors of the target population, including thepoorest and most vulnerable groups? Are both women and men
benefiting? Are there any ethnic or religious groups who do not benefit?
• Why have these observed changes occurred? Are the conditions whichfacilitated these changes likely to continue and are the impacts likely tobe sustainable?
• Were the target communities or groups reasonably typical of broader
population groups (such as poor farmers or urban slum dwellers) and isPart II37

it likely that the same impacts could be achieved if the project werereplicated on a larger scale?
The key design questions concern such issues as: (a) careful measurementof the key impact indicators; (b) ensuring that reliable information is obtainedon participation and access to benefits by vulnerable groups such as womenand ethnic minorities; (c) understanding the economic, social and politicalcontext within which the project is being implemented; and (d) ensuring thatthe observed changes are due to the project and not to unrelated factors. In
many cases good estimates on most or all of these questions can be obtainedwith relatively simple evaluation designs. For the evaluation of larger andmore complex projects, more rigorous and costly designs may be required.
◆ Reducing sample size and simplifying the sample design

Sample size can often be reduced by accepting a lower level of precision for
the estimates or by reducing the types of disaggregated data analysis; forexample, accepting global estimates of project impacts and not comparing
impacts in different regions. The use of cluster sampling can often
significantly reduce interviewer costs by reducing distances and travel time
between interviews. Of course, it will be necessary to assess the trade-off ineach case between reduced cost and lower precision or less detailed analysis.
◆ Rapid and low cost methods for data collection and analysis

A wide range of rapid and low cost data collection methods are available(Kumar 1993, Valadez and Bamberger 1994 Chapter 7), including: direct
observation, automatic counters, focus groups and community fora, key
informants, survey instruments that respondents can complete by themselves,use of secondary sources rather than interviews, etc. (see Table 2). PRA andrelated participatory methods are a potentially effective way to reduce datacollection cost. Through these, estimates of the degree and direction ofcommunity level impacts can be obtained from a few carefully designed andexecuted group sessions rather than through large numbers of individualinterviews.
◆ Integrating quantitative and qualitative approaches

The integration of quantitative and qualitative data collection and analysis

38

methods is a requirement for any good evaluation design. Integration isparticularly important to cost-effectiveness, as the use of a number of
independent estimators can help validate methods which reduce sample size or
the cost of data collection. This is an important application of the triangulation
principle. Integrated approaches are particularly valuable also forunderstanding the contextual factors discussed above. Bamberger 2000
(Chapter 1) argues that an integrated evaluation approach is more than simply
combining different data collection methods; it affects the way in which
research hypotheses are generated, how the research team is constituted, how
the research budget is allocated, and how time is allocated among different
phases of the research process.
Participatory (largely qualitative) methods also can increase the validity andutility of information. Discussions with intended beneficiaries or groups who
may have been negatively affected by projects can often identify unanticipatedconsequences of projects which may not be captured in surveys. Participatory
methods are very useful for understanding contextual factors which may
influence the level and distribution of project impacts and for assessing
sustainability and replicability (Hentschel 1999).

Part II

39

Table 2: Rapid and Low Cost Methods for Impact Evaluations
Simplify evaluation
designNote: There is an important trade-off between reducing the costs
and time of data collection. on the one hand, and the quality andvalidity of the evaluation findings, on the other. Each successivemodel described below sacrifices methodological rigor and issubject to an increasing range of threats to validity (see Chapter5). It is important to compensate for some of these threats byreconstructing baseline and control group data through the use of
secondary sources and the other measures discussed later in thishandbook. Commonly used approaches for simplifying evaluation
designs include:• eliminate data collection for pre-test control group (Model 2)• eliminate data collections for pre-test and post-rest control group(Model 3)• eliminate pre-test measurement for both project and control
groups (Model 4)• eliminate all baseline measurements and also post-test control
group (model 5)

Reduce sample sizeand data collection
costs• lower the level of required precision• reduce the types of disaggregation required• stratify sample designs• use cluster sampling• use university students, student nurses and community residents
to reduce data collection costs

Reconstruct baseline
data and control
groups• use secondary data
• redesign project records to incorporate impact indicators• use recall• use key informants• use PRA and other participatory methods

Reduce the cost ofquantitative datacollection• use self-administered questionnaires• reduce length and complexity of survey instruments

Qualitative datacollection methods• direct observation• automatic counters and other non-obtrusive methods• focus groups and community fora
• key informants• PRA and other participatory methods

Integrated, multimethod datacollection• use triangulation (multi-method approaches) so that throughindependent estimates of key variables it may be possible toreduce sample size while at the same time increase reliability
and validity.

40

Chapter 4
Reconstructing Baseline Data
and Control Groups

Reliable information on the condition of project participants or control
groups is often not readily available at the time the project is launched (the“baseline’’ period referred to as T1). There are a number of sources which canbe drawn on to reconstruct baseline conditions, including the following:• Secondary data on factors such as morbidity, access to health services,
school attendance, farm prices, and travel time and mode of transport
can often be obtained from surveys conducted by sectoral agencies orfrom household surveys conducted by central statistical agencies. NGOs
also may have conducted studies in some of the project areas. Many ofthese secondary sources are less than ideal for baseline references,lacking the exactly desired coverage, not conducted at the right time or
not including all of the required information. Consequently, while thesesources can provide a useful approximation of baseline conditions, theirstrengths and weaknesses as proxy baselines must always be assessed.
Factors to be assessed include the following: differences in time periods
and their significance (for example did economic conditions change
significantly between the survey date and the project launch);differences in the populations covered (for example did the surveys
include employment in the informal as well as in the formal sector andwere both women and men interviewed); was information collected on
all key project variables and potential impacts.
• Project records from micro-credit agencies, health centers, schools andwater projects often contain information on conditions prevailing before

Part II41

the project began. Again,the reliability of each source must be
assessed10.
• Recall: Asking individuals, key informants or focus groups to recall thesocial and economic conditions at an earlier point in time can besometimes be used to estimate household or community conditions prior
to the launch of the project. While recall is generally not reliable forobtaining precise numerical data on income, numbers of incidences ofdiarrhea or farm prices etc., it may provide useable information on majorchanges in the welfare conditions of the household. For example,
families can usually recall which children traveled outside thecommunity to attend school before the community village school was
opened, how the children traveled to school, and the travel time andcost. Families may also be able to provide reasonably reliable
information on use of health facilities prior to the project, or where theypreviously obtained water, how much they used and how much it cost.
As relatively few studies have been conducted to assess the reliability ofthese kinds of recall estimates in developing countries, it is particularly
important to identify and assess potential sources of bias in theestimates. For example, families might be reluctant to admit that theirchildren had not been attending school, or that they had been using
certain kinds of traditional medicine. They might also wish tounderestimate how much they had spent on water if they are trying toconvince the project that they are too poor to pay the proposed water
charges.
The U.S. research literature has identified two common sources of recall
bias. First, there is substantial evidence that the underestimation of small androutine expenditures increases as the recall period increases. Second, there is atelescoping of recall concerning major expenditures, so that major
expenditures made outside of the recall period will often be reported as having
10

For example, in the Eritrea impact evaluation referred to in Chapter 2, it was found that the project healthcenters kept records only of individual patient visits, it did not have records of the number of different
patients or families visiting the center over a given period of time. These records would have been
extremely time-consuming and expensive to sort by individual patient or to arrange by family. For this reason, it was not possible to use the records to estimate changes in the proportion of the population using
health services.

42

occurred within the period. Most of the systematic research on recall bias has
been carried out by U.S. government survey agencies (such as the U.S
Expenditure Survey) or by universities; but some of their findings are relevant
to developing country research. The LSMS (Living Standards Measurement
Study) program also has conducted some assessments on the use of recall for
estimating consumption in developing countries11.
• Key informants such as community leaders, doctors, teachers, local
government agencies, NGOs and religious organizations may be able toprovide useful reference data on baseline condition . However, many ofthese sources have potential biases (such as health officials or NGOs
wishing to exaggerate health or social problems, or community leaders
downplaying community problems in the past by romanticizing
conditions in the “good old days”).
• Participatory methods such as PRA can be used to help the community
to reconstruct information on past conditions and to identify criticalincidents in the history of the community or region.
◆ Reconstructing control groups

Many of the above methods can also be used to reconstruct control groups.
However, this poses additional difficulties; it becomes necessary to identify
appropriate control areas as well as to assess the conditions in these areas.
With few exceptions project areas are selected purposively (for example totarget the poorest areas or those with the greatest development potential) rather
than randomly12, so it can be a challenge to identify reasonably similar control
areas.
Many ex-post quantitative impact assessments use statistical techniques tocontrol for differences in individual and household characteristics, and hence
LSMS is a World Bank program extending over more than 10 years to develop standard survey instruments and guidelines for the measurement of living standards in developing countries. For a review of theliterature see Angus Deaton and Margaret Grosh “Consumption” Chapter 5 Margaret Grosh and Paul
Glewwe (editors) Designing Household Survey Questionnaires for Developing Countries: Lessons from
15 years of the Living Standards Measurement Study. World Bank 2000.
12
One of the situations in which randomization is used in the selection of project areas occurs when demandsignificantly exceeds supply and some kind of lottery or other random process is used. This sometimes
occurs with social funds (see Baker 2000 for a discussion of the Bolivia Social Fund) or with community
supported schools (see for example Kim, Alderman and Orazem 1999 for a discussion of the Pakistan
Community School project).
11Part II

43

to approximate a control group, by identifying households or individuals who
did not receive particular project services or who received less of theservices13. While this kind of multivariate analysis offers a useful statisticalcontrol for individual characteristics, it cannot usually control for historicalevents or for differences in non-household attributes (such as different
employment opportunities).

13

For example subjects may be categorized according to their distance from a project-constructed road orwater source, by whether any family attended literacy classes, or by the amount of food aid they received.

44

Chapter 5
Identifying and Addressing
Threats to Validity

5-1 Threats to validity
In their efforts to reduce time and costs, evaluators frequently ignore someof the basic principles of evaluation design such as: random sampling,
specification of the evaluation model, instrument development and fulldocumentation of the data collection and analysis process. As a consequence,
many rapid evaluations suffer from serious methodological weaknesses which
threaten the validity or generalizability of their findings.
It is often assumed that evaluators working under budget and time
constraints cannot be held to the same high methodological standards as would
be the case normally when designing an impact evaluation. Two central
premises of the shoestring evaluation approach address this point:• The evaluator must make every effort to achieve the maximum possible
methodological rigor in a given research context.
• The evaluator must explicitly recognize and make every effort tocontrol for methodological weaknesses in the evaluation design.
To guide the researcher in identifying potential weaknesses in theevaluation design, a checklist of “threats to validity in the interpretation ofevaluation findings” has been developed (Annex 1). The checklist is based on
the recently published update and expansion of the seminal work of Cook andCampbell (1979) (Shadish, Cook and Campbell 2002). The table in Annex 1 isbased on the four categories of threats to validity in Shadish, Cook andCampbell (2002). Three of the four categories have been expanded here toPart II

45

include additional factors of particular importance for assessing the validity ofshoestring evaluation designs. The discussion in this chapter also applies tomany other impact evaluation designs which have relaxed or eliminated someof the requirements of a fully randomized experimental design (Valadez andBamberger 1994 pp. 244-45). The four categories of threats to validity are
discussed below.
◆ Threats to statistical conclusion validity (Annex 1 Section 1)

This deals with why the statistical design and analysis may incorrectly
assume that program interventions have contributed to the observed changes(impacts). It also considers situations in which the statistical analysis may haveoverlooked some potential impacts. Of the 10 reasons given in Annex 1
Section 1, some of the most common issues for shoestring evaluations are thefollowing:• [1.1] Low statistical power: If the sample is too small, the conclusion
may be incorrectly drawn that the relationship between treatment(project input) and outcome (project impact) is not significant. For
example, many community water supply and health projects include
reductions in infant mortality as one of the intended impacts. Infantmortality rates tend to change quite slowly because they are influenced
by many factors and because only a small proportion of households havechildren born in a given year; for these reasons, even a successful
initiative will probably produce only a small reduction in the short-run.
It will often be necessary to have a sample size of several thousandhouseholds in order to identify a significant change but many
evaluations of community projects only use samples of several hundredhouseholds. Thus, it will usually be impossible to identify a significant
change, particularly over a 2-3 period year which is the reference period
for many evaluations.
• [1.4] Unreliability of measures: Measurement error weakens therelationship between two variables. This is a common problem inproject evaluations where administrative records on who received which
benefits can be unreliable. For example, local health centers may
provide treatment for common local diseases and infections and they
may also provide advice to mothers on child-care and nutrition. Due toheavy work pressure on health center staff and unreliable supplies of

46

medicines, it frequently happens that one mother receives all required
medicines as well as guidance on child-care and nutrition while anothermother does not receive all of the medicines and other mothers do notreceive guidance. Health center records normally will not record thesedetails so in the analysis it is difficult to determine whether differencesin impacts are due to contextual factors or simply to the fact thatdifferent people received different services.
• [1.5] Restriction of range: When a sample is truncated and covers only
part of the population range, usually the relationship is weakened
between the impact indicators and another variables, such as householdincome or family size. For example, if the sample is targeted to cover
only low-income families, detecting a statistically significant
relationship may be more difficult. This can be an important statisticalissue when assessing the impact of poverty reduction programs whichtarget the poorest segments of the population.
• [1.10] Extrapolation from a truncated or incomplete data base:Statistical analysis is often based on data sets which exclude part of thetarget population. Linear extrapolation or weighting methods used toestimate values for the excluded groups can provide biased population
estimates. For example, enterprise surveys often cover only firms withmore than a certain number of employees (often 10 or 25), excluding thelarge number of small informal enterprises where most of the poor work.
Similarly, censuses and household surveys often miss the very poorest
households, such as pavement dwellers, illegal immigrants or slum
dwellers living outside the money economy.
◆ Threats to internal validity (Annex 1 Section 2)

This deals with reasons why inferences of causality in the observed
relationship between a project intervention and an output/impact may beincorrect. For example, the analysis may find a positive association between
membership in a village bank and women’s income and control over household
resources and decision-making. It might be inferred that membership in thevillage bank caused women’s income to increase and caused their enhanced
control over household income. However, for a number of reasons, the assumed
causal relationship may not be correct. Possibly women’s decision-making
power in the household had increased even before they joined the village bankPart II

47

(perhaps as the result of their participation in a literacy or leadership training
program). Or there may be a selection bias, so that women with higher incomesare more likely to be accepted by the village bank. The following figure
presents the evaluation design hypothesis about the impact of village bankmembership on women’s income and control of household resources:

Women join the village bank

Increases their income

when they receive loans, learn
skills and gain self-confidence

Increases their control

WHICH

over household resources

The figure below presents an alternative explanation of the observedchanges in which the temporal sequence of events is changed.
Women who have taken
literacy training are more
Some women takeliteracy training
which increases
their self-

likely to join the village bank.
Their self-confidence andliteracy makes them more
effective entrepreneurs.

Women’s income andcontrol over household
resources increased as
a combined result ofconfidence andself-confidence,

also work skills.

literacy and loans.

The Annex 1 table lists a range of other factors which could result inincorrect inferences about causal relationships. Of the 12 reasons given in
Section 2 of the table, some of the most common issues for shoestring
evaluations include the following:• [2.1] Ambiguous temporal precedence (confusion as to whether“impacts” occurred before or after the start of the project). Lack ofclarity about precisely when the assumed impacts occurred may yield
confusion about which is cause and which is effect.

48• [2.10] Inappropriate proxy indicators. Proxy indicators are frequently
used when it is not possible to directly measure the desired change or
impact. For example, ownership of consumer durables, quality of
housing construction or ownership of land may each be used as a proxy
to measure changes in household welfare even though it is understoodthat none will fully capture all of the dimensions of welfare. Proxies
frequently rely on project documents or other secondary data collected
for purposes other than the evaluation, thus these indicators may notadequately measure access or impact. A typical example is the use ofincome as a proxy for welfare or economic condition. In many rural
areas the majority of families practice subsistence agriculture and barter;
they use very little money. Clearly in this context monetary income is
not a good proxy for welfare or economic condition.
• [2.11] Reliance on qualitative indicators. The evaluation may rely, forideological14, methodological or logistical reasons, on qualitative indicators
which may not permit generalization nor control for other explanations of
the hypothesized causal relations. For example, participants in acommunity discussion group may be asked to agree on whether the qualityof life of the community has improved or worsened over the past years.
Normally it would be impossible to include this kind of very general
qualitative indicator in a statistical analysis of factors determining changes
in the social and economic conditions of a community.
• [2.12] Unreliable respondent memory or deliberate distortion. Recall,which is frequently used to recreate baseline conditions (see Chapter 4),
is subject to biases due to memory failure and to deliberate distortion.
◆ Threats to external validity (Annex 1 Section 4)

This deals with reasons why inferences about how to generalize study
findings to other settings (regions, social or economic groups, etc.) may not bevalid. For example, the evaluation report on a community-managed village watersupply project may say that the approach used in the projects was successful.
The report may also recommend that the same approach be applied at the14

For example, some NGOs, women’s organizations and academic researchers question the appropriatenessof using quantitative surveys to understand living conditions in a village or local community. They argue
that research must be based on empathy with the community and personal experience with their way oflife.

Part II

49

national level. However, the pilot areas may have been selected because ofcharacteristics likely to make the project a success (for example, traditions ofcommunity management of social infrastructure projects) and thesecharacteristics might not exist in other regions of the country. Consequently thelarge scale replication of this project may be much less successful than expected.
Of the 8 reasons given in this section, some of the most common issues forshoestring evaluations include the following:• [4.6] Policymaker indifference. Policymakers may impede or fail to
implement a program they perceive to be irrelevant or threatening totheir own priorities. Due to this kind of hindrance from local officials, a
potentially effective project model may have disappointing results. Sucha situation may result in under-estimation of the project’s potential
impact, which might have been achieved without the interference.
• [4.7] Pro-active political interference. The opposite may also occur.
Policymakers and politicians may be anxious for the project to succeed
and through their direct or indirect support may change the program
model in ways that managers cannot control. For example, additional
resources (equipment, materials, staff etc.) may be provided withoutcharge, or they may remove normal administrative bottlenecks. This
may result in an over-estimate of how successful the project would be if
replicated in other areas that might lack this special level of support15.
• [4.8] Seasonal cycles. Results may be attributable to irregular seasonalvariations rather than to the program impact. This is particularly criticalin assessing the impact and replicability of agricultural and ruraldevelopment projects.

5-2 Strategies for addressing common threats to validity
Table 3 gives examples of approaches which can be used to address someof the common threats to validity once they have been identified. The fourth
15

For example, internationally funded low-cost housing programs in the 1970s and 1980s sometimes foundthat governments (often in the hope of ensuring continued funding) wanting to demonstrate these
programs’ success would transfer to the project areas schools, health centers and other facilities originallyintended for other areas of the city. As a consequence, the evaluation of the housing projects might findunexpectedly large improvements in health or education indicators which should have been attributed, atleast in part, to these non-project interventions.

50

column indicates the stage of the evaluation at which different correction
measures can be used. Most of the measures should be used during the designof the evaluation, though some can be used while the evaluation is being
implemented (to conduct quality control on the validity of the information
being collected or to consult with key informants and others to understand or
correct some of the data collection issues). Still other measures are used during
the analysis phase (for example to find secondary data sources to compensate
for a lack of control groups). Time and resources should be allocated to allow
for rapid follow-up field visits to check on inconsistencies or questions arisingfrom the analysis or to explore in depth some of the interesting issues arisingfrom the analysis.
The following are examples of the corrective measures which can be taken:
Threat 1.4: Unreliability of measures: This is an issue to be addressed in thedesign phase of the evaluation. Three possible approaches are recommended:• Ensure that sufficient time and resources are allocated for developing
and testing data collection instruments.
• Incorporate multi-method data collection approaches so that at least two
independent measures are used for all key variables.
• Use triangulation to check the reliability of the information.
Threat 2.2: Selection biases: This refers to the possible differentiation ofproject participants from non-participants with respect to unique characteristics
such as age, sex, economic status, ethnicity or motivation. Four possible
approaches are recommended, some of which can be used during theevaluation design, others while the evaluation is being implemented, and stillothers during the analysis phase:• Compare participant characteristics with non-participants, either
informally or, if possible, through the construction of a control group.
Ideally this should be done during evaluation design; but if not done at
this stage, it is important to build some kind of comparison into theanalysis and interpretation of the findings.
• During analysis, statistically control for participant characteristics toassess how they affect impact indicators.

Part II51

• If there is no formal control group, use key informants and otheravailable sources to compare participant characteristics with those ofother people/families in the project and similar areas. This can be done
during evaluation design, implementation, or analysis.
Use direct observation in focus groups and other settings to assess theunique psychological characteristics of participants, such as self-confidence or
motivation, which might help explain project outcomes.

Table 3: Strategies for Addressing Common Threats to Validity
in Shoestring Evaluations
No

Threat

How to address the threat

1. Threats to Statistical Conclusion Validity• Increase sample size
Low statistical• Use stratified sample to increase number of
power
observations in critical cells
1.4 Unreliability of • Spend sufficient time and resources to develop
and test data collection instruments
measures• Multi-method approaches to provide at least two
independent measures of key variables• Use triangulation to check on reliability ofmeasures• Broaden sample to cover wider segment of the
1.5 Restriction of
population
range• Use secondary data for comparative purposes
1.1• Conduct small, rapid sample of excluded
population• Same as for 1.5

1.10 Extrapolationfrom truncatedor incomplete
data base2. Threats to Internal Validity• Conduct rapid studies to clarify temporal
2.1 Ambiguous
sequence
temporal
precedence• Compare participant characteristics (age, sex,
2.2 Selection Biasit ) itht lApplicableevaluation
stage• Design• Design• Design• Design• Design• Design• Design andanalysis• Design andanalysis• Design andanalysis• Design• Design and
l i

52

No

Threat

How to address the threat

Applicableevaluation
stage• Design and• Compare participant characteristics (age, sex,
analysisincome, etc.) with control group• Statistically control for particular characteristics • Analysis
to assess how this affects impact indicators• Design• If no control group, use key informants toimplemencompare participant characteristics with those oftation andthe total project area
analysis• Use direct observation in focus groups and other • Design andimplemensettings to assess psychological characteristics
tationof participants such as self-confidence andmotivation• Design and
2.3 Local or national • Collect secondary data on market prices,
analysis
educational enrolment, morbidity patterns, wage
history
rates, etc.
• Design and• Consult key informants and experts on national
analysistrends.
• Design• Same as for 1.4.
2.10 Inappropriate
indicators• Design and• Find and use independent sources (key2.12 Unreliable
implemeninformants, secondary sources, etc.) to check
respondent
tationreliability for information.
memory or• Design and• Combine individual recall questions with groupdeliberate
implemeninterviews to rest for consistency of information
distortion
tation3. Threats to Construct Validity• Design• Meet with key stakeholders to understand more
3.1 Inadequate
fully the implicit project model.
explanation of• Design• Develop a project theory model and meet
constructs
individually or in groups with stakeholders tocheck and elaborate the model.
• Design
3.8 Reactivity to the • Use exploratory studies, observation, etc., tounderstand respondent expectations and to
experimental
identify potential response bias.
situation
4. Threats to External Validity• Design• If the project is implemented in different
4.6 Policymakerlocations, identify differences in the attitudes ofindifference
policy makers in each location (throughinterviews, secondary sources or key
informants) and assess how these differences
appear to affect the project.

2.2

Selection Bias

Part II

No

Threat4.7

Pro-active
political
interference
Seasonal cycles

4.8

How to address the threat

53

Applicableevaluation
stage• Same as for 4.6• Design andanalysis• Try to construct a control group.
• Consult key informants• Design• Design andimplementation• Design andimplementation• Use secondary data to identify seasonal patterns.

Part III
Using the Shoestring Evaluation
Approach at the Project, Sector,
and National/Policy Levels

Part II

57

Chapter 6
Shoestring Evaluation
at the Project Level

6-1 Applications
There is a growing demand for rapid and low cost impact evaluations. In allareas evaluators face time and budget constraints as well as limited access todata; in this regard, shoestring approaches can make a major contribution. The
following are some of its applications at the project level:• During project preparation: rapid assessment of the outputs and impactsof earlier projects• During the mid-term review: rapid assessment of actual and potential
project impacts and of factors potentially limiting impacts• Project completion report: better assessment of actual and potential
impacts and identification of factors affecting future impacts andsustainability• Ex-post evaluations: conducted by internal evaluation and audit units or
by external consultants• PRSP: periodic evaluations of the outputs and impacts of PRSP
components.
• Gender mainstreaming strategy: evaluating the impacts of integrating
gender into projects, programs and policies• Pilot projects: rapid assessment of the actual and potential impacts ofpilot projects

58

6-2 Challenges
Evaluators face all of the methodological and organizational challenges
discussed in Chapter 1.

6-3 Useful approaches◆ Simplifying the evaluation design

There are normally two questions to be addressed with respect to the types
of QED discussed in Chapter 2:• Which design to use?
• What are the implications for the purposes of a given study of using oneof the simpler but less robust evaluation designs?
The range of options is greatest when the evaluator is called in at the start ofthe project. Ideally the evaluation would use Model 1 with pre- and post-test
measurement of project and control groups and with documentation on the
project implementation process. The designs can be strengthened if they are
combined with the development of a project theory model (see Chapter 2) sothat the process of project implementation can be studied and so that the
influence of contextual factors can be taken into account. However if time orbudgets are constrained or if finding a control group will be difficult, theevaluator can consult with project management to clarify the precise purposeof the impact study and on this basis decide how important it is to have acontrol group:• If the project is a relatively small pilot project whose purpose is to assess
whether a particular new service or delivery system works, then it may
be less important to have a control group.
• If, on the other hand, the purpose of the evaluation is to assess the costeffectiveness of the project compared to alternative approaches, and torecommend whether the project could be replicated on a larger scale;then a control group may be more important.

Part III

59

When the evaluator is not called in until the project is completed or
implementation is well advanced, the options for the evaluation design are
more 1imited. However, it is still important to consult with the key
stakeholders to answer the following questions:• How will the evaluation findings be used?
• What are the key questions to be answered?
• What level of detail and precision is required?
The evaluator should then use the evaluation worksheet (see following
section) to assess how well the current evaluation design can address thepriority information needs of the key stakeholders. If the evaluation design
does not provide the required analysis and information, the evaluator should
consider using one or more of the following approaches:• Reconstruct project baseline data using the methods described in
Chapter 4• Define a control group and conduct a rapid ex-post survey• Reconstruct baseline data for the control group using the methods
described in Chapter 4• Conduct rapid assessment studies or focus groups to provide more
information on contextual factors and how they affect project
performance• Complement ex-post survey information with other independent datacollection methods to test the reliability and validity of the information
being collected◆ Check sample design and statistical power (threats to statisticalconclusion validity
If the sample size is too small to for statistically significant associations
between project inputs and intended outputs/impacts, there are various possible
options:• Assuming the interviews have not yet been conducted, increase thesample size or improve the efficiency of the sample throughstratification

60• Reduce the planned level of disaggregation (for example by presenting
estimates for the whole program only, and not for each project or region)or reduce the required level of precision• Complement the sample survey with qualitative group interviews or
consultations to obtain independent estimates of key variables or
impacts◆ Threats to internal validity

Check the potential causes of threats to internal validity in Annex 1 Section
2. Depending on the stage at which the evaluation is conducted, the analysis
can either be used to improve project performance or to better understand thestrengths and weaknesses of the program model. For example:• [2.1] Conduct rapid assessment studies to clarify temporal precedence of
project inputs and changes in dependent variables• [2.2] Remove biases from participant selection or recruitment
procedures• [2.6] Try to reduce attrition or note the characteristics of those who dropout so as to take account of this in the analysis• [2.10] Review carefully the key indicators and assess theirappropriateness for defining and measuring the key constructs◆ Threats to construct validity

A key recommendation is to work closely with all key stakeholders todevelop a detailed and logical program theory model that specifies a set of
logically consistent hypotheses about the processes through which the projectis intended to produce its impacts. The model will also permit clear
specification of the indicators required to measure all of the constructs. Oncethe model has been developed and agreement reached with stakeholders on the
project design model and the key hypotheses, the evaluation design can bechecked to assess which threats to construct validity exist (see Annex 1
Section 3) and how important they are for the objectives of the evaluation.
◆ Threats to external validity

The evaluator should consult with stakeholders to define what kinds ofgeneralizations and projections are to be made on the basis of the evaluation

Part III61

findings. For example, will the evaluation results be used for the following
purposes:• To recommend whether the project could be replicated on a larger scale?
• To define for which population groups or in which regions the project
could be replicated?
• To define the contextual factors which positively and negatively affectthe success of the project and which should be taken into consideration
when recommending how and where it could be replicated?
The threats to external validity checklist (see Annex 1 Section 4) should be
used to identify any factors affecting the reliability of these projections.
Depending on the stage of the project, appropriate measures should then be
taken to correct factors affecting the reliability of projections. For example, if
the attitude (positive or negative) of local politicians and policy makers
appears to be a key determinant of project success, an analysis should be madeof the implementation experience and outcomes in areas with different types ofpolitical participation. This will provide a firmer basis for anticipating theextent to which project outcomes are likely to be affected by the policy andpolitical environment.

6-4 Using the Shoestring Project Evaluation Worksheet
Annex 2 presents a Shoestring Project Evaluation Worksheet which can beused as a checklist to assess the strengths and weaknesses of evaluation
designs (particularly rapid and low cost designs) to identify methodologicalweaknesses which might affect the validity of findings and reduce theirpractical utility. Annex 5 gives an example of how to apply the worksheet tothe evaluation of the impact of low-cost housing on income and employment
discussed in Chapter 2 (as an example of QED Model 1). The worksheet
contains the following sections:◆ Coversheet
1. Stage of the evaluation at which the worksheet is being used. This isimportant because the types of changes and interventions which can beeffected will vary depending on how advanced the evaluation is.

62

2. Evaluation design. The design should be identified according to theclassification system presented in Chapter 2. Potential problems should be
identified concerning the following:• baseline data
• control group• data collection methods and quality of data
• analysis of contextual factors• other potential problems
The problems can be listed on the front page of the worksheet or on a
separate page of notes are to be added.
3. Objectives of the evaluation. It is essential for the reviewer to fully
understand the objectives of the evaluation. If the objectives are notexplained in the evaluation design, the evaluator should meet with projectstakeholders to discuss their expectations and to understand the decisions or
actions to which the evaluation findings will contribute. It is particularly
important to understand whether the evaluation is intended to be any of thefollowing:• An initial exploratory study• An evaluation of a small pilot project• A rigorous multivariate statistical analysis of a large and complex
project
Determining which of these options best describes the purpose of theevaluation is critical to understanding the information needs and the required
level of methodological rigor.
4. Time and resource constraints. The reviewer must understand the primaryconcern of project management:• To comply with a tight time and budget deadline• To comply with a tight deadline, but where additional resources might
be available• To produce a high quality product which can withstand scrutiny from

Part III63

evaluation professionals and the project’s critics
The recommendations which can be made on possible revisions to thedesign will be different under each of these scenarios.
◆ Analysis of individual threats to validity

A separate page should be completed for each potentially important threat
to validity. Importance is a judgment call which will be based on thereviewer’s understanding of the objectives of the evaluation. For example, amore rigorous assessment of the sample design and the standardization of datacollection procedures will be required for a large-scale statistical evaluation.
The points to be covered for each threat to validity include the following:
A. How is the problem manifested in the evaluation design? For example:• Very few women attended the focus group session or community
meeting in which feedback on the project was obtained.
• The village banks were launched in communities with previous
entrepreneurial experience so that the project is expected to be more
successful than in typical communities.
B. What are the potential effects on the study findings and generalizations?
For example:• The finding that the community was generally satisfied with the choiceof projects may not reflect the views of most women.
• The findings may over-estimate the potential for replication of the pilot
project.
C. How big a problem is this for the evaluation? Reasons should be given forthe judgment. For each threat the reviewer must decide whether it is critical,
important, of minor importance or unimportant for the evaluation.
D. What are the proposed actions? The proposed actions will vary according
to the seriousness of the problem and the resource constraints. In somecases the reviewer may be in a position to initiate the recommended actions,in other cases recommendations are made to the client concerning actions
which he could take. Examples of possible actions include the following:

64• Prepare a report for the client indicating areas in which the results andconclusions of the evaluation might be misleading or where they should
be interpreted with reservations.
• Secondary data could be analyzed to assess whether the observedchanges in the project population are similar to or different from thoseoccurring in the wider population.
• Rapid assessment studies could be conducted to validate findings or toprovide a stronger framework for the interpretation of the findings.
E. How adequate are the proposed actions? A note should indicate whether
the actions are likely to correct, reduce, or have little impact on the
problem. The note should also indicate why the problem cannot be resolved
at this stage of the analysis. Is it a question of time and resource constraints,
is it due to resistance or lack of interest from the client, or are there external
political factors involved?

Part III

65

Chapter 7
Shoestring Evaluation at the
Sector and Program Level

7-1 Applications
There is also a growing demand for rapid and low cost impact evaluations
at the program and sector level. The following are some of the present andpotential applications:• Assessments by development agencies of their sector work (forexample, social funds, irrigation, transport)• Assessments of the agencys’ social policies such as gendermainstreaming, indigenous peoples, etc., at the country, regional orinternational levels• Assessment of the impacts of rural development programs on the poor
• Evaluating the impacts of the initiatives to combat HIV/AIDS

7-2 Challenges
Evaluators face all of the methodological and organizational challenges
discussed in Chapter 6, with the following additional complications:• As sector programs usually comprise many different interventions, all of
the methodological questions discussed in the previous chapters(sampling, constructing baseline and control data, etc.) become evenmore complicated.
• The issue of identifying a control group is particularly difficult as
different projects and programs in the sector are organized differently

66

and address different groups.
• Many different agencies are involved, in most sectors – each withdifferent objectives and ways of organizing projects.
• Several additional organizational complications must be addressed.
First, different agencies organize projects and programs differently.
Second, each funding agency has a slightly different set of questions ofconcern to its national clients (Parliament, Ministry of Finance, NGOs,etc.) or international stakeholders. Consequently, developing countries
are often required to divide their 1imited within-country evaluation
capacity resources among many different projects. It has proved
extremely difficult to achieve the cooperation required for any kind of
coordinated evaluation effort.

7-3 Useful approaches◆ Simplifying the evaluation design

Sector and program evaluations frequently combine global information on
the sector with data compiled from a number of different project levelevaluations. The guidelines for conducting and reviewing individual projectlevel evaluations will be similar to those discussed in Chapter 6. However,there are added complications when the findings of different project
evaluations are combined.
New challenges must be faced when compiling sector-wide information.
The analysis is easiest when the findings of similar projects are combined (forexample when different donors fund similar education or water projects indifferent regions). The analysis becomes much more difficult whensynthesizing projects with different objectives, scope and implementation
methods. Rapid assessment methods offer a useful way to obtain a quick
overview of project level experiences. Some of the possible evaluation
approaches include:• Focus groups and workshops with representatives from different
projects and agencies• Using recall at the project (rather than household) level to obtain overall
assessments of implementation experience and impact• Using project records and secondary data to reconstruct baseline

Part III67

conditions• Control group information (for example, on education, health, watersupply, etc.) is sometimes easier to obtain at the sector level than at the
project level◆ Assess sample design and statistical power (threats to statisticalconclusion validity
In many cases the challenge is to compare data from different projects todetermine whether observed differences in outputs and impacts are attributableto project design or to other factors. This often is very difficult because thenumber of observations is quite small. Due to the small number ofobservations, frequently an interpretation of differences must rely more on
qualitative data than on rigorous statistical analysis.
◆ Threats to internal validity

The issues and approaches are similar to Chapter 4 with the addedcomplication that the assessment often must be made on a number of different
projects. The following are some of the special issues which may have to beaddressed:• [2.1] Temporal Precedence: The same set of checks on temporal
precedence (see Chapter 5) often must be applied to different projects.
There is the additional complication that the implementation schedule ofdifferent projects may interfere with each other. For example, if one
agency is organizing women’s leadership training programs in areas
where another is starting a micro-credit program; it becomes difficult toassess whether women’s enhanced empowerment is due to the impactsof the second project, or whether the women participating in the second
project had already been empowered by the leadership training of thefirst project.
• [2.2] Selection bias: Selection procedures and biases are likely to bedifferent for different projects. Biases must be identified and theimplications of different selection procedures compared.
• [2.6] Attrition: Again experiences with different projects must becompared.
• [2.10] Inappropriate indicators: Frequently different projects use

68slightly different indicators, something must be controlled for. While itis desirable to use standard indicators, care must be taken not to obscuredifferences in approach between different projects16.
◆ Threats to construct validity

Where projects using different models are being compared, the importanceof well defined theory models becomes even more critical. These should
define the program theory models used by each project, but also must identify
critical differences between models.
◆ Threats to external validity

In addition to the project level analysis discussed in Chapter 6, market or
demand saturation is a factor which must be addressed at the sector level.
When projects are analyzed separately each one may appear to have a capacity
for replication, but when an aggregate analysis is conducted, the issue ofsaturation, or competition between projects, must also be assessed.

7-4 Using the Shoestring Program Evaluation Worksheet
Annex 3 presents a Shoestring Program Evaluation Worksheet. As with the
project worksheet in Annex 2, this can be used either to plan an evaluation or
to review and modify an ongoing evaluation. The program worksheet issimilar to the project worksheet except for the following:◆ Coversheet

Type of evaluation study: Clarify whether the study evaluates a single
program implemented in a standard way in many sites, or whether there are anumber of different projects each using a different design.
◆ Analysis of individual threats to validity

Level of generality: Does the discussion refer to a threat to one or moreindividual projects or interventions, or to the overall program?

16

For example, one micro credit program may provide only training on financial management, while anothermight include also entrepreneurial development or empowerment training. It is obviously important tomake sure that while indicators are comparable, they also capture these differences.

Part III

69

Chapter 8
Using Shoestring Evaluation at theCountry and Policy Level

8-1 Applications
There is also a growing demand by development agencies for rapid and low
cost evaluations of the effectiveness of their country programs and policy
work. Although there is less experience in the use of shoestring evaluation at
the national level, the shoestring approach has potential to contribute to theunique methodological challenges facing effectiveness analysis at this level.
Some potential applications include the following:• Evaluating the effectiveness of country assistance strategies.
• Contributing to periodic reviews of agency performance; for example,
the World Bank Annual Review of Development Effectivenessprepared by the Bank’s Independent Evaluation Group (IEG)• Evaluating the effectiveness of special financing mechanisms, such as
targeted development grants, and cooperative financing mechanisms;
for example, the World Bank recently evaluated the special PHRD
project preparation facility financed by the Government of Japan• Evaluating the effectiveness of the contributions of an agency to theinternational HIV/AIDS initiative

8-2 Challenges
Country and policy evaluation poses an additional set of challenges on topof the methodological and organizational questions discussed in earlier

70

chapters. These include the following:• Finding the right counterfactual• How to make before-and-after comparisons for complex programs
which include many components and operate on many different levels.
Frequently this involves aggregating the impacts of different projects,
often using different methodologies, which form part of a national
program.
• How to distinguish the impact of development agency interventions
from:- Changes attributable to country performance- Changes due to other development partners- Changes due to exogenous factors• Identifying and measuring often subtle changes in policies

8-3 Useful approaches◆ Distinguishing levels of impact

It is important to define which of the following levels of effectiveness are to
be examined:• Outcomes and impacts within a development agency (corporate
responsiveness). For example, the World Bank evaluation of theimplementation of its gender policies includes, among other things, theincreasing use of gender analysis by the Bank and the allocation offunds for Bank gender initiatives.
• Mainstreaming policies and approaches within client county agencies• Evaluating impacts of policies on target populations within client
countries◆ Finding the right counterfactual

A counterfactual is an estimation of what would have been the situation if aparticular policy or country program had not been implemented. There are twomain approaches:• Comparing before-and-after conditions

Part III71

• Analysis of client country performance relative to:- Similar countries- Development indicators defined in country assistance strategies,
United Nations Human Development Indicators, country gender
reviews, PRSPs, country sector work and policy documents, etc.
- Endogenous factors (economic conditions, demographic changes,
terms of trade, etc.)◆ Approach 1: Before-and-after studies

Country program and policy evaluations can be assessed in either or both of
the following ways:• Aggregating the outcomes and impacts of a number of different projectand sectoral interventions, all of which form part of a national program;for example, programs which: provide grants to communities and local
level organizations in different regions to construct or upgrade
community infrastructure, or which providing loans and tax breaks tosmall entrepreneurs to offer transport services to rural communities. In
both cases the assessment of the effectiveness of the country program(or policy) requires evaluating and aggregating the impacts of large
numbers of different projects.
• Directly assessing a program or policy which is implemented in a
uniform manner throughout the country
There are three main types of evaluation studies which can be used toevaluate either of these kinds of national programs/policies:• Exploratory evaluations• Evaluations of pilot interventions, at either the project or program levels• Large-scale quantitative analysis of (large-scale) program/policy
interventions
Each of these evaluation types must address the following threemethodological challenges:• Defining indicators for each dimension

72• Selecting cost-effective and reliable data collection methods• Assessing the extent to which observed changes are due to the program
strategy◆ Exploratory studies

Exploratory studies provide initial feedback on the ability of individual
projects or national programs/policies to achieve their objectives. Thesestudies normally use rapid participatory assessment methods with relatively
small samples and are completed at a modest cost in a relatively short periodof time. The sample size and cost will normally increase for larger and more
complex programs. Rapid assessment methods typically include the following:• Focus groups and other community consultations• Participant observation• Unstructured interviews with individuals, households and community
groups• Key informant interviews• Stakeholder analysis• Rapid surveys to obtain basic quantitative data
◆ Evaluations of pilot interventions at the project or program levels

The assessment of (usually small) pilot interventions is intended to assess
the potential of the model for achieving its objectives and impacts. The
assessments are also intended to help understand factors contributing to thesuccess, accessibility, and potential replicability of the project model. Again,
rapid assessment studies are used, but usually in combination with rapid
sample surveys to provide a stronger base for generalization.
◆ Large-scale quantitative analysis of (large-scale) interventions

Large-scale quantitative evaluations are normally justified only for large
projects or programs which have been operating for several years and which
are already producing clearly defined, uniform, and quantifiable results. They
are also particularly useful where comparative analysis is required for projects
or programs operating in several different contexts. These expensive studies
are justified only if the findings can contribute to decisions on future
investments.

Part III73

Large-scale evaluations will normally have the following stages:
•
•
•
•
•
•
•

Exploratory studies to identify key variables and issues
Analysis of secondary data
Definition of evaluation design model
Definition of control group
Selection of a representative sample of project beneficiaries
Application of the data instruments
Use of a multimethod approach that combines surveys with qualitative
methods

Shoestring methods can be introduced at most stages to reduce cost or time,
for purposes of triangulation, or to help fill gaps in the available data (forexample, reconstructing baseline studies or creating control groups).
◆ Approach 2: Comparing client performance to benchmarks

In order to assess the potential impacts of ODA interventions, client
performance in countries where the agency is active can be assessed relative tothe following:• Similar countries• Development indicators included in country assistance strategies,
country gender reviews, United Nations Human Development
Indicators, etc.
• Client performance relative to endogenous factors such as national andinternational economic environment, demographic changes, etc.
◆ Strengthening the logical robustness of attributions

Several approaches can be used to strengthen the logical robustness of
attributions:• Self-assessment by agencies• Independent assessment by other agencies• Independent assessment by internal evaluation and audit departments
and other development partners• Comparison of impacts in different policy and program areas

74• Rapid assessment studies to obtain feedback from different stakeholders(intended beneficiaries, implementing agencies, civil society, etc.)
The assessments can include short-, medium-, and long-term indicators. It is
important to define clearly the time-horizon over which different types ofimpacts are expected to occur.
Example of short-term indicators (within the first 12 months):• Are required inputs being mobilized and used effectively?
- Were funds approved and released?
- Were staff and consultants hired?
- Were training/capacity building activities launched?
Examples of medium-term indicators for projects or programs (within 2-3 years)• Were intended products created (schools built, health workers trained,
community environmental action plans prepared, etc.)?
• Do all sectors of the target group have access to services and benefits?
Examples of long-term indicators (beyond 2-3 years)• Welfare indicators for the target populations• Indicators of sustainability:- There is accessibility and use of services.
- There is regular maintenance by community and responsible
agencies.
- The community makes the required financial contribution.
- Finance and other resources are contributed by responsible agencies.
- There is capacity building at the community and government levels.
◆ Evaluating the effectiveness of programs in achieving five broad develop-

ment criteria
The evaluation should also assess program performance in terms of broaddevelopment criteria. The following five criteria are used by the World Bank
Independent Evaluation Group in all of its program evaluations, and similar
criteria are used by many other development agencies:• Efficiency

Part III

•
•
•
•

75

Efficacy
Development Impact
Relevance
Sustainability◆ Threats to validity

The issues to be addressed here are similar to those for program and sector
evaluations. Some points to be kept in mind are the following:Threats to statistical conclusion validity:
The need to combine estimates from a number of different studies requires
caution in the interpretation of statistical conclusions about outcomes andimpacts, and particularly their attribution to projects.
Threats to internal validity:
Issues of internal validity are particularly difficult to assess because theadministration of programs and policies is often not done in a sufficiently
systematic and controlled way. This may be due to the often weakadministrative structures and the need to compromise with different groups
who oppose the programs.
Threats to construct validity:
It is particularly important to have a clearly defined program model with
explicit definition of the intended links between inputs, processes, outputs andexpected impacts. This will help define key indicators and assess theirappropriateness for measuring the underlying constructs.
Threats to external validity:
As most policies are intended for application at the national level, it isusually not possible to generalize them to a broader population group in thesame way that projects and sector programs are generalized. However, some of
the generalizability issues which must be addressed include the following:• Policies may be implemented first in certain sectors or regions with theintention of subsequent replication in other sectors or regions. Because
the characteristics of different sectors often vary in important ways, it

76

can be difficult to assess how easily this replication can be achieved. For
example, a policy to reduce subsidies and increase user charges for theprovision of public services is likely to have a very different
implementation experience in primary health services, education, andpublic transport. If this is the case, experience in the health sector may
not provide good guidelines for assessing what will happen in education.
• Are the policies sustainable?

8-3 Using the Shoestring Policy and Country Evaluation
Worksheet (Annex 4)
The policy and country program worksheet is very similar to the sector
worksheet (Annex 3) except for the following:◆ Covershee

1. Type of evaluation: Is this an evaluation of a policy or of a country
program?
2. Evaluation design: Does the evaluation use a before-and-after design, or
does it assess effectiveness by comparing client performance with similarcountries, development indicators or exogenous factors?
◆ Analysis of individual threats to validity

Level of generality: Does the discussion refer to the overall policy or tocertain components of it?

Part III

77

Chapter 9
Building Evaluation into Project
and Program Design

9-1 Building evaluation elements into project and program
designMany of the problems facing evaluators who are called in late in the project
cycle occur because no information was collected on the characteristics of the
future project beneficiaries and non-beneficiaries before the project began.
Consequently, one of the main challenges facing the evaluator is to reconstruct
information on pre-project conditions to form the basis for comparison with
post-project data on beneficiaries and (hopefully) a control group. If a way
could be found to convince policy makers and project managers to collectbaseline data, the task of the future evaluator would be much easier and thequality of the evaluation analysis would be much better. Unfortunately, at thetime when new projects and programs are being planned and launched,
concerns about future impact evaluations are very low on the project
manager’s list of priorities. When a project is launched there is no guarantee itwill survive or that there will be any results to measure. In most cases theevaluation staff are not even recruited and trained until the project has been
underway for some time.
Much of the basic baseline data could be collected with almost no
additional effort or cost. The question addressed in this chapter is thefollowing: Given the real-life conditions under which projects are launched –keeping in mind that the evaluation staff have probably not yet been recruitedand the manager has no time to worry about future evaluations –are there are
any simple measures that could be taken to ensure that the data is generated?

78

9-1 Strengthening the treatment of evaluation at four phases ofthe project cycle
Focus in this regard is on four critical moments in the project cycle
described below. All of the following recommendations are considered to beminimal guidelines, to be followed by project management regardless ofwhether or not there is a plan to conduct an impact evaluation and whether or
not an evaluation unit has been established. In other words, the proposed
measures provide useful information for strengthening program management.
If management does choose to plan for a future evaluation, then theseguidelines can be used to prepare it.
◆ During project planning

Despite the fact that extensive data collection and analysis is frequently
undertaken during project appraisal and planning, very little attention isnormally given to how the information could be used for future evaluations.
The concern is to produce the information required for the project approval
and launch. Even though many development agencies require that projectplanners produce a LogFrame and a set of performance indicators, little serious
thought is normally given to laying the basis for a future evaluation. The
following are some simple measures which could be taken during projectplanning:• Require that the project planning process include a discussion on how tocollect a minimum set of evaluation indicators. The plan should include:- Definition of a preliminary theory model- A simple evaluation design- A discussion of sampling (how to select a representative sample offuture project beneficiaries and a control group of similar families
who will not benefit from the project)- A preliminary set of measurable performance indicators- Definition of organizational responsibilities for both monitoring andevaluation• Ensure that all surveys and other forms of data collection and analysis
conducted during appraisal are archived and documented so that they

Part III79

will be accessible to and usable by future evaluators.
• Require that the project planning documents include at least a brief
discussion of the characteristics of the target population, and theirsimilarities and differences from the rest of the low-income population.
Ideally there should be some minimal collection of information on thecontrol population and a discussion of how a control group could becreated.
• Recommendations on how routinely collected administrative
information on the project could be organized to make it more useable
for a possible future evaluation. For example:- Records of local health facilities could be organized so that there is afile for households and not just for individuals. In this way it would
be possible to quickly estimate what proportion of households use thehealth facilities.
- Consideration should also be given to collecting brief information on
the basic living conditions of patients using the health facilities(quality of housing, type of water supply and sanitation, and possibly
a simple indicator of economic status), although it will often not be
feasible to collect all of the information.
- Schools might be asked to collect some basic social and economic
information on students (similar to the above).
- Village banks and credit programs, many of which already collect a
lot of information, might be asked to prepare a brief socio-economic
profile of borrowers◆ During project implementation• Ensure that the proposed revisions to the administrative records(discussed above) are put into place.
◆ In conjunction with the mid-term review (MTR)

In many cases rapid assessment studies are conducted in preparation for the
MTR in order to provide an initial assessment of potential impacts and to
identify potential constraints to the achievement of project objectives. A
number of simple measures could be taken to ensure the information cancontribute to a possible future impact evaluation:

80• When rapid assessment studies are commissioned, attention should be
paid to the sample procedures (however simple they may be) to ensure
the representativeness of the findings. These studies can often provide
an approximation of a baseline study.
• Data should be archived and documented so that it can be easily locatedand used for future evaluations.
• The MTR should produce estimates on the status of all of the critical
performance indicators identified during project planning.
• The Logic Model should be reviewed and the validity of its key
assumptions assessed and, if necessary, updated.
◆ In conjunction with the project Completion Report

Similar guidelines should be followed as for the MTR.

Part III

81

References
Alderman, Harold, Jooseop Kim and Peter Orozem. 2000. Design, Evaluationand Sustainability of Private School for the Poor: The Pakistan Urban
and Rural Fellowship School Experiments. HDNED.
Baker, Judy. 2000. Evaluating the impact of Development Projects on Poverty:
a Handbook for Practitioners. Directions in Development. WorldBank.
Bamberger, Michael. 2002. Longitudinal Impact Evaluation of the Community
Development Fund Feeder Road Component. Unpublished.
Bamberger, Michael. 2000. Integrating Quantitative and Qualitative Researchin Development Projects. Directions in Development. World Bank.
Chambers, Robert. 1995. Poverty and Livelihoods: Whose Reality Counts?
Institute for Development Studies. Discussion Paper No. 347.
University of Sussex.
Deaton, Angus and Margaret Grosh. 2000. “Consumption” Chapter 5 Margeret
Grosh and Paul Glewwe (editors) Designing Household Survey
Questionnaires for Developing Countries: Lessons from Fifteen Yearsof the Living Standards Measurement Study. Oxford University Press.
FASID 2000. Project Cycle ManagementÅDMonitoring and Evaluation basedon the PCM Method.
Grosh, Margaret and Paul Glewwe. 2000. Designing Household Survey
Questionnaires for Developing Countries: Lessons from Fifteen Yearsof Living Standard Measurement Study. Oxford University Press.
Khandker,Shahidur 1998 Fighting Poverty with Microcredit: Experience in
Bangladesh. New York. Oxford University Press for the World Bank.
Kumar, Krishna (editor). 1993. Rapid Appraisal Methods. World Bank.
Reitbergen-McCracken and Deepa Narayan. 1997. Participatory Tools andTechniques: A Resource Kit for Participation and Social Assessment.
Environment Department. The World Bank. Washington D.C.
Miles, Matthew and Michael Huberman. 1994. Qualitative Data Analysis.
Sage Publications.
Narayan, Deepa and Patti Petesch. 2002. Voices of the Poor: From Many
Lands. World Bank. Washington D.C.
Newman, Constance. 2001. Gender Time-Use, and Change: Impacts ofAgricultural Export Employment in Ecuador. Policy Research Report

82

on Gender and Development. Working Paper Series No. 18. URL:
www. worldbank. org/gender/prr
Nizam, Ahmed. 2000. Study on Gender Dimensions of the Second Bangladesh
Rural Roads and Markets Improvement and Maintenance Project.
URL: www. worldbank. org/gender/transport/grantsandpilots
Patton, Michael. 2002. Qualitative Research and Evaluation Methods (3rdedition). Sage Publications.
Picciotto, Robert. 2002. Development Cooperation and Performance
Evaluation: the Monterrey Challenge. OED, World Bank.
Pitt, Mark; Shahidur Khandker, Signe-Mary McKernan, M. Abdul Latif. 1999.
“Credit Programs for the Poor and Reproductive Behaviour in LowIncome Countries: Are the Reported Causal Relationships the Result ofHeterogeneity Bias?” Demography 36 (1) 1-2 1
Pitt, Mark and Shahidur Khandker. 1998. “The Impact of Group-based Credit
Programs on Poor Households in Bangladesh: Does the Gender ofParticipants Matter?” Journal of Political Economy. 106: 958-96.
Shadish, William, Thomas Cook and Donald Campbell. 2002. Experimental
and Quasi-experimental Designs for Generalized Causal Inference.
Houghton Mifflin. New York.
Valadez, Joseph and Michael Bamberger. 1994. Monitoring and Evaluating
Social Programs in Developing Countries. World Bank.
World Bank. 2000. Attacking Poverty: World Development Report 2000/2001.
Oxford University Press.
World Bank. 2001. Engendering Development through Gender Equality in
Rights, Resources and Voice. pp. 160-162. A World Bank Policy
Research Report.
World Bank. 2002. Poverty Reduction Strategies Sourcebook. World Bank.
Washington D.C.

Annexes

Annex85

Annex 1Threats to Validity in the Interpretationof Evaluation Findings

Threats to Validity in the Interpretation of Evaluation Findings
1. Threats To Statistical Conclusion Validity17: Reasons why inferences
about covariation between two variables may be incorrect
1.1 Low Statistical Power
1.2 Violated Assumptions of Statistical Tests
1.3 Fishing and the Error-Rate Problem
1.4 Unreliability of Measures
1.5 Restriction of Range
1.6 Unreliability of Treatment Implementation
1.7 Extraneous Variance in the Experimental Setting
1.8 Heterogeneity of Units
1.9 Inaccurate Effect Size Estimation
1.10 Extrapolation from a Truncated or Incomplete Data Base
2. Threats to Internal Validity18 : Reasons why inferences that therelationships between two variables is causal may be incorrect
2.1 Ambiguous Temporal Precedence
Source: Adapted from Shadish, Cook and Campbell 2002. Table 2.2 page 45. The threats in italics,
representing additional factors of particular importance for Shoestring Evaluations, have been added by thepresent authors.
18
Source: Adapted from Shadish, Cook and Campbell 2002. Table 2.4 page 55. The final 3 threats (in italics)
have been added by the present authors.
17

86

2.2 Selection
2.3 History
2.4 Maturation
2.5 Regression
2.6 Attrition
2.7 Testing
2.8 Instrumentation
2.9 Additive and Interactive Effects Of Threats To Internal Validity.
2.10 Inappropriate proxy indicators
2.11 Reliance on qualitative indicators
2.12 Unreliable respondent memory or deliberate distortion
3. Threats To Construct Validity19 : Reasons why inferences about theconstructs that characterize study operations may be incorrect
3.1 Inadequate explanation of constructs
3.2 Construct confounding
3.3 Mono-operation bias
3.4 Mono-method bias
3.5 Confounding Constructs with Levels of Constructs
3.6 Treatment sensitive factorial structure
3.7 Reactive self-report changes
3.8 Reactivity to the experimental situation
3.9 Experimental expectancies
3.10 Novelty and disruption effects
4. Threats To External Validity20 : Reasons why inferences about how
study results would hold over variations in persons, settings, treatments
and outcomes may be incorrect
4.1 Interaction of the causal relationship with units

19
20

Source: Adapted from Shadish, Cook and Campbell 2002. Table 3.1 page 73.
Source: Adapted from Shadish, Cook and Campbell Table 3.2 Page 87. The three final threats (in italics)
have been added by the present authors.

Foundation for Advanced Studies on International Development