Introducing the Impact Evaluability Assessment Tool

This	checklist	is	a	tool	designed	to	help	organizations	assess	and	discuss	a	program’s	readiness	to	

participate	in a	rigorous	impact	evaluation,	particularly	using	a quasi‐experimental	or	experimental	

design	study	to	measure	effectiveness.	It	can be	used	for	assessment,	planning,	and	communication	

purposes.	Rigorous	impact	evaluations	require	resources,	expertise	and	necessary	conditions	as	

well	as	thorough	planning	and	execution.	This	checklist	covers	the	range	of	necessary	elements	for	

conducting	an	impact	study	on	programs of	interest.	The	elements	are	covered	in	groups	or	

sections.	A	program	may	not	address	all	items	or	meet	all	requirements	noted	within	a	section,	

however	missing	items	or lack	of	readiness	on a	number	of	dimensions	may	indicate	potential	areas	

for	discussion,	development,	or	technical	assistance	prior	to engagement	in	a rigorous	impact	

evaluation.		

The	focus areas	covered	in	this	checklist	are	organized	into	three	broad	content	categories:	

Organization	Readiness;	Program	Readiness;	and	Evaluation Readiness.	Each	category	addresses	

key	elements	of	readiness.	Readiness	in	all	three	areas	covered	is	important	for	successful	planning	

and	implementation of	an	impact evaluation.

Organizational Readiness	addresses intentionality,	commitment	and	prioritization at	the

organization	level.	Core	to	these are	existing	support	for	evaluation,	capacity	building (as	needed),	

learning,	and use	of	data	for	decision	making within	the	organization,	especially	at	the	leadership	

level.	Additionally,	organizational	readiness	means	existence	and	support	for	the	infrastructure	

necessary	to	conduct	related	activities	within	the	organization.	

Program Readiness	addresses	elements	that need	to	be	in place	at the	program	level	for

conducting	rigorous	impact	evaluations.	These	cover	several	areas:	existing support	for	evaluation	

and	evidence	building	at	the	program	and	stakeholder	level,	operational	readiness,	program	scale,	

maturity	and	stability.

Evaluation Readiness addresses	three	areas of	focus	that include	a	history	of,	and	focus	on,	

evaluation,	as	well	as	the	resources,	structure,	capacity,	scope,	and	size	to	engage	in	rigorous	impact	

evaluation.	In	addition,	the	program has	an	evaluation	partner/team	in	place	that	has	the	

experience	and	skills	necessary for	this	type	of evaluation.		

Sources for Evaluability Checklist:

Evaluability Assessment: Examining the Readiness of a Program for Evaluation. Juvenile Justice Evaluation

Center. Justice Research and Statistics Association. www.jrsa.org/jjec. May 2003.

Evaluability Assessment: A Tool for Program Development In Corrections. Patricia Van Voorhis, Professor.

Division of Criminal Justice, University of Cincinnati.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.131.362&rep=rep1&type=pdf.

Anon. 2011. “Using the Evaluability Assessment Tool. Guidance Note 11.” ILO.

http://www.ilo.org/wcmsp5/groups/public/‐‐‐ed_mas/‐‐‐eval/documents/publication/wcms_165984.pdf

Sources for Proposed Evaluation Partners Checklist:

Modified from: Dunn, E. 2008. “Planning for Cost Effective Evaluation with Evaluability Assessment”.

USAID. http://pdf.usaid.gov/pdf_docs/PNADN200.pdf

1
Organizational Readiness (existing support for evaluation, capacity building [as needed],

learning, use of data for decision making within the organization, especially at the leadership level,

and existence of requisite infrastructure to support related activities.)

Indicate to what ex

Not at

all true

Some

tr

Leadership CommitmentThere is support for the evaluation and evaluation capacity building, as needed, at the leadership level (CEO

and/or Board of Directors).

  

Leadership demonstrates commitment to evaluation and evidence‐based or data‐driven decision making.   Leadership supports staff positions/activities that focus on evaluation, learning, and improvement.   

Organization and its Board of Directors demonstrate interest in learning about the effectiveness of the

program by rigorously evaluating program effectiveness.

  

Learning EnvironmentThe organization provides opportunities for and fosters a culture of information sharing, discussion, reflection,learning, and improvement in order to support informed decision‐making and practice.

  

Staff makes decisions based on regular assessment and use of data, information, evidence and feedback. For

example, if a program was evaluated in the past, information that came from the evaluation was utilized.

  

Resource CommitmentLeadership is willing and committed to devoting necessary resources (e.g. staff positions and time andfinancial or other non‐financial resources) to the evaluation.

  

Tools and Systems

There are systems, structures, tools, and processes in place for data collection, storage, processing, analysis,

and reporting.

  

There are systems, structures, tools, and processes in place for information sharing, reflection, knowledge

building, and evaluation use.

  

Additional Comments:
Program Readiness(existing support for evaluation and evidence building at the program and

stakeholder level, operational readiness, program scale, maturity and stability.)

Indicate to what e

Not at all

true

Some

tr

Theory of Change

There is a coherent, logical program theory. Strategies and activities are designed to address a clearly identified

and defined problem or need. There is a logical connection between the program strategies and activities and

the intended outcomes or desired changes. Goals and objectives are articulated and attainable with the

available resources. (The program has a logic model.)

  

Program participation is clearly defined and distinguishable from nonparticipation. There is no ambiguity about

who is in the program and who is not.

  

There is a shared understanding among program leadership and staff about the core elements of the programand the context in which the program operates.

  

There is agreement across the program leadership and staff as to what the expected program outcomes are.   

Clear Time Frame for the Program

The intervention has a clearly defined timeframe.   

There is a reasonable and shared expectation around the timeframe for when observable/measurable

outcomes in the short, intermediate or long term will occur.

  

Support for Evaluation and Evidence Building

The program leadership and staff have a learning agenda for the implementation and effectiveness of the

program so as to inform the evaluation.

  

There is interest and support among stakeholders in advancing a program’s level of evidence by conducting an

impact evaluation. Stakeholders see the value of evaluation and have ideas about how the program couldbenefit.

  
Program Readiness(existing support for evaluation and evidence building at the program and

stakeholder level, operational readiness, program scale, maturity and stability.)

Indicate to what e

Not at all

true

Some

tr

Program and stakeholders are likely to agree (or are open to discussion) on what levels of evidence should be

currently targeted, including the need for the development of evaluations that are designed to limit threats to

internal validity1 and/or enhance external validity2 to the greatest extent possible.

  

There is allocation of a reasonable level of staff time and resources to conduct an impact evaluation at the

program level.

  

Program ImplementationIf the program is based on a model or logical program theory, it is implemented with fidelity to that model andhas a well‐planned sequence of activities.

  

If the program is currently being adapted, it is being adapted using theory/systematically‐obtained field‐based

knowledge, and along lines that can be quantified and documented.

  

Staff members are qualified and properly trained to operate the program. There are enough qualified staff

members on site to implement the planned activities.

  

Frontline workers who deliver the services provided by the program have sufficient qualifications to execute

the program. There are enough qualified frontline workers on site to successfully execute the program.

  

There are systems in place to track program implementation: There are procedures in place to determine if the target population is being served (referral system, intake

process).

 Data that track service usage is collected (attendance lists, case management logs). Input is sought on a regular basis to understand how participants experience the services and to identify

and address any problems in a timely manner.





1 Definition of Internal Validity: For a given design, the extent to which the observed difference in the average group outcomes (usu

versus control or comparison group members) can be causally attributed to the intervention or program. Randomized controlled tria

attribution because of their ability to rule out alternative explanations (usually unobserved characteristics) other than the interventio

observed affect.

2 Definition of External Validity: The extent to which evaluation results, statistically, are applicable to groups other than those in the 

technically, it refers to how well the results obtained from analyzing a sample of study participants from a population can be generaliThe strongest basis for applying results obtained from a sample to a population is when the sample is randomly selected from that po

generalization must be made on extra‐statistical ground – that is, on a non‐ statistical basis.
Program Readiness(existing support for evaluation and evidence building at the program and

stakeholder level, operational readiness, program scale, maturity and stability.)

Indicate to what e

Not at all

true

Some

tr

Program Maturity and Stability

The program has been in operation for a reasonable length of time and is known in the target community, or

has clear evidence of both uptake and effectiveness in other, similar communities.

  

The program is relatively mature and stable and is not undergoing refinements or changes that are expected to

occur in early stages of program development and delivery (i.e. the intervention/experiment is repeatable and

likely to produce the same effects over time).

  

Risks/threats to program delivery (e.g. recruitment of participants/deliverers, constancy of necessary

partnerships) have been identified, and risk monitoring and mitigation processes have been proposed or arecurrently in place.

  

External/contextual influences and factors are accounted for and assessed as relatively stable. These forces

(e.g. policy environment) are not expected to affect the program and its participants in a significantly different

way over time.

  

Scale/participation numbersThe program’s intentions for expanding or advancing the model/intervention are clearly planned out, and

sufficiently resourced and feasible.

  

The program is being delivered at a scale that allows for reasonable impact measurement against a

counterfactual/comparison group controlling for potential biasing factors, such as demographic characteristicsof participants (i.e. there is adequate statistical power for a statistical analysis in accordance with evidence

standards).

  
Program Readiness(existing support for evaluation and evidence building at the program and

stakeholder level, operational readiness, program scale, maturity and stability.)

Indicate to what e

Not at all

true

Some

tr

Additional Comments on Program Readiness:
Evaluation Readiness (Program has a history of, and focus on, evaluation, as well as the

resources, structure, capacity, scope, and size to engage in rigorous impact evaluation. In addition,the program has an evaluation partner/team in place that has the experience and skills necessary for

this type of evaluation.)

Indicate to what e

Not at all

true

Some

tr

Past Evidence/Evaluation WorkA process evaluation or implementation analysis has been (or is currently being) conducted to ensure that the

program intervention is implemented as envisioned, reaching the expected target group, leading to expected

results, and to assess program participation, engagement, satisfaction, quality, and efficiency. For example, The process evaluation showed (or shows) that the program is serving/reaching its target population. The process evaluation showed (or shows) that planned activities are implemented as intended; services

are delivered in sufficient amount and quality.

 The process evaluation showed (or shows) that participants and other key stakeholders are satisfied with

the program.







There is agreement across the program and stakeholders as well as evaluation partner(s) as to what program

outcomes are, and on what types of outcomes data should be collected.

  

Outcomes are relevant to the services and program’s objectives, and clearly expressed in the program’s logic

model. Outcomes may be expressed as short‐term, intermediate, or long‐term objectives.

 Current outcome measures are relevant and valid indicators of progress toward program objectives. For

example, pre‐ post data show evidence that program beneficiaries experience a change in attitude,

awareness, knowledge, behavior or condition.

 The program selects its current outcome measures and targets in conjunction with external standards of

effectiveness.





Outcomes are defined in quantifiable, measurable terms, and procedures for measuring outcomes have been

implemented.

  

Performance data (i.e., performance measures) are routinely collected.   

Prior outcome evaluations have been conducted and there is supporting evidence that the program isproducing the desired results for participants/beneficiaries, and there is a compelling case for allocating

resources for conducting an impact evaluation.

  

There is a shared understanding regarding the existing evidence behind the intervention/model by program

stakeholders and evaluation partner(s).

  

Evaluation Questions
Evaluation Readiness (Program has a history of, and focus on, evaluation, as well as the

resources, structure, capacity, scope, and size to engage in rigorous impact evaluation. In addition,the program has an evaluation partner/team in place that has the experience and skills necessary for

this type of evaluation.)

Indicate to what e

Not at all

true

Some

tr

Evaluation questions are clearly stated and they cover what key stakeholders (including program staff) want to

learn about the program.

  

Evaluation questions are in line with proposed methods of evaluation and program design.   

Any current evaluation questions consider both process and outcomes and seek to understand the why, how,

and cause and effect of program impact.

  

Program Evaluation ReadinessThe program’s intentions for expanding or advancing the outcomes/impacts to be measured for the current

project are clearly planned out, and sufficiently resourced and feasible.

  

The program has capacity (expertise, skills, staff time) to conduct an evaluation internally or in partnership withan external partner, or to work with an external evaluator to plan and implement an impact evaluation.

  

If the evaluation will involve contracting with an external and independent evaluator, the program must have

the capacity to effectively contract with and monitor work of the external evaluators such that: The program has or can set clear criteria for selecting which evaluator will be hired.

 The program has or can set a clear plan for effective communication with the evaluation contractor, and

means (e.g. staff time and knowledge) for monitoring evaluator activities.

 The program is capable of developing a contract that meets the needs and requirements of parties

involved, which include the program itself, funding partner(s), and the evaluation partner.





The program has internal evaluation capabilities and processes in place to allow for clear communication with

evaluation partner(s).

  

Proposed Evaluation Partner(s)

The proposed evaluation partner has previous experience in the following sub‐categories:

 Has substantial experience with the logistics of running rigorous experimental or quasi‐experimental

evaluations.

 Has a team of trained and experienced evaluators.

 Is experienced with evaluations of comparable programs (similar size, scope, and focus)

 Is experienced with conducting data collection of the type anticipated for the evaluation.

 Is experienced with conducting data collection with the target population for the study.




Evaluation Readiness (Program has a history of, and focus on, evaluation, as well as the

resources, structure, capacity, scope, and size to engage in rigorous impact evaluation. In addition,the program has an evaluation partner/team in place that has the experience and skills necessary for

this type of evaluation.)

Indicate to what e

Not at all

true

Some

tr

 Is experienced with the collection and analysis of impact data from more than one time point (e.g. pre‐post

or time series), including management of data at the level of analysis anticipated for the evaluation (e.g.

individual, group, multi‐level).

 Has no conflicts of interest, if hired.







The proposed evaluation partner: Is able to mobilize data collection and management teams in the regions where the study is to be

conducted, with evaluators able to conduct interviews, administer surveys, and collect other forms of data.

 Is able to place a qualified researcher/evaluator in charge. This individual must be willing and able to work

closely with the program evaluation team and provide informed input.

 Is able to provide qualified and experienced individuals to support the desired evaluation design.





The proposed evaluation partner: Is willing to commit to a study that will most likely have multiple modes of data collection, at multiple time

points.

 Is willing/able to respond to requirements, criteria and input from the program, the key program funding

partner, and any funding evaluation partners (e.g. in the case of multi‐site evaluations of initiatives).





An appropriate or reasonable budget is available and allocated to the evaluation. The evaluation budget is

adequate for the type of evaluation design envisioned and in line with expectations about the work.

  

There are agreements and capabilities across the program and stakeholders for developing a timeline for, andtimely production of evaluation deliverables, and to publish, communicate, and/or disseminate

deliverables/findings.

  

Specific Evaluation Requirements/Logistics/Pre‐Requisites

There is agreement and commitment from all necessary program staff and stakeholders regarding the

collection and use of data that is needed for evaluation purposes, including data relating to

participant/beneficiary satisfaction, outcomes and impacts.

  

If the program is going to rely on administrative/secondary data, access to such data is possible and the needed

agreements (e.g. Memoranda of Understanding) can be secured.

  

If a randomized controlled trial is to be conducted, sites are on board with the approach and ready to work with

evaluators to assign an eligible pool of potential participants into treatment and control groups.

  
Evaluation Readiness (Program has a history of, and focus on, evaluation, as well as the

resources, structure, capacity, scope, and size to engage in rigorous impact evaluation. In addition,the program has an evaluation partner/team in place that has the experience and skills necessary for

this type of evaluation.)

Indicate to what e

Not at all

true

Some

tr

The program design is such that periods of baseline and follow‐up data collection can be defined for evaluation

purposes (i.e. participant baseline measures can be collected or obtained prior to program service delivery.)

  

The program (and stakeholders, if necessary) has a viable management information system, and efficientrecord‐keeping processes. The program has a demonstrated capacity to generate data (e.g. client records,survey data, progress reports) that can be exported to others and merged for evaluation use.

  

Data will be effectively updated, archived, and securely stored.   

Staff members are well trained to collect data and use the information system.   

Risks/threats to rigorous evaluation have been identified, and risk monitoring and mitigation strategies have

been proposed or are currently in place.

  

Human SubjectsRequirements around human subject protection are considered and addressed in line with the properInstitutional Review Board (IRB).

  

The program has adequately outlined a plan for obtaining consent forms if needed as well as handling, securely

storing, and sufficiently destroying personally identifiable data.

  

Evaluation TimeframeThe timing of the evaluation is commensurate with the timeframe of the program and when intended

outcomes can be measured or observed (e.g. longer than program intervention timeframe) and in line with theduration of the grant (i.e. before the grant period ends).

  

Comparison or Control Group

There are enough individuals and/or sites participating in the program (depending on the unit at which

program participation is assigned) to allow for comparison group analysis. In other words, the program is of

sufficient size that can leave enough potential participants and/or sites unserved to allowing the formation of a

matched comparison or randomly assigned control group of sufficient size to make statistical comparisons

possible.

  

The comparison or control group can be formed from individuals who are within the same school, community,

or other comparable grouping as the participating group.

  
Evaluation Readiness (Program has a history of, and focus on, evaluation, as well as the

resources, structure, capacity, scope, and size to engage in rigorous impact evaluation. In addition,the program has an evaluation partner/team in place that has the experience and skills necessary for

this type of evaluation.)

Indicate to what e

Not at all

true

Some

tr

If the program cannot be assigned randomly, a sufficient sample size and amount of background data will beavailable for statistical adjustment and analysis during the evaluation.

  

If one evaluation design will not address key threats to internal validity, there is a way to construct a combined

design where two or more separate study components combine to sufficiently reduce multiple threats to

internal validity (e.g. History‐, Time‐, and Site‐related factors are potential confounds, meaning they could be

omitted factors that caused the results) and allow justifiable causal claims.

  

Additional Comments:

Impact Evaluability Assessment Tool is a product of the Office 

at the Corporation for National and Community Service and is Commons Attribution‐NonCommercial‐NoDerivatives (CC BY N

License.Created by Lily Zandniapour (CNCS) with support from JBS InteSuggested citation: Zandniapour, Lily, and JBS International. 20

Assessment Tool”. Washington, DC: Corporation for National C