Accessing and Building

Local Evaluation Networks

Kathryn Newcomer 

November 10, 20161 
Questions to Address Today

 How might the American Evaluation Association 

and its affiliates help strengthen evaluation capacity in local nonprofit service providers? 

 How might local nonprofit service providers 

access other resources to help them strengthen capacity? 

 Why is the demand to supply evidence so 

prevalent? 

2 
AEA’s Participation in the Internat

Evaluation Agenda

How AEA can 

strengthen the: 

AEA and the United States 

Within: 

Other Volun

Public Ev

Oth

Enabling Environment 

Institutional Capacities 

Individual Capacities 
AEA Will Support All Six Ar

How AEA can 

strengthen the: 

AEA and the United States 

Within: 

Other V

Enabling Environment 



Institutional Capacities 



Individual Capacities 


AEA will work to 

strengthen the: 

Within AEA and the United States Within Other V

Co

1. Ask the Evaluation Policy Task Force 6. Find ways to learn 

(EPTF) to identify 1‐2 key gaps in federal other VOPEs around 

Enabling legislation, regulations, or practices, then strengthening their o

Environment work to correct those gaps environments

2. Work to strengthen the demand for 7. Identify emerging 

evaluation within governments at all the world, and use A

Institutional levels of government, the private sector, and/or TIGs to twin/

Capacities nonprofits, and foundations development (IPP P

3. Use Federal Executive Institute and

other courses to train incoming federal‐

level political officers and SES candidates

and leaders in evaluation

5

13 Specific Actions
Individual 

Capacities

4. Offer relevantinternational topics to be

featured in AEA’s e‐studiesprograms5. Expand in‐person training

opportunities beyond the

conference and summer

training institute, including

online courses

6 

8. Waive the conference and workshop developing country evaluator awarded 

funds by EvalPartners

9. Solicit webinar speakers from USA. Offer relevant topics to bAEA’s e-studies programs (increoffering at different times to azones) 10. Step up the marketing for the Silent 

recruiting more corporate donations

11. Match the travel funds raised during doubling the number of evaluators from 

AEA supports12. Offer access to AEA online services t

outside the USA (price/promotion to be 

management)13. Continue to explore other ways AEA 

Global Evaluation Agenda 2016‐2020 in partnerships with other VOPEs around t
Evaluation Resources

• American Evaluation Association – on-line reso• Local American Evaluation Association Affiliat• Local Colleges and Universities – in Schools or 

Departments of Education, or Public AdministPublic Policy Departments or Schools (especiaprograms that have required capstones!) 

• Betterevaluation.org • Innonet.org 

7 
“Evidence-based Policy” 

 “Evidence-Based Policy” is a Mantra affecting 

governmental decision-makers, foundations, nonprofit boards, intermediaries and --

-

evaluation practice! 

 Myth or reality?  Advantages and disadvantages for evaluators and 

nonprofit service providers? Meeting expectations! 

8 
Contrasting Views about

Evidence-Based Policy

Fixed Mindset Growth Mindset 

1. We need to collect data to test if programs 1. We need to learn which program mechanisms 

work or do not work. work for whom, where and under what 

circumstances. 

2. Policy should be made at the top and based on 2. Policy is “made” through implementation evidence. processes at multiple levels by multiple actors with 

different types of data available to them. 

3. Program impact can be measured precisely. 3. Measuring program impact is difficult as 

programs and impactees change and evolve. 

4. Random Control Trials (RCTs) are the gold 4. Research designs must be matched to answer standard for research and evaluation design. the question raised; RCTs are appropriate for 

certain impact questions. 

5. Proven program models can be replicated 5. Program mechanisms may be replicated 

successfully in multiple locations as long as they successfully in multiple locations as long as they are implemented with fidelity to the original are adapted to meet local conditions. design. 6. Benefit-cost analysis should be used to compare 6. Benefit-cost analysis is difficult to use to 

social programs. compare social programs given the challenge of 9 costing out benefits, especially those accruing 

over time. 
What are Challenges for Evidence 

to Inform Policymaking?

Expectations regarding: 

 What constitutes evidence?  How transferable is evidence?  When and where do we underestimate the role played 

by the “impactees?” 

 Where is the capacity to support both the demand and 

supply of evidence? 

10 
What are the Opportunities for Evidenc

to Inform Decision-making?

 Analyses of “performance” data collected by agencies 

(or delegated service delivery agents such as grantees)

 Implementation, Outcome and Impact evaluations 

typically performed by other agents for government

 Manipulations of services in experiments by agencies –

“behavioral economics”

 Syntheses or systematic reviews of impact evaluations 

by external agents, e.g. websites like “What Works”

11 
Why isn’t There Agreement About 

the Quality of Evidence?

 Differing professional standards and “rules” or criteria for 

evidence, e.g., lawyers, accountants, engineers, economists 

 Disagreements about methodologies within professional 

groups, e.g., RCTs 

 The constancy of change in problems and the 

characteristics of the targeted impactees 

12 
We Underestimate the Evolving Sources o

Complexity Affecting the Production of 

Relevant Evidence

 Change in the nature of problems to be addressed by 

government, e.g., the nature of natural security threats, the use of the internet in crime 

 Change in the context in which programs and policies are 

implemented, e.g., increasingly complicated service delivery networks, PPPs 

 Changing priorities of political leaders (and would-be 

leaders) 

13 
It plays a wide 

(enough) causal 

role 

Study conclusion: It plays Policy prediction: It 

a causal role there will play a causal 

role here

We Overstate the Ease of Flow of Eviden

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel. Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112. 

14 
What is needed for a well-supported 

effectiveness (impact) prediction? 

It will play a positive causal role here 

The support factors We have w, y, z It can play a positive 

for it are w, y, z here causal role here 

It can It plays a 

play the positive causal 

same role role there (and 

as there there)

? ? ? ? ? ? ? 

S R R

D ? 

T C C

Y U T T 

15 

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel.Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112.
We Underestimate the Role of Volition Among

Impactees and their Own Heuristics

MECHANISM CHANGE 

PREMISES IMPLEMENTATION PLAN 

Based on information 

and evidence we decide that... 

IF we implement 

successfully certain activities... 

...THEN we trigger 

desired behaviors and 

processes in 

a target group... 

...AND THAT leads 

to positive, 

sustainable 

change 

DETAILED Heuristics OUTPUTS 

PREMISES and biases Infrastructure 

Facts INPUTS EFFECTS ISSUE ACTIONS Services 

Research result Opportunity Money Operations Reaction of Structural and Information Personnel long term Information Need Procedures Disincentives subjects 

Earlier Problem Other Processes change Incentives 

resources experiences Choice 

Opinions Context architecture 

UNDERLYING THEORIES THEORY OF IMPLEMENTATION THEORY OF CHANGE 

On what premises do we base our 

decisions? 

How do we want to use inputs to produce 

desired outputs? 

How will positive change be produced? 

16 
Evaluation Capacity = 

Both Demand and Supply

 Who is asking for the evidence?  How clear is the understanding between providers and requestors on 

what sort of data (evidence) is needed? 

 Are there sufficient resources to respond to demand for data?  What about the lack of interaction and synergies among the different 

potential providers of evidence - such as at the U.S. federal level GPRA/GPRAMA reporting staff, internal evaluation staff, external evaluation contractors, SBST, data.gov teams, etc.! 

17 
Promising Practices from

the Obama Administration

Promising Affects Supply or Needed Support Practice Demand? Factors Knowledge Brokers, Both Brokers have e.g., Chief technical expertise, Evaluation Officers interpersonal skills, 

and contextual wisdom 

Learning Agendas Demand Strong leadership 

backing and encouragement to be innovative 

Quarterly Reviews Supply Credible data, 

stress on learning, no punitive actions 

Strategic Reviews Both Encouragement to 

be innovative, stress on learning not accountability 

18 
A simple framework.... 

19 

Informed 

decisions 

& learning 

from Data 

Develop and 

Address 

information 

needs 

Cultivate an 

organizational 

learning 

culture 

Cater to 

individual 

information 

processing 
Remember Evaluation Capacity = 

Both Demand and Supply

 Consider who is asking for the data/evidence and who 

might use the information provided and how and when they may use it 

 Probe the extent to which there is a clear understanding 

between providers and requestors for what sorts of evidence is needed, e.g., brokering 

 Assess whether or not sufficient resources are available to 

meet demand 

 Address the lack of interaction and facilitate synergies 

among the different potential providers of evidence - such as monitoring and reporting staff, internal evaluation 20 staff, external evaluation contractors, etc. 
What are Evaluation-Receptive

Organizational Cultures? 

 Engage in self-reflection & self-examination 

 Deliberately seek evidence on what it’s doing  Use results information to challenge or support what it’s doing  Promote candor, challenge and genuine dialogue 

 Engage in evidence-based learning 

 Make time to learn  Learn from mistakes and failures  Encourage knowledge sharing 

 Encourage experimentation and change 

 Support deliberate risk-taking  Seek out new ways of doing business (See John Mayne, 2010) 

21 
Move To Strategic and Synergistic 

Use of Evaluation!

Evaluation 

Monitoring 

Impact 

Evaluation 

Implementation 

Evaluation 

Behavioral 

Economics 

22
Help Information Users Frame 

Pertinent Questions and then 

Match the Questions with the 

Appropriate Evaluation Approach 

Questions Relevant 

to Users 

Evaluation 

Design 

23 
Match Evaluation Approach to Questions 

Objective Illustrative Questions Possible Design #1: Describe  How extensive and costly are the program activities?  Monitoring program  How do implementation efforts vary across sites, beneficiaries,  Exploratory Evaluations regions?  Evaluability Assessments activities  Has the program been implemented sufficiently to be evaluated?  Multiple Case Studies #2: targeting  How closely are the protocols implemented with fidelity to the  Multiple Case Studies Probe & implementation original design?  Implementation or Process 

 What key contextual factors are likely to affect achievement of evaluations 

intended outcomes?  Performance Audits  How do contextual constraints affect the implementation of a  Compliance Audits intervention?  Problem-Driven Iterative  How does a new intervention interact with other potential Adaptation solutions to recognized problems? 

#3: Measure the  What are the average effects across different implementations of  Experimental Designs/RCTs 

the intervention?  Non-experimental Designs: impact of policies &  Has implementation of the program or policy produced results Difference-in-difference, Propensity programs consistent with its design (espoused purpose)? score matching, etc. 

 Is the implementation strategy more (or less) effective in relation  Cost-effectiveness & Benefit Cost 

to its costs? Analysis 

 Systematic Reviews & MetaAnalyses 

#4 : Explain how/  How/why did the program have the intended effects?  Impact Pathways and Process 

 To what extent has implementation of the program had important why programs & tracing unanticipated negative spillover effects? policies produce  System dynamics 24  How likely is it that the program will have similar effects in other  Configurational analysis, (un)intended effects communities or in the future? 
Relevant References 

 Dahler-Larsen, Peter. 2012. The Evaluation Society. Stanford University Press.  Donaldson, S., C. Christie, and M. Mark (editors) 2015. Credible and Actionable Evidence, 

2nd Edition. Sage. 

 Head, B. 2015. “Toward More “Evidence-Informed” Policy Making?” Public Administration 

Review. Vol.76, Issue 3, pp. 472-484. 

 Kahneman, D. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux Publishers.  Mayne, J. 2010. “Building an evaluative culture: The key to effective evaluation and 

results management.” Canadian Journal of Program Evaluation, 24(2), 1-30. 

 Newcomer, K. and C. Brass. 2O16. “Forging a Strategic and Comprehensive Approach to 

Evaluation within Public and Nonprofit Organizations: Integrating Measurement and Analytics within Evaluation.” American Journal of Evaluation, Vol. 37 (1), 80-99. 

 Olejniczak, K., E. Raimondo, and T. Kupiec. 2016. “Evaluation units as knowledge brokers: 

Testing and calibrating an innovative framework.” Evaluation, Volume 22 (2)., 168-189. 

 Sunstein. C. and R. Hastie. 2015. Wiser: Getting Beyond Groupthink to Make Groups 

Smarter. Harvard Business Review Press. 

 World Bank Group. Mind, Society and Behavior. 2015. 25 
Thank You! 

I can be reached at newcomer@gwu.edu 

26