Accessing and Building Local Evaluation Networks

Kathryn Newcomer 

November 10, 2016

1

1

Questions to Address Today

How might the American Evaluation Association and its affiliates help strengthen evaluation capacity in local nonprofit service providers?

How might local nonprofit service providers access other resources to help them strengthen capacity?

Why is the demand to supply evidence so prevalent?

2

| How AEA can strengthen the: | Within:                   |                                                                              |
|-----------------------------|---------------------------|------------------------------------------------------------------------------|
|                             | AEA and the United States | Other Voluntary Organizations of Public Evaluation(VOPEs) in Other Countries |
| Enabling Environment        |                           |                                                                              |
| Institutional Capacities    |                           |                                                                              |
| Individual Capacities       |                           |                                                                              |

AEA’s Participation in the International Evaluation Agenda

| How AEA can strengthen the: | Within:                   |                                 |
|-----------------------------|---------------------------|---------------------------------|
|                             | AEA and the United States | Other VOPEs and Other Countries |
| Enabling Environment        |                           |                                 |
| Institutional Capacities    |                           |                                 |
| Individual Capacities       |                           |                                 |

AEA Will Support All Six Areas

13 Specific Actions
| AEA will work to strengthen the: | Within AEA and the United States                                                                                                                                                                                                                                                                      | Within Other VOPEs and Other Countries                                                                                                               |
|----------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------|
| Enabling Environment             | 1.  Ask the Evaluation Policy Task Force (EPTF) to identify 1-2 key gaps in federal legislation, regulations, or practices, then work to correct those gaps                                                                                                                                           | 6. Find ways to learn systematically from other VOPEs around the world how they are strengthening their own enabling environments                    |
| Institutional Capacities         | 2. Work to strengthen the demand for evaluation within governments at all levels of government, the private sector, nonprofits, and foundations 3. Use Federal Executive Institute and other courses to train incoming federal-level political officers and SES candidates and leaders in evaluation  | 7. Identify emerging VOPEs in other parts of the world, and use AEA, Local Affiliates, and/or TIGs  to twin/mentor their development   (IPP Program) |

5

| Individual Capacities | 4. Offer relevant international topics to be featured in AEA’s e-studies programs 5. Expand in-person training opportunities beyond the conference and summer training institute, including online courses    | 8. Waive the conference and workshop fees for any developing country evaluator awarded conference travel funds by EvalPartners9. Solicit webinar speakers from outside the USA. Offer relevant topics to be featured in AEA’s e-studies programs (increase access by offering at different times to accommodate time zones)   10. Step up the marketing for the Silent Auction, including recruiting more corporate donations11. Match the travel funds raised during the Silent Auction, doubling the number of evaluators from developing countries AEA supports   12. Offer access to AEA online services to selected evaluators outside the USA (price/promotion to be determined by management)13. Continue to explore other ways AEA can promote the Global Evaluation Agenda 2016-2020 in general and partnerships with other VOPEs around the world in particular.   |
|-----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|

6

6

7

Evaluation Resources

American Evaluation Association – on-line resources

Local American Evaluation Association Affiliates

Local Colleges and Universities – in Schools or Departments of Education, or Public Administration or Public Policy Departments or Schools (especially programs that have required capstones!)

Betterevaluation.org

Innonet.org

“Evidence-based Policy” 

“Evidence-Based Policy” is a Mantra affecting governmental decision-makers, foundations, nonprofit boards, intermediaries and --- evaluation practice!

Myth or reality?

Advantages and disadvantages for evaluators and nonprofit service providers? Meeting expectations!  

8

Contrasting Views about Evidence-Based Policy
| Fixed Mindset                                                                                                                                       | Growth Mindset                                                                                                                                             |
|-----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1. We need to collect data to test if programs  work or do not work.                                                                                | 1. We need to learn which program mechanisms work for whom, where and under what circumstances.                                                            |
| 2. Policy should be made at the top and based on evidence.                                                                                          | 2. Policy is “made” through implementation processes at multiple levels by multiple actors with different types of data available to them.                 |
| 3. Program impact can be measured precisely.                                                                                                        | 3. Measuring program impact is difficult as programs and impactees change and evolve.                                                                      |
| 4. Random Control Trials (RCTs) are the gold standard for research and evaluation design.                                                           | 4. Research designs must be matched to answer the question raised; RCTs are appropriate for certain impact questions.                                      |
| 5. Proven program models can be replicated successfully in multiple locations as long as they are implemented with fidelity to the original design. | 5. Program mechanisms may be replicated successfully in multiple locations as long as they are adapted to meet local conditions.                           |
| 6. Benefit-cost analysis should be used to compare social programs.                                                                                 | 6. Benefit-cost analysis is difficult to use to compare social programs given the challenge of costing out benefits, especially those accruing  over time. |

9

What are Challenges for Evidence to Inform Policymaking?

Expectations regarding:

What constitutes evidence?

How transferable is evidence?

When and where do we underestimate the role played by the “impactees?”

Where is the capacity to support both the demand  and supply of evidence?

10

What are the Opportunities for Evidence to Inform Decision-making?

Analyses of “performance” data collected by agencies (or delegated service delivery agents such as grantees)

Implementation, Outcome and Impact evaluations typically performed by other agents for government

Manipulations of services in experiments by agencies – “behavioral economics”

Syntheses or systematic reviews of impact evaluations by external agents, e.g. websites like “What Works”

11

Why isn’t There Agreement About the Quality of Evidence?

Differing professional standards and “rules” or criteria for evidence, e.g., lawyers, accountants, engineers, economists

Disagreements about methodologies within professional groups, e.g., RCTs

The constancy of change in problems and the characteristics of the targeted impactees

12

We Underestimate the Evolving Sources of Complexity Affecting the Production of Relevant Evidence

Change in the nature of problems to be addressed by government, e.g., the nature of natural security threats, the use of the internet in crime

Change in the context in which programs and policies are implemented, e.g., increasingly complicated service delivery networks, PPPs

Changing priorities of political leaders (and would-be leaders)   

13

We Overstate the Ease of Flow of Evidence

It plays a wide (enough) causal role

Study conclusion: It plays a causal role there

Policy prediction: It will play a causal role here

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel. Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112.

14

14

What is needed for a well-supported effectiveness (impact) prediction? 

?

?

?

?

?

?

?

It can play the same role as there

?

It plays a positive causal role there (and there) 

STUDY

It will play a positive causal role here

The support factors for it are w, y, z

We have w, y, z here

It can play a positive causal role here

RCT

RCT

Source: Cartwright, N. (2013). Knowing what we are talking about: why evidence doesn't always travel. Evidence & Policy: A Journal of Research, Debate and Practice, 9(1), 97-112.

15

We Underestimate the Role of Volition Among Impactees and their Own Heuristics 

16

Evaluation Capacity = Both Demand and Supply

Who is asking for the evidence?

How clear is the understanding between providers and requestors on what sort of data (evidence) is needed?

Are there sufficient resources to respond to demand for data?

What about the lack of interaction and synergies among the different potential providers of evidence - such as at the U.S. federal level GPRA/GPRAMA reporting staff, internal evaluation staff, external evaluation contractors, SBST, data.gov teams, etc.! 

17

18

Promising Practices from the Obama Administration
| Promising Practice                                 | Affects Supply or Demand? | Needed Support Factors                                                        |
|----------------------------------------------------|---------------------------|-------------------------------------------------------------------------------|
| Knowledge Brokers, e.g., Chief Evaluation Officers | Both                      | Brokers have technical expertise, interpersonal skills, and contextual wisdom |
| Learning Agendas                                   | Demand                    | Strong leadership backing and encouragement to be innovative                  |
| Quarterly Reviews                                  | Supply                    | Credible data, stress on learning, no punitive actions                        |
| Strategic Reviews                                  | Both                      | Encouragement to be innovative, stress on learning not accountability         |

A simple framework....

19

Informed decisions & learning from Data

Develop and Address  information needs 

Cultivate an organizational  learning culture 

Cater to individual information processing

Remember Evaluation Capacity = Both Demand and Supply

Consider who is asking for the data/evidence and who might use the information provided and how and when they may use it

Probe the extent to which there is a clear understanding between providers and requestors for what sorts of evidence is needed, e.g., brokering

Assess whether or not sufficient resources are available to meet demand

Address the lack of interaction and facilitate synergies among the different potential providers of evidence - such as monitoring and reporting staff, internal evaluation staff, external evaluation contractors, etc. 

20

What are Evaluation-Receptive Organizational Cultures? 

Engage in self-reflection & self-examination

Deliberately seek evidence on what it’s doing

Use results information to challenge or support what it’s doing

Promote candor, challenge and genuine dialogue

Engage in evidence-based learning

Make time to learn 

Learn from mistakes and failures

Encourage knowledge sharing

Encourage experimentation and change

Support deliberate risk-taking

Seek out new ways of doing business

(See John Mayne, 2010) 

21

Move To Strategic and Synergistic Use of Evaluation!

22

Monitoring

Impact Evaluation

Implementation Evaluation

Behavioral Economics

Evaluation

Help Information Users Frame Pertinent Questions and then Match the Questions with the Appropriate Evaluation Approach

Questions Relevant

 to Users  

Evaluation Design

23

Match Evaluation Approach to Questions
| Objective                                                                   | Illustrative Questions                                                                                                                                                                                                                                                                                                                   | Possible Design                                                                                                                                                                                   |
|-----------------------------------------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| #1:  Describe  program activities                                           | How extensive and costly are the program activities?How do implementation efforts vary across sites, beneficiaries, regions?Has the  program been implemented sufficiently to be evaluated?                                                                                                                                              | MonitoringExploratory EvaluationsEvaluability AssessmentsMultiple Case Studies                                                                                                                    |
| #2:  Probe  targeting & implementation                                  | How closely are the protocols implemented with fidelity to the original design?What key contextual factors are likely to affect achievement of intended outcomes?How do contextual constraints affect the implementation  of a intervention?How does a  new intervention interact with other potential solutions to recognized problems? | Multiple Case StudiesImplementation or Process evaluationsPerformance AuditsCompliance AuditsProblem-Driven Iterative Adaptation                                                                  |
| #3:  Measure  the impact  of policies & programs                        | What are the average effects across different implementations of the intervention?Has implementation of the program or policy produced results consistent with its design (espoused purpose)?Is the implementation strategy more (or less) effective in relation to its costs?                                                           | Experimental  Designs/RCTsNon-experimental Designs: Difference-in-difference, Propensity score matching, etc.Cost-effectiveness & Benefit Cost AnalysisSystematic Reviews & Meta-Analyses |
| #4 :  Explain how/ why programs & policies produce (un)intended effects | How/why did the program have the intended effects?To what extent has implementation of the program had important unanticipated negative spillover effects?How likely is it that the program will have similar effects in other communities or  in the future?                                                                            | Impact Pathways and Process tracingSystem dynamicsConfigurational analysis,                                                                                                                       |

24

24

12/8/2015

Hart: Insights from Program Evaluation for  Retrospective Evaluation of Regulations

Relevant References

Dahler-Larsen, Peter.  2012. The Evaluation Society. Stanford University Press.

Donaldson, S., C. Christie, and M. Mark (editors) 2015. Credible and Actionable Evidence, 2nd Edition. Sage. 

Head, B. 2015. “Toward More “Evidence-Informed” Policy Making?” Public Administration Review. Vol.76, Issue 3, pp. 472-484.

Kahneman, D. 2011. Thinking, Fast and Slow. Farrar, Straus and Giroux Publishers.

Mayne, J. 2010. “Building an evaluative culture: The key to effective evaluation and results management.” Canadian Journal of Program Evaluation, 24(2), 1-30.

Newcomer, K. and C. Brass. 2O16. “Forging a Strategic and Comprehensive Approach to Evaluation within Public and Nonprofit Organizations:  Integrating Measurement and Analytics within Evaluation.” American Journal of Evaluation, Vol. 37 (1), 80-99.

Olejniczak, K., E. Raimondo, and T. Kupiec. 2016. “Evaluation units as knowledge brokers: Testing and calibrating an innovative framework.” Evaluation, Volume 22 (2)., 168-189.

Sunstein. C. and R. Hastie. 2015. Wiser: Getting Beyond Groupthink to Make Groups Smarter. Harvard Business Review Press.

World Bank Group. Mind, Society and Behavior.  2015. 

25

Thank You! 

I can be reached at  newcomer@gwu.edu

26

26

# image1.png

# image2.png

# image3.emf