Using Evaluation Results to Learn 

Improve 
Objectives 

• Hear findings from high quality 

AmeriCorps evaluations acrosrange of focus areas and gran

• Learn how AmeriCorps grante

using evaluation results to imptheir programs 
Panelists 

• Stephanie Biegler, Child Abuse Prevention Coun• Joel Krogstad, St. Paul Neighborhood Network • Peter Rumsey, Habitat for Humanity, Internation• Amy Hetrick, CNCS • Allyson Augustin, City Year 
  ‐  

Birth & Beyond 

Family

Resource 

Center 

Parenting

Workshops 

Crisis 

Intervention 

Services 

School 

Readiness 

NPP Home 

Visitation 

Sacramento County

Child Protective

Services

Walk‐ins, Hospitals

Schools, Communit

Partners

Differential

Response and

Aftercare for

families with

children 0 5 years

S

Family Resource Centers 
Sacramento County 

CPS Referrals 

by Zip Code 
Birth & Beyond | Theory of Cha

• NEED: Documented high rates of child abuse and neglect 

attributable to deficits in parenting knowledge/practices prevent a child’s healthy emotional/physical developmen

• INTERVENTION: Members provide at least 8 hours of par

education through home visits and workshops to high ris

using the Nurturing Parenting Program to teach parents c

development, building empathy, parent/child roles, estabfamily routines, and replacements to corporal punishmen

parenting skills as an alternative to abuse and neglect.

• OUTCOMES: Parents demonstrate reduced risk for child a

neglect and do not enter/re‐enter the child welfare syste
Prior Annual Evaluation|

Did Birth & Beyond Parents Have a

Reduced Rate of Child Welfare Refer

• Child welfare ‘look up’ study annually since 2• Consistently positive results for child welfare

outcomes

• Half of B&B parents had a prior history of chi

welfare referral (including as victim)

• All B&B parents with at least 8 hours of Hom

Visitation had a reduced rate of child welfare 

one year after their case closure
Quasi‐experimental Study|

Will any parent participating in Birth & Be

have a reduced rate of CPS referrals?

• Quasi‐experimental study includes all Home

Visitation parents, with child welfare history, 

were served by an AmeriCorps member in th2015 program years

• Impact evaluation allows for the direct attrib

positive changes to Home Visitation program provides ‘higher‐level of evidence’ of program

effectiveness)
Quasi‐ Experimental Study

• Compares HV parents (n=496) with previous 

welfare history to non‐HV parents (n=985) w

similar child welfare history

• Controls for bias that some individuals are m

willing/likely to participate in HV program

• Accounts for rolling program entries and exi• Holds all demographic factors constant to co

child welfare recidivism over time
Evaluation | Findings

• All parents participating in HV were

41% less likely to have a substantiated child 

referral over a 4‐year period than non‐HV pa

• Parents with 25‐36 hours of HV were 173% le

likely to have a substantiated child welfare 

over a four‐year period than non‐HV parents

(statistically significant at p < 0.05)

• Parents with at least 8 hours of HV showed a

average decrease in risk/increase in parenti

as measured by the AAPI (statistically significant at p 

Current Evaluation | Summary

• CPS Recidivism | All parents participating in 

a reduced likelihood of new child welfare ref

• Optimal Hours | Those parents receiving 25‐

hours of HV have the greatest reduction in th

likelihood new child welfare referrals.

The findings of this study provide strong evidence that & Beyond home visitation supports Sacramento Cparents and their families by reducing the risk of welfare referrals after participating in the program.
Evaluation |Recommendation

• Consider aligning home visitation dosage tow

the optimal 25‐34 hour range.

• Explore strategies to address program attritio

ensure parents are receiving optimal level of 

visitation program dosage.

• Consider additional research to explore rema

and emergent questions from this study abo

effectiveness and impact.
Evaluation Tips from 

Community Technology Empowerment Project AmeriCorps Symposium Sept 22, 2016 

Joel Krogstad, Program Director krogstad@spnn.org 550 Vandalia Street, Suite 170 

 Saint Paul, MN 5511

www.spnn.org/ctep 
35 AmeriCorps members 

Teaching technologyskills for economic, social and civic empowerment at 28 community-basedpartner agenciesin Twin Cities, Minnesota 

Target Population:  Low-income  Unemployed or

Underemployed 

 Have low technolo

skills 
MEMBER ACTIVITIES/

THEORY OF CHANGE

 Teach classes with NorthStar Digital Literacy 

Standards 

 Proctor Assessments to gain basic tech skill 

certifications for 1500 community members 

 Assisting community members to use acquired

tech skills for job placement: 600 to gain employment 

RESEARCH QUESTIONS 

FROM A REPRESENTATIVE SAMPLE OF ADULT LEARNERS IN CTEP PROGRAMWE SOUGHT TO DETERMINE....  the employment rate change after attending at lea

four hours of computer skills programming in a CTprogram. 

 the employers and job types of those who received 

employment 

 how the CTEP employment result compares to a 

metro wide employment comparison group (DEED

 the Return on Investment for the value of all new j

received. 
STUDY DESIGN AND COST 

 Quasi-experimental study with a comparison 

group (DEED) 

 CTEP members did data collection over course 

one program year with evaluator doing studydesign, instrument creation, training, qualitycontrol, analysis, and report write up. 

 208 surveys administered in 28 of 30 CTEP sit

confidence level 95% 

 Evaluation cost $7000, for time of the lead 

evaluator. 
KEY FINDINGS

 80% reported computer class helped look for employment.  71% reported attending computer class helped in getting 

offered job. 

 $31,266: average salary, mostly office/admin, sales, person

care 

 50% found a job within 4 months after class  Beat Employment Result from Comparison Group (DEED)

41% (339/818) 

 $5.7 million total estimated economic impact of (value of n

jobs created) 
Learn More! Study posted at 

www.spnn.org/ctep 
G

Eva

AmeriCorp

Septe
We build strength, stability and self-reliance 

What’s the 

you want to 

Can you prove 

ddd

ddd
• qwerty Start with... 
Add some ph.D’s... 
Research Questions: added va

• Do host affiliates (through members) inc

capacity compared to similar, non-hostinaffiliates? 

– Serve more families? – Engage more volunteers? – Increase new builds and rehabs? 

• Do members benefit?: 

– Obtain full workforce development training 

mentorship? 

– Learn new and transferrable skills? – Increase new and lasting civic – engagement interests? 
Surveys and Data Sourc

• Affiliate and Host Affiliate (Pre-Post) 

– Completed by leadership – Matched comparison – Impact and added capacity of members o

affiliate outcomes. 

• Member (Pre-Post) 

– Fall of 2014 to Fall of 2015 – Member perceptions 

• Alumni (Post-only) 

– Long-term perceptions 
Assumptions should be 

The data will set yo

Be courageous! 
• Do host affiliates experience increase

capacity compared to similar, non-hosaffiliates? 

– Serve more families? YES – Engage more volunteers? YES – Increase new builds and rehabs? YES 

• Do members benefit? 

– Value workforce development training and 

mentorship? YES

– Learn new and transferrable skills? YES 

– Increase new and lasting civic engagement 

interests? YES
Phew...
• More than 600 AmeriCorps members were appro

provide interventions that are aligned with schooturnaround plans in more than 70 schools in 15 s

School Turnaround AmeriCorps Overview • A partnership between the U.S. Department of E

and CNCS 

• 13 programs funded in 2013 • Programs operate in School Improvement Grant 

schools and Priority schools 

• Programs coordinate with school leaders and us

student data to target interventions 
School Turnaround AmeriCorps 

Overview• Members focus on one or more of th

following:

– Family and Community Engagement – Addressing Non-Academic Factors that Impact Studen

Achievement (attendance, discipline, school safety, soemotional needs, etc.) 

– Reading and Math knowledge and skill acquisition – Increase Graduation Rates – Increase College Enrollment Rates – Increase Learning Time 
School Turnaround AmeriCorps 

Overview • Austin Independent School District • Berea College • Blackfoot Community Center • City Year, Inc. • Communities in Schools of Miami • Denver Public Schools • Detroit Parent Network • Duluth Area Family YMCA • Learning Works • MN Alliance With Youth • ReNEW-Reinventing Education (ReNEW Schools) • Springfield College • Teach For America 
School Turnaround AmeriCorps 

Study Design • Evaluation Design(s): Comparative Case Study, 

Implementation Study. 

• Study Population: School leaders and staff, gran

AmeriCorps members, and parents. 

• Evaluator: Abt Associates. 
School Turnaround AmeriCorps 

Research Questions • How was the School Turnaround AmeriCorps pr

implemented in schools? 

• How did school leaders perceive the impact of 

AmeriCorps services? 

• What about the program model and AmeriCorps 

contributed to or hindered these results? 
School Turnaround AmeriCorps 

Findings Findings: How was the School Turnaround AmeriCprogram implemented in schools? • Overall, AmeriCorps members and program part

implemented program interventions as intended. 

• Program start-up and grant administration challe

typically associated with launching a new prograbecame less prevalent. 

• Other common challenges to the AmeriCorps pro

model persisted, including member recruitment, retention, and role definition. 
School Turnaround AmeriCorps 

Findings 

Findings: How did stakeholders perceive the impact AmeriCorps services? 
School Turnaround AmeriCorps 

Findings

Findings: What about the program model and AmeriCservice contributed to or hindered these results? • The quality of communication and relationship bui

school stakeholders 

• The quality of member training and consistency of • The quality of the school-based service delivery 
PREPARATION FOR IMPACT STUDY 

 Having strong performance measures th

can be easily collected, have methods in place of following up with participantsmonths after intervention 

 Have your AmeriCorps staff undergo

training on evaluation fundamentals
FIVE TAKE-AWAYS FOR AFFORDABLE, 

STRONG AMERICORPS EVALUATION #1 Choose Quasi-Experimental Design!#2 Evaluator does not need a lot of extravagant credentials. 
FIVE TAKE-AWAYS FOR AFFORDABLE, STRON

AMERICORPS EVALUATION 

#3 Two Year Eval Process Year 1: study designed and tested Year 2: study carried out, with leadership from returning members
FIVE TAKE-AWAYS FOR AFFORDABLE, STRON

AMERICORPS EVALUATION 

#4 Have Members do Data Collection. Fina Comparison Group that already collectsthe data you need #5 Buy-in from members from beginning service year, not something thrown on themembers mid-year 
School Turnaround AmeriCorps 

Key Learnings - CNCS

• Planning year allowed the evaluation to be redes

order to provide meaningful information.

• Clarified member roles and rolled out new expecta

update written partnership agreement annually weincluded in 2016 NOFO.

• Portfolio calls were held on year 1 evaluation fin

discussion also held at Symposium. 

• Clarifying GPR instructions to grantees, includin

expectations for when goals are exceeded and the report on School Turnaround AmeriCorps programdiscretely. 
School Turnaround AmeriCorps 

Key Learnings – City Year 

• Reduce data burden for members and standardiz

school trainings 

• Continue investing in on-site staff coordinator per What learnings did City Year take away from being the School Turnaround evaluation, and what changwe made as a result? • Focus on youth development training for our mem
School Turnaround AmeriCorps 

Key Opportunities – City Year

What opportunities did participation in the School Tuevaluation represent for City Year? • Learning from and share with the STA community • Using findings to validate and promote our progra
School Turnaround AmeriCorps 

Evaluation Planning 

• Strategies for managing cost • Tips for hiring and managing an external evaluato• Using STA evaluation as part of body of evidence