AmeriCorps Office of Research and Evaluation

February 2021

Impact Evaluation of AmeriCorps 

National Civilian Community Program 

(NCCC)
We’ll get started in just a 

couple of minutes.

Please be advised that 

there is no dial-in for this 

webinar; all audio is 

provided directly through 

the Adobe Connect 

platform.

Welcome!

2

Impact Evaluation of AmeriCorps NCCC
Introductory Remarks

Impact Evaluation of AmeriCorps NCCC

3

Dr. Melissa Gouge

Research and Evaluation Manager

Office of Research and Evaluation/ 

AmeriCorps NCCC

Jacob Sgambati

Acting Deputy Director

AmeriCorps NCCC
Dr. Annie Georges

Senior Research Associate

JBS International

Speaker

Impact Evaluation of AmeriCorps NCCC

4
Evaluation Objectives

Identify how participation in NCCC promotes leadership qualities and skills among its Members and strengthens the 

communities in which NCCC Members serve.

5

Leadership Development 

• Determine how participation in NCCC promotes 

leadership qualities and skills (professional skills, life skills, 

teamwork, and civic engagement)

Strengthening Communities

• Understand how NCCC strengthens 

communities where Corps Members serve

• investigate and define what strengthening 

communities means in the context of NCCC

Member Retention

• Understand factors that promote retention 

and minimize member attrition
Leadership Development and Member 

Retention

6

Components of 

Evaluation

Design

Design Overview

Mixed methods quasi-experimental design; 

multiple data collection time points

Data Sources

Surveys, interviews, focus groups, administrative data 

on Members’ service projects 

Survey Participants

Current NCCC Members and Team Leaders, Members who 

leave early, accepted applicants who decline to serve

Interview and Focus Group Participants 

NCCC Members, Team Leaders and staff, sponsors and 

community stakeholders
Data Collection Timeline

Leadership Development and Member Retention

7
8

Gain professional skills

Able to work as a team

Engage in civic life

Participants successfully complete their trainings and service projects and receive the support and 

mentoring from staff, team leaders and sponsors, then they can improve on four core outcomes

Develop life skills

Theory of Change - Leadership
9

Short Term Leadership Outcomes

Life Skills

•Plan, prepare,

execute, and

assess an

endeavor

•Attitude of

action or change

orientation to the

world

•Positive selforientation, selfefficacy

Professional 

Skills

•Hard & soft skills

to succeed in

workplace

•Continuing

education

•Education &

service awards

Upon completion of NCCC service

Work as a Team

•Effective

communication

•Awareness,

attitude,

knowledge, and

skills in dealing

with people

different from

oneself

•Employ positive

team behavior

Civic 

Engagement

•Knowledge

related to civic

participation

•Civic orientation

and civic

attitudes,

knowledge of

civic

opportunities and

outlets.
10

Medium Term Leadership Outcomes

Life Skills

•Belief in positive 

self-orientation 

and sense of selfefficacy

Professional 

Skills

•Career choices, 

affirmed career 

interests

• Complete 

education 

requirements

•Usage of 

education awards

One year after completion of NCCC service

Work as a 

Team

•Demonstrate 

positive team 

behavior

•Ability in dealing 

with people 

different from 

oneself

Civic 

Engagement

•Belief in the 

ability to affect 

change through 

civic action

•Belief in civic 

participation and 

civic orientation 

and attitudes
11

Time point 1, 2 and 3

(Likert scale questions) Time point 2

(open ended responses)

• Communicate ideas to other people, 

listen to other people’s opinions, 

collaboration, resolve conflicts thru 

discussion & dialog

• Courteous & respectful, 

understanding other people’s ideas 

& opinions, encourage different 

point of views, encourage 

participation

• Finding solutions to unexpected 

problems, forming relationships with 

groups different from oneself. 

• Importance of serving with 

Members from diverse 

backgrounds

• Personal definition of diversity

• Recommendations to attain a 

diverse pool of NCCC Members

Cultural Competency Measures
12

12

• Survey is clear, relevant and easily 

completed on mobile devices and 

computers.

• Certain questions are redundant or 

repetitive and should be deleted.

• Format and design of surveys are 

easy to follow.

• Additional questions are needed to 

capture socio-economic status and 

mental health.

• Neutral response options are needed 

for certain questions.

Data Collection Progress

Leadership Development Study

Note: The distribution of participants is preliminary. A small minority of participants might transfer to another class or region. We review 

and reclassify these cases as appropriate. 

Treatment Comparison

Region/Class

Corps 

Members 

Roster

Completed & 

Partial with 

sufficient 

information

Partial 

incomplete 

information Refusal

Completion 

rate Contacted

Completed & 

Partial with 

sufficient 

information

Partial 

incomplete 

information Refusal

Completion 

rate

NC Winter CL 26A 174 114 7 8 70% 171 87 10 18 57%

SN Winter FEMA CL 26A 166 132 10 6 86% 235 116 7 5 52%

SW Summer CL 26B 91 61 4 2 71% 64 32 3 3 55%

PC Summer CL 26B 132 95 3 6 74% 296 175 2 5 60%

SN Summer CL 26B 160 117 9 2 79% 134 69 4 6 54%

NC Summer CL 26B 81 47 5 1 64% 76 40 2 2 55%

PC Fall CL 27A 229 178 8 10 81% 154 99 5 7 68%

SW Fall CL 27A 243 187 11 10 81% 179 115 5 11 67%

NC Winter CL 27A (in progress) 268 175 14 4 71% 152 81 0 0 53%

SN Winter FEMA CL 27A (in 

progress) 155 30 0 0 19% 87 28 0 0 32%

Total Participants 1699 1136 71 49 71% 1548 842 38 57 57%
13

Data Collection Progress

Member Retention Study

Note: The distribution of participants is preliminary. A small minority of early exit participants might join 

another class following their exit. We review and reclassify these cases as appropriate. 

Early Exit Contacted

Completed & Partial 

with sufficient 

information

Partial incomplete 

information Refusal Completion rate

NC Winter CL 26A 55 27 0 0 49%

SN Winter FEMA CL 26A 41 18 0 1 44%

SW Summer CL 26B 17 10 0 1 59%

PC Summer CL 26B 27 16 0 1 59%

SN Summer CL 26B 7 2 0 0 29%

NC Summer CL 26B 9 5 0 0 56%

PC Fall CL 27A 10 7 0 0 70%

SW Fall CL 27A 23 3 0 1 13%

NC Winter CL 27A (in progress)

SN Winter FEMA CL 27A (not yet started)

Total Early Exit 189 88 0 4 47%
14

Change in Participants Incentives

• Increase incentive to $40 for comparison 

participants.

• Provide incentive to participants for completing a 

short survey upon early exit from NCCC.

• Provide incentive to de-activated Corps Members to 

participate in a follow-up phone interview after their 

term of service.

• Provide incentive to early exit to participate in a 

follow-up phone interview.

• .
15

Change in Incentives and the Completion Rate

• Comparison participants leadership development study

 Prior to change, over a period of 7-8 weeks, completion rate at 43% 

(NC winter 2020) and 26% (SN FEMA winter 2020)

 After change, over a period of 3 weeks, 1 follow-up email or phone 

contact - 20% (NC winter 2020). Over a period of 7-8 weeks, 4 followup emails or phone contacts – 32% (SN FEMA winter 2020)

 NC winter 2021 over a period of 6 weeks – 53%

• Early exit member retention study

 Prior to change, over a period of 5 to 6 months, completion rate at 

43%

 After change, over a period of 3 months, completion rate at 55%
16

Leadership Development Study

Challenges encountered and solutions implemented
17

External Shock Due to Pandemic

Challenge

• Pose threat to the study’s internal 

validity. 

• Participants may not be impacted 

equally.

Solution

• Added survey questions to 

gather information from 

participants.

• Modify focus groups and 

interview protocols.

• Statistical analysis and 

adjustments to account for 

external shock.
18

Tracking and Monitoring

Track and monitor all participants so that each participant is counted 

once through the entire study.

Since this is a national longitudinal study across four regions covering 

multiple NCCC classes, individual’s status may change from being in the 

comparison group to accepting and serving as a Corps Member.

Some Corps Members may also exit one class and enroll in a 

subsequent class in a different region. For example, a Corps Member 

exits the Winter Class and enrolls in the next Summer Class in a 

different region.
19

Strengthening Communities Study
20

Strengthening Communities

20

Components of 

Study Design

Exploratory Study Design Overview

Exploratory mixed methods consisting of review of

documentation from completed and current projects, 

case studies of selected current projects

Primary Data Collection

Service projects database, site visits with interviews, 

focus groups, phone interviews, and participatory 

research methods 

Case Studies

A selected number of completed and current projects will be the subjects of indepth case studies, including review of documents, site visits, focus group and 

interviews 

Administrative Data Review 

Quantitative overview of project outputs, analysis of project information, 

recommendations for data collection instruments
21

Primary Data Collection

• Site visits, ethnographic techniques, review of NCCC service 

projects database, service projects documentation, and multiple 

interviews and focus groups.

• Conduct remote interviews, review of NCCC service 

projects database, service projects documentation. 

Restricted case studies

In-depth comprehensive case studies 
22

Primary Data Collection Timeline 

22

In-depth Case Studies of Current Projects - Interviews and Focus Groups

Time point 1

• While project is active

Time point 2

• A year after project ends 

Restricted Case Studies of Completed and Current Projects - Interviews

Time point 1

• Current and completed 

projects 
23

Document Outputs and Outcomes 

of NCCC Service Projects

23

 Outputs: Increased number of volunteers; increased 

partnerships between organizations; improved physical 

infrastructure for community-serving agencies

 Short-term Outcomes: Increased volunteer engagement; more 

help to establish connections, or increase connections, 

between community-serving organizations; greater ability to 

serve more beneficiaries

• Intermediate Outcomes: Increased efficiency and effectiveness 

of community-led volunteer efforts; increased ability of 

community serving organizations to leverage community 

assets; increased capacity of community and government 

agencies to serve community members, including ability to 

serve greater numbers of clients, to serve the needs of existing 

clients more holistically, and/or to deliver existing services 

more efficiently and effectively

• Long-term Outcomes: Increased organizational and 

interorganizational capacity to serve community needs; 

increased community member health and well-being (e.g., due 

to ability to use trails, reduced exposure to mold).

Given the short time frame for the 

proposed data collection, 

intermediate and long-term 

outcomes will not be measured, but 

can be explored prospectively with 

key stakeholders
24

Service Projects Database

24

• Includes all service projects completed since 2012

• Captures key data about NCCC service projects 

such as project issue area, characteristics, sponsor 

organization information, and accomplishments 

and goals

• Service projects goals and outcomes are 

thematically coded, and key outcomes are 

systematically categorized. The thematic 

classification of service projects goals and 

outcomes reveal service projects’ community and 

member impacts

• SPD will be used in this evaluation to:

– Identify service projects for case studies

– Examine association between the service 

projects experiences, leadership skills and 

retention
25

Strengthening Communities Study

Potential Challenges and Planned Solutions
26

External Shock Due to Pandemic

Challenge

• Potential shift in types and 

characteristics of service projects.

• Potential shift in community 

stakeholders served.

Solution

• Modify focus groups and 

interview protocols.

• Added survey questions to gather 

information from Corps Members 

and Team Leaders.
27

Ongoing Data Collection Activities 

and Next Steps
Survey 

Administration

• NC Winter Class 27A 

& SN FEMA Winter 

Class 27A (baseline)

• First Follow-up Survey

• SN FEMA Winter 

Class 26A

• Planning for SW, 

SN, NC, & PC 

summer Class 26B

Focus Group & 

Interview

• SN FEMA Winter Class 

26A

• Planning for SW, SN, 

NC, & PC Summer 

Classes 26B

• Planning for PC & SW 

Fall Classes 27A

Case Studies

• Analyze the service 

projects database

• Select service 

projects for case 

studies

• Retrieve, review and 

code service projects 

documents

• Conduct interviews

Ongoing Data Collection Activities

28
• Baseline survey for the leadership development and retention studies

• Begin data collection for the strengthening communities study - Select 12 

service projects for case studies, review service projects documents, conduct 

interviews

• Timeline

• July 2021 - Preliminary results based on baseline survey for the leadership 

development study 

• September 2021 – Preliminary results of the analysis of the 12 service projects

• September 2021 - Office of Research and Evaluation webinar presentation on 

the preliminary results for the leadership development study

Next Steps

29
Next Steps 

Leadership Development Study

Research Questions, Data Sources and Sample, analysis

15
31

Research Questions Data sources/Sample Description of analysis Analysis

1. What is the profile of NCCC 

members, including their demographic 

characteristics, sources to learn about 

NCCC, and motivations for serving? How 

do members perceive the NCCC 

experience prior to the start of their 

service? How do Traditional Corps and 

FEMA Corps differ?

Baseline survey / sample of members. Examine members' 

background characteristics, 

incoming leadership skills, 

motivation, sources where and 

how they learn of NCCC, 

knowledge and perception of 

serving with NCCC (the degree 

of mismatch in how they 

perceive the service 

experience).

Descriptive analysis, chi-square test, 

t-test; latent class analysis (LCA) to 

identify types of members base on 

motivation, sources where 

members learn about NCCC.

2. How do NCCC members differ in 

interpersonal skills, life skills, including 

attitudes and behavior in group settings 

prior to the start of service compare to 

youth of similar background who do not 

serve with NCCC? How do Traditional 

Corps and FEMA Corps differ?

Baseline survey / two groups: 

members and accepted applicants 

who do not serve (comparison group)

Examine how members differ 

from comparison group in 

background, interpersonal 

skills, life skills; establish 

baseline equivalence between 

the two groups using 

propensity score methods to 

construct matched comparison 

cases. 

Descriptive analysis, t-test or nonparametric test (e.g., Wilcoxon 

signed-rank test) to assess 

differences in observed 

characteristics; propensity score 

methods.

Base Year Analysis
Next Steps 

Strengthening Communities

Research Questions, Data Sources and Sample, analysis

15
33

33

• Survey is clear, relevant and easily 

completed on mobile devices and 

computers.

• Certain questions are redundant or

repetitive and should be deleted.

• Format and design of surveys are

easy to follow.

• Additional questions are needed to

capture socio-economic status and

mental health.

• Neutral response options are needed

for certain questions.

Research Questions, Data Sources and Analysis

Research Questions Data sources / Sample Description of analysis Analysis

1. How do NCCC projects strengthen communities?

How is success in strengthening communities

defined by different stakeholders? How do

community members perceive Traditional NCCC and

FEMA Corps? How do NCCC members affect

sponsoring and partnering organizations ability to

serve and strengthen communities?

2. What are some lessons learned to strengthen

communities? What do members, sponsors and site

supervisors, and community stakeholders believe

are the most important factors influencing the

ability for NCCC teams to strengthen communities?

What are some of the direct, indirect, unintended,

and long-term effects of NCCC’s presence and

project work on communities? How can these be

assessed, increased, or reduced in the future?

3. Which approaches and projects have been most or

least effective for the goals of strengthening

communities? How do sponsoring and partnering

organizations perceive project success? Are some

project types more likely to lead to community

benefits? What testable hypotheses can be formed

to guide impact research investigating the

relationship between project, sponsor, and

community characteristics and community

strengthening?

Sample of active projects / Review of 

administrative data; comprehensive and 

restricted case studies of sample projects; CNCS 

service projects database

Analysis of service projects database 

to examine characteristics of service 

projects over time; quantitative 

reporting of administrative data will 

be descriptive and will focus on 

project and sponsor characteristics 

and project outputs. Qualitative 

analysis will include individual case 

descriptions, analysis and reporting 

of emergent themes within each case 

(within-case analysis), and thematic 

analysis across cases (cross-case 

analysis). Case studies will also 

include descriptive quantitative data 

relevant to each project using the 

service projects databases. 

Content and thematic analysis; 

descriptive analysis; bivariate and 

multivariate 
Closing Remarks

Impact Evaluation of AmeriCorps NCCC

34

Dr. Melissa Gouge

Research and Evaluation Manager

Office of Research and Evaluation/ 

AmeriCorps NCCC

Jacob Sgambati

Acting Deputy Director

AmeriCorps NCCC
Questions?

Dr. Melissa Gouge

Research and Evaluation Manager, 

Office of Research and Evaluation/

AmeriCorps NCCC

Email: mgouge@cns.gov

Q&A

Impact Evaluation of AmeriCorps NCCC

35