**AmeriCorps State and National Evaluation Plan**

****Organization Name: ABC Institute 

Program Name: ABC Youth Corps (AYC) Program 

# 1. Introductory Sections and Program Description

## 1.1 Theory of Change

AYC is designed to enhance the developmental assets of young people and reduce the likelihood of participation in risky behaviors. The program is supported by existing research which suggests that interventions, especially the introduction of caring adults into a youth’s life, can increase protective factors and mitigate risk factors that may reduce the likelihood of participation in negative behaviors, such as gang participation and substance use. In addition, there is existing research that supports the use of afterschool programming to enhance youth’s positive assets, including career and educational aspirations and development, positive mental and emotional health, and substance-use prevention. Interventions also can reduce the negative effects of adverse childhood experiences (ACEs) and the likelihood that adolescents will engage in unhealthy behaviors in their day-to-day lives. The AYC program’s Theory of Change is that youth engaged in AmeriCorps-supported, program activities (academic and career support, community service, mentoring by caring adults, and participation in positive peer groups), are able to: 1) build developmental assets, including commitment to learning, positive values, social competencies, and positive identity; 2) enhance their self-efficacy and other critical protective factors; and 3) reduce their participation in risky behaviors, such as gang involvement, substance abuse, and other unhealthy behaviors.

## 1.2 Scope of the Evaluation

The goal of this evaluation is to establish further supporting evidence of the AYC program’s theory of change. The evaluation will show that at-risk youth participating in the program’s activities provided by AmeriCorps members will experience improvements in their academic engagement and performance, education and career goals, and self-efficacy, while also reducing their participation in negative behaviors, such as violence and substance abuse.

# 2. Evaluation Outcome(s) of Interest

In this evaluation, the AYC wants to demonstrate that youth participants in our program experience improvements (increases in positive behaviors and reductions in negative behaviors) in specific outcomes, including:  school attendance; academic performance; educational and career aspirations; self-efficacy; school behavioral incidents; and substance use.  

# 3. Research Question(s)

Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in school attendance and performance? Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in educational and career aspirations?Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in self-efficacy? Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience reductions in the number of school behavioral incidents and self-reported substance use?

# 4. Evaluation Design

## 4.1 Evaluation type

The AYC will employ a non-experimental outcome evaluation. We are utilizing this design in order to collect further evidence that supports our program’s theory of change that youth who receive one-on-one academic and career support, community service, and mentoring from AmeriCorps members through the AYC program on average will experience improvements in important protective factors and positive outcomes and also will experience a reduction in their participation in negative behaviors. 

# 5. Sampling Methods

## 5.1 Sample Selection

We estimate up to 12 sites of AYC’s 20 current sites will be selected for this evaluation. An even number of program sites will be selected from each of two types of program settings, school-based and community-based. Within each of these program settings, we will select sites that represent a broad range of the demographics of youth participant groups served, including by race, gender, and socio-economic status. We will attempt to survey all of the AYC participants at the selected AYC sites.

## 5.2 *Sample Size Justification*

AYC is anticipated to serve approximately 600 youth across its programming in 2022-2023 across the 20 current sites. The number of youth served ranges between 25 and 35 youth per site. Therefore, we expect the 12 selected sites to provide at least 50% of the program participant population (300 or more youth participants). We plan to survey all of the AYC participants at a selected sample of sites. We will limit the site selection to our high school-age programs (serving ages 14-18), which represent a large majority of our program participants (88%). Each site is implementing the same curriculum with older youth, and AmeriCorps members receive the same core program training prior to assignment to sites. In addition, a recent process evaluation of the AYC program confirmed similar implementation and participant experiences across sites (CFR, 2020). Therefore, we feel confident that consistent programming will be occuring across the selected program sites. To ensure that we have representation from various types of participant groups, we will select up to 12 sites that provide representation from important participant groups by race, gender, and socio-economic status in order to gain a full understanding of the participant experience across our program sites. Each of our programs collect basic demographic information on our youth participants, so we will examine the diversity of youth within sites to ensure we include participants that best represent the diversity of youth in all of our programming. 

# 6. Data Collection Procedures, Data Sources, and Measurement Tools

This study will utilize a Pre- and Post- Survey to measure changes in outcomes among the youth participants at selected program sites.The first (pre) survey will be administered at the beginning of the school year and the second (post) survey will be administered nine months later at the end of the school year. Both surveys will be designed to take between 10 and 20 minutes to complete. The program evaluators will develop data collection procedures and measurement tools consistent with the language and literacy levels of our youth participants (ages 14-18). Both surveys will be administered on-site to the participant group via paper and pencil during “homework time” at the sites. Homework time is 90 to 120 minutes each day, allowing sufficient time to complete a short survey. Surveys will be distributed and collected by AmeriCorps members and entered into an Access database by the site supervisors. A script and a frequently asked questions document will be drafted for use by members when administering the survey to youth. A one hour training will be provided with staff and members at each site to ensure that similar procedures are used for administering the surveys and that the data entry is consistently completed. We anticipate an 80% response rate or higher to both the pre and post surveys since these will be administered in-person at the program sites. The Pre-Survey will consist of approximately 20 questions, and will include both open- and closed-ended question types. This survey will be administered in September 2022 and will serve as a baseline measure of opinions, beliefs, and self-reported behaviors in a number of domains, including: school attendance; academic performance; educational and career aspirations; dimensions of self-efficacy; school behavior; and substance use. Under each of these domains, respondents will rate themselves on various items using a 5-point Likert scale. The Pre-Survey will also use open-ended questions to collect qualitative data on youth’s judgments of self-capability, likely reactions to different situational circumstances, and descriptive characteristics about the youth respondent. The Post-Survey will serve as an endpoint measure of similar outcomes and will largely mirror the Pre-Survey. The Post-Survey will also include open-ended questions about youth’s program experiences and how it has contributed to the potential achievement of their personal goals.The surveys will include quantitative and qualitative measures of participant experiences to understand each of the research questions. School attendance, academic performance, and educational and career aspirations (RQ1) Self-efficacy (RQ2) School behavioral incidents and substance use (RQ3)If possible, we will work with the local schools at each site to obtain de-identified data on student attendance, academic performance, and school behavioral incidents. Whether community- or school-based, our programs have close relationships with the local schools, and we have obtained these types of data from schools in the past for performance measurement purposes. For this reason, we are fairly confident that we can obtain these data from the schools. In the past, once we were able to obtain permission from the local school district’s Human Subjects Review Board, we have been able to provide a list of student participants’ names and grades, and then the schools have provided de-identified data in response for our analysis. We hope to be able to enhance our analysis for RQ1 and RQ3 with an analysis of these administrative data. 

# 7. Analysis Plan

The program evaluator will conduct a detailed descriptive summary and interpretation of survey findings and school administrative data (if available) for each of the outcomes associated with the three research questions. Descriptive statistics will be calculated (e.g., frequencies, means, median, etc.) for each survey question both on the pre-survey and post-survey. The administrative data variables will be treated similarly by calculating descriptive statistics. Survey questions and administrative data variables will be grouped by the major outcome area that they address (e.g., school attendance, school performance, etc.) and qualitatively compared pre- and post-program to provide a broad picture of participant change, growth, and experiences for each of the major outcome areas. When there are two sources for the same outcome (self-reported via survey and administrative data from the school district) both findings will be reported and similarities or differences will be discussed and clarified. If appropriate for some questions, we anticipating using the one-sample Student's t-test to assess statistical significance. Open-ended responses focused on youth judgement and reactions and also post-survey comments on program experiences and personal goals will be analyzed for common themes and interpreted to inform the development and improvement of program practices and training for AmeriCorps members.The evaluator will provide a final report with technical appendices to the grantee for review at the end of 2023 (See timeline below) prior to the submission of the grantee’s next application for AmeriCorps funding. The evaluator will make edits based on any comments or questions received from the grantee and/or other program stakeholders. A final report will then be provided. 

# 8. Evaluator Qualifications

The Center for Research (CFR) is a nonprofit, research organization that addresses social and economic problems occurring in communities. Based in the College of Sciences at the University of America, CFR’s research is divided into five main areas: Community and Economic Development, Community Data, Human Services and Education, Program Evaluation, and Agriculture. CFR offers its clients professional assistance with: community planning, focus groups, outreach and capacity building, community data, survey research, and program evaluation.  

# 9. Timeline

Click or tap here to enter text.

# 10.  Budget

The budget will cover all costs of evaluation services by the Center for Research, including:Final evaluation designCoordination with school district, including Human Subjects Review Board submission and approvalStaff and member training for data collectionDevelopment of pre- and post-program survey instrumentData collection and analysisTechnical report writing, submission and presentation to AYCThis is anticipated to cost $60,000.  

Page 2 of 2