**AmeriCorps State and National Evaluation Plan**

****Organization Name: ABC Institute 

Program Name: ABC Youth Corps (AYC) Program 

Application ID: 22ED123456

# 1. Introductory Sections and Program Description

## 1.1 Theory of Change	Comment by Megan Edgin (she/her): Describe the nature of your service activities (interventions) and why they are expected to produce the desired outcomes. This section should be short but include enough detail to assess how well the proposed evaluation aligns with your program's logic model and proposed outcomes of interest.

AYC is designed to enhance the developmental assets of young people and reduce the likelihood of participation in risky behaviors. The program is supported by existing research which suggests that interventions, especially the introduction of caring adults into a youth’s life, can increase protective factors and mitigate risk factors that may reduce the likelihood of participation in negative behaviors, such as gang participation and substance use. In addition, there is existing research that supports the use of afterschool programming to enhance youth’s positive assets, including career and educational aspirations and development, positive mental and emotional health, and substance-use prevention. Interventions also can reduce the negative effects of adverse childhood experiences (ACEs) and the likelihood that adolescents will engage in unhealthy behaviors in their day-to-day lives. The AYC program’s Theory of Change is that youth engaged in AmeriCorps-supported, program activities (academic and career support, community service, mentoring by caring adults, and participation in positive peer groups), are able to: 1) build developmental assets, including commitment to learning, positive values, social competencies, and positive identity; 2) enhance their self-efficacy and other critical protective factors; and 3) reduce their participation in risky behaviors, such as gang involvement, substance abuse, and other unhealthy behaviors.

## 1.2 Scope of the Evaluation	Comment by Megan Edgin (she/her): State concisely the goal(s) of the evaluation and specify which service activity/ies will be assessed. AmeriCorps does not require you to evaluate all components of your theory of change; your evaluation may focus on a sub-set of program activities. 

The goal of this evaluation is to establish further supporting evidence of the AYC program’s theory of change. The evaluation will show that at-risk youth participating in the program’s activities provided by AmeriCorps members will experience improvements in their academic engagement and performance, education and career goals, and self-efficacy, while also reducing their participation in negative behaviors, such as violence and substance abuse.

# 2. Evaluation Outcome(s) of Interest	Comment by Megan Edgin (she/her): List the outcome(s) your evaluation will measure. The outcomes identified should align with the outcomes described in the program's logic mode. Outcomes must also align with the theory of change, scope in #1, and must be feasible to measure, based on the source(s) of data needed and level of effort required. Given the length of a single grant cycle, it can be challenging to evaluate long-term program outcomes; evaluating short- and medium-term program outcomes may be more practical. For non-experimental outcome evaluations, outcomes must involve a change in knowledge, attitude, behavior, or condition.  Metrics that measure the amount of service provided (e.g. number of students tutored/volunteers recruited/organizations served) should not be listed as outcomes for this type of evaluation. However, for process or implementation evaluations, these metrics may be appropriate.

In this evaluation, the AYC wants to demonstrate that youth participants in our program experience improvements (increases in positive behaviors and reductions in negative behaviors) in specific outcomes, including:  school attendance; academic performance; educational and career aspirations; self-efficacy; school behavioral incidents; and substance use.  

# 3. Research Question(s)	Comment by Megan Edgin (she/her): List the research question(s) that will guide your evaluation. Research questions must be clearly connected to the outcomes in #2 and aligned with the theory of change and scope in #1. Research questions are phrased in accordance with the research design.For more information on creating good research questions, see the evaluation core curriculum webinar on “Asking the Right Research Questions.” 	Comment by Megan Edgin (she/her): [ASN TO INSERT LINK]

Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in school attendance and performance? Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in educational and career aspirations?Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience improvements in self-efficacy? Do youth who participate in AYC-sponsored, one-on-one academic support and mentoring during the 2022-23 school year experience reductions in the number of school behavioral incidents and self-reported substance use?

# 4. Evaluation Design

## 4.1 Evaluation type	Comment by Megan Edgin (she/her): State and fully describe the type(s) of evaluation design that will be used, and explain why this is the most appropriate design to achieve the evaluation goal(s) in #1 and answer the research question(s) in #3. Possible evaluation designs include but are not limited to:Process or implementation evaluationNon-experimental outcome evaluation

The AYC will employ a non-experimental outcome evaluation. We are utilizing this design in order to collect further evidence that supports our program’s theory of change that youth who receive one-on-one academic and career support, community service, and mentoring from AmeriCorps members through the AYC program on average will experience improvements in important protective factors and positive outcomes and also will experience a reduction in their participation in negative behaviors. 4.2 Comparison Group FormationN/A

# 5. Sampling Methods

## 5.1 Sample Selection	Comment by Megan Edgin (she/her): Describe the population from which the sample will be drawn, the estimated sample sizes for treatment and (if applicable) comparison groups, and how the sample will be selected, including any eligibility criteria for inclusion in the study sample (e.g.., simple random, stratified random, purposive, convenience sampling). Alternatively, the plan clearly states that no sample will be drawn, as applicable (i.e., the entire program population will be included in the study). If the grantee plans to use the full population, an estimate of the number still needs to be provided. Specify any consent procedures (i.e., parental/guardian consent, opt-in/opt-out) or data use agreements that will be necessary to gather or obtain data. For a non-experimental evaluation that will use a comparison group, describe the group that will be used and explain why this selection is appropriate in Section 4.2: Control or Comparison Group Formation (if applicable).

We estimate up to 12 sites of AYC’s 20 current sites will be selected for this evaluation. An even number of program sites will be selected from each of two types of program settings, school-based and community-based. Within each of these program settings, we will select sites that represent a broad range of the demographics of youth participant groups served, including by race, gender, and socio-economic status. We will attempt to survey all of the AYC participants at the selected AYC sites.

## 5.2 *Sample Size Justification*	Comment by Megan Edgin (she/her): For non-experimental evaluations, explain the basis for selecting the sample sizes in #5.1 and how the size will be adequate to answer the research questions.

AYC is anticipated to serve approximately 600 youth across its programming in 2022-2023 across the 20 current sites. The number of youth served ranges between 25 and 35 youth per site. Therefore, we expect the 12 selected sites to provide at least 50% of the program participant population (300 or more youth participants). We plan to survey all of the AYC participants at a selected sample of sites. We will limit the site selection to our high school-age programs (serving ages 14-18), which represent a large majority of our program participants (88%). Each site is implementing the same curriculum with older youth, and AmeriCorps members receive the same core program training prior to assignment to sites. In addition, a recent process evaluation of the AYC program confirmed similar implementation and participant experiences across sites (CFR, 2020). Therefore, we feel confident that consistent programming will be occuring across the selected program sites. To ensure that we have representation from various types of participant groups, we will select up to 12 sites that provide representation from important participant groups by race, gender, and socio-economic status in order to gain a full understanding of the participant experience across our program sites. Each of our programs collect basic demographic information on our youth participants, so we will examine the diversity of youth within sites to ensure we include participants that best represent the diversity of youth in all of our programming. 

# 6. Data Collection Procedures, Data Sources, and Measurement Tools	Comment by Megan Edgin (she/her): Describe each data source and measurement tool and the procedures that will be used to collect or extract data, including when, how often, and by what mode (i.e., paper/pencil, phone, or web survey; administrative data extract). Explain how the proposed data sources and tools are adequate for addressing all of the research question(s) and how the data align with the evaluation’s outcome(s) of interest. Either a description of the tools for collecting data or a detailed plan for extracting administrative data is provided.

This study will utilize a Pre- and Post- Survey to measure changes in outcomes among the youth participants at selected program sites.The first (pre) survey will be administered at the beginning of the school year and the second (post) survey will be administered nine months later at the end of the school year. Both surveys will be designed to take between 10 and 20 minutes to complete. The program evaluators will develop data collection procedures and measurement tools consistent with the language and literacy levels of our youth participants (ages 14-18). Both surveys will be administered on-site to the participant group via paper and pencil during “homework time” at the sites. Homework time is 90 to 120 minutes each day, allowing sufficient time to complete a short survey. Surveys will be distributed and collected by AmeriCorps members and entered into an Access database by the site supervisors. A script and a frequently asked questions document will be drafted for use by members when administering the survey to youth. A one hour training will be provided with staff and members at each site to ensure that similar procedures are used for administering the surveys and that the data entry is consistently completed. We anticipate an 80% response rate or higher to both the pre and post surveys since these will be administered in-person at the program sites. The Pre-Survey will consist of approximately 20 questions, and will include both open- and closed-ended question types. This survey will be administered in September 2022 and will serve as a baseline measure of opinions, beliefs, and self-reported behaviors in a number of domains, including: school attendance; academic performance; educational and career aspirations; dimensions of self-efficacy; school behavior; and substance use. Under each of these domains, respondents will rate themselves on various items using a 5-point Likert scale. The Pre-Survey will also use open-ended questions to collect qualitative data on youth’s judgments of self-capability, likely reactions to different situational circumstances, and descriptive characteristics about the youth respondent. The Post-Survey will serve as an endpoint measure of similar outcomes and will largely mirror the Pre-Survey. The Post-Survey will also include open-ended questions about youth’s program experiences and how it has contributed to the potential achievement of their personal goals.The surveys will include quantitative and qualitative measures of participant experiences to understand each of the research questions. School attendance, academic performance, and educational and career aspirations (RQ1) Self-efficacy (RQ2) School behavioral incidents and substance use (RQ3)If possible, we will work with the local schools at each site to obtain de-identified data on student attendance, academic performance, and school behavioral incidents. Whether community- or school-based, our programs have close relationships with the local schools, and we have obtained these types of data from schools in the past for performance measurement purposes. For this reason, we are fairly confident that we can obtain these data from the schools. In the past, once we were able to obtain permission from the local school district’s Human Subjects Review Board, we have been able to provide a list of student participants’ names and grades, and then the schools have provided de-identified data in response for our analysis. We hope to be able to enhance our analysis for RQ1 and RQ3 with an analysis of these administrative data. 

# 7. Analysis Plan	Comment by Megan Edgin (she/her): Describe an analysis that is appropriate for the evaluation’s design and data sources (e.g., statistical testing for quantitative data or descriptive analysis methods for qualitative data). Explain how the analysis will address all of the evaluation’s research questions.Descriptive analyses, correlational analyses, and t-tests/chi-square tests are appropriate for quantitative data used for non-experimental designs. Qualitative analysis methods (e.g., content analysis, thematic analysis) are appropriate for qualitative data.

The program evaluator will conduct a detailed descriptive summary and interpretation of survey findings and school administrative data (if available) for each of the outcomes associated with the three research questions. Descriptive statistics will be calculated (e.g., frequencies, means, median, etc.) for each survey question both on the pre-survey and post-survey. The administrative data variables will be treated similarly by calculating descriptive statistics. Survey questions and administrative data variables will be grouped by the major outcome area that they address (e.g., school attendance, school performance, etc.) and qualitatively compared pre- and post-program to provide a broad picture of participant change, growth, and experiences for each of the major outcome areas. When there are two sources for the same outcome (self-reported via survey and administrative data from the school district) both findings will be reported and similarities or differences will be discussed and clarified. If appropriate for some questions, we anticipating using the one-sample Student's t-test to assess statistical significance. Open-ended responses focused on youth judgement and reactions and also post-survey comments on program experiences and personal goals will be analyzed for common themes and interpreted to inform the development and improvement of program practices and training for AmeriCorps members.The evaluator will provide a final report with technical appendices to the grantee for review at the end of 2023 (See timeline below) prior to the submission of the grantee’s next application for AmeriCorps funding. The evaluator will make edits based on any comments or questions received from the grantee and/or other program stakeholders. A final report will then be provided. 

# 8. Evaluator Qualifications	Comment by Megan Edgin (she/her): Describe how the person(s) who will conduct the evaluation are sufficiently qualified to conduct the proposed evaluation (e.g., have experience and technical qualifications that align with the planned evaluation design). The internal/external evaluator is sufficiently qualified and meets AmeriCorps evaluation requirement for the applicant (large or small). Whether the evaluator is internal or external to the program is clearly stated. If an external evaluator has been identified, the description supports the conclusions that the evaluator is independent from the program and qualified to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts).  If the evaluator is not yet identified or hired, describe the required and/or preferred qualifications for an evaluator. 

The Center for Research (CFR) is a nonprofit, research organization that addresses social and economic problems occurring in communities. Based in the College of Sciences at the University of America, CFR’s research is divided into five main areas: Community and Economic Development, Community Data, Human Services and Education, Program Evaluation, and Agriculture. CFR offers its clients professional assistance with: community planning, focus groups, outreach and capacity building, community data, survey research, and program evaluation.  

# 9. Timeline	Comment by Megan Edgin (she/her): Provide a timeline for all of the major evaluation activities (e.g., finalizing evaluation design, hiring evaluator, developing data collection instruments, collecting pre-intervention data, collecting post-intervention data, analyzing data, writing report). Delineate the timeline by month and year or on a quarterly basis (e.g., fall 2023, spring 2024). The timeline must show how all evaluation activities and a final report will be completed before your next recompete application.AmeriCorps recommends using the first program year for evaluation planning (including gaining final approval of the plan) and data collection instrument development; the second program year for data collection; and the remaining time in the third program year to analyze data and complete the evaluation report. Since grantees have unique programs and recompete application deadlines may vary by state, exact evaluation timelines may vary. 

Click or tap here to enter text.

# 10.  Budget	Comment by Megan Edgin (she/her): Specify the overall budget allotted for the evaluation, including the cost of engaging an external evaluator if applicable. If you will be utilizing staff time for conducting an internal evaluation, provide a description of those in-kind resources.

The budget will cover all costs of evaluation services by the Center for Research, including:Final evaluation designCoordination with school district, including Human Subjects Review Board submission and approvalStaff and member training for data collectionDevelopment of pre- and post-program survey instrumentData collection and analysisTechnical report writing, submission and presentation to AYCThis is anticipated to cost $60,000.  

Page 2 of 2