**AmeriCorps State and National Evaluation Plan**

Organization Name: All Star Foundation

Program Name: All Star Youth (ASY) AmeriCorps Program

Application ID: 22ND123456

# 1. Introductory Sections and Program Description

## 1.1 Theory of Change

While high school graduation rates have steadily risen over the last decade, the graduation rates among students of color, low-income, English Learners (EL), homeless students, and students with disabilities remain unacceptably low. Black and Hispanic students are still graduating about 6.3 and 4.3 percentage points behind the national average, respectively (National Center for Education Statistics, 2019). Our theory of change is that providing low-income and underrepresented elementary and middle school students with sequential programming that includes academic enrichment, 1:1 and group tutoring and homework help, mentoring, academic advising, and social/cultural events will help them develop academically, socially, and personally, and will increase their liksselihood of completing high school and applying and attending post-secondary institutions. To deliver these interventions, 50 part-time AmeriCorps members assist after-school-program teachers in providing out-of-school time supports to students attending after-school programs at 15 elementary and middle schools that serve majority low-income children (85% of the students receive free or reduced-cost meals). In addition to academic support activities (tutoring, homework assistance, etc.), members are also trained to teach the program’s All-Star Youth after school curriculum, a 10-week course designed for students ages 6 to 14 that uses structured games and activities to teach concepts such as cause and effect, how to control reactive behaviors, the value of sharing with others, and the importance of making an effort. Through its curriculum, the program aims to increase prosocial behaviors and reduce antisocial and problem behaviors.  

## 1.2 Scope of the Evaluation

This evaluation will focus on measuring the impact of the ASY AmeriCorps Program on the targeted short-term outcomes (behavioral and attitudinal) for elementary and middle school students who participate in the program during the 2022-2023 school year. 

# 1. Evaluation Outcome(s) of Interest

The outcomes of interest for the evaluation are the program’s expected short-term outcomes that consist of both behavioral and attitudinal changes, specifically: 

· Increase in prosocial behaviors 

· Increase in self-control behaviors

· Fewer reported problem behaviors

# 1. Research Question(s)

This evaluation seeks to answer the following questions:

1) Do students who participate in the ASY AmeriCorps program demonstrate a greater increase in prosocial behaviors than a similar group of peers who did not participate in the program? 

2) Do students who participate in the ASY AmeriCorps program demonstrate a greater increase in self-control behaviors than a similar group of peers who did not participate in the program? 

3) Do students who participate in the ASY AmeriCorps program demonstrate fewer problem behaviors than a similar group of peers who did not participate in the program?

# 1. Evaluation Design

## 4.1 Evaluation type

**To address our research questions, we will employ a quasi-experimental design in which propensity score matching is used to create a comparison group composed of similar students enrolled at the same school sites served by ASY AmeriCorps members. 

## 4.2 Comparison Group Formation

ASY AmeriCorps currently partners with schools where the number of students who are eligible for support from an ASY member is more than twice as large as the total number of students who could be served within a single school year. At the conclusion of the school year (2022-23), the evaluator will use **propensity score matching** to identify a pool of students within the broader population of a given school who are statistically similar to students served by the program. 

The matching process involves two analytic steps. First, a logistic regression model will be used to calculate each students’ propensity for treatment assignment (i.e., participated in ASY AmeriCorps or not). The model will include relevant covariates, such as students’ race/ethnicity, gender, grade level, EL status, academic achievement, and pre-test scores on the outcome measure. In the second analytic step, the evaluator will match cases according to their propensity scores, where each student participating in ASY AmeriCorps will be matched with a student in their same school who did not participate in the program, but had a similar propensity for receiving ASY services. The evaluator will use nearest neighbor matching without replacement to pair cases based on their likelihood of participating in the ASY AmeriCorps program. Comparison cases not matched to a treatment case will be excluded from further analysis. Propensity score matching will be limited to the students who have a complete record of data needed for the study (i.e., pre-post outcome data and data on identified covariates).

# 1. Sampling Methods

## 5.1 Sample Selection

For our school sample, we will prioritize including the schools that agree to have teachers administer our chosen assessment tool, the Behavior Assessment System for Children, Third Edition (BASC-3), to all students in the school at two time points (fall and spring) during the 2023-2024 school year. Schools that participate in the study will also agree to provide de-identified student level data that includes both demographic, academic, and behavioral data elements. Our program already has data-sharing MOUs in place with most of our school partners and we have strong relationships with the administrators at each of our partner schools, thus we do not anticipate any challenges in recruiting the minimum number of schools (n=6) needed for our study. However, if necessary, we will consider offering incentives (e.g., gift cards for teachers) to encourage their participation.  

For our student sample, the treatment group will consist of students who participate in the ASY program for a minimum of four months during the 2023-24 school year and attend one of the six partner schools that agree to participate in the study. The comparison group sample will consist of the students who did not participate in the program and had a similar propensity for receiving ASY services as another student in their same school. 

## 5.2 *Sample Size Justification*

Each ASY AmeriCorps member is assigned a caseload of approximately 30 students that they work with during out-of-school time sessions throughout the year, and all school sites have multiple AmeriCorps members (average of 3 members per site). Thus, ASY estimates serving a total of 1,350 students each year, an average of 90 students per school site. ASY AmeriCorps currently partners with schools where the number of students who are eligible for support from an ASY member is more than twice as large as the total number of students who could be served within the school year. Given this, the potential size of the comparison group pool exceeds 1,350 students. 

To determine the minimum sample sizes needed for our study to detect an expected effect size of .20[footnoteRef:1] on our selected outcomes of interest, we conducted a power analysis using the following assumptions:  [1:  Based on prior pre/post outcomes assessment data of ASY participants using the Behavior Assessment System for Children, Third Edition (BASC-3). ] 

· Expected effect size: .20

· Power level: .80

· Significance level (two-tailed test): .05

· ICC: .10

· ANOVA / multiple regression model

· Two study groups program (T) and comparison (C)

· 180 eligible students per school (90 T and 90 C)

· Pre-test used, which is highly correlated with the post-test (.80)

· At least one covariate would be used at both the school (e.g., size, type, FRPL rate, etc.) and student level (e.g., race, gender, income, etc.) in the analysis model to explain variation in the data

Based on these assumptions, the power analysis results indicate that our study requires 6 schools of approximately 180 students, which is 1,080 students total (540 in the treatment group and 540 in the comparison group). Given the estimated number of ASY participants per year and the size of the comparison group pool, our study will be able to meet the required minimum sample size needed to detect an effect of participation in the ASY AmeriCorps Program. 

# 1. Data Collection Procedures, Data Sources, and Measurement Tools

To measure the outcomes of interest, we will use a common published instrument, the Behavior Assessment System for Children, Third Edition (BASC-3) Teacher Rating Scales (TRS). The BASC–3 Teacher Rating Scales (TRS) is a comprehensive measure of both adaptive and problem behaviors in the school setting and directly aligns with our outcomes of interest. It is designed for use by teachers or others who fill a similar role, such as teacher assistants or preschool caregivers. The TRS has three forms, with items targeted at three age levels: preschool (ages 2 through 5), child (ages 6 through 11), and adolescent (ages 12 through 21). Given the age range of the students served by our program, the child and adolescent forms will be used for this study, depending on the age of the subject child. The forms contain descriptors of behaviors that the respondent rates on a four-point scale of frequency, ranging from Never to Almost always.  The TRS takes 10 to 15 minutes to complete for teachers with experience completing rating scales. 

As noted above, the schools that participate in the study will agree to have classroom teachers administer the assessments to all students in their home room classroom at two time points – early fall 2023 for the pre-test and late spring 2024 for the post-test. All item responses will be entered by hand into the BASC-3 ASSISTTM software, and routine quality checks will be conducted. All scoring will be computed by the BASC-3 ASSIST software.

Through a data-sharing agreement with each participating school, our evaluator will have access to de-identified student-level administrative data, including demographic data, achievement data, and behavioral data (i.e., the BASC assessment scores). 

# 1. Analysis Plan

The external evaluator will first conduct a set of baseline analyses, comparing the treatment and comparison groups to ensure that they are similar on average in terms of students’ race/ethnicity, gender, grade level, ELL status, and academic achievement. Baseline levels of short-term outcomes will also be examined to ensure that the comparison group is similar across these outcomes at baseline relative to the treatment group. Two-sample t-tests and chi-square tests (where appropriate) will be utilized to verify that the treatment and comparison groups are comparable. Variables where treatment and comparison groups differ significantly will be included as covariates in the analytic analyses.

To answer each of our research questions, we will use multiple regression to assess the impact of the intervention by estimating the difference between the baseline and follow-up short-term outcomes among the comparison group relative to the treatment group. The regression model will include the following variables: (1) time (baseline and follow-up) and (2) the group (treatment vs. comparison group), and (3) an interaction term between time and group. The impact will be estimated through the coefficient of the interaction term. Important covariates that are noted in the preliminary analyses also will be included in the model. Models will be run separately for each of the short-term outcomes.

If sample size permits, additional subgroup analyses may be used to assess whether students equally benefited from their participation in the ASY Program or whether there are important differences based on socio-demographic background. To help ensure that the results of the regression models are the result of the ASY Program as opposed to unique aspects of each school and program site, sites will be controlled for in regression models.

# 1. Evaluator Qualifications

ASY AmeriCorps has established a relationship with faculty in the Education Department at the State University. Faculty in the department have been engaged in a wide range of quantitative evaluation projects that involve analyzing data to demonstrate evidence of impact, inform continuous improvement, and compliance. We plan to hire an assistant professor and graduate student to serve as evaluators for our study. The evaluators will work with us to finalize the evaluation design, analyze the collected data, and write the final evaluation report. 

# 1. Timeline

# 1. Budget

We have budgeted a total of $70,000 for external evaluation services, which is approximately 10% of the ASY AmeriCorps budget per year.