**AmeriCorps State and National Evaluation Plan**

Organization Name: All Star Foundation

Program Name: All Star Youth (ASY) AmeriCorps Program

Application ID: 22ND123456

# 1. Introductory Sections and Program Description

## 1.1 Theory of Change	Comment by Megan Edgin (she/her): A program’s theory of change is the general underlying idea of how you believe your intervention will create change and why the desired change is expected to come about. Your theory of change articulates the assumptions underlying your choice of activities. A good program evaluation should test some part of your theory of change. You can focus your evaluation on one or more components of your theory of change.

While high school graduation rates have steadily risen over the last decade, the graduation rates among students of color, low-income, English Learners (EL), homeless students, and students with disabilities remain unacceptably low. Black and Hispanic students are still graduating about 6.3 and 4.3 percentage points behind the national average, respectively (National Center for Education Statistics, 2019). Our theory of change is that providing low-income and underrepresented elementary and middle school students with sequential programming that includes academic enrichment, 1:1 and group tutoring and homework help, mentoring, academic advising, and social/cultural events will help them develop academically, socially, and personally, and will increase their likelihood of completing high school and applying and attending post-secondary institutions. To deliver these interventions, 50 part-time AmeriCorps members assist after-school-program teachers in providing out-of-school time supports to students attending after-school programs at 15 elementary and middle schools that serve majority low-income children (85% of the students receive free or reduced-cost meals). In addition to academic support activities (tutoring, homework assistance, etc.), members are also trained to teach the program’s All-Star Youth after school curriculum, a 10-week course designed for students ages 6 to 14 that uses structured games and activities to teach concepts such as cause and effect, how to control reactive behaviors, the value of sharing with others, and the importance of making an effort. Through its curriculum, the program aims to increase prosocial behaviors and reduce antisocial and problem behaviors.  	Comment by Megan Edgin (she/her): Defining the problem your program addresses helps clarify why you are implementing your intervention. Your intervention should logically follow from your problem statement; alignment between your targeted problem and intervention is critical for producing the change you desire. In this example, the problem definition comes before the program background in order to make the connection between the problem and intervention more clear. 	Comment by Megan Edgin (she/her): There is a written description of the program’s theory of change (i.e., how a program intends to achieve its desired outcomes) with enough detail to assess its alignment with the program's logic model and proposed outcomes of interest. 	Comment by Megan Edgin (she/her): A clearly stated, well-defined program description will enable you to make decisions about what components of the program to focus on in the evaluation and to select evaluation objectives that are feasible. It sets a common understanding of what the program is (and is not), what resources and components are utilized, what products are produced or activities conducted, and what is expected to happen as a result of those activities. 	Comment by Megan Edgin (she/her): This statement concisely states the aims of focus for the evaluation. While the intervention aims to help students develop academically, socially, and personally, this sentence provides a focus for the evaluation, clarifying that all outcomes will not be evaluated (such as academic outcomes). 

## 1.2 Scope of the Evaluation	Comment by Megan Edgin (she/her): State concisely the goal(s) of the evaluation and specify which service activity/ies will be assessed. AmeriCorps does not require grantees to evaluate all components of their theory of change; evaluation may focus on a sub-set of program activities.

This evaluation will focus on measuring the impact of the ASY AmeriCorps Program on the targeted short-term outcomes (behavioral and attitudinal) for elementary and middle school students who participate in the program during the 2022-2023 school year. 	Comment by Megan Edgin (she/her): This statement clearly and concisely describes the focus of the impact evaluation, which outcomes from the logic model will be evaluated, which students will be included, and the timeframe of the evaluation.

# 1. Evaluation Outcome(s) of Interest	Comment by Megan Edgin (she/her): The evaluation outcome(s) of interest section specifies the outcome(s) the evaluation will measure; outcomes are aligned with the theory of change and scope of study and are feasible to measure, based on the source(s) of data needed and level of effort required.For impact evaluations, outcomes must involve a change in knowledge, attitude, behavior, or condition.  Metrics that measure the amount of service provided (e.g. number of students tutored/volunteers recruited/organizations served) should not be listed as outcomes for this type of evaluation.

The outcomes of interest for the evaluation are the program’s expected short-term outcomes that consist of both behavioral and attitudinal changes, specifically: 	Comment by Megan Edgin (she/her): This description explicitly states that short-term outcomes will be assessed. Given the length of a single grant cycle, it can be challenging to evaluate long-term program outcomes; evaluating short- and medium-term program outcomes may be more practical. 

· Increase in prosocial behaviors 	Comment by Megan Edgin (she/her): The specific behaviors are noted that will be assessed to understand behavioral and attitudinal change. 

· Increase in self-control behaviors

· Fewer reported problem behaviors

# 1. Research Question(s)	Comment by Megan Edgin (she/her): Written research questions are provided and are clearly connected to measurable outcomes that are aligned with the theory of change. Research questions are phrased in accordance with the research design (e.g., reference a control/comparison group for impact designs). In this example, the program has chosen to focus only on impact questions, or questions that are concerned with the program’s effect on outcomes. Impact evaluation questions seek to examine cause and effect, and focus on attributing the changes in outcomes directly to the intervention. Each research question is aligned to the items noted in the "Outcomes of Interest" section. For help on creating good research questions, see the evaluation core curriculum webinar on “Asking the Right Research Questions.” 	Comment by Megan Edgin (she/her): [ASN TO INSERT LINK]

This evaluation seeks to answer the following questions:

1) Do students who participate in the ASY AmeriCorps program demonstrate a greater increase in prosocial behaviors than a similar group of peers who did not participate in the program? 	Comment by Megan Edgin (she/her): For impact evaluations, research questions must involve a comparison between beneficiaries receiving the intervention or aspect of the intervention being studied (program/treatment group) and those that do not receive the intervention or aspect of the intervention (comparison/control group) and compare data on the outcomes of interest for both groups at two different time points, ideally at baseline (pre-intervention) and follow-up (post-intervention). The research questions here clearly include both groups.

2) Do students who participate in the ASY AmeriCorps program demonstrate a greater increase in self-control behaviors than a similar group of peers who did not participate in the program? 

3) Do students who participate in the ASY AmeriCorps program demonstrate fewer problem behaviors than a similar group of peers who did not participate in the program?

# 1. Evaluation Design	Comment by Megan Edgin (she/her): It is especially important when designing an impact evaluation that the selected evaluation design follow directly from your evaluation goal(s) in #1 and research question(s) in #3. Explain why this is the most appropriate design. For impact evaluations, the evaluation must include a Quasi-experimental design (QED) or Randomized controlled trial (RCT) design. For more information, see the evaluation core curriculum webinar “Overview of Evaluation Designs.” 

## 4.1 Evaluation type	Comment by Megan Edgin (she/her): The evaluation type section is where the design is clearly stated (e.g., experimental, quasi-experimental) and fully described. The design choice aligns with the goals for the evaluation.

**To address our research questions, we will employ a quasi-experimental design in which propensity score matching is used to create a comparison group composed of similar students enrolled at the same school sites served by ASY AmeriCorps members. 

## 4.2 Comparison Group Formation	Comment by Megan Edgin (she/her): Since impact evaluations seek to attribute results directly to the intervention, you need to include a comparison or control group, which helps to rule out other explanations for observed changes in the group that participates in your program. An evaluation that does not utilize a comparison or control group leaves open the possibility of alternative explanations for why change might have occurred. 

ASY AmeriCorps currently partners with schools where the number of students who are eligible for support from an ASY member is more than twice as large as the total number of students who could be served within a single school year. At the conclusion of the school year (2022-23), the evaluator will use **propensity score matching** to identify a pool of students within the broader population of a given school who are statistically similar to students served by the program. 	Comment by Megan Edgin (she/her): For a QED study, describe the approach for identifying a matched comparison group (e.g., propensity score matching, nearest-neighbor matching, etc.). Include the procedures for identifying a pool of similar individuals, organizations, or locations from which to draw a comparison group. Also include a description of the procedures and a list of the variables (covariates) you will use to match treatment and comparison groups, including either a baseline measure of the outcome (e.g., pre-intervention outcome score) or a proxy measure for the outcome of interest (e.g., grade point average to estimate future HS graduation).For an RCT study, describe the eligibility criteria for inclusion in the study and the randomization process, which will result in two or more study groups (i.e., treatment and control). 

The matching process involves two analytic steps. First, a logistic regression model will be used to calculate each students’ propensity for treatment assignment (i.e., participated in ASY AmeriCorps or not). The model will include relevant covariates, such as students’ race/ethnicity, gender, grade level, EL status, academic achievement, and pre-test scores on the outcome measure. In the second analytic step, the evaluator will match cases according to their propensity scores, where each student participating in ASY AmeriCorps will be matched with a student in their same school who did not participate in the program, but had a similar propensity for receiving ASY services. The evaluator will use nearest neighbor matching without replacement to pair cases based on their likelihood of participating in the ASY AmeriCorps program. Comparison cases not matched to a treatment case will be excluded from further analysis. Propensity score matching will be limited to the students who have a complete record of data needed for the study (i.e., pre-post outcome data and data on identified covariates).

# 1. Sampling Methods

## 5.1 Sample Selection	Comment by Megan Edgin (she/her): Describe the population from which the sample will be drawn, the estimated sample sizes for treatment and comparison groups, and how the sample will be selected. Include any eligibility criteria for inclusion in the study sample (e.g.., simple random, stratified random, purposive, convenience sampling). Alternatively, the plan clearly states that no sample will be drawn, as applicable (i.e., the entire program population will be included in the study). 

For our school sample, we will prioritize including the schools that agree to have teachers administer our chosen assessment tool, the Behavior Assessment System for Children, Third Edition (BASC-3), to all students in the school at two time points (fall and spring) during the 2023-2024 school year. Schools that participate in the study will also agree to provide de-identified student level data that includes both demographic, academic, and behavioral data elements. Our program already has data-sharing MOUs in place with most of our school partners and we have strong relationships with the administrators at each of our partner schools, thus we do not anticipate any challenges in recruiting the minimum number of schools (n=6) needed for our study. However, if necessary, we will consider offering incentives (e.g., gift cards for teachers) to encourage their participation.  	Comment by Megan Edgin (she/her): Here there is a description of the MOU in place, which addresses any consent procedures (i.e., parental/guardian consent, opt-in/opt-out) or data use agreements that will be necessary to gather or obtain data. This description also shows being thoughtful of recruitment procedures, such as a potential need for incentives. The number of schools is listed. 

For our student sample, the treatment group will consist of students who participate in the ASY program for a minimum of four months during the 2023-24 school year and attend one of the six partner schools that agree to participate in the study. The comparison group sample will consist of the students who did not participate in the program and had a similar propensity for receiving ASY services as another student in their same school. 	Comment by Megan Edgin (she/her): This description outlines the student sample approach for both treatment and comparison groups. The sample size is further described in the "Sample Size Justification" section.

## 5.2 *Sample Size Justification*	Comment by Megan Edgin (she/her): For impact evaluations, describe how a power analysis was used to determine (1) how large a sample is needed to enable statistical tests that are accurate and reliable (i.e., required minimum sample size), and (2) the likelihood the statistical tests used in the analysis will be able to detect effects of a given size in a particular situation. Include detail on the assumptions used to conduct the power analysis (including how a minimum detectable effect size (MDES) was identified) and specify the results of the power analysis. Note that the values used to calculate the a priori power analysis must be included (e.g., minimum detectable effect size). If subgroup analyses are anticipated, ensure that the sample size is sufficient to allow for these analyses.The specific sample size for treatment and comparison groups is noted here. An estimation is sufficient.

Each ASY AmeriCorps member is assigned a caseload of approximately 30 students that they work with during out-of-school time sessions throughout the year, and all school sites have multiple AmeriCorps members (average of 3 members per site). Thus, ASY estimates serving a total of 1,350 students each year, an average of 90 students per school site. ASY AmeriCorps currently partners with schools where the number of students who are eligible for support from an ASY member is more than twice as large as the total number of students who could be served within the school year. Given this, the potential size of the comparison group pool exceeds 1,350 students. 

To determine the minimum sample sizes needed for our study to detect an expected effect size of .20[footnoteRef:1] on our selected outcomes of interest, we conducted a power analysis using the following assumptions:  [1:  Based on prior pre/post outcomes assessment data of ASY participants using the Behavior Assessment System for Children, Third Edition (BASC-3). ] 

· Expected effect size: .20

· Power level: .80

· Significance level (two-tailed test): .05

· ICC: .10

· ANOVA / multiple regression model

· Two study groups program (T) and comparison (C)

· 180 eligible students per school (90 T and 90 C)

· Pre-test used, which is highly correlated with the post-test (.80)

· At least one covariate would be used at both the school (e.g., size, type, FRPL rate, etc.) and student level (e.g., race, gender, income, etc.) in the analysis model to explain variation in the data

Based on these assumptions, the power analysis results indicate that our study requires 6 schools of approximately 180 students, which is 1,080 students total (540 in the treatment group and 540 in the comparison group). Given the estimated number of ASY participants per year and the size of the comparison group pool, our study will be able to meet the required minimum sample size needed to detect an effect of participation in the ASY AmeriCorps Program. 

# 1. Data Collection Procedures, Data Sources, and Measurement Tools	Comment by Megan Edgin (she/her): Describe each data source and measurement tool and the procedures that will be used to collect or extract data, including when, how often, and by what mode (i.e., paper/pencil, phone, or web survey; administrative data extract). Explain how the proposed data sources and tools are adequate for addressing all of the research question(s) and how the data align with the evaluation’s outcome(s) of interest. 

To measure the outcomes of interest, we will use a common published instrument, the Behavior Assessment System for Children, Third Edition (BASC-3) Teacher Rating Scales (TRS). The BASC–3 Teacher Rating Scales (TRS) is a comprehensive measure of both adaptive and problem behaviors in the school setting and directly aligns with our outcomes of interest. It is designed for use by teachers or others who fill a similar role, such as teacher assistants or preschool caregivers. The TRS has three forms, with items targeted at three age levels: preschool (ages 2 through 5), child (ages 6 through 11), and adolescent (ages 12 through 21). Given the age range of the students served by our program, the child and adolescent forms will be used for this study, depending on the age of the subject child. The forms contain descriptors of behaviors that the respondent rates on a four-point scale of frequency, ranging from Never to Almost always.  The TRS takes 10 to 15 minutes to complete for teachers with experience completing rating scales. 	Comment by Megan Edgin (she/her): There is a detailed description of each data source that will be collected or extracted and when and how often data will be collected or used. The data sources are adequate for addressing all of the study's research questions, and either a description of the tools for collecting data or a detailed plan for extracting administrative data is provided. This section clearly describes the data collection instrument revealing developmental appropriateness for the students included in the evaluation through the use of different forms,. This section also notes topics covered in the assessment to show alignment with the outcomes of interest, scope of evaluation, and research questions.

As noted above, the schools that participate in the study will agree to have classroom teachers administer the assessments to all students in their home room classroom at two time points – early fall 2023 for the pre-test and late spring 2024 for the post-test. All item responses will be entered by hand into the BASC-3 ASSISTTM software, and routine quality checks will be conducted. All scoring will be computed by the BASC-3 ASSIST software.	Comment by Megan Edgin (she/her): This section clearly describes who will conduct data collection, when data collection will be conducted, where data collection will occur, and data entry/quality assurance procedures.

Through a data-sharing agreement with each participating school, our evaluator will have access to de-identified student-level administrative data, including demographic data, achievement data, and behavioral data (i.e., the BASC assessment scores). 	Comment by Megan Edgin (she/her): This statement is important to show how access to data will be granted.

# 1. Analysis Plan	Comment by Megan Edgin (she/her): Explain how the analysis will address all of the evaluation's research questions. Multiple regression or ANOVA analysis is appropriate for quantitative data used for impact designs. Qualitative analysis methods (e.g., content analysis, thematic analysis) are appropriate for qualitative data.

The external evaluator will first conduct a set of baseline analyses, comparing the treatment and comparison groups to ensure that they are similar on average in terms of students’ race/ethnicity, gender, grade level, ELL status, and academic achievement. Baseline levels of short-term outcomes will also be examined to ensure that the comparison group is similar across these outcomes at baseline relative to the treatment group. Two-sample t-tests and chi-square tests (where appropriate) will be utilized to verify that the treatment and comparison groups are comparable. Variables where treatment and comparison groups differ significantly will be included as covariates in the analytic analyses.	Comment by Megan Edgin (she/her): Describe how baseline equivalency test(s) will be conducted to demonstrate that the sample groups (i.e., treatment and comparison/control) do not differ significantly at baseline or, if differences exist, how the necessary statistical adjustment will be made to address any group differences identified.

To answer each of our research questions, we will use multiple regression to assess the impact of the intervention by estimating the difference between the baseline and follow-up short-term outcomes among the comparison group relative to the treatment group. The regression model will include the following variables: (1) time (baseline and follow-up) and (2) the group (treatment vs. comparison group), and (3) an interaction term between time and group. The impact will be estimated through the coefficient of the interaction term. Important covariates that are noted in the preliminary analyses also will be included in the model. Models will be run separately for each of the short-term outcomes.	Comment by Megan Edgin (she/her): Describe an analysis that is appropriate for the evaluation’s design and data sources (e.g., statistical testing for quantitative data or descriptive analysis methods for qualitative data). Describe how you will use multivariate analysis techniques (e.g., M/ANOVA or ANCOVA or regression models) for analyzing the pre-post data for the purpose of answering the research questions; t-tests of statistical significance are not sufficient because covariates cannot be utilized for these statistical tests, though they are useful for baseline equivalency test(s).In this case, it is clear how the analysis will yield answers to the research questions. For evaluation that include mixed methods, multiple types of evaluation, or various focus, it can be helpful to map the analysis by data source for each research question.The proposed analysis is appropriate for the evaluation’s design and data sources and should produce findings that address all of the evaluation’s research questions. 

If sample size permits, additional subgroup analyses may be used to assess whether students equally benefited from their participation in the ASY Program or whether there are important differences based on socio-demographic background. To help ensure that the results of the regression models are the result of the ASY Program as opposed to unique aspects of each school and program site, sites will be controlled for in regression models.

# 1. Evaluator Qualifications	Comment by Megan Edgin (she/her): Describe how the person(s) who will conduct the evaluation are sufficiently qualified to conduct the proposed evaluation (e.g., have experience and technical qualifications that align with the planned evaluation design). Include details for how the evaluator is sufficiently qualified and meets AmeriCorps evaluation requirement for the applicant (large or small). Whether the evaluator is internal or external to the program is clearly stated. However, for impact evaluations, an external evaluator is strongly recommended for small grantees and may be required by the terms of your grant award if you are a large grantee.If an external evaluator has been identified, the description supports the conclusions that the evaluator is independent from the program and qualified to conduct an objective and unbiased evaluation (e.g., no conflicts of interest, nor appearance of conflicts). This also means that evaluators that have any affiliation to the program’s host agency or organization may not be considered external. Grantees that are uncertain of their evaluator’s affiliation, should confirm with their AmeriCorps Portfolio Manager or with their State Commission representative if a State Commission sub-grantee.If the evaluator is not yet identified or hired, describe the required and/or preferred qualifications for an evaluator. For more information on hiring an external evaluator, please see the evaluation core curriculum “Managing an External Evaluation” course. 

ASY AmeriCorps has established a relationship with faculty in the Education Department at the State University. Faculty in the department have been engaged in a wide range of quantitative evaluation projects that involve analyzing data to demonstrate evidence of impact, inform continuous improvement, and compliance. We plan to hire an assistant professor and graduate student to serve as evaluators for our study. The evaluators will work with us to finalize the evaluation design, analyze the collected data, and write the final evaluation report. 

# 1. Timeline	Comment by Megan Edgin (she/her): Provide a timeline for all of the major evaluation activities (e.g., finalizing evaluation design, hiring evaluator, developing data collection instruments, collecting pre-intervention data, collecting post-intervention data, analyzing data, writing report). Delineate the timeline by month and year or on a quarterly basis (e.g., fall 2023, spring 2024). The timeline must show how all evaluation activities and a final report will be completed before your next recompete application.AmeriCorps recommends using the first program year for evaluation planning (including gaining final approval of the plan) and data collection instrument development; the second program year for data collection; and the remaining time in the third program year to analyze data and complete the evaluation report. Since grantees have unique programs and recompete application deadlines may vary by state, exact evaluation timelines may vary. 

# 1. Budget	Comment by Megan Edgin (she/her): Specify the overall budget allotted for the evaluation, including the cost of engaging an external evaluator if applicable. If you will be utilizing staff time for conducting an internal evaluation, provide a description of those in-kind resources. Estimated budget is reasonable given the scope of the evaluation.

We have budgeted a total of $70,000 for external evaluation services, which is approximately 10% of the ASY AmeriCorps budget per year.