Impact Evaluation of the Minnesota Alliance With 

Youth Statewide AmeriCorps Promise Fellows 

Program 

Prepared for Minnesota Alliance With Youth 

on October 7, 2022 by Ethan R. Van Norman, PhD 
 

4 

Table of Contents 

Background 3 

Present Evaluation 3 

Data from 2021-2022 

Table 1. Summary of Students Characteristics in Entire Dataset 4 

Table 2. Demographic Information for Entire Sample 5 

Data Analysis Plan 6 

Matching Process 6 

Overall Impact 7 

Analyses for Student Sub-Groups 7 

Results 7 

Overall Impact of Promise Fellow Program 8 

Table 3. Overall Change in Attendance Percentage 8 

Impact of the Promise Fellow Program across Student Sub-Groups 8 

Table 4. Change in Attendance Percentage by Grade Level 8 

Table 5. Change in Attendance Percentage by Gender 8 

Table 6. Change in Attendance Percentage by Race 9 

Summary of Program Impact 9 

References 10 

Appendix 11 

Table 7. Generalized Estimating Equation Output for Overall Program Impact (Log-Odd 

Units) 11 

Table 8. Logistic Regression Analysis for Impact of Grade Level (Log-Odd Units) 11 

Table 9. Logistic Regression Analysis for Impact of Gender (Log-Odd Units) 11 

Table 10. Logistic Regression Analysis for Impact of Race/Ethnicity (Log-Odd Units) 12 
 

 

Background The Promise Fellow Program is an intervention designed to prevent dropout amongst 

students in grades 6-12 in Minnesota that show early warning signs of school 

disengagement. 

The program consists of three components, led by interventionists, or Promise Fellows. 

The program draws from a menu of research-based supports. Each category of 

support, caring adults, service-learning, and out-of-school supports align with the 

National Dropout Prevention Center’s “Basic Core Strategies”. 

Caring adult strategies aim to develop strong, positive relationships between the 

student and the Promise Fellow as well as other adults in the student’s life. Servicelearning involves the Promise Fellow helping the student serve as a mentor, volunteer, 

or to engage in a service-learning project. Finally, for out-of-school supports, the 

Promise Fellow helps the student participate in things like after-school clubs, activities, 

and tutoring. 

Each Promise Fellow generally supports at least 30 youth for a minimum of 12 weeks. 

Promise Fellows regularly meet with the Youth Success Team, where data collected by 

Fellows (e.g., Attendance, Engagement, Work Completion) are shared to create a 

focus list and develop targeted supports based upon student need. 

Present Evaluation 

The purpose of this report is to evaluate the impact of the Promise Fellow Program on 

student school attendance. More specifically, this report will determine whether there 

is a statistically significant difference in the percentage of days attended between 

students served by the Promise Fellow Program and students not served by the 

program during the 2021-2022 academic year. In addition, this report will explore the 

impact of the Promise Fellow Program on school attendance for various student 

subgroups including gender, race, and grade level. 
 

Data from 2021-2022 

At the conclusion of the 2021-2022 academic school year, a total of 11 schools that 

participated in the Promise Fellow Program shared attendance data with the 

Minnesota Alliance With Youth. The table below shows the number of students present 

in the dataset shared with the evaluator. 

Table 1. Summary of Students Characteristics in Entire Dataset 

School ID Promise Fellow (n) Typical Practice (n) Total (n) 

1 16 694 710 

2 25 232 257 

3 22 36 58 

4 11 114 125 

5 11 80 91 

6 4 983 987 

7 15 15 30 

8 3 57 60 

9 225 1086 1311 

10 11 57 68 

11 31 1956 1987 

Total (%) 374 (6.58%) 5310 (93.42%) 5684 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Demographic information for the entire sample as well as by Promise Fellow Program 

students and non-participating students (Typical Practice) are presented below: 

Table 2. Demographic Information for Entire Sample 

Promise Fellow Typical Practice Total 

(n = 374) (n = 5310) (n = 5684) 

Gender 

Female 3.96% 42.93% 46.89% 

Male 2.52% 48.89% 51.41% 

Other 0.11% 0.23% 0.33% 

Not Reported -- 1.37% 1.37% 

Grade 

5 -- 0.12% 0.12% 

6 0.48% 4.45% 4.93% 

7 0.76% 12.30% 13.05% 

8 1.02% 13.83% 14.85% 

9 0.67% 19.16% 19.83% 

10 1.20% 12.97% 14.16% 

11 1.13% 13.48% 14.60% 

12 1.34% 17.10% 18.44% 

Not Reported - 0.02% 0.02% 

Race 

AI 0.14% 1.74% 1.88% 

Asian 0.12% 4.68% 4.80% 

Black 1.76% 11.59% 13.35% 

Latinx 2.57% 13.37% 15.94% 

Multi-Racial 0.14% 2.41% 2.55% 

NHPI - 0.05% 0.05% 

Oromo - 0.02% 0.02% 

Somali 0.02% 1.06% 1.07% 

White 1.83% 57.14% 58.97% 

Not Reported - 1.35% 1.35% 

Note. AI – Native Alaskan / American Indian, NHPI – Native Hawaiian / 

Pacific Islander 

Reviewing Tables 1 and 2 there is a clear imbalance in the number of students that 

participated in the Promise Fellow Program relative to students that did not. Combined 

with the fact that students were not randomly selected to participate in the program, 

statistical adjustments were necessary to determine the impact of the intervention. 
 

 

 

Data Analysis Plan 

Propensity score matching is a method in which the effect of an intervention can be 

estimated when students are not randomly assigned to conditions (Rosenbaum & 

Rubin, 1983). Random assignment in experiments helps to ensure that treatment and 

control groups are balanced on factors that may predict treatment response other 

than the intervention. Propensity score matching can be used to meaningfully select 

students that did not participate in the Promise Fellow Program to compare against 

students that did participate. 

Propensity score matching, in general, matches students from the experimental group 

with those from the control group by estimating a logistic regression model in which 

relevant student characteristics are used to predict treatment assignment. Depending 

on the matching method used, students from each group are matched based upon 

their probability of receiving the treatment from the logistic regression analysis. Doing 

so reduces bias in the estimation of treatment effects caused by confounding factors 

that influence whether students were or were not selected to participate in the 

intervention. 

For the present evaluation 1:1 nearest neighbor propensity score matching without 

replacement was conducted using the MatchIt package (Ho et al., 2011) in the 

computer program R (R Core Team, 2021). Data available for most students included 

grade level, race, gender, ethnicity, and the percentage of school days attended the 

previous year. One complexity in the present evaluation was the nested nature of 

outcomes. Namely, students were nested within schools. This is important because any 

number of unmeasured factors within individual school sites may have influenced 

student participation in the program. To account for this, propensity score matching 

was conducted separately within each school site (Cannas & Arpino, 2019). 

Matching Process 

Unfortunately, data from three schools (3, 7 and 10) could not be used for a variety of 

reasons including limited or no outcome data (end of year attendance rates) and/or 

large volumes of missing student data (e.g., race, gender). In addition, sites 8 and 9 

did not have sufficient information to match students in grade five that participated in 

the program. In turn, those students were matched out of grade level within their sites. 

There was a total of 309 students that participated in the Promise Fellow Program 

included in the analytic sample and 309 comparable peers that did not participate. 

The average attendance rate was 79.16% for the control group the previous year 

compared to an average rate of 78.19% in the Promise Fellow group. That difference 

was not statistically significant (t = 0.867) at the p < .05 level (p = .385). 

Follow-up chi-square tests of independence suggest that groups did not differ as a 

2 2 function of gender (χdf=2 = 0.174, p = .917) or race (χdf=6 = 8.989, p = .174). This 
 

suggests that the matched groups are sufficiently comparable to permit an evaluation 

of the impact of the Promise Fellow Program. 

Overall Impact 

Given the nested nature of available data, a Generalized Estimating Equation (GEE) 

was estimated to quantify the overall average impact of participating in the program 

across available student characteristics and previous attendance history. By using a 

GEE the unique influence of each school site is statistically taken into account to 

determine the overall impact of the program (McNeish et al., 2017). The geepack 

package (Højsgaard et al., 2006) in the computer program R (R Core Team, 2021) was 

used for analyses. 

Analyses for Student Sub-Groups 

In addition to understanding the typical, or average, impact of the program on school 

attendance, follow-up analyses were conducted to determine whether the Promise 

Fellow Program differentially impacted student sub-groups. Separate logistic regression 

models were estimated to determine the impact of the program teased apart by 

gender, grade level, and race. These analyses considered the impact across school 

sites. Logistic regression analyses were conducting using the glm function in R (R Core 

Team, 2021). 

Results 

The Appendix shows the raw output of the GEE and logistic regression analyses. In 

each case a binomial link function was used to model the percentage of school days 

students attended in the academic year with weights equal to the number of days 

school was in session. The output of those analyses, log-odd coefficients, are difficult to 

interpret. Therefore, in the main report log-odds are converted to probabilities which 

are converted to percentages to quantify the average percentage of schooldays 

attended for each group of interest. In addition, assuming a 170-day academic year, 

the average number of days differences in percentages translate to are also reported. 
Overall Impact of Promise Fellow Program 

Table 3. Overall Change in Attendance Percentage 

Typical Practice Promise Fellow Difference Difference 

(School Days) 

80.38% 83.20% + 2.82% + 4.79 

The results of the GEE suggest that the net effect, or overall average effect of the 

program yielded a statistically significant (p < .001) difference in the attendance rates 

of students that participated in the Promise Fellow Program compared to students that 

did not. Converting the outcomes of that analysis suggest that, on average, students 

that participated in the program attended one more week, or roughly five days of 

school, relative to students that did not participate in the program (assuming a 170 

day school year). 

Impact of the Promise Fellow Program across Student Sub-Groups 

Table 4. Change in Attendance Percentage by Grade Level 

Grade Typical Practice Promise Fellow Difference Difference 

(School Days) 

7 82.55 83.26 +0.71 +1.21 

8 86.33 80.60 -5.73 -9.74 

9 84.08 86.92 +2.84 +4.83 

10 82.16 85.08 +2.92 +4.96 

11 76.24 83.4 +7.16 +12.17 

12 74.36 80.85 +6.49 +11.03 

The results of this analysis suggest the program is relatively most impactful for older 

students. Note low participation rates in grade five and six forced them to be omitted 

from this follow-up analysis. 

Table 5. Change in Attendance Percentage by Gender 

Gender Typical Practice Promise Fellow Difference Difference 

(School Days) 

Female 80.83 83.64 +2.81 +4.77 

Male 79.93 83.01 +3.08 +5.24 

The results of this analysis suggest that the program works similarly for male and female 

students. Note there were a few students that identified as non-binary in the sample 

that could not be adequately matched within individual school sites. 
 

Table 6. Change in Attendance Percentage by Race 

Race Typical Practice Promise Fellow Difference Difference 

(School Days) 

AI 79.87 73.52 -6.35 -10.80 

Asian 85.22 91.16 +5.94 +10.10 

Black 80.98 81.35 +0.37 +0.63 

Latinx 78.40 84.20 +5.80 +9.86 

Multi 82.72 89.44 +6.72 +11.42 

White 82.62 83.45 +0.83 +1.41 

Note – AI: American Indian. Only racial categories with a sufficient sample size were 

included. 

The results of this analysis suggest that the program tends to have differential impact as 

a function of student race. Namely, American Indian students that participated in the 

program attended school less than non-participants. Black students tended to 

experience little increase in school attendance whereas the program seems to be 

relatively more effective for students that were Asian, Latinx, and multi-racial. 

Summary of Program Impact 

Overall, the results of this impact evaluation suggest that the Promise Fellow Program 

had on average, a statistically significant positive effect on school attendance for 

students that participated. 

Further, the program seems to be relatively most effective for students at higher grade 

levels, as well as Asian, Latinx, and multi-racial students. 

Moving forward, it may be worthwhile to conduct internal reviews to determine how 

the program may be optimized for younger students as well as American Indian and 

Black students. 
References 

Cannas, M., & Arpino, B. (2019). Matching with clustered data: the CMatching 

Package in R. The R Journal, 11. 

Ho, E. E., Imai, K., King, G., & Stuart, E. A. (2011). MatchIt: Nonparametric preprocessing 

for parametric causal inference. Journal of Statistical Software, 42, 1-28. 

Højsgaard,S., Halekoh, U., & Yan, J. (2006). The r package geepack for generalized 

estimating equations. Journal of Statistical Software, 15, 1-11. 

McNeish, D., Stapleton, L., M., & Silverman, R. D. (2017). On the unnecessary ubiquity of 

hierarchical linear modeling. Psychological Methods, 22, 114-140. 

R Core Team (2021). R: A language and environment for statistical computing. R 

Foundation for Statistical Computing. https://www.R-project.org/ 

Rosenbaum, P. R. & Rubin, D. B. (1983). The central role of the propensity score in 

observational studies for causal effects. Biometrika, 70, 41-55. 
Appendix 

Table 7. Generalized Estimating Equation Output for Overall Program Impact (Log-Odd 

Units) 

Coefficient B SE Wald p 

Intercept 1.410 0.027 2645.44 <.001 

Treatment 0.191 0.037 26.396 <.001 

Table 8. Logistic Regression Analysis for Impact of Grade Level (Log-Odd Units) 

Coefficient B SE p 

Intercept 1.554 0.027 <.001 

Treatment 0.053 0.043 .219 

Grade 8 0.289 0.045 <.001 

Grade 9 0.109 0.430 .011 

Grade 10 -0.028 0.038 .470 

Grade 11 -0.388 0.039 <.001 

Grade 12 -0.485 0.035 <.001 

Grade 8 x Treat -0.469 0.062 <.001 

Grade 9 x Treat 0.183 0.075 .014 

Grade 10 x Treat 0.161 0.058 .005 

Grade 11 x Treat 0.398 0.058 <.001 

Grade 12 x Treat 0.322 0.055 <.001 

Model Fit Deviance (Null) Deviance (Residual) AIC 

10168.30 9316.80 1221 

Table 9. Logistic Regression Analysis for Impact of Gender (Log-Odd Units) 

Coefficient B SE p 

Intercept 1.440 0.015 <.001 

Treatment 0.193 0.021 <.001 

Male -0.056 -0.057 .023 

Male x Treat 0.013 0.014 .681 

Model Fit Deviance (Null) Deviance (Residual) AIC 

10539 10159 13175 
 

Coefficient B SE p 

Intercept 1.379 0.191 <.001 

Treatment -0.357 0.202 .077 

Asian 0.373 0.206 .070 

Black 0.066 0.194 .733 

Latinx -0.093 0.192 .630 

Multi 0.189 0.224 .405 

White 0.180 0.193 .351 

Treat x Asian 0.938 0.248 <.001 

Treat x Black 0.381 0.205 .062 

Treat x Latinx 0.741 0.204 <.001 

Train x Multi 0.923 0.265 <.001 

Train x White 0.416 0.204 .042 

Model Fit Deviance (Null) Deviance (Residual) AIC 

10539 10036 13066 

Table 10. Logistic Regression Analysis for Impact of Race/Ethnicity (Log-Odd Units)