FY19 End of Grant Evaluation (2019-21)

Jefferson County Board of Education 

Program Name: REACH Corps

Evaluation Completed: July 16, 2021 

Evaluators: 

Stephen Lin, PhD

Thomas Reece, M.A.
2 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

Description of the Program

REACH Corps aims to reduce disciplinary incidences and increase attendance rates by 

implementing Check and Connect engagement model strategies. Increased student engagement 

through targeted interventions to build relationships will promote change in attendance rates and 

disciplinary incidences. 

Goal and Research Questions

1. To what effect does the REACH Corps program improve school attendance for students 

who participate in the program? 

2. To what effect does the REACH Corps program reduce the number of behavior 

incidences for students who participate in the program? 

Approach

This evaluation used a quasi-experimental design, where students are not randomly assigned to 

treatment and control groups, but rather a matched control group was created post hoc through 

multivariate matching. Each student in the intervention group was matched with similar students 

who did not participate in the program. Selection of the comparison group used a two-step 

process. First, students were matched by grade, school, demographic variables (age, gender, race, 

free/reduced lunch status, ECE or special education status), and baseline rates of unexcused 

school absences and behavior referrals. This ensured the comparison group was as similar in 

background as possible to the treatment group. Not all of the students could be matched on all 

variables, however, so the remainder of the students were matched solely by grade and baseline 

rates of school absences and behavior referrals. All students in the comparison group were from 

schools participating in the REACH Corps program. 

Due to the unique challenges introduced by the COVID-19 pandemic, the evaluation team 

decided to limit the evaluation to the time period occurring before the pandemic began to have 

a widespread impact on the school district. During the 2019-2020 school year, all schools in 

the district transitioned to Non-Traditional Instruction (NTI) starting March 16, 2020 and 

instruction continued in virtual environments through the remainder of the school year. The 

outcomes targeted by this evaluation (absenteeism and behavior referrals) were fundamentally 

affected by the transition to NTI. The state department of education released guidance that 

resulted in behavior incidents being treated differently (e.g. suspensions were no longer a 

resolution option) and what constituted attendance was also redefined by the state department 

of education. Given these challenges, this evaluation is based solely on the time period before 

NTI began. Preintervention control variables and the identification of a comparison group were

based on data from SY 18-19, but instead of using data for the full school year, the data were

based on the same number of instructional days as were in the Pre-NTI 19-20 school year (130 

instructional days).
3 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

Data Analysis 

Both research questions were addressed using multivariate regression models to determine 

significant differences between the two groups. This analysis method allows us to control for any 

differences that may be present from the pre-test measures (behavior referrals and unexcused

absences), student demographics, or student grade. To determine if the effect of the program was 

moderated by grade, a program participation by grade interaction term was also included in the 

regression model. This had the benefit of testing for a differential effect of the program 

depending on the grade of the student while also preserving statistical power, and controlling the 

Type I error rate, by conducting only one analysis per outcome. 

Results

Table 1 shows demographic information for both groups. Student demographic variables were 

similar between intervention and comparison groups and preintervention differences in 

unexcused absences and behavior referrals were acceptably similar (d = .02, a small difference). 

This difference in pretest scores is within acceptable limits as defined by the What Works 

Clearinghouse standards (version 4.1) for quasi-experimental research designs.

Table 1

Demographics 

REACH Comparison

Variable M (SD) M (SD)

Prior Year Unexcused Absences 13.32 (6.89) 13.17 (6.94)

Prior Year Behavior Referrals 3.27 (6.26) 3.20 (6.22)

Intervention Year Unexcused 

Absences 7.53 (6.49) 8.38 (7.32)

Intervention Year Behavior Referrals 4.45 (8.89) 4.54 (10.00)

Race N N

 African American 324 293

 Asian 2 5 

 Hispanic 43 44

 Two or More 23 24

 White 97 123

 Other 1 1 

Gender

 Male 237 242

 Female 253 248

Grade

 Kindergarten 20 20
4 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

REACH Comparison

Variable M (SD) M (SD)

 1 19 19

 2 29 28

 3 55 56

 4 82 82

 5 56 56

 6 62 63

 7 89 89

 8 78 77

% F/R Lunch 18% 17%

% ECE 90% 91%

Outcome 1: To what effect does the REACH Corps program improve school attendance for 

students who participate in the program? 

 

To understand the effect of the REACH Corps program on unexcused absences, an OLS

regression analysis was used. Table 2 shows the final model. The highlighted rows in Table 2 

indicate variables of interest, specifically, the variable related to participation in the program and 

the interaction effect of program participation by student grade. The unhighlighted variables 

represent variables added to the model to control for factors that may affect the outcome, but are 

not related to the program itself. As can be seen in Table 2, the REACH Corps membership by 

grade interaction term was statistically significant. This indicates that the effect of the REACH 

Corps program on absenteeism varied based on the grade of the participant. 

Table 2 

Model Parameter Estimates for Model Predicting Unexcused Absences During 

The Intervention Year

 Unexcused Absences

Predictors Estimate se 95% CI p 

Intercept -2.47 1.12 -4.66 – -0.29 0.03

REACH Corps Membership 1.50 0.98 -0.42 – 3.43 0.12

REACH X Grade -0.49 0.18 -0.83 – -0.14 0.01

Grade 0.87 0.14 0.60 – 1.14 <0.001

F/R Lunch 0.79 0.66 -0.51 – 2.08 0.24
5 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

 Unexcused Absences

Predictors Estimate se 95% CI p 

ECE 0.57 0.53 -0.47 – 1.61 0.28

Prior Year Behavior Referrals 0.07 0.03 0.00 – 0.13 0.04

Prior Year Unexcused Absences 0.42 0.03 0.36 – 0.48 <0.001

Sex (Male) -0.31 0.41 -1.12 – 0.50 0.46

Race (White) a 0.34 0.49 -0.62 – 1.30 0.49

Race (Hispanic) a 0.32 0.71 -1.07 – 1.71 0.66

Race (Asian) a -3.28 2.34 -7.86 – 1.31 0.16

Race (2+ Races) a 0.74 0.92 -1.07 – 2.54 0.42

Race (Other) a 3.43 4.28 -4.96 – 11.81 0.42

Random Effects

σ2 35.99 

τ00 school 0.81 

N school 24 

Observations 980 

Marginal R2 / Conditional R2 0.228 / 0.245 

Note. Parameters of interest highlighted a Referent = African American

Because of the statistically significant interaction effect in the model, the easiest way to examine 

the effect of the program is graphically. Figure 1 shows the effect of the program on current year 

unexcused absences for each grade. The blue bars represent the predicted unexcused absences, 

given a particular grade, for the average student who participated in the REACH program. The 

red bars represent the average student in the matched comparison group. The lines extending 

above and below the top of each bar represent the 95% confidence intervals. For practical 

purposes, when the line for one group includes the top of the bar for the other group, there is not 

a significant difference between the two groups at that level. As can be seen in Figure 1, there 

was no difference in unexcused absences between the groups for students in 4th grade and below. 

Starting with 5th grade, however, students in the REACH program had fewer unexcused absences 
6 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

than the comparison students. This suggests that the REACH program was more effective when 

working with older students (d at grade 8 = 0.18, or 2.39 fewer unexcused absences). 

Outcome 2: To what effect does the REACH Corps program reduce the number of behavior 

incidences for students who participate in the program? 

To understand the effect of the REACH Corps program on the number of behavior incidences, a 

Poisson regression analysis was used. Table 3 shows the final model. The highlighted rows in 

Table 3 indicate variables of interest, specifically, the variable related to participation in the 

program and the interaction effect of program participation by student grade. The unhighlighted 

variables represent variables added to the model to control for factors that may affect the 

outcome, but are not related to the program itself. As can be seen in Table 3, the REACH Corps 

membership by grade interaction term was statistically significant. This indicates that the effect 

of the REACH Corps program on the number of behavior referrals varied based on the grade of 

the participant.

Table 3

Model Parameter Estimates for Model Predicting the Number of Behavior Referrals During The 

Intervention Year

 Behavior Referrals

Predictors Count Ratio se 95% CI p 

Intercept 0.72 0.12 0.52 – 1.00 0.05 

REACH Corps Membership 1.65 0.15 1.39 – 1.97 <0.001
7 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

 Behavior Referrals

Predictors Count Ratio se 95% CI p 

REACH X Grade 0.93 0.01 0.90 – 0.96 <0.001

Grade 1.11 0.02 1.07 – 1.15 <0.001

F/R Lunch 1.36 0.09 1.20 – 1.54 <0.001

ECE 1.33 0.05 1.24 – 1.42 <0.001

Prior Year Behavior Referrals 1.08 0.00 1.07 – 1.08 <0.001

Prior Year Unexcused Absences 1.01 0.00 1.01 – 1.02 <0.001

Sex (Male) 1.38 0.05 1.29 – 1.48 <0.001

Race (White) a 0.55 0.03 0.50 – 0.60 <0.001

Race (Hispanic) a 0.45 0.04 0.38 – 0.53 <0.001

Race (Asian) a 0.16 0.09 0.05 – 0.50 <0.001

Race (2+ Races) a 0.57 0.06 0.47 – 0.70 <0.001

Race (Other) a 0.20 0.14 0.05 – 0.78 0.02

Random Effects

σ2 0.27 

τ00 intervention.schoolID 0.33 

N intervention.schoolID 24 

Observations 980 

Marginal R2 / Conditional R2 0.486 / 0.768 

Note. Parameters of interest highlighted a Referent = African American

Because of the statistically significant interaction effect in the model, the easiest way to examine 

the effect of the program is graphically. Figure 2 shows the effect of the program on current year 

unexcused absences for each grade. The blue bars represent the count ratio of behavior referrals, 

given a particular grade, for the average student who participated in the REACH program. The 

red bars represent the average student in the matched comparison group. The lines extending 

above and below the top of each bar represent the 95% confidence intervals. For practical 
8 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

purposes, when the line for one group includes the top of the bar for the other group, there is not 

a significant difference between the two groups at that level. As can be seen in Figure 2, there 

was a statistically significant difference in the count ratio of behavior referrals between the 

groups for students in 3rd grade and below. Starting with 4th grade students, however, the count 

ratio of behavior referrals for students in the REACH program was not statistically significantly 

different from the comparison students. This suggests that the REACH program was not 

effective in reducing the number of behavior referrals. 

Recommendations

The REACH program was effective in reducing the number of unexcused absences when 

working with older students (grades 5-8). However, the program was not effective in reducing 

the number of unexcused absences for younger students (grades K-4) and did not reduce the 

number of behavior referrals at any grade. These findings suggest a number of 

changes/improvements are possible: 

1. It may be beneficial for the REACH program to focus primarily on older students. 

2. Alternatively, the REACH program may need to adjust the intervention protocol for 

younger students. This may involve providing training to REACH Corps members who 

work with younger students that is more specific to their needs. 

3. The current analysis can not distinguish program effects from the variations in 

implementation of the program by individual REACH Corps members. A possible 

question to be addressed in a process evaluation would be what characteristics successful 

REACH Corps members share. This information may be useful to identify future 

members.
9 

ARSI:SL:TR 16-July-2021

REACH Corps End of Grant Evaluation

Evaluation Brief 

16-July-2021 

4. The process of selecting program schools should be revised. Identifying students for the 

comparison group was made more difficult for some participating schools due to the 

limited number of students who met program eligibility requirements. Relatedly, this

means these schools would also have difficulty recruiting a sufficient number of eligible 

students into the program. The results of this evaluation can be used to identify schools 

with larger populations of in-need students, ensuring REACH Corps members are 

deployed where they will make the most difference.