October 2020

CIS OF CENTRAL TEXAS

Austin ISD AmeriCorps Outcomes Evaluation
CIS-AmeriCorps Evaluation 2020 1 

Table of Contents 

Executive Summary ............................................................................................................... 2

Introduction .......................................................................................................................... 3

The CIS Model................................................................................................................................4

AmeriCorps Partnership.................................................................................................................5

Background Significance.................................................................................................................6

Research Questions .......................................................................................................................7

Key Findings.......................................................................................................................... 8

Question 1: Academics...................................................................................................................8

Question 2: Behaviors....................................................................................................................9

Question 3: Climate .......................................................................................................................9

Conclusions & Recommendations ....................................................................................... 10

Limitations and Future Research ..................................................................................................13

Study Details....................................................................................................................... 14

Methodology ...............................................................................................................................14

Results.........................................................................................................................................17

Elementary School Grades................................................................................................................................. 17

STAAR................................................................................................................................................................. 19

Attendance & Discipline.....................................................................................................................................20

Climate ...............................................................................................................................................................21

References........................................................................................................................... 23

About the Evaluators........................................................................................................... 24

Acknowledgements............................................................................................................. 24

 
CIS-AmeriCorps Evaluation 2020 2

Executive Summary 
CIS-AmeriCorps Evaluation 2020 3 

Introduction 

Communities In Schools (CIS) is a national organization serving students with a diverse 

range of needs across the country. 

Broadly, CIS Central Texas is 

focused on dropout 

prevention, but it is so much 

more than that. CIS Central 

Texas serves the most 

vulnerable students and their 

families in the region to ensure 

they have the resources 

needed to overcome any 

barriers that may keep 

students from remaining in 

school.

An integral component of their 

work is to establish 

relationships with individual 

students to support them when 

an obstacle is encountered in an effort to keep them in school and succeed in life. 

These supports may include counseling, life skill development, academic tutoring, and 

other enrichment opportunities. Common to these supports is providing students with 

ample opportunities to relate with each other, build connections with mentors, and 

engage in activities that will set them up for future success in life and in school. 

In the Austin Independent 

School District (AISD), CIS staff 

work directly with students at 24 

elementary schools, 10 middle 

schools, 7 high schools, and one 

alternative school to help 

students and their families 

identify, gain access to, and navigate existing resources in the community and services 

most needed to help students succeed. 
CIS-AmeriCorps Evaluation 2020 4 

The CIS Model

To help achieve these critical 

goals, CIS staff use a specific 

model (see figure below) 

wherein program managers, 

trained in social services, work 

with students’ and their 

respective schools to support 

students’ needs. 

As a first step in this process, 

students are referred to CIS 

program managers by teachers 

or administrators at their 

respective school. After 

students are referred, CIS 

Program Managers receive parental consent, assess their students’ individual needs 

(academic or non-academic needs), and collaboratively develop a service plan to help 

them succeed. 

Figure 1. The CIS program model is centered on building relationships.

Source. Communities in Schools Central Texas

Program managers either directly provide their assigned students with services (such as 

counseling) or connect them with volunteers or external services through existing 
CIS-AmeriCorps Evaluation 2020 5 

partnerships such as support groups, tutoring, mentoring, or other enrichment 

opportunities. 

Many students – and their families – receive wraparound services to help them succeed 

in school and life. Although the main goal of CIS is to prevent high risk students from 

dropping out of school, CIS staff also help students achieve other positive outcomes

such as graduating, college and career attainment, increased attendance, and helping 

high risk students succeed. 

Additionally, CIS Central Texas provides specific programming to help meet their 

students’ needs such as AmeriCorps, ASPIRE Family Literacy (literacy services for the 

family), Care Coordination (wraparound family services), Pebble Project (child abuse 

prevention/education), SmartKids (afterschool programming public housing), and the XY 

zone (leadership program for high school males). 

Through these partnerships and programs, CIS staff effectively build relationships not 

only with students and families but also with the community including businesses, 

volunteers, and other partners. Doing so models relationship building from the top down 

ensuring all students’ needs are met – no matter how big or how small. Indeed, the 

success of CIS hinges on the relationships developed between CIS program managers 

and students, as well as between program managers and the community (see Figure 1

above). 

AmeriCorps Partnership

AmeriCorps is a network of national service 

programs, comprised of three primary programs 

taking a different approach to improving people’s 

lives and fostering civic engagement. Members 

commit their time to address critical community 

needs like increasing students’ academic 

achievement, mentoring youth, fighting poverty, 

sustaining national parks, preparing for disasters, 

and more. 

CIS of Central Texas is an AmeriCorps State and 

National grant recipient and has partnered with 

AmeriCorps for nearly 20 years. CIS of Central 

Texas recruits, trains, and supports nearly 80 fulltime, half-time and Education Award Only quarter-time AmeriCorps members every 

year. Members provide both 1:1 and small group (3:1) support to over 1,000 

economically disadvantaged students in public schools across central Texas. 

AmeriCorps members’ primary focus is providing mentorship and supportive guidance 

to students. In doing so, they become a trusted adult who students know they can share 
CIS-AmeriCorps Evaluation 2020 6 

successes and concerns with while at school. This relationship can then increase 

students’ sense of belonging, self-confidence, and motivation to do well in school. 

Members also provide tutoring and academic support to students, enrichment activities, 

and college and career guidance, among other activities.

In this report, we examine how AmeriCorps members impact student academics, 

behaviors, and perceptions of school. The report compares students served by an 

AmeriCorps member in Grade 4 and 5 in 2017-2018 to students from a carefully 

matched control group that attended schools that were not served by CIS during this 

time period.

Background Significance

In the 96 schools currently served by CIS of Central Texas (CISCT), over 46,000 

students are at-risk for school dropout because they have repeated one or more grades; 

failed the state standardized test; have limited English proficiency; are homeless or in 

foster care; are pregnant or parenting; or have been set back academically by other 

challenges. More than 52,000 students at these schools are economically 

disadvantaged (Texas Education Agency, 2017-18 Texas Academic Performance 

Reports). Students of color (particularly male), English Language Learners (ELL), and 

low-income students are overrepresented in dropout rates in Central Texas (E3 

Alliance, 2020). 

The individual and societal consequences of dropping out have been well-documented. 

Students who drop out before graduation are more than twice as likely to live in poverty, 

over three times more likely to be arrested, and eight times more likely to be 

incarcerated (National Center for Educational Statistics, 2011). While the graduation 

rates are improving nationally, in Central Texas disparities in graduation rates persist 

among students of color, low-income, English Language Learners, and students with 

disabilities (E3 Alliance, 2020).

CISCT provides year-round, dropout prevention services at 96 high-need schools in 7 

Central Texas Independent School Districts (ISD)-Austin, Bastrop, Elgin, Lockhart, 

Manor, Hays Consolidated, and San Marcos. CISCT targets schools in underserved 

communities with large percentages of at-risk and economically disadvantaged 

students. In 2019-2020 86.4% of CISCT students received free and price lunch; 25% 

were ELL; 10.3% had an incarcerated parent; 13.4% were in special education; and .6% 

were pregnant or parenting. 

 The CISCT theory of change is centered on providing students with opportunities to 

develop a positive and safe relationships with a caring adult. We believe that 

relationships are crucial to our work and that every student needs and deserves a oneon-one relationship with a caring adult. The CISCT AmeriCorps program plays a critical 

role in this model by providing supportive mentoring relationships to at-risk, 

economically disadvantaged students attending a CISCT supported school.
CIS-AmeriCorps Evaluation 2020 7 

Previous studies conducted by Agile Analytics examining the CIS 9th grade transition 

program have found that the more hours students spend in CIS-related activities, the 

better the outcomes (Hutson & Hawk, 2012). For example, CIS students who spent 

more time engaged in direct and indirect services had improved attendance (and fewer 

tardies), an increase likelihood of passing social studies, improved behavior, and 

increased promotion to the 10th grade.

Research Questions

The purpose of this report is to analyze the impact of working with AmeriCorps 

members on CIS students’ academic outcomes.

Working with CIS Central Texas staff, we developed three broad research questions to 

document the impact of the AmeriCorps program in 2017-2018. To analyze these key 

questions, we compared outcomes for 171 Grade 4 and 5 students served by 

AmeriCorps members and a statistically similar group of 171 matched non-CIS students

For more information on the matching process and for other methodological details, see 

Study Details: Methodology.

The research questions are based on the program model. We had three primary 

research questions:

Questions 

Question 1: Did CIS Grade 4 and 5 students 

have more academic improvement (grades and 

STAAR) in 2017-2018 compared to the 

matched control group?

Question 2: Did CIS Grade 4 and 5 students 

have behavioral improvements (attendance 

and discipline) in 2017-2018 compared to the 

matched control group?

Question 3: Were CIS students’ perceptions of 

school climate better than those of their peers?
CIS-AmeriCorps Evaluation 2020 8 

Key Findings 

Question 1: Academics

The first question explored changes in 

academics. First, student grades for 2017-

2018 in math, ELA, science, and social 

studies were compared between groups. 

Next, we compared students’ 2018 State of 

Texas Assessments of Academic Readiness 

(STAAR) Reading and Math performance by 

group.

Social studies grades were similar for both 

CIS and their matched non-CIS peers in 

2016-2017. However, the following year CIS 

students’ social studies grades not only 

improved over time but were also 

significantly higher than that of their matched 

non-CIS peers (Figure 1; t (315) = -1.9, p = 

0.056). 

When prior year social studies grade, prior STAAR score, and other demographic 

variables were controlled, students served by AmeriCorps members continued to have 

higher grades than their peers, (B = 1.91, p <.01). 

Figure 1. CIS Students served by AmeriCorps members had higher social studies grades 

than did their matched non-CIS peers. 

Source. 2016-2017 to 2017-2018 AISD Grades

Note. 2018 difference is significant, p = 0.056; difference was stronger (p < .01) when prior grades and other 

factors were controlled in a linear model.

Question 1: Did CIS Grade 4 and 5 

students have more academic 

improvement (grades and STAAR) 

in 2017-2018 compared to the 

matched control group?

Quick answer: Yes – in one area. 

Students served by AmeriCorps 

members had better grades in social 

studies. 

There were no significant differences 

between CIS students and their matched 

non-CIS peers in ELA, Math, or Science, 

or on the STAAR Reading and Math 

examinations.

Students who were served by an 

AmeriCorps member in 2017-2018 

had higher social studies grades 

than students in the matched 

control sample.
CIS-AmeriCorps Evaluation 2020 9 

There were no differences for other subjects, or when we examined 2018 STAAR 

performance. Detailed results are explored in Study Details: Results.

Question 2: Behaviors 

Next, we examined students’ attendance and 

disciplinary infractions to determine if service 

from AmeriCorps members predicted 

improvements in these areas.

We did not find any differences between groups.

• Overall attendance rates over time were 

consistent over time and between 

groups. 

• The number of disciplinary incidents was 

small and decreased for each group over 

time to near 0 for 2017-2018.

• For more information, see Study Details: 

Results.

Question 3: Climate 

Finally, we theorized that students’ perceptions of 

school climate in 2017-2018 might differ based on 

CIS participation. The Austin ISD climate survey 

was broken into three scales for our analyses:

• Perceptions of self

• Perceptions of other students

• Perceptions of teachers and staff

We found no differences in the climate data between 

students for any of the three categories. There was not 2016-17 climate data available 

to compare, so it is not possible to know if there were positive changes over time in 

these areas. For more details, see Study Details: Results.

Question 2: Did CIS Grade 

4 and 5 students have 

behavioral improvements 

(attendance and 

discipline) in 2017-2018 

compared to the matched 

control group?

Quick answer: No. We found 

no significant differences 

between CIS students and 

their matched non-CIS peers.

Question 3: Were CIS 

students’ perceptions of 

school climate better than 

those of their peers?

Quick answer: No. There 

were no differences between 

groups.
CIS-AmeriCorps Evaluation 2020 10

Conclusions & Recommendations 

CIS Central Texas focuses on providing direct and indirect services to the region’s most 

in-need students. The results from this outcomes evaluation showed a small 

improvement in social studies grades for the students served by AmeriCorps 

members, but no other differences emerged in terms of remaining subject grades, 

STAAR performance, behaviors, or perceptions of school climate.

AmeriCorps members spend only a small amount of time in academic 

support services.

As part of our analysis, we collected the number of service hours provided to each 

student and the categories in which that student was served. Students received 

services (both direct and indirect) in seven documented areas:

• Academic Enhancement & Support

• Case Management

• College & Career Readiness

• Enrichment Activities

• Health & Human Services

• Parental & Family Engagement

• Supportive Guidance

On average, each student was served by an AmeriCorps member for 31.2 hours. The 

majority of time spent (18.9 hours or 61%) was in supportive guidance. A very small 

amount of time, 2.4 hours or about 8% of the total participation time, was in receiving 

academic support and tutoring from members (see Figure 2, next page).
CIS-AmeriCorps Evaluation 2020 11

It is likely 

that supportive guidance and enrichment activities relate to academic achievement, but 

we suspect that this relationship is indirect. That is, students receiving supportive 

guidance will form a trusting and supportive relationship with an adult, gain confidence, 

and eventually improve their academic performance. However – we imagine that the 

academic improvement caused by this indirect relationship may take one or more years 

to manifest. Our first recommendation is to examine more years of academic and 

behavioral data in the next outcomes study.

Additionally, a brief analysis of internal data revealed that students who were referred to 

CIS because they did not meet assessment standards were not more likely to receive 

academic enhancement and support services than those who were referred for other 

reasons. Therefore, our second recommendation is to provide more intense 

academic support services for students who are referred for academic reasons. 

To enhance the tutoring services members can provide, CIS staff may want to pick a 

specific problem area (e.g., Grade 4 math) and provide intensive training to members 

on how to best help in that area. They should then monitor the service logs to ensure 

that students are receiving the additional academic support that they need. Then CIS 

Figure 2. Students received most support from CIS staff in programming 

related to supportive guidance.

Note. The “other” category includes case management activities, health and human services, and 

parental/family engagement services.

Source. 2017-2018 CIS internal service data

Academic enhancement support, 8%

College career awareness, 12%

Other, 16%

Enrichment activities, 22%

Supportive guidance, 61%

Service data suggest that 

AmeriCorps members spend 

the majority of the ~31 hours 

with students giving 

supportive guidance.

A very small amount of time 

(less than 3 hours per 

student, on average) is spent 

in direct academic support
CIS-AmeriCorps Evaluation 2020 12

staff can measure progress using their own BOY and EOY data to determine if the 

approach was successful. 

The data available from Austin ISD were unlikely to show large differences 

between groups.

Given that the majority of hours spent working with AmeriCorps members is in 

supportive guidance, improvements in PEIMS1-level academic and behavioral variables 

within one year is unlikely. A third recommendation is to identify outcomes more 

directly linked to supportive guidance and enrichment activities provided by 

AmeriCorps members. Refocusing outcomes and research questions to focus on the

specific activities CIS staff engage in most frequently with their students will more 

clearly delineate how these services relate to outcomes of interest.

To address this issue, we recommend the following ideas:

• Within Austin ISD, linked student climate survey data are available for the 2017-

2018 school year and beyond, so future studies could include both pre- and posttest climate data to assess changes over time. 

• In years where an impact/outcomes evaluation is not required, a closer look at 

subjective student experience using targeted focus group and internal survey 

questions would be helpful.

• Align research aims and objectives with the CIS logic model and/or program 

model. Focus particularly on short term outcomes

The two groups of students, those served by CIS and their matched nonCIS peers, may have been different from outset.

Although great care was made to ensure that both student groups – CIS and their 

matched control non-CIS served peers – were statistically similar prior to analyses, it is 

quite possible that these two groups of students were quite different from each other. 

Conversations with CIS program staff corroborated this assertion stating that many 

students selected to participate in CIS services were referred to CIS based on needs 

that are not measured in PEIMS data (e.g., homelessness, disengagement with school, 

low self-esteem, extreme behavioral problems, family conflict/abuse, etc.). Many of 

these characteristics are not measurable and are likely influence the outcomes 

examined in this study including attendance, academic achievement, and perceptions of 

school climate.

1 PEIMS is the Public Information Management System of Texas. All data that are submitted to the state 

are collected in a standardized format and are what are usually available for evaluation and research 

studies.
CIS-AmeriCorps Evaluation 2020 13

Limitations and Future Research

While this study helped shed light on students served by AmeriCorps members within 

CIS Central Texas, there were some limitation of this study. For example, we were 

unable to link students served by AmeriCorps back to their direct mentoring data

to determine which specific mentoring services were most beneficial to students. 

Gaining access to this more granular level of data will help identify these best practices 

and also remove students who only received a few mentoring services. This refined 

sample might speak more directly to the true experiences of students served by CIS 

Central Texas.

Relatedly, a factor we were unable to include in our analyses was examining

AmeriCorps member type (e.g., full or half-time) to determine if there is a difference in 

student outcomes based on the amount of time members work with students. For 

example, we might see more increases in student outcomes for those students paired 

with a full-time mentor since full time mentors more immersed in the school, more able 

to build connections and relationships with students, and engage with the community.

Finally, expanding the number of years examined in the study will allow for more 

time to see the full effects of AmeriCorps members’ mentoring. Many effective 

dropout intervention programs work with students from middle school through high 

school (Wang & Fredricks, 2014; Hutson & Lamb, 2020); therefore, following students 

for a longer period of time will more likely show improvements in their experiences in 

school. For example, a recent evaluation of a similar program in Central Texas gathered 

data from participating students for a 7-year time period (middle school through 

graduation), and as a result found positive results (Hutson & Lamb, 2020). Using a 

similar type of methodology will likely benefit the overall evaluation of CIS. 

In the next section, we provide details on the methodology of the evaluation and the 

outcomes that were not significant.
CIS-AmeriCorps Evaluation 2020 14

Study Details 

Methodology 

In this section of the report, we describe in more detail 

the analyses and results conducted for this evaluation. First, 

we will explain in more detail our methods for creating our 

sample. Ultimately, we conducted a quasi-experimental design 

with propensity score matching (see sidebar).

The sample. As described earlier in the report, in order to 

examine the impact of AmeriCorps members on academic and 

behavioral outcomes for CIS participants within AISD, we 

collected student-level data from the 2016-2017 school year 

for students enrolled in grades 3 through 5. We used this data 

to create a group of CIS students and a group of similar 

matched students who were not enrolled in CIS. 

Propensity Score 

Matching (PSM)

Propensity Score Matching 

(PSM) is used when a true 

treatment and control group are 

difficult or impossible to obtain in 

a study. PSM uses existing data 

to create a one-to-one match to 

create treatment and control 

groups that are similar.

Step 1: The match. CIS

students were matched to nonCIS students on gender, 

ethnicity, free/reduced lunch 

status, and 2016-2017 STAAR 

math/reading scores.

Step 2: Baseline equivalency.

Initially, there were differences 

in groups. After reexamining the 

data, we removed students who 

attended a CIS school in 2017-

2018 but did not in 2016-2017. 

After rebalancing the data, we 

confirmed that there were no 

differences between the CIS

students and the matched 

control students on any 

demographic or prior 

achievement variables. 

Step 3: Analyze groups for 

post-intervention differences. 

We assessed differences 

between the groups in 2017-

2018 data including grades, 

discipline, attendance, STAAR 

performance, using linear 

models that included 

demographic variables and prior 

achievement. 
CIS-AmeriCorps Evaluation 2020 15

We selected a quasi-experimental research design 

utilizing propensity score matching for several reasons

because they generally considered to be among the most 

robust available aside from Randomized Control Trial 

studies (which were not able to be conducted in this case 

for ethical and practical reasons). In a PSM study, pretest variables are carefully matched, and data are checked 

to ensure that groups are statistically similar at baseline. 

Differences at post-test can thus be inferred as changes 

caused by program participation. Further, in our design, 

we created linear models that controlled for pre-test 

scores and certain demographic / prior year data. These 

steps ensured that our findings were as reliable as 

possible.

Specifically, we matched students in grades 3 through 5 

based on their 2016-2017 demographic information (e.g., 

Hispanic/non-Hispanic, free/reduced price lunch status, 

STAAR reading and math, attendance, and disciplinary 

referrals; see sidebar on previous page). Notably, students 

in the comparison group could not have attended a CIS 

school. In this way we could determine the overall effects 

of CIS on students’ experiences in school among students 

with similar backgrounds. After we received the data from 

AISD, we computed Propensity Score Matching (PSM) 

using the Nearest Neighbor matching procedure (Rassen, 

Shelat, Myers, Glynn, Rothman, & Schneeweiss, 2012).

Balancing. Once our sample was created, we 

computed statistical checks to ensure both 

groups were balanced, or equivalent to each 

other in terms of background characteristics such as race, 

ethnicity, free/reduced lunch status, and 2016-17 grades, 

STAAR performance, and behaviors, prior to conducting

analyses using the 2017-2018 data2 (all difference effect 

sizes <.25, with the majority < .05). Doing so would allow

us to have more confidence in stating that any differences 

between the groups in 2017-2018 was because of 

engaging with CIS, rather than due to chance alone. 

2 Baseline equivalency is defined by the What Works Clearinghouse guidelines as a very small difference between groups, with the 

difference effect size being 0.05 or less. Slightly larger differences of 0.06 to 0.25 are acceptable as long as the variables with these 

differences are included in all models. Groups that have differences with effect sizes >0.25 are not considered equivalent. 

The data

The following de-identified 

student-level data elements 

were obtained from AISD’s 

Department of Research and 

Evaluation (DREA unique study 

ID was assigned to each student 

to ink disparate data sources.

Demographics: 2016-2017 and 

2017-2018 de-identified studentlevel gender, ethnicity, free and 

reduced-price lunch status, at 

risk status (a composite 

indicator computed by the Texas 

Education Agency), special 

education status, and limited 

English proficiency status. 

Behavior: 2016-2017 and 2017-

2018 de-identified student-level 

attendance and discipline data.

Grades: 2016-2017 and 2017-

2018 de-identified student-level 

grades in English, Math, 

Science, and Social Studies.

STAAR: 2016-2017 and 2017-

2018 de-identified student-level 

STAAR performance data for 

English, Math, Science, and 

Social Studies.

Student climate: 2017-2018 

student-;level responses to 

AISD’s Student Climate Survey 

(conduected by AISD’s DRE). 

More information is on page25.
CIS-AmeriCorps Evaluation 2020 16

Rebalancing. After balancing the data, we ran descriptive statistics to 

determine that our samples were similar. Next, we joined this data set back to 

students’ remaining data requested from AISD, including additional 2016-2017 

demographic information including gender, at risk status, SPED stats, LEP status, gifted 

and talented participation, and grades (reading, math, science, and social studies). 

Additionally, we received AISD data from 2017-2018 including grades, STAAR reading 

and math, attendance, disciplinary infractions, and students’ ratings of school climate. 

We again ran analyses to ensure our groups (e.g., CIS and non-CIS students) were 

statistically similar. Unfortunately, we found that many non-CIS students enrolled in 5th

grade in 2016-2017 went on to attend a middle school offering CIS services in 2017-

2018. As a result, these students had to be removed from our sample to ensure that we 

were matching CIS to students who were never served by CIS. Doing so is critical 

because we know that many CIS students are identified because they are experiencing 

specific needs including family conflict, homelessness, and academic issues that many 

students do not experience. This means that CIS students – even if they are attending 

the same school as non-CIS students – are likely quite different than their non-CIS 

peers. It should also be noted that CIS of Central Texas focus their services on the 

lowest performing schools, also limiting the sample. For these reasons, we decided to 

exclude students enrolled in 6th grade in 2017-2018 from our final sample. 

After the balancing and rebalancing, our final sample included 171 CIS students in 4th

and 5th grades who were expertly matched to 171 similar peers who were not served by 

CIS, making a total of 342 students in the sample. Importantly, after rebalancing the 

data, we found that – after including additional students from which to draw a sample 

from – the final sample was well balanced in terms of grades and all other variables of 

interest (effect sizes < .025; Table 1).

Table 1. After re-balancing and removing 6th grade students from the sample, CIS and 

matched non-CIS students were demographically similar. 

Freereduced 

price lunch

% Hispanic % LEP % SPED 

status % At risk

CIS 

(n = 171) 96% 78% 47% 16% 68%

Non-CIS

(n = 171) 96% 78% 50% 20% 68%

Source. 2016-2017 AISD demographic data
CIS-AmeriCorps Evaluation 2020 17

Results

This section outlines in more detail results from our analyses. To analyze the effects of 

AmeriCorps members on CIS students’ experiences in school in the 2017-2018 school 

year, we created linear models to explore the potential relationship between CIS 

participation and students’ academic achievement, behaviors, and attitudes/perceptions 

about school. Specifically, our linear models included the following 2016-2017 variables 

(all effect sizes >.05): 

• math, reading and science grades in 2016-17, 

• special education status, 

• number of disciplinary infractions in 2016-17, and 

• LEP status. 

Additionally, we included variables in our models if they were significantly related to the 

outcome of interest (e.g., social studies grades in 2016-2017 were included in the model 

predicting 2017-2018 social studies grades). We computed 11 individual regression 

models examining student outcomes. We first describe analyses relating to subject-level 

grades and STAAR performance at the elementary school level. 

Elementary School Grades 

Math. In 2016-2017, non-CIS students’ grades were slightly (but not statistically 

significantly) higher than their CIS peers, and although both groups improved, non-CIS 

students’ grades continued to be slightly higher than that of their CIS peers (Figure 3; t

(314) = 1.99, p < .05). However, when a regression model was created that controlled 

for demographic variables and prior math achievement (prior math grade, prior STAAR 

performance), there was no longer a significant difference between CIS and non-CIS 

students.

Figure 3. At both time points, math grades for CIS students were lower than that 

of their matched non-CIS peers. Change over time was similar for both groups.
CIS-AmeriCorps Evaluation 2020 18

Source. 2016-2017 to.2017-2018 AISD math grade data

Note. * p<.05; differences are significant across group, within year

English Language Arts (ELA). Similar to math, elementary school students 

participating in CIS had lower ELA grades in 2016-2017, and although they experienced 

a slight increase in grades, matched non-CIS students also experienced an increase in 

ELA grades in 2017-2018 (Figure 4). These differences, however, were not statistically 

significant. Additionally, there were no significant differences in the regression models 

created.

Figure 4. Students participating in CIS and their matched non-CIS peers had 

similar ELA grades in 2016-2017 and 2017-2018.

Source. 2016-2017 to.2017-2018 AISD ELA grade data

Science. Finally, students’ science grades were similar for both CIS students and their 

matched non-CIS peers (Figure 5). 
CIS-AmeriCorps Evaluation 2020 19

Figure 5. In both 2016-2017 and 2017-2018, participating CIS students and 

their matched non-CIS peers had similar science grades in 2016-2017 and 

2017-2018.

Source. 2016-2017 to.2017-2018 AISD science grade data

STAAR 

Next, we conducted a similar set of analyses to determine the effects of AmeriCorps 

members influence on CIS on students’ STAAR reading and math performance in 2017-

2018. To do so, we created and analyzed linear models that included the following 

2016-2017 variables (all effect sizes >.05): STAAR math and reading, special education 

status, number of disciplinary infractions, and LEP status on STAAR Percentile.

STAAR Percentile was calculated using frequency distribution data from the Texas 

Education Agency.3 Frequency distributions were computed using a formula available 

Our results found no differences between groups, either in the t-test models or in the 

linear models (Figures 6 and 7). 

Figure 6. While not significant, participating CIS students had lower STAAR 

math scores at both time periods than did their matched non-CIS peers.

3 https://tea.texas.gov/student-assessment/testing/staar/staar-frequency-distributions
CIS-AmeriCorps Evaluation 2020 20

Source. 2016-2017 to.2017-2018 STAAR math percentiles

Figure 7. Although not significant, participating CIS students experienced an 

increase in STAAR reading performance in 2017-2018 compared with their 

matched non-CIS peers.

Source. 2016-2017 to.2017-2018 STAAR reading percentiles

Attendance & Discipline
CIS-AmeriCorps Evaluation 2020 21

Next, we examined students’ attendance rates and disciplinary infractions over time 

based on group. As mentioned previously in the report, we found no significant 

differences based on CIS group. Attendance levels remained high for both groups 

(Figure 8). 

In addition, disciplinary infractions were few and as a result we were unable to analyze 

group differences over time.

Climate

Finally, we conducted analyses to determine the effects of AmeriCorps members 

influence on CIS on students’ perceptions of school climate in 2017-2018 using the 

Austin Independent School District’s (AISD) student climate survey. Each year, the 

AISD’s Department of Research and Evaluation (DRE) asks students in grades 3-11 to 

complete a survey that provides district staff and key stakeholders with feedback 

regarding their perceptions of and experiences with school climate. Beginning in 2017-

2018, the survey was administered online and linked to students’ individual records. As 

a result, Agile Analytics was able to obtain de-identified student-level data for those 

students in our sample. Students’ responses on the survey ranged from 1-4 (strongly 

disagree to strongly agree). For more information on AISD’s student climate survey, 

please click here (after clicking, please select “interactive reports” and then “student 

climate survey”). 

Utilizing a factor analysis technique, items from the student climate survey were divided 

into three small scales:

• Perceptions of self

o I use ways to calm myself down. 

o I don’t give up even when I feel frustrated. 

Figure 8. Both CIS and matched non-CIS students had similar attendance rates in 

2016-2017 and 2017-2018.

Source. 2016-2017 to.2017-2018 AISD attendance rates
CIS-AmeriCorps Evaluation 2020 22

o I can do even the hardest schoolwork if I try.

o I enjoy doing my schoolwork.

o During my school day, I am aware of when my feelings change.

o I try hard to do my best work.

o I feel successful in my schoolwork.

o During my school day I am aware of when my thoughts change from 

positive to negative.

o I can reach the goals I set for myself. 

o My homework helps me learn things I need to know.

o During my school day, I am aware of how my body feels when my feelings 

change.

o I receive recognition or praise for doing good work.

o If I get angry with a classmate, we can talk about it and make it better.

• Perceptions of other students

o My classmates show respect to each other.

o My classmates show respect to other students who are different. 

o I am happy with the way my classmates treat me. 

o My classmates behave the way my teachers want them to.

o At my school, students take care of each other.

• Perceptions of teachers and staff

o Teachers at this school care about their students.

o Adults at this school listen to student ideas and opinions.

o Adults at this school treat all students fairly.

o The staff in the front office show respect to students.

o At my school, there is respect for different cultures.

o At my school, there is respect for students who speak languages other 

than English. 

o My teachers are fair to everyone. 

o My teachers expect me to think hard about the things we read.

o My teachers expect everybody to work hard.

o Teachers at this school know who I am.

o My teachers know what I am good at.

Results yielded no significant differences between groups. Data for 2016-17 were not 

available for comparison; in future years these data will be available and may make for 

interesting analyses.

 
CIS-AmeriCorps Evaluation 2020 23

References 

Huston, A., & Lamb, L. (2020). Breakthrough Central Texas: Austin Independent School 

District (AISD) Impact report. In preparation. 

Hutson, A., & Hawk, C. (2012). 2011 Service Log Analyses. Prepared for Communities 

in Schools.

Rassen, J. A., Shelat, A. A., Myers, J., Glynn, R. J., Rothman, K. J., & Schneeweiss, S. 

(2012). One-to-many propensity score matching in cohort studies. 

Pharmacoepidemiol Drug and Safety. (2) 69-80.

Texas Education Agency (2020). Converting a STAAR Scale Score to a Percentile. 

Retrieved from

https://tea.texas.gov/WorkArea/linkit.aspx?LinkIdentifier=id&ItemID=25769806946 on 27 

Oct 2020. 

Texas Education Agency (2018). STAAR Frequency Distributions. Retrieved from

https://tea.texas.gov/student-assessment/testing/staar/staar-frequencydistributions on 27 Oct 2020.

Wang, M. Fredricks, J. (2014). The Reciprocal Links Between School Engagement, 

Youth 

Problem Behaviors, and School Dropout During Adolescence. Child 

Development, 85 (2), 722-737.

 
CIS-AmeriCorps Evaluation 2020 24

About the Evaluators 

Agile Analytics is a woman-owned boutique research and evaluation firm. We work with 

a select group of non-profit organizations that improve people’s lives through innovative 

programs.

We use advanced quantitative and qualitative techniques to measure the impact 

programs have on the people they serve, and to find areas of opportunity for growth.

Our goal is to empower organizations to use their data to improve their programs, and 

ultimately to change people’s lives for the better.

Andrea Hutson, Ph.D. is the founder and President of Agile Analytics. She has a Doctorate in 

Developmental Psychology from the University of Texas at Austin, with minors in Statistics and 

Cognition. Andrea founded Agile Analytics with the focus on the improvement of education for 

students in grade PK-16 in August of 2011. 

Lindsay Lamb, Ph.D. is the Director of Evaluation for Agile Analytics and has a Doctorate in 

Educational Psychology from the University of Texas at Austin within the Human Development 

and Education division with a Master of Arts focusing on Quantitative Methods. Prior to joining 

Agile Analytics, Lindsay worked in the Austin Independent School District’s (AISD) Department 

of Research and Evaluation for nearly 10 years, most recently as an Evaluation Coordinator for 

Social and Emotional Learning (SEL).

Acknowledgements 

We are incredibly grateful for the assistance of Jay Brown, Kaylee Lawrence, Jessi 

Engelke in the preparation of this manuscript.