August	2015

AmeriCorps	 	 	 	 	 at	CIS of Central Texas:

Year 3	Impact Analysis
	 	

	 	

	 	

	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	

	 	

	 	

	 	

2

Table	of Contents

Introduction...................................................................................................................... 3

Research	Questions.......................................................................................................... 4

Methodology .................................................................................................................... 5

Question	1:	What	are	characteristics	of	AmeriCorps	members?................................... 6

Service	Type ................................................................................................................................... 6

Gender and Race/Ethnicity............................................................................................................ 6

Education	Level.............................................................................................................................. 6

Question	2:	What	are	characteristics	of	students	served	by	AmeriCorps	Members? . 7

Background	Characteristics / Demographics ................................................................................7

Gender.........................................................................................................................................7

Race & Ethnicity...........................................................................................................................7

Campuses ....................................................................................................................................7

Grade ...........................................................................................................................................8

Because of these differences in	population, all outcome analyses conducted	were conducted	

first with all students, and then with Elementary School students only. ...................................8

Living Situation ...........................................................................................................................9

School Attendance .........................................................................................................................9

Disciplinary incidents ......................................................................................................................9

Course Grades.................................................................................................................................9

Question	3:	How	do	outcomes	for	students	served	by	AmeriCorps	members

compare	to	those	not	served	by	the	program?..............................................................11

CIS Data ..........................................................................................................................................11

Hours Served ..............................................................................................................................11

Recommendation	to	continue services......................................................................................11

Documented improvement in Key	Areas ...................................................................................11

Attendance................................................................................................................................ 13

Discipline ................................................................................................................................... 13

Grades........................................................................................................................................ 13

STAAR.........................................................................................................................................15

Question	4:	What	is	the	relationship	between	the	amount	and	type	of	services	

students	receive	and	outcome	variables?......................................................................16

Hours of Participation ..................................................................................................................16

Number of Sessions...................................................................................................................... 17

Member Status ............................................................................................................................. 17

Question	4:	When	pre-test	differences	are	controlled,	are	there	any	significant	

outcome	differences	of	students	served	by	AmeriCorps	vs	not?................................18

Sample	Characteristics .................................................................................................................18

Outcomes......................................................................................................................................19

Conclusion....................................................................................................................... 20

AmeriCorps	Year	3	Report
	 	 	

	 	 	

3

Introduction

Established	in 1985,	Communities	In	Schools	of	Central	Texas	(CIS)	is	a	dropout	

prevention	program that encourages students	to	stay	in	school	through	campusbased	programs offering a	network	of	volunteers,	social	services,	and business	and

community	resources.	The	mission of	the	organization is	to provide	a support system	

to	at-risk	students,	encouraging	them	to	be	successful	in	school.	Its	parent	

organization	describes itself	as	the	“nation’s	largest	and	most	effective	organization

for	keeping	kids	in	school	and	helping	them	succeed	in life.”

The	AmeriCorps	program	“engages	more	than	75,000	Americans	in	intensive	service	

each	year	at	non	profits,	schools,	public	agencies,	and	community	and	faith-based

groups	across	the	country” Members	dedicate	one	year	to	the	program,	and,	in

return,	receive	money	for	college,	work	experience,	and	(at	times)	a	small	living	

stipend.

The	AmeriCorps	program	directly	supports	CIS’s	efforts	by	providing	members	who	

mentor	and	counsel	students.	 Almost one	hundred	young	people	serve	CIS	of	

Central	Texas	each	year	through	the	AmeriCorps	program	at	over	two	dozen	

campuses,	working	with	over	1200	students.	Members	serve	fullr time,	partr time,	

or	quarterr time,	working	directly	with	students	as	mentors	and	tutors.	

The	purpose	of	this	evaluation	report	is	to	detail	the	findings	of	an	impact	analysis	

examining	the	unique	effects	of	working	with AmeriCorps	members.	Specifically,	we	

were	most	interested to	see	if	students	served	by	AmeriCorps	members	would	have	

better	outcomes	compared	to	students	served	by	other	CIS	staff	that	were not	

AmeriCorps	members.
 

 

 

 

 

 

 

 

 

 

 

 

 

 

4

Research	Questions

This	analysis	was	formed	around	five distinct	Research	Questions.	 These	questions,	

listed	below,	are	each	addressed	in turn in	this	report.

1. What were the characteristics of AmeriCorps 

members in 2013-2014? 

2. What are characteristics of students served by 

AmeriCorps members? How are they 

similar/different to other students served by CIS? 

3. How do outcomes for students served by 

AmeriCorps members compare to those not served 

by the program? 

4. Do students in AmeriCorps improve over time? Is 

there a relationship to the number of hours served, 

or any student characteristics? 

5. When pre-test differences are controlled, are there 

any significant outcome differences of students 

served by AmeriCorps vs not? 

AmeriCorps	Year	3	Report
	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	

	 		

	 	 	

	 	 	

	 	

	 	 	 	

	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	

	 	

	 	 	 	 		

	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	

Methodology

Data	on	CIS	students	and	members	from	the	2013-2014 school	year were obtained.	

We	obtained a variety of data on these students including the items listed in the	box	

below.

We were primarily

interested in	

discovering if	there	

were	differences	in

outcomes	of	

students	who	were	

served	by	an	

AmeriCorps	member	

compared	to	those

who	were	served	by

other	CIS staff.	

Our	previous	

findings (Hutson et

al.,	2014)	indicated

that	students	that	

were	served	by	

AmeriCorps	

members	are	

typically targeted for

academic issues.	

Therefore,	we	were	

certain	students	at	

the	same	campus		

who	were	served	by	

AmeriCorps	

members	and those

who were not would

not be comparable.	

Our solution to	this	

5

Data obtained for CIS Students

Demographics/Background Information:

Gender

Ethnicity/Race

Does student live at home?

What is the family structure at home?

Does the student have an incarcerated parent?

Outcomes Information:

Grades: (Pre, 6th Six Weeks, Post)

• Math

• Science

• Social Studies

• Reading

Attendance

Tardies

STAAR Assessment

• Math

• Science

• Social Studies

• Reading

• Writing

Improvement in various areas, including:

• Social Skills

• Testing

• Basic Needs

• Classroom Participation

problem was to compare students working with AmeriCorps members to students at

campuses that did not have AmeriCorps members.	 We hoped that this sampling

method would create two groups that were more similar than dissimilar at pre-test.	
	 	

	 	 	

	 	

6

Question 1: What	are characteristics of	AmeriCorps members?

Our	first	goal	was	to	describe the	characteristics	of	AmeriCorps	members	that	served	

students	in	2013-2014.	We	obtained	data	on	service	type,	gender,	race,	ethnicity,	and	

education	level	for	all	members.

Service	Type

There	were	101 members	in the	data set	for	the	2013-2014	school	year.	Members	had

the	option	to	serve	 full-time,	part-time,	or	quarter-time through an	Education	Only	

Award	(EAO).	The	majority	of	members	were	EAO	and	served	quarter-time.

# %

Full	Time 30 29.7%

Part	Time 19 18.9%

EAO 52 51.4%

Gender and	Race/Ethnicity

The	majority	of	members	were	female	(91	of	101,	or	90%).		Most	were	also	white	

(79.2%).		About	a	quarter	of	members (24.8%)	were	Hispanic.		

Education	Level

Members	were highly	educated,	overall.	92%	had	attended	at	least	some	college,	and	

over	half	were	college	graduates.	

High	School	Graduate	

Associates	Degree	

Some	College	

College	Graduate	

Some	Graduate	School	

Graduate/Professional	Degree	

1	

8	

34	

36	

16	

6	

AmeriCorps	Year	3	Report
	 	 	 	 	 	 	 	 	

	 	

7

Question 2: What are characteristics of students served by

AmeriCorps Members?

Next,	we	wanted	to	examine	the	characteristics	of	students	served	by	AmeriCorps	

members.	Even	more	importantly,	these	characteristics	should	be	compared	to	those	

students	who	were	not	served	by	AmeriCorps	members	to	see	if	our	sample	if	

equivalent.

Background	Characteristics	/ Demographics

There	were	2584	students	in the	sample.	1295	(50.1%)	were	served	by	AmeriCorps	

members,	and	1289	were	not.	

Gender

There	were	slightly	more	girls	than	boys	served	by	CIS;	approximately	53%	(vs	47%).	

There	were	no differences	in gender	for	students	served	by AmeriCorps	or	not.

Race	& Ethnicity

There	were	significant	differences	in	race/ethnicity	by	group.	Students	served	by	

AmeriCorps	members	were	more	likely	to	be	Hispanic	or	White and	less	likely	to	be	

Black	than	students	that	were	not	served	by	AmeriCorps	members.

NonAmeriCorps AmeriCorpsSignificant	

difference?

Asian/Pacific	Islander 0.2% 0.5% NO

Black/African	American 36.0% 25.9% YES

Hispanic/Latino 59.5% 68.4% YES

White/Caucasian 5.6% 7.4% YES

Campuses

Students	in	the	sample	were	from	six	school	districts	in	Central	Texas.	The	majority	of	

students	in	both	groups	were	from	Austin	ISD,	but	students	were	also	enrolled	in Del	

Valle	ISD	(non-AmeriCorps	only),	Hays	(AmeriCorps),	IDEA	(AmeriCorps),	and	

Lockhart	(primarily	non-AmeriCorps;	a	few	AmeriCorps	students	attended	Lockhart	

Junior	High)	
8

NonAmeriCorps AmeriCorps

Austin	ISD 807 1000

Del	Valle	ISD 207 0

Hays	ISD 0 178

IDEA	 0 12

Lockhart ISD 95 8

Manor ISD 184 100

Grade

There	were	large	differences	between	students	served	by	AmeriCorps	and those

served	by	other	CIS staff	by	grade	level. The	sample	of	students	was	drawn such	that	

there	were	no	schools	that	had	both	students	served	by	AmeriCorps	members	and	

those	not	served	by	AmeriCorps	members.	Doing	so	meant	that	a	large	number	of	

non-AmeriCorps	students	were	from	Elementary	and	High	school,	while	the	majority	

of	students	in	served	by	AmeriCorps	were	in	Middle	school.

Differences	in	grade	level	by	condition.	The	majority of	students	served	by

AmeriCorps	were	in	Middle	School,	followed	by	Elementary	and	then High

School.	The	majority	of	students	in	our	sample	that	were	not	served	by	

AmeriCorps	were	in	Elementary	school.

non-AmeriCorps	 AmeriCorps	

56%

10%

34%

32%

60%

8%

Elementary	

Middle	School	

High	School	

Because	of	these	differences	in	population,	all	outcome analyses	conducted	were	

conducted	first	with	all	students,	and	then with	Elementary	students only.

AmeriCorps	Year	3	Report
		

	 	 	 	

9

Living	Situation

Does	the	student	live	at	home	or	with	another	family	member,	friend,	in	a	foster	

home,	or	by	themselves?		Slightly	more	students	served	by AmeriCorps	lived	at	home

than	those	not	served	by	the	program (92.4%	vs	88.4%).

Students	were	also	asked	to	indicate	who	they	lived	with	(parents,	relatives,	etc.).	

There	were	significant	differences	in	students	who	lived	with	two	parents,	a	single	

parent,	or	with	non-parents	(e.g.,	with	another	relative	or	in	a	foster	home).		

Students	working	with	AmeriCorps	members	were	more	likely	to	live	with two	

parents,	and	those	that	did	not	were	more	likely	to	live	with	a	single	parent	or	

without	a	parent	in	the	household.

Non- Significant	

AmeriCorps AmeriCorps difference?

No	Parents 8.3% 4.9% YES

Single	Parent 48.3% 43.1% YES

Two	Parents 41.5% 50.8% YES

Additionally,	students	served	by	AmeriCorps	members were	twice	as	likely	to	have	an

incarcerated	parent	(8.9%	vs	3.6%).	

School Attendance

Information on absences and	tardies were	available	at	both	pre-and	post	test.	

Students	in	served	by	AmeriCorps	members	had	fewer	absences	at	pre-test	than	

those	not	served	by	AmeriCorps	(1.68	vs	2.23) for	both	the	entire	sample	and	for	the	

elementary	only	sample	(1.34	vs	1.88).		Both	groups	had	about	the	same	number	of	

tardies	(2.85)	in	the	entire	sample,	but	in	the	elementary	sample,	students	that	were	

served	by	AmeriCorps	members	had	fewer	tardies	(1.22	vs	1.73).

Disciplinary incidents

The	number	of	disciplinary	incidents	for	both	groups	was	very	small.		Students	served	

by	AmeriCorps	members	had	slightly	fewer	incidents	at	pre-test	on	average	in	both	

the	entire	sample	(0.30	vs	0.53)		and	the	elementary	sample	(0.08	vs	0.26).

Course	Grades

We	were	able	to	obtain	data	on	grades	in	English	Language	Arts	(ELA),	Math,	

Science,	Social	Studies,	and	Other	classes.	
10

Students	had	similar	grades	at	pre-test	in	Math,	Science,	and	Social	Studies,	but	

students	who	were	not	served	by	AmeriCorps	members	tended	to	have	higher

grades	in	English/Language	Arts	and	Other	subjects.	

ELA/Reading

Math

Science

Social	Studies

Other

All	Students

NonAmeriCorps AmeriCorpsSig.	

diff?

80.34 79.25 YES

79.24 78.38 NO

82.21 82.20 NO

82.89 82.69 NO

84.64 79.84 YES

Elementary	Only

NonAmeriCorps AmeriCorpsSig

diff?

80.95 79.55 YES

79.95 77.99 YES

83.69 83.36 NO

85.51 87.22 YES

78.67 80.83 NO

AmeriCorps	Year	3	Report
	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	

		

	 	

11

Question 3: How do outcomes for students served by

AmeriCorps members compare to those not served	by the

program?

We	know	from	the	previous	section	that	students	who	were	served	by AmeriCorps

members	and	those	who	were	served	by	other	CIS	staff	had	large	pre-existing

differences.	We	wanted	to	see	if	working with	AmeriCorps	members	caused the

differences	between the	groups	to lessen	or	disappear. We	were	also	interested	in

seeing	if	students	served	by	AmeriCorps	members	had	greater	academic	growth	than	

students	served	by	non-members.

CIS	Data

Hours	Served

On average,	students	received	services	for about	47	hours	in	the	2013-2014	school	

year. There	were	no differences	between students	served	by	AmeriCorps	and	those	

that	were	not	when	we	examined	all	students. However,	when	only	Elementary	

students	were	examined,	we	found	that students	served	by	AmeriCorps	members	

received	about	6	fewer	hours	of	services	(35.8	hours	vs.	42.0).	

Recommendation	to	continue	services

Teachers	were	asked	if	each	student	who	participated	in	CIS	should	continue	

receiving CIS	services for	the	next year. About	the	same	number	of	teachers	returned	

the	form	for	each	group	(96.2%	overall). Teachers	of	students	who	were	served	by	

AmeriCorps	members	were	significantly	more	likely	to	say	“yes”	to	this	question,	

91.7%	vs.	79.8%.		In	the	Elementary	sample,	the	same	trend	held,	although	the	

difference	was	smaller	– 94.2%	vs	86.9%

Documented	improvement	in	Key	Areas

CIS	Program	staff	take	copious	notes	on	each	of	their	students	and	document,	by	

end-of-year,	progress	in	these	areas.	Results	are	listed	in	the	table	on	the	next	page.

Importantly,	students	only	had	progress	noted	in	those	areas	that	were	of	concern

for	them.	 Students	who	were	served	by	AmeriCorps	members	made	fewer	gains	in

Social	Skills,	Basic	Needs,	Emotional	Crisis,	Career/Employment,	and	Health	than

those	served	by	non-members.
	 	 	 	 	

12

CIS Outcomes by Prevalence and Improvement No

n-Am

eriC

orps

Am

eriC

orps

Significa

nt

differe

nce?

N

Social Skills 2591 3.57 2.93 YES

Academic Readiness 1255 3.84 3.77 NO

Self Esteem 1175 3.88 3.82 NO

Basic Needs 437 3.24 2.80 YES

Classroom Conduct 372 4.56 4.62 NO

Grades 345 3.29 3.43 NO

Family Crisis 344 3.57 3.73 NO

College Readiness 276 2.78 2.84 NO

Absences 249 3.17 3.07 NO

Mental Health 201 2.91 2.97 NO

Tardies 195 3.36 3.19 NO

Homework Completion 178 3.50 3.54 NO

Classroom Participation 118 3.66 3.71 NO

Emotional Crisis 117 3.94 3.61 YES

Career / Employment 105 1.82 1.53 YES

Life Skills 82 2.99 2.97 NO

Grief / Death 76 1.78 1.97 NO

Testing 64 3.3 3.32 NO

Language (ESL/LAP) 64 2.05 1.91 NO

Health 50 3.89 3.40 YES

Delinquent Conduct 36 3.62 3.00 NO

Housing 29 2.25 1.80 NO

Suspected Substance Abuse 12 2.38 2.00 NO

Violence 8 1.50 3.00 NO

Suspected Gang Involvement 6 2.00 2.00 NO

Day Care 1 NA 5 NO

AmeriCorps	Year	3	Report
	 	

13

Attendance

Both	groups	of	students	had	similar	absences	at	the	six-week	mark.	Whereas	at	the	

beginning	of	year	there	was	a	significant	difference	in	the	number	of	absences	of	

students	served	by	AmeriCorps	members	and	those	that	were	not,	by	end	of	year,	

this	difference	was	not	significant	for	either	the	entire	sample	or	the	Elementary	

sample.

There	were	significant	differences	at	the	six	week	mark	and	end	of	the	year	for	

tardies,	even	though	there	were	not	significant	differences	at	beginning	of	year.		

Students	served	by	AmeriCorps	members	had	significantly	more	tardies: by	end of	

year,	almost	6	more,	on	average,	than	those	not	served	by	AmeriCorps	members.

There	were	no	significant	differences,	however,	in	the	elementary	sample.

NonAmeriCorps AmeriCorpsSignificant	

difference?

Absences:	Pre-Test 2.23 1.68 YES

Absences:	Six	Weeks 2.85 2.67 NO

Absences:	Final 9.97 9.20 NO

Tardies:	Pre-Test 2.87 2.84 NO

Tardies:	Six	Weeks 2.52 3.67 YES

Tardies:	Final 8.43 14.34 YES

Discipline

The	number	of	disciplinary	incidents	at	end-of-year continued	to	be	lower	for	

students	served	by	AmeriCorps	members	and	was	about	half	that	for	students	not	

served	by	AmeriCorps	members,	0.52	incidents	to	1.25.			

Grades

At	the	beginning	of	the	year,	there	were	significant	differences	in	grades	for	English

Language	Arts	and	for	Other	(with students	not	served	by	AmeriCorps	members	

having	higher	grades	in	these	areas).		By	the	sixth	six	week	grading	period,	students	

who	were	served	by	AmeriCorps	members	had	significantly	lower	grades	than	those	

not	served	by	members.
14

ELA/Reading

Math

Science

Social	Studies

Other

All	Students

NonAmeriCorps AmeriCorpsSig.	

diff?

81.10 79.70 YES

81.10 80.19 YES

83.38 82.19 YES

84.18 82.34 YES

83.7 75.00 YES

Elementary	Only

NonAmeriCorps AmeriCorpsSig

diff?

82.72 79.57 YES

82.46 79.96 YES

85.98 83.54 YES

87.08 85.73 YES

83.00 66.63 YES

Final	grades,	however,	told	a	different	story.	There	were	no	significant	differences	in

grades	for	English/Language	Arts,	Social	Studies,	or	Other,	but	there	were	differences	

in	Math	and	Science,	with	students	served	by AmeriCorps	members	scoring higher	

than	those	served	by	other	CIS	staff.	Students	in	elementary	school	only	had	sixweek	grades	recorded;	final	grades	only	apply	to	students	served	in	middle	and	high

school.

NonAmeriCorps AmeriCorpsSignificant	

difference?

Final	Grades	(MS/HS	only)

English/Language	Arts 78.53 79.00 NO

Math 77.12 78.37 YES

Science 78.43 80.09 YES

Social	Studies 78.79 79.65 NO

Other 85.38 83.58 NO

There	were	significant	differences	at	the	six-week	mark	and	at	end-of-year,	but	were	

there	significant	changes	over	time?		The	answer	was	complicated.		In	middle	and	

high	school,	final	grades	generally	decreased	compared	to	pre-test	grades	in all	

subjects.	Students	served	AmeriCorps	had	smaller	decreases	compared	to students	

not	served	by	AmeriCorps	in	English/Language	Arts	and	Math.

However,	for	elementary	students,	there	was	generally	positive	growth	from	the	Pretest	to	the	6th six	weeks.	However,	there	was	less	growth	for	students	served	by	

AmeriCorps	members	in	Science.	Students	served	by	AmeriCorps	members	showed	

no	increases	in	averages	in	English/Language	Arts,	while	those	served	by	other	CIS	

staff	had	improvements.		In	Social	Studies,	the	grades	for	students	served	by	

AmeriCorps	members	actually	decreased,	while	grades	for	students	not	served	by	

members	increased.		There	were	no	significant	differences	for	Math	or	Other,	where	

both	groups	had	significant	increases	from	pre-test	to	the	sixth	six	weeks.

AmeriCorps	Year	3	Report
		

15

ELA/Reading

Math

Science

Social	Studies

Other

All	Students (Final	Grades)

Non- Sig.	

AmeriCorps AmeriCorps diff?

-1.32 -0.02 YES

-1.70 -0.12 YES

-2.55 -1.56 NO

-0.79 -0.74 NO

-1.35 +1.56 NO

Elementary	Only (6th 6 Weeks)

Non- Sig

AmeriCorps AmeriCorps diff?

+1.67 +.01 YES

+2.43 +2.01 NO

+2.16 +0.17 YES

+1.42 -1.48 YES

+5.42 +6.61 NO

STAAR

The	percentage	of	students	that	passed	STAAR	varied	significantly	between	students	

served	by	AmeriCorps	members	and	those	served	by	other	staff.		Data	for	previous	

STAAR	tests	was	not	included	in	the	data	set,	so	it	is	likely	that	many	of	these	

differences	were	pre-existing. Interestingly,	at	the	Elementary	level,	there	were	no	

significant	differences	in	STAAR	scores	for	any	subject	but	Science.	

ELA/Reading

Math

Science

Social	Studies

Writing

All	Students

NonAmeriCorps AmeriCorpsSig.	

diff?

73.3 58.8 YES

69.4 54.4 YES

80.2 54.8 YES

81.3 30.3 YES

68.2 45.1 YES

Elementary	Only

NonAmeriCorps AmeriCorpsSig

diff?

66.5 60.5 NO

63.0 58.7 NO

74.8 60.5 YES

N/A N/A NO

59.2 50.4 NO
	 	

	 	 	 	

	 	

		 	 	

		

	 	

16

Question 4:	What is the	relationship	between	the	amount and

type of services students receive	and outcome	variables?

Previous	evaluations	have	found	that	dosage	is	very	important	when	it	comes	to	the	

effectiveness	of	CIS.	Our	own	AmeriCorps	Evaluation	(Agile	Analytics,	2014)	found	

that	students	who	were	served	more	hours	had	better	outcomes	and	more	

improvement	over	time.	We	wanted	to	see	if	this	was	the	case	for	data	for	the	2013-

2014	school	year.	

Hours of Participation

The number	of	hours	that	students	participated in	CIS	was	positively	correlated with

grades,	and	negatively	correlated with	absences,	tardies,	and	disciplinary	incidents.		

In	particular,	students	who	participated	for	more	hours	had	higher	grades	in	the	last	

six	weeks	in	Science	and	Social	Studies.		Additionally,	middle/high	school	students	

who	participated	more	hours	showed	more	improvements	from	pre-	to	post-test	in

Social	Studies.	They	had	fewer	absences	and	tardies	at	the	end	of	the	year.		They	also	

had more	improvements	in	Social	Skills.		Finally,	teachers	were	more	likely	to	

recommend	that	students	who	participated	more	hours	should	continue	receiving	

CIS	services	next	year.		Students	who	participated	more	hours	were	also	less	likely	to	

pass	the	Reading	and	Social	Studies	STAAR.		

In	the	Elementary	sample,	these	trends	were	similar,	except	there	was	no	

relationship	between	the	number	of	hours	of	participation	and	6th Six	Week	grades

or	a	relationship	with	STAAR	passing	rates. There	was	also	a	negative correlation

between	the	number	of	hours	of	services	and	Classroom	Participation.	

However,	we	found	that	the	number	of	hours	that	students	participated	in	CIS	was	

also	positively	correlated	with	pre-test	grades	in	ELA,	Science,	and	Social	Studies,	and	

with	the	number	of	disciplinary	incidents	at	pre-test.		This	finding	indicates	that	there	

is	likely	a	factor	that	is	unaccounted	for	in	the	data	that	explains	why	students	both

have	better	academic	performance	&	behaviors	and	why	they	attend	more	CIS	hours.	

For	example,	these	students	could	be	more	motivated	than	other	students,	or	better	

able	to	focus,	which	might	cause	them	to	both	attend	more	CIS	sessions	and	to	do

well academically.

Due	to	the	finding	that	some	pre-test	variables	were	highly	correlated	with	the	

number	of	hours	of	participation,	we	also	examined other	background	variables	to	

see	if	they were	related	to	the	number	of	hours	of	participation.	The	item with the	

highest	correlation between	the	number	of	hours	of	participation	in	CIS and outcome

variables	was student	participation	in	after school	services.	Students	who	were	

AmeriCorps	Year	3	Report
	 	 	 	 	 	 	 	 	 	 	 	 	

		 	 	 	

	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	

	 	

	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	

	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	

17

served	for	more	hours	by CIS were also more likely to participate in at least one after

school	activity.	 No other student variables,	including gender,	race/ethnicity,	living

situation,	parental	status,	etc.	had a relationship	with	the	number	of	hours	that	

students were served.	 In the Elementary sample,	students who received more hours

of services were significantly more likely to live in a single-parent household and less

likely to have two parents at home.

Next,	we separated the data and examined correlations between the number of

hours	served and	improvements	for	students	who	were	served	by	AmeriCorps	

members and those that were not served by the program.	 Surprisingly,	there were

no correlations at all	between the number	of	hours	of	attendance	and outcomes	for

AmeriCorps students.	

Number of Sessions

For students who were served by an AmeriCorps member only,	we had data on the

sessions attended by month (e.g.,	“September sessions”).	We counted the number

of	sessions	(average = 6.43,	minimum=0,	maximum=10)	and examined the correlation

between session attendance and some outcome data that were reported only for

AmeriCorps students:	student pre- and	post-surveys	and	teacher	pre-post surveys,	

and	a	measure	of	increased	engagement (YES/NO).	

We found that the number of sessions attended was positively correlated with

improvements	in	engagement	and	improvement	on	the	student	survey	from	pre- to	

post-test.	 There was also a positive correlation between the number of sessions

attended and	teacher	pre- and	post-survey results.	

Member Status

Next,	we wanted to see if member term (EAO,	PT or FT)	affected student outcomes.	

Pre- and	post-test	teacher	and	student	surveys	and	student	engagement	were	

examined.	We found significant differences	in student	engagement	improvement	by

status.	 Only about 8.7% of students showed increased engagement overall.	14% of

students served by part time members,	compared to 10% of students served by full	

time	members	and	4%	of	students	served	by	EAO members,	increased engagement.	

Additionally,	EAO members had students whose teachers’	pre-test	surveys	were	

higher.	There were no other differences by term.
	 	 	

	 	 	 	 	 	 	 	

	 	 	 	

	 	

	 	 	 	 	

	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 		 	 	 	

	 	 	 	 	 	 	 	 	

18

Question 4: When pre-test	differences are controlled, are there

any	significant outcome	differences of students served by	

AmeriCorps vs not?

Because we found large pre-existing differences	between students	served	by

AmeriCorps	members	and	those	not	served	by	AmeriCorps,	we created	a small	

sample of students that was carefully matched on several	pre-test characteristics.	

Our hope was that by using matching scores that we would be able to create a

sample with no differences in pre-test	scores	or	demographic/background	

characteristics.

Sample	Characteristics

Our	final	matched sample contained 133 students (71 in the control	group,	62 in the

treatment	group).	 The sample was chosen so that each student had a match in the

following demographic and academic categories:	

• Grade

• Lives	at	Home

• Pre-test	absences

• Pre-ELA	Grade

• Pre-Math	Grade

Details on the final	sample are below.	It is important to note that this sample looks

very different from the sample examined in Question #2,	and also that the students in

the sample were very closely matched on these characteristics.

Non- AmeriCorps Significant

AmeriCorps difference?

Pre-test	Absences 0.32 0.29 NO

Pre-ELA	Grade 80.3 80.5 NO

Pre-Math	Grade 79.0 78.0 NO

Lives at Home? 98.6% 98.4% NO

Grade 3.50 3.92 NO

Of course,	matching data is not perfect.	There may have been other factors that we

did not control	for that are creating differences that are still	in the sample;	however,	

we hoped that matching data in this way would eliminate most differences.	Indeed

most of the differences disappeared.	 The only significant pre-test	difference	that	

remained	was ethnicity:	None of the students in the AmeriCorps groups were White,	

compared	to	7%	of	the	students	in	the	non-AmeriCorps	group.

AmeriCorps Year 3 Report
	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	

	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 		

	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	

	 	 	 	 	 	 	 	 	 	 	 	 	

	 	 	 	 	 	 	

19

Outcomes

There were a few significant differences in this small	sample.	 The first was that Social	

Skills increased significantly less for students	served	by the	AmeriCorps	Members	

(2.74 vs 3.60).	 Secondly,	CIS staff were less likely to say that students served by

AmeriCorps had grades that improved,	3.23 vs 4.50.	 This was supported by one piece

of	data	-- students	who	were	served	by	AmeriCorps	members	had	grades in Social	

Studies that	decreased from	pre-test to the last six weeks (by 3.0 points),	but

students	served	by	other	CIS	staff had a slight increase (by 0.5 points).

This lack of findings should not be interpreted to mean that	AmeriCorps	was	

ineffective.	AmeriCorps members target students that are struggling in school,	and

therefore it should be expected that these students will	not grow at the same rate

academically as their peers who were not in CIS for academic reasons.	 All of	the

findings should be interpreted with caution.	The matched sample size used was a

small	subset of students.	They were matched on academic and demographic

variables,	but it is possible that there are other underlying differences that were not

controlled for that are in the data.	
	 	

		

	 	

	 	

20

Conclusion

The	report	details	an	Impact	Analysis	of	the	AmeriCorps	program	at	Communities	in

Schools	of	Central	Texas.		The	purpose	of	the	evaluation	was	to	determine	if	there	

were	any	positive	changes	that	could	be	attributed	to	the	AmeriCorps	program.	

Specifically,	we	were	most	interested	to	see if students	served	by	AmeriCorps	

members	would	have	better	outcomes	compared	to	students	served	by	other	CIS

staff	that	were	not	AmeriCorps	members.

The	sample	drawn	for	this study	included	all	CIS	students	served	by	AmeriCorps	

members	in	Central	Texas	as	well	as	CIS	students	in	schools	that	did	not	have	an

AmeriCorps	program.	We	had	hoped	that	creating	the	sample	in	this	way	would	

create	two	groups	that	were	highly	similar;	however,	this	was	not	the	case.		The	two	

groups	of	students	were	very	different. First,	the	grade	level	of	the	students	varied	

significantly	by	condition.		The	AmeriCorps	sample	had	few	students	in High	School,	

and the	non-AmeriCorps	sample	had	few	Middle	School	students.		Because	of	this,	

we	conducted	all	analyses	with	the	entire	group	of	students,	and	then separately	

with	Elementary	students	only.		We	also	found	demographic	differences	between

students	(students	served	by	AmeriCorps	were	more	likely	to	be Hispanic,	to	have	an

incarcerated	parent,	and	to	live	at	home	with	two	parents).		They	also	had	fewer	

absences	and	disciplinary	incidents,	but	lower	grades.

When	we	examined	outcomes	for	these	two	very	different	groups,	it	was	not	

surprising	to	see	differences	persist.	Students	working	with	AmeriCorps	members	

were	more	likely	to	be	recommended	to	continue	services	next	year.		They	showed	

less	progress	in	five	areas	documented	by	CIS	staff.		They	continued	to	have	fewer	

disciplinary	incidents,	but	had	more tardies at	end-of-year	than	students	not	served

by	AmeriCorps.		They	had	lower	grades	in	all	subject	areas	at	the	six-week	mark,	but	

middle	and	high	school	students	had	higher	final	grades	in	Math	and	Science.			When

we	examined	changes	in	grades,	we	found	that	Middle	and	High	School	students	

showed	decreases	overall	from	pre-to	post-test,	but	students	served	by	AmeriCorps	

members	had	smaller	decreases.	At	the	Elementary	level,	there	were	generally	

improvements	in	grades	from	pre-test	to	the	6th Six	weeks, but	students	served by	

AmeriCorps	members	had	smaller	improvements.	

We	next	looked	at	the	impact	that	the	number	of	hours	of	participation	had	on

outcomes.	For	students	who	were	not	served	by	AmeriCorps	members,	we	found	

positive	correlations	between	number	of	hours	of	services	received	and	grades,	

improvements	in	Social	Skills,	and	negatively	correlated	to	absences,	tardies,	and	

disciplinary	incidents.	However,	there	were	also	strong	pre-test	correlations,	leading	

us	to	believe	that	there	were	inherent	differences	between students	who attended	

CIS	more	often	and	those	who	did	not.	When	we examined students	served	by	

AmeriCorps	members,	we	found	no	significant	relationships	between	the	number	

AmeriCorps	Year	3	Report
		

21

of	hours	of	services	received	and	any	of	the	outcome	variables. That	is,	students	

served	by	AmeriCorps	members	had	similar	outcomes	regardless	of	the	amount	of	

participation	in	programming.

Because	students	served	by	AmeriCorps	members	were	so	different	than	students	

not	served	by	members,	we	created	a	small	sample	of	students	that	were	matched

on	key	variables	(grade	level,	pre-test	absences,	pre-test	ELA/math	grades,	and	

housing	status.)	We	found	that	students	in	this	small	sample	were	highly	similar	at	

pre-test.		At	post-test,	students	served	by	AmeriCorps	members had	smaller	

documented	improvements	in Social	Skills	and	Grades.		They	had	a	slight	decrease	in

pre-test	to	6th Six	Week	Grades	for	Social	Studies,	whereas	students	not	served	by	

AmeriCorps	had	a	slight	increase.	

Based	on	these	analyses,	we	can	conclude that	students	served	by	the	AmeriCorps	

program	are	fundamentally	different	than	those	not	served	by	AmeriCorps,	even

when	they	are	not	at	the	same	school.		They	had	smaller	improvements	in	many	areas	

than	students	not	served	by	the	program,	but	this	may	be	due	to	their	pre-existing

differences.	Data	for	students	not	served	by	CIS	but	with	similar	characteristics	to	

students	served	by	AmeriCorps	were	not	available,	but	would	help	determine	if	

AmeriCorps	members	have	had	a	positive	impact	on	their	students.	We	can	be

reasonably	confident	from	our	careful	examination	of	the	data	that	the	AmeriCorps	

program	is	no	more	effective	at	improving	student	academic	outcomes	than	

traditional	CIS	services.

However,	these	data	do	not	tell	the	whole	story	of	AmeriCorps	members’	impact	

with	students.	 When	we	examined	data	for	the	number	of	months	in	which	students	

received	AmeriCorps	sessions,	we	saw	positive	correlations	in	student	engagement	

and change	from	pre- to	post-test	improvement	on	the	student	survey.	There	may	be	

other	areas	in	which	AmeriCorps	members	positively	affected	the	students	they	

served	that	were	not	measured	by	the	data	available.