Evaluation Report 

Breakthrough 

Grant Application ID: 17AC189783 

In 2011, independent researchers at Technology & Education Consulting Associates began a long-term,

quasi-experimental study on the impact of our AmeriCorps intervention on participants in the 

Breakthrough program in the Manor Independent School District. This Evaluation Report describes the 

preliminary results of this impact study. 

The study compares Breakthrough participants to non-participants with similar backgrounds through 

year-over-year surveys and de-identified student data, including grades, attendance, standardized test 

scores, and selection of rigorous courses. Breakthrough intentionally aligned the launch of this study with 

the beginning of our partnership with Manor ISD, and the first year of our AmeriCorps programming. 

The evaluation examines survey results and academic outcomes over time.

This report summarizes two sets of preliminary results: 1) academic outcomes during the first two years 

of Breakthrough’s program, and 2) educational and career goals and non-cognitive skill development. 

This study compares the results for Breakthrough program participants to other students with similar 

backgrounds at the same grade levels in Manor ISD. 

Purpose

The purpose of the evaluation is to study the effects of the Breakthrough AmeriCorps program in Manor 

ISD on academic achievement and non-cognitive factors of participants and non-participants over time. 

Specifically, this study attempts to answer the following questions: 

1. How do participants compare to non-participants on academic measures (grades and test scores) 

after matching for demographic characteristics, prior academic performance and attendance?

2. How do participants compare to non-participants on attendance after matching for demographic 

characteristics, prior academic performance and attendance?

3. How do participants compare to non-participants on self-reported non-cognitive measures, such as 

motivation, educational and professional goals, and self-discipline, after matching for demographic 

characteristics, prior academic performance and attendance?

4. How do participants compare to non-participants on teacher-reported non-cognitive measures after 

matching for demographic characteristics, prior academic performance and attendance?

5. How do program participants compare to similar non-participants on changes in non-cognitive 

measures?

Background and Theory of Change

Low-income, first-generation students graduate from high school, enroll in college, and persist and 

graduate from college at significantly lower rates than other students (The Pell Institute, 2006). Through 

a combination of out-of-school learning time and individualized case management support, 

Breakthrough’s AmeriCorps program targets low academic achievement, college-ready milestone 

completion, high school graduation, and college enrollment rates for low-income students whose parents 
or guardians have never been to college. The Breakthrough program is based on best practices in the 

field and research that demonstrates the interventions low-income, first-generation students need to 

make it to and through college. 

This Evaluation Report includes results collected during the 2012-2013 school year and 2014-2015 school 

year and focuses on the program in Manor ISD. During each of these school years, Breakthrough utilized 

year-round AmeriCorps members to serve an 11.5 month term supporting the Breakthrough Manor ISD 

program and quarter-time members to serve a 9-week summer term in the Breakthrough Manor Middle 

School Summer Program. 

With AmeriCorps member support, Breakthrough implemented a robust year-round program for students 

in Manor ISD that included individualized case management support alongside academic out-of-school 

learning time. AmeriCorps members, supported by Breakthrough staff members, provided out-of-schooltime learning and academic case management to Manor Breakthrough students to support their 

development in the following areas: academics, social and emotional development, and college 

knowledge. Because research demonstrates that low-income students often lose academic ground during 

the summer compared to their more affluent peers, Breakthrough utilized AmeriCorps members to deliver 

a rigorous summer program for participating students. The summer program included six full weeks of 

intensive, full-day academic programming, including a nutritious breakfast, lunch and snack, wellorganized workshops, field trips, and counseling sessions. Throughout the program, AmeriCorps 

members provided instruction in all core subjects, direct instruction on social-emotional learning, physical 

fitness, and teacher-crafted electives. The goals of the summer program were to address the academic 

deficits experienced by low-income, first-generation students, while fostering mission-critical 

relationships between students and staff and boosting motivation for college at an early age. In addition, 

full-time AmeriCorps members supported Breakthrough Program Coordinators by delivering year-round 

interventions, providing individualized counseling and mentorship, and helping to plan and implement 

Saturday, after school, and summer programs for our Manor ISD students. Full-time members provided 

critical academic support, help students build college-preparatory non-academic skills, and ensure 

students take necessary steps to enroll in and succeed in college. 

With AmeriCorps member support, Breakthrough ensured students met important benchmarks on the path 

to college graduation. Breakthrough tracks each student’s progress toward short and long-term outcomes, 

and our ultimate goal is for students to attain a college degree. 

Target Population

For the 2013 analysis, the population sample includes the 563 students who, according to district records, 

were enrolled as fifth graders in the district during the 2010-2011 school year. Within the population, 55 

students were enrolled in Breakthrough. In order to be eligible for the program, neither of a student’s 

parents may have graduated from a four-year college or university. Students who chose to apply to the 

program were then selected for participation based upon their grades, state assessment performance, 

regular school attendance, and teacher recommendations. The remaining 508 students comprise all other 

fifth grade students in the district during 2010-2011. To answer the pre-participation research questions 

for this study, and to conduct the selectivity analysis, we included all students for whom relevant data 

were available in fifth grade. Post-participation research questions compared the seventh grade academic, 

attendance and non-cognitive outcomes of program participants and a matched group of students who 

were not program participants. The students included in these analyses were restricted to those who were 

enrolled in the school district in both 2010-2011 and 2012-2013 and to those who had pertinent academic, 

attendance and questionnaire data available.
In addition, this report includes survey results from 1,209 Manor ISD students, of which 166 participated 

in the Breakthrough program, who were in 5th, 7th, and 9th grade during the 2014-2015 school year. 

Past Research

There is considerable evidence that intensive and early intervention programs (Fenske, Geranios & 

Keller, 1997), personalized case management (Dynarski, Clark, Cobb, Finn, Rumberger & Smink,

2008), and out-of-school support (Beckett, Borman, Capizzano, Parsley, Ross, Schirm & Taylor, 2009) 

are associated with success in school.

There is also a growing body of research linking non-cognitive factors, academic performance and

success in school. Recent work by the University of Chicago Consortium on School Research

(Farrington et al., 2012) highlighted the important roles of non-cognitive factors in academic 

performance, focusing especially on their potential as levers for increasing student educational attainment 

and providing educators with clear strategies to promote positive academic mindsets and behaviors in a 

school and classroom context. The non-cognitive factors measured as a part of this study were selected 

because of their previously demonstrated associations with educational outcomes (Eccles, 2009; Eccles & 

Wigfield, 2001; Weiner, 1992) and because of their potential to indicate longitudinal change between 

program participants and non-participants over the course of secondary school.

According to Conley (2007), college and career readiness consists of four key elements: cognitive strategies, 

content knowledge, learning skills and techniques, and transition knowledge and skills. Many college 

preparation programs and much of the related research have focused on specific aspects of college 

readiness, such as academic preparation, college knowledge, mentoring, or financial aid. Few programs 

have integrated these critical components and thus little research has been conducted with all of these 

interrelated factors in mind. 

Evaluation Approach and Design

This impact study examines the program’s intermediate objectives by comparing program participants

and non-participants on academic achievement, attendance and non-cognitive factors in 5th grade and

then every two years thereafter. 

For the 2013 results, the study examined academic achievement, attendance and non-cognitive factors 

in 5th grade and then in 7th grade for both program participants and non-participants. Because it was not 

possible to design a random assignment experiment to answer the research questions posed, several 

non-experimental methods were used to compare differences between program participants and nonparticipants on several variables. Using student survey data coupled with demographic, academic 

achievement and attendance data, participating students were compared with a matched group of nonparticipants. 

Data sources include administrative records and student and teacher questionnaire responses. Because 

the school district and community-based organization that are the focus of this study have a 

comprehensive memorandum of understanding and data sharing agreement, demographic, academic 

and attendance data were obtained directly from the district’s central office. In order to collect 

information regarding non-cognitive factors, student questionnaires were administered in 5th grade 

homeroom classes during the spring of 2011 and student and teacher questionnaires were completed in 

7th grade English Language Arts classes during the spring of 2013 at the request of the district’s 

Director of College Readiness, who serves as the liaison between the district and the community-based 

organization. Questionnaire participation for program participants was 78% in 2011 and 2013. For 

various reasons, questionnaire participation for program non-participants was inconsistent, 31% in 
2011 and 45% in 2013. 

The student questionnaires administered for both the 2013 analysis and the 2015 analysis included 

items based on a review of the literature and adapted from questionnaires developed as part of the 

University of Michigan Childhood and Beyond Study (Eccles, 1986-1995), as well as a Brief SelfControl Scale developed by Tangney, Baumeister & Boone (2004). The Childhood and Beyond Study 

was a large scale, longitudinal study that examined children’s achievement and self-perceptions in 

various domains and the roles that parents and teacher play in socializing those beliefs. Items included 

from that study included those that asked students to report their intrinsic, extrinsic, and moral 

motivations for doing school work, and the likelihood of achieving various long-term educational 

goals. The Brief Self-Control Scale focuses on factors related to impulse control that have been 

correlated with higher grade point averages and better adjustment. Although many items on the student 

questionnaire administered in 2011 were replicated on the questionnaire in 2013 and 2015, there were 

some changes and additions. For example, there was a set of items regarding problem-solving abilities 

that was not included on the seventh grade questionnaire because most students rated themselves very 

highly on those skills. The teacher questionnaire was developed for this study based on a review of the 

literature regarding non-cognitive factors that are associated with success in school.

Independent Variables

























Program participation

Gender (male or female)

Ethnicity (African-American, Hispanic, White, or Other, including multiple races)

Economic disadvantage (parent or guardian applied for and student was approved to 

receive free or reduced lunch according to criteria set by U.S. Department of Agriculture)

State assessment scores in reading and math in 5th grade in 2010-11—scaled scores on 

Texas Assessment of Knowledge and Skills (TAKS)

Final course grades in reading and math in 5th grade in 2010-2011 (0-100% scale) 

School enrollment in 5th grade in 2010-2011 (days enrolled; maximum 175 days) 

School attendance in 5th grade in 2010-2011 (days attended divided by days enrolled) 

Special education designation in 5th grade 

Limited English Proficient (LEP) designation in 5th grade

First or second year monitored in 5th grade (student in first or second year after exit from

Limited English Proficiency status)

Gifted and Talented designation in 5th grade

Dependent Variables 











State assessment scores in reading, math and writing in 7th grade in 2012-2013—scaled 

scores on State of Texas Assessment of Academic Readiness (STAAR)

Final course grades in reading and math in 7th grade in 2012-2013—(0-100% scale) 

School attendance in 7th grade in 2012-2013 (days attended divided by days enrolled) 

Motivation for doing schoolwork in 7th grade in 2012-2013 and 7th grade and 9th grade in 

2014-2015—student questionnaire adapted from Childhood and Beyond Study (Eccles, 

1986-1995). This 7-item self-report measure indicates the degree to which students have 

intrinsic, extrinsic or moral reasons for doing schoolwork. This questionnaire was also 

administered to a subset of 5th graders in 2011 and 2013. 

Educational goals in 7th grade in 2012-2013 and 7th grade and 9th grade in 2014-2015—

student questionnaire adapted from Childhood and Beyond Study (Eccles). This 6-item 
self-report measure asked students to individually rate the likelihood of their achieving 

various educational outcomes, ranging from dropping out of school to attending a graduate 

or professional school. This questionnaire was also administered to a subset of 5th graders 

in 2011 and 2013. 

 Self-perceptions of non-cognitive skills in 7th grade in 2012-2013 and 7th grade and 9th

grade in 2014-2015—student questionnaire items developed for this project. This 5-item 

self-report measure was designed to measure students’ own assessment of their level of 

tenacity with schoolwork that have been identified in the literature as important for success 

in school. 

 Self-discipline in 7th grade in 2012-2013 and 7th grade and 9th grade in 2014-2015—student 

questionnaire items adapted from Brief Self-Control Scale (Tagney, Boone & Baumeister,

2004). This 11-item self-report measure was designed to assess student perceptions of their 

own self-discipline. Items are rated on a scale from 1 (not at all like me) to 7 (very like 

me).

 Teacher ratings of students’ non-cognitive skills in 7th grade in 2012-2013 and 7th grade and 9th

grade in 2014-2015—teacher questionnaire developed for this study. This 10-item questionnaire 

asked teachers to rate each positive statement on a scale from 1 (rarely) to 5 (almost all of the 

time).

 Changes in motivation and educational goals from 5th grade to 7th grade to 9th grade as measured 

by comparing mean differences.

Data Analysis 

Descriptive statistics were used to compare the demographic, academic and attendance characteristics of 

program participants to all other 5th grade students in the district in 2011-2012. Significance tests were 

conducted using the chi-square or t-tests as appropriate. These analyses allowed us to identify the ways in 

which program participants were similar or different from the student body overall. Because there was 

evidence of group differences, we confirmed which student characteristics were correlated with program 

participation and used those variables to construct a comparison group. 

The comparison group was created by using the propensity score procedure, pscore, in STATA 12. 

Students were matched on the independent variables from their 5th grade year that were significantly 

different between the two groups of students before program participation: gender, LEP monitored status, 

gifted and talented, special education, number of days enrolled in 2010-2011, and TAKS scores in reading 

and math. Students also differed on their reading and math grades, however, because those were highly 

correlated with TAKS performance, they were excluded from the matching procedure

Program outcomes were assessed by comparing program participants to matched non-participants on 7th

grade academic performance (final reading and math grades, reading and math state assessment 

performance), school attendance, motivations for doing schoolwork, educational goals, self-perceptions, 

self-discipline, and teacher ratings of non-cognitive skills. Most of these comparisons were conducted 

using t-tests. However, because even after the propensity matching procedure there was still evidence of 

higher rates of state assessment performance among program participants in 5th grade, we used linear 

regression to examine the association between program participation and 7th grade STAAR scores while 

controlling for the students’ own TAKS scores in 5th grade.

Scores on the student motivation and educational goals questions of the student questionnaire were 

available for a subset of program participants and matched non-participants in both fifth and seventh 
grades, which allowed us to calculate change scores for each item. Mean differences in change from fifth 

to seventh grade were examined using t-tests.

Limitations

The results of this study should be considered in light of the following potential limitations. First, because 

students elected whether or not to participate in the program, it was not possible to conduct a controlled 

experiment. This created the potential for self-selection bias. Second, students in the comparison group 

were matched on demographic, academic and attendance factors only and differed from program 

participants on some non-cognitive factors. Third, although attempts were made to account for a variety 

of student factors in fifth grade, some important variables, such as parent and teacher support, or other 

program participation, were not measured. Fourth, although the questionnaires used in this study were 

adapted from other studies or based upon a review of the literature, they were not subject to prior 

validation and were the result of student self-reports and teachers perceptions. Fifth, the size of the 

participant group was quite small in comparison to non-participants and only represented one cohort of 

the program. Sixth, because questionnaires were only received from a portion of the campuses and 

students, the possibility of systematic missing data is a limitation. Seventh, because the researchers had 

no way of accessing data for students who left the district between fifth and seventh grades, student 

attrition, mobility and data quality are potential limitations. Finally, although the community-based 

college preparation program that is the focus of this study uses a variety of research supported best 

practices for helping low-income and underserved students succeed in school and eventually graduate 

from high school and enroll in college, this study did not directly examine the quality or quantity of 

program features which students received. 

Findings

2013 Results 

In 2011, data were collected to establish baseline measures of student characteristics on demographic, 

academic, attendance and specific non-cognitive measures and to determine ways in which program 

participants were similar to or different from their peers prior to program participation. Table 1 shows the 

comparison between program participants and all other fifth graders in the district on demographic, 

academic and attendance measures. Table 2 shows the correlations among demographic, academic and 

attendance measures and program participation. 

Table 1 

Pre-participation demographic and academic characteristics of 

program participants and all other 5th grade students in the district in 2011

Student 

Characteristics

Program Participants

N=55

Other Students

N=507

Significance tests

Frequency Percentage Frequency Percentage

Demographics

African-American 11 20% 147 29% χ2(1, N = 562) = 

1.99, p = .16

Hispanic 35 64% 301 59% χ 2(1, N = 562) = 

.38, p = .54

White 8 14% 42 8% χ 2(1, N = 562) = 
2.40, p = .121

Other 1 2% 17 3% χ 2(1, N = 562) = 

.01, p = .94

Female 37 67% 230 45% χ 2(1, n= 559) = 

9.31, p < .01

Male 18 33% 274 54%

Economic 

Disadvantage

46 84% 395 84% χ 2(1, N = 524) = 

0.01, p = .91

Academics

Gifted and Talented 12 22% 51 11% χ 2(1, N = 522) = 

5.51, p < .05

Special Education 1 2% 48 10% χ 2(1, N = 524) = 

4.11, p < .05

Limited English 

Proficient

10 18% 113 24% χ 2(1, N = 524) = 

.96, p = .33

1st or 2nd Year 

Monitored

15 27% 56 12% χ 2(1, N = 524) = 

9.88, p < .01

State Assessment 

TAKS

Scaled Score

 Reading 733 673 t(469)= 4.60, 

p<.001

 Math 736 664 t(469)= 5.02, 

p<.001

Final Course Grades 0-100% scale

 Reading 87.9 81.9 t(460)= 5.46, 

p<.001

 Math 87.7 81.9 t(460)= 4.67, 

p<.001

Enrollment/

Attendance

Average Days 

Enrolled in 2010-

2011 (maximum of 

175)

173 155

t(303) = 7.25, 

p<.001

Attendance Rate in 

2010-2011

97.4% 96.6%

t(560) = 1.35, ns

Note: The frequencies represent the number of students with each characteristic and may not add up to the 

total ‘n’ because of missing data. The denominator was adjusted in order to reflect valid percentages for 

each indicator. 
Table 2 

Correlations between program participation and 5th grade demographic, attendance and academic variables 

Program 

participation

Program 

participation

-

White Black Hispanic

Other raceethnicity Female

Economic 

disadvantage

Limited 

English 

Proficiency Monitored 

Gifted & 

Talented

Special 

Education

Days 

enrolled 

Attendance 

rate

TAKS 

reading 

scale 

score

TAKS math 

scale 

score

Final 

reading 

grade

Final 

math 

grade

White .066 -

Black -.059 -.195** -

Hispanic .025 -.381** -.763** -

Other raceethnicity -.003 -.044 -.088* -.172** -

Female .128** .039 .051 -.072 -.007 -

Economic 

disadvantage -.005 -.376** .022 .225** -.063 -.007 -

Limited English 

Proficiency -.044 -.174** -.333** .414** .030 -.056 .180** -

Monitored .138** -.124** -.232** .263** .119** -.061 .126** -.220** -

Gifted & 

Talented .103* .113** -.097* .010 .087* -.067 -.147** -.179** .196** -

Special 

Education -.088* -.009 .065 -.061 .008 -.141** -.022 .022 -.108* -.116** -

Days enrolled .125** .028 -.053 .014 .027 .032 -.072 .044 .084 .064 -.008 -

Attendance rate .057 -.113** -.034 .084* .030 -.010 -.035 .147** .146** .095* -.034 .333** -

TAKS reading 

scale score .208** .148** -.076 -.049 .098* .113* -.134** -.391** .192** .379** -.042 .038 .040 -

TAKS math 

scale score .227** .198** -.235** .065 .109* -.030 -.126** -.219** .269** .466** -.064 .136** .229** .597** -

Final reading 

grade .246** .152** -.151** .024 .128** .198** -.066 -.229** .220** .393** -.120* .066 .222** .636** .621** -

Final math 

grade .213** .122** -.202** .098* .099* .096* -.014 -.091 .229** .393** -.130** .008 .335** .492** .722** .756** -

*p < .05. **p < .01. 
The results indicate that program participants were similar to other 5th grade students with regard to ethnicity, economic 

disadvantage, attendance rate and Limited English Proficiency status. The majority of students in both groups were 

Hispanic and the majority of students in both groups were economically disadvantaged. Program participants and nonparticipants were also different in many ways. Program participants were more likely to be female and designated as 

Gifted and Talented. Program participants were enrolled in more days of school and had higher state assessment scores 

and final course grades in reading and math. In addition, students enrolled in the program were less likely to be in Special 

Education and more likely to have been exited from Limited English Proficiency status. 

Tables 3 and 4 show the results comparing program participants and non-participants in fifth grade, prior to program 

participation, on students’ motivation for doing schoolwork and future educational goals. Program participants and nonparticipants rated equally their likelihood of attending a 4-year college or attending graduate or professional school. 

Program participants were more likely than other students to say they do their school work because they “feel bad it it’s 

not done” and less likely than other students to say they do their schoolwork because “my parents told me I have to.” In 

addition, program participants were less likely than other students to say they would “dropout of high school” or “work 

full time right after high school.” 

Table 3 

Average responses in 5th grade prior to program participation to the question:

“Circle the number that shows WHY you do your schoolwork...?” 

Why you do your school work Program Participants

N=44

Other Students

N=157

Significance tests

Mean Mean

I want to learn new things 5.727 5.819 t(197) = .38, ns

It’s fun and interesting 5.136 4.928 t(195) = .68, ns

I feel bad if it’s not done 5.614 4.707 t(192) = 3.04, p < .01

I think I should 5.977 5.667 t(188) = -1.11, ns

The teacher says have to 5.095 5.527 t(188) = 1.24, ns

It makes my parents happy 5.791 6.285 t(54) = 1.70, ns

My parents told me I have to 4.227 5.273 t(196)) = 2.68, 

p < .01

Note. Response scale ranged from 1 “This is not like me at all” to 7 “This is a lot like me.” 

Table 4 

Average responses in 5th grade prior to program participation to the question: 

“We can’t always do what we most want to do. 

How likely do you think each of the following will be?” 

Outcome Program Participants

N=44

Other Students

N=157

Significance tests

Mean Mean

You will drop out of high school before 

you finish 1.091 1.484 t(196) = 3.06, p < .01

You will go to work full time right after 

high school 4.159 4.949 t(198) = 2.12, p < .05

You will go into the military right after 

high school 2.070 2.242 t(194) = .51, ns

You will graduate from a 4-year college 6.636 6.420 t(93) = 1.26, ns

You will attend a graduate or professional 

school after college 5.818 5.378 t(82) = 1.75, ns

Note. Response scale ranged from 1 “Very unlikely” to 7 “Very likely.” 

Table 5 shows the demographic, academic and attendance characteristics of program participants and the comparison 

group of non-participants after the propensity match. The one program participant who was enrolled in special education 

was dropped from the participant group because it was interfering with the matching procedure. After the matching 

procedures, program participants still had significantly higher scores than the comparison group on TAKS math and final 
course grades in reading and math. These differences were controlled for in subsequent tests of academic achievement by 

including students’ fifth grade TAKS scores in reading or math as covariates in regression analyses. 

Table 5 

Comparison of program participants and propensity matched non-participants 

on demographic, academic and attendance variables 

Student Characteristics Program 

participants 

(n=54)

Matched nonparticipants 

(n=265)

Significance tests

Average or 

Percentage

Average or 

Percentage

Demographics

African-American 20% 24% χ 2(1, N = 319) = .43, 

p = .51

Hispanic 63% 60% χ 2(1, N = 319) = .17, 

p = .69

White 15% 11% χ 2(1, N = 319) = .66, 

p = .121

Other 2% 5% χ 2(1, N = 319) = .11, 

p = .42

Female 67% 63% χ 2(1, n= 319) = .26, 

p = .61

Male 33% 37%

Economic Disadvantage 83% 81 χ 2(1, N = 319) = .14, 

p = .70

Academics

Gifted and Talented 22% 18% χ 2(1, N = 319) = .56, 

p = .44

Special Education 0% 0% n/a no SPED students in 

analysis group

Limited English Proficient 17% 17% χ 2(1, N = 319) = .04, 

p = .85

1st or 2nd Year Monitored 28% 20% χ 2(1, N = 319) = 1.62, 

p = .20

State Assessment-TAKS

 Reading 733 710 t(317) = 1.89 , ns

 Math 736 707 t(317) = 2.16, p < .05

Final Course Grades 

 Reading 88 85 t(98) = 3.90, p < .001

 Math 88 85 t(297) = 2.60, p < .05

Enrollment/Attendance

Average Days Enrolled in 2010-

2011 (maximum of 175)

173 173 t(317) = .19, ns

Attendance Rate in 2010-2011 97.3% 97.3% t(317) = .15, ns

Table 6 shows how program participants and a matching group of non-participants compare on grades, test scores and 

attendance in seventh grade, after two years of the program (Research Question 2). In comparison to the propensitymatched group, program participants had significantly higher state assessment (STAAR) scores in seventh grade 

reading and math but no significant differences in final course grades in reading and math. No significant differences 

were found between the attendance rates of seventh grade participants and non-participants. 
Table 6 

Comparison of program participants and matching group of non-participants on test scores, grades and 

attendance in 7th grade after two years of the program 

Measure Program 

Participants

(n=49) 

Matched 

NonParticipants 

(n=190)

Significance tests

Mean

Final course grade in reading 86.94 85.67 t(235) = 1.21, ns

Final course grade in math 81.90 82.25 t(236) = .32, ns

Scaled Score

STAAR reading* 1676 1639 t(246) = 2.39, p < .05

STAAR math* 1670 1630 t(245) = 2.45, p < .05

Average Daily Attendance

Attendance Rate 93.6% 93.4% t(257) = .28, ns

Note. The number of matched non-participants varied slightly depending upon the indicator. 

In order to account for baseline differences in academic achievement between the groups, regression analyses were used 

to examine the associations between program participation and seventh grade state assessment scores while controlling 

for students’ state assessment performance in fifth grade. In this more rigorous analysis shown in Table 7, program 

participation was significantly associated with higher state assessment (STAAR) scores in reading but not in math or 

writing in seventh grade. 

Table 7 

Summary of regression analyses for seventh grade state assessment outcomes

Variable 7th grade STAAR 

reading score

7th grade STAAR 

math score 

7th grade STAAR 

writing score

B SE B β B SE B β B SE B β

Program 

participation 30.2 12.69 .124** 19.84 10.10 .077 71.37 52.52 .077 

5th grade 

TAKS

reading 

score

.693 .063 .567*** 2.125 .262 .460*** 

5th grade 

TAKS math 

score

.861 .048 .740*** 

R2 .586 .566 .220 

F F(2, 245) = 64.23, p < .001 F(2, 244) = 159.15, p < .001 F(2, 243) = 34.35, p < .001 

**p < .01. ***p < .001. 

Tables 8, 9, 10 and 11 show how participants and matched non-participants compare on self-reported non-cognitive

measures after matching for demographic characteristics, prior academic performance and attendance (Research 

Question 4). Results indicate that program participants and matched non-participants reported similar motivation, 

educational goals and self-discipline except for a few key areas. Program participants were significantly less likely to 

report doing their schoolwork because it makes their parents happy, an external motivation. Program participants also 

were significantly less likely to report they will go into the military right after high school. Although both groups had 

high average ratings of likely college graduation, program participants had significantly higher ratings on this goal. 

Surprisingly, program participants were significantly less likely to report that they completed their schoolwork on 
time. Also surprisingly, program participants were less likely than the comparison group to describe themselves as 

having a lot of self-discipline. No significant differences were found between participants and match non-participants 

on the self-discipline scale.

Table 8 

Comparison of program participants and matching group of non-participants to the question: “Circle the number 

that shows WHY you do your schoolwork...?” 

in 7th grade 

Why you do your school work Program Participants

n=43

Matched NonParticipants n=168Significance tests

Mean Mean

I want to learn new things 5.023 5.226 t(209) = .76 , ns

It’s fun and interesting 3.744 3.982 t(208) = .77 , ns

I feel bad if it’s not done 3.977 4.533 t(208) = 1.65 , ns

I think I should 4.930 5.357 t(209) = .1.62, ns

The teacher says have to 4.767 5.120 t(207) = .1.10, ns

It makes my parents happy 4.907 5.554 t(209) = 2.15, p < .05

My parents told me I have to 4.558 5.244 t(209) = 1.97 , ns

Note. Response scale ranged from 1 “This is not like me at all” to 7 “This is a lot like me.” 

Table 9 

Comparison of program participants and matching group of non-participants to the question: “We can’t always 

do what we most want to do. How likely do you think each of the following will be?” in 7th grade 

Outcome Program Participants

n=44

Matched NonParticipants n=157Significance tests

Mean Mean

You will drop out of high school before 

you finish 1.093 1.320 t(210) = 1.51 , ns

You will go to work full time right after 

high school 3.814 4.118 t(210) = .90, ns

You will go into the military right after 

high school 1.488 2.036 t(129) = 2.83, p < .05

You will graduate from a technical program 

or 2-year college 3.605 3.754 t(208) = .42 , ns

You will graduate from a 4-year college 6.233 5.574 t(92) = 2.88, p < .05

You will attend a graduate or professional 

school after college 5.071 4.625 t(208) = 1.37, ns

Note. Response scale ranged from 1 “Very unlikely” to 7 “Very likely.” 
Table 10 

Comparison of program participants and matching group of non-participants to questions related to selfperceptions of non-cognitive skills in 7th grade 

Question Program Participants

n=43 

Matched NonParticipants 

n=168

Significance tests

Mean Mean

How often did you complete your 

schoolwork on time? 5.047 5.375 t(81) = 2.01 , p < .05

How often were you excited about learning 

new things in at school? 4.302 4.574 t(210) = 1.08 , ns

How often did you stick with your 

schoolwork even when it was difficult? 4.907 5.000 t(209) = .39, ns

How often did you get along with the other

students at school? 5.256 5.349 t(210) = ..41 , ns

How often did you get along with the adults 

at school? 5.262 5.186 t(201) = .28, ns

Note. Response scale ranged from 1 “never” to 7 “always.” 

Table 11 

Comparison of program participants and matching group of non-participants on student self-reported selfdiscipline in 7th grade 

Question Program Participants

n=433

Matched NonParticipants n=168Significance tests

Mean Mean

I have a hard time breaking bad habits. 3.024 3.163 t(206) = .79 , ns

I say inappropriate things. 3.349 3.554 t(209) = 1.09 , ns

I refuse things that are bad for me. 3.837 3.842 t(206) = .03, ns

I wish I had more self- discipline. 3.349 3.452 t(20) = .52 , ns

I am good at resisting temptation. 3.674 3.436 t(204) = 1.33 , ns

People would say that I have a lot of selfdiscipline 2.837 3.217 t(207) = 1.93 , ns

Pleasure and fun sometimes keep me from

getting work done 2.814 2.695 t(205) = .61 , ns

I have trouble concentrating. 2.930 3.137 t(209) = .99 , ns

I am able to work effectively toward longterm goals 3.605 3.750 t(209) = .91 , ns

Sometimes I can't stop myself from doing

something, even if I know it is wrong.

3.628 3.405 t(209) = 1.15, ns

I often act without thinking through all the

alternatives. 3.372 3.149 t(209) = 1.08 , ns

Note. Response scale ranged from 1 “Not like me at all” to 5 “Very like me.” 

Table 12 shows teacher ratings of program participants and matching non-participants in 7th grade in 2013 on noncognitive measures related to school success. No significant differences in teacher ratings were detected between the two 

groups. 
Table 12 

Comparison of teacher ratings of program participants and matching group of non-participants on student noncognitive skills in 7th grade 

This student.... Program Participants

n=34

Matched NonParticipants n=157Significance tests

Mean Mean

is well organized 4.152 4.108 t(188) = .20, ns

comes to class prepared for the day’s

lesson 4.118 4.025 t(189) = .42 , ns

handlesfrustration well 4.088 3.949 t(190) = .61 , ns

seeks help when it’s needed 4.364 4.217 t(188) = .80 , ns

is curious about the world 4.258 4.275 t(171) = .10 , ns

is a creative problem-solver 4.152 4.119 t(174) = .18 , ns

thinks before they act (not impulsive) 4.212 4.095 t(174) = .54 , ns

shows empathy for others 4.333 4.184 t(183) = .82 , ns

has positive relationships with peers at

school 4.471 4.386 t(190) = .57 , ns

has positive relationships with adults at

school 4.471 4.323 t(190) = .87 , ns

Note. Response scale ranged from 1 “Rarely” to 5 “Almost all of the time” 

Tables 13 and 14 compare how program participants and matched non-participants changed between 5th and 7th grade on 

self-reported scores of student motivation and educational goals. 

Table 13 

Changes in self-reported motivation from 5th grade to 7th grade 

Why you do your school work Program Participants

n=36

Other Students

n=55

Significance tests

Mean Difference Mean Difference

I want to learn new things -0.944 -0.782 t(89) = .44 , ns

It’s fun and interesting* -1.694 -0.566 t(87) = 2.56 , p < .05

I feel bad if it’s not done** -1.861 0.157 t(85) = 3.68 , p < .001

I think I should -1.229 -0.580 t(83) = 1.42 , ns

The teacher says have to -0.412 -0.600 t(82) = .29 , ns

It makes my parents happy -0.914 -0.519 t(87) = .89 , ns

My parents told me I have to 0.056 0.278 t(88) = .39 , ns

Note. Response scale ranged from 1 “This is not like me at all” to 7 “This is a lot like me.” 

Table 14 

Changes in self-reported educational goals from 5th grade to 7th grade 

Outcome Program Participants

n=36

Other Students

n=55

Significance tests

Mean Difference Mean Difference

You will drop out of high school before 

you finish 0.000 -0.278 t(88) = 1.08 , ns

You will go to work full time right after 

high school -0.083 -0.963 t(88) = 1.52, ns

You will go into the military right after 

high school -0.556 0.038 t(86) = 1.37, ns

You will graduate from a 4-year college -0.472 -0.782 t(89) = .97, ns

You will attend a graduate or professional 

school after college -0.971 -1.000 t(86) = .07, ns

Note. Response scale ranged from 1 “Very unlikely” to 7 “Very likely.” 
Additional Findings: 2015 Results

These results pertain to Manor ISD 5th, 7th, and 9th grade student self-ratings of educational and career goals and noncognitive skills, and teacher ratings of non-cognitive skills. Additionally, it compares these ratings for Breakthrough 

program participants in comparison to other students at the same grade levels. A summary of these survey results follow.

Over 1,200 students participated in the survey; approximately 14% of the students are in Breakthrough program. Some 

students did not provide a student Identification number, so not all of these records will be included in the subsequent 

academic analysis. Too many 5th grade student surveys were attributed to Manor Elementary School, this error will be 

corrected upon receipt of the academic records which will allow us to merge in the correct school for each student. 

Teachers completed just over 700 surveys about their students. Approximately 12% of those surveys were completed 

about students in the Breakthrough program. 

Students’ Long-Term Educational and Career Goals

•

•

•

•

•

Across all grade levels, students rated graduating from a 4-year college and their most likely educational outcome; 

however, the average rating was slightly lower for older students than it was for younger students. These ratings 

were statistically equivalent among 7th and 9th graders whether they were in Breakthrough or not; however older 

Breakthrough students rated a higher likelihood of going to graduate or professional schools. 

Breakthrough students consistently rated the risk of dropping out lower than did other students. Among 7th and 9th

graders they also rated the likelihood of working full-time after high school graduation lower than their district 

peers. 

Only the Breakthrough 7th grade cohort rated the likelihood of entering the military significantly lower than their 

district peers.

When asked what type of job they would most like to have, the majority (50% or more) of students at all grade 

levels named a job that would require a 4-year degree (professional).

Breakthrough students listed professional jobs at a somewhat higher rate than other Manor ISD students at the 

same grade level.

Manor ISD students rated graduating from a 4-year college as their most likely educational outcome

However older students gave slightly average lower ratings than younger students

1.6

4.5

2.3

4.1

5.3

4.6

1.4

4.0

2.0

3.5

5.0

4.5

1.6

4.0

2.3

3.9

4.7

4.3

1.0

2.0

3.0

4.0

5.0

6.0

7.0

 Drop out of

high school

before you

finish.

Go to work full

time right after

high school.

Go into the

military right

after high

school.

Graduate from

a technical

program or 2-

year college

Graduate from

a 4-year college.

Attend a

graduate or

professional

school after

college.

5th graders 7th graders 9th graders

Note. Students rated this item on a scale from (1) very unlikely to (7) very likely.
Manor ISD students in Breakthrough rated graduating from a 4-year college as their most likely educational outcome

However, older students gave lower average ratings than younger students

1.3

4.4

2.1

4.2

6.1

5.4

1.1

3.1

1.5

3.3

5.8

4.8

1.1

2.7

2.0

3.2

4.8

4.3

1.0

2.0

3.0

4.0

5.0

6.0

7.0

You will drop out

of high school

before you finish.

You will go to

work full time

right after high

school.

You will go into

the military right

after high school.

You will graduate

from a technical

program or 2-

year college

You will graduate

from a 4-year

college.

You will attend a

graduate or

professional

school after

college.

5th graders 7th graders 9th graders

↓ ↓ ↓

↑

↓ ↓

↑

↓

↑

Note. Students rated this item on a scale from (1) very unlikely to (7) very likely. Arrows indicate whether Breakthrough student 

averages were lower (↓) or higher (↑) than other Manor ISD students at the in the same grade.

5th grade 7th grade 9th grade

Job type Breakthrough Other students Breakthrough Other students Breakthrough Other students

Professional 72.2% 51.7% 67.7% 52.5% 70.8% 56.1%

Skilled 8.3% 14.4% 10.8% 14.0% 4.2% 16.2%

Athlete 6.9% 13.2% 4.6% 14.3% 12.5% 6.3%

Artist/entertainer/musician 4.2% 6.9% 9.2% 5.4% 4.2% 3.3%

Unclear 1.4% 3.7% 4.6% 3.0% 4.2% 3.7%

Don't know 1.4% 3.2% 0.0% 2.1% 4.2% 6.3%

Military 2.8% 4.6% 1.5% 2.7% 0.0% 3.3%

Retail/food service 1.4% 1.7% 0.0% 3.0% 0.0% 2.2%

Entrepreneur 1.4% 0.6% 1.5% 1.5% 0.0% 1.8%

Unskilled 0.0% 0.0% 0.0% 1.5% 0.0% 0.7%

Note. If a student listed more than one profession, their first example was recoded into one of these categories. 

Academic Motivation

The student survey included a subscale that measured three types of motivation for doing schoolwork: intrinsic 

motivation (I want to learn new things, It’s fun and interesting), conscientiousness (I feel bad if it’s not done, I think I 

should) and extrinsic motivation (The teacher says I have to, It makes my parents happy, My parents told me I have to).

•

•

•

Motivation results for Breakthrough and other students were not significantly different except at 5th grade. We 

believe this can be attributed to pre-participation differences between students who are admitted to the program 

and other students, so the results for all students are combined in the figure below. 

All motivations are somewhat lower among secondary students than elementary students, with the exception of 

wanting to learn new things.

The largest differences between elementary and secondary students are in the intrinsic motivation of schoolwork 

being fun and interesting and all three of the extrinsic motivations.
External motivations are lower among secondary students than elementary students

The motivation to learning new things is similar across the grade level groups

5.3

4.9 4.7

5.4 5.4

6.0

5.4 5.5

4.7 4.5

5.1

4.7

5.7

5.0 5.1

4.1

4.4

5.1

4.6

5.1 4.8

1.00

2.00

3.00

4.00

5.00

6.00

7.00

I want to

learn new

things.

It's fun and

interesting.

I feel bad if

it's not done.

I think I

should.

The teacher

says I have

to.

It makes my

parents

happy.

My parents

told me I

have to.

5th graders 7th graders 9th graders

•

•

5.4 5.3 5.2 5.6 5.7

5.0

4.5 4.8 5.1 5.0 5.1

4.4 4.7

5.2 5.1

1.00

2.00

3.00

4.00

5.00

6.00

7.00

How often did

you complete

your school

work on time?

How often

were you

excited about

learning new

things at

school?

How often did

you stick with

your

schoolwork

even when it

was difficult?

How often did

you get along

with the other

students at

school?

How often did

you get along

with the adults

at school?

5th graders 7th graders 9th graders

Student Self Ratings of Non-Cognitive Skills

On average, district students at all grade levels give themselves moderately high ratings on the non-cognitive

skills; elementary students rated themselves more highly than did secondary students.

At 5th and 7th grade, Breakthrough gave self-ratings that were significantly higher than other students with regard 

to completing their schoolwork on time and getting along with other students at school. 

District elementary students rate themselves slightly higher on these noncognitive skills than do secondary students

Note. Rated on a scale from (1) never to (7) always. 
Breakthrough elementary students provide higher self-ratings of noncognitive skills than do secondary students

5th and 7th grade students are more likely to say they complete their schoolwork on time and get along with other student than do 

other district students

5.8

5.4

5.9 5.9 6.2

5.4

4.8 5.2 5.5 5.4

5.0

4.2 4.5

5.1 4.9

1.00

2.00

3.00

4.00

5.00

6.00

7.00

How often did

you complete

your school

work on time?

How often

were you

excited about

learning new

things at

school?

How often did

you stick with

your

schoolwork

even when it

was difficult?

How often did

you get along

with the other

students at

school?

How often did

you get along

with the adults

at school?

5th graders 7th graders 9th graders

↑ ↑ ↑ ↑

↑ ↑

Note. Rated on a scale from (1) never to (7) always. Arrows indicate when Breakthrough student averages were higher (↑) than 

other Manor ISD students at the in the same grade.
Student Self Ratings of Self-Control 

•

•

•

•

Overall, students reported moderate levels of self-control. 

There were few grade level differences in student reported aspects of self-control. The one notable exception is that elementary students report lower levels 

of saying inappropriate things than do secondary students. 

There also were very few differences in self-control ratings between Breakthrough participants and other district students; for this reason, results are 

combined across groups. 

Although results from prior administration of the Brief Self Control Scale (BSCS) showed that lower ratings of self-control are associated with lower 

levels of student achievement, the present results indicate that the BSCS does not consistently distinguish between students who participate in 

Breakthrough and those who do not. 

o One exception was that 5th and 7th grade Breakthrough students both gave significantly higher ratings to their ability to work effectively toward 

long-term goals.

Across grade levels, students give fairly similar ratings to various aspects of self-control

Elementary students were slightly less likely to say inappropriate things than secondary students

3.4

4.0

3.5 3.2 3.3

2.9 2.7

3.2

3.8 3.5 3.3 3.3 3.4 3.6

3.3 3.3 3.2

2.8 3.0

3.7 3.5 3.2 3.0 3.0

3.5 3.2 3.5 3.3

2.6 2.8

3.7

3.2 3.1

1.00

2.00

3.00

4.00

5.00

6.00

7.00

I have a hard

time breaking

bad habits.

(reverse)

I say

inappropriate

things.

(reverse)

I refuse things

that are bad

for me.

I wish I had

more selfdiscipline.

(reverse)

I am good at

resisting

temptation.

People would

say that I have

a lot of selfdiscipline.

Pleasure and

fun

sometimes

keep me from

getting work

done.

(reverse)

I have trouble

concentrating.

(reverse)

I am able to

work

effectively

toward longterm goals.

Sometimes I

can't stop

myself from

doing

something,

even if I know

it is wrong.

(reverse)

I often act

without

thinking

through all

the

alternatives.

(reverse)

5th grade 7th grade 9th grade

Note. Negative items were reverse coded so that higher scores indicate higher levels of self-control.

Rated on a scale from (1) this is not at all like me to (5) this is a lot like me.
Teachers’ Ratings of Students’ Non-Cognitive Skills

•

•

•

Across grade levels, teachers gave moderate ratings to the non-cognitive skills Manor ISD students; some of their highest average ratings were for 

students’ relationships with peers and adults at school.

Across grade levels and program groups, many of the lowest average ratings were provided by 7th grade teachers.

At 5th and 7th grade, Breakthrough students were consistently rated higher than were other district students. 

Unlike some of the students self-rated measures, non-cognitive ratings from teachers consistently distinguish between Breakthrough and non-program students 

both before they participate in the program (in 5th grade) and after two years of participation (at the end of 7th grade).

Across grade levels, teachers gave higher ratings to the noncognitive skills of Breakthrough students

7th grader Breakthrough students were consistently rated higher than their district peers

4.4 4.5 4.4

4.0 4.1 3.9

4.3 4.4 4.5 4.5

3.9 3.9 3.9 3.8 3.8 3.6 3.9 4.1 4.2 4.2 4.0 4.0

4.4

4.0

3.6 3.4

4.8 5.0 4.8 4.8

1.00

2.00

3.00

4.00

5.00

6.00

7.00

is well

organized.

comes to class 

prepared for 

the day’s 

lesson.

handles

frustration well.

seeks help 

when it’s 

needed.

is curious about

the world.

is a creative

problem-solver.

thinks before

they act (not

impulsive).

shows empathy

for others.

has positive

relationships

with peers at

school.

has positive

relationships

with adults at

school.

5th graders 7th graders 9th graders

↑

↑

↑ ↑

↑

↑ ↑

↑ ↑

↑

↑

↑

↑

↑ ↑

Note. Rated on a scale from (1) rarely to (7) always.
Conclusions

Key findings from the 2013 Results:

•

o

•

•

No significant findings were found between participants in performance on math and writing standardized 

assessments, attendance rates, and behavior.

However, this study provides evidence of an association between student participation in the 

Breakthrough program in 6th and 7th grades and improved reading test scores. 

The study also provides evidence of an association between student program participation and specific noncognitive factors.

Most notably, program participants were significantly more likely to report a high likelihood of graduating from 

college. 

 Key findings from 2015 Survey Results:

•

•

•

•

•

o

The survey results indicate that both Breakthrough and non-program students aspire to high levels of education, 

but that on average, non-program students consider 2-year and technical programs or working right after high 

school more likely outcomes.

The majority of non-program students listed a career goal that required a 4-year degree; however, the proportion 

of students with these aspirations was lower than among Breakthrough students. 

Breakthrough students also reported a significantly lower likelihood of dropping out of high school and of 

working full time after high school, which also may indicate a stronger commitment to full-time postsecondary 

education than among non-program students. 

Student self-reported motivations for doing schoolwork, noncognitive skills, and self-control were not 

consistently different between Breakthrough and non-program students.

However, teacher reports of non-cognitive skills were significantly different for the two groups.

Teacher consistently gave Breakthrough students higher ratings with regard to being organized, coming 

prepared for class, handling frustration well, and having positive relationships with peers and adults. 

Initial results from the 2013 and 2015 reports indicate an association between participation in the program and improved 

academic and noncognitive skills associated with college success. The continuation and expansion of this longitudinal 

study will allow further examination of the association between program participation and intended and unintended

outcomes, as well as potentially serve as a model for future evaluation studies. Since the program that is the focus of this 

model works with a new cohort of students each year, from the summer prior to sixth grade through college graduation, 

this is a unique opportunity to gather and analyze student, program and district data over time.