An Evaluation of the Jeffco Summer of Early 

Literacy (JSEL) Program 

June 2015
1

Abstract 

The Jeffco Summer of Early Literacy (JSEL) program is a six-week intensive summer school designed to meet the 

needs of at-risk students for targeted literacy instruction and encourage good literacy practices for students 

outside of the classrooms. Students at six selected elementary school sites were placed in small classes by grade 

level or by reading level. This evaluation report contains two key pieces:

1. An implementation evaluation study conducted in summer 2014 by the Jeffco Public Schools

Assessment and Research and Assessment Office. This study was designed to ascertain the extent to 

which the JSEL program was implemented with fidelity and to identify formative recommendations to 

support the program; and

2. An impact evaluation study conducted by Augenblick, Palaich and Associates (APA). This impact study 

was designed to determine the impact of the JSEL program on student academic performance as 

measured by assessment scores. The impact study was part of a planned, multiyear impact study that 

would gather student performance data over multiple years of JSEL program operation.

All material in this report regarding program implementation was authored by the Jeffco Public Schools research 

and assessment office. This material was incorporated by APA into the current report to provide JSEL with a 

unified evaluation document. The Jeffco implementation study utilized data from focus groups, interviews, and 

survey responses to track program implementation during the summer of 2014. Data from these sources 

demonstrated that most teachers, principals and parents valued JSEL for student academic opportunities and 

teacher professional learning. Suggestions from teachers, principals, parents, and coordinators and resultant

recommendations are discussed. All data collection instruments used by Jeffco are provided in the appendices.

All material in this report regarding program impact on student achievement was authored by APA Consulting. 

APA’s impact evaluation presents findings from statistical models comparing outcomes for JSEL participants and 

similar Jeffco students who did not participate in JSEL. These data demonstrate that JSEL participants make 

greater progress in their reading proficiency than similar non-participants, with a difference between groups 

equivalent to moving from the 50th to the 54th percentile on the DIBELS Next assessment.
2

Executive Summary 

This report provides impact and implementation evaluations of the Jeffco Summer of Early Literacy (JSEL) 

program, using summer 2014 program data collected between June 12 and July 25, 2014. The Jeffco Public 

Schools Assessment & Research Office conducted the implementation analysis and APA Consulting (APA) 

conducted the impact analysis. The impact evaluation focuses on kindergarten through third grade students, 

and does not include students in grades four through six. The implementation analysis covers all JSEL students.

JSEL is a six-week summer school program for elementary students who will be entering kindergarten through 

sixth grade in the fall. In summer 2014, JSEL was implemented in six elementary school sites: Edgewater, Lasley 

(housed at Molhom), Pennington, Stevens, and Swanson. Across these sites, the program served 660 students. 

Most these were kindergarten through third grade students (539 students, or 81.7 percent), with 

kindergarteners making up the largest number of students at any one grade level (241, or 36.5 

percent).Attendance rates varied between 80 and 90 percent. Across all sites, the number days absent for 

students ranged from zero to 30, with 30.9 percent of students missing zero days, 66.8 percent of students 

missing zero to four days, and only 33.2 percent of students missing five or more days. 

JSEL provides a number of major services to students, families, and teachers. For students, JSEL offers targeted 

literacy instruction through small group and individual lessons. Teachers use Jeffco produced Comprehensive 

Approach to Literacy Instruction (CALI) and Curriculum Alignment Project (CAP) documents as guidelines for 

literacy instruction. Teachers also conduct regular assessments so that they can continually tailor instruction and 

interventions to their students. Additionally, the JSEL program includes interactions between students’ families 

and program staff and conversations about student progress. This interaction takes place during the two field 

trips in summer 2014, which were open to families as well as students. Interactions also take place at structured 

events like free breakfast (before the school day) and lunch (after the school day). Additionally, “Friday Folders” 

are sent home on a weekly basis to keep families informed, provide academic tools, and gather feedback. For 

teachers, JSEL provides professional learning (PL) before the start of classes and throughout the summer during 

hour-long PL sessions.

For this evaluation, researchers aimed to (1) assess the fidelity of the program, (2) lift up promising practices 

that could be replicated in other, similar sorts of programs, and (3) identify considerations for improvement. 

Based on focus groups, interviews, and survey responses, the implementation analysis found that most 

teachers, principals, and parents valued JSEL for student academic opportunities and teacher professional 

learning. Based on statistical models comparing outcomes for JSEL participants versus similar Jeffco students 

who did not participate in JSEL, the impact analysis found that JSEL participants make greater progress in their 

reading abilitythan similar non-participants, with a difference between groups equivalent to moving from the 

50th to the 54th percentile on the DIBELS Next assessment. Research questions and key findings are summarized 

in the table below.
3

Summary Table

Question 

Number Research Question

Question Type

& Responsible 

Organization Key Findings

C1

Do JSEL participants make 

greater progress in their 

reading proficiency than 

similar non-participants?

Impact: 

ConfirmatoryAPA

APA finds that participation in only one year of JSEL 

provides a statistically significant and positive effect on 

student reading performance. This effect is equivalent to 

moving students from the 50th to the 54th percentile on 

the DIBELS Next literacy assessment.

C2

Does the magnitude of the 

difference between JSEL 

participants and nonparticipants vary as a 

function of the number of 

years students 

participated?

Impact: 

ConfirmatoryAPA

There are small numbers of students with multiple years 

of JSEL participation in the current study sample, making it 

difficult to detect an added effect associated with multiple 

years of JSEL participation. A larger sample of students 

with multiple years of participation is necessary to detect 

a statistically significant effect, if one exists.

C3

Are JSEL participants more 

likely to score “proficient” 

on third grade reading 

TCAP than nonparticipants?

Impact: 

ConfirmatoryAPA

With limited years of data available, only 129 JSEL 

participant students had third grade TCAP scores. With 

this small sample, APA did not find a statistically significant 

difference between JSEL participants and non-participants 

on TCAP performance. A larger sample of students, which 

was called for in APA’s evaluation plan, is necessary to 

detect a statistically significant effect of JSEL on TCAP 

results, if one exists.

C4

Does the proportion of 

participants scoring 

“proficient” increase with 

the number of years 

enrolled in JSEL?

Impact: 

ConfirmatoryAPA

The number of JSEL students with TCAP data is insufficient 

to find a statistically significant association between TCAP 

performance and the number of years of JSEL 

participation. Because of the small number of students 

who participated in JSEL in multiple and have available 

TCAP scores, these results should not be interpreted as 

meaningful.

C5

Do students who attend 

regularly show greater 

growth in reading skills 

than students who attend 

less regularly?

Impact: 

Exploratory-APA

Data is insufficient to find a statistically significant 

association between days of JSEL attendance and 

differences in DIBELS Next fall scores. Because of the small 

number of students available in the study sample, these 

results should not be interpreted as meaningful.

E6

How has participation in 

the program changed 

parents’ perception of their 

Impact: 

ExploratoryJeffco

Families are provided with home literacy environment

information through Friday folders. The majority of 

families responding to the survey also indicated that they 

found the school environments welcoming (96%) and that 

school staff contacted them often (77%). Also, the survey 
4

home literacy 

environment?

showed that 41.4% of responding families read with their 

children often and 39.1% read with them always. 83.8% 

did a homework activity with their child often or always.

E7

How has participation in 

the program changed 

teachers’ approach to 

literacy instruction? Impact: 

ExploratoryJeffco

Teachers reported positive experiences with instructional 

support provided by coaches, which included 

administering assessments and interventions as well as 

availability to respond to questions and concerns. 

E8

What is the overall fidelity 

of implementation of the 

summer school program?

Impact: 

ExploratoryJeffco

Practices related to literacy block and CALI 

implementation such as, mini lessons, whole group, 

shared, guided, and independent reading were reported as 

being, “used on a regular basis.” However, differences 

were reported in the level of classroom readiness, writing 

instruction, administration of interventions, walkthrough 

practices, mealtime practices, and community 

involvement elements.

E9

Do teachers implement 

CALI with fidelity?

ImplementationJeffco

All sites used CALI structures. CALI implementation 

practices, such as mini lessons, whole group, shared, 

guided, and independent reading were “used on a regular 

basis” across sites. However, teachers had different 

practices related to writing instruction and interventions. 

Participants cited different reasons for variations in 

implementation, including resource availability, school 

culture and practice, and logistical complications related 

to funding sources (as discussed previously).

E10

What proportion of 

students attends regularly 

(i.e. receive the 

recommended dosage of 

the intervention)?

ImplementationJeffco

Across sites, the number of days absent for students in 

JSEL ranged from 0 to 30 days with 66.8 percent of 

students missing zero to four days. The average number of 

days missed was 4.29. Most students attended JSEL 

regularly, receiving adequate dosages of the intervention. 

The expanded and new school sites had the three highest 

averages of student absences. The three existing K-3 sites 

had the lowest average of absences. There were higher 

attendance rates mid-week (Tuesday, Wednesday, and 

Thursday) each week, and there were slight declines in 

attendance towards the end of the program.
5

Table of Contents 

Abstract ...................................................................................................................................................................... 1

Executive Summary .................................................................................................................................................... 2

Summary Table....................................................................................................................................................... 2

Overview of Jeffco Public Schools Summer of Early Literacy Program...................................................................... 7

Programmatic Structures....................................................................................................................................... 7

Instruction & Student Assessment.................................................................................................................... 10

Teacher Professional Development & Meetings.............................................................................................. 10

Family Support, Participation, & Engagement................................................................................................. 10

Methods & Results................................................................................................................................................... 11

Student Demographics......................................................................................................................................... 11

Enrollment........................................................................................................................................................ 11

Gender Composition......................................................................................................................................... 12

Race & Ethnicity................................................................................................................................................ 13

Individual Education plan................................................................................................................................. 14

Attendance........................................................................................................................................................... 15

DRA Assessment................................................................................................................................................... 19

Focus Groups and Interviews............................................................................................................................... 21

Parent Feedback................................................................................................................................................... 31

End-of-Year Surveys. ............................................................................................................................................ 32

Recommendations ................................................................................................................................................... 35

Improved Communication ................................................................................................................................... 35

Adjust Professional Learning................................................................................................................................ 36

Increase Teacher-driven Planning time................................................................................................................ 36

Provide IT Support................................................................................................................................................ 36

Create a Consistent Student Behavior System..................................................................................................... 36

Consider Planning of Field Trips........................................................................................................................... 36

Impact Analysis......................................................................................................................................................... 37

Impact Research Questions.................................................................................................................................. 38

Overview of Analysis ............................................................................................................................................ 38

Sampling Eligibility................................................................................................................................................ 39

Constructing Comparison Groups........................................................................................................................ 40
6

Statistical Analysis................................................................................................................................................ 43

Question 1: Do JSEL participants make greater progress in their reading proficiency than similar nonparticipants?..................................................................................................................................................... 44

Question 2: Does the magnitude of the difference between JSEL participants and non-participants vary as a 

function of the number of years students participated?................................................................................. 45

Question 3: Are JSEL participants more likely to score “proficient” on third grade reading TCAP than nonparticipants?..................................................................................................................................................... 46

Question 4: Does the proportion of participants scoring “proficient” on third-grade reading TCAP increase 

with the number of years enrolled in JSEL?..................................................................................................... 47

Question 5: Do students who regularly attend JSEL show greater growth in reading skills than students who 

attend less regularly? ....................................................................................................................................... 48

Next Steps................................................................................................................................................................. 49

Conclusion ................................................................................................................................................................ 50

Appendix A: Teacher Focus Group Protocol ............................................................................................................ 52

Appendix B: Principal Interview Protocol................................................................................................................. 55

Appendix C: Instructional Coach Focus Group Protocol .......................................................................................... 57

Appendix D: Program Coordinator Interview Protocol............................................................................................ 60

Appendix E: Weekly Parent Survey .......................................................................................................................... 62

Appendix F: Parent End-of-Year Survey ................................................................................................................... 63

Appendix G: Statistical Model Output ..................................................................................................................... 66
Overview of Jeffco Public Schools Summer of Early Literacy Program 

The Jeffco summer of Early Literacy (JSEL) program is a six-week summer school program that provides targeted 

literacy instruction to elementary school students. Over the course of the summer, students are given small 

group and individually targeted instruction. In prior years of the program’s operation (summer of 2012 & 2013), 

JSEL served K-3

rd grade students at four Jeffco Elementary schools: Edgewater, Lasley, Molholm and Swanson. In 

2014, the program was expanded to include two new school sites, Pennington and Stevens, as well as 4

th

-6

th

grade students in certain schools. Figure 1 depicts the JSEL program school sites and the grades they served in 

the summer of 2014.

Figure 1. Jeffco Summer of Early Literacy participating schools and grades 

Schools Grades Served

Edgewater K-3

Lasley* K-3

Molholm K-3

Pennington K-6

Stevens K-4

Swanson K-6

*Due to construction the students, teachers, and administrators from Lasley were housed at Molholm.

Figure 2 depicts the theory of change for the program, outlining the planned immediate, intermediate, and 

ultimate intended outcomes. The only changes to this model throughout the study period were an expansion of 

the program to 4th to 6th grade students in certain schools in 2014. While some 4th to 6th grade students were 

served in that year, the core focus of the program remained K through 3rd grade student, who made up about 

89% of students served.
Figure 2: Theory of Change for Jeffco Early Literacy program 

The ultimate intended outcome of the Jeffco Early Literacy program is that 100% of students 

across the district, regardless of economic, minority, language or other status, will graduate high 

school on-time, prepared for the world of work and/or higher education. 

The intermediate outcomes include that 3rd

grade state-mandated test scores in reading at 

pilot school sites will improve by at least 25% 

from those recorded in the 2011-2012 school 

year to those in the 2015-2016. 

The intermediate outcomes include that Jeffco 

Public Schools will implement a district-wide 

summer literacy program, accessible by all P-3 

students when/if financial resources become 

available. 

The immediate outcomes

include that pilot project P-3 

students will demonstrate 

improved literacy skills 

acquisition in the five core 

areas of literacy development 

each year of the program’s 

pilot implementation (2012-

2016)

The immediate outcomes

include that pilot project P-3 

teachers will demonstrate 

improved instructional skills 

and literacy block7

competency each year of the 

program’s pilot 

implementation (2012-2016) 

The immediate outcomes

include that families of pilot 

project P-3 students will 

develop at-home literacy 

skills and practices to support 

their children’s literacy and 

educational development 

during the program’s pilot 

implementation (2012-2016)
9

Strategy 1: Students will be assessed at the beginning of 

the program to determine benchmark literacy; each 

summer from preschool to 3

rd grade, students will 

participate in 90 hours of classroom instruction and 

curricula and at-home practice tailored to their specific 

literacy deficiencies; summer instructional programming 

will complement and build from and to regular schoolyear curricula and literacy instruction.

Strategy 2: Teachers will receive eight additional hours of dedicated professional 

development around literacy instruction; will work with students’ regular school 

teachers to develop and implement individual interventions for specific literacy core 

competencies; will develop familiarity and comfort with assessment and feedback; will 

receive support from on-site coaches and professional development providers; and will 

assist students’ new teachers each fall with continuing to build on students’ literacy 

skills development.

Strategy 3: Families of students participating in the program will attend at least two parent 

night events at their child’s school to learn more about classroom activities and effective 

literacy practice activities to be used in the home; parents will work with teachers at parent 

night events on modeling literacy behaviors; parents will receive materials and coaching from 

teachers to encourage family literacy outside the classroom.

Assumptions: (A) Early literacy deficiencies can be overcome with additional instructional time to 

help students catch up and then not fall behind over the summer months; (B) Teachers who are 

given appropriate baselines, goals and strategies for success with an individual child can affect the 

academic development of that child; (C) Teachers who receive additional coaching and professional 

development can become more skilled and effective teachers in the classroom; (D) Families who 

are given coaching, materials and an opportunity to practice literacy with their children will choose 

to encourage their children’s development of literacy skills; (E) Students who can read proficiently 

at 3rd grade will continue advancing in their academic achievement and will graduate high school 

on time; (F) School districts that see positive results from pilot projects will prioritize the 

implementation of the project across all appropriate schools
10

Programmatic Structures 

This section provides an overview of programmatic structures including instruction and assessment practices, 

professional learning for teachers, and elements of community engagement. From an instructional lens, student 

expectations and the day-to-day practices of the program are described. Professional learning is explained in 

detail including the mechanisms created to develop teacher skills and approaches, and structures designed to 

encourage community involvement with families.

Instruction & Student Assessment

In JSEL, classroom assignments depend on student ability, age/grade, student-teacher ratio, and instructional 

decisions made at each individual school. Classes were designed to have a small student to teacher ratio, with 

no more than fifteen students assigned to a classroom. 

Comprehensive Approach to Literacy Instruction (CALI) and Curriculum Alignment Project (CAP) documents were 

used as guidelines for literacy instruction. For instance, literacy block structures included the use of mini lessons, 

whole group, shared, guided, and independent reading with a written response, as well as the use of oral 

language expression and the listening CAP document. The format of the block and its segments differed 

between student levels. For Kindergarten classrooms, the block lasted 90 minutes. In 1

st through 6th grade

classrooms the literacy block lasted 180 minutes. However, the time spent on each segment differed between 

1

st – 3rd grade students and 4th through 5th grade students. 

Teachers were also asked to conduct assessments on a regular basis and were encouraged to use the 

information to tailor instruction and assign intervention groupings. Running records were administered weekly, 

as well as DIBELS progress monitoring for students who were categorized as strategic and intensive. Jeffco best 

practices were explained and communicated to teachers during professional learning (PL) provided in June and 

reinforced throughout the summer session through on-going PL meetings.

Teacher Professional Development & Meetings

Teachers were provided with professional learning (PL) before the start of classes. Over a span of three days,

teachers spent nine hours reviewing program expectations including: literacy block structures (using CAP and 

CALI), and ready classroom practices. These initial meetings also provided teachers with opportunities to learn 

more about differentiated instructional strategies. For deeper learning, teachers were given a choice of two 

topics; guided reading and running records. 

Ongoing learning occurred through hour-long PL opportunities scheduled throughout the summer. This 

included three all-site meetings and four site-based meetings. The content of the meetings was based on 

demonstrated teacher need uncovered during walkthroughs. During these walkthroughs, coaches and 

administrators observed and recorded practices related to the literacy block structures, ready classroom 

environment, and oral language. As an example, two PL topics were using oral language and turn-and-talks. 

Family Support, Participation, & Engagement

Structures to encourage the support and engagement of families were established. These structures were 

designed to foster communication, build community, and offer opportunities for family members to participate. 

Community meals (breakfast and lunch), field trips, and sending home Friday Folders were all a part of the 

family engagement efforts.
11

Breakfast was served before the start of the school day and lunch at the end of the school day at no cost to 

family members. The purpose was to encourage families to interact with program staff (principals, teachers, and 

coordinators), to participate in conversations about students’ academic progress, and to create a welcoming 

environment for students and families.

Additionally, two field trips to the Downtown Aquarium in Denver and the Wildlife Experience in Parker were 

planned during the summer session. Families were allowed to attend field trips at no cost. There was no limit on 

the number of family members allowed to attend. The field trip locations were chosen based on their relevance 

to the program’s ocean theme.

Friday folders were sent out on a weekly basis. The purpose of this folder was to keep families informed about 

their child’s progress, provide tools for families to cultivate home literacy environments, and to gather feedback 

about family participation in program activities. As such, folders contained examples of student work, suggested 

literacy activities for families, and included participation feedback questionnaires. Feedback questionnaires 

asked families if they read with their child at home, participated in literacy activities (e.g. homework), attended

field trips, and attended community meals.

Methods & Results 

The purpose of the Jeffco implementation evaluation was to assess the fidelity of the program, lift up promising 

practices that could be replicated, and identify considerations for improvement. Data were collected using 

multiple sources and methods including, interviews, focus groups, surveys, district student demographic data, 

attendance records, and DRA 2 progress monitoring data. The following section describes the methods and data 

sources, as well as findings and insights gained from their analysis.

Student Demographics 

Student demographic data were gathered for every JSEL student from a district enrollment database. Students 

who were enrolled after the beginning of the summer session (June 25th), as well as students who enrolled 

before the beginning of the summer session were included in this analysis. This section focuses on the 

composition of the JSEL’s student population and its comparison to the composition of participating sites during 

the regular school year. Completion of the program was not a prerequisite for being included in these analyses.

Enrollment 

Principals and schools were asked to recruit students to participate in the program and used various methods to

encourage enrollment. Recruitment activities included contacting parents, having teachers identify students 

who might benefit from inclusion, and disseminating communication via home folders, parent information 

nights, and other forms of parent outreach. There was no pre-determined enrollment cap for any of the 

participating schools. 

The program served a total of 660 students across all sites. Sixty-nine percent of students (n = 458) were 

enrolled at the four original school sites (Edgewater, Lasley, Molholm, and Swanson). Meanwhile, thirty-one 

percent of students (n = 202) were enrolled at the new school sites (Pennington and Stevens). Figure 3 shows 

the number of students enrolled at each school site. 

Figure 3. Student Enrollment by School 
12

School Name Number of Students

Edgewater 136

Lasley 44

Molholm 124

Pennington 142

Stevens 60

Swanson 154

Kindergarten had the largest population of students with 241 (36.5%) students and 6th grade had the smallest 

population of students with 11 (1.7%). The previously existing grades, Kindergarten through 3rd grade, made up 

88.6% of the student population. Meanwhile grades that were added this past summer, 4

th through 6th grade,

made up 11.4% of the population. Figure 4 shows the number of students enrolled by grade. The impact 

evaluation conducted by APA Consulting focuses on students in Kindergarten through 3rd grade, and does not 

include those in 4th through 6th grade.

Figure 4. Student Enrollment by Grade 

Grade Number of Students

(Percentage)

Kindergarten* 241

(36.5%)

1

st Grade 153

(23.2%)

2

nd Grade 145

(22.0%)

3

rd Grade 46

(7.0%)

4

th Grade 33

(5.0%)

5

th Grade 31

(4.7%)

6

th Grade 11

(1.7%)

*This count includes preschool students going into Kindergarten.

Gender Composition

All but two JSEL school sites (Stevens and Molholm) had more boys than girls enrolled. The gender composition

between the JSEL enrollment and enrollment during the school year tended to be similar. Stevens and Molholm 

were the only two schools with gender enrollment disproportionalities greater than 3%. Figure 5 shows the 

gender composition of enrollment by school site comparing the JSEL program to overall school enrollment.

Overall JSEL percentage of 51.5% boys was similar to the overall school population of 52.8% boys.

Figure 5. Student Gender by school 
13

School Name Number of Boys

(Percent)

Number of Girls

(Percent)

Number of Boys

(Percent)

Number of Girls

(Percent)

Edgewater 74

(54.4)

62

(45.6)

233

(52.4)

212

(47.6)

Lasley 24

(54.5)

20

(45.5)

301

(54.4)

252

(45.6)

Molholm 54

(43.5)

70

(56.5)

246

(52.6)

222

(47.4)

Pennington 77

(54.2)

65

(45.8)

127

(51.4)

120

(48.6)

Stevens 24

(40.0)

36

(60.0)

153

(51.5)

144

(48.5)

Swanson 87

(56.5)

67

(43.5)

218

(52.9)

194

(47.1)

Race & Ethnicity 

Among students for whom race and ethnicity data were available, Hispanic students represented the largest 

ethnic subgroup across all six school sites. Hispanic students made up 63.9% of the JSEL Enrollment population. 

The second largest ethnic subgroup was Caucasian; accounting for 27.4% of the JSEL Enrollment population. 

Asian Pacific Islander was the smallest ethnic subgroup in enrollment; only representing .2% of the student 

population. Figure 6 shows the racial and ethnic composition by school site for JSEL and Figure 7 shows the racial 

and ethnic composition of participating school sites during the regular year.

Figure 6. Race and Ethnicity of student enrollment by school in the JSEL Program 

School Name AI

(%)

API

(%)

AS

(%)

B

(%)

H

(%)

M

(%)

W

(%)

Edgewater 1

(0.7)

1

(0.7)

0

(0.0)

3

(2.2)

113

(83.1)

2

(1.5)

16

(11.8)

Lasley 3

(6.8)

0

(0.0)

5

(11.4)

2

(4.5)

21

(47.7)

2

(4.5)

11

(25.0)

Molholm 2

(1.6)

0

(0.0)

3

(2.4)

1

(.8)

98

(79.0)

3

(2.4)

17

(13.7)

Pennington 0

(0.0)

0

(0.0)

0

(0.0)

2

(1.4)

75

(52.8)

1

(.7)

64

(45.1)

Stevens 2

(3.3)

0

(0.0)

0

(0.0)

0

(0.0)

39

(65.0)

5

(8.3)

14

(23.3)

Swanson 1

(0.6)

0

(0.0)

13

(8.4)

2

(1.3)

76

(49.4)

3

(1.9)

59

(38.3)

*AI= American Indian or Alaska Native; API = Native Hawaiian or Other Pacific Islander; AS = Asian; B = Black or African American; H = Hispanic or Latino;

M = Two or more races; W = White

Figure 7. Race and Ethnicity of student enrollment by school in the during the 13/14 school year 

Overall School Enrollment

School Name AI

(%)

API

(%)

AS

(%)

B

(%)

H

(%)

M

(%)

W

(%)

Edgewater 3

(0.7)

3

(0.7)

0

(0.0)

12

(2.7)

358

(80.4)

7

(1.6)

62

(13.9)
14

Lasley 8

(1.4)

2

(0.4)

42

(7.6)

7

(1.3)

361

(65.3)

16

(2.9)

117

(21.2)

Molholm 7

(1.5)

0

(0.0)

6

(1.3)

10

(2.1)

366

(78.2)

12

(2.6)

67

(14.3)

Pennington 5

(2.0)

0

(0.0)

0

(0.0)

2

(0.8)

124

(50.2)

2

(0.8)

114

(46.2)

Stevens 7

(2.4)

0

(0.0)

3

(1.0)

4

(1.3)

168

(56.6)

9

(3.0)

106

(35.7)

Swanson 3

(0.7)

2

(0.5)

28

(6.8)

10

(2.4)

185

(44.9)

11

(2.7)

173

(42.0)

*AI= American Indian or Alaska Native; API = Native Hawaiian or Other Pacific Islander; AS = Asian; B = Black or African American; H = Hispanic or Latino; 

M = Two or more races; W = White

Individual Education plan

Students with individual education plans (IEPs) made up 15.3% of the student population. Pennington had the 

highest number and percentage of students with an IEP; 30 students or 21.1% of Pennington students had an 

IEP. Stevens had the second largest percentage (20%) and the third largest number of students with IEPs.

Swanson had the third largest percentage of students with IEPs at 16.5%. Figure 8 depicts the number and 

percent of JSEL students who had an IEP.

Figure 8. Individual Education Plans by school among students enrolled in JSEL 

School Name Number of Students with an IEP

(Percent)

Edgewater 16

(11.8)

Lasley 4

(9.1)

Molholm 14

(11.3)

Pennington 30

(21.1)

Stevens 12

(20.0)

Swanson 25

(16.2)

English Proficiency 

The JSEL program was open to all students regardless of English proficiency and had a total of 104 (15.8%) 

students enrolled that were designated as Not English Proficient (NEP) and 116 (17.6%) designated as Limited 

English Proficiency (LEP). Fifty-five percent of Edgewater students were designated as either NEP or LEP, which 

was the highest percentage of students with NEPs or LEPs among the JSEL schools sites. Figure 9 shows the 

number and percent of NEP and LEP students by site.

Figure 9. Non-English proficiency and limited English proficiency by school among students enrolled in JSEL
15

School Name Number of LEP Students

(Percent)

Number of NEP Students

(Percent)

Edgewater 37

(27.2)

38

(27.9)

Lasley 7

(15.9)

3

(6.8)

Molholm 21

(16.9)

35

(28.2)

Pennington 11

(7.7)

7

(4.9)

Stevens 13

(21.7)

8

(13.3)

Swanson 27

(17.5)

13

(8.4)

All JSEL sites except Lasley had a higher percentage of NEP/LEP students enrolled in JSEL than the 2013/14

school year. Figure 10 shows the number and percent of NEP/LEP students during the summer session and 

2013/14 school year.

Figure 10. Non-English proficiency and limited English proficiency by school among students enrolled in JSEL 

and enrollment during the regular school year 

School Name

JSEL Enrollment

Number of NEP/LEP Students

(# of students)

2013/14 Enrollment

Number of NEP/LEP Students

Edgewater 55.1%

(75)

47.2%

Lasley 22.7%

(10)

27.7%

Molholm 45.2%

(56)

43.8%

Pennington 12.7%

(18)

11.3%

Stevens 35.0%

(21) 20.2%

Swanson 26.0%

(40)

23.5%

Attendance 

Attendance was gathered in each classroom and recorded using Infinite Campus. Attendance records were 

analyzed to understand trends in student attendance. Three student records showed 31 days of absence (every 

day of the program’s operation) and these instances were excluded from the following analysis. Across all school 

sites, the average number of days missed was 4.29 days. The expanded and new school sites had the three 

highest averages of student absences. At Stevens students averaged 5.32 days of absence, the highest of all 

school sites. In contrast, the three existing K-3 sites had the lowest average of absences. Edgewater Elementary 
16

students were absent on average 2.65 days, the lowest average of all school sites. Figure 11 shows the mean of 

absences for each school site.

APA’s impact evaluation, described later in this report, examines whether variations in student attendance are 

related to test score outcomes

Figure 11. Average number of absences by school site

School Site Average Days Absent

(n)

Edgewater 2.65

(136)

Lasley 2.82

(44)

Molholm 4.69

(124)

Pennington 4.85

(140)

Stevens 5.32

(60)

Swanson 4.94

(153)

Across all school sites, the number of days absent for students in JSEL ranged from 0 to 30 days with 66.8% of 

students missing 0-4 days. Additionally, 30.9% of students had no absences. Meanwhile, two students (.3%) 

were absent thirty days and only 33.2% of students had five or more absences. Figure 12 illustrates student 

absences.
17

Figure 12. Student Absences

An examination of student attendance over time demonstrated a slight decline in student attendance toward 

the end of the program’s operation. Aside from the first day of school, (June 12th) the day with the lowest

attendance was July 21st with 81.3% of the student population in attendance. Records also demonstrate that, 

generally, attendance is higher during the mid-portion of the week (Tuesday, Wednesday, and Thursday) and 

lower at the beginning and end of the week (Monday and Friday). Figure 13 depicts student attendance over 

time across all sites.

203

88

49 56

43 36

27 19 12

21

11 15 15 13 6 7 8 4 2

7 2 2 4 2 1 1 1 2

0

50

100

150

200

250

0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 27 29 30

Number of students

Days Absent
18

Figure 13. Percent of Students Present by day for all sites

When comparing newly introduced school sites to existing school sites (including Swanson), attendance in the 

first few days was higher at new school sites than existing school sites. Ninety-one percent of students at new 

school sites attended on June 12th, the first day of school. Meanwhile, 74 % of students at existing sites attended

that day. However, as the summer session progressed attendance at new school sites decreased more sharply 

than existing school sites. At new sites, 76.2% of students were present July 24th, the second to the last day and 

80.2% were present the last day of school. Existing sites had 85.4% of students present on June 24th and 82.8% 

of students present on the last day of school. Figure 14 depicts a comparison of student attendance between 

existing school sites and new school sites.

0.0

10.0

20.0

30.0

40.0

50.0

60.0

70.0

80.0

90.0

100.0

Percentage of Students Present

Program Dates
19

Figure 14. Numbers of Student Present by week

DRA Assessment

Teachers were asked to keep daily running records and at the end of the summer, students were assessed using 

the DRA 2 progress monitoring tool. Out of 660 students, six student records were incomplete (e.g. invalid 

student IDs, missing or incorrectly entered data) and 353 students did not have data available. There are various 

reasons for missing DRA 2 data including: absences, grade level (e.g. DRA 2 is not administered to preschool 

students), and logistical issues recording data. Administrators, teachers, and coaches proctoring the assessment 

were asked to record data using a Google form. Since administrators, teachers, and coaches could not edit the 

spreadsheet in which the data populated, proctors were told that if a mistake was made when entering data

then they could make a corrected entry. If corrected entries were recorded, then the most recent entry would 

be chosen as the final. However, no delineation was made between corrected entries and a student being 

assessed twice. 

DRA 2 examines a variety of literacy components including reading fluency and comprehension. From these 

components students are placed in levels ranging from A – 80 and within those levels students are categorized 

as, intervention, instructional, independent, or advanced. In order to be considered proficient at grade level a 

student must be categorized as independent or advanced within the DRA Level cut points which correspond to 

0

10

20

30

40

50

60

70

80

90

100

Percentage of Students Present

New School Sites

Previously

Existing Sites
20

their grade.1 Below, figure 15 is a table adapted from the DRA 2 Technical Manual that depicts the DRA Level cut 

points for each grade.

Figure 15. DRA 2 Level Cut Points by Student Grade 

Grade Time Proficient/Independent

Kindergarten Fall Pre A

Mid-Year 1

Spring 3

1

st Grade Fall 3

Mid-Year 8

Spring 16

2

nd Grade Fall 16

Mid-Year 20

Spring 28

3

rd Grade Fall 28

Mid-Year 34

Spring 38

4

th Grade Fall 38

Mid-Year 38

Spring 40

5

th Grade Fall 40

Mid-Year 40

Spring 50

6

th Grade Fall 50

Mid-Year 50

Spring 60

Since the program takes place during the summer, proficiency for students was calculated based on the spring 

cut points outlined in the DRA 2 Technical Manual. Kindergarten had the highest percentage (70.9%) of students 

that scored at or above proficient, while 6th grade had the lowest percentage (11.1%) of students that scored at 

or above proficient. Figure 16 depicts the number and percentage of students that scored proficient or above 

and the number and percentage of students not proficient in each grade.

 

1 See Pearson (2011). DRA 2 K-8 Manual Developmental Reading Assessment. Retrieved from 

http://assets.pearsonschool.com/asset_mgr/current/20139/DRA2_Technical_Manual_2012.pdf
21

Figure 16. Reading proficiency by Student grade

Student Grade

(n)

Number of Students Proficient or 

Above

(%)

Number of Students Not 

Proficient

(%)

Kindergarten

(86)

61

(70.9%)

25

(29.1%)

1

st Grade

(95)

40

(42.1%)

55

(57.9%)

2

nd Grade

(77)

31

(40.3%)

46

(59.7%)

3

rd Grade

(20)

4

(40.3%)

16

(80.0%)

4

th Grade

(13)

5

(38.5%)

8

(61.5%)

5

th Grade

(4)

1

(25%)

3

(75%)

6

th Grade

(9)

1

(11.1%)

8

(88.9%)

The DRA 2 data used in APA’s impact study described later in this report was not from the DRA administration 

during the JSEL program, but was from the fall administration of the DRA 2 in each student’s home school.

Focus Groups and Interviews 

Focus groups with teachers and coaches, as well as interviews with coordinators and principals, were conducted 

during the final two weeks of the program. This included three teacher focus groups, two instructional coach 

focus groups, and interviews with each principal and coordinator. A total of 44 staff participated in interviews or 

focus groups including: 29 teachers, 5 coaches, 6 principals, and 4 coordinators. The following table shows the 

focus groups, the number of participants in each focus group, and what school sites were represented in each.

Figure 17. Focus Group Participants

Location of Focus 

Group/Participant Type

Number of 

Participants

Sites Represented

Edgewater/Coaches 3 Edgewater, Stevens, Swanson

Edgewater/Teachers 10 Edgewater, Lasley, Molholm, Pennington

Molholm/Coaches 2 Lasley, Molholm

Pennington/Teachers 9 Pennington, Stevens

Swanson/Teachers 10 Swanson

All Participants were asked questions pertaining to program structures, literacy instruction, professional 

development, and community involvement (for interview and focus group questions see Appendix A-D). Each 

discussion was analyzed for themes across all sites and between participant groups.
22

Themes 

Across all groups the following four themes emerged: the need for improvement in communication structure 

and delivery, cultivation of community through meals, field trips, and small class size, considerate planning of 

events and field trips, and the value of supportive instructional coaches. Although all groups provided different 

perspectives, the themes mentioned above were a common thread found in interviews with principals and 

coordinators, as well as the focus groups with teachers and coaches. Other overlapping and independent 

themes emerged. Figure 18 shows the independent and overlapping themes from each participant group. A 

discussion of all themes is presented in this section. 

Figure 18. Focus Group Themes Across all Participant Groups 

Overlapping Themes Across all groups 

Communication. Among themes that were present across all participant groups the most common was 

communication difficulties, especially in relation to the timing and clarity of information. Many participants 

described the timing of communication as, “frustrating,” “stressful,” and “annoying.” Difficulties such as these 

were especially salient for school sites that were new to JSEL. When discussing the timing of the information one 
23

participant said, “We kept on asking for a calendar with the dates and the times and the commitment and we 

didn’t get that until like two days before JSEL started.” While another said, “It felt like a lot of times they forgot 

us...Then we had to be rushed to finish what they needed.” Participants also reported having received various 

incongruent messages. In one instance a participant reported receiving confusing information about 

implementation of CAP, “They were telling us use the first unit of CAP and then use the second unit of CAP. And 

by the end of the third day we figured out which unit to use when.” Information regarding field trips, schedules, 

meetings, professional development, walkthroughs, time commitment, and expectations were among the 

information all schools felt were not communicated effectively. Similarly, coordinators needed information 

pertaining to student and teacher recruitment that they felt they were unable to ascertain in a timely manner.

Although various suggestions were made to improve communication, participants’ recommendations

demonstrate that comprehensive documentation about program structures, schedules, and expectations, as 

well as a considerable amount of lead time for information is desired.

Program Events. Another theme that emerged across all participant groups was the need for improved planning 

of program events. For instance, field trips were seen as needing more “thoughtful planning” aimed at

improving issues related to transportation, location, and schedule. Many reported that the combination of a 

short school day and the distance to field trip locations caused the field trips to feel “rushed.” One participant 

explained that, “The time allotted to travel to the field trip did not match. So the kids didn't even get to do the 

field trip. They only got to do IMAX.” While others suggested altering the schedule to allow more time for field 

trips, “I would like to see more flexibility in the school day on a field trip day... [They] only got a short little snap 

shot of it because of the rigidity of the school day.” A number of suggestions were posed by participants

including: more autonomy for schools to choose local field trips, allow more parent autonomy in transportation 

choices (e.g. school bus or families to drive together), and altering the length of the school day to allow students 

and their families to receive the full experience of the field trips.

In spite of this, participants viewed the field trips as an excellent conduit for relationship building. In fact, field 

trips, mealtimes, and smaller class sizes were reported as contributing to a sense of community. This theme was 

discussed across all participant groups. For example, participants reported an “excess” of parent volunteers for 

field trips. Participants also reported meaningful conversations with families about students’ education while on 

field trips. One participant described parents’ experience saying, “They were becoming so much more 

vulnerable with us. Like admitting their faults and how we could help them...I think they had a huge learning.” 

Participant discussion also conveyed that small class sizes allowed teachers, principals, and coordinators to 

spend more time learning about students, while breakfast and lunch provided a place for participants to interact 

with families. Positive comments related to breakfast and lunch included the following: “[During the school year] 

you never get to talk to parents... But being able to sit with the parents and eat breakfast with them and eat 

lunch with [th]em. I know so much more about our families. And I love that.” And, “I think one really rewarding 

part of JSEL is when we’re doing the breakfast and the lunch and getting a chance to talk with parents... It was 

less time for me to plan but it was really an opportunity for meeting with parents.” Participants responded

positively to these aspects of the program and expressed a desire for them to continue in the future.

The theme discussed across all groups was positive experiences with instructional support provided by coaches.

Participants said that coaches provided necessary support by administering assessments and interventions, as 

well as being readily available to respond to questions or concerns instructors had throughout the program. One 
24

participant described coaches as, “The most effective component of the summer school structures.” While 

another expressed appreciation for the level of support they received from coaches, “They’ve been in our 

classrooms...Really helping more than in the past few years.” These comments demonstrate participants

appreciated instructional support from coaches and a desire to continue in future years.

Overlapping Themes Across Particular Groups

Figure 19. Overlapping themes for distinct participant groups

In addition to the four themes that appeared throughout all participant groups, overlapping themes across 

differing participant groups also emerged. For teachers, instructional coaches, and principals the improvement 

of professional learning through differentiation was a common theme, while difficulties with logistical tasks was 

a common theme among instructional coaches, coordinators, and principals. Another overlapping theme for 

teachers and coaches was finding and accessing resources.
25

Professional Learning. The theme of professional learning for principals, teachers, and coaches focused on 

discussion of improvement through differentiation and depth of knowledge. Although all-site professional 

learning (PL) was described as “good reminders,” “organized,” and “presented well,” participants viewed sitebased PL as, “targeted,” “relevant,” and “valuable.” When asked to elaborate, participants said that they 

preferred the site-based PD because it, “pertained to a particular school.” Additionally participants described allsite PL as “rushed” and “a social scene” and conveyed that these meetings were less productive than the sitebased meetings. For these reasons participants suggested altering the PL to be “cafeteria-style” where each site 

presents a topic and teachers can choose what PL to attend. They also suggested having “less frequent” but 

“longer” meetings in order to adequately discuss the subject matter, as well as changing the schedule of the 

initial June PL to allow teachers more time for planning and classroom set-up. Topics that were of interest for 

participants include writing, using resources, and guided reading within a skill set (e.g. grouping by fluency).

Logistics. Meanwhile difficulties with logistics, such as invoices, acquiring subs, and ordering supplies, was a 

common theme for principals, coaches, and coordinators. These difficulties were attributed to communication 

problems (as mentioned in the previous section) and unforeseen complications related to having two separate 

funding sources. One participant said, “We were able to bring on more schools through the Title component. 

But it... brought on another level of complexity with logistics of field trips and what schools are doing.” Another

participant described the problems saying, “The supplies and materials was a nightmare... Because some of 

them were ordered through Title and some were ordered through JSEL. They’re not two separate programs but 

it’s those little budgeting and finance pieces that are a pain.” In light of this participants expressed a desire for 

improved communication pertaining to logistical considerations when accomplishing administrative tasks. 

Another suggestion was refining instructions and administrative tasks in order that there might be a more 

systematic way of processing paperwork for separate funders.

Resources. Another overlapping theme for teachers and coaches was difficulty in finding and using resources. 

Teachers and coaches expressed that leveled books and other resources were difficult to find, especially for 

participants who were unfamiliar with a particular school site, “The only thing that the school had was MONDO 

and so all of my personal stuff was located at my other school... I just didn’t feel like there was a whole [lot of] 

resources provided for a kindergarten teacher or for me.” Teachers who were familiar with a site’s resources

reported having an easier time finding and navigating resources. In addition to books, teachers and coaches also 

mentioned needing technological resources (e.g. iPads, computers with more battery life or bandwidth, 

document cameras, access to software), as well as IT support. One participant described this saying “I could not 

get the computers. I couldn’t get the code, [because I’m] an outsider.” Participants reported that finding and 

using resources were a considerable challenge in administering interventions, planning lessons, and 

accomplishing instructional tasks.
26

Independent Group Themes

Figure 20. Independent Themes for Particular Participant Groups

•

•

•

Frustration with non-standardized procedures for student discipline and accountability for homework.

Teachers, instructional coaches, and coordinators each had key themes emerge that were unique to their role. 

Planning time, student behavior, and burnout/stress were common themes among teachers. Meanwhile, 

support for new teachers was a common theme in discussions with coaches. Finally, program planning and 

creation was a topic shared by coordinators.

Independent themes emerging from teacher discussions demonstrated that teachers experience:

Diminished self-driven planning time.

Burn out and stress related to competing expectations and rigor of the summer program.

Behavior. Teachers reported difficulty with students exhibiting disruptive and/or disobedient behavior, as well 

as non-compliance with rules related to homework and Friday Folder submittal. Teachers found that frustration 

with disciplinary and accountability issues were exacerbated because, “It didn’t seem like there was a clear set-
27

up as far as what consequences could be.” One participant described the difficulties and said, “Whether it’s 

behavior or homework. Kids not doing [th]em. My hands were tied.” In this way, teachers described uncertainty 

for how to handle disciplinary issues that they encountered. This uncertainty extended to enforcing rules related 

to homework and Friday Folders. Discussions pertaining to accountability for homework assignments were

mixed. Some teachers advocated for both rewards and punishments, while others only desired a rewards based 

system. Meanwhile, another group urged eliminating homework altogether. Regardless, teachers felt they 

needed a system that kept students accountable for their schoolwork and provided clear consequences for 

disobedient and disruptive behavior.

Planning Time. Limited planning time was another theme extensively discussed among teachers. Participants 

reported that although the 30-minute coaching time was intended to assist teachers in lesson planning, it was 

often more like a “meeting” used to discuss schedules and program updates. Although coaches were seen as 

helpful, participants also expressed a desire for more autonomy during planning time. One participant said, “The 

first week I was like, great I have this thirty minute period that I can get done some of these things, but our 

planning was planned for us.” This diminished time for planning and lack of independence when making lessons 

was a sentiment that was shared in discussions among teachers. 

Stress & Burnout. Diminished planning time, rigor of program, and competing expectations were all mentioned 

as contributors to a feeling of stress and burnout among teachers. The need for respite from the burnout of the 

program was a theme pervasive in teacher focus groups. Although most teachers found community involvement 

and student achievement to be rewarding, they also communicated that the workload was large and oftentimes 

stressful, “This program is what’s right for kids, but the exhaustion it takes out on the adults.” Although some 

participants’ commented that the changed schedule helped alleviate this, many still relayed a feeling that more 

time was needed to help teachers recuperate from the stress and fatigue of the summer session, “We’re really 

beaten up through the school year... We do need some time to re-coop and some of us don’t have that time 

now.” Participants mentioned that some of the contributing factors that added to the stress were, diminished 

planning time (as mentioned above), the amount of assessments, added time commitment (e.g. meetings), and 

the duration of the program’s session. Various suggestions were made by participants to help alleviate the stress 

and burnout felt by teachers including, shortening the program to 4 or 5 weeks, having a 4-day school week with 

a teacher planning/meeting day on Friday, and reducing the amount of assessments administered (e.g. choosing 

to administer only DIBELS or only running records).

New Teachers. Coaches also discussed these stressors and found that they were especially difficult for new 

teachers. A common thread among coaches was the support needed for new teachers. Participants felt that

navigating the school site, acclimating to Jeffco practices, and learning teaching skills (e.g. classroom 

management) was a challenge for new teachers, “When you had teachers that were new to Jeffco... [You have 

to get] them connected to DIBELS and having access to SOARS... And so then you’re not only teaching them 

instruction. You’re teaching them the Jeffco way.” In this way, coaches communicated a struggle with addressing 

the needs of new teachers while still providing enough support for more seasoned teachers. New teachers were 

also reported as having more stress, “[I had] new teachers... They felt that it was demanding... [I tried] to make it 

easier for them and taking some load off them, but it was just really challenging to do that.” For this reason 

coaches felt that the rigor and expectations of the program were too demanding for many new teachers.
28

Planning & Logistics. The independent themes discussed above surrounded instruction and instructional 

support. In contrast, the theme most salient in Coordinator discussions surrounded the logistical tasks related to 

program operations. Discussion with coordinators demonstrated that program planning around its ocean theme

was a challenge, “Ocean which in a landlocked state we found was trickier than we anticipated.” Participants 

described difficulty finding activities related to the ocean, “I think we should not choose the theme until we see 

what’s going on around Denver for field trips... Just with doing oceans and then trying to find [something] 

applicable [was difficult].” For this reason, coordinators suggested more thoughtful planning of the theme and 

the corresponding activities.

Participant Recommendations 

A wide array of recommendations for program improvement were mentioned by participants. In addition to the 

suggestions and recommendations already discussed in the sections above the following recommendations 

were the most commonly discussed:

•

•

•

Improving student engagement and attendance through incorporating more non-fiction, increasing 

activities (e.g. special presentations and field trips), and incentives.

Increasing monetary support for teachers.

Conducting an early coaches meeting to discuss program structures and expectations.

Student Engagement. Participants, especially teachers, desired flexibility in the way they foster student 

engagement. For instance, many suggested incorporating more non-fiction during instruction, “All the 

assessments were [supposed to] be fiction and the writing was [supposed to] be fiction. We felt like maybe the 

kids would’ve benefited from more non-fiction.” Participants expressed that students had an interest in science 

and social studies and found that instruction would have been more enjoyable if non-fiction were also utilized in 

the classroom. Also, many participants felt that activities such as the field trips were an incentive for students to 

attend, “We ended up having both of our field trips on the same week. And our attendance plummeted 

afterwards.” For this reason participants suggested increasing local field trips (e.g. parks, public library), parties 

for students with perfect attendance, and pervasive advertising of attendance incentives.

Teacher Compensation. Principals and teachers also recommended increasing monetary support for teachers.

Principals reasoned that, “It’s hard for teachers, because that’s their summer and that’s why they need extra 

incentive. To, ya know, really motivate themselves.” Meanwhile, teachers explained that their workload

required more than the 4-hour time frame of the school day. Teachers found that lesson planning, administering 

assessments, professional development, and meetings were a larger time commitment than initially anticipated. 

For this reason, the salary did not feel commensurate with their workload.

Communication Structures. Finally, coordinators recommended improving the program by creating 

communication structures that encouraged dialogue between coaches and coordinators. Coordinators 

expressed that, “Something else we learned this year was, meet with our coaches early... They need to be a part 

of the planning process prior to that first day and we lost a coach as a result of that. They felt like they didn’t 

know what was going on and didn’t want to be a part of it.” Coordinators felt that program expectations, 

structures, and activities were all information that coaches needed prior to the teachers’ professional learning. 
29

Coordinators felt that this early meeting would help alleviate anxiety related to missing information and better 

prepare coaches to address the needs of their school sites. 

Fidelity 

During focus group discussion similarities and differences of how each school site implemented the JSEL 

program emerged. Practices related to literacy block and CALI implementation such as, mini lessons, whole 

group, shared, guided, and independent reading were reported as being, “used on a regular basis.” However, 

differences were reported in the level of classroom readiness, writing instruction, administration of 

interventions, walkthrough practices, mealtime practices, and community involvement elements. Participants 

cite a number of different reasons for these including familiarity with school site, familiarity with JSEL structures, 

resource availability (as discussed previously), school culture and practice, and logistical complications related to 

funding sources (as discussed previously). Figure 20 illustrates the differences and commonalities between 

school sites.

Figure 21. Participant comments pertaining to program implementation by school

Classroom Readiness. Inconsistencies with ready classrooms were reported in both new and existing JSEL school 

sites. The two primary barriers to classroom readiness were lack of resources and time for preparation, as 

discussed in the previous section. For instance, principals and teachers reported that teachers who were not in 
30

their original classroom had difficulty finding resources as a result of site resources being stored for summer. For 

this reason, teachers were not able to easily find supplies for creating a ready classroom. Meanwhile, the threeday professional learning opportunity was cited as occupying time teachers needed for planning and classroom 

set-up. These two factors were seen as the main reasons for difficulty with readiness and contributed to the 

inconsistencies related to readiness.

Writing. Additionally, difficulties with writing were reported across participant groups. For instances, participant 

commented, “I felt like writing... It’s just out there. I just felt like I need more. I wanted some consistent 

structures and practices for writing that really were not there.” Challenges related to multi-age and multi-grade 

classrooms were another reported contributor to inconsistency with use of CAP. Teachers reported a wide range 

of ages and skill levels and expressed that it posed difficulty when choosing rubrics, “One of the messages they 

kept sending us was to use the first unit for instruction in CAP whatever grade you were going into. Not taking 

into account that many of us were doing multi-grade levels... It didn’t flow nicely... So we ended up making a 

decision here to move a few units ahead.” Participants suggested having more in-depth professional 

development on writing and choosing rubrics for multi-age and multi-grade classrooms.

Interventions. The manner in which interventions were administered differed between Pennington and other 

school sites. Although all school sites said that they used assessments (running records and DIBELS progress 

monitoring) in the decision making process for assigning interventions, the scheduling of interventions differed. 

It was reported that at Pennington interventions were administered using a flood model in which students have 

a 40-minute block every day in which interventions were administered. One participant cited the reason for this 

practice was, “It was an extension of what we were doing in our regular school year and we didn’t want to stop 

that because it’s been very effective.” At other sites the scheduling of interventions were at the discretion of the 

teacher and could take place at any time during the 3-hour school day.

Walkthroughs & Feedback. Participant discussion concerning walkthroughs demonstrated differing practices and 

opinions related to the delivery of teacher feedback. At some school sites, individual feedback was given to 

teachers and at other sites, only group feedback was given teachers. Additionally, the preferred manner of 

feedback delivery was mixed. At some school sites, participants preferred receiving individualized feedback and 

did not receive it, “’Cause I feel like if you’re coming in my room every day. If you’re not [going to] give me 

feedback why are you there.” Meanwhile, participants that received individualized feedback demonstrated 

mixed opinions with some describing it as, “negative,” “evaluative,” and “stressful,” while others found it helpful 

and expressed a desire for more “growth producing” feedback. The timing of walkthroughs was also seen as 

needing improvement by some participants reporting that walkthroughs came too soon after professional 

development and did not allow them enough time to process the information before implementation. 

Participants suggested clarifying the purpose and process of walkthroughs during the initial professional 

development and disseminating feedback in a consistent manner (e.g. group or individualized feedback).

Community Meals. The implementation of community meals and perceptions about the nature of the program 

were elements with distinct differences between new school sites and existing sites. In contrast to existing sites 

where families were welcome to eat together, new sites hindered parents from eating with their children. 

Participants report that parents could neither buy a meal nor eat a meal at no-cost with their children,

“[Cafeteria staff] put their foot down. No parents can eat. Only kids.” Participants also report that cafeteria staff 
31

were “very rude” when enforcing this policy. Confusion surrounding payment source was seen as the primary 

reason for this practice, “It just wasn’t clear as to who was paying for the lunches. So then the lunch people 

didn’t give adults lunches.” Participants suggested having the lunch staff from the regular school year to work 

during the summer, as well as improved communication to help alleviate difficulties related to meal times.

Program Rigor. The second element that distinguished new from existing sites was the perceived nature of the 

program. New school sites report having the impression that the program was akin to “camp” with a less rigid in 

structure compared to the regular school year. One participant describes this when she says, “We did sell it as 

summer camp... That it would be fun... The struggle is that it wasn’t different than what the regular school year 

looked like.” Participants report that this message of a less rigid structure was relayed to both teachers and 

students and as a result, the rigor of the program was unexpected at new school sites. Meanwhile existing sites 

say that the program was advertised as, “Summer school... It was never described as camp to anyone.” 

Participants attributed this perception as due to miscommunication and commented that improved 

communication during the student and teacher recruiting process could help alleviate the confusion.

Unique School Approaches. New school sites also described elements that were unique to their school. For 

Pennington, a community read-a-loud, in which all students at the site read the same book, was a structure 

unique to the school. Participants reported that the read-a-loud at Pennington was an added element that had a 

positive effect, “We were very hesitant about [the read-a-loud] for the younger kids. And I would have to say 

that was the best part I think of summer school.” Participants describe this added structure as contributing to 

the sense of community at Pennington. When describing one student’s reaction a participant says, “Everyday 

she wanted to talk about [it] and the kids. And I think that's where we kept the theme alive.” Overall, 

participants responded positively to this structure. Meanwhile for Stevens, added lesson review meetings were

a structure exclusive to the site. Teachers were scheduled to meet with administration and instructional coaches 

to review and revise lesson plans. Some participants commented that this practice was for the purpose of 

helping teachers plan lessons, acclimate to the program structures, and provide extra support for newer 

teachers. However, other participants viewed this practice negatively. In particular, participants commented 

that this structure was not always necessary and gave teachers less autonomy in designing lessons. 

Parent Feedback

To gather information about parent participation, weekly surveys were sent home with Friday Folders for 

parents to fill out and return the following Monday (Appendix E). In addition to the weekly surveys, an end-ofthe year survey gathered feedback about the summer school program in general to provide insight into parents 

and their children’s experiences in the program (Appendix F). The following section discusses the findings from 

both the weekly and end-of-year survey.

Weekly surveys 

Weekly surveys asked parents the following four questions: (1) “Did you read with your child this summer?”, (2) 

“Did you do a homework activity with your child?”, (3) “Did you attend program breakfasts/lunches?”, and (4) 

“Did you serve as chaperone on field trips?”. Parents were asked to mark each option in which they participated 

that week. Pennington Elementary was the only school site with alternative wording for the weekly survey. This 

revision was a school based decision. In addition to the alternative wording, Pennington added an “other” 

response option in which participants were able to write down an activity. As a result of the added response 
32

option the school site was omitted from the analysis of the weekly surveys shown in figure 22. Records with 

ambiguous and missing data were also excluded in this analysis. Over the entire summer session, only seven 

parents submitted all weekly surveys for every week of the program’s operation. The analysis shows the average 

number of activities participated in per week. As a result of the discrepancies related to missing identifiers and 

incomplete or ambiguous records, averages were calculated based on the number of complete responses 

received that week. 

Although the average number of activities reported by parents increased from 3 in the first week to 3.22 in the 

sixth week, the number of parents submitting weekly surveys also reduces from 116 respondents in the first 

week to 23 in the sixth. The week with the highest number of responses (121), week 2, had the lowest average

of reported participation (2.97). Whereas, week six had the lowest number of respondents (23) and the highest 

average of activities in which parents participated (3.22). Figure 22 shows the average number of activities 

parents reported participating in by week. Nonetheless, the number of home learning activities reported by 

parents had an upward trend throughout the program. 

Figure 22. Average number of activities in which parents participated in by week

Program Week

Mean

(n)

Week 1

3.00

(116)

Week 2

2.97

(121)

Week 3

3.04

(92)

Week 4

3.04

(75)

Week 5

3.23

(47)

Week 6

3.22

(23)

 

End-of-Year Surveys. 

The end-of-the-year survey gathered information in five main categories: (1) increase in student’s academic 

performance, (2) parent communication and connection to the program, (3) parent-student needs, (4) clarity of 

program expectations, and (5) participation in program activities (See Appendix E). One-hundred seventy-one

parents responded to the survey. Of the parents that responded to the survey, approximately 59.6% indicated 

that it was their child’s first year in the program, approximately 14.6% indicated it was their 2nd year in the 

program, and approximately 13.5% indicated it was their child’s 3

rd year in the program (7.6% of respondents 
33

did not respond to this question and 2.9% entered an ambiguous response). These responses correspond to data 

from APA’s impact study, which found that the majority of students attended the program for one year, with 

fewer students attending for two or three years. The majority of respondents (approximately 70%) indicated

that they have “lived in this community” for more than 4 years.

In the first category, increase in student’s academic performance, parents were asked about perceived increase 

of student’s skills in reading and writing. A majority of parents (approximately 92%) report feeling that their 

child’s reading skills had improved as a result of the program. Meanwhile, 85% of parent respondents report 

their child’s writing skills improving as a result of the program. Less than 1% of parents felt that their child’s 

reading skills had not improved and approximately 1% felt that their child’s writing skills had not improved.

Overall, responses demonstrate that parents felt participation in the program led to increased academic skills 

and reading behavior at home.

In the second category, parent communication and connection, parents were asked about the communication 

they received from staff and the environment of the school site. The majority of parents (approximately 96%) 

responded that they found the school environment welcoming. A smaller percentage (approximately 77%) of 

parents indicated that they felt like they were “contacted by program staff often.” Overall, responses 

demonstrate that the majority of parents felt that the school environment was welcoming and that the staff 

communicated with them often.

In the third category, student-parent needs, parents were asked if students were able to receive needed 

supplies, if the summer school schedule met their needs as caregivers, and if the summer school instructors 

knew student’s educational needs. The majority of parents (approximately 94%) indicated that they agreed that 

their child received necessary school supplies. Although slightly lower, the majority of parents (approximately 

89%) agreed that the program schedule met their needs as caregivers. Finally, approximately 93% of 

respondents indicated that program staff understood the educational needs of their children. 

In the fourth category, clarity of program expectations, the majority of parents (approximately 92%) indicated 

that they agreed that they knew the expectations of the program.

In the final category, parent participation, respondents were asked questions about the activities they 

participated in during the program. Similar to the weekly survey, parents were asked about participation in athome reading, homework activities, breakfast/lunch, and field trips. Perhaps not surprisingly, serving as a 

chaperone on field trips had the largest percentage of parents (48.24%) responding that they “never” 

participated. Meanwhile, breakfast and lunch was the activity that had the largest percent of parents responding 

that they “always” participated. Figure 23 illustrates parent’s responses to questions about participation in 

program activities.
34

Figure 23. Parent participation questions in the end-of-year-survey

Parents were asked two open-ended questions. The first asked respondents about program elements they liked 

and the second asked for suggestions for changes in the future. The academic opportunities, special activities, 

and meals were the three most common components which parents said they liked about the program.

Academic opportunities were the most popular program element that parents said they liked. This category 

included responses pertaining to academic growth, participation in academic work, and references to ‘summer 

slip’. Parent responses demonstrate an affinity for the reading, writing, and oral language components of the 

program. Responses include: “It’s great for extra learning.” And, “My child was on track with the reading that 

she had to do.” Meanwhile, special activities, which includes field trips and special school events (e.g. class 

parties, guest lectures), was the second most cited program element mentioned in responses. Activities were 

described by parents as, “great” and “fun.” Finally, mealtimes was the third most cited program element that 

parents liked. Participants comment on being able to eat with their kids. Some describe meals as, “excellent” 

and express appreciation for opportunity to have a meal at the school.

Aside from many responses saying they would not change anything about the program, parents mentioned 

adding leisurely activities at school, lengthening the schedule, and added field trips as the most cited changes. 

Parents express that they want to increase the amount of “fun” activities such as, sports, playtime, and other 

0.59%

2.37%

12.43%

48.24%

15.98%

18.34%

21.89%

13.53%

41.42%

43.79%

24.85%

8.82%

39.05%

31.95% 31.95%

25.29%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

How often did you

read with your child

this summer?

How often did you

do a homework

activity with your

child?

How often did you

attend program

breakfasts/lunches?

How often did you

serve as chaperone

on field trips.

%Never

%Rarely

%Sometimes

%Often

%Always
35

non-academic activities. Many respondents also suggest longer days and a longer session. In these instances 

respondents suggest having the school-day resemble the full-day schedule of the regular school year, longer 

field trip days, or a longer summer session. Increasing field trips was the third most referenced change 

mentioned by parents. Responses demonstrate that parents found field trips “fun” and would like the 

opportunity to have more in the future.

Recommendations 

Overall, responses from focus groups, interviews, and surveys demonstrate coordinators, administrators, 

teachers, and parents communicate positive program experiences. However, feedback also shows that 

improvements to professional learning, communications structures, events and activities, teacher planning, and 

access to resources are desired.

Improved Communication 

In light of the challenges in communication, (see pg. 16-17) it is recommended that communication structures 

be improved through:

•

•

•

•

Creating a more comprehensive program ‘handbook’

Bolstering communication structure around program updates

Creating structures for dialogue between coordinators, coaches, and principals

Clarifying communication regarding financial logistics for new schools

Findings demonstrate that administrators and participating teachers need information about time commitment, 

schedules for professional learning, meetings, focus groups, and program expectations. Other information 

needed by administrators is around payment/invoices, acquiring substitutes, and logistical aspects of the 

program. All this information should be available to principals and teachers inside the program handbook in 

order that it may be easily referenced. It is also suggested that this information be given to teachers and 

principals during the recruitment process and prior to the start of the summer session.

Another avenue where communication could be improved is around program updates and information. 

Participants expressed that information about weekly changes or upcoming events could be better 

disseminated. To diminish confusion about administrator leave, coordinator visits, field trip details, and schedule 

changes it is suggested that structures be created for relaying information around program changes and site 

updates. 

Another suggestion to improve communication is to create structures that encourage a dialogue prior to the 

start of the summer session between coordinators, coaches, and administrators in order to crystalize 

expectations, roles, and procedures of the program. This would create an avenue for staff to ask questions, as 

well as clarify roles, procedures, and expectations before the start of the summer session.

Lastly, focus group discussions demonstrate that challenges with breakfast and lunches emerged as a result of 

confusion around the financial logistics of the cafeteria’s operation. Communication with lunchroom staff and 

management about payment and procedures will aid in consistent lunch and breakfast structures. 
36

Adjust Professional Learning 

Many teachers, coaches, and principals expressed a desire for increased differentiation and more autonomy in 

choosing professional learning. Although discussion of program expectations was perceived as a good reminder, 

participants want the opportunity to choose from a variety of options. For the on-going PL, teachers expressed a 

desire to have more input in the topics discussed and suggested that all on-going sessions be site-based while 

giving teachers the autonomy to choose which off-site PL to attend. Furthermore, in order to gain more depth of 

knowledge about the topics discussed it is suggested that fewer on-going PLs be scheduled, but having those PLs 

be longer in length in order to more deeply explore topics.

Increase Teacher-driven Planning time 

Participant responses demonstrated that teacher-driven planning time was diminished due to additional teacher 

responsibilities (e.g. lunch duty, recess duty), as well as the use of planning time for program updates. For this 

reason, it is suggested that teachers be given more time to plan by either increasing the number of hours in 

teacher schedules or creating structures for teacher-driven planning time while decreasing coach-led planning 

time.

Provide IT Support 

Challenges related to finding and accessing instructional resources could be improved by providing IT support 

for school sites and instructional resources. IT support would be able to aid teachers and coaches with accessing 

and updating software, allocating devices, and troubleshooting. Teachers expressed a need for being able to

access necessary supplies for creating a ready classroom as well.

Create a Consistent Student Behavior System 

Teachers expressed challenges related to student discipline and the lack of clear procedures for addressing 

issues related to unacceptable behavior and accountability for student homework. Participants expressed a 

desire for a site-wide or program-wide system for addressing these challenges and suggest that such a system 

should include incentives for students to turn in homework.

Consider Planning of Field Trips 

Participant responses demonstrate that transportation, location, and school schedule resulted in diminished 

time for students and their families to enjoy field trips. A number of suggestions can be considered to improve 

field trips including allowing schools more autonomy to choose local destinations, more autonomy for families 

to choose transportation, and lengthening the school day to allow families more time. 

Another matter for consideration is increasing the number of field trips. Not only does survey feedback 

demonstrate that parents want more opportunities for their children to participate in leisurely activities, but 

focus group discussion shows that participants observed a decrease in attendance after field trips and view it as 

an incentive for student attendance. Thus, increasing the number of field trips and spreading them throughout 

the summer session could encourage consistent student attendance.
37

Impact Analysis 

The first impact evaluation of the Jeffco Summer of Early Literacy (JSEL) program was an evaluation of the 2012 

program year by the Buechner Institute for Governance at the University of Colorado Denver’s School of Public 

Affairs (Buechner), with support from the Jeffco Schools Foundation and MHUW. Buechner conducted an impact 

evaluation for JSEL participants in the first and second program years. Due to the departure of key personnel 

from Buechner, the evaluation of JSEL for the third program year was transferred to APA Consulting (APA) in 

2014. 

JSEL was funded as part of the five-year Social Innovation Fund (SIF) grant from the Corporation for National and 

Community Service (CNCS) to MHUW. However, MHUW graduated JSEL from their SIF portfolio after year three. 

Because of this change, the originally envisioned five-year evaluation plan has been truncated to three years and 

the original Sub-grantee Evaluation Plan required by CNCS has been modified. The most important modification 

is a reduction in the number of JSEL students and comparison students included in the analysis This reduces the 

statistical power of the analysis and decreases the likelihood that it will detect a statistically significant effect of 

the JSEL program. Aside from the truncated timeline, this report follows the original analysis plan.

The analysis plan for this evaluation compares students who participated in JSEL to similar students from the 

Jefferson County School District (Jeffco) who did not participate in JSEL. The group of comparison students is 

constructed using a statistical technique called propensity score matching (PSM), which uses the demographic 

characteristics and pre-test scores of JSEL participants to match them to non-JSEL participants with similar 

demographics and pre-test scores. Comparing these groups allows APA researchers to estimate the effect of the 

JSEL group by comparing the differences in outcomes between JSEL participants and similar non-JSEL 

participants. By using pre-test scores, the analysis examines the change in test scores over the summer 

(between spring and fall).

APA examined two primary outcomes: scores on the DIBELS Next reading assessment that were collected both 

in the spring and fall prior to and following JSEL participation and third-grade scores on the reading components 

of the Transitional Colorado Assessment Program (TCAP) standardized assessment.2 APA used a multi-level 

analysis technique called hierarchical linear modeling (HLM) to control for the school-level characteristics of the 

different schools attended by students in the treatment and comparison group. 

APA’s impact analysis does not address the fidelity of JSEL implementation or many of the themes raised by 

teachers, instructional coaches, coordinators, and principals in the implementation evaluation conducted by 

Jeffco public schools that is incorporated in prior sections of this report. Neither does the impact evaluation 

consider the full range of student skill increases, reflected by the 92% of parents who reported seeing an 

increase in their child’s reading and academic skills. This impact evaluation is focused only on student increases 

in test scores on the DIBELS Next and TCAP assessments.

This impact analysis was designed to achieve a moderate level of evidence of JSEL’s impact. While an 

experimental design using random assignment was not possible for this intervention, which would be necessary 

 

2 The technical manuals with construction and validation information for the TCAP is available at 

https://www.cde.state.co.us/assessment/2014tcaptechrep. The technical manual for the DIBELS Next, with similar 

information, is available at https://dibels.org/dibelsnext.html.
38

to achieve the highest level of evidence. To provide a moderate level of evidence, quasi-experimental designs 

such as the PSM approach used here, are appropriate. The PSM approach reduces threats to internal validity by 

comparing treatment students to comparison students who are similar on a range of observed characteristics. 

Impact Research Questions 

The impact analysis is intended to explore whether JSEL has a positive effect on student learning. All three years 

of evaluation activities have focused on the same research questions, reviewed in Table 1.

Table 1: Confirmatory and Exploratory Research Questions

Question 

Number Research Question Question Type

1

Do JSEL participants make greater progress in their reading 

proficiency than similar non-participants? Confirmatory

2

Does the magnitude of the difference between JSEL participants and 

non-participants vary as a function of the number of years students 

participated? Confirmatory

3

Are JSEL participants more likely to score “proficient” on third grade 

reading TCAP than non-participants? Confirmatory

4

Does the proportion of participants scoring “proficient” increase with 

the number of years enrolled in JSEL? Confirmatory

5

Do students who regularly attend JSEL show greater growth in reading 

skills than students who attend less regularly? Exploratory

Research questions focused on implementation and fidelity are addressed by the implementation study 

conducted by Jeffco Public Schools, which is incorporated into this report.

Overview of Analysis 

Research questions 1 through 3 compare JSEL students to similar students identified using PSM techniques. The 

same group of comparison students was used for each analysis. APA identified the comparison group using a 

propensity score match that employed demographic variables and the students’ DIBELS Next pre-test score from 

the spring immediately prior to the JSEL program. Each comparison group student is uniquely matched to a JSEL 

participant student with similar demographic characteristics and pre-test scores. Because the resulting 

treatment and comparison groups are similar, any differences in outcomes between the two groups are likely to 

be attributable to JSEL participation, rather than to other differences between the groups of students. APA used 

the same PSM technique for year three JSEL data as had been used in the previous two years’ impact analyses. 

After generating the comparison group, APA ran hierarchical linear modeling (HLM) regressions comparing 

outcomes between comparison group students and JSEL participant students. HLM is used when the outcome is 

measured at the individual level, but the treatment (ie. JSEL) is provided to groups, such as at schools. For the 

research questions 1, 2, and 3, APA looked at differences in scores on outcome measures, controlling for 

background variables, pre-test scores, and school characteristics. These controls are intended to help separate 

out JSEL’s effect from students’ reading ability prior to JSEL (using the pre-test scores) or known demographic 
39

descriptors that are known to be associated with differences in reading performance (e.g. gender). For questions 

1 and 2, APA used the students’ DIBELS Next scores for the fall following the summer of JSEL participation (or 

non-participation for comparison group students) as the outcome measure. For question 3, APA used students’ 

scores from the reading portion of the third-grade statewide assessment, in this case the Transitional Colorado 

Assessment Program (TCAP), administered in the spring of the school year following JSEL participation (or nonparticipation). For all three of these research questions, the variable indicating whether a student participated in 

JSEL was the variable of interest. In all of the models, students’ spring DIBELS Next results were used as the pretest score.

Questions 4 and 5 compare JSEL students to other JSEL students based on the amount of time they have spent in 

JSEL. Question 4 examines the effect of multiple years of JSEL participation, while question 5 compares students 

based on the number of sessions attended during a summer of JSEL participation. Questions 4 and 5 apply only 

to students who attended JSEL. Because of the reduced number of students in the JSEL-only models, these 

models have reduced statistical power and are less likely to find statistically significant effects.

All of the analysis for questions 1, 2, and 5 contain pooled data from students in years 1, 2, and 3 of the analysis. 

Aggregating students from multiple years of treatment and comparison groups increases the total number of 

students in the analysis and is more likely to yield statistically significant results. The models addressing 

questions 3 and 4, using TCAP scores as the outcome variable include only students who participated years 1 

and 2 of JSEL, since TCAP scores for students who participated in year 3 of the program are not yet available. 

TCAP scores were not available to the Buechner researchers when they conducted their analysis, so this report 

represents the first time that questions 3 and 4 have been analyzed examining the impact of JSEL using TCAP 

scores as an outcome variable.

APA obtained administrative data on the demographics and assessment results of JSEL participants and matched 

students from the Jeffco Director of Assessment and Research pursuant to an MOU between APA and that 

office. APA staff physically visited the Jeffco office to obtain the file of JSEL participants with demographic and 

assessment data. While at Jeffco, APA performed the PSM match (described further below) to obtain a file of 

matched comparison group students. This matching technique meant that data were largely clean when 

obtained, because students with missing demographic or assessment data were not eligible to be matched. The 

de-identified student data were stored on a flash drive and not stored on any server or hard drive to facilitate 

complete destruction and erasure of the student data at the end of the analysis. 

Sampling Eligibility 

For all five research questions, students were included in the analysis sample only if they took the DIBELS Next 

exam in the spring prior to their participation (or non-participation) in JSEL and also in the fall immediately 

following their participation (or non-participation) in JSEL. For questions 3 and 4, students were included in the 

analysis only if they also took the third-grade TCAP reading exam in the year following their summer of JSEL 

participation (or non-participation). Because of these restrictions, the sample size for these analyses of 

questions 3 and 4 consisted of a small subset of the total JSEL participants, limiting the likelihood that the 

models will produce statistically significant results.
40

Although Buechner excluded students who had any JSEL exposure prior to entering Kindergarten from their 

analysis, both for the students’ first year of JSEL and all subsequent years, APA included later years of 

participation from these students in order to better identify the impact of JSEL. Both Buechner and APA 

excluded students in grade levels higher than third grade from all analyses, since the focus of the study is 

improving reading skills for students in their first few years of elementary school. In this final analysis year, no 

students were excluded because of having DIBELS pre-scores instead of DIBELS Next pre-scores and no students 

were excluded for having enrolled in the program prior to their kindergarten year. Students were excluded for 

lacking a DIBELS pre-test (28 students lacked a pre-test). No students were missing a DIBELS post-test, so none 

were excluded for that reason.

Constructing Comparison Groups

APA used propensity score matching (PSM) techniques to create comparison groups of non-participating 

students for each year of the analysis.3In PSM, a propensity score is generated for each student – both JSEL 

participants and non-participants – describing the student’s probability of attending JSEL based on the similarity 

of her demographic characteristics and DIBELS Next pre-test score with those of a student in JSEL. In order to 

identify the comparison group using PSM, APA gathered demographic and test score information for all students 

in Jeffco who were in Kindergarten through second grade prior to their participation (or non-participation) in 

JSEL. There was no contamination between treatment and comparison groups, so JSEL students were 

completely distinct from non-JSEL students. APA then matched students based on their propensity scores using 

a logit to predict likelihood of attending JSEL and a 1-to-1 nearest-neighbor matching estimator using 

Mahlanobis distance computation, without replacement. In this method, each treatment student is matched 

with the comparison student with the most similar propensity score, based on their background characteristics. 

The matched comparison student is then removed from the pool of potential matches, so that every treatment 

student is matched with a unique comparison student. This leads to equal sized treatment and comparison 

groups. Note that this propensity score cannot account for student characteristics that are not captured by 

available data, including unobservable traits such as personal motivation or parental engagement. 

One of the primary variables used to match students in the PSM was students’ composite score on the DIBELS 

Next assessment in the spring before their summer of JSEL participation (or non-participation). Because the 

DIBELS Next composite score includes different subcomponents depending on the grade level of the student 

being tested, the composite scores for students in different grades are on different scales, and are not directly 

comparable. In order to ensure comparability of DIBELS Next scores across grade levels, APA converted the raw 

composite scores into z-scores, standardizing within grade level. Because of this standardization, a Kindergarten 

student who scored in the 90th percentile of her peers would have a similar z-score to a second-grade student 

 

3

In year 3 of the analysis, APA adopted matching techniques chosen by the previous evaluator and implemented in the Year 1 and Year 2 

analysis and reports, in order to ensure consistency of matching techniques across years. Additionally, the data restrictions that Jeffco 

uses to protect student privacy meant that APA could not test and compare various matching techniques. In order to preserve the 

security of the data, APA obtained outcome data for both comparison and treatment students by physically going to Jeffco offices and 

doing a single match while in the office. APA then received outcome data only for comparison students matched during that process. 

Because of this process, APA could not test and implement various matching techniques.
41

scoring in the 90th percentile of her peers, even though the students’ raw composite scores would likely be quite 

different. No other adjustments were made to the DIBELS Next scores.

Though APA obtained data from Buechner’s previous analyses, APA was unable to obtain a list of the exact 

Jeffco students who were included in the comparison group in the first two evaluation reports. Instead, APA 

conducted its PSM using the same technique and variables as Buechner, resulting in new comparison groups for 

each of the first two years of JSEL data. In keeping with the PSM performed by Buechner in previous years, APA 

performed its PSM using each student’s standardized DIBELS Next pre-score, an indicator variable indicating 

whether the student was white or non-white, indicator variables indicating whether the student was eligible for 

free or reduced lunch, and the student’s grade level. Although the comparison groups for APA’s year 1 and 2 

analysis do not contain exactly the same students as the comparison group used by Buechner, APA’s models 

produce nearly identical results to those found in Buechner’s report for years 1 and 2. This suggests that the 

comparison groups produced by APA’s analysis are extremely similar to the groups used in the analysis from 

previous years. 

The PSM analysis performed by Buechner matched students only on student-level characteristics and did not 

include any school-level characteristics. Although APA was able to conduct a rematch of students from the first 

two years of the impact study, school-level characteristics were not available for those students. This means that 

none of the PSM matches for any of the years of the impact analysis contained school-level variables.

After performing a separate PSM for each year of JSEL implementation, APA pooled the students from all three 

years of the JSEL evaluation. Table 2 reviews the characteristics of the resulting sample. The comparison sample 

was demographically similar to the JSEL students, however they generally did not attend the same schools as the 

JSEL students. 

Table 2: Characteristics of the Pooled Sample

JSEL Participants Comparison Group

Kindergarten 160 160

First Grade 150 150

Second Grade 142 142

Edgewater 32.1% 2.4%

Lasley 19.4% 5.8%

Molholm 19.8% 1.7%

Pennington 7.2% 0.5%

Stevens 4.9% 1.6%

Swanson 16.0% 0.6%

Non-JSEL Schools 0.6% 87.3%
42

To test the quality of the PSM, APA used the standardized mean difference of each demographic variable for the 

treatment and comparison groups, which is a common tool for measuring differences between groups.4 When 

reviewing the quality of a match, the absolute standardized mean difference should be no larger than 0.25 and 

preferably less than 0.1. Table 3, below, details the characteristics of the pooled treatment and pooled 

comparison groups generated by the PSM. The two groups are very similar in terms of raw demographics and 

the standardized mean differences between the groups fall below the recommended thresholds.

Table 3: JSEL Comparison and Participation Groups are Similar

JSEL Participants Comparison Group

Absolute 

Standardized 

Mean Difference

Free and reduced lunch eligibility 87.0% 89.6% 0.079

Special Education 10.6% 13.7% 0.096

Gifted and Talented 2.0% 2.0% 0

English Language Learner 40.0% 30.8% 0.192

Male 50% 49.3% 0.013

Standardized Spring DIBELS Next Score -0.01 0 0.01

Although students were well-matched on student-level variables, the exclusion of school-level variables from 

the match procedure means that the two groups were not well matched on those school-level variables. For 

example, the average JSEL student attended a school where 82% of students received free or reduced school 

lunch, while comparison students attended schools where 33% of students were eligible for free or reduced 

lunch. The schools attended by comparison students also had slightly higher average ratings in overall academic 

achievement, reading, and writing. While the inclusion of these school-level variables in the analysis models, 

discussed below, controls for these differences, there are meaningful differences between the treatment and 

comparison groups on these school-level variables. 

There was no differential attrition in the comparison group. This is because the universe of students eligible for 

matching to become part of the comparison groups was limited to those with a pre- and post-test score. All 

comparison students were included in the analysis. As described above, only 28 of 452 treatment students, or 

6%, were excluded from the analysis due to a missing pre- or post-test. 

 

4 This statistic is calculated by determining the raw difference between the treatment and control groups on a certain 

variable, then dividing the result by the standard deviation of the pooled group on that same variable. For example, a 

standardized mean difference of 0.05 would indicate that JSEL students’ mean for that variable is 0.05 standard deviations 

higher than the mean of the matched comparison students.
43

Variable Full Treatment Group Analysis Treatment Group

Kindergarten 35.4% 34.9%

1

st Grade 33.2% 34.2%

2

nd Grade 31.4% 30.9%

Non-white 75.2% 75.2%

FRL 87.0% 87.3%

Male 50.0% 49.5%

ELL 40.0% 40.3%

Statistical Analysis 

The statistical analysis for all five research questions included a set of control variables at the student level and 

another set of control variables at the school level. These controls are intended to isolate the effect of JSEL on 

student outcomes from the effects of other variables, including demographics and past student performance. 

The student-level control variables are: 

•

•

•

•

•

•

•

Race and ethnicity, described with an indicator variable indicating whether the student is white; 

Socioeconomic status, measured by eligibility for free or reduced lunch;

Participation in school programs: gifted and talented, English language learner, and special education;

Grade level;

Gender; 

Standardized DIBELS Next spring pre-test scores

Year of JSEL participation (or non-participation).5

The analysis for all of the research questions used HLM models that control for school-level effects on student 

summer learning. All of the HLM models included the following school-level variables:

•

•

•

•

•

•

•

•

Overall state-level accountability rating;

Overall achievement ratings;

Overall growth ratings;

Overall progress toward closing the achievement gap;

Reading achievement;

Writing achievement;

Percentage of students eligible for free or reduced lunch;

Stability rates; and

 

5

Including this variable allowed APA to control for any possible differences in DIBELS Next scaling and implementation from 

one year to the next.
44

• Mobility rates.

The following sections include the coefficients on the JSEL participation variables and the statistically significant 

coefficients on key variables of interest. These coefficients describe the relationship between the variable and 

the outcome assessment scores. Appendix G provides more details of the results in each of the five models, 

including the coefficients associated with all variables. Effect sizes are also reported for statistically significant 

coefficients on key variables of interest.

Question 1: Do JSEL participants make greater progress in their reading proficiency than similar 

non-participants? 

For this research question the outcome being examined was a student’s standardized DIBELS Next composite 

score from the fall following the summer of JSEL participation (or non-participation). 

All statistically significant coefficients for model 1 are reported in Table 4, below. The coefficient of 0.10 for the 

JSEL participation variable indicates that students who participated in JSEL scored, on average, 0.10 standard 

deviations higher on the fall DIBELS Next assessment than comparable students who did not attend JSEL, 

controlling for other demographic characteristics and students’ DIBELS Next pre-test scores. This coefficient is 

statistically significant, indicated by the p-value of less than .05. This is equivalent to an effect size of 0.10.

Statistical significance means that APA is confident that the mean difference between DIBELS Next scores for 

JSEL participants and non-participants is not due to random chance, but is a real difference between students in 

the two groups. This difference is equivalent to moving from the 50th to the 54th percentile on the DIBELS Next 

assessment. As expected, the coefficient on students’ DIBELS Next pre-test score was also a statistically 

significant predictor of fall assessment scores, but none of the coefficients on the other control variables were 

statistically significant. 

Table 4: Participation in JSEL is Associated with Significantly Higher DIBELS Next Scores

Coefficient P-value

JSEL participation *0.10 0.009

Standardized spring DIBELS Next score *0.85 < 0.001

* indicates statistical significance

The proportion of variance attributable to school-level variables was only 0.6 percent, This suggests that there is 

very little relationship between school-level characteristics and summer learning. This lack of variance 

attributable to the school-level variables suggests that matching JSEL students with students in non-participating 

Jeffco schools was a reasonable way to match JSEL participants with comparable students who did not 

participate in JSEL. 
45

Question 2: Does the magnitude of the difference between JSEL participants and non-participants 

vary as a function of the number of years students participated? 

Table 5: Participation in JSEL in Multiple Years

Overall Finding for Research Question 1: APA finds that participation in only one year of 

JSEL provides a statistically significant and positive effect on student reading 

performance. This effect is equivalent to moving students from the 50th to the 54th 

percentile on the DIBELS Next literacy assessment. This is the equivalent of an effect size 

of 0.10.

To address this research question, APA used a model similar to that for question 1, detailed above. Instead of 

including an indicator variable indicating whether the student had participated in JSEL, APA used three indicator 

variables indicating whether a student had participated in JSEL for a total of one, two, or three years. All other 

individual- and school-level variables were the same as in question 1. In this analysis, the sample size is slightly 

smaller than in question 1, since each row of data represents a unique student, and students therefore do not 

appear in this model multiple times if they participated in JSEL in more than one year (as was done in the model 

for question 1). The results of the question 2 analysis are shown in Table 5, below.

Coefficient P-value

First year of JSEL *0.14 0.002

Second year of JSEL -0.07 0.261

Third year of JSEL 0.03 0.836

White *0.07 0.049

Standardized spring DIBELS Next score *0.86 < 0.001

* indicates statistical significance

The results of the model indicate that students who attend a single year of JSEL perform roughly 0.14 standard 

deviations higher on the fall DIBELS Next assessment than similar students who did not attend JSEL, controlling 

for the other demographic characteristics and the DIBELS Next pre-test scores. This difference is statistically 

significant, meaning that APA is confident that the result is associated with JSEL participation and not due to 

random chance. This result is equivalent to moving from the 50th to the 56th percentile on the DIBELS Next 

assessment. 

This is a slightly larger coefficient for JSEL participation than the Question 1 analysis: 0.14 compared to 0.10. 

However, given the imprecision of these estimates, the difference between these two coefficients is not 

statistically significant and APA cannot conclude that there is a meaningful difference between the effects of any

year of JSEL participation (addressed in question 1) and the effects of a student’s first year of JSEL participation 

(addressed in question 2). 
46

The coefficients for participating in JSEL for 2 or 3 years are not statistically significant and APA therefore cannot 

conclude that there is a marginal effect on reading scores from participating in an additional year of JSEL. The 

lack of statistical significance for the additional effects of multiple years of JSEL participation is likely influenced 

by the relatively small number of students who participated in multiple years of JSEL. Of the 760 students who 

attended JSEL at any point, 78 percent attended JSEL for only one year, while 20 percent (148 students), 

attended for two years and only 2 percent (17 students) attended for three years. The small numbers of 

students with multiple years of participation make it difficult to detect an effect of multiple years of 

participation. While the results suggest that the first year of JSEL participation may be the most important, these 

results should be interpreted with caution because of the limited sample size.

Overall Finding for Research Question 2: There are small numbers of students with 

multiple years of participation in JSEL in the current study sample., making it difficult to 

detect an added effect associated with multiple years of JSEL participation versus the 

positive effect yielded by a single year of participation as shown in Research Question 1. 

A larger sample of students with multiple years of participation is necessary to detect a 

statistically significant effect, if one exists.

Question 3: Are JSEL participants more likely to score “proficient” on third grade reading TCAP than 

non-participants? 

This research question considers results for only a small subset of students in the treatment and comparison 

groups. Because TCAP is administered only in the third grade, only students who finished second grade in the 

spring prior to their summer of JSEL participation (or non-participation) were eligible to take the TCAP. As a 

result, students who finished Kindergarten and first grade just before participating in JSEL are excluded from the 

analysis, as they do not have TCAP scores for the year following their JSEL participation. Additionally, the TCAP is 

administered in the spring of third grade and some students who participated in JSEL left the district prior to 

TCAP administration or otherwise lacked TCAP scores. The analysis for this question therefore includes only the 

129 JSEL participant students and 126 comparison group students who had third grade TCAP scores in the year 

following their summer of JSEL participation. The relatively low number of students included in this analysis 

reduces the statistical power, making it less likely that the analysis can detect a statistically significant effect of 

JSEL participation.

The principal model for this research question is also slightly different from the models for questions 1 and 2. 

While those questions considered a continuous outcome variable, this research question considers a binary 

outcome: whether a student is or is not proficient on the third grade reading TCAP. APA did not use the raw 

scale scores from TCAP but instead used the indicator of whether the student’s score was considered 

“proficient” or not. Because of the binary nature of the outcome, a linear regression model is not appropriate. 

Instead, APA used a two-level logistic regression model to predict the likelihood of an individual student being 

proficient on the third-grade TCAP, controlling for DIBELS Next pre-test scores and other individual- and school-
47

level variables. As a check of this change in models APA also ran a traditional (non-logistic) HLM model with 

TCAP scale scores as the continuous outcome variable and found similar results to those reported below.

In both models, APA did not find a statistically significant difference between JSEL participants and nonparticipants, either in their likelihood of being proficient on the third-grade TCAP reading assessment or in the 

mean TCAP reading scale score for the two groups. The lack of statistically significant results was likely 

influenced by the small number of students who were included in this model. A larger sample of students with 

JSEL participant and subsequent third grade standardized test scores would probably be necessary to detect a 

statistically significant effect of JSEL on TCAP results, if one exists. To obtain a sample equal to that used for 

question 1, APA would need data on approximately 700 additional students who participated in JSEL between 

2

nd and 3rd grade and who had subsequent 3rd grade TCAP scores. 

Overall Finding for Research Question 3: With limited years of data available, only 129 

JSEL participant students had third grade TCAP scores. With this small sample, APA did 

not find a statistically significant difference between JSEL participants and nonparticipants on TCAP performance. A larger sample of students, which was called for in 

APA’s evaluation plan, is necessary to detect a statistically significant effect of JSEL on 

TCAP results, if one exists.

Question 4: Does the proportion of participants scoring “proficient” on third-grade reading TCAP 

increase with the number of years enrolled in JSEL? 

The analysis for this question includes an even smaller subset of the students included in the analysis for 

question 3, which itself included only a subset of students from questions 1 and 2. The analysis compares 

students with TCAP scores to each other, rather than comparing JSEL participants with a group of nonparticipating students. As a result, this analysis includes a total of only 129 students, greatly reducing the 

statistical power of the model.

The model for this research question is a multilevel logistic regression, similar to the model used in question 3. 

The only difference is that the question 4 model includes three indicator variables indicating whether each 

student participated in JSEL for a total of one, two, or three years, rather than a single variable indicating 

whether a student participated in JSEL at all. All other individual- and school-level control variables are the same 

as in question 3.

The model for question 4 did not yield a statistically significant relationship between the number of years of JSEL 

attendance and the likelihood of scoring proficient on the third-grade TCAP reading exam. Similarly, APA’s 

analysis did not show a statistically significant association between scaled third-grade TCAP reading scores and 

the number of years of JSEL participation. However, because of the small number of students who took the 

third- grade Overall Finding for Research Question 4: The number of JSEL students with TCAP data is 

insufficient to find a statistically significant association between TCAP performance and 

the number of years of JSEL participation. Because of the small number of students who 

participated in JSEL in multiple and have available TCAP scores, these results should not 

be interpreted as meaningful.
48

TCAP and participated in JSEL in multiple years, these results should not be interpreted as meaningful.

Question 5: Do students who regularly attend JSEL show greater growth in reading skills than 

students who attend less regularly? 

The analysis for this question again uses the standardized DIBELS Next fall score as an outcome variable, as in 

the analyses for questions 1 and 2. The analysis for question 5 focuses only on students who participated in JSEL, 

comparing those students to each other, rather than a comparison group of non-participating students. In this 

model, the dummy variable indicating whether a student participated in JSEL has been replaced by a continuous 

variable indicating the number of JSEL days a student attended within a summer. The coefficient on this variable 

would indicate the unique effect of attending an additional day of JSEL, controlling for the individual- and 

school-level background characteristics.

In this model, the coefficient on days of JSEL attendance was not statistically significant, meaning that APA 

cannot be confident that differences in DIBELS Next fall scores are associated with variation in student 

attendance at JSEL. Again, this finding may be due to the limited sample size of students included in this analysis 

and should not be interpreted as meaningful. 

Overall Finding for Research Question 5: Data is insufficient to find a statistically 

significant association between days of JSEL attendance and differences in DIBELS Next 

fall scores. Because of the small number of students available in the study sample, these 

results should not be interpreted as meaningful.

Analysis Limitations 

This analysis is limited by the school-level differences between the treatment and comparison groups, where 

comparison students attended schools with lower rates of free or reduced lunch receipt and higher school 

performance framework ratings on overall academic achievement, reading, and writing. This means that, on 

average, comparison students attended better schools than students who attended JSEL, introducing a 

meaningful difference between the treatment and comparison groups. This problem is somewhat mitigated by 

the inclusion of school-level variables in the analysis models, to parse out differences associated with JSEL 

attendance from those associated with differences in the quality of school attended during the school year. 

Although these controls cannot completely eliminate the bias introduced by the lack of equivalence between 

the treatment and comparison groups, it is instructive to consider the direction of bias that is introduced. The 

comparison students, who attend somewhat better schools during the school year, could have been expected to 

have higher academic performance than their demographically-matched peers who attended lower performing 

schools. This means that the bias introduced by the non-equivalence is likely underestimating the difference 

between the performance of treatment and comparison students, underestimating the positive impact of 

attending JSEL. While this non-equivalence is a limitation of the analysis in terms of precisely estimating the 
49

exact magnitude of the positive effect of the JSEL program, it likely does not minimize or affect the finding that 

there is a positive significant impact derived from JSEL attendance.

Next Steps 

During the 2014-15 school year, JSEL was graduated from the MHUW SIF portfolio. This will be the final report to 

MHUW and CNCS for the SIF project. However, the Jeffco Foundation has growth JSEL during the summer of 

2015 to 30 schools using support from Jeffco Public Schools including allocations from Title 1 funding. The 

evidence of program effectiveness provided through the impact evaluations was part of the information used by 

Jeffco Public Schools to support JSEL’s growth and expansion. 

Several changes were made to the 2015 programing in response to the implementation evaluation and changes 

in the amount of resources that program planners had at their disposal. Several of the structures of the program 

changes, specifically structures related to family programming, transportation, resources, and assessment were 

altered or added including:

•

•

•

•

o

o

o

Utilizing an universal intervention (Leveled Literacy Intervention) and resource (Reading A to Z);

Adding the TRC/Reading 3D to assessments administered to participating students;

Adding a transportation system between schools and host site; and

Replacing field trips with on-site enrichment;

Hawk Quest: a presentation about birds of prey;

Author Reading: a reading and meet & greet with local author; and

Library visit: a visit from the local library.

In spite of the changes made as a result of the expansion, many of the original structures remained intact 

including:

•

•

•

Utilizing running records, CALI, and CAP;

Family breakfast & lunch; and

Ongoing professional development and learning opportunities for teachers, led and developed by 

coaches and principals.

APA and the Jeffco Assessment and Research Office will continue to provide evaluation support to the Jeffco 

Foundation on the JSEL project. This will include both formative feedback on their implementation and on-going 

impact evaluation using the study design used in this analysis. 
50

Conclusion 

This evaluation report contains two key pieces:

1. An implementation evaluation study conducted in summer 2014 by the Jeffco Public Schools research 

and assessment office. This study sought to ascertain the extent to which the JSEL program was 

implemented with fidelity and to identify formative recommendations to support the program; and

2. An impact evaluation study conducted by Augenblick, Palaich and Associates (APA). This impact study 

was designed to determine impacts of the JSEL program on student academic performance. The impact 

study was part of a planned, multiyear impact study that would gather student performance data over 

multiple years of JSEL program operation.

All material in this report regarding program implementation was authored by the Jeffco Public Schools research 

and assessment office. This material was incorporated by APA into the current report to provide JSEL with a 

unified evaluation document.

The Jeffco implementation study drew on data from a series of focus groups, interviews, and surveys of 

coordinators, administrators, teachers, and parents. Overall, Jeffco found that these key players communicate 

positive program experiences. However, feedback also showed that improvements to professional learning, 

communications structures, events and activities, teacher planning, and access to resources are desired.

APA’s impact study examined the relationship between JSEL participation and subsequent student performance 

on the DIBELS Next literacy assessment and Colorado TCAP assessment scores. The impact evaluation and 

approved Subgrantee Evaluation Plan for JSEL were designed to provide five years of student data to analyze in 

order to have appropriate numbers of JSEL-participating students to provide sufficient statistical power to study 

a series of key impact research questions.

However, at the current time, only three years of DIBELS Next data and only two years of TCAP data are 

available to answer research questions regarding JSEL’s impact on student The lack of additional years of data 

mean APA’s impact analysis is incomplete within the design of the original evaluation plan.

Despite these limitations, APA found a statistically significant impact of JSEL on student reading performance. In 

particular, APA found that participation in one summer of JSEL provides an effect equivalent to moving students 

from the 50th to the 54th percentile on the DIBELS Next literacy assessment. This finding offers strongly positive 

evidence of the program’s effectiveness.

Data limitations, however, greatly reduced APA’s overall ability to identify statistically significant results on its 

remaining impact study research questions. For these questions, which sought to identify impacts associated 

with multiple years of participation in JSEL, with higher student attendance in the program, and with reading 

proficiency on a statewide reading assessment, data was insufficient to identify effects, and a larger sample of 
51

students generated through additional years of JSEL operation is necessary for further study. Because of the 

small number of students available in the study sample, these results should not be interpreted as meaningful.
52

Appendix A: Teacher Focus Group Protocol 

First, I want to thank you for taking the time to participate in this focus group. My name is <insert name> 

and this is <notetaker name> and we are from Assessment and Research here at Jeffco. I’ll be leading our 

discussion, and <notetaker name> will take notes and sometimes ask clarifying questions.

The purpose of our conversation today is to talk about your experience as an instructor with the Jeffco 

Summer of Early Literacy program.

Everything we talk about is confidential – only our research group is allowed to review the notes from this 

session. All comments will be de-identified. Any information that is shared with outside sources will be 

used solely for the purpose of improving this program in future years.

When we write up this report – no individual names will be mentioned nor will names be associated with 

individual comments. Each of you has important things to say, so I’m going to make sure each of you gets 

time to tell us your thoughts and feelings about the program. It’s ok to disagree with anything I say, or 

things that other people say. The research needs your honest thoughts and feelings.

Our conversation will take approximately an hour. I want to respect the time that you are putting into this 

important work. In order to move the conversation along efficiently, there may be times when I need to 

stop you or indicate the need to move on to another question. This is not meant to be disrespectful in any 

way, but, rather, is done to ensure that we end this group discussion on time. If you have additional 

thoughts or ideas that you feel you did not have an opportunity to share during the focus group, I would 

be happy to stay and talk with you after the group has finished. Do you have any questions about anything 

I’ve said so far?

I know that many of you have participated in focus groups before, but I want to make sure we’re all on the 

same page. So, I want to establish the ground rules during our discussion. 

•

•

•

•

•

Ground Rules/Group Norms

There are no right or wrong answers to questions – just ideas, experiences and opinions, which 

are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

Since it is important for us to hear everyone’s ideas and opinions. Only one person talks at a time. 

Everyone participates, but no one dominates.

What we share is confidential. What is said in the room stays in the room. The report will not 

identify schools’ or individuals’ comments.

JSEL Teacher Focus Group: Guiding Questions

Introductions/Warm-up (15 min)
53

I want to start with getting a sense of your experience as teachers in JSEL. So lets start with going around 

the table and saying your name, site, and why you decided to participate in the summer school. I’ll start 

and then we can go around the room.

1. Describe your experience with this program? How did it go? 

•

•

•

•

•

•

•

•

What did you find challenging?

What did you find rewarding or really like?

Professional Development (30 min)

I want to talk about your professional development that you participating in June in preparation for JSEL. 

2. How did you apply what you learned from professional development in your classroom?

What parts of the professional development did you use on a regular basis?

What parts of the professional development were difficult to apply?

Examples?

3. What professional development did you need or would have wanted for JSEL?

• Please be specific 

4. Thinking about the regular school year compared to JSEL. How did your instruction differ?

How did you apply the Comprehensive Approach to Literacy Instruction (CALI) or CAP to your instruction 

in JSEL?

What are examples of you using CALI or CAPin your class?

What are challenges you faced when using CAP or CALI ?

5. Every week, you were asked to complete a running record for each student in your classroom. 

Describe that process and how you used that data?

•

•

•

•

•

•

How did you assign interventions to students based on their weekly reading performance? 

How did you decide how to best help each student?

What resources did you draw from? 

How did you implement these ideas for each student? 

What did you find most difficult about completing running records?

Prompt, if necessary: [What experience did each of you have with running records before 

beginning this program?

Instructional Support (20 Min)

I’d like to talk a little bit about the instructional support you received as a JSEL teacher.

6. During JSEL, what types of support did you receive from the coordinators and instructional coaches?
54

•

•

•

•

•

•

•

•

•

•

•

•

•

•

What type of instructional support did you receive when you faced challenges or difficulties?

Where did you seek help from? ( your colleagues, from online resources, or the instructional 

team?)

What is the role of Coordinators? /Instructional Coaches?

7. What about peer support? How did you collaborate as teachers?

How did you collaborate with colleagues during the summer session? (If not, why?)

8. Talk about the weekly meetings. How helpful were the school based meetings and the all JSEL school 

meetings?

What did you find valuable and why?

9. Did you got enough instructional support and professional development in the JSEL program?

What else did you need?

Was the feedback from coaches, coordinators and principals valuable to you?

Parent/Student engagement & Participation (15 min)

10. How did you engage with parents and students?

What are some parent conversations that stand out to you? 

What role did parents play in the program?

How is this similar or different to the regular school year? (Probe w/ why do you think this is? Can 

you give examples? Can you tell me a little bit more about that?)

(New additions) Did you see a difference between 4-6 parents or parents from the newer school 

sites?

Program Recommendations (10 min)

11. Overall, what would you recommend continuing next summer? What would you recommend not 

doing next year?

What advice would you give teachers just starting with the JSEL program?

What things did you wish you had known before the beginning of the program?

12. What parts of what you learned here will you use as a teacher in the fall semester?

What are you going to share back with colleagues in the fall from the JSEL program. 

13. Is there anything else about the JSEL program that you would like to talk about?
55

Appendix B: Principal Interview Protocol 

My name is < name> and I work with the Jeffco Assessment and Research department and we are 

conducting an evaluation of the implementation of the Summer Early Literacy program. I’ll be asking you a 

few questions, taking notes, and sometimes asking clarifying questions.

We are particularly interested in learning the experiences and insights of educators who have participated 

in the JSEL program this summer. The interview will take approximately 40 minutes, and consists of about 

one dozen questions. Please feel free to tell me if you do not want to respond to a question.

It is important to note that we will be sharing this information with the research team only. Everything you 

say here will remain anonymous and confidential. In our report to the district, respondents will not be 

identified in any way.

I want to pay attention to the time. In order to move the conversation along efficiently, there may be times 

when I may need to stop you or indicate the need to move on to another question. This is not meant to be 

disrespectful in any way, but, rather, is done to ensure that we end this discussion on time.

We would also like to audio-record this discussion to ensure accuracy of our notes. Just a reminder, all of 

your answers will remain confidential. Only the research team will be reviewing the tape and the transcript. 

Do I have your permission to tape this conversation? 

Thank you.

1. How long have you been an administrator in Jeffco Schools?

a. Why did you decide to participate in JSEL?

NEW SCHOOLS: 2. What was the process of developing a plan for implementation of the program prior to 

the start of the summer school session about the early literacy program?

a. How did the plan for implementation different with the actual teaching and learning this 

summer.

3. Tell me about the June professional development for JSEL. How valuable was it for you? For your 

teachers?

4. Let’s talk about running records. What is the impact of running records on instructional practice?

•

•

•

Is this something you do during the school year? 

What effect has running records had on student learning or parent involvement?

Can you give me an example?

6. During your walkthroughs in classrooms, what type of activities relating to literacy have you observed?
56

a. What changes have you seen in your teachers since the beginning of the program?

b. Can you give examples?

8. What was your process for conducting weekly meetings with teachers or any other type of professional 

development, on an ongoing basis?

9. Do you plan on using any of the activities or information you received during the summer school term 

this upcoming school year? How so?

10.What similarities or differences of the literacy program across the four schools have your observed?

• Is it consistent across schools or are their site based adaptations? Describe your adaptations. 

11. What have been your experiences with parents and their reaction to the program?

a. How has the program impacted parent involvement? 

How will JSEL impact parent involvement into the school year?

12.How did the JSEL program help support your goals associated with your UIP?

13. Finally, do you have any recommendations on how to improve the JSEL program for future years?

14. Is there anything else about the JSEL program that you would like to talk about?
57

Appendix C: Instructional Coach Focus Group Protocol 

JSEL Instructional Coach Focus Group (5 min) 

First, I want to thank you for taking the time to participate in this focus group. My name is <insert name> 

and this is <notetaker name> and we are from Assessment and Research here at Jeffco. I’ll be leading our 

discussion, and <notetaker name> will take notes and sometimes ask clarifying questions.

The purpose of our conversation today is to talk about your experience as an instructor with the Jeffco 

Summer of Early Literacy program.

Everything we talk about is confidential – only our research group is allowed to review the notes from this 

session. All comments will be de-identified. Any information that is shared with outside sources will be 

used solely for the purpose of improving this program in future years.

When we write up this report – no individual names will be mentioned nor will names be associated with 

individual comments. Each of you has important things to say, so I’m going to make sure each of you gets 

time to tell us your thoughts and feelings about the program. It’s ok to disagree with anything I say, or 

things that other people say. The research needs your honest thoughts and feelings.

Our conversation will take approximately an hour. I want to respect the time that you are putting into this 

important work. In order to move the conversation along efficiently, there may be times when I need to 

stop you or indicate the need to move on to another question. This is not meant to be disrespectful in any 

way, but, rather, is done to ensure that we end this group discussion on time. If you have additional 

thoughts or ideas that you feel you did not have an opportunity to share during the focus group, I would 

be happy to stay and talk with you after the group has finished. Do you have any questions about anything 

I’ve said so far?

I know that many of you have participated in focus groups before, but I want to make sure we’re all on the 

same page. So, I want to establish the ground rules during our discussion. 

•

•

•

•

•

Ground Rules/Group Norms

There are no right or wrong answers to questions – just ideas, experiences and opinions, which 

are all valuable.

It is important for us to hear all sides of an issue – both the positive and the negative.

Since it is important for us to hear everyone’s ideas and opinions. Only one person talks at a time. 

Everyone participates, but no one dominates.

What we share is confidential. What is said in the room stays in the room. The report will not 

identify schools’ or individuals’ comments.

JSEL Instructional Coach Focus Group: Guiding Questions 

Introductions/Warm-up (15 min)
58

I want to start with getting a sense of your experience as coaches in JSEL. So lets start with going around 

the table and saying your name, site, and why you decided to participate in the summer school. I’ll start 

and then we can go around the room.

1. Describe your experience with this program? How did it go? 

•

•

•

•

•

•

•

•

•

•

•

•

•

•

What did you find challenging?

What did you find rewarding or really like?

Professional Development (30 min)

I want to talk about your professional development that you participating in June in preparation for JSEL. 

2. How did how did teachers apply what they learned in the professional development in their classroom 

this summer?

What parts of the professional development did was used on a regular basis?

What parts of the professional development were difficult to apply?

Examples?

3. What professional development is you want or need for JSEL?

• Please be specific 

4. Thinking about the regular school year compared to JSEL. How did instruction differ?

How was the CALI or CAP to applied in JSEL?

What are examples of how CALI and CAP was used?

What are challenges did you and your teachers experience when using CAP or CALI ?

5. Every week, teachers were asked to complete a running record for each student in your classroom. 

Describe that process and how you used that data?

How did you help teachers assign interventions to students based on their weekly reading 

performance? 

How did you help teachers decide how to best help each student?

What resources did you draw from? 

How did you help implement these ideas for each student? 

What did you find most difficult about completing running records?

Prompt, if necessary: [What experience did each of you have with running records before 

beginning this program?

Instructional Support (20 Min)

I’d like to talk a little bit about the instructional support you gave to JSEL teacher.

6. During JSEL, what types of support did teachers need?
59

•

•

•

•

•

•

•

•

•

•

•

What kinds of challenges did teachers face?

7. Where did you seek help from?

your colleagues, from online resources, or the instructional team?

8. Talk about the weekly meetings. How helpful were the school based meetings and the all JSEL school 

meetings?

What did you find valuable and why?

9. Did feel like you got enough personal time with your teachers?

What else did you need to support your teachers?

Parent/Student engagement & Participation (15 min)

10. How did you engage with parents and students?

What are some parent conversations that stand out to you? 

What role did parents play in the program?

How is this similar or different to the regular school year? (Probe w/ why do you think this is? Can 

you give examples? Can you tell me a little bit more about that?)

(New additions) Did you see a difference between 4-6 parents or parents from the newer school 

sites?

Program Recommendations (10 min)

11. Overall, what would you recommend continuing next summer? What would you recommend not 

doing next year?

What advice would you give teachers just starting with the JSEL program? What advice would 

you give coaches?

What things did you wish you had known before the beginning of the program?

12. What parts of what you learned here will you use as a teacher in the fall semester?

What are you going to share back with colleagues in the fall from the JSEL program. 

13. Is there anything else about the JSEL program that you would like to talk about?
60

Appendix D: Program Coordinator Interview Protocol 

JSEL Summer School Coordinator Interview Questions

My name is < name> and I work with the Jeffco Assessment and Research department and we are 

conducting an evaluation of the implementation of the Summer Early Literacy program. We are particularly 

interested in learning the experiences and insights of educators who have participated in the JSEL program 

this summer.

The interview will take approximately 40 minutes, and consists of about one dozen questions. Please feel 

free to tell me if you do not want to respond to a question. I’ll be asking you a few questions, taking notes, 

and sometimes asking clarifying questions.

It is important to note that we will be sharing this information with the research team only. Everything you 

say here will remain anonymous and confidential. In our report to the district, respondents will not be 

identified in any way.

I want to pay attention to the time. In order to move the conversation along efficiently, there may be times 

when I may need to stop you or indicate the need to move on to another question. This is not meant to be 

disrespectful in any way, but, rather, is done to ensure that we end this discussion on time.

We would also like to audio-record this discussion to ensure accuracy of our notes. Just a reminder, all of 

your answers will remain confidential. Only the research team will be reviewing the tape and the transcript. 

Do I have your permission to tape this conversation? 

Thank you.

1. How long have you been associated with Jeffco Schools?

•

•

•

•

•

•

Why did you decide to participate in JSEL?

2. Tell me about the June professional development for JSEL. How valuable was it for you? For your 

teachers?

3. Let’s talk about running records. Can you describe what that process is like for your teachers?

Is this something you do during the school year?

What effect has running records had on student learning or parent involvement?

Can you give me an example?

4. During your walkthroughs in classrooms, what type of activities relating to literacy have you observed?

What changes have you seen in your teachers since the beginning of the program?

Can you give examples?
61

5. What was your process for conducting weekly meetings with teachers or any other type of professional 

development, on an ongoing basis?

•

•

•

•

6. What have been your experiences with parents and their reaction to the program?

Can you give me examples?

7. Do you plan on using any of the activities or information you received during the summer school term 

this upcoming school year? How so?

8. What similarities or differences of the literacy program across the four schools have your observed?

Is it consistent across schools or are their site based adaptations? Describe your adaptations.

9. What have been your experiences with parents and their reaction to the program?

How has the program impacted parent involvement? 

How will JSEL impact parent involvement into the school year?

10. How did the JSEL program help support your goals associated with your UIP?

11. Do you have any recommendations on how to improve the JSEL program for future years?

12. Is there anything else about the JSEL program that you would like to talk about?
62

Appendix E: Weekly Parent Survey 

Version 1 

Version 2 
63

Appendix F: Parent End-of-Year Survey 

I feel that I have a clear understanding of the Program’s goals.

Jeffco Summer of Early Literacy 

Parent Feedback Survey 

Strongly 

Agree Agree Disagree Strongly Disagree

No

Opinion

1. The hours of the summer school meets 

my needs as a caregiver.

2. The Program staff contacts my family 

often.

3. I know the expectations of the 

program. 

4. My student is reading more at home 

due to the program. 

5. I feel welcome to visit the school.

6. My child has access to reading supplies 

related to their class. 

7. The summer school teacher knows my 

child’s educational needs.

8. My child’s reading skills have improved 

as a result of the summer literacy 

program.

9. My child’s writing skills have improved 

as a result of the summer literacy 

program.

10. The teacher helps my child understand 

his or her homework. 
64

Always Often Sometimes Rarely Never

11. How often did you read with your child 

this summer? 

12. How often did you do a homework 

activity with your child?

13. How often did you attend program 

breakfasts/lunches?

14. How often did you serve as chaperone 

on field trips?

15. What three (3) things do you like about the Program?

16. What three (3) changes would you like to see in the Program? 
65

17. I have lived in this community for:

 Less than one year

 1 - 2 years

 2 1⁄2 - 3 years

 3 1⁄2 - 4 years

 More than 4 years

18. My child has been in the Summer School Literacy Program for:

 1 Year

 2 Years

 3 Years

19. Is the student of Latino/Hispanic origin?

 Yes

 No

20. Which of the following best describes the student’s race? 

 American Indian or Alaskan Native

 Asian

 Black or African American 

 Native Hawaiian or other Pacific Islander

 White
66

Appendix G: Statistical Model Output 

Question 1: Do JSEL participants make greater progress in their reading proficiency than similar nonparticipants? 

Mixed-effects ML 

regression Number of obs 1742

Group variable: schoolid Number of groups 94

Obs per group: 

min 1

avg 18.5

max 306

Wald chi2(19) 4849.14

Log likelihood = -

1295.886 Prob > chi2 0

Standardized Fall DIBELS Coef. Error z P>z

JSEL participation 0.10 0.04 2.61 0.009

2012 -0.01 0.04 -0.28 0.781

2013 0.03 0.04 0.68 0.498

White 0.06 0.03 1.87 0.062

Free and Reduced Lunch -0.01 0.04 -0.31 0.754

Grade Level 0.00 0.02 -0.13 0.900

Male 0.00 0.02 0.04 0.964

Standardized Spring 

DIBELS 0.85 0.01 62.47 0.000

Gifted and Talented 0.08 0.10 0.8 0.423

ELL -0.01 0.03 -0.23 0.822

IEP -0.06 0.04 -1.48 0.140

SPF Total 0.08 0.05 1.79 0.073

SPF Achieve 0.02 0.07 0.26 0.799

SPF Growth -0.06 0.06 -1.15 0.250

SPF Gap 0.06 0.05 1.22 0.221

SPF Reading -0.06 0.07 -0.9 0.367

SPF Writing 0.01 0.05 0.23 0.819

Percent FRL -0.02 0.15 -0.15 0.877

Stability rate 0.26 0.64 0.4 0.689

Mobility rate 0.00 (omitted)

Constant -0.36 0.62 -0.58 0.563

Random-effects 

Parameters Estimate Std. Err. [95% Conf. Interval]

schoolid: Identity
67

var(_cons) 0.002 0.002 0.000 0.019

var(Residual) 0.258 0.009 0.241 0.276

Question 2: Does the magnitude of the difference between JSEL participants and non-participants vary 

as a function of the number of years students participated? 

Mixed-effects ML regression Number of obs 1512

Group variable: schoolid

Number of 

groups 94

Obs per group: 

min 1

avg 16.1

max 224

Wald chi2(21) 4340.93

Log likelihood = -1106.9212 Prob > chi2 0

Standardized Fall DIBELS Coef. Std. Err. z P>z

JSEL in 1st year 0.14 0.04 3.11 0.002

JSEL in 2nd year -0.07 0.06 -1.12 0.261

JSEL in 3rd year 0.03 0.13 0.21 0.836

2012 0.07 0.05 1.46 0.145

2013 0.08 0.04 1.89 0.059

White 0.07 0.03 1.97 0.049

Free and Reduced Lunch -0.01 0.04 -0.14 0.892

Grade Level 0.01 0.02 0.73 0.468

Male 0.00 0.03 0.11 0.915

Standardized Spring DIBELS 0.86 0.01 59.16 0.000

Gifted and Talented 0.05 0.10 0.46 0.642

ELL -0.01 0.03 -0.34 0.733

IEP -0.03 0.04 -0.67 0.505

SPF Total 0.09 0.05 1.84 0.066

SPF Achieve 0.00 0.08 -0.03 0.980

SPF Growth -0.07 0.06 -1.29 0.196

SPF Gap 0.06 0.05 1.29 0.199

SPF Reading -0.03 0.07 -0.41 0.684

SPF Writing 0.01 0.06 0.27 0.788

Percent FRL 0.00 0.16 0.00 0.999

Stability rate -0.09 0.69 -0.13 0.899

Mobility rate 0.00 (omitted)

Constant -0.18 0.67 -0.26 0.792

Random-effects Parameters Estimate Std. Err. [95% Conf. Interval]
68

schoolid: Identity

var(_cons) 0.003 0.003 0.001 0.016

var(Residual) 0.251 0.009 0.233 0.270

Question 3: Are JSEL participants more likely to score “proficient” on third grade reading TCAP than 

non-participants?

Mixed-effects logistic 

regression Number of obs 254

Group variable: schoolid

Number of 

groups 43

Obs per group: 

min 1

avg 5.9

max 60

Integration points = 10 Wald chi2(18) 64.56

Log likelihood = -104.86651 Prob > chi2 0

Proficient on TCAP Coef. Std. Err. z P>z

JSEL participation 0.89 0.51 1.73 0.084

2012 -0.79 0.41 -1.93 0.053

White -0.50 0.50 -1 0.318

Free and Reduced Lunch -0.78 0.75 -1.04 0.3

Grade Level 12.02 345.59 0.03 0.972

Male -1.00 0.38 -2.65 0.008

Standardized Spring DIBELS 1.99 0.29 6.88 0

Gifted and Talented -0.05 1.07 -0.05 0.96

ELL -0.29 0.41 -0.69 0.488

IEP -0.08 0.67 -0.11 0.91

SPF Total -0.03 0.87 -0.04 0.971

SPF Achieve 1.77 1.08 1.63 0.103

SPF Growth -0.08 0.90 -0.09 0.925

SPF Gap -0.76 0.80 -0.95 0.34

SPF Reading 0.02 1.08 0.02 0.984

SPF Writing -0.29 0.87 -0.34 0.735

Percent FRL 3.03 2.30 1.32 0.188

Stability rate -14.95 8.76 -1.71 0.088

Mobility rate 0.00 (omitted)

Constant -10.11 691.17 -0.01 0.988

Random-effects Parameters Estimate Std. Err.

schoolid: Identity
69

sd(_cons) 1.81E-06 0.2656566

Mixed-effects ML 

regression Number of obs 254

Group variable: schoolid Number of groups 43

Obs per group: 

min 1

avg 5.9

max 60

Wald chi2(18) 271.47

Log likelihood = -

257.28991 Prob > chi2 0

Standardized TCAP Scale Coef. Std. Err. z P>z

JSEL participation 0.03 0.13 0.23 0.821

2012 -0.07 0.10 -0.77 0.444

White -0.09 0.12 -0.7 0.484

Free and Reduced Lunch -0.06 0.14 -0.4 0.691

Grade Level 0.39 0.71 0.55 0.584

Male -0.15 0.09 -1.72 0.085

Standardized Spring 

DIBELS 0.69 0.05 14.12 0

Gifted and Talented -0.08 0.25 -0.31 0.756

ELL 0.01 0.10 0.09 0.926

IEP -0.03 0.16 -0.23 0.822

SPF Total -0.20 0.22 -0.91 0.361

SPF Achieve 0.14 0.26 0.52 0.606

SPF Growth 0.05 0.22 0.21 0.834

SPF Gap -0.04 0.20 -0.19 0.852

SPF Reading 0.24 0.25 0.95 0.34

SPF Writing 0.06 0.20 0.3 0.763

Percent FRL 0.82 0.54 1.54 0.123

Stability rate -2.32 2.19 -1.06 0.29

Mobility rate 0.00 (omitted)

Constant 0.83 2.48 0.34 0.736

Random-effects 

Parameters Estimate Std. Err. [95% Conf. Interval]

schoolid: Identity

var(_cons) 0.009 0.013 0.000 0.165

var(Residual) 0.437 0.040 0.366 0.522
70

Question 4: Does the proportion of participants scoring “proficient” on third grade reading TCAP 

increase with the number of years enrolled in JSEL?

Mixed-effects logistic 

regression Number of obs 254

Group variable: schoolid Number of groups 43

Obs per group: min 1

avg 5.9

max 60

Integration points = 10 Wald chi2(19) 64.45

Log likelihood = -104.72465 Prob > chi2 0

Proficient on TCAP Coef. Std. Err. z P>z

JSEL in 1st year 0.949 0.527 1.8 0.072

JSEL in 2nd year 0.589 0.763 0.77 0.44

2012 -0.698 0.443 -1.58 0.115

White -0.497 0.503 -0.99 0.323

Free and Reduced Lunch -0.771 0.750 -1.03 0.304

Grade Level 11.243 472.384 0.02 0.981

Male -0.978 0.377 -2.6 0.009

Standardized Spring DIBELS 1.987 0.289 6.88 0

Gifted and Talented -0.027 1.087 -0.03 0.98

ELL -0.277 0.413 -0.67 0.504

IEP -0.054 0.670 -0.08 0.936

SPF Total -0.015 0.867 -0.02 0.986

SPF Achieve 1.926 1.124 1.71 0.087

SPF Growth -0.101 0.896 -0.11 0.91

SPF Gap -0.726 0.801 -0.91 0.364

SPF Reading -0.042 1.085 -0.04 0.969

SPF Writing -0.338 0.867 -0.39 0.696

Percent FRL 3.207 2.328 1.38 0.168

Stability rate -14.982 8.765 -1.71 0.087

Mobility rate 0.000 (omitted)

Constant -8.841 944.774 -0.01 0.993

Random-effects Parameters Estimate Std. Err.

schoolid: Identity

sd(_cons) 0.000 0.264
71

Question 5: Do students who attend JSEL regularly show greater growth in reading skills than 

students who attend less regularly?

Mixed-effects ML 

regression Number of obs 1742

Group variable: schoolid Number of groups 94

Obs per group: 

min 1

avg 18.5

max 306

Wald chi2(19) 4817.03

Log likelihood = -1297.95 Prob > chi2 0

Standardized Fall DIBELS Coef. Std. Err. z P>z

JSEL participation 0.00 0.00 1.48 0.14

2012 -0.03 0.04 -0.66 0.511

2013 0.01 0.04 0.16 0.869

White 0.07 0.03 2.04 0.042

Free and Reduced Lunch -0.02 0.04 -0.47 0.636

Grade Level 0.00 0.02 -0.16 0.875

Male 0.00 0.02 0.12 0.906

Standardized Spring 

DIBELS 0.85 0.01 62.34 0

Gifted and Talented 0.08 0.10 0.84 0.403

ELL 0.00 0.03 -0.09 0.928

IEP -0.06 0.04 -1.44 0.149

SPF Total 0.07 0.05 1.57 0.115

SPF Achieve 0.01 0.07 0.14 0.886

SPF Growth -0.06 0.06 -1.00 0.317

SPF Gap 0.06 0.05 1.22 0.224

SPF Reading -0.05 0.07 -0.67 0.501

SPF Writing 0.00 0.05 0.05 0.959

Percent FRL 0.04 0.15 0.27 0.789

Stability rate 0.49 0.66 0.74 0.46

Mobility rate 0.00 (omitted)

Constant -0.57 0.64 -0.89 0.372

Random-effects 

Parameters Estimate Std. Err. [95% Conf. Interval]

schoolid: Identity

var(_cons) 0.003 0.002 0.000 0.015

var(Residual) 0.258 0.009 0.241 0.276