Literacy First K–1 Report 

The 2017-18 Impact Evaluation 

JULY 2019 

Sarah Caverly | Ginger Stoker | Dong Hoon Lee | Grace 

Fleming 

MAKING RESEARCH RELEVANT 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 

 

 

 

 

 

Literacy First K–1 Report 

The 2017-18 Impact Evaluation 

APRIL 2019 

Sarah Caverly | Ginger Stoker | Dong Hoon Lee | Grace 

Fleming 

1000 Thomas Jefferson Street NW 

Washington, DC 20007-3835 

202.403.5000 

www.air.org 

Copyright © 2019 American Institutes for Research. All rights reserved. 

7895_04/19 

 

Contents 

Page 

Introduction ............................................................................................................................... 1 

Background ................................................................................................ ................................ 1 

Research Questions.................................................................................................................... 2 

Research Design ......................................................................................................................... 3 

Sample ................................................................................................ ....................................... 4 

Sample Characteristics................................................................................................................ 6 

Analysis ................................................................................................ ...................................... 9 

Results................................................................................................ ...................................... 10 

References................................................................................................ ................................ 13 

Appendix A. Data Tables............................................................................................................. 1 
 

 

 

 

 

 

 

 

 

 

Tables 

Page 

Table 1. Kindergarten Sample Descriptive Statistics, Statistical Significance, and Effect 

Size Differences.......................................................................................................................... 7 

Table 2. Grade 1 Sample Descriptive Statistics, Statistical Significance, and Effect Size 

Differences................................................................................................ .................................. 8 

Table 3. Treatment effects of Literacy First for Kindergarten Students...................................... 11 

Table 4. Treatment Effects of Literacy First for Grade 1 Students.............................................. 11 

Figures 

Page 

Figure 1. Consort Diagram Displaying Participation Status of Kindergarten Students.................. 5 

Figure 2. Consort Diagram Displaying Participation Status of Grade 1 Students.......................... 6 

Exhibits 

Page 

Exhibit A-1. Impact Analysis Results for Letter Sound Fluency, Kindergarten Students.............A-1 

Exhibit A-2. Impact Analysis Results for Kindergarten Decoding Fluency, Kindergarten 

Students................................................................................................ ...................................A-1 

Exhibit A-3. Impact Analysis Results for Nonsense Word Fluency, Grade 1 Students................A-2 

Exhibit A-4. Impact Analysis Results for Oral Reading Fluency, Grade 1 Students.....................A-2 

Exhibit A-5. Impact Analysis Results for Whole Words Read, Grade 1 Students........................A-3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

Introduction 

Literacy First is an AmeriCorps program that provides trained tutors for students in 

kindergarten through Grade 2 in the Austin, Texas area. Literacy First’s early literacy program is 

designed to strengthen students’ early reading and comprehension skills (e.g., phonemic 

awareness, letter sound identification) through daily, 30-minute tutoring sessions. The program 

tracks student data, including benchmark assessments, weekly progress monitoring, 

attendance, and demographics, as well as program fidelity of implementation data from tutors. 

Literacy First is currently implemented in more than 30 elementary schools in three districts in 

the Austin area, serving more than 1,600 K-2 students annually. Fifty percent of Literacy First’s 

tutors are bilingual, allowing the program to provide support to both Spanish and English 

speakers. The program identifies struggling readers each fall and provides tutoring across the 

school year. Literacy First has established itself as a program that supports highly trained 

volunteers to implement an intensive tutoring intervention that is relies upon data to guide 

instruction. 

During the 2018–19 school year, American Institutes for Research (AIR) partnered with Literacy 

First to conduct an independent evaluation of the impact of the Literacy First program on 

student outcomes. The evaluation used a multi-site randomized controlled trial design across 

22 schools in the Austin Independent School District (AISD). This report presents findings from 

analyses designed to estimate the impact of participation in Literacy First on the early reading 

skills of struggling readers in kindergarten and Grade 1. 

Background 

Over the past two decades, national attention has emphasized the critical role of early literacy 

and language instruction in preventing reading difficulties and improved our understanding of 

how young children learn to read (Connor, Alberto, Compton, & O’Connor, 2014; National 

Reading Panel, 2000; Raynor, Foorman, Perfetti, Pesetsky, & Seidenberg, 2001). The research 

has established that students who are not strong readers by Grade 3 are less likely to build 

vocabulary and interact with a wide variety of texts (Good, Simmons, & Kame’enui, 2001). In 

addition, recent national data indicate that 64% of fourth graders fail to reach proficient-level 

reading scores as measured by the National Assessment of Educational Progress, which is 

designed to measure students’ reading comprehension (National Center for Education 

Statistics, 2015). A recent study by Jacob, Armstrong, Bowden, and Pan (2016) demonstrated the 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 1 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

impact of a volunteer tutoring program on Grade 2 students’ reading comprehension skills. 

Despite these findings, few supplemental early literacy tutoring programs have conducted 

efficacy trials to demonstrate their potential impact on early literacy, comprehension, and 

language skills. 

Literacy First has established itself as a unique tutoring program in three key areas. First, Literacy 

First tutors are highly trained volunteers who receive more than 70 hours of training in best 

practices, as well as weekly follow-up visits from Literacy First experts and coaches. Second, 

Literacy First is intensive; each child is seen daily for approximately 30 minutes and receives 

tailored tutoring to address literacy/language needs. Third, Literacy First uses data to drive 

instruction. Tutors use a response-to-intervention model, with benchmark assessments (which 

take place three times per year) and weekly progress monitoring. 

Literacy First has invested in rigorous research to evaluate program impacts by implementing 

regression discontinuity design (RDD) studies over the past 3 years, using students’ beginningof-the-year (BOY) reading ability to determine program eligibility. Findings from the evaluations 

provide strong evidence that Literacy First significantly accelerated students’ reading skills 

across Grades K–2 in two Austin-area school districts and charter schools (Tackett, Leroux, & 

McFarland, 2013; Tidd, 2014, 2015). Effect sizes found across the annual evaluations range 

from .20 to .40 (Tidd, 2014, 2015). Additionally, a quasi-experimental study investigating the 

impact of Literacy First on students’ reading skills found that first- and second-grade students 

who graduated from Literacy First scored statistically significantly higher than matched 

comparison students on the Development Reading Assessment (Agile Analytics, 2018). 

However, there are limitations to this research base: None of the studies used an experimental 

design, and the RDD and propensity score matching studies have the potential for sampling and 

selection bias. The current study addresses this issue by employing a well-designed and RCT 

that will meet the WWC standards (version 4.0) in 22 schools with approximately 439 

kindergarten and 433 first grade students. 

Research Questions 

The study was designed to answer the following research questions: 

1. Does participation in Literacy First one-on-one tutoring have significant impacts on 

kindergarten students’ early reading skills, as measured by the Letter Sound Fluency (LSF) 

and Kindergarten Decoding Fluency (KDF) assessments? 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 2 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

2. Does participation in Literacy First one-on-one tutoring have significant impacts on Grade 1 

students’ early reading skills, as measured by the Dynamic Indicators of Basic Early Literacy 

Skills (DIBELS) Nonsense Word Fluency (NWF), DIBELS Oral Reading Fluency (ORF), and 

Whole Words Read (WWR) assessments? 

Research Design 

The evaluation employed a multi-site randomized controlled trial to estimate the impact of 

Literacy First on kindergarten and Grade 1 students’ reading outcomes. The study was 

conducted within 22 elementary schools in AISD. Each of the participating schools was 

oversubscribed by a minimum of six students and would not have been able to provide services 

to all eligible students. As such, the evaluation study design did not cause a reduction in 

services provided to students. Each of the schools continued to serve the same number of 

students as it would have in the absence of the evaluation. 

Following established procedure, all students in kindergarten and Grade 1 from participating 

schools were screened for Literacy First eligibility by the Literacy First tutors during the first 

week of September 2017. Eligibility for participation in Literacy First tutoring was determined 

by students’ scores on the BOY assessments. Assessments were administered in Spanish or 

English, depending on students’ current language abilities as determined by the elementary 

school. Following the assessments, parental consent1 was obtained for students identified as 

eligible to receive the program. 

Kindergarten students completed the KDF and LSF assessments, which were created by the 

program administrators. The KDF was created to measure decoding skills aligned with program 

content, while the LSF is a modified version of the AIMSweb LSF assessment. Grade 1 students 

completed the DIBELS NWF, DIBELS ORF, and WWR assessments.2 The WWR is a component of 

the DIBELS NWF. 

All students whose parents did not opt them out of the study and who scored at the Tier 2 level 

on the assessment were eligible to receive Literacy First services. Students in both the 

treatment and control group completed the same assessments at the end of the school year in 

1 Parental consent was passive, as approved by our Institutional Review Board (IRB) for this study. Parent opt-out letters were 

sent home to all families approximately 3 weeks before Literacy First’s administration of the BOY assessments. 

2 Students completing the assessments in Spanish completed the Indicadores Dinámicos del Éxito en la Lectura (IDEL) Fluidez en 

las Palabras sin Sentido and IDEL Fluidez en el Relato Oral. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

May 2018. Students completed the assessments in the language in which they were currently 

being tutored. 

Students’ assessment scores were converted to standardized z-scores using sample-based 

means and standard deviations on each of the assessments. Z-scores were created separately 

for Spanish and English versions of the assessments. This was necessary in order to place 

students’ scores on the DIBELS and Literacy First assessments on the same scales. While the 

Spanish and English versions of these assessments are substantively similar, they are not 

considered equivalent. 

Sample 

Figures 1 and 2 provide consort diagrams for the study summarizing student progression 

through the study for kindergarten and Grade 1 students, respectively. Across all 22 AISD 

schools, 520 kindergarten students were determined to be eligible based on their BOY 

assessment scores and were randomly assigned to either the treatment group or the control 

group (Figure 1). Using simple random assignment, 260 students were assigned to each group.3 

Following random assignment, four parents (three in the treatment group and one in the 

control group) opted out of the study prior to knowing their student’s group assignment. 

Additionally, one student died. These students were excluded from any data analysis. Pretest 

data were therefore available for 256 treatment group and 259 control group kindergarten 

students. At the end of the year, 224 treatment group and 218 control group students 

completed the end-of-year (EOY) assessments. Three of these students (two in the treatment 

group and one in the control group) completed the BOY assessments in Spanish and the EOY 

assessments in English. These students were not included in the outcomes analyses due to 

issues related to the use of standardized scores at BOY and EOY.4 The final sample of 

kindergarten students consisted of 222 treatment group students and 217 control group 

students. The overall attrition rate for kindergarten students was 14.8%, while the differential 

attrition rate between treatment and control groups was 2.9%. This is indicative of low attrition 

(What Works Clearinghouse, 2017). 

3 Due to small sample sizes within each school, as well as data availability, it was not possible to use a stratified random 

assignment procedure. 

4 Z-scores based on the study sample were constructed separately for the students completing the English and Spanish versions 

of each of the assessments. For students who completed the BOY assessments in Spanish and EOY assessments in English, the 

z-scores for the BOY and EOY assessments were not calculated relative to the same groups of students or based on the same 

scales. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 4 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

Figure 1. Consort Diagram Displaying Participation Status of Kindergarten Students 

All Students 

520 

Randomized 

256 

Randomized 

259 

Beginning of Year 

LSF & KDF Scores 

Treatment Control 

Beginning of Year 

LSF & KDF Scores 

Parent 

Denial 

Before 

Knowing 

Status 

1 

Parent 

Denial 

Before 

Knowing 

Status 

3 

Died 

1 

All= 256 All= 259 

English= 130 English= 133 

Spanish= 126 Spanish= 126 

End of Year LSF & End of Year LSF & 

KDF Scores KDF Scores 

All= 222 Included in Analyses All= 217 

English= 108 English= 105 

Spanish= 114 Spanish= 112 

Completed 

SPN BOY & 

ENG EOY 

1 

For Grade 1, 504 students were determined to be eligible based on their BOY assessment 

scores (Figure 2). Using the same procedure, 252 students were assigned to the treatment 

group and 252 were assigned to the control group. Following random assignment, four parents 

(one in the treatment group and three in the control group) opted their students out of the 

evaluation prior to knowing their student’s group assignment. These students were excluded 

from any data analysis. Pretest data were therefore available for 251 treatment group and 249 

control group Grade 1 students. At the end of the year, 219 treatment group and 217 control 

group students completed the EOY assessments. Three of the treatment group students 

completed the BOY assessments in Spanish and the EOY assessments in English. These students 

were not included in the outcomes analyses due to issues related to the use of standardized 

scores at the beginning and end of the year.

5 The final sample of Grade 1 students consisted of 

5 Z-scores based on the study sample were constructed separately for the students completing the English and Spanish versions 

of each of the assessments. For students who completed the BOY assessments in Spanish and EOY assessments in English, the 

z-scores for the BOY and EOY assessments were not calculated relative to the same groups of students or based on the same 

scales. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 5 
 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

216 treatment group students and 217 control group students. The overall attrition rate for 

grade 1 students was 13.4%, while the differential attrition rate between treatment and control 

groups was 1.1%. These estimates are indicative of low attrition (What Works Clearinghouse 

2017). 

Figure 2. Consort Diagram Displaying Participation Status of Grade 1 Students 

3 0 

Sample Characteristics 

The analytic samples used to examine the impact of Literacy First on the outcome assessments 

consisted of: (1) 222 treatment group and 217 control group kindergarten students, and (2) 216 

treatment group and 217 control group Grade 1 students. Baseline equivalence on key 

demographic and achievement variables was assessed using Hedge’s g for continuous variables 

(i.e., ORF scores) and Cox’s index for dichotomous variables (i.e., gender and previous Literacy 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 6 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

First tutoring). Tests of statistical significance between the treatment and control groups on 

baseline characteristics were also conducted. 

Tables 1 and 2 present descriptive data, mean, percentage, and effect size differences between 

the treatment and comparison groups for kindergarten and Grade 1 students, respectively. As 

shown, most of the effect size differences are in the acceptable range but are large enough to 

warrant inclusion in the final analytic impact models to control for baseline differences, based 

on What Works Clearinghouse (WWC) threshold standards (What Works Clearinghouse, 2017).

6 

The only variable exceeding WWC thresholds was race/ethnicity. Additionally, Tables 1 and 2 

present p values from analyses comparing the percentages of students in each category for the 

treatment and control groups, as well as differences in students’ BOY assessment scores. There 

were no statistically significant differences between students in the treatment or control 

groups on any of the student characteristics or outcomes. 

Table 1 presents the descriptive statistics for kindergarten students in the treatment and control 

groups. Overall, the kindergarten students were primarily Hispanic (85%) and economically 

disadvantaged (94.5%), had limited English proficiency (57.5%), and did not participate in special 

education (89.5%). 

Table 1. Kindergarten Sample Descriptive Statistics, Statistical Significance, and Effect Size 

Differences 

Treatment Control P Value Effect Size 

N % N % 

Race/Ethnicity 

Black 19 8.6 18 8.3 0.71 0.08+ 

Hispanic 182 82.0 190 88.0 0.08 -0.29++ 

White 14 6.3 7 3.2 0.13 0.44++ 

Other race/ethnicity 7 3.2 2 0.9 0.14 0.74++ 

Limited English Proficiency 

Yes 134 60.4 119 54.8 0.29 0.12+ 

6 The WWC does not require variables with effect size differences less than 0.05 to be included in analytic models. 

The WWC requires variables with values between 0.05 and 0.25 to be included in analytic models. The WWC 

considers variables with effect size differences larger than 0.25 to not be equivalent. Absolute values are used for 

effect size differences. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 7 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

No 88 39.6 98 45.2 0.29 -0.12+ 

Special Education 

Yes 22 9.9 25 11.5 0.50 -0.12+ 

No 200 90.1 192 88.5 0.50 0.12+ 

Gender 

Male 114 51.4 107 49.3 0.68 0.05+ 

Female 108 48.6 110 50.7 0.68 -0.05+ 

Economic Disadvantage 

Yes 206 93.6 204 94.9 .0.65 -0.12+ 

No 16 6.4 13 5.1 0.65 0.12+ 

Beginning-of-Year Assessment Scores 

Mean Std Dev Mean Std Dev P Value Effect Size 

LSF score 

(standardized) 

0.01 0.99 -0.02 1.01 0.75 0.03 

KDF score 

(standardized) 

0.02 1.12 -0.01 0.86 0.76 0.03 

*p<.05, **p<.01 

+0.05< absolute value of effect size ≤0.25, ++Absolute value of effect size >0.25 

Table 2 presents the descriptive statistics for Grade 1 students in the treatment and control 

groups. Similar to the kindergarten students, Grade 1 students were primarily Hispanic (84%) 

and economically disadvantaged (92%), had limited English proficiency (52%), and did not 

participate in special education (88.5%). The majority of students had not previously received 

Literacy First tutoring in kindergarten (58.8%). 

Table 2. Grade 1 Sample Descriptive Statistics, Statistical Significance, and Effect Size Differences 

Treatment Control P Value Effect Size 

N % N % 

Race/Ethnicity 

Black 21 9.7 20 9.2 0.72 0.07+ 

Hispanic 178 82.4 186 85.7 0.26 -0.18+ 

White 12 5.6 5 2.3 0.03* 0.69++ 

Other race/ethnicity 4 1.9 5 2.3 1.00 0.00 

Limited English Proficiency 

Yes 113 52.3 113 52.1 1.00 0.00 

No 103 47.7 104 47.9 1.00 0.00 

Special Education 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 8 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

Yes 28 13.0 21 9.7 0.33 0.18+ 

No 188 87.0 196 90.3 0.33 -0.18+ 

Gender 

Male 111 51.4 114 52.5 0.68 -0.05+ 

Female 105 48.6 103 47.5 0.68 0.05+ 

Economic Disadvantage 

Yes 198 92.1 199 91.7 1.00 0.00 

No 18 7.9 18 8.3 1.00 0.00 

Prior Literacy First Tutoring 

Yes 93 43.0 86 39.6 0.53 0.07+ 

No 123 57.0 131 60.4 0.53 -0.07+ 

Beginning-of-Year Assessment Scores 

Mean Std Dev Mean Std Dev P Value Effect Size 

ORF score 

(standardized) 

-0.02 0.92 0.02 1.07 0.64 -0.04 

NWF score 

(standardized) 

-0.03 1.03 0.03 0.97 0.55 -0.06+ 

WWR score 

(standardized) 

0.03 0.93 -0.03 1.06 0.58 0.06+ 

*p<.05, **p<.01 

+0.05< absolute value of effect size ≤0.25, ++Absolute value of effect size >0.25 

Analysis 

The study used a multi-site design in which students were randomly assigned within schools to 

treatment and control groups. Since students were nested within schools, a two-level hierarchical 

linear model was used to analyze the data, with students at level 1 and schools at level 2. The 

equation below was used to estimate the impact of Literacy First tutoring on Kindergarten and 

Grade 1 student outcomes.7 

Level-1 Students 

Yij = β0j + β1j*(Treatmentij) + β2j*(Tutoring 2016-17ij) + β3j*(Blackij) + β4j*(Whiteij) 

+ β5j*(Other Raceij) + β6j*(Maleij) + β7j*(Special Educationij) + β8j*(Economic disadvantageij) 

+ β9j*(BOY Assessmentij) + β10j*(Spanishij) + β11j*(LEPij) + rij 

Level-2 Schools 

7 The Tutoring 2016-17 variable was omitted from impact analyses using data for Kindergarten students. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 9 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

β0j = γ00 + u0j 

β1j = γ10 + u1j 

β2j = γ20 

β3j = γ30 

β4j = γ40 

β5j = γ50 

β6j = γ60 

β7j = γ70 

β8j = γ80 

β9j = γ90 

β10j = γ100 

β11j = γ100 

where Yij is the student outcome (i.e., LSF, ORF), β0j is the adjusted mean outcome score across 

school sites, and β1j is the adjusted treatment effect across school sites. A dummy variable 

indicator, Spanish, was included to control for differences on English and Spanish versions of 

the assessments. All variables, with exception of the treatment indicator, were group-mean 

centered. As shown, the treatment effect was allowed to vary across sites. 

Results 

Results of the analyses showed statistically significant, positive effects of Literacy First 

participation on kindergarten students’ letter sound fluency and decoding fluency relative to 

students who received supplemental reading supports through their classroom teacher (e.g., 

tutoring, and small group instruction). Results of the analyses also showed statistically 

significant, positive effects of Literacy First on participation Grade 1 students’ nonsense word 

fluency, oral reading fluency, and whole word reading relative to students that received 

supplemental reading supports from their teachers. For all outcomes, additional analyses were 

conducted to assess whether there was a significant interaction between the treatment effect 

and taking the Spanish versions of the assessments. These analyses did not reveal any 

statistically significant differences in treatment effect for students who completed Spanish 

versions of the assessments. 

Table 3 shows the treatment effects for kindergarten students. As shown in Table 3,8 treatment 

group students scored 0.83 standard deviations higher than control group students on the EOY 

LSF assessment on average across sites, controlling for baseline student characteristics. 

Similarly, treatment group students scored 0.53 standard deviations higher than control group 

8 Detailed data tables are provided in Appendix A. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 10 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

students on the EOY KDF assessment on average across sites (see Appendix A for data tables). 

The results were statistically significant. 

Table 3. Treatment effects of Literacy First for Kindergarten Students 

Kindergarten Assessment 

Treatment Effect 

(in standard deviation units) 

Letter Sound Fluency (LSF) 0.83** 

Kindergarten Decoding Fluency (KDF) 0.53** 

*p<.05, **p<.01 

Table 4 shows the results of the analyses examining the effect of Literacy First on Grade 1 

students’ nonsense word fluency, oral reading fluency, and whole word reading. Grade 1 

students who received Literacy First tutoring scored 0.33 standard deviations higher on the EOY 

DIBELS NWF assessment than students who did not receive Literacy First tutoring, on average 

across sites (see Appendix A). Similarly, Grade 1 students in the treatment group scored 0.38 

standard deviations higher on the EOY DIBELS ORF assessment than students in the control 

group, on average across sites. Grade 1 students in the treatment group also scored 0.24 

standard deviations higher on the EOY WWR assessment than students in the control group, on 

average across sites. The results were statistically significant. 

Table 4. Treatment Effects of Literacy First for Grade 1 Students 

First Grade Assessment 

Treatment Effect 

(in standard deviation units) 

Nonsense Word Fluency (NWF) 0.33** 

Oral Reading Fluency (ORF) 0.38** 

Whole Words Read (WWR) 0.24** 

*p<.05, **p<.01 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 11 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

Summary 

This study employed a multi-site, randomized, experimental design to estimate the impact of 

Literacy First on kindergarten and Grade 1 student outcomes. Kindergarten and Grade 1 

students were screened for eligibility in September 2018, based on student scores on BOY 

assessments measuring letter sound fluency and decoding fluency for kindergarten students 

and nonsense word fluency, oral reading fluency, and whole word reading for Grade 1 students. 

Eligible students were randomly assigned to a treatment condition in which students received 

Literacy First tutoring or a “business as usual” control condition. Both groups of students 

continued to receive any additional support normally received. At the end of the year, students 

completed the same assessments. Multilevel modeling was used to estimate the impact of 

participation in Literacy First on student outcomes. 

Overall, the study shows that students who received Literacy First tutoring outperformed 

students who did not receive Literacy First tutoring on all five of the outcome measures 

included in the evaluation. Results of the analyses showed statistically significantly, positive 

effects on kindergarten students’ letter sound fluency and decoding fluency. On average across 

sites, kindergarten students in the treatment group scored 0.83 standard deviations higher on 

the EOY LSF and 0.53 standard deviations higher on the EOY KDF than students in the control 

group, controlling for baseline student characteristics. Similarly, results of the analyses showed 

statistically significant, positive effects of Literacy First on Grade 1 students’ nonsense word 

fluency, oral reading fluency, and whole word reading. Grade 1 students in the treatment group 

scored 0.33 standard deviations higher on the DIBELS NWF, 0.38 standard deviations higher on 

the DIBELS ORF, and 0.24 standard deviations higher on the WWR than control group students 

on average across sites, controlling for baseline student characteristics. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 12 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Literacy First K–1 Report 

References 

Agile Analytics. (2018). Literacy First evaluation: Austin ISD impact analysis. Austin, TX: Author. 

Connor, C. M., Alberto, P. A., Compton, D. L., & O’Connor, R. E. (2014). Improving reading 

outcomes for students with or at risk for reading disabilities: A synthesis of the 

contributions from the Institute of Education Sciences Research Centers (NCSER 2014-

3000). Washington, DC: U.S. Department of Education, Institute of Education Sciences, 

National Center for Special Education Research. 

Good, R. H., Simmons, D. C., & Kame’enui, E. J. (2001). The importance and decision-making 

utility of a continuum of fluency-based indicators of foundational reading skills for thirdgrade high-stakes outcomes. Scientific Studies of Reading, 5(3), 257–288. 

Jacob, R., Armstrong, C., Bowden, A. B., & Pan, Y. (2016). Leveraging volunteers: An 

experimental evaluation of a tutoring program for struggling readers. Journal of 

Research on Educational Effectiveness, 9(1), 67–92. 

National Center for Education Statistics. (2015). A first look: 2015 mathematics and reading 

(National Assessment of Educational Progress at grades 4 and 8). Washington, DC: U.S. 

Department of Education, Institute of Education Sciences. 

National Reading Panel, National Institute of Child Health and Human Development. (2000). 

Teaching children to read: An evidence-based assessment of the scientific research 

literature on reading and its implication for reading instruction (NH Publication No. 00-

4769). Washington, DC: U.S. Government Printing Office. 

Raynor, K., Foorman, B. R., Perfetti, C. A., Pesetsky, D., & Seidenberg, M. S. (2001). How 

psychological science informs the teaching of reading. Psychological Science in the Public 

Interest, 2(2), 31–37. 

Tackett, K. K., Leroux, A. J., & McFarland, L. (2013). ACE: A Community for Education— 

evaluation report. Austin, TX: Author. 

Tidd, S. (2014). ACE: A Community for Education evaluation report, 2013–2014 school year. 

Austin, TX: Author. 

Tidd, S. (2015). ACE: A Community for Education evaluation report, 2014–2015 school year. 

Austin, TX: Author. 

What Works Clearinghouse. (2017). Standards handbook (Version 4.0). Washington, DC: U.S. 

Department of Education, Institute of Education Sciences. 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG 13 
 

 

 

 

 

 

 

Literacy First K–1 Report 

Appendix A. Data Tables 

Exhibit A-1. Impact Analysis Results for Letter Sound Fluency, Kindergarten Students 

Fixed Effect Coefficient Standard 

Error 

t-ratio Approx. d.f. P Value 

Intercept -0.45 0.12 -3.76 21 0.00** 

Treatment 0.83 0.76 10.91 21 0.00** 

BOY LSF 0.25 0.04 6.86 386 0.00** 

Black -0.11 0.14 -0.78 386 0.44 

White -0.08 0.19 -0.41 386 0.68 

Other race -0.27 0.27 -0.99 386 0.32 

Male -0.04 0.07 -0.54 386 0.59 

Limited English 

Proficiency (LEP) 

-0.13 0.17 -0.77 386 0.44 

Special education -0.37 0.12 -3.06 386 0.00** 

Economic 

disadvantage 

-0.28 0.17 -1.63 386 0.11 

Spanish 0.14 0.17 0.88 386 0.38 

*p<.05, **p<.01 

Exhibit A-2. Impact Analysis Results for Kindergarten Decoding Fluency, Kindergarten Students 

Fixed Effect Coefficient Standard 

Error 

t-ratio Approx. 

d.f. 

P Value 

Intercept -0.29 0.12 -2.32 21 0.03* 

Treatment 0.53 0.08 6.39 21 0.00** 

BOY KDF 0.18 0.04 4.38 385 0.00** 

Black -0.00 0.16 -0.03 385 0.98 

White -0.06 0.22 -0.29 385 0.77 

Other race 0.18 0.30 0.61 385 0.54 

Male -0.06 0.08 -0.70 385 0.49 

LEP -0.28 0.19 -1.51 385 0.13 

Special education -0.35 0.14 -2.59 385 0.10 

Economic 

disadvantage 

-0.53 0.19 -2.81 385 0.01** 

Spanish 0.37 0.19 1.98 385 0.05 

*p<.05, **p<.01 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG A–1 
 

 

 

 

 

 

Literacy First K–1 Report 

Exhibit A-3. Impact Analysis Results for Nonsense Word Fluency, Grade 1 Students 

Fixed Effect Coefficient Standard 

Error 

t-ratio Approx. 

d.f. 

P Value 

Intercept -0.14 0.07 -2.05 21 0.05* 

Treatment 0.33 0.09 3.70 21 0.00** 

BOY NWF 0.42 0.04 9.63 378 0.00** 

Tutoring 2016–17 -0.10 0.09 -1.05 378 0.30 

Black 0.10 0.07 -2.05 378 0.54 

White -0.03 0.24 -0.13 378 0.90 

Other race 0.34 0.32 1.10 378 0.27 

Male 0.20 0.09 2.34 378 0.02* 

LEP -0.07 0.09 -0.77 378 0.44 

Special education 0.01 0.14 0.05 378 0.96 

Economic 

disadvantage 

-0.05 0.17 -0.30 378 0.77 

Spanish -0.01 0.11 -0.10 378 0.92 

*p<.05, **p<.01 

Exhibit A-4. Impact Analysis Results for Oral Reading Fluency, Grade 1 Students 

Fixed Effect Coefficient Standard 

Error 

t-ratio Approx. 

d.f. 

P Value 

Intercept -0.17 0.08 -2.26 21 0.04* 

Treatment 0.38 0.08 4.75 21 0.00** 

BOY ORF 0.51 0.04 12.39 378 0.00** 

Tutoring 2016–17 0.08 0.08 0.91 378 0.36 

Black 0.01 0.15 0.09 378 0.93 

White 0.20 0.22 0.90 378 0.37 

Other race 0.28 0.29 0.96 378 0.34 

Male 0.07 0.08 0.90 378 0.37 

LEP -0.05 0.09 -0.57 378 0.57 

Special education 0.03 0.13 0.21 378 0.84 

Economic 

disadvantage 

-0.49 0.16 -3.08 378 0.00** 

Spanish -0.06 0.10 -0.56 378 0.57 

*p<.05, **p<.01 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG A–2 
 

 

 

 

Literacy First K–1 Report 

Exhibit A-5. Impact Analysis Results for Whole Words Read, Grade 1 Students 

Fixed Effect Coefficient Standard 

Error 

t-ratio Approx. 

d.f. 

P Value 

Intercept -0.11 0.07 -1.46 21 0.16 

Treatment 0.24 0.09 2.67 21 0.01** 

BOY WWR 0.38 0.05 8.39 378 0.00** 

Tutoring 2016–17 -0.10 0.09 -0.99 378 0.33 

Black 0.15 0.17 0.87 378 0.38 

White 0.07 0.25 0.28 378 0.78 

Other race 0.50 0.32 1.56 378 0.12 

Male 0.22 0.09 2.53 378 0.01* 

LEP 0.03 0.09 0.32 378 0.75 

Special education -0.04 0.14 2.67 378 0.77 

Economic 

disadvantage 

0.05 0.17 0.28 378 0.78 

Spanish -0.00 0.11 -0.04 378 0.97 

*p<.05, **p<.01 

AMERICAN INSTITUTES FOR RESEARCH | AIR.ORG A–3 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

_

-

Established in 1946, the American Institutes for 

Research (AIR) is an independent, nonpartisan, 

not-for-profit organization that conducts behavioral 

and social science research on important social 

issues and delivers technical assistance, both 

domestically and internationally, in the areas of 

education, health, and workforce productivity. 

MAKING RESEARCH RELEVANT 

AMERICAN INSTITUTES FOR RESEARCH 

1000 Thomas Jefferson Street NW 

Washington, DC 20007 3835 | 202.403.5000 

www.air.org 

LOCATIONS 

Domestic: Washington, DC (HQ) | Monterey, Sacramento, and San Mateo, CA | Atlanta, GA | Honolulu, HI | Chicago and Naperville, IL 

Indianapolis, IN | Metairie, LA | Waltham, MA | Frederick and Rockville, MD | Chapel Hill, NC | New York, NY | Columbus, OH | Cayce, SC 

Austin, TX | Reston, VA 

International: El Salvador | Ethiopia | Haiti | Honduras | Zambia 

7895 04/19