Literacy Lab 

Early Literacy Intervention

Treatment and Comparison Group

FAST Bridge Assessment Differences

2017 – 2018

Prepared by:

Sarah Pepper

Erica Fissel

Kirk Knestis
2 | P a g e

Table of Contents 

Summary and Recommendations .............................................................................................3

Study Design..............................................................................................................................3

Group Comparison Analysis......................................................................................................5

Change in Assessment Scores for Treatment and Comparison Students .................................................5

Assessment Scores for Treatment and Comparison Students at Baseline ................................................6

Using Propensity Scores to Construct Comparison Sample for Analyses................................................7

Reading Growth for Treatment vs. Comparison Students in Propensity Score Sample ........................8

Regional Analysis .................................................................................................................... 11

Washington D.C...............................................................................................................................................11

Virginia ..............................................................................................................................................................15

Missouri.............................................................................................................................................................19

Maryland............................................................................................................................................................23
3 | P a g e

Summary and Recommendations 

The Literacy Lab intervention is providing valuable support for students who enter the 

school year reading below grade level targets. After receiving tutoring support, these students 

achieve more reading growth than their peers who did not participate in the intervention. These 

advantages in reading growth to students who receive the literacy intervention extend to students in 

all four regions: Washington D.C., Virginia, Missouri and Maryland. Unfortunately, despite greater 

reading growth than their peers without tutoring, many of these students are still not reaching the 

grade level average weekly growth targets. 

In order to ensure that a greater number of students who receive tutoring support approach 

grade level weekly growth targets, Literacy Lab may wish to consider piloting variations to their 

model in different locations. It is possible that an intervention across more months may be useful in 

maintaining a favorable growth trajectory beyond the active intervention period. Another alternative 

might be to shift to lighter touch intervention for maintenance rather than removing all intervention 

supports once a student achieves their initial reading goal.

Study Design 

The Literacy Lab is an AmeriCorps grantee and a national replication partner of the Reading 

Corps model. The study described in the following sections examines The Literacy Lab's replication 

of the K-3 Reading Corps program in Washington D.C., Virginia, Missouri, and Maryland. Reading 

Corps treatment and comparison youth were compared using FAST Bridge Assessment data 

collected by the Literacy Lab in selected schools during the 2017-2018 academic year. The data file, 

which was shared with Carson Research Consulting (now operating under the tradename Inciter), 

included assessment scores at two or three time points for each kindergarten to third grade student. 

The analyses that are outlined here look at changes in the assessment scores or the average weekly 

reading growth between two time points.

Kindergarten students took two assessments—letter names and letter sounds—in the fall 

and in the spring. Scores from each assessment taken in the fall are compared to those from the 

spring. First grade students took the nonsense words assessment in the fall and winter. Scores from 

the fall assessment are compared to scores from the winter assessment. First graders also took the 

CBM reading assessment in the winter and spring; scores were compared between the two time 

points. Finally, second and third grade students took the CBM Reading assessment in the fall and 

spring; scores were compared between the two time points. 
4 | P a g e

In order to identify the most appropriate students to target for intervention within a school 

in which Literacy Lab tutors were working, students first completed the fall assessment associated 

with their current grade level. In most cases, students who earned a baseline score above the 

indicated benchmark for their grade did not receive treatment because they were already at grade 

level.1 Across students who earned a baseline score below their grade level threshold, Literacy Lab 

tutors typically began targeted 20 minute per day reading intervention with the students who were 

closest to meeting their grade level benchmark. As these students reached their goal, they were 

cycled out of the program and were replaced in the treatment pool by students who had initially 

tested lower.

The goal of this document is to present results for the analyses that sought to answer the 

following research questions:

 Do students who receive targeted intervention from Literacy Lab tutors have significant 

improvement in their reading scores?

 Do comparison students who do not receive targeted intervention from Literacy Lab 

tutors have significant improvement in their reading scores?

 Do collected data support the hypothesis that gains to reading are greater for students 

who receive targeted intervention from Literacy Lab tutors?

 Do students who receive targeted intervention from Literacy Lab tutors exceed average 

weekly reading growth targets for their grade level?

 Does the effectiveness of Literacy Lab targeted intervention vary by grade?

 Does the effectiveness of Literacy Lab targeted intervention vary by geographic region?

Some students in the original data file were not eligible for Literacy Lab intervention given 

the outlined criteria for treatment, thus, these ineligible students were removed from the analysis 

sample. The initial analysis sample was defined as all treatment youth plus all comparison youth who 

did not score above the fall targeted benchmark for their grade, meaning they were eligible for 

treatment. For first grade, youth were also included in the analysis sample when they scored below 

the winter targeted benchmark on the CBM Reading assessment.

 

1 The treatment group includes 52 cases that were above the benchmark for eligibility, but received treatment anyway (13

in Kindergarten, 14 in Grade 1, 18 in Grade 2, and 7 in Grade 3; across all four regions).
5 | P a g e

Group Comparison Analysis 

Using the analysis sample defined in this way, we present data to address the first two 

research questions.

Change in Assessment Scores for Treatment and Comparison Students 

Table 1 presents the baseline and final assessment scores for all treatment youth in the 

shared data file. It also includes the number of students in each grade level with two assessment 

points (N), and the average change on the assessments within each grade level with associated 

statistical significance as calculated through a paired t-test. Within each grade level, and on each 

assessment, we see that treatment youth show significant gains over the school year. These gains 

may be due to regular classroom instruction, the Literacy Lab targeted intervention, or a 

combination of the two. Subsequent tables will present results of analyses that seek to determine if 

there is added value from the Literacy Lab intervention.

Table 1: Average Baseline FAST Bridge Assessments by Grade Level for Treatment Youth,

All

Grade Level Assessment Treatment Group 

Average Pre-test 

Score

Treatment Group 

Average Post-test 

Score

Average 

Difference Across 

Time on 

Assessments2

N

Kindergarten Letter Names 13.51 39.37 25.86 *** 501

Letter Sounds 2.38 38.43 36.05 *** 501

1

st Grade Nonsense Words 21.07 49.05 27.99 *** 838

CBM Reading 22.90 41.91 19.00 *** 868

2

nd Grade CBM Reading 30.78 71.78 41.01 *** 777

3

rd Grade CBM Reading 60.05 97.71 37.66 *** 836

*** p < .001

Table 2 mimics the data presented in Table 1 for the comparison youth in the file who were 

part of the initial analysis sample. These youth had baseline scores on their grade level assessment 

low enough to mark them as eligible for treatment. As with the treatment youth, we see that within 

each grade level, on each assessment, comparison youth in the analysis sample show significant gains 

across the school year. 

 

2 Statistically significant differences in means are noted below each table, referencing applicable p-values. Differences in 

means without notes are not significant.
6 | P a g e

Table 2: Average Baseline FAST Bridge Assessments by Grade Level for Comparison 

Youth, Initial Analysis Sample

Grade Level Assessment Comparison 

Group Average 

Pre-test Score3

Comparison 

Group Average 

Post-test Score

Average 

Difference Across 

Time on 

Assessments

N

Kindergarten Letter Names 11.50 37.31 25.82 *** 508

Letter Sounds 1.49 23.82 22.33 *** 508

1

st Grade Nonsense Words 19.79 37.17 17.38 *** 144

CBM Reading 24.26 38.97 14.70 *** 115

2

nd Grade CBM Reading 23.31 53.36 30.05 *** 590

3

rd Grade CBM Reading 50.13 77.88 27.74 *** 588

*** p < .001

Assessment Scores for Treatment and Comparison Students at Baseline 

Table 3 compares the baseline scores on each assessment for the treatment and comparison 

youth. If we see that the baseline scores are equivalent for treatment and comparison youth, then the 

initial comparison group sample provides a fair test of the effectiveness, or added value provided by 

the Literacy Lab intervention. If they are not equivalent, we would want to statistically construct a 

more appropriate comparison group to more accurately answer whether the targeted intervention 

was more effective than regular classroom instruction alone in promoting growth in reading. As 

shown in Table 3, there is no significant difference between baseline scores for treatment and 

comparison youth in first grade on either of the two assessments. Kindergartners, second and third 

grade students who received treatment, however, tested significantly higher at baseline on average 

than the comparison youth in the initial analysis sample. 

 

3 The constructed comparison group includes all youth who did not receive treatment, but had a baseline score below 

the target for their grade level.
7 | P a g e

Table 3: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Comparison Youth, Initial Analysis Sample

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Comparison 

Group 

Average 

Pre-test Score

Difference 

Between 

Groups on 

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter Names 13.51 11.50 2.01 ** 501 508

Letter Sounds 2.38 1.49 0.89 *** 501 508

1

st Grade Nonsense 

Words

21.07 19.79 1.28 838 144

CBM Reading 22.90 24.26 1.36 868 115

2

nd Grade CBM Reading 30.78 23.31 7.47 *** 777 590

3

rd Grade CBM Reading 60.05 50.13 9.91 *** 836 588

** p < .01, *** p < .001

Using Propensity Scores to Construct Comparison Sample for Analyses

Except in the case of first grade—because the treatment and comparison samples are not 

equivalent at baseline, even after eliminating all comparison youth who scored above the threshold 

at baseline—we conducted propensity score analysis to construct equivalent comparison groups for 

the kindergarten, second and third grade samples. Within each grade, propensity score analysis is 

conducted with replacement, and resulting weights are applied to all subsequent analyses.4 For the 

kindergarten sample, propensity scores are calculated using district ID, and the fall scores on both 

the letter names and letter sounds assessments. For the second and third grade samples, propensity 

scores are calculated using district ID and the fall CBM Reading score. In each case, propensity 

scores with replacement are used rather than one to one matching because there were either an 

insufficient number of comparison youth or an insufficient number of comparison youth with 

similar baseline scores in the file. Using school ID rather than district ID was considered, but was 

rejected because it resulted in some extremely high weights for comparison youth where virtually all 

students in a given school received treatment. District ID is considered useful because it is expected 

that students within the same grade in the same district are experiencing the same district specified 

curriculum.

 

4 When conducting the propensity score analysis, all students in the file are considered for matching, including those 

who were not part of the previously defined analysis sample. 
8 | P a g e

Table 4 presents the baseline scores for treatment and comparison youth in the weighted 

propensity score sample. Three treatment cases from third grade were eliminated from the analysis 

sample because there were no appropriate comparison matches available. All other treatment cases 

are retained for the analysis. As shown in Table 4, the propensity score analysis successfully 

identified a comparison sample that is equivalent to the treatment group at baseline within each 

grade level and on each of the reading assessments. This propensity score sample will allow an 

accurate test of whether the targeted Literacy Lab intervention results in enhanced reading growth 

above and beyond that resulting from classroom instruction alone.

Table 4: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Constructed Comparison Group Youth, Weighted Propensity Score Sample

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Propensity 

Score 

Comparison 

Group5

Average 

Pre-test Score

Difference 

Between 

Groups on 

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter Names 13.51 13.99 0.48 501 250

Letter Sounds 2.38 2.38 0.04 501 250

1

st Grade Nonsense 

Words

21.07 19.79 1.28 838 144

CBM Reading 22.90 24.26 1.36 868 115

2

nd Grade CBM Reading 30.78 28.95 1.83 777 305

3

rd Grade CBM Reading 60.05 58.09 1.97 833 343

Reading Growth for Treatment vs. Comparison Students in Propensity Score Sample 

As shown in Table 5, which presents data for the propensity score sample, treatment and 

comparison youth in each grade level have shown growth across the school year on their grade level

reading assessments. For kindergarten youth, there is not a significant difference between treatment 

and comparison youth in their growth on the letter names assessment. Both treatment and 

comparison youth increased 26 points on the assessment, on average. On the letter sounds 

assessment, however, kindergarten youth in the treatment sample increased significantly more than 

youth in the constructed comparison group. While youth in the constructed comparison group 

 

5 The constructed comparison group is the group identified through propensity score matching, except for first grade 

where the original analysis sample is used.
9 | P a g e

showed gains of 26.4 points across the school year, gains for treatment youth were higher at 36.0, on 

average. Similarly, first grade youth who received treatment experienced greater gains than did first 

grade comparison youth on both the nonsense words assessment and the CBM reading assessment.

The findings are similar for second and third grade students, where treatment youth 

experienced greater gains on the CBM reading assessment than did the comparison youth who were 

comparable at baseline. It is important to note that the gains on the CBM reading assessment for 

first graders are of a smaller magnitude than those presented for second and third grade because 

they are calculated from winter to spring, while the growth for the two older grades is calculated 

from fall to spring.

Table 5: Comparison of Baseline FAST Bridge Assessments Reading Growth by Grade 

Level for Treatment and Constructed Comparison Group Youth, Weighted Propensity Score 

Sample

Grade Level Assessment Treatment 

Group 

Average 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Reading 

Growth

Difference 

Between 

Groups in 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter Names 25.86 26.24 0.38 501 250

Letter Sounds 36.05 26.37 9.68 *** 501 250

1

st Grade Nonsense 

Words

27.99 17.38 10.61 *** 838 144

CBM Reading 19.00 14.70 4.30 ** 868 115

2

nd Grade CBM Reading 41.01 32.06 8.95 *** 777 305

3

rd Grade CBM Reading 37.57 28.34 9.23 *** 833 343

** p < .01, *** p < .001

In addition to examining the absolute growth in reading scores across the school year, we 

calculated the average weekly reading growth for students in the weighted propensity score sample. 

These are presented in Table 6. As shown in the table, for each grade, the average weekly reading 

growth is greater for the students who received the intervention compared to their comparison 

group peers in the weighted propensity score sample. Even among treatment youth, however, many 

did not exceed the targeted weekly growth rates on the reading assessment for their grade. First 

grade treatment youth showed an average weekly growth of 1.65 on the nonsense words assessment, 

which exceeded the growth target of 1.59. On each of the other grade level assessments presented in 
10 | P a g e

Table 6, however, the average weekly growth rate for the average treatment student was lower than 

the targeted weekly growth for that grade.

Table 6: Comparison of Average Weekly Growth on Baseline FAST Bridge Assessments by 

Grade Level for Treatment and Constructed Comparison Group Youth, Weighted 

Propensity Score Sample

Grade Level Assessment Target 

Weekly 

Growth

Treatment 

Group 

Average 

Weekly 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Weekly 

Reading 

Growth

Difference 

Between 

Groups in 

Average 

Weekly 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Sounds

1.21 1.09 0.80 0.29 *** 501 250

1

st Grade Nonsense 

Words

1.59 1.65 1.02 0.62 *** 838 144

CBM 

Reading

1.88 1.19 0.92 0.27 ** 868 115

2

nd Grade CBM 

Reading

1.61 1.24 0.97 0.27 *** 777 305

3

rd Grade CBM 

Reading

1.06 1.14 0.86 0.28 *** 833 343

** p < .01, *** p < .001

Recognizing that the averages across all treatment youth were often shy of the target, we 

explored the percentage of youth in each group who were exceeding the grade level weekly growth 

targets. These are presented in Figure 1. For all paired comparisons, except for first grade on the 

CBM Reading assessment, a significantly higher percentage of treatment students compared to the 

comparison students exceed the average weekly targeted growth on the indicated assessment. For 

the first grade CBM Reading assessment, the difference between average weekly growth for 

treatment and comparison youth is not statistically significant. On each bar, the orange portion 

indicates the percentage of students who exceeded the targeted weekly growth on the specified 

assessment. Only among third graders did at least one-half of treatment youth exceed the target. 

Kindergartners on the letter sounds assessment and first graders on the nonsense words assessment 

are close behind at almost 42 percent and a little over 48 percent, respectively.
11 | P a g e

58.5%

80.8%

51.7%

76.4% 81.4% 85.2% 78.0%

90.1%

41.8%

67.2%

41.5%

19.2%

48.3%

23.6% 18.6% 14.8% 22.0%

9.9%

58.2%

32.8%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment (***)

Comparison

Treatment (***)

Comparison

Treatment (n.s.)

Comparison

Treatment (***)

Comparison

Treatment (***)

Comparison

Kindergarten: Letter

Sounds

1st Grade: Nonsense

Words

1st Grade: CBM

Reading

2nd Grade: CBM

Reading

3rd Grade: CBM

Reading

Figure 1: Average Weekly Growth in FAST Bridge Assessment Scores Relative to Grade 

Level Targets for Treatment and Comparison Youth by Grade 

Above Average

Target Weekly

Growth

Below Average

Target Weekly

Growth

Regional Analysis 

The section that follows explores whether the overall results are consistent across each of 

the four regions in which Literacy Lab tutors served students. Within each region we use the 

previously defined propensity score sample to address this question and present three tables and one 

figure. The first table in each region explores whether or not the treatment and comparison youth in 

the propensity score sample in the given region are equivalent at baseline. In many cases they are, 

but there are exceptions which must be considered when evaluating the second table presented for 

each region. The second table in each region compares the growth on each reading assessment for 

youth in the propensity score sample. The third table compares average weekly growth on the 

reading assessments for treatment and comparison youth, and the figure presents the percentage of 

students in each group who exceed the grade level average weekly growth targets.

Washington D.C. 

Tables 7 and 8 present the data for students from Washington D.C. Table 7 compares the 

baseline scores for treatment and comparison youth in the propensity score sample from 

Washington D.C. Second and third grade youth in the treatment and comparison groups are not 

significantly different at baseline. In addition, kindergarten youth in the two groups are not 
12 | P a g e

significantly different with respect to the letter sounds assessment. Comparison kindergarten youth 

do test slightly higher on the letter names assessment compared to their treatment group peers. Most 

important to note is that there are no first-grade comparison youth from Washington D.C. Also 

important to remember is that the first-grade analysis group includes all youth in the original file 

provided by Literacy Lab who tested below their grade level benchmark.

Table 7: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Constructed Comparison Group Youth, Weighted Propensity Score Sample, 

Washington D.C.

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Propensity 

Score 

Constructed 

Comparison 

Group 

Average Pretest Score

Difference 

Between 

Groups on 

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter 

Names

12.74 14.95 2.20 + 289 137

Letter 

Sounds

2.01 2.46 0.45 289 137

1

st Grade Nonsense 

Words

22.35 – – 303 0

CBM 

Reading

22.41 – – 312 0

2

nd Grade CBM 

Reading

30.11 30.82 0.71 296 112

3

rd Grade CBM 

Reading

54.27 56.76 2.49 294 113

+ p < .10

Table 8 presents a comparison of the growth on each reading assessment for treatment and 

comparison youth in the weighted propensity score sample from Washington D.C. Although they 

began at slightly different points, the growth for kindergarten students on the letter names 

assessment is equivalent. For the letter sounds assessment, growth is 8.6 points higher on average 

for kindergarten students who received the targeted reading intervention. In second grade, growth 

on the CBM reading assessment is 5.6 points higher for students receiving the targeted intervention, 

and in third grade, it is 9.9 points higher for students receiving the targeted intervention. Apart from 

the change on the letter names assessment, each of these differences shows statistically greater 

improvement for youth receiving the targeted literacy intervention from Literacy Lab tutors.
13 | P a g e

Table 8: Comparison of Baseline FAST Bridge Assessments Reading Growth by Grade 

Level for Treatment and Constructed Comparison Group Youth, Weighted Propensity Score 

Sample, Washington D.C.

Grade Level Assessment Treatment 

Group 

Average 

Reading 

Growth

Propensity 

Score 

Constructed 

Comparison 

Group 

Average 

Reading 

Growth

Difference 

Between 

Groups in 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Names

25.80 25.25 0.56 289 137

Letter 

Sounds

37.51 28.87 8.64 *** 289 137

1

st Grade Nonsense 

Words

27.05 – – 303 0

CBM 

Reading

17.87 – – 312 0

2

nd Grade CBM 

Reading

37.62 32.00 5.62 * 296 112

3

rd Grade CBM 

Reading

35.68 25.74 9.94 *** 294 113

* p < .05, *** p < .001

As shown in Table 9, the average weekly reading growth for students receiving the tutoring 

intervention exceeds that for the Washington D.C. students in the constructed comparison group. 

In addition, third grade treatment youth met the weekly growth targets on the CBM reading 

assessment and first grade students met the target on the nonsense words assessment. Since data 

were not available for comparison youth in Washington D.C., values are only presented for 

treatment youth for the two first grade assessments.
14 | P a g e

Table 9: Comparison of Average Weekly Growth on Baseline FAST Bridge Assessments by 

Grade Level for Treatment and Constructed Comparison Group Youth, Weighted 

Propensity Score Sample, Washington D.C.

Grade Level Assessment Target 

Weekly 

Growth

Treatment 

Group 

Average 

Weekly 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Weekly 

Reading 

Growth

Difference 

Between 

Groups in 

Average 

Weekly 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Sounds

1.21 1.14 0.87 0.26 *** 289 137

1

st Grade Nonsense 

Words

1.59 1.59 – – 303 0

CBM 

Reading

1.88 1.12 – – 312 0

2

nd Grade CBM 

Reading

1.61 1.14 0.97 0.17 * 296 112

3

rd Grade CBM 

Reading

1.06 1.08 0.78 0.30 *** 294 113

* p < .05, *** p < .001

Figure 2 presents a comparison of the percentage of students in the treatment and 

comparison groups who exceed or fall below the average weekly reading growth targets for their 

grade. Higher percentages of Washington D.C. treatment youth, compared to comparison youth, in 

kindergarten, second and third grade exceed the average weekly growth targets. Over one-half of 

third grade treatment youth exceed the weekly growth targets, as do just over 48 percent of 

kindergarteners and almost 46 percent of first graders on the nonsense words assessment.
15 | P a g e

Figure 2: Average Weekly Growth in FAST Bridge Assessment Scores Relative to Grade 

Level Targets for Treatment and Comparison Youth by Grade, Washington D.C. 

51.9%

75.7%

54.1%

83.0% 84.1% 89.0%

44.6%

79.7%

48.1%

24.3%

45.9%

17.0% 15.9% 11.0%

55.4%

20.3%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment (***)

Comparison

Treatment

Comparison

Treatment

Comparison

Treatment (+)

Comparison

Treatment (***)

Comparison

Kindergarten: Letter

Sounds

1st Grade: Nonsense

Words

1st Grade: CBM

Reading

2nd Grade: CBM

Reading

3rd Grade: CBM

Reading

Above Average

Target Weekly

Growth

Below Average

Target Weekly

Growth

Virginia

Table 10 compares the baseline scores for treatment and comparison youth in the propensity 

score sample from Virginia. As shown in Table 10, when we restrict the sample to students from 

Virginia, we see that the treatment and comparison groups are not equivalent at baseline on any of 

the assessments. In each case, the students in the treatment groups score higher on their baseline 

assessment than students in the comparison group. As was the case in Washington D.C., there are 

no first-grade comparison youth from Virginia. Since the groups in Virginia are not equivalent at 

baseline, the test of the effectiveness of the Literacy Lab intervention in this location is not as clean, 

and should be interpreted cautiously.
16 | P a g e

Table 10: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Constructed Comparison Group Youth, Weighted Propensity Score Sample, 

Virginia

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Propensity 

Score

Constructed 

Comparison 

Group 

Average Pretest Score

Difference 

Between 

Groups on 

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter 

Names

14.27 11.26 3.01 + 130 69

Letter 

Sounds

2.91 1.72 1.19 *** 130 69

1

st Grade Nonsense 

Words

20.41 – – 216 0

CBM 

Reading

22.74 – – 230 0

2

nd Grade CBM 

Reading

34.28 27.74 6.54 ** 195 89

3

rd Grade CBM 

Reading

67.77 56.89 10.88 *** 229 118

+ p < .10, ** p < .01, *** p < .001

Table 11 presents a comparison of the growth on each reading assessment for treatment and 

comparison youth in the weighted propensity score sample from Virginia. Although they began at 

slightly different points, the growth for kindergarten students on the letter names assessment is 

equivalent. For the kindergarten letter sounds assessment, as well as the second and third grade 

CBM Reading assessments, growth is significantly higher for the treatment youth than for youth in 

the comparison youth. It is important to note that these treatment youth were more proficient 

readers at baseline, so may have progressed more quickly even without the intervention. 
17 | P a g e

Table 11: Comparison of Baseline FAST Bridge Assessments Reading Growth by Grade 

Level for Treatment and Constructed Comparison Group Youth, Weighted Propensity Score 

Sample, Virginia

Grade Level Assessment Treatment 

Group 

Average 

Reading 

Growth

Propensity 

Score

Constructed 

Comparison 

Group 

Average 

Reading

Growth

Difference 

Between 

Groups in 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Names

27.77 30.18 2.41 130 69

Letter 

Sounds

34.58 22.02 12.56 *** 130 69

1

st Grade Nonsense 

Words

25.04 – – 216 0

CBM 

Reading

19.33 – – 230 0

2

nd Grade CBM 

Reading

44.01 34.73 9.27 *** 195 89

3

rd Grade CBM 

Reading

36.24 27.29 8.95 *** 229 118

*** p < .001

Table 12 presents the average weekly reading growth scores for treatment and comparison 

group students in Virginia. Kindergarten, second and third grade students who participated in the 

Learning Lab intervention had significantly higher average weekly reading score growth than their 

peers in the constructed comparison group, but this should be interpreted cautiously because the 

two groups were not equivalent at baseline. In Virginia, treatment students scored more favorably 

on the baseline assessment than their peers in the constructed comparison group. While treatment 

youth exceeded comparison youth on the average weekly reading growth, only third grade treatment 

youth met the weekly growth target for their grade.
18 | P a g e

Table 12: Comparison of Average Weekly Growth on Baseline FAST Bridge Assessments by 

Grade Level for Treatment and Constructed Comparison Group Youth, Weighted 

Propensity Score Sample, Virginia

Grade Level Assessment Target 

Weekly 

Growth

Treatment 

Group 

Average 

Weekly 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Weekly 

Reading 

Growth

Difference 

Between 

Groups in 

Average 

Weekly 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Sounds

1.21 1.05 0.67 0.38 *** 130 69

1

st Grade Nonsense 

Words

1.59 1.47 – – 216 0

CBM 

Reading

1.88 1.21 – – 230 0

2

nd Grade CBM 

Reading

1.61 1.33 1.05 0.28 *** 195 89

3

rd Grade CBM 

Reading

1.06 1.10 0.83 0.27 *** 229 118

*** p < .001

As shown in Figure 3, kindergarten, second and third grade students in Virginia who 

received the Literacy Lab intervention were more likely than their peers in the constructed 

comparison group to exceed the average weekly growth targets on their reading assessments. Over 

one-half of third grade treatment youth exceeded the average weekly growth targets for their grade, 

and 43 percent of first grade treatment youth exceeded the average weekly growth target on the 

nonsense word assessment. Fewer students exceeded the grade level weekly growth target on the 

other assessments.
19 | P a g e

Figure 3: Average Weekly Growth in FAST Bridge Assessment Scores Relative to Grade 

Level Targets for Treatment and Comparison Youth by Grade, Virginia

70.8%

93.1%

56.9%

83.5%

74.4%

92.0%

46.3%

67.4%

29.2%

6.9%

43.1%

16.5%

25.6%

8.0%

53.7%

32.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment (***)

Comparison

Treatment

Comparison

Treatment

Comparison

Treatment (***)

Comparison

Treatment (***)

Comparison

Kindergarten: Letter

Sounds

1st Grade: Nonsense

Words

1st Grade: CBM

Reading

2nd Grade: CBM

Reading

3rd Grade: CBM

Reading

Above Average

Target Weekly

Growth

Below Average

Target Weekly

Growth

Missouri

Table 13 compares the baseline scores for treatment and comparison youth in the propensity 

score sample from Missouri. As shown in Table 13, kindergarten, second and third grade youth in 

the treatment and comparison groups, where propensity scores were used, are not significantly 

different at baseline. Missouri is the only region with comparison first graders. While these 

comparison youth were not significantly different when compared to the treatment youth across all 

regions simultaneously, they do test slightly lower at baseline than first graders who received the 

targeted intervention in Missouri.
20 | P a g e

Table 13: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Constructed Comparison Group Youth, Weighted Propensity Score Sample, 

Missouri

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Propensity 

Score

Constructed 

Comparison 

Group 

Average Pretest Score

Difference 

Between 

Groups on

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter 

Names

16.05 16.65 0.59 37 19

Letter 

Sounds

2.22 3.26 1.04 37 19

1

st Grade Nonsense 

Words

22.27 19.79 2.48 + 212 144

CBM 

Reading

28.27 24.26 4.01 + 212 115

2

nd Grade CBM 

Reading

34.89 34.34 0.55 187 62

3

rd Grade CBM 

Reading

66.28 70.53 4.25 220 72

+ p < .10

Table 14 presents a comparison of the growth on each reading assessment for treatment and 

comparison youth in the weighted propensity score sample from Missouri. For kindergarten youth, 

there is not a statistically significant difference in the level of growth between the treatment and 

comparison youth. On both tests, however, growth is higher for the treatment youth, and the 

relatively small sample size for this grade level may in part explain the non-detection of a significant 

difference. Among first graders, growth on both the nonsense words and CBM Reading assessments 

was higher for treatment youth than for youth in the comparison group, but in each case, these 

treatment youth were more proficient readers at baseline, so may have progressed more quickly even 

without the intervention. Among second graders, where the two groups were equivalent at baseline, 

treatment youth show a 12-point higher gain on the CBM Reading assessment than comparison 

youth, on average. For third grade youth in Missouri, there is no evidence of enhanced growth in 

reading as a result of targeted intervention from Literacy Lab tutors. 
21 | P a g e

Table 14: Comparison of Baseline FAST Bridge Assessments Reading Growth by Grade 

Level for Treatment and Constructed Comparison Group Youth, Weighted Propensity Score 

Sample, Missouri

Grade Level Assessment Treatment 

Group 

Average 

Reading 

Growth

Propensity 

Score

Constructed 

Comparison 

Group 

Average 

Reading 

Growth

Difference 

Between 

Groups in 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Names

27.78 24.68 3.11 37 19

Letter 

Sounds

34.32 27.97 6.36 37 19

1

st Grade Nonsense 

Words

32.62 17.38 15.25 *** 212 144

CBM 

Reading

22.04 14.70 7.34 *** 212 115

2

nd Grade CBM 

Reading

46.95 34.58 12.37 *** 187 62

3

rd Grade CBM 

Reading

41.19 36.90 4.29 220 72

*** p < .001

Table 15 presents the average weekly reading growth treatment and comparison youth on 

each assessment together with the target weekly growth for each grade. First and second grade 

students in Missouri who participated in the Literacy Lab intervention showed higher average weekly 

growth on their reading assessments than their peers in the comparison group. Among students in 

kindergarten and third grade, the difference in average weekly growth between treatment and 

comparison youth was not statistically significant. Although they were not significantly different 

from comparison youth, both treatment and comparison group third graders exceeded the weekly 

grade level growth targets. The only other group of treatment youth exceeding the weekly growth 

targets in Missouri was first grade students on the nonsense words assessment.
22 | P a g e

Table 15: Comparison of Average Weekly Growth on Baseline FAST Bridge Assessments by 

Grade Level for Treatment and Constructed Comparison Group Youth, Weighted 

Propensity Score Sample, Missouri

Grade Level Assessment Target 

Weekly 

Growth

Treatment 

Group 

Average 

Weekly 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Weekly 

Reading 

Growth

Difference 

Between 

Groups in 

Average 

Weekly 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Sounds

1.21 1.04 0.85 0.19 37 19

1

st Grade Nonsense 

Words

1.59 1.92 1.02 0.90 *** 212 144

CBM 

Reading

1.88 1.38 0.92 0.46 *** 212 115

2

nd Grade CBM 

Reading

1.61 1.42 1.05 0.37 *** 187 62

3

rd Grade CBM 

Reading

1.06 1.25 1.12 0.13 220 72

*** p < .001

Figure 4 shows the percentage for Missouri treatment and comparison students who 

exceeded each of the average weekly growth targets in orange. As shown there, a higher percentage 

of Missouri students in first, second and third grade who participated in the Literacy Lab 

intervention exceeded the average weekly reading growth targets as compared to students in the 

constructed comparison samples. Among third grade students, nearly 67 percent of treatment youth 

exceeded the average weekly growth targets, while 56 percent of first grade treatments did so on the 

nonsense words assessment. 
23 | P a g e

56.8%

67.7%

43.9%

76.4% 73.6%

85.2%

69.0%

88.1%

33.2%

44.0%

43.2%

32.3%

56.1%

23.6% 26.4%

14.8%

31.0%

11.9%

66.8%

56.0%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment (n.s.)

Comparison

Treatment (***)

Comparison

Treatment (*)

Comparison

Treatment (***)

Comparison

Treatment (*)

Comparison

Kindergarten: Letter

Sounds

1st Grade: Nonsense

Words

1st Grade: CBM

Reading

2nd Grade: CBM

Reading

3rd Grade: CBM

Reading

Level Targets for Treatment and Comparison Youth by Grade, Missouri

Above Average

Target Weekly

Growth

Below Average

Target Weekly

Growth

Figure 4: Average Weekly Growth in FAST Bridge Assessment Scores Relative to Grade 

Maryland 

Table 16 compares the baseline scores for treatment and comparison youth in the propensity 

score sample from Maryland. As shown in Table 16, kindergarten, second and third grade youth in 

the treatment and comparison groups are not significantly different at baseline. As was the case for 

Washington D.C. and Virginia, there are no first-grade comparison youth from Maryland. 
24 | P a g e

Table 16: Comparison of Average Baseline FAST Bridge Assessments by Grade Level for 

Treatment and Constructed Comparison Group Youth, Weighted Propensity Score Sample, 

Maryland

Grade Level Assessment Treatment 

Group 

Average Pretest Score

Propensity 

Score

Constructed 

Comparison 

Group 

Average Pretest Score

Difference 

Between 

Groups on 

Pre-test

Treatment 

N

Control 

N

Kindergarten Letter 

Names

14.16 12.62 1.53 45 25

Letter 

Sounds

3.31 2.47 0.84 45 25

1

st Grade Nonsense 

Words

16.39 – – 107 0

CBM 

Reading

14.60 – – 114 0

2

nd Grade CBM 

Reading

18.09 19.90 1.81 99 42

3

rd Grade CBM 

Reading

44.07 42.46 1.61 90 40

Table 17 presents a comparison of the growth on each reading assessment for treatment and 

comparison youth in the weighted propensity score sample from Maryland. There is no significant 

difference in growth between treatment and comparison kindergartners on the letter names 

assessment, but kindergarten youth who received the targeted literacy intervention improved an 

average of 13 points more on the letter sounds assessment than their peers in the comparison group. 

In second grade, growth on the CBM Reading assessment is 9 points higher for students receiving 

the targeted intervention, and in third grade, it is 12 points higher for students receiving the targeted 

intervention. 
25 | P a g e

Table 17: Comparison of Baseline FAST Bridge Assessments Reading Growth by Grade 

Level for Treatment and Constructed Comparison Group Youth, Weighted Propensity Score 

Sample, Maryland

Grade Level Assessment Treatment 

Group 

Average 

Reading 

Growth

Propensity 

Score

Constructed 

Comparison 

Group 

Average 

Reading 

Growth

Difference 

Between 

Groups in 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Names

19.11 23.93 4.82 45 25

Letter 

Sounds

32.33 19.27 13.07 *** 45 25

1

st Grade Nonsense 

Words

27.41 – – 107 0

CBM 

Reading

15.79 – – 114 0

2

nd Grade CBM 

Reading

34.00 24.99 9.01 ** 99 42

3

rd Grade CBM 

Reading

38.23 25.99 12.25 ** 90 40

** p < .01, *** p < .001

Table 18 presents the average weekly reading growth for treatment and comparison students 

together with the grade level weekly growth targets. In Maryland, kindergarten, second and third 

grade students who participated in the Literacy Lab intervention show significantly higher levels of 

average weekly reading growth as compared to their peers in the constructed comparison group. 

Third grade students who received the intervention exceeded the weekly growth targets as did first 

grade treatment youth on the nonsense words assessment.
26 | P a g e

Table 18: Comparison of Average Weekly Growth on Baseline FAST Bridge Assessments by 

Grade Level for Treatment and Constructed Comparison Group Youth, Weighted 

Propensity Score Sample, Maryland

Grade Level Assessment Target 

Weekly 

Growth

Treatment 

Group 

Average 

Weekly 

Reading 

Growth

Propensity 

Score 

Comparison 

Group 

Average 

Weekly 

Reading 

Growth

Difference 

Between 

Groups in 

Average 

Weekly 

Reading 

Growth

Treatment 

N

Control 

N

Kindergarten Letter 

Sounds

1.21 0.98 0.58 0.40 *** 45 25

1

st Grade Nonsense 

Words

1.59 1.61 – – 107 0

CBM 

Reading

1.88 0.99 – – 114 0

2

nd Grade CBM 

Reading

1.61 1.03 0.76 0.27 ** 99 42

3

rd Grade CBM 

Reading

1.06 1.16 0.79 0.37 ** 90 40

** p < .01, *** p < .001

As shown in Figure 5, higher percentages of Maryland treatment than comparison youth in 

kindergarten, second and third grade exceeded the average weekly reading growth targets. Among 

third grade students receiving the intervention, 58 percent exceeded the weekly growth targets as did 

50 percent of first graders on the nonsense words assessment and one-third of kindergarten students 

on the letter sounds assessment. Only a small subset of first and second grade youth receiving the 

intervention met the average weekly growth targets on the CBM assessment for their grade.
27 | P a g e

20181024

66.7%

93.3%

49.5%

87.7% 83.8%

92.2%

42.2%

61.4%

33.3%

6.7%

50.5%

12.3% 16.2%

7.8%

57.8%

38.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Treatment (**)

Comparison

Treatment

Comparison

Treatment

Comparison

Treatment (+)

Comparison

Treatment (*)

Comparison

Kindergarten: Letter

Sounds

1st Grade: Nonsense

Words

1st Grade: CBM

Reading

2nd Grade: CBM

Reading

3rd Grade: CBM

Reading

Above Average

Target Weekly

Growth

Below Average

Target Weekly

Growth

Figure 5: Average Weekly Growth in FAST Bridge Assessment Scores Relative to Grade 

Level Targets for Treatment and Comparison Youth by Grade, Maryland