U.S. DEPARTMENT OF EDUCATION

Evaluation of the DC Opportunity 

Scholarship Program

Impacts After Three Years

Executive Summary

NCEE 2009-4051
Evaluation of the DC Opportunity 

Scholarship Program 

Impacts After Three Years

Executive Summary

March 2009

Patrick Wolf, Principal Investigator, University of Arkansas

Babette Gutmann, Project Director, Westat

Michael Puma, Chesapeake Research Associates

Brian Kisida, University of Arkansas

Lou Rizzo, Westat

Nada Eissa, Georgetown University

Marsha Silverberg, Project Officer, Institute of Education Sciences

NCEE 2009-4051 

U.S. Department of Education
U.S. Department of Education

Arne Duncan

Secretary 

Institute of Education Sciences

Sue Betka

Acting Director

National Center for Education Evaluation and Regional Assistance

Phoebe Cottingham

Commissioner

March 2009

This report was prepared for the Institute of Education Sciences under Contract No. ED-04-CO-0126. The 

project officer was Marsha Silverberg in the National Center for Education Evaluation and Regional 

Assistance. 

 

IES evaluation reports present objective information on the conditions of implementation and impacts of 

the programs being evaluated. IES evaluation reports do not include conclusions or recommendations or 

views with regard to actions policymakers or practitioners should take in light of the findings in the 

reports.

This report is in the public domain. Authorization to reproduce it in whole or in part is granted. While 

permission to reprint this publication is not necessary, the citation should be: Wolf, Patrick, Babette 

Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, and Nada Eissa. Evaluation of the DC Opportunity 

Scholarship Program: Impacts After Three Years Executive Summary (NCEE 2009-4051). Washington, 

DC: National Center for Education Evaluation and Regional Assistance, Institute of Education Sciences, 

U.S. Department of Education.

To order copies of this report, 

• Write to ED Pubs, Education Publications Center, U.S. Department of Education, P.O. Box 1398, 

Jessup, MD 20794-1398.

• Call in your request toll free to 1-877-4ED-Pubs. If 877 service is not yet available in your area, 

call 800-872-5327 (800-USA-LEARN). Those who use a telecommunications device for the deaf 

(TDD) or a teletypewriter (TTY) should call 800-437-0833.

• Fax your request to 301-470-1244. 

• Order online at www.edpubs.org. 

This report also is available on the IES website at http://ies.ed.gov/ncee. 

Upon request, this report is available in alternate formats such as Braille, large print, audiotape, or 

computer diskette. For more information, please contact the Department’s Alternate Format Center at 

202-260-9895 or 202-205-8113.
iii

Acknowledgments 

This report is the fifth of a series of annual reports mandated by Congress. We gratefully 

acknowledge the contributions of a significant number of individuals in its preparation and production.

Staff from the Washington Scholarship Fund provided helpful information and have always 

been available to answer our questions. 

We are also fortunate to have the advice of an Expert Advisory Panel. Members include: 

Julian Betts, University of California, San Diego; Thomas Cook, Northwestern University; Jeffrey Henig, 

Columbia University; William Howell, University of Chicago; Guido Imbens, Harvard University; 

Rebecca Maynard, University of Pennsylvania; and Larry Orr, formerly of Abt Associates and now an 

independent consultant.

The challenging task of assembling the analysis files was capably undertaken by Yong Lee, 

Quinn Yang, and Yu Cao at Westat. The management and conduct of the data collection was performed 

by Juanita Lucas-McLean and Sabria Hardy of Westat. Expert editorial and production assistance was 

provided by Evarilla Cover and Saunders Freeland of Westat. Jeffery Dean of the University of Arkansas 

ably assisted with the intermediate outcomes analysis and the drafting of chapter 4 and appendix F. 
iv

Disclosure of Potential Conflicts of Interests1

The research team for this evaluation consists of a prime contractor, Westat, and two 

subcontractors, Patrick Wolf (formerly at Georgetown University) and his team at the University of 

Arkansas Department of Education Reform and Michael Puma of Chesapeake Research Associates 

(CRA). None of these organizations or their key staff has financial interests that could be affected by 

findings from the evaluation of the DC Opportunity Scholarship Program (OSP). No one on the sevenmember Technical Working Group convened by the research team once a year to provide advice and 

guidance has financial interests that could be affected by findings from the evaluation.

 

1 Contractors carrying out research and evaluation projects for IES frequently need to obtain expert advice and technical assistance from 

individuals and entities whose other professional work may not be entirely independent of or separable from the particular tasks they are 

carrying out for the IES contractor. Contractors endeavor not to put such individuals or entities in positions in which they could bias the 

analysis and reporting of results, and their potential conflicts of interest are disclosed.
v 

Executive Summary

The District of Columbia School Choice Incentive Act of 2003, passed by Congress in 

January 2004, established the first federally funded, private school voucher program in the United States. 

As part of this legislation, Congress mandated a rigorous evaluation of the impacts of the Program, now 

called the DC Opportunity Scholarship Program (OSP). This report presents findings from the evaluation 

of the impacts 3 years after families who applied were given the option to move from a public school to a 

participating private school of their choice. 

The evaluation is based on a randomized controlled trial design that compares the outcomes 

of eligible applicants randomly assigned to receive (treatment group) or not receive (control group) a 

scholarship through a series of lotteries. The main findings of the evaluation so far include: 

• After 3 years, there was a statistically significant positive impact on reading test 

scores, but not math test scores. Overall, those offered a scholarship were performing 

at statistically higher levels in readingequivalent to 3.1 months of additional 

learningbut at similar levels in math compared to students not offered a scholarship 

(table 3). Analysis in prior years indicated no significant impacts overall on either 

reading or math achievement. 

• The OSP had a positive impact overall on parents’ reports of school satisfaction 

and safety (figures 3 and 4), but not on students’ reports (figures 3 and 4). Parents 

were more satisfied with their child’s school (as measured by the percentage giving the 

school a grade of A or B) and viewed their child’s school as safer and more orderly if 

the child was offered a scholarship. Students had a different view of their schools than 

did their parents. Reports of safety and school climate were comparable for students in 

the treatment and control groups. Overall, student satisfaction was unaffected by the 

Program. 

• This same pattern of findings holds when the analysis is conducted to determine 

the impact of using a scholarship rather than being offered a scholarship. Fourteen 

percent of students in our impact sample who were randomly assigned by lottery to 

receive a scholarship and who responded to year 3 data collection chose not to use their 

scholarship at any point over the 3-year period after applying to the Program.1 We use 

a common statistical technique to take those “never users” into account; it assumes that 

the students had zero impact from the OSP, but it does not change the statistical 

significance of the original impact estimates. Therefore, the positive impacts on 

reading achievement, parent views of school safety and climate, and parent views of 

 

1 This 14 percent “never user” rate among year 3 respondents in the impact sample differs from the 25 percent “never user” rate for the impact 

sample as a whole (Figure 1) because scholarship “never users” in the impact sample responded to year 3 data collection events at lower rates 

than did scholarship “ever users.” 
vi

satisfaction all increase in size, and there remains no impact on math achievement and 

no overall impact on students’ perceptions of school safety and climate or satisfaction 

from using an OSP scholarship. 

• The OSP improved reading achievement for 5 of the 10 subgroups examined.

2

Being offered or using a scholarship led to higher reading test scores for participants 

who applied from schools that were not classified as “schools in need of improvement” 

(non-SINI). There were also positive impacts for students who applied to the Program

with relatively higher levels of academic performance, female students, students 

entering grades K-8 at the time of application, and students from the first cohort of 

applicants. These impacts translate into 1/3 to 2 years of additional learning growth. 

However, the positive subgroup reading impacts for female students and the first 

cohort of applicants should be interpreted with caution, as reliability tests suggest that 

they could be false discoveries. 

• No achievement impacts were observed for five other subgroups of students, 

including those who entered the Program with relative academic disadvantage. 

Subgroups of students who applied from SINI schools (designated by Congress as the 

highest priority group for the Program) or were in the lower third of the test score 

distribution among applicants did not demonstrate significant impacts on reading test 

scores if they were offered or used a scholarship. In addition, male students, those 

entering high school grades upon application, and those in application cohort 2 showed 

no significant impacts in either reading or math after 3 years. 

DC Opportunity Scholarship Program 

The purpose of the new scholarship program was to provide low-income residents, 

particularly those whose children attend schools in need of improvement or corrective action under the 

Elementary and Secondary Education Act, with “expanded opportunities to attend higher performing 

schools in the District of Columbia” (Sec. 303). The scholarship, worth up to $7,500, could be used to 

cover the costs of tuition, school fees, and transportation to a participating private school. The statute also 

prescribed how scholarships would be awarded: (1) in a given year, if there are more eligible applicants 

than available scholarships or open slots in private schools, scholarships are to be awarded by random 

selection (e.g., by lottery), and (2) priority for scholarships is given first to students attending SINI public 

schools and then to families that lack the resources to take advantage of school choice options. 

 

2 The subgroups that are analyzed in this study were designated prior to the collection and analysis of data and are of particular policy interest 

based on the Program statute and education policy literature. The subgroups are: (1) whether students attended a school designated as in need 

of improvement (SINI) under the No Child Left Behind Act prior to application to the Programstudents were either attending a SINI-ever or 

SINI-never school; (2) whether students were relatively lower performing or relatively higher performing at baselinestudents were either in 

the bottom one-third or the top two-thirds of the test score distribution; (3) student gender; (4) whether students were entering grades K-8 or 9-

12 at the time of application; and (5) whether students were in application cohort 1 (applied in 2004) or application cohort 2 (applied in 2005).
vii

The Program is operated by the Washington Scholarship Fund (WSF). To date, there have 

been five rounds of applications to the OSP (table 1). Applicants in spring 2004 (cohort 1) and spring 

2005 (cohort 2) represent the majority of Program applicants; the evaluation sample was drawn from 

these two groups.3 A smaller number of applicants in spring 2006 (cohort 3), spring 2007 (cohort 4), and 

spring 2008 (cohort 5) were recruited and enrolled by WSF in order to keep the Program operating at 

capacity each year.

Table 1. OSP Applicants by Program Status, Cohorts 1 Through 5, Years 2004-2008

Cohort 1 

(Spring 2004)

Cohort 2 

(Spring 2005)

Total 

Cohort 1 and 

Cohort 2

Cohort 3 

(Spring 2006), 

Cohort 4 

(Spring 2007), 

and Cohort 5 

(Spring 2008)

Total, All 

Cohorts

Applicants 2,692 3,126 5,818 2,034 7,852

Eligible applicants 1,848 2,199 4,047 1,284 5,331

Scholarship awardees 1,366 1,088 2,454 1,284 3,738

Scholarship users in initial year of receipt 1,027 797 1,824 1,057 2,881

Scholarship users fall 2005 919 797 1,716 NA 1,716

Scholarship users fall 2006 788 684 1,472 333 1,805

Scholarship users fall 2007 678 581 1,259 671 1,930

Scholarship users fall 2008 496 411 909 807 1,714

NOTES: Because most participating private schools closed their enrollments by mid-spring, applicants generally had their eligibility 

determined based on income and residency, and the lotteries were held prior to the administration of baseline tests. Therefore, 

baseline testing was not a condition of eligibility for most applicants. The exception was applicants entering the highly 

oversubscribed grades 6-12 in cohort 2. Those who did not participate in baseline testing were deemed ineligible for the lottery and 

were not included in the eligible applicant figure presented above, though they were counted in the applicant total. In other words, 

the cohort 2 applicants in grades 6-12 had to satisfy income, residency, and baseline testing requirements before they were 

designated eligible applicants and entered in the lottery. 

The initial year of scholarship receipt was fall 2004 for cohort 1, fall 2005 for cohort 2, fall 2006 for cohort 3, fall 2007 for cohort 

4, and fall 2008 for cohort 5.

SOURCES: OSP applications and WSF’s enrollment and payment files.

Mandated Evaluation of the OSP

In addition to establishing the OSP, Congress mandated an independent evaluation of it be 

conducted, with annual reports on the progress of the study. The legislation indicated the evaluation 

should analyze the effects of the Program on various academic and nonacademic outcomes of concern to 

policymakers and use “. . . the strongest possible research design for determining the effectiveness” of the 

 

3 Descriptive reports on each of the first 2 years of implementation and cohorts of students have been previously prepared and released (Wolf, 

Gutmann, Eissa, Puma, and Silverberg 2005; Wolf, Gutmann, Puma, and Silverberg 2006) and are available on the Institute of Education 

Sciences’ website at http://ies.ed.gov/ncee.
viii

Program. The current evaluation was developed to be responsive to these requirements. In particular, the

foundation of the evaluation is a randomized controlled trial (RCT) that compares outcomes of eligible 

applicants (students and their parents) randomly assigned to receive or not receive a scholarship. This 

decision was based on the mandate to use rigorous evaluation methods, the expectation that there would 

be more applicants than funds and private school spaces available, and the statute’s requirement that 

random selection be the vehicle for determining who receives a scholarship. An RCT design is widely 

viewed as the best method for identifying the independent effect of programs on subsequent outcomes 

(e.g., Boruch, de Moya, and Snyder 2002, p. 74). Random assignment has been used by researchers 

conducting impact evaluations of other scholarship programs in Charlotte, NC; New York City; Dayton, 

OH; and Washington, DC (Greene 2001; Howell et al. 2002; Mayer et al. 2002).

The recruitment, application, and lottery process conducted by WSF with guidance from the 

evaluation team created the foundation for the evaluation’s randomized trial and determined the group of 

students for whom impacts of the Program are analyzed in this report. Because the goal of the evaluation 

was to assess both the short-term and longer term impacts of the Program, it was necessary to focus the 

study on early applicants to the Program (cohorts 1 and 2) whose outcomes could be tracked over at least 

3 years during the evaluation period. During the first 2 years of recruitment, WSF received applications 

from 5,818 students. Of these, approximately 70 percent (4,047 of 5,818) were eligible to enter the 

Program (table 1). Of the total pool of eligible applicants, 2,308 students who were attending public 

schools or were rising kindergarteners entered lotteries (492 in cohort 1; 1,816 in cohort 2), resulting in 

1,387 students assigned to the treatment condition and 921 assigned to the control condition. These 

students constitute the evaluation’s impact analysis sample and represent three-quarters of all students in 

cohorts 1 and 2 who were not already attending a private school when they applied to the OSP.

Data are collected from the impact sample each year, starting with the spring in which 

students applied to the OSP (baseline) and each spring thereafter. These data include assessments of 

student achievement in reading and mathematics using the Stanford Achievement Test version 9 (SAT9),4 surveys of parents, and surveys of students in grade 4 and above―administered by the evaluation 

team in central District of Columbia (DC) locations on Saturdays or weekday evenings because neither 

the public nor private schools would allow data collection on their campuses during the school day. In 

addition, the evaluation surveys all DC public and private schools each spring in order to address the 

statute’s interest in understanding how the schools are responding to the OSP.

 

4 Stanford Abbreviated Achievement Test (Form S), Ninth Edition. San Antonio, TX: Harcourt Educational Measurement, Harcourt Assessment, 

Inc., 1997.
ix

Participation in the OSP

In interpreting the impacts of the OSP, it is useful to examine the characteristics of the 

private schools that participate in the Program and the extent to which students offered scholarships (the 

treatment group) moved into and out of them during the first 3 years.

School Participation

The private schools participating in the OSP represent the choice set available to parents 

whose children received scholarships. That group of schools had mostly stabilized by the 2005-06 school 

year. The schools that offered the most slots to OSP students, and in which OSP students and the impact 

sample’s treatment group were clustered, have characteristics that differed somewhat from the average 

participating OSP school. Although 56 percent of all participating schools were faith-based (39 percent 

were part of the Catholic Archdiocese of Washington), 82 percent of the treatment group attended a faithbased school, with 59 percent of them attending the 22 participating Catholic parochial schools (table 2). 

Twenty-two percent of treatment group students were attending a school that charged tuition above the 

statutory cap of $7,500 during their third year in the Program (table 2) even though 38 percent and 46 

percent of participating schools charged tuitions above that cap in 2006-07 and 2007-08, respectively. 

Table 2. Features of Participating OSP Private Schools Attended by the Treatment Group in 

Year 3

Characteristic

Weighted 

Mean Highest Lowest Valid N

Archdiocesan Catholic schools (percent 

of treatment students attending)

59.2 NA NA 66

Other faith-based schools (percent of 

treatment students attending)

22.5 NA NA 66

Charging over $7,500 tuition (percent of 

treatment students attending) 

22.3 NA NA 48

Tuition $6,620 $29,902 $3,600 48

Enrollment 260.5 1,072 10 43

Student N 701

NOTES: “Valid N” refers to the number of schools for which information on a particular characteristic was available. When a tuition range was 

provided, the mid-point of the range was used. The weighted mean was generated by associating each student with the characteristics 

of the school he/she was attending, and then computing the average of these student-level characteristics. 

SOURCE: OSP School Directory information, 2004-05, 2005-06, 2006-07, and 2007-08, Washington Scholarship Fund.

While the characteristics of the participating private schools are important considerations for 

parents, in many respects it is how the schools differ from the public school options available to them that 

matters most. In the third year after applying to the OSP, students in the treatment and control groups did 
x 

not differ significantly regarding the proportion attending schools that offered a separate library (88 vs. 91 

percent), gyms (71 and 72 percent), and art programs (89 and 87 percent). There were the following 

statistically significant differences (at the .01 level): 

• Students in the treatment group were more likely than those in the control group to 

attend schools with a computer lab (96 vs. 87 percent), with special programs for 

advanced learners (48 vs. 32 percent), and that offered a music program (89 vs. 82

percent).

• Students in the treatment group were less likely than the control group to attend a 

school with a cafeteria facility (79 vs. 88 percent) or a nurse’s office (30 vs. 81 

percent). 

• Students in the treatment group were also less likely than those in the control group to 

attend a school that offered special programs for non-English speakers (26 vs. 57 

percent), special programs for students with learning problems (71 vs. 88 percent), 

counselors (69 vs. 82 percent), tutors (50 vs. 67 percent), and after-school programs

(86 vs. 92 percent).

Student Participation

As has been true in similar programs, not all students offered an OSP scholarship actually 

used it to enroll in a private school. For students assigned to the treatment group, during the first 3 years 

of the Program (figure 1):

• 25 percent (346 out of 1,387 students) of those offered an OSP scholarship never used 

it;

• 34 percent (473 students) used their scholarship during some but not all of the first 3 

years after the award; and

• The remaining 41 percent (568 students) used their scholarship consistently for the 

entire 3 years after the lottery.

The reasons for not using the scholarshipeither initially or consistentlyvaried. The most 

common reasons cited by parents whose child never used their scholarship at anytime in year 3 and who 

completed surveys were (figure 2): 

• Lack of available space in the private school they wanted their child to attend (22 

percent of these parents); 

• Child moved out of DC (21 percent of these parents);

• Child was accepted into a public charter school (19 percent of these parents); and 
xi

• Participating schools did not offer services for their child’s learning or physical 

disability or other special needs (16 percent of these parents).

Figure 1. Proportions of Treatment Group Students Who Experienced 

Various Categories of Usage in First 3 Years

Never used

25%

Consistently used

41%

Partially used

34%

NOTES: Data are not weighted. Valid N = 1,387. Students were identified as scholarship users based upon 

information from WSF’s payment files. Because some schools use a range of tuitions and some 

students had alternative sources of funding, students were classified as full users if WSF made 

payments on their behalf that equaled at least 80 percent of the school’s annual tuition. Otherwise, 

students were identified as partial users (1 percent to 79 percent of tuition paid) or nonusers (no 

payments). 

SOURCES: OSP applications and WSF’s payment files.

The most common responses given by parents whose child initially used a scholarship in year 3 but 

dropped out of the OSP include:

• Lack of academic support that the child needed (39 percent of these parents);

• "Child did not like the private school" (25 percent);

• There was another private school the child liked better (13 percent);

• Work at the private school was too hard (11 percent);

• It was too difficult to get the child to the private school each day (11 percent); and

• The discipline or rules at the private school were too strict (7 percent).
xii

Figure 2. Most Common Reasons Given by Parents for Declining to Use 

the OSP Scholarship in Year 3

22.0%

20.8%

18.5%

15.5%

10.1%

0%

5%

10%

15%

20%

25%

Lack of space Moved out of DC Got into charter

school

Lack of special

needs services

Transportation

issues

NOTES: Responses are unweighted. Respondents were able to select multiple responses, which generated a 

total of 180 responses provided by 153 parents. This equates to an average of 1.2 responses per 

parent. 

SOURCE: Impact Evaluation Parent Surveys.

Students who were partial users were more likely to have special needs and those entering the higher 

grades averaged lower baseline test scores than students who participated consistently across the 3 years.5

Students who never used the OSP scholarship offered to them, or who did not use the 

scholarship consistently, could have found their way into other (non-OSP-participating) private schools, 

public charter schools, or traditional DC public schools. The same alternatives were available to students 

who applied to the OSP but were never offered a scholarship (the impact sample’s control group). Both 

the treatment and control groups moved between public (both traditional and charter) and private schools 

or between SINI and non-SINI schools. As a result, over the 3 years after they applied to the OSP: 

• Among the treatment group, 3 percent remained in the same school they were in when 

they applied to the Program; 46 percent switched schools once; 40 percent switched 

schools twice; and 11 percent switched three times. 

 

5 At baseline, partial users in grades 9-12 were lower performing in reading (27 National Percentile Ranks (NPRs) vs. 40 NPRs for full users, 

statistically significant at the .05 level) and in math (29 NPRs vs. 49 NPRs for full users, statistically significant at the .01 level); partial users 

in grades 6-8 were lower performing in math (34 NPRs vs. 41 NPRs for full users, statistically significant at the .01 level); and partial users 

were more likely to have special needs (5 percent vs. 10 percent for full users, statistically significant at the .05 level). 
xiii

• Among the control group, 15 percent remained in the same school they were in when 

they applied to the Program; 40 percent switched schools once; 37 percent switched 

schools twice; and 8 percent switched three times.

These patterns of student mobility are important because previous studies suggest that 

switching schools has an initial short-term negative effect on student achievement (Hanushek, Kain, and 

Rivkin 2004).

Impact of the Program After 3 Years: Key Outcomes

The statute that authorized the OSP mandated that the Program be evaluated with regard to 

its impact on student test scores and school safety, as well as the “success” of the Program, which, in the 

design of this study, includes satisfaction with school choices. The impacts of the Program on these 

outcomes are presented in two ways: (1) the impact of the offer of an OSP scholarship, derived straight 

from comparing outcomes of the treatment and control groups, and (2) the impact of using an OSP 

scholarship, calculated from the unbiased treatment-control group comparison, but statistically adjusting 

for students who declined to use their scholarships.6 The main focus of this study was on the overall 

group of students, with a secondary interest in students who applied from SINI schools, followed by other 

subgroups of students (e.g., defined by their academic performance at application, their gender, or their 

grade level).

Previous reports released in spring 2007 and spring 2008 indicated that 1 and 2 years after 

application, there were no statistically significant impacts on overall academic achievement or on student 

perceptions of school safety or satisfaction (Wolf et al. 2007; Wolf et al. 2008). Parents were more 

satisfied if their child was in the Program and viewed their child’s school as safer and more orderly. 

Among the secondary analyses of subgroups, there were impacts on math test scores in year 1 for students 

who applied from non-SINI schools and those with relatively higher pre-Program test scores, and impacts 

in reading test scores (but not math) in year 2 for those same two subgroups plus students who applied in 

the first year of Program implementation. However, these findings were no longer statistically significant 

when subjected to a reliability test to adjust for the multiple comparisons of treatment and control group 

students across 10 subgroups; the results may be “false discoveries” and should therefore be interpreted 

and used with caution. Throughout this report, the phrases “appears to have an impact” and “may have 

 

6 This analysis uses straightforward statistical adjustments to account not only for the approximately 14 percent of impact sample year 3 

respondents who received the offer of a scholarship but declined to use it over the 3-year period after application (the “never users”), but also 

the estimated 1.6 percent of the control group who never received a scholarship offer but who, by virtue of having a sibling with an OSP 

scholarship, ended up in a participating private school (we call this “program-enabled crossover”). These adjustments increase the size of the 

scholarship offer effect estimates, but do not alter the statistical significance of the impact estimate.
xiv

had an impact” are used to caution readers regarding statistically significant impacts that may have been 

false discoveries.

The analyses in this report were conducted using data collected on students 3 years after they 

applied to the OSP.

7

 

Impacts on Students and Parents Overall

• Across the full sample, there was a statistically significant impact on reading 

achievement of 4.5 scale score points (effect size (ES) = .13)8 from the offer of a 

scholarship and 5.3 scale score points (ES = .15) from the use of a scholarship 

(table 3). These impacts are equivalent to 3.1 and 3.7 months of additional learning, 

respectively.9

• There was no statistically significant impact on math achievement, overall (ES = .03) 

from the offer of a scholarship nor from the use of a scholarship (table 3).10

• Parents of students offered a scholarship were more likely to report their child’s school 

to be safer and have a more orderly school climate (ES = .29) compared to parents of 

students not offered a scholarship (figure 3); the same was true for parents of students 

who chose to use their scholarships (ES = .34). 

• On the other hand, students who were offered a scholarship reported similar levels of 

school safety and an orderly climate compared to those in the control group (ES = .06; 

figure 3); there was also no significant impact on student reports of school safety and 

an orderly climate from using a scholarship (ES = .07). 

• The Program produced a positive impact on parent satisfaction with their child’s school 

as measured by the likelihood of grading the school an “A” or “B,” both for the impact 

of a scholarship offer (ES = .22; figure 4) and the impact of scholarship use (ES = .26). 

 

7 Specifically, year 3 test scores were obtained from 69 percent of study participants, whereas parent survey data were gathered from 68 percent 

of participants and student survey data from 67 percent of participants. Response rates to the principal survey varied between 51.8 percent and 

57.3 percent, depending on academic year and school sector. Missing outcome data create the potential for nonresponse bias in a longitudinal 

evaluation such as this one, if the nonrespondent portions of the sample are different between the treatment and control groups. Response rates 

differed by less than 2 percent between the treatment and control groups for the tests and parent and student surveys, meaning that similar 

proportions of the treatment and control groups provided outcome data. In addition, nonresponse weights were used to equate the two groups 

on important baseline characteristics, thereby reducing the threat of nonresponse bias in this case. 

8 An effect size (ES) is a standardized measure of the relative size of a program impact. In this report, effect sizes are expressed as a proportion 

of a standard deviation of the distribution of values observed for the study control group. One full standard deviation above and below the 

average value for a variable such as outcome test scores contains 64 percent of the observations in the distribution. Two full standard 

deviations above and below the average contain 95 percent of the observations.

9 Scale score impacts were converted to approximate months of learning first by dividing the impact ES by the ES of the weighted (by grade) 

average annual increase in reading scale scores for the control group. The result was the proportion of a typical year of achievement gain 

represented by the programmatic impact. That number was further divided by nine to convert the magnitude of the gain to months, since the 

official school year in the District of Columbia comprises 9 months of instruction.

10 The magnitudes of these estimated achievement effects are below the threshold of .12 standard deviations, estimated by the power analysis to 

be the study’s Minimum Detectable Effect (MDE) size.
xv

Table 3. Year 3 Impact Estimates of the Offer and Use of a Scholarship on the Full Sample: 

Academic Achievement

Impact of the Scholarship Offer (ITT)

Impact of Scholarship 

Use (IOT)

Student 

Achievement

Treatment 

Group 

Mean

Control 

Group 

Mean

Difference 

(Estimated 

Impact) Effect Size

Adjusted 

Impact 

Estimate Effect Size

p-value of 

estimates

Reading 635.44 630.98 4.46* .13 5.27* .15 .01

Math 630.15 629.35 .81 .03 .95 .03 .62

*Statistically significant at the 95 percent confidence level.

NOTES: Means are regression adjusted using a consistent set of baseline covariates. Impacts are displayed in terms of scale scores. Effect sizes 

are in terms of standard deviations. Valid N for reading = 1,460; math = 1,468. Separate reading and math sample weights used.

Figure 3. Parent Perceptions and Student Reports of Safety and an 

Orderly School Climate 

Year 3 Group Means: Student 

Reports

6.17 6.05

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Treatment Control

Year 3 Group Means: Parent 

Perceptions

7.07

8.08**

0.00

1.00

2.00

3.00

4.00

5.00

6.00

7.00

8.00

9.00

10.00

Treatment Control

**Statistically significant at the 99 percent confidence level.

NOTES: Parent perceptions are based on a ten-point scale; student reports are based on an eight-point scale. For 

parent perceptions, valid N = 1,423; parent survey weights were used; the ten-point index of indicators 

of school safety and an orderly environment includes the absence of property destruction, tardiness, 

truancy, fighting, cheating, racial conflict, weapons, drug distribution, drug/alcohol use, and teacher 

absenteeism. For student reports, valid N = 1,098; student survey weights were used; the survey was 

given to students in grades 4-12; the means represent the absence of incidents on an eight-item index 

for student reports of students being a victim of theft, drug-dealing, assaults, threats, bullying or 

taunting, or had observed weapons at school. Means are regression adjusted using a consistent set of 

baseline covariates.
xvi

Figure 4. Parent and Student Reports of School Satisfaction

Year 3 Group Means: Percentage 

of Students Who Gave School 

Grade A or B

71% 73%

0%

20%

40%

60%

80%

100%

Treatment Control

Year 3 Group Means: Percentage 

of Parents Who Gave School 

Grade A or B

63%

74%**

0%

20%

40%

60%

80%

100%

Treatment Control

**Statistically significant at the 99 percent confidence level.

NOTES: For parent reports, valid N = 1,410; parent survey weights were used. For student reports, valid N = 

1,014; student survey weights were used; the survey was given to students in grades 4-12. Means are 

regression adjusted using a consistent set of baseline covariates.

• Overall, there were no impacts of the OSP from being offered (ES = -.06; figure 4) or 

using a scholarship (ES = -.07) on students’ satisfaction with their schools as measured 

by the likelihood of assigning their school a grade of “A” or “B.”

Impacts on Subgroups

In addition to determining the general impacts of the OSP on all study participants, this 

evaluation also reports programmatic impacts on policy-relevant subgroups of students. The subgroups 

were designated prior to data collection and include students who were attending SINI versus non-SINI 

schools at application, those relatively higher or lower performing at baseline, girls or boys, elementary 

versus high school students, and those from application cohort 1 or cohort 2. Since the subgroup analysis 

involves significance tests across multiple comparisons of treatment and control students, some of which 

may be statistically significant merely by chance, these subgroup-specific results should be interpreted 

with caution. Specifically:
xvii

Subgroup Achievement Impacts 

• There were no statistically significant reading (ES = .05) or math (ES = .01) 

achievement impacts for the high-priority subgroup of students who had attended a 

SINI public school under No Child Left Behind (NCLB) before applying to the 

Program. 

• There were statistically significant impacts on reading test scores in year 3 for five 

subgroups of students, although the statistical significance of two of the subgroup 

findings was not robust to adjustments for multiple comparisons: 

o Students who attended non-SINI public schools prior to application to the 

Program (56 percent of the impact sample) scored an average of 6.6 scale score 

points higher in reading (ES = .19) if they were offered the scholarship 

compared to not being offered a scholarship and 7.7 scale score points higher 

(ES = .22) if they used their scholarship compared to not being offered a 

scholarship. These scale score differences between the treatment and control 

groups translate into 4.1 and 4.9 additional months of learning, or half a year of 

schooling based on a typical 9-month school year.

o Students who entered the Program in the higher two-thirds of the test-score 

performance distribution at baseline (66 percent of the impact sample) scored an 

average of 5.5 scale score points higher in reading (ES = .17) if they were 

offered a scholarship and 6.2 scale score points higher (ES = .19) if they used 

their scholarship, impacts equivalent to 4.0 and 4.6 months of learning gains. 

o Female students scored an average of 5.1 scale score points higher in reading 

(ES = .15) if they were offered a scholarship and 5.8 scale score points higher 

(ES = .17) if they used their scholarship. These impacts represent 3.1 and 3.6 

months of additional learning, respectively. The statistical significance of this 

finding was not robust to adjustments for multiple comparisons. 

o Students who entered the Program in grades K-8 (81 percent of the impact 

sample) scored an average of 5.2 scale score points higher in reading (ES = .15) 

or 2.9 months of additional learning if they were offered a scholarship compared 

to not being offered a scholarship and 6.0 scale score points higher (ES = .17) or 

3.3 months of additional learning if they used their scholarship compared to not 

being offered a scholarship. 

o Students from the first cohort of applicants (21 percent of the impact sample) 

scored an average of 8.7 scale score points higher in reading (ES = .31) if they 

were offered a scholarship compared to not being offered a scholarship and 11.7 

scale score points higher (ES = .42) if they used their scholarship compared to 

not being offered a scholarship. These impacts translate into 14.1 and 18.9 

months of additional learning (1.5 to 2 years of typical schooling). The statistical 

significance of this finding was not robust to adjustments for multiple 

comparisons. 
xviii

• The OSP had no statistically significant reading impacts for other subgroups of 

participating students, including those in the lower third of the test-score performance 

distribution at baseline, boys, secondary students, and students from the second cohort 

of applicants (ES ranging from -.00 to .11).

• The OSP had no statistically significant math impacts for any of the 10 subgroups (ES 

ranging from -.16 to .23).

Subgroup Safety and Satisfaction Impacts

• All of the 10 subgroups analyzed, including parents of the high-priority subgroup of 

students who had attended SINI schools at baseline, reported viewing their child’s 

school as safer and more orderly if the child was offered or using an OSP scholarship 

compared to not being offered a scholarship. Effect sizes for the impact of an offer of a 

scholarship on parent perceptions of safety and an orderly school climate for the 10 

subgroups ranged from .27 to .40. Adjustments for multiple comparisons indicate that 

these 10 subgroup impacts on parental perceptions of safety and school climate are not 

likely to be false discoveries. 

• Consistent with the finding for students overall, none of the subgroups of students 

reported experiencing differences in safety and an orderly school climate if they were 

offered (ES range from -.03 to .08) or using an OSP scholarship. 

• In addition to an overall impact on parental satisfaction with their child’s school, the 

Program produced satisfaction impacts on 7 of the 10 subgroups analyzed. Effect sizes 

for the impact of an offer of a scholarship on the likelihood of a parent grading his/her 

child’s school “A” or “B” for these seven subgroups ranged from .16 to .41. 

Adjustments for multiple comparisons indicated that none of these parent satisfaction 

subgroup impacts may have been a false discovery. The parents of students who had 

attended SINI schools, parents of students in the lower one-third of the test score 

distribution, and parents of high school students generally did not report higher levels 

of school satisfaction that were statistically significant as a result of the treatment (ES

ranged from -.03 to .13). 

• There were no statistically significant differences between the treatment group and the 

control group for all 10 subgroups in the likelihood that students gave their school a 

grade of A or B (ES ranged from -.18 to .05). 

The Impact of the Program on Intermediate Outcomes

Understanding the mechanisms through which the OSP does or does not affect student 

outcomes requires examining the expectations, experiences, and educational environments made possible 

by Program participation. The analysis here estimates the impact of the Program on a set of “intermediate 

outcomes” that may be influenced by parents’ choice of whether to use an OSP scholarship and where to 

use it, but are not end outcomes themselves. The method used to estimate the impacts on intermediate 
xix

outcomes is identical to that used to estimate impacts on the key Program outcomes, such as academic 

achievement.

Prior to data analysis, possible intermediate outcomes of the OSP were selected based on 

existing research and theory regarding scholarship programs and educational achievement. Because 24 

intermediate outcome candidates were identified through this process, the variables were organized into 

four conceptual groups or clusters, as described below, to aid in the analysis. 

There is no way to rigorously evaluate the linkages between the intermediate outcomes and 

achievement―students are not randomly assigned to the experience of various educational conditions and 

programs. That is why any findings from this element of the study do not suggest that we have learned 

what specific factors “caused” any observed test score impacts, only that certain factors emerge from the 

analysis as possible candidates for mediating influence because the Program affected students’ experience 

of these factors. The analyses are exploratory, and, given the number of factors analyzed, some of the 

statistically significant findings may be “false discoveries” (due to chance). 

Overall, 3 years after applying to the Program, the offer of an Opportunity Scholarship

appears to have had an impact on 8 of the 24 intermediate outcomes examined, 7 of which remained 

statistically significant after adjustments for multiple comparisons:

• Home Educational Supports. Of the four intermediate outcomes in this category, the 

offer of a scholarship had an impact on one of them. There was a significant negative 

impact on tutor usage outside of school (ES = -.14), and this impact remained 

statistically significant after adjustments for multiple comparisons. There were no 

statistically significant differences between the treatment and control groups on 

parents’ reports of their involvement in school in year 3 (ES = -.11), parents' 

aspirations for how far in school their children would go (ES = .02), or time required 

for the student to get to school (odds ratio = 1.13).11

• Student Motivation and Engagement. Of the six intermediate outcomes in this 

category, the offer of a scholarship may have had an impact on one of them. Based on 

student surveys, the offer of a scholarship seems to have had a significant negative 

impact on whether students read for fun (ES = -.16). Adjustments for multiple 

comparisons, however, indicate that this result could be a false discovery, so it should 

be interpreted with caution. There were no statistically significant differences between 

the treatment and control groups in their reported aspirations for future schooling 

(ES = -.14), engagement in extracurricular activities (ES = .04), and frequency of 

doing homework (ES = .08), or in their parents’ reports of student attendance 

(odds ratio = 1.11) or tardiness rates (odds ratio = 1.19). 

 

11 The effect size for this categorical variable is expressed as an odds ratio, which describes the extent to which being in the treatment group 

increases (if above 1.0) or decreases (if below 1.0) the likelihood of giving a higher-category response.
xx

• Instructional Characteristics. The offer of a scholarship had a statistically significant 

impact on 5 of the 10 intermediate outcomes in this group of indicators. Students 

offered a scholarship experienced a lower likelihood that their school offered tutoring 

(ES = -.38), special programs for children who were English language learners 

(ES = -.61), or special programs for students with learning problems (ES = -.36) 

compared to control group students; these impacts remained statistically significant 

after adjustments for multiple comparisons. Students offered a scholarship experienced 

a higher likelihood that their school offered programs for advanced learners (ES = .27) 

and such enrichment programs as art, music, and foreign language (ES = .23); these 

two impact estimates also remained statistically significant after adjustments for 

multiple comparisons. There were no significant differences between the treatment and 

control groups in student/teacher ratio (ES = .01), how students rated their teacher’s 

attitude (ES = -.04), the school’s use of ability grouping (ES = .02), in-school tutor 

usage (ES = .04), or the availability of before- and after-school programs (ES = -.11).

• School Environment. The offer of a scholarship affected one of four measures of 

school environment. Students offered a scholarship experienced schools that were 

smaller by an average of 182 students (ES = -.29) than the schools attended by students 

in the control group; this impact remained statistically significant after adjustments for 

multiple comparisons. There were no statistically significant differences between the 

treatment and control groups, on average, in school reports of parent/school 

communication practices (ES = -.06), the percentage of minority students at the school 

(ES = -.10), or the classroom behavior of peers (ES = .09) based on student reports.

It is important to note that the findings regarding the impacts of the OSP reflect the 

particular Program elements that evolved from the law passed by Congress and the characteristics of 

students, families, and schools―public and private―that exist in the Nation’s capital. The same program 

implemented in another city could yield different results, and a scholarship program in Washington, DC, 

with different design features than the OSP might also produce different outcomes. 

 
xxi

References

Boruch, Robert, Dorothy de Moya, and Brooke Snyder. “The Importance of Randomized Field Trials in 

Education and Related Areas.” Evidence Matters: Randomized Trials in Education Research, 

Frederick Mosteller and Robert Boruch, editors. Washington, DC: The Brookings Institution 

Press, 2002.

Greene, Jay P. “Vouchers in Charlotte.” Education Matters 2001, 1(2): 55-60.

Hanushek, Eric A., John F. Kain, and Steven G. Rivkin. “Disruption Versus Tiebout Improvement: The 

Costs and Benefits of Switching Schools.” Journal of Public Economics 2004, 88: 1721-1746.

Howell, William G., Patrick J. Wolf, David E. Campbell, and Paul E. Peterson. “School Vouchers and 

Academic Performance: Results from Three Randomized Field Trials.” Journal of Policy 

Analysis and Management 2002, 21(2): 191-217.

Mayer, Daniel P., Paul E. Peterson, David E. Myers, Christina Clark Tuttle, and William G. Howell. 

School Choice in New York City After Three Years: An Evaluation of the School Choice 

Scholarships Program. MPR Reference No. 8404-045. Cambridge, MA: Mathematica Policy 

Research, 2002.

Wolf, Patrick, Babette Gutmann, Nada Eissa, Michael Puma, and Marsha Silverberg. Evaluation of the 

DC Opportunity Scholarship Program: First Year Report on Participation. U.S. Department of 

Education, National Center for Education Evaluation and Regional Assistance. Washington, DC: 

U.S. Government Printing Office, 2005. Available online at [http://ies.ed.gov/ncee/].

Wolf, Patrick, Babette Gutmann, Michael Puma, and Marsha Silverberg. Evaluation of the DC 

Opportunity Scholarship Program: Second Year Report on Participation. U.S. Department of 

Education, National Center for Education Evaluation and Regional Assistance, NCEE 2006-4003. 

Washington, DC: U.S. Government Printing Office, 2006. Available online at 

[http://ies.ed.gov/ncee/].

Wolf, Patrick, Babette Gutmann, Michael Puma, Lou Rizzo, Nada Eissa, and Marsha Silverberg. 

Evaluation of the DC Opportunity Scholarship Program: Impacts After One Year. U.S. 

Department of Education, National Center for Education Evaluation and Regional Assistance, 

NCEE 2007-4009. Washington, DC: U.S. Government Printing Office, 2007. Available online at 

[http://ies.ed.gov/ncee/].

Wolf, Patrick, Babette Gutmann, Michael Puma, Brian Kisida, Lou Rizzo, and Nada Eissa. Evaluation of 

the DC Opportunity Scholarship Program: Impacts After Two Years. (NCEE 2000-4023). U.S. 

Department of Education, National Center for Education Evaluation and Regional Assistance, 

NCEE 2008-4023. Washington, DC: U.S. Government Printing Office, 2008. Available online at 

[http://ies.ed.gov/ncee/].