Contract No.: K-5547-5-00-80-30 

MPR Reference No.: 8279-093 

The Quantum 

Opportunity Program 

Demonstration: 

Short-Term Impacts 

August 2003

Allen Schirm 

Nuria Rodriguez-Planas

Myles Maxfield 

Christina Tuttle 

Submitted to: 

U.S. Department of Labor 

Employment and Training Administration 

200 Constitution Avenue, N.W. 

Room N-5637 

Washington, DC 20210 

Project Officer: Eileen Pederson 

Submitted by: 

Mathematica Policy Research, Inc. 

600 Maryland Avenue, S.W. 

Suite 550 

Washington, DC 20024 

(202) 484-9220 

Project Director: Allen Schirm 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
iii 

D ISCLAIMER

This report was prepared under Contract No. K-5547-5-00-80-30 from the U.S. Department of 

Labor. The views expressed herein do not necessarily reflect the policies or opinions of the U.S. 

Department of Labor. 

A CKNOWLEDGMENTS

The authors would like to acknowledge the contributions of David Lah, Eileen Pederson, Daniel 

Ryan, and James Woods of the U.S. Department of Labor. Several members of the Mathematica Policy 

Research staff also contributed to this report. Susan Mitchell and Julie Young directed the data 

collection. Walter Corson, David Myers, Peter Schochet, and Michael Sinclair of MPR, as well as Alan 

Zaslavsky of Harvard University, provided valuable technical guidance and comments on draft sections 

of the report. Alfreda Holmes prepared the manuscript. Errors and omissions are the sole responsibility 

of the authors. 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
v 

CONTENTS 

Chapter Page 

DISCLAIMER.......................................................................................................................iii

 ACKNOWLEDGMENTS..................................................................................................iii

 EXECUTIVE SUMMARY .................................................................................................xi

I INTRODUCTION................................................................................................................1

Plan of the Report ..................................................................................................................2 

II WHAT IS QOP? ....................................................................................................................5

Target Population and Sample Selection ............................................................................5

Overview of Program Components and Services .............................................................6

What Type of a Youth Program Is QOP?........................................................................11

QOP Funding and Structure...............................................................................................13 

III IMPACT EVALUATION DESIGN AND METHODS.............................................15

Research Questions, Outcomes, and Data Sources ........................................................15

Obtaining an Evaluation Sample and Conducting Random Assignment....................18

Estimating Program Impacts ..............................................................................................19 

IV PARTICIPATION IN QOP .............................................................................................25

Participation in QOP ...........................................................................................................27

Profiles of Dedicated Enrollees and Disenchanted Enrollees.......................................30

Reasons for Low Participation ...........................................................................................32 

V SHORT-TERM IMPACTS ON HIGH SCHOOL COMPLETION, HIGH 

SCHOOL PERFORMANCE, AND POSTSECONDARY ACTIVITIES ..............35

High School Completion and Performance .....................................................................35

Postsecondary Activities......................................................................................................36 

VI SHORT-TERM IMPACTS ON RISKY BEHAVIORS AND RESILIENCY 

FACTORS.............................................................................................................................43

Risky Behaviors.....................................................................................................................44

Risk and Resiliency Factors.................................................................................................44 
CONTENTS (cont.)

Chapter Page 

vi 

VII SHORT-TERM IMPACTS ON SUBGROUPS.............................................................53

Short-Term Impacts by Sex ................................................................................................54

Short-Term Impacts by Age when Entering Ninth Grade ............................................54

Short-Term Impacts by Rank in the Baseline Grade Distribution ...............................55

Short-Term Impacts by Site................................................................................................56

Short-Term Impacts By Funding Source..........................................................................57 

 REFERENCES ....................................................................................................................67

APPENDIX A: OBTAINING AN EVALUATION SAMPLE AND 

CONDUCTING RANDOM ASSIGNMENT 

APPENDIX B: THE BASELINE DATA 

APPENDIX C: FOLLOW-UP DATA 

APPENDIX D: OUTCOMES AND SUBGROUPS 

APPENDIX E: WEIGHTING, IMPACT ESTIMATION, AND VARIANCE 

ESTIMATION 

APPENDIX F: SENSITIVITY ANALYSES 

APPENDIX G: SHORT-TERM IMPACTS ON PARTICIPANTS 
vii 

TABLES 

Table Page 

II.1 DEVELOPMENTAL ACTIVITIES OFFERED BY QOP 

DEMONSTRATION SITES ............................................................................................... 10

IV.1 PROFILES OF ENROLLEES ATTENDING MANY QOP ACTIVITIES 

AND ENROLLEES ATTENDING FEW QOP ACTIVITIES................................... 31

V.1 SHORT-TERM IMPACTS ON HIGH SCHOOL COMPLETION............................ 38

V.2 SHORT-TERM IMPACTS ON HIGH SCHOOL PERFORMANCE........................ 39

V.3 GROUP MEANS FOR POSTSECONDARY ACTIVITIES........................................ 40

V.4 SHORT-TERM IMPACTS ON POSTSECONDARY ACTIVITIES.......................... 41

VI.1 SHORT-TERM IMPACTS ON SUBSTANCE ABUSE, IN-PERSON 

SURVEY................................................................................................................................... 46

VI.2 SHORT-TERM IMPACTS ON SUBSTANCE ABUSE, TELEPHONE 

SURVEY................................................................................................................................... 46

VI.3 SHORT-TERM IMPACTS ON GANG ACTIVITY, CRIME, AND 

INVOLVEMENT WITH THE CRIMINAL JUSTICE SYSTEM, IN-PERSON 

SURVEY................................................................................................................................... 47

VI.4 SHORT-TERM IMPACTS ON CRIME AND INVOLVEMENT WITH THE 

CRIMINAL JUSTICE SYSTEM, TELEPHONE SURVEY.......................................... 47

VI.5 SHORT-TERM IMPACTS ON SEXUAL ACTIVITY................................................... 48

VI.6 SHORT-TERM IMPACTS ON HOW AN ADULT POSITIVELY 

INFLUENCED THE YOUTHíS LIFE............................................................................. 49

VI.7 SHORT-TERM IMPACTS ON PARTICIPATION IN SPECIAL 

PROGRAMS TO HELP STUDENTS ............................................................................... 50

VI.8 SHORT-TERM IMPACTS ON ATTITUDES TOWARD RISKY 

BEHAVIORS AND OUTLOOK ON LIFE .................................................................... 51

VII.1 SHORT-TERM IMPACTS BY SEX................................................................................... 59

VII.2 SHORT-TERM IMPACTS BY AGE WHEN ENTERING NINTH GRADE......... 60

VII.3 SHORT-TERM IMPACTS BY RANK IN THE BASELINE GRADE 

DISTRIBUTION.................................................................................................................... 61
TABLES(cont.)

Table Page 

viii 

VII.4 SHORT-TERM IMPACTS BY SITE.................................................................................. 62

VII.5 QOP-GROUP MEANS BY SITE....................................................................................... 63

VII.6 CONTROL-GROUP MEANS BY SITE........................................................................... 64

VII.7 SHORT-TERM IMPACTS ON RISKY BEHAVIORS BY SITE, 

TELEPHONE SURVEY ...................................................................................................... 65

VII.8 SHORT-TERM IMPACTS BY FUNDING SOURCE................................................... 66
ix 

FIGURES 

Figure Page 

IV.1 DISTRIBUTION OF HOURS BY PERIOD ................................................................... 28

IV.2 DISTRIBUTION OF EDUCATION HOURS BY PERIOD....................................... 28

IV.3 DISTRIBUTION OF DEVELOPMENT HOURS BY PERIOD................................ 29

IV.4 DISTRIBUTION OF COMMUNITY SERVICE HOURS BY PERIOD .................. 29

IV.5 REASONS FOR LIMITED PARTICIPATION BY DISENCHANTED 

ENROLLEES.......................................................................................................................... 33

IV.6 REASONS FOR STOPPING PARTICIPATION........................................................... 33

IV.7 REASONS FOR LIMITED PARTICIPATION BY ALL ENROLLEES 

WHO LIMITED THEIR PARTICIPATION .................................................................. 34
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
xi 

E XECUTIVE S UMMARY

From July 1995 through September 2001, the U.S. Department of Labor (DOL) and The Ford 

Foundation (Ford) operated a demonstration of the Quantum Opportunity Program (QOP). QOP 

offered intensive and comprehensive services to help at-risk youth graduate from high school and enroll 

in postsecondary education or training. The QOP demonstration included several features of Workforce 

Investment Act (WIA) youth programs, and findings from the demonstration might provide some 

insight about the implementation challenges that such WIA programs will encounter and the potential 

effectiveness of those programs. 

The QOP demonstration targeted youth with low grades entering high schools with high dropout 

rates. Randomly selected eligible youth were enrolled in QOP and served even if they transferred to 

other schools, dropped out of school, became incarcerated, or became inactive in QOP for a long time. 

QOPís primary goals were to increase the rates of high school graduation and enrollment in 

postsecondary education or training. Its secondary goals were to improve high school grades and 

achievement test scores and to reduce risky behaviors, such as substance abuse, crime, and teen 

parenting. 

QOP was mainly an after-school program providing case management and mentoring, 

supplemental education, developmental activities, community service activities, supportive services, and 

financial incentives. These services were provided year-round for five years to enrollees who had not 

graduated from high school, and were designed to be comprehensive enough to address all barriers to 

success and to be intensive. The program model specified roughly 15 to 25 enrollees per case manager, 

and it prescribed an annual participation goal of 750 hours for each enrollee who had not graduated. 

From graduation to the end of the demonstration, enrollees who had graduated received limited 

servicesósome mentoring and assistance with enrolling in postsecondary education or training. 

Community-based organizations (CBOs) in seven sites operated QOP demonstration programs. 

Five sitesóCleveland, Fort Worth, Houston, Memphis, and Washington, D.C.ówere funded by DOL. 

Four of the five served 100 youth each, and the Washington, D.C., site served 80 youth. The other two 

sitesóPhiladelphia and Yakimaóserved 50 youth each with funding from Ford. DOL has also funded 

an evaluation of the QOP demonstration. By the end of the demonstration, enrollees were in a variety 

of statuses, including attending college or another postsecondary training program, still attending high 

school, attending a general educational development (GED) certification program, working after 

finishing high school, and working or unemployed after dropping out of high school. 

The purpose of this report is to present the short-term impacts of QOP. To estimate impacts, we 

translated each program goal, such as high school graduation, into a quantifiable outcome, such as 

whether a youth graduated from high school. We measured each outcome for a group of youth enrolled 

in QOP and a group of statistically identical youth, called the control group. We formed the QOP 

group and the control group at the start of the demonstration by randomly assigning each youth eligible 

for the program to one group or the other. All members of the QOP group were enrolled in QOP. 
xii 

Members of the control group were not allowed to participate in QOP and, thus, show what would have 

happened to the enrollees had they not been enrolled. 

The impact of QOP on the enrolled youth is the difference between the average outcome for the 

group of QOP enrollees and the average outcome for the control group. The impacts are short-term 

impacts because we estimated them from data collected during the fourth and fifth years of the 

demonstration, that is, before the demonstration was over and when many youth were either still 

attending high school or had only recently graduated. Longer-term impacts, which may be a more 

appropriate basis for policy decisions, might be more or less beneficial than the short-term impacts 

presented in this report. To measure longer-term impacts, DOL is having us collect data in fall 2002 

and fall 2004. The fall 2002 data collection is roughly seven years after the youth in the demonstration 

sample entered the ninth grade and two years after the end of the QOP demonstration. 

The QOP Target Group and Program Model 

The target group in the QOP demonstration was youth entering the ninth grade in fall 1995 (1996 

in the Washington, D.C., site) who met the following criteria: 

• Began the ninth grade at a high school selected for the QOP demonstration. Each high 

school had a dropout rate of 40 percent or more. 

• Were not repeating the ninth grade. 

• Were not so physically disabled or learning disabled that participation in the program would 

not have been appropriate, as determined by the school. 

• Had a grade point average (GPA) below the 67th percentile among the students meeting the 

first three requirements at the participating high school. (The GPA was calculated from 

grades received in the eighth grade.) 

The QOP model consisted of four primary components: (1) case management and mentoring, (2) 

education, (3) developmental activities, and (4) community service. Secondary aspects of the program 

model included financial incentivesóstipends, accrual accounts, enrollee bonuses, and staff bonusesó

and supportive servicesósnacks, transportation assistance, and other services as needed. Compared to 

the models for most other youth programs, the QOP model required more intensive case management 

and mentoring in four ways: 

1. Enrollees were to have greater access to case managers and were to be involved in more 

program activities for longer periods of time. Each case manager was to have a caseload of 

approximately 15 to 25 enrollees. The QOP model set a target of 250 hours per year for 

activities in each of three service componentsóeducation, developmental activities, and 

community serviceófor a total of 750 hours per year until an enrollee graduated from high 

school. Enrollees who took full advantage of QOP received services for five years. Most 

case managers were available during off hours for enrollees to call in emergencies. 

2. Enrollees were to interact with case managers for longer periods of time because program 

eligibility was not contingent on enrollee behavior. Youth continued to be enrolled in QOP 

even if they transferred to another school, dropped out of school, became incarcerated, or 
xiii 

became inactive in QOP for a long time. In contrast to some other youth programs, QOP 

did not accept or retain only those youth who were sufficiently motivated to apply and 

actively participate. QOPís approach of enrolling all randomly selected eligible youth 

reflected the programís philosophy that the least-motivated youth might benefit the most 

from receiving help. 

3. Enrollees were to receive more comprehensive services because the scope of case 

management called for addressing all barriers that enrolled youth faced. Case managers 

either addressed a barrier directlyóby arranging transportation to program activities, for 

exampleóor referred the enrollee to another community resource, such as a substance 

abuse treatment program. 

4. Enrollees were to receive services throughout school vacations and the summer. Enrollees 

who failed a class during the school year were encouraged to attend summer school. Case 

managers assisted other enrollees who were age 16 or older to find summer jobs. 

Developmental and community service activities continued throughout the summer for all 

enrollees. 

Each of the other three components of the QOP model was geared toward achieving a specific 

program goal. 

• Educational activities were intended to improve academic achievement, increase the 

likelihood of completing high school, and increase the likelihood of going on to college or 

some other postsecondary training program. After an academic assessment, which formed 

the basis of an individual education plan, educational services were to consist of one-onone tutoring and computer-assisted instruction in specific coursework as well as in basic 

reading and mathematics. Educational services also included visiting nearby college 

campuses and other activities designed to promote awareness of and planning for college or 

other postsecondary training. 

• Developmental activities were designed to reduce risky behaviors. They also promoted 

cultural awareness and provided recreation. 

• Community service activities, such as visiting the residents of a local nursing home or 

volunteering at a local food bank, were designed to help youth develop a sense of 

responsibility for the quality of life of others in their neighborhood. 

The QOP model addressed numerous barriers to success by specifying that supportive services 

were to be provided either directly or indirectly through referrals to other resources in the community. 

QOP case managers referred enrollees to community health and mental health services; summer jobs 

programs; and local agencies that provide housing, food, income support, or child care. 

In addition to supportive services, QOP provided youth with three types of financial incentives to 

attend program activities. The first was a stipend of approximately $1.25 for every hour devoted to 

educational activities, developmental activities that were not purely recreational, and community service. 

A matching amount was deposited in an accrual account and promised to the enrollee when he or she 

earned a high school diploma or GED certificate and enrolled in college, a certified apprenticeship 
xiv 

program, an accredited vocational/technical training program, or the armed forces. Enrollees in some 

sites also received bonuses for completing major program activities. 

QOP also provided financial incentives to program staff. The two Ford-funded sites compensated 

staff entirely through incentive payments based on the time enrollees spent on program activities, while 

some DOL-funded sites provided bonuses to staff based at least partly on enrollee participation. 

Evaluation Methods 

At the start of the demonstration, we formed the QOP group and the control group by randomly 

assigning each of the nearly 1,100 youth eligible for the program to one group or the other. In the 

spring of the fourth academic year of the demonstration, we interviewed enrollees and control-group 

members in-person. The survey collected data on risky behaviors and factors that assist a youth in 

resisting negative influences in his or her social environment. At the same time, we administered 

achievement tests in reading and mathematics. Seven to ten months later, we conducted a telephone 

survey covering high school graduation, postsecondary activities, risky behaviors, and (for the enrollee 

group) attitudes toward QOP. Shortly thereafter, we requested transcripts from the high schools that 

sample members had attended since the beginning of the demonstration. From information provided 

by QOP staff, we measured how much enrollees participated in QOP. 

After conducting the two surveys, administering the achievement tests, and collecting transcripts, 

we measured the impact of QOP on an outcome by subtracting the mean outcome for the control 

group from the mean outcome for the QOP group. Because the available data were obtained before the 

end of the demonstration and when many youth were either still attending high school or had only 

recently graduated, the impacts estimated from those data and presented in this report should be 

interpreted as short-term impacts for many of the outcomes considered, as noted above. The data that 

we collect in 2002 and 2004 will reveal whether QOPís longer-term impacts are more or less favorable 

than its short-term impacts. 

Participation in QOP 

Despite QOPís goal of engaging a broad cross-section of eligible youth, most enrollees attended 

relatively few program activities. 

• Enrollees spent an average of 174 hours per year on QOP activitiesó23 percent of the 

annual goal of 750 hoursóthrough the first four years of the demonstration. 

• Enrollees spent an average of 72 hours per year on education (29 percent of the goal), 76 

hours on developmental activities (30 percent of the goal), and 26 hours on community 

service (11 percent of the goal). 

• The average time spent on QOP activities fell steadily from 247 hours in the first year of 

the demonstration to 89 hours in the fourth year, while the fraction of enrollees spending 

no time at all on QOP activities rose steadily from 1 percent to 36 percent. 

• The most dedicated enrolleesóthose spending at least 1,300 hours on QOP activitiesó

tended to have higher grades at baseline (eighth grade), be younger when entering the ninth 
xv 

grade, be in families receiving welfare, and be in families headed by a single parent. The 

most disenchanted enrolleesóthose spending 100 or fewer hours on QOP activitiesó

tended to have lower baseline grades, be male, not speak English at home, and be older 

when entering the ninth grade. 

• The most disenchanted enrollees reported being uninterested in QOP activities or having 

other after-school activities such as playing a sport, working, or caring for other family 

members. 

Short-Term Impacts of QOP 

Primary Outcomes: High School Completion and Postsecondary Education or 

Training 

• QOP increased by a statistically significant seven percentage points the likelihood 

that enrollees graduated from high school with a diploma. 

• QOP increased the likelihood of engaging in postsecondary education or training, 

although the size and statistical significance of the impact depends on how this 

outcome was measured and how the impact was estimated. 

- QOP significantly increased by six percentage points the likelihood of engaging in 

postsecondary education or training when education or training was defined to 

include college attendance, vocational or technical school attendance, apprenticeship 

enrollment, and armed forces enlistment. The impact became smaller and 

insignificant when this measure was either narrowed to include only college 

attendance or broadened to include employment. It also became smaller and 

insignificant when we used regression methods to adjust for random differences 

between the baseline characteristics of the QOP group and the control group. 

- When we included acceptance into collegeóin addition to current attendance at 

collegeóin the definition of postsecondary education or training, QOP significantly 

increased the likelihood of engaging in postsecondary education or training by six to 

nine percentage points for all but one measure of postsecondary activity. 

Secondary Outcomes: High School Performance, Risky Behaviors, and Resiliency 

Factors 

• QOP did not significantly improve enrollee performance while in high school. 

- QOP did not significantly raise reading or mathematics achievement test scores or 

high school grades. 

- QOP did not significantly increase the number of credits earned by enrollees or 

reduce disciplinary actions taken against enrollees in high school. 

• QOP did not significantly reduce risky behaviors.
xvi 

- QOP did not significantly reduce any risky behavior, including gang activity, crime, 

and teen parenting. 

- According to data from the in-person survey, QOP significantly increased by seven 

percentage points the fraction of enrollees who had a drink and the fraction who 

used an illegal drug in the 30 days before the survey. However, some evidence 

suggests that there were differences between QOP enrollees and control-group 

youth in the accuracy with which they reported risky behaviors. Those differences 

might have contributed substantially to the estimated detrimental impacts on 

drinking and drug use. That QOP might not have increased drinking and drug use 

is also suggested by data from the telephone survey. According to those data, QOP 

had beneficialóbut not significantóimpacts on drinking and drug use. 

• QOP significantly increased one resiliency factor.

- QOP significantly increased by 31 percentage points the fraction of enrollees 

reporting participation in a special program that helped them. Nevertheless, slightly 

less than half (47 percent) of QOP enrollees reported participating in ìspecial 

programs other than your normal high school classes Ö[that try] to help students 

stay in school, make good grades, stay away from drugs, prepare for work or college, 

and make good decisions in life.î This might reflect the fact that participation in 

QOP activities fell substantially short of the programís goal, especially by the fourth 

year of the demonstration when we asked the youth in the evaluation sample about 

their participation in special programs. 

- QOP did not significantly increase the likelihood that an enrollee perceived himself 

or herself as being positively influenced by a caring adult. It also did not 

significantly improve resiliency factors such as having an optimistic outlook on the 

future or believing that risky behaviors are wrong. 

Short-Term Impacts on Subgroups 

• QOP was more beneficial for enrollees in the middle of the eligible grade 

distribution than for enrollees at the top or bottom of the distribution. 

- QOP had several significant impacts on enrollees in the middle third of the eligible 

grade distribution, and all of those impacts were beneficial. They included a 14-

percentage-point increase in the likelihood of receiving a diploma, a 13-percentagepoint increase in the likelihood of college attendance or acceptance, and an 8-

percentage-point decrease in the likelihood of having a child. 

- QOP had both significant beneficial and detrimental impacts on enrollees in the 

bottom third of the distribution. It increased by 9 percentage points the likelihood 

of engaging in postsecondary education or training and decreased by 11 percentage 

points the likelihood of ever being arrested or charged with a crime. However, 

QOP also increased by 14 percentage points the likelihood of using an illegal drug. 

- QOP had only one significant impactóa detrimental impactóon enrollees in the 

top third of the distribution. It increased by eight percentage points the likelihood 

of binge drinking. 
xvii 

• QOP had significant beneficial impacts on both older and younger enrollees, and it did not 

consistently benefit one age group more than the other. (The older enrollees were over age 

14 when they entered the ninth grade, whereas the younger enrollees were age 14 or 

younger.) The impact on younger enrollees was significantly different from the impact on 

older enrollees for just one outcome. QOP decreased by nine percentage points the 

fraction of younger enrollees who had a child. This impact was significantly different from 

both zero and the (insignificant) six-percentage-point increase in the fraction of older 

enrollees who had a child. 

• Some of QOPís impacts on females and some of its impacts on males were significantly 

different from zero. Although the significant impacts were beneficial for females and 

detrimental for males, QOPís impact on females was significantly different from its impact 

on males for only one key outcome, the likelihood of engaging in postsecondary education 

or training, attending high school or a GED class, or working. 

Short-Term Impacts by Site 

• QOP’s impacts varied from site to site. And, only one of the seven sitesóthe 

Cleveland siteóhad significant beneficial impacts and no significant detrimental impacts. 

The Cleveland site significantly increased the likelihood of graduating from high school, 

significantly increased the likelihood of attending or being accepted by a college, and 

significantly decreased the likelihood of binge drinking. 

• The impacts for the whole QOP demonstration were substantially—but not 

entirely—attributable to the impacts of the Philadelphia site alone or the 

Philadelphia and Yakima sites, the Ford-funded sites, together. 

- The five DOL-funded sites together had one significant impactóthey increased by 

seven percentage points the likelihood that a QOP enrollee graduated from high 

school. This impact on one of QOPís primary outcomes was not significantly 

different from the impact for the two Ford-funded sites. 

- The Ford-funded sites had four significant beneficial impacts: a 2-percentile-point 

increase in the mathematics achievement test score, a 14-percentage-point increase 

in the likelihood of engaging in postsecondary education or training, a 17-

percentage-point increase in the likelihood of engaging in postsecondary education 

or training or working at a good job, and a 14-percentage-point decrease in the 

likelihood of having a child. 

- The Ford-funded sites also had three significant detrimental impacts: 17-, 14-, and 

16-percentage-point increases in the likelihood of engaging in binge drinking, using 

an illegal drug, and committing a crime, respectively. As discussed above, however, 

these detrimental impacts on risky behaviors might not have been attributable to 

QOP. 
xviii 

Conclusions 

• QOP achieved some short-term success in meeting its two primary goals of raising rates of 

high school completion and enrollment in postsecondary education or training. It had 

statistically significant beneficial impacts of modest size on at least some measures of both 

outcomes. 

• QOP was not successful in meeting its secondary goals of improving grades and 

achievement test scores and reducing risky behaviors. 

• QOP was not an effective resiliency factor. Although it significantly increased the fraction 

of enrollees participating in a program designed to help youth succeed in life, QOP did not 

improve enrolleeís optimism about life or attitudes toward risky behaviors, and it did not 

reduce their risky behaviors. 

• QOP was more beneficial in the short-run for enrollees in the middle of the eligible grade 

distribution than for enrollees at the top or bottom of the distribution. 

• QOPís impacts varied from site to site, and the impacts for the whole QOP demonstration 

were substantially, but not entirely, attributable to the impacts of the Philadelphia site alone 

or the Philadelphia and Yakima sites (the Ford-funded sites) together. The DOL-funded 

sites significantly increased the likelihood of graduating from high school, one of QOPís 

primary goals, but had no other statistically significant impacts. 

• Participation in QOP activities was substantially less than the program goal and declined 

steadily throughout the demonstration. 

As noted, the impacts presented in this report are short-term impacts that we estimated from data 

collected during the fourth and fifth years of the demonstration, that is, before the demonstration was 

over and when many youth were either still attending high school or had only recently graduated. 

Longer-term impacts, which may be a more appropriate basis for policy decisions, might be more or less 

favorable than the short-term impacts. To measure longer-term impacts, DOL is having us collect data 

in fall 2002 and fall 2004. The fall 2002 data collection is roughly seven years after the youth in the 

demonstration sample entered the ninth grade and two years after the end of the QOP demonstration. 
1 

C HAPTER I 

I NTRODUCTION

From July 1995 through September 2001, the U.S. Department of Labor (DOL) and The Ford 

Foundation (Ford) operated a demonstration of the Quantum Opportunity Program (QOP). QOP was 

mainly an after-school program providing at-risk high-school-age youth with intensive case management 

and mentoring, supplemental academic education, developmental activities, community service activities, 

supportive services, and financial incentives.1 

The QOP demonstration served a single cohort of youth from the beginning of the ninth grade in 

the fall of 1995 through the fall of 2000.2 A local community-based organization (CBO) in each of six 

inner-city communities and one rural community implemented and operated a QOP program. Each 

CBO teamed with from one to three high schools, and had 50, 80, or 100 youth enrolled in the 

program. By the end of the demonstration, enrollees were in a variety of statuses, including attending 

college or another postsecondary training program, still attending high school, attending a general 

educational development (GED) certification program, working after completing high school, and 

working or unemployed after dropping out of high school. 

 

The primary objectives of the demonstration were to increase the likelihood of high school 

graduation and to increase enrollment in postsecondary education or training. Its secondary objectives 

were to increase academic achievement while in high school and to reduce risky behaviors, such as 

substance abuse, crime, and teenage childbearing. 

The purpose of this report is to present the short-term impacts of the QOP demonstration. To 

obtain impact estimates, we randomly assigned each youth who was eligible for QOP to either a QOP 

group or a control group. Youth in the QOP group were enrolled in QOP and allowed to participate in 

program activities while youth in the control group were not. Thus, the youth in the control group 

show what would have happened to the QOP youth had the QOP youth not been enrolled in QOP. 

For each of many outcomes that pertain to the goals of QOP, such as high school graduation and 

engagement in postsecondary education or training, we estimated an impact of QOP by subtracting the 

average outcome for youth in the control group from the average outcome for youth in the QOP group. 

Impacts reported here are short-term because we estimated them from data collected before the 

demonstration was over and when many youth were either still attending high school or had only 

recently graduated. We conducted an in-person survey and administered reading and mathematics 

 1 At-risk youth are at a greater risk of substance abuse, criminal activity, teenage childbearing, not completing high 

school, or not enrolling in a postsecondary education or training program, compared to the average high-school-age youth in 

the United States. 

2 All events occurred one year later in the Washington, D.C., site. 
2 

achievement tests near the end of the fourth academic year of the demonstration. During the fifth 

academic year, we conducted a telephone survey and collected high school transcripts. While the data 

enabled us to measure impacts on, for example, substance abuse and crime while sample members were 

still of or very close to high school age, they do not allow us to measure impacts on substance abuse and 

crime after the youth became young adults. Similarly, some sample members who had just graduated 

from high school at the time of data collection may not yet have enrolled in college or another 

postsecondary education or training activity, whereas those who had enrolled at that time may have 

stopped their training after a few months. Furthermore, about 16 percent of the youth in the 

demonstration were still in high school when they were last interviewed; therefore, the data analyzed for 

this report do not indicate whether these youth successfully completed high school and then engaged in 

postsecondary education or training. 

To obtain longer-term impact estimates, DOL is having us conduct two additional telephone 

surveys in fall 2002 and fall 2004. The fall 2002 data collection is roughly seven years after the youth in 

the demonstration sample entered the ninth grade and two years after the end of the QOP 

demonstration. After all of the youth in the demonstration sample have left high schoolówith or 

without graduatingówe might find that QOPís impacts on high school completion and postsecondary 

activities, for example, are different from the impacts presented in this report. The longer-term impacts, 

which may be a more appropriate basis for policy decisions, might be higher or lower than the shortterm impacts. The importance of estimating longer-term impacts is illustrated by the National Job 

Corps Study, which obtained longer-term impacts that were substantially more favorable than the shortterm impacts (Schochet et al. 2000, 2001). 

PLAN OF THE REPORT

The report is organized as follows: Chapter II discusses the QOP program model, including how 

QOP fits into the spectrum of youth programs, the history of QOP, and the organizational structure of 

the demonstration. Chapter III summarizes the methods used to estimate program impacts. The 

methods are presented in detail in the several appendices listed below. Chapter IV describes the degree 

to which enrolled youth participated in QOP activities. 

Chapters V though VII present the short-term impacts of the QOP demonstration on each of 

several sets of outcomes. Chapter V covers high school completion and postsecondary activities; 

Chapter VI examines risky behaviors and resiliency factors; and Chapter VII presents impacts on 

subgroups of enrollees as well as impacts by demonstration site. 

The report concludes with a set of appendices that provide comprehensive documentation of our 

methods. The documentation is designed to enable other analysts to replicate our findings. 

• Appendix A describes how we obtained a sample of youth for the evaluation and how we 

conducted random assignment. 
3 

• Appendix B documents the baseline database. Given that DOL elected not to fund a 

baseline survey, baseline characteristics are taken from eligibility data and from later surveys 

for those demographic characteristics that do not change. 

• Appendix C presents the survey questionnaires, achievement tests, and school recordscollection protocol as well as the fielding procedures and response rates for each datacollection activity. 

• Appendix D defines each outcome variable and the subgroups for which impacts are 

estimated. 

• Appendix E presents the technical details pertaining to how we computed weights for the 

QOP demonstration sample, estimated difference-of-means impacts, and estimated the 

variances of those impacts. 

• Appendix F presents our findings from assessing the sensitivity of the impact estimates to 

alternative estimation approaches. 

• Appendix G presents QOPís impacts on participants, the subgroup of enrollees who 

attended some QOP activities. 

A companion report (Maxfield et al. 2003a) documents how well the QOP program model, 

described in Chapter II, was implemented in each of the seven demonstration sites and, thus, provides a 

context for interpreting the program impacts presented in this report. Another companion report 

(Maxfield et al. 2003b) summarizes our findings pertaining to program implementation and short-term 

impacts. 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
5 

C HAPTER II 

W HAT I S QOP? 

The description of the QOP model in this chapter covers the programís target population, the 

components of the program model, a brief comparison of QOP to other youth programs, a short 

history of the program, and the institutional structure of the demonstration. 

TARGET POPULATION AND SAMPLE SELECTION

The target group in the QOP demonstration was youth entering the ninth grade in fall 1995 who 

met the following criteria: 

• Began the ninth grade at a high school selected for the QOP demonstration. Each high 

school had a dropout rate of 40 percent or more. (The dropout rate was defined as the 

proportion of students entering ninth grade who had not earned a diploma or a GED four 

years later.) 

• Were not repeating the ninth grade. 

• Were not so physically disabled or learning disabled that participation in the program would 

not be appropriate, as determined by the school. 

• Had a grade point average (GPA) below the 67th percentile among the students meeting the 

first three requirements. (The GPA was calculated from grades received in the eighth 

grade.) 

A sample of youth was drawn from a list of students meeting the above criteria in each school 

participating in the program.3 Sampling was necessary because, in most sites, the number of eligible 

youth exceeded the number of slots available for the demonstration evaluation, which was the number 

of program slots plus a roughly equal number of control-group slots. Sampled youth were aggressively 

recruited to participate in the study, and about 98 percent of those who were located agreed to 

participate.4 Youth who agreed to participate were randomly assigned to a QOP group or to a control 

group. The high recruitment rate meant that the youth in the evaluation were representative of all youth 

meeting the eligibility criteria, not simply those who were motivated, had strong self-esteem, and were 

 3 In two schools, we selected all eligible youth for the evaluation. 

4 Staff could not locate seven percent of sampled eligible youth, reporting that most of these individuals did not really 

attend the QOP schools and were erroneously on school enrollment lists. 
6 

optimistic about the future―characteristics likely to be associated with seeking out and volunteering for 

a program such as QOP. 

The fact that youth in the evaluation are representative of all eligible youthórather than only the 

most motivated youthóis an important feature of the demonstration. One of the guiding principles of 

QOP is that youth with low-self esteem, little motivation, and a pessimistic outlook may be most in 

need of the program. But these youth are least likely to make an effort to enroll in a program like QOP. 

 Had QOP recruiting efforts depended only on the youth themselves to take the initiative to seek out 

the program office and apply, the program caseload would have consisted of youth who were motivated 

to improve their lives. This type of recruiting is common among other youth programs but contrary to 

the ideology of QOP. 

Most of the youth in the QOP demonstration were 13 to 15 years old when the demonstration 

began and 18 to 20 years old when it ended. The preponderance of youth were members of low-income 

families and were black, Hispanic, or both. The group included youth who, at some point during the 

demonstration period, were in special education programs, had disabilities, were teen parents, were 

involved in the juvenile justice system, did not attend high school, were one or more grades behind in 

basic skill levels, or were undocumented residents. 

OVERVIEW OF PROGRAM COMPONENTS AND SERVICES

The QOP program model consisted of four primary components: (1) case management and 

mentoring, (2) education, (3) developmental activities, and (4) community service. Secondary aspects of 

the program model included financial incentivesóstipends, accrual accounts, enrollee bonuses, and staff 

bonusesóand supportive servicesósnacks, transportation assistance, and other services as needed. 

This section describes each component of the program model. A description of how well the model 

was implemented at each of the demonstrations sites may be found in Maxfield et al. (2003). As we 

learned from annual site visits, annual QOP conferences, and conference calls with QOP staff, two sites 

implemented a version of QOP that deviated substantially from the program model, and the other five 

sites implemented versions that deviated moderately from the model. 

Intensive Case Management and Mentoring 

Intensive case management and mentoring was the central program component. Case management 

included assessing the needs of each enrolled youth and structuring a service mix appropriate to meeting 

those needs. With mentoring, a case manager was to establish a personal relationship with each youth. 

QOP regarded case management and mentoring as inseparable activities to be undertaken by one 

person. 

Compared to the models for most other youth programs, the QOP model required more intensive 

case management and mentoring in four ways: 

1. Enrollees were to have greater access to case managers and were to be involved in more 

program activities. 

2. Enrollees were to interact with case managers for longer periods of time because program 

eligibility was not contingent on enrollee behavior. 
7 

3. Enrollees were to receive more comprehensive services because the scope of case 

management called for addressing all barriers facing enrolled youth. 

4. Enrollees were to receive services throughout school vacations and the summer. 

Each of these is discussed below. 

Greater Access to Case Managers and More Involvement in Program Activities Were 

Provided to Enrollees. QOP CBOs employed a case manager for approximately every 15 to 25 

enrollees. Case managers were generally available to enrollees every weekday, and many were accessible 

on one weekend day. Most case managers were also available to their enrollees by telephone or pager 

during off hours to respond to urgent situations confronting their youth. 

Absent staff turnover and the rare case of voluntary reassignment of an enrollee to a different case 

manager, each enrollee remained with the same case manager for the first four years of the 

demonstration. During the final year, services were reduced and focused on those youth who had not 

yet completed high school.5 In most instances, case managersí caseloads changed from the fourth to the 

fifth year of the demonstration. 

Enrolled youth were expected to spend large amounts of time engaged in QOP activities. The 

program set a participation goal for each enrolled youth at 750 hours per year until the enrollee 

graduated from high school, or more than 14 hours per week on average throughout the year. Onethird of that time was to be spent on educational activities, including tutoring and computer-assisted 

instruction; one-third on community service; and one-third on developmental activities, including 

decision-making skills, cultural activities such as visiting museums, and lifestyle-related activities such as 

family planning seminars. In most sites, a typical schedule included meetings with case managers, most 

of whom were available at the school during the school day; program activities at the CBOís facility 

from 3 p.m. to 6 p.m. each weekday; and program activities for half a day on most weekends. 

It is important to emphasize that the target of 750 hours per year overall was a feature of the 

program model for the QOP demonstration. As we report in Chapter IV, the average enrollee spent 23 

percent of the target number of hours on QOP activities. 

Continuing Program Eligibility Was Not Contingent on Enrollee Behavior or Life 

Circumstances. Unlike continuing eligibility for most programs of its type, continuing eligibility for 

QOP was not contingent on the youthís behavior, residence, or health status. Other youth programs 

limit continuing eligibility to youth living in a specific neighborhood, to in-school youth or out-ofschool youth, to youth in or not in the criminal justice system, or to youth who complete some 

minimum number of program activities.6 In contrast, QOP continued to serve youth in a wide variety 

of circumstances, including those who: 

• Dropped out of school. Program staff attempted to get dropouts to re-enroll in school 

or, failing that, to earn a GED. 

 5 Enrollees who had graduated from high school received some mentoring and assistance in enrolling in postsecondary 

education or training between graduation and the end of the fifth year of the demonstration. 

6 Where QOP fits within the broad spectrum of youth programs is presented in the next section. 
8 

• Moved to a different school or neighborhood. Case managers continued to provide 

services to those who moved to another neighborhood in the metropolitan area, to make 

one-on-one visits to those who moved outside the metropolitan area but remained within 

driving distance, and to call those who moved beyond driving distance. However, the 

farther away an enrollee moved, the less intense services became. 

• Became incarcerated. Case managers visited and called incarcerated enrollees and sent 

them educational materials that they could complete. 

• Became ill or disabled. Case managers visited and called enrollees who were hospitalized 

or confined to their homes because of illness or disability. Case managers also adapted 

program activities to enable such enrollees to particpate. 

• Became inactive. Once a youth was enrolled in QOP, a program slot was held for that 

person for the full five years of the demonstration regardless of how much time he or she 

spent in program activities. For example, some enrollees were ìactiveî in the 9th grade 

(spent a large amount of time in program activities) but became ìinactiveî in the 10th and 

11th grades (did not participate in program activities). Case managers continually tried to 

maintain contact with inactive enrollees and to re-engage them in the program. 

Consequently, some youth who became inactive became active again in subsequent years. 

The dual policy of unconditional eligibility and continuing enrollment was summarized by the 

program motto, ìOnce in QOP, Always in QOP.î This motto was based on the philosophy that the 

need for mentoring may not diminish and indeed may increase when an enrollee drops out of school, 

moves, or becomes incarcerated, disabled, ill, or disenchanted with the program. This philosophy also 

reflects the fact that the lives of many disadvantaged youth pass through several of these states during 

their high school years. For example, an enrollee might be in school and actively participating in the 

program in the ninth grade, then drop out of school and become inactive in the program, move several 

times, become incarcerated, and finally become active in the program again at the age of 18 or 19 as he 

or she works toward a GED certificate. Throughout, the case manager was expected to maintain, or 

attempt to maintain, a stabilizing presence in the youthís life. 

Case Management and Mentoring Were to Address All Barriers. QOP case management and 

mentoring was more intense than in many other youth programs because of the depth and breadth of 

the youthsí relationships with their case manager. The relationships were intended to be personal, longterm, and comprehensive. While many QOP activities were conducted in a group setting, case managers 

spent one-on-one time with every enrollee whom they could locate. Case managers were expected to 

help enrollees overcome a broad range of barriers to achieving the program goals. Common barriers 

included low educational achievement, alienation from school, substance abuse, physical and mental 

health problems, gang membership, criminal activities, teen parenthood, an unstable or abusive family 

environment, and insufficient funds to pay for necessities. For some enrollees, these barriers were 

compounded by the fact that their parents or guardians faced similar problems. 

Services Were to Be Provided Throughout School Vacations and the Summer. Case 

managers encouraged and made arrangements for enrollees who failed a class during the school year to 

attend summer school. Case managers assisted other enrollees who were age 16 or older to find 

summer jobs. Developmental and community service activities continued throughout the summer for 

all enrollees. 
9 

Educational, Developmental, and Community Service Activities 

In addition to intensive case management and mentoring, QOP offered educational, 

developmental, and community service activities. Each of these components was geared toward 

achieving a specific program goal. 

• Educational activities were intended to improve academic achievement, increase the 

likelihood of completing high school, and increase the likelihood of going on to college or 

some other postsecondary training program. After an academic assessment, which formed 

the basis of an individualized education plan, educational services were to consist of oneon-one tutoring and computer-assisted instruction in specific coursework as well as in basic 

reading and mathematics. Educational services also included visiting nearby college 

campuses and other activities designed to promote awareness of and planning for college or 

other postsecondary training. 

• Developmental activities were designed to reduce risky behaviors. They also promoted 

cultural awareness and provided recreation. A list of developmental activities is presented 

in Table II.1. 

• Community service activities, such as visiting the residents of a local nursing home or 

volunteering at a local food bank, were designed to help youth develop a sense of 

responsibility for the quality of life of others in their neighborhood. 

Supportive Services and Financial Incentives 

The QOP model addressed numerous barriers to success by specifying that supportive services 

were to be provided either directly or indirectly through referrals to other resources in the community. 

Most QOP sites provided afternoon snacks and transportation to program activities. QOP case 

managers referred enrollees to community health and mental health services; summer jobs programs; 

and local agencies that provide housing, food, or income support. 

In addition to supportive services, QOP provided youth with three types of financial incentives to 

participate in program activities. The first was a stipend of approximately $1.25 for every hour devoted 

to nonrecreational program activities, such as time spent being tutored and time spent on community 

service activities. A matching amount was deposited in an accrual account and promised to the enrollee 

when he or she earned a high school diploma or GED certificate and enrolled in college, an 

apprenticeship program certified by DOL, an accredited vocational/technical training program, or the 

armed forces. The purpose of the accrual accounts was to provide financial support for college or other 

postsecondary training and to teach enrollees about planning, saving, and investing for the future. By 

the end of the demonstration, accrual account balances ranged from a few hundred dollars to nearly 

$10,000, with most being in the range of $1,000 to $3,000. Final payments were made directly to the 

enrollee rather than to the postsecondary institution or to the enrolleeís parents.7 Bonuses awarded by 

some sites for the completion of major program activities were the third type of financial incentive 

provided to enrollees. 

 7 While accrued funds were usually paid directly to the enrollee, case managers, with written support from DOL, 

retained the option to make exceptions to this procedure if they felt direct payment would not be appropriate. 
10 

TABLE II.1 

DEVELOPMENTAL ACTIVITIES OFFERED BY QOP DEMONSTRATION SITES 

Lifeskills 

activities/discussion 

topics 

Public/Private Ventures life skills 

curriculum 

Contraception, family planning, and 

abstinence 

Budgeting 

Money management 

Financial planning 

Menu planning/grocery shopping 

Business ownership 

Personal hygiene 

Nutrition 

Overcoming adversity 

Conflict resolution training 

Managing anger 

Avoiding drug abuse 

Gang prevention 

CPR training 

Peer mediation training 

Behavioral skills 

Self-esteem 

Sexual harassment 

Sexual abuse 

Dating behavior and decision making 

Date rape 

Male parenting roles 

Importance of education 

Current magazine reading assignments 

Prison tours 

Decision-making/problem-solving activities 

Pre-employment training World-of-work basic skills 

Telephone etiquette 

Mock interviews 

Resume writing workshops 

On-site corporate tours 

Career exposure through guest speakers 

Summer placements in: 

Hospitals, nursing homes 

Federal, state, and county offices (e.g., health 

department, park service) 

Day care centers 

Local schools 

Restaurants 

Grocery stores 

Social service agencies (Goodwill, United 

Way) 

Cultural activities Museums 

Theater 

Ballet 

Symphonies, concerts 

Civic events 

Zoo 

State capitol tours 

Music and dance lessons 

Public lectures 

History videos 

Arts and crafts projects 

Fund raisers 

Workshops on topics such as African 

heritage, AIDS awareness, volunteerism, 

civic participation, entrepreneurship 

Classes in cooking, photography, arts and 

crafts 

Recreational activities Movies 

Ice skating 

Bowling 

Swimming 

Sailing 

Golfing 

Mountain biking 

Amusement/water parks 

Haunted houses 

Board/computer games 

Local fairs 

Picnics 

Attending sporting events 

Pizza lunches 

Restaurant dinners 

SOURCE: Annual site visits. 
11 

QOP also provided financial incentives to program staff. The two Ford-funded sites compensated 

staff entirely through incentive payments based on the time enrollees spent on program activities, while 

some DOL-funded sites provided bonuses to staff based at least partly on enrollee participation. 

WHAT TYPE OF A YOUTH PROGRAM IS QOP? 

Understanding QOP requires understanding how it compares with other types of youth programs. 

Youth programs are broadly defined here as public programs intended to solve a social problem 

experienced by substantial numbers of individuals from age 12 through 21. This definition excludes 

programs not open to the public, programs targeting other age groups, programs designed to address 

problems not considered to be social, and programs targeting small numbers of individuals. It also 

excludes defunct federal youth programs, such as the Civilian Conservation Corps of the late 1930s. 

Of the many programs that would fall within our broad definition, we focus on programs 

sponsored by the federal government that are designed to help at-risk youth. For fiscal year 1998, the 

U.S. General Accounting Office counted 117 federal youth programs with collective appropriations of 

more than $4 billion. Among these programs, which were sponsored by 15 federal departments and 

agencies, 45 provided mentoring and 35 provided academic tutoring (U.S. General Accounting Office 

1999). 

QOP can be placed in the spectrum of other youth programs in terms of the social ìunitî it treats, 

its goals and service mix, its target population, the relationship between case managers and enrollees, 

and its cost. The social unit that QOP treated was the youth, as opposed to other relevant social units 

such as the school, the family, the labor market, and the community. The QOP designers focused on 

youth because they interpreted the problem of poverty and career failure from the human capital 

perspective. The human capital perspective comes from the field of economics and assumes that if the 

skills of a youth are improved, he or she will be more employable and better able to avoid poverty. 

Thus, QOP can be viewed in terms of the social units that it does not treat: 

• QOP was not a school-reform program, in which the treated unit is the school, school 

district, or state education agency. Although QOP provided tutoring and computerassisted instruction, it was not designed to influence the structure, policies, or operation of 

the high schools with which local QOP programs were associated. 

• QOP was not a family counseling or therapy program. While case managers sought to 

involve the parents of enrollees and communicated regularly with many of them, QOP was 

not designed to address the problem of poverty and career failure by providing counseling 

or therapy to troubled families. 

• QOP did not focus on employers. QOP addressed the supply side of the labor market, 

that is, the skills that enrollees bring to the labor market as young adults. It did not address 

the demand side of the labor market, as does the Work Opportunity Tax Credit program 

by, for example, offering tax incentives for employers to hire disadvantaged persons. 
12 

• QOP was not a community development program. Although most QOP sites were 

located within high-poverty communities, QOP was not intended to attract new businesses 

to the community or to address other social problems on a community-wide basis. 

QOP also differed from other programs of its type in its goals and service mix. QOP focused more 

sharply on educational outcomes and provided more education services than do most other 

DOL/Employment and Training Administration-sponsored youth programs, which tend to focus on 

employment outcomes. QOP provided basic education skills as opposed to vocational training, workreadiness training, job search assistance, job development, or direct placement. Under QOP, getting a 

job right after graduation from high school was not an objective. Instead, QOP was premised on the 

belief that postsecondary education or training is required for long-term economic self-sufficiency in the 

modern labor market. 

QOP was also more comprehensive than most other federal youth programs. It provided services 

related directly and indirectly to academic skills; college planning and applications (including financial 

aid); physical and mental health; substance abuse; conflict resolution; family planning; cultural and ethnic 

awareness; career awareness and planning; issues related to gang membership and involvement in the 

criminal justice system; coping with dysfunctional, abusive, or unsupportive family environments; 

summer jobs; transportation; nutrition; and paying bills in family emergencies. 

In addition to its scope of services, QOP differed from other youth programs in the scope of its 

target population, which was not limited to highly motivated eligible youth. While no youth program is 

legally mandatory, QOP enrolled all randomly selected eligible youth. Therefore, QOP enrolled many 

at-risk youth who would not have ended up in such a program had the recruitment procedures been 

more passive and required an expression of interest in the program from eligible youth, as in many 

youth programs. 

QOP differed from other programs of its type in terms of the relationship between case managers 

and participating youth. This highly personal, long-lasting connection mirrored the relationship between 

a teenager and a nurturing, supportive older relative such as an aunt, uncle, or grandparent. Enrollees 

could confide in their case managers more freely than they might confide in a parent, and case managers 

could provide guidance on how to handle a situation without risking the traditional parent-teen conflict 

that often works against the acceptance of such guidance. And like an older relative who has made a 

commitment to a child, case managers made every attempt to sustain the relationship with youth despite 

resistance and distance. Case managers acted as advocates by negotiating on behalf of youth with the 

high school and with criminal justice and other public agencies. 

Finally, QOP was more intensive and expensive than most other federal youth programs. QOP 

provided a case manager for roughly every 15 to 25 enrollees, provided services to each enrollee for five 

years, set a goal of 750 hours of participation per year until graduation, and cost about $25,000 per 

enrollee, on average. 
13 

QOP FUNDING AND STRUCTURE

Under the pilot and demonstration authority of the Job Training Partnership Act (JTPA)8, DOL 

funded QOP operations in five sites through grants to local public service delivery organizations. For 

each of the first four years of the demonstration, each site received a grant of $200,000 and was obliged 

to provide local matching funds of an equal amount, for a total budget of $400,000 per year.9 In the 

fifth year, each DOL-funded site received a grant of $200,000 but no local matching funds. 

Through a grant to Opportunities Industrialization Centers of America (OICA), The Ford 

Foundation funded program operations in two demonstration sites as well as technical assistance for 

program operations in all seven sites. While there was no formal contractual arrangement between 

DOL and Ford, the two organizations coordinated their activities throughout the demonstration. DOL 

also funded the evaluation of all seven sites in the demonstration through a contract with Mathematica 

Policy Research, Inc. (MPR). 

The DOL sites were located in Cleveland, Ohio; Fort Worth, Texas; Houston, Texas; Memphis, 

Tennessee; and Washington, D.C. Each DOL grantee was the public agency that administered JTPA 

programs in that area. That local public agency was known under JTPA as the service delivery area 

(SDA). SDAs did not operate QOP directly; instead, each SDA contracted with a single local CBO. 

Typically, an SDA passed on 80 to 90 percent of the QOP grant to the CBO to reimburse the CBO for 

program administration and operating expenses. Each CBO in a DOL-funded site hired a QOP 

coordinator and five case managers to serve 100 enrollees.10

The two Ford-funded sites were located in Philadelphia, Pennsylvania, and Yakima, Washington. 

The structure of the two Ford-funded sites differed in several ways from that of the five DOL-funded 

sites. OICA operated QOP in Philadelphia directly and had an informal agreement with its local OIC 

affiliate in Yakima to oversee operations there. The relationship between the Yakima affiliate and the 

Philadelphia OICA was the same as the relationship between a franchise and its national headquarters. 

Each Ford-funded site had 50 enrollees, half the number in each DOL-funded site (except Washington, 

D.C.), and each had three case managers who also had responsibilities for programs other than QOP. 

Finally, while case managers at DOL-funded sites received a salary, those at Ford-funded sites were 

compensated through incentive payments based on the time that enrollees spent on program activities. 

As noted above, each CBO teamed with between one and three high schools, which participated in 

QOP in several ways. First, they provided the population of eligible youth from which the evaluation 

sample was selected. Second, they gave case managers access to enrolleesí teachers for monitoring 

enrolleesí academic performance. Third, with the informed consent of enrolleesí parents, the schools 

gave case managers access to enrolleesí school records. Finally, some schools provided office space for 

QOP case managers. Thus, case managers at some sites provided services during school hours in an 

office on school grounds. Case managers at other sites spent time in the school during school hours but 

did not have an office there. At still other sites, case managers had no significant in-school presence. 

For after-school activities in four sites, case managers and enrollees met at the CBOís facility. At three 

 8 JTPA expired in 2000, replaced by the Workforce Investment Act of 1998 (WIA). 

9 The CBO at the Houston site operated its QOP program with more limited local matching funds for the third and 

fourth years of the demonstration. 

10 The Washington, D.C., site had 4 case managers and served 80 enrollees. 
14 

sites, case managers met with enrollees for after-school activities in the schools themselves. Participating 

schools did not enter into a contractual arrangement with either the SDA or the CBO and were not 

reimbursed from the QOP grant. 
15 

C HAPTER III 

I MPACT E VALUATION D ESIGN AND M ETHODS

In principle, the goal of a program evaluation is to measure a programís effectiveness in achieving 

its policy objectives. In practice, effectiveness is measured as the impact of the program on outcomes 

that quantify the policy objectives. Accordingly, the purpose of the QOP impact evaluation is to 

measure the impact of QOP on outcomes such as high school graduation and engagement in 

postsecondary education or training. An impact is the causal effect of QOP enrollment, that is, what 

happens to a youth solely because he or she enrolled in QOP. In other words, it is the difference 

between what happens to the enrollee and what would have happened had the youth not enrolled in the 

program. 

The obvious problem in measuring this difference was that during a youthís high school years, the 

youth either did or did not enroll in QOP. Therefore, we had to design a procedure that allowed us to 

infer, rather than directly measure, what would have happened to a QOP enrollee had he or she not 

enrolled in the program. The procedure used for the QOP impact evaluation was random assignment. 

Under random assignment, each eligible youth was assigned at random to either a QOP group or a 

control group. Youth in the QOP group were enrolled in QOP and allowed to participate in program 

activities while youth in the control group were not. 

Random assignment ensured that the QOP and control groups were statistically equivalent. In 

practical terms, statistical equivalence means that at the time of random assignment, the only differences 

in measured or unmeasured characteristics between the two groups were purely random (and typically 

small). Therefore, subsequent differences were attributable to QOP. Comparing the average outcome 

for enrollees to the average outcome for the control group measured the impact of QOP on the 

outcome under consideration. Because the groups were statistically equivalent, the youth in the control 

group revealed what would have happened to enrollees had they not been enrolled in QOP. 

In the remainder of this chapter, we describe in further detail the design and methods of the impact 

evaluation. We discuss research questions, outcomes, and data sources in the next section. In the 

following section, we describe how we obtained an evaluation sample and randomly assigned youth to a 

QOP group and a control group. In the third section, we describe how we estimated the impacts of 

QOP. 

RESEARCH QUESTIONS, OUTCOMES, AND DATA SOURCES

The impact evaluation of the QOP demonstration was designed to address the following research 

questions: 
16 

• Does QOP increase the likelihood that enrollees graduate from high school? 

• Does QOP increase the likelihood that enrollees engage in postsecondary education or 

training? 

• Does QOP increase the likelihood that enrollees engage in some gainful activity, which can 

include employment and attending high school as well as engaging in postsecondary 

education or training? 

• Does QOP improve the high school performance as well as the graduation rate of 

enrollees? 

• Does QOP decrease the likelihood that enrollees engage in risky behaviors, such as 

substance abuse, crime, and teenage childbearing? 

• Does QOP increase the likelihood that there will be resiliency factors that might protect an 

enrollee from negative influences in the enrolleeís social environment? 

• Does the effectiveness of QOP vary across subgroups defined by enrolleesí characteristics? 

• Does the effectiveness of QOP vary across program sites? 

The first two questions correspond to the two main objectives of the QOP demonstration. To 

measure engagement in postsecondary education or training, the evaluation used the same four activities 

that QOP used to determine whether a QOP youth qualified to receive the funds from his or her 

accrual account. These activities were (1) enrolling in an accredited two- or four-year college or 

university, (2) enrolling in an accredited vocational or technical school, (3) enrolling in a certified 

apprenticeship program, and (4) enlisting in the armed forces. 

To answer the research questions, we developed a list of outcomes and defined several subgroups 

of sample members. The subgroups are described later in this chapter, and a complete list of outcomes 

is presented in Appendix D. The outcomes are grouped into five broad categories: 

• High school completion. The outcomes in this category measure receipt of a high school 

diploma or receipt of a GED certificate. They also measure whether a youth was still 

attending high school. 

• Postsecondary activity. The outcomes in this category measure engagement in 

postsecondary education or training and employment. 

• High school performance. The outcomes in this category include achievement test 

scores, grade point average, credits earned, and suspensions or expulsions from high 

school. 

• Risky behaviors. The outcomes in this category measure substance abuse, including the 

consumption of alcohol and illegal drugs; gang activity; criminal activity; involvement with 

the criminal justice system; sexual activity; and childbearing. 
17 

• Resiliency factors. The outcomes in this category measure whether there are factors in a 

youthís social environment that increase the likelihood that the youth will achieve the goals 

of QOP. The presence of a caring adult mentor is an example of a factor that might 

improve a youthís resiliency to negative influences. 

We obtained data on these outcomes from four sources: 

• An in-person survey. The survey, administered mainly during the spring of the fourth 

year of the demonstration, focused on risky behaviors and resiliency factors. About 59 

percent of respondents completed the survey in group sessions while the rest completed 

the survey during a one-on-one visit by a member of the data-collection staff. The 

response rates were 88 percent for QOP-group youth, 80 percent for control-group youth, 

and 84 percent overall. 

• Achievement tests in reading and mathematics. The tests were administered 

immediately before the in-person survey. The tests were developed from National 

Education Longitudinal Study (NELS) tests and scored by the Educational Testing Service 

(ETS). Because only a very few youth completed the in-person survey but not the 

achievement tests or vice versa, the response rates for the achievement tests were 

essentially the same as for the in-person survey. 

• A telephone survey. The computer-assisted survey was administered mainly during the 

fall and winter of the fifth year of the demonstration and focused on high school 

completion status, postsecondary activity, and risky behaviors. The response rates were 87 

percent for QOP-group youth, 80 percent for control-group youth, and 83 percent overall. 

• High school transcripts. Data from transcripts were used to measure high school 

performance and graduation. From the telephone survey, we obtained a list of any high 

schools attended by a youth after the original QOP school. We obtained academic records 

for 86 percent of QOP-group youth, 77 percent of control-group youth, and 82 percent of 

all youth in the evaluation sample. 

Appendix C describes in greater detail our data-collection procedures and provides a more in-depth 

analysis of response patterns. 

 We measured risky behaviors twice, once in the in-person survey and once in the telephone survey. 

Most survey respondents consider information about substance abuse, criminal activity, and sexual 

activity to be personal and private. Research on alternative survey modes (mail, telephone, in-person) 

suggests that responses to such sensitive questions are more accurate when the questions are asked in 

person rather than by mail or telephone (Aquilino, 1994). To minimize interaction with the datacollection staff and provide a greater sense of privacy, each youth completed the paper survey on his or 

her own and returned the completed questionnaire to the data-collection staff in a sealed envelope. 

 The purpose of covering many of the same outcomes seven to ten months later in the telephone 

survey was to establish a baseline for comparisons to future surveys designed to measure longer-term 

program impacts. The future follow-up surveys will be conducted by telephone (in fall 2002 and fall 

2004). Comparing the results of those future telephone surveys to data previously collected by 

telephone will enable us to estimate trends unbiased by the influence of changing the mode of the 

survey. 
18 

OBTAINING AN EVALUATION SAMPLE AND CONDUCTING RANDOM ASSIGNMENT

The impact evaluation sample for the QOP demonstration consists of 1,069 youth, 580 in the QOP 

group and 489 in the control group. In this section, we describe how we obtained the sample and split 

it into the QOP and control groups. 

QOP CBOs and schools identified eligible youthówith the assistance of the evaluation teamó

during the fall of 1995 when the youth were entering ninth grade, and the CBOs provided services to 

randomly selected eligible youth from December 1995 through fall 2000. The District of Columbia site 

was the exception. Because the first CBO selected to operate QOP in that site became financially 

insolvent and had to be replaced, all program and data-collection activities took place one calendar year 

later than in the other six sites. 

As presented in Chapter II, a youth was eligible for QOP if he or she was attending a QOP school, 

was entering ninth grade for the first time, was appropriate for QOP in accordance with applicable laws 

and regulations regarding disabilities, and was in the bottom two-thirds of the grade distribution based 

on grades from eighth grade (among students satisfying the first three criteria). The second criterion 

excluded students repeating the ninth grade, and the third criterion excluded students for whom QOP 

would have been inappropriate because of severe physical or learning disabilities. In addition to these 

four eligibility criteria based on a youthís characteristics, a fifth criterion was implicit: a youth had to be 

selected at random for QOP if the number of youth meeting the first four criteria exceeded the number 

of slots allocated to the school. Although this criterion clearly pertained to the demonstration program 

because a random assignment evaluation was to be conducted, we believe that if there were ever a 

regular, ongoing QOP program, itólike the demonstration programówould likely use random 

selection for selecting enrollees from among eligible youth if, as seems fairly probable, there were more 

eligibles than program slots.11 Of course, if there were more slots than eligibles, all eligible youth would 

be enrolled. 

In the QOP demonstration, CBO and school staff applied the first three eligibility criteria and were 

responsible for obtaining informed consent from a parent or guardian for a youthís enrollment in the 

evaluation and the program. Evaluation team staff typically applied the fourth eligibility criterion and 

always carried out random assignment. The rest of this section summarizes the procedures for applying 

the eligibility criteria, which are described in detail in Appendix A. 

For the 580 available QOP slots in the 11 QOP schools, there were 2,550 ìGPA eligiblesîóyouth 

meeting the four (explicit) eligibility criteria. Although an ongoing QOP program would have selected 

as many youth from this group as there were available slots in each school, more youth were needed in 

the demonstration program to form a control group for the impact evaluation. Accordingly, we 

randomly selected over 1,200 of the GPA-eligible youth for an initial sample. Then, for all youth in the 

initial sample, we instructed QOP staff to obtain consent for participation in the evaluation. 

 11 This belief is based on two critical elements of the program model: (1) QOP seeks to serve youth who might not be 

interested in participating as well as youth who are interested in participating, and (2) the motto of the program is ìOnce in 

QOP, Always in QOP.î For QOP to serve all youth meeting the four (explicit) eligibility criteria regardless of motivation 

means that youth cannot enter the program through a nonrandom process, for example, through applications or referrals. 

QOPís motto means that enrollees cannot drop out of or be expelled from the program and then be replaced by other youth. 
19 

As it turned out, about five percent of the youth in the initial sample were determined to be 

ineligible for QOP based, in most instances, on evidence from school records indicating that a youth 

had never attended the QOP school or had transferred to another school early in the school year before 

QOP eligibility was determined. The parents/guardians of about another seven percent of the youth in 

the initial sample never responded to QOP staffís attempts to obtain consent. There was strongly 

suggestive evidence from school staff or the youthsí relatives, friends, and neighborsóbut not definitive 

evidence from school recordsóthat many of the youth were in fact ineligible. However, in some 

instances, the failure to respond probably was a passive denial of consent. Parents/guardians actively 

denied consent for another two percent of the initial QOP sample. Before we conducted random 

assignment for a school, QOP staff had to document that they had made substantial efforts to contact 

and obtain consent from the nonrespondents. 

From among the 1,069 ìconsentersîóeligible youth in the initial sample for whom consent to 

enroll in the evaluation was obtainedówe filled the available QOP slots in each school by random 

assignment. The 580 youth randomly selected for QOP constitute the QOP group. The 489 youth who 

were not selected for QOP are the control group. QOP-group members were allowed and encouraged 

to participate in QOP activities. Control-group members were not allowed to participate, although they 

could participate in the activities of other programs in their schools and communities. 

ESTIMATING PROGRAM IMPACTS

Earlier in this chapter, we described our basic approach to estimating QOPís impact on an 

outcome: we subtracted the mean outcome for youth in the control group from the mean outcome for 

youth in the QOP group. Before describing some of the technical details pertaining to how we derived 

this ìdifference-of-meansî impact estimate, we discuss several issues pertaining to the interpretation of 

the estimate. 

Interpreting Impact Estimates 

As discussed before, the control group provides the ìcounterfactualî to the QOP group by 

showing what would have happened to the QOP group had its members not been enrolled in QOP. 

Although members of the control group were not enrolled in QOP, they were allowed to enroll and 

participate in other programs offered in their schools and communities. Thus, in the evaluation of the 

QOP demonstration, the counterfactual is an environment in which other programs might be available 

and members of the control group might participate in those programs. Thus, an impact estimate 

measures the incremental effect of QOP relative to the effects of other programs in which youth would 

participate if QOP were not available. As documented in Chapter VI, 16 percent of control-group 

members participated in a youth program other than QOP. Given that most youth programs are 

substantially less intensive than QOP, participation in programs by members of the control group was 

probably not so extensive or intensive (for those who did participate) that the counterfactual to QOP 

closely resembles QOP. 

Once we have obtained impact estimates, we face the question of whether we can generalize our 

findings beyond the seven CBOs in the QOP demonstration, that is, are the findings ìexternally valid.î 

The answer is no. The CBOs in the demonstration were not selected by using any type of probability 

sampling. Thus, they are not statistically representative of a universe of potential CBOs. The 

procedures that were used to select CBOs are described in Maxfield et al. (2003). 
20 

We have not raised the issue of external validity to criticize the procedures for selecting CBOs or 

the design of the demonstration. Rather, we have sought to clarify which interpretations of the 

evaluation findings are valid and which are not. We are not aware of any demonstration of a social 

program in which sites have been selected so as to ensure external validity of evaluation findings. 

Finally, it is important to remember that random assignment was conducted successfully and that 

its integrity was maintained in the sense that, based on our knowledge from several monitoring activities, 

no control-group youth participated in QOP. Thus, for the fixed set of CBOs in the QOP 

demonstration, the impact estimates measure the causal effects of QOP, that is, they are ìinternally 

valid.î 

Estimating Impacts 

In subsequent chapters, we present impact estimates for the whole QOP demonstration, that is, for 

the seven sites combined. We also present impact estimates for each of the seven sites considered 

separately. All of these estimates were obtained from school-level estimates. We took this approach so 

that we could measure accurately the statistical uncertainty in our impact estimates for sites. 

To obtain an impact estimate for a school, we subtracted the mean outcome among youth in the 

control group for that school from the mean outcome among QOP enrollees for that school. Random 

assignment in each school was conducted separatelyóthat is, independentlyófrom the random 

assignments in other schools. Although at least some of the youth in both groups no longer attended 

the original QOP school by the end of the demonstration, all youth remained members of the QOP 

group or control group to which they were originally assigned. 

For each of the four sites in which youth were selected from only one school, the site-level impact 

equals the school-level impact. For the other three sites, the site-level impact equals a weighted average 

of the school-level impacts, where the weight placed on the impact estimate for a given school equals 

the proportionate number of QOP slots assigned to that school. For the District of Columbia site, the 

impact estimates from the two QOP schools were weighted equallyówith weights of 0.5 and 0.5ó

because the schools had equal numbers of slots. This was also true for the Houston site. For the 

Memphis site, the impact estimates for the three QOP schools had weights of 0.27, 0.35, and 0.38, 

reflecting the slightly unequal allocation of QOP slots. 

To obtain an impact estimate for the entire demonstration, we calculated a weighted average of the 

site-level impact estimates. However, our approach to weighting site-level estimates to obtain a 

demonstration-level estimate was different from our approach to weighting school-level estimates to 

obtain a site-level estimate. We weighted site-level estimates equally rather than according to the 

number of QOP slots available in each site. Our equal weighting of sites was based on the belief that if 

QOP were implemented as an ongoing, national program, CBOs would have roughly equal numbers of 

QOP slots. We believe that the variation in the demonstration site program sizes would not be 

replicated in an on-going QOP program, as discussed in Appendix E. To determine whether our 

conclusions would be different if site-level impacts were weighted unequally rather than equally, we 

derived impact estimates for the whole QOP demonstration by weighting each siteís impact estimate 

according to the proportionate number of slots at that site. Appendix F presents the estimates. 

Appendix E presents the mathematical expressions for how we derived all of our impact estimates. 

Because the QOP and control groups for each school were the product of random assignment, the 

difference-of-means impact estimator is statistically unbiased. This means that if it were possible to 

repeat the random assignment process many times for each school, the average impact estimate would 
21 

equal the true impact of QOP in that school. The reason is that the only differences between the QOP 

and control groups at the time of random assignment are purely random, and those differences (and 

their effects on impact estimates) will ìaverage outî if random assignment were performed many times. 

 Thus, ìhighî and ìlowî impact estimates will average to the true impact. 

Of course, it is not possible to perform random assignment many times. Instead, it is performed 

just once for each school. Although random assignment ensures that any differences in average baseline 

characteristics between QOP enrollees and control-group youth are purely random, random assignment 

cannot ensure that the QOP and control groups are perfectly balanced across all baseline characteristics. 

 For example, there is no guarantee that 100 females will be split 50/50 between the two groups. There 

is even some chance that they will be split very unevenly. Then, if the true impacts are different for 

females and males, we could obtain an estimate of the overall impact that is ìtoo highî or ìtoo low,î 

even though our estimator is unbiased. 

To attempt to correct for purely random baseline differences between the QOP and control 

groups, we have derived ìregression-adjustedî impact estimates. These estimates are presented in 

Appendix F, where we also examine the baseline differences between the QOP and control groups for 

which our regression adjustments sought to compensate. Because DOL elected not to have a baseline 

survey conducted for the QOP demonstration, we have data for only a small set of baseline 

characteristics: age, sex, race/ethnicity, and grade point average from the eighth grade. As described in 

Appendix B, our data on these characteristics were obtained when we determined eligibility for random 

assignment and when we conducted the telephone survey. 

Because random assignment occurred after eligibility for QOP was determined but before youth 

started participating in QOP, enrollees consisted of those who participated in QOP activities (about 88 

percent) and those who did not, the so-called ìno-showsî (about 12 percent). The impact estimates are 

based on all enrolleesóthe QOP target populationórather than on only those youth sufficiently 

motivated to participate in the programís activities. For two reasons, it is appropriate to evaluate QOP 

according to its impacts on enrollees rather than its impacts on participants. 

First, the QOP mottoóìOnce in QOP, Always in QOPîóreflects one of the most fundamental 

philosophical underpinnings of the program model, namely, that the least motivated youth might have 

the greatest need for assistance and that the program is designed to serve and will make every reasonable 

effort to serve those youth. That no-shows remain members of the QOP group and should continue to 

receive substantial attention from QOP staff is a requirement of the QOP model. 

Second, estimating impacts for enrollees is faithful to the experimental design and does not require 

that we assume that QOP had no impact on no-shows. Avoiding such an assumption is, we believe, 

prudent because QOP staff often invested substantial time in trying to engage no-shows. Such efforts 

might have had nontrivial, albeit maybe still small, effects even if the efforts did not result in active 

participation in QOP. 

Although we believe that the arguments are strong for presenting impact estimates for enrollees 

only, Appendix G presents impact estimates for participants. To derive such estimates, we assumed that 

QOP had no impact on no-shows. Then, the impact for participants was obtained by dividing the 

impact for enrollees by the participation rate among enrollees (Bloom, 1984). 
22 

Weighting to Adjust for Nonresponse 

As noted before, about 16 percent of youth in the evaluation sample did not complete the inperson survey or take the achievement tests. About 17 percent did not complete the telephone survey, 

and no transcript data at all were obtained for 18 percent of youth. When we calculated the mean 

outcome among QOP enrollees for an outcome measured in the telephone survey, for example, we had 

data only for the enrollees who responded to the survey. Therefore, although we wanted to obtain the 

mean for all QOP enrolleesórespondents and nonrespondentsóthe mean that we were able to 

calculate was based on the respondents only. As reported in Appendix E, differences existed in the 

baseline characteristics of respondents and nonrespondents. Thus, outcomes might also have differed 

as well, although we could not measure those differences. To compensate for those differences and 

obtain mean estimates from respondents that were as close as possible to the means for all youthóboth 

respondents and nonrespondentsówe assigned weights to the respondents (the weights for 

nonrespondents equal zero). 

As described in Appendix E, sample membersí weights were based on their estimated response 

probabilities. Respondents who had lower response probabilities and ìlooked more likeî 

nonrespondents in terms of their baseline characteristics received greater weights than respondents who 

had higher response probabilities and ìlooked less likeî nonrespondents. The objective was to make 

the weighted sample of respondents closely resemble the sample of all youth. We used the assigned 

weights to calculate mean outcomes. Because the mechanisms and patterns of nonresponse differed 

across data-collection activities, we developed three sets of weights. The three sets adjusted for 

nonresponse to, respectively, the (1) in-person survey and achievement tests, (2) the telephone survey, 

and (3) high school transcripts. 

Estimating the Precision of Impact Estimates 

In addition to estimating impacts, we estimated standard errors to measure the potential error in the 

impact estimates. Such error is largely attributable to the relatively small number of youth from each 

school. Appendix E presents the mathematical expressions that show how we estimated standard 

errors. 

We used the standard error for an impact to conduct a t-test to determine whether the estimated 

impact was large relative to the error in that estimate. Because we conducted two-sided t-tests, either a 

positive impact or a negative impact could be judged large if it was sufficiently far from zero. In fact, in 

subsequent chapters, we report whether each estimated impact is ìsignificantly different from zeroî at 

each of three conventional significance levelsó1 percent, 5 percent, and 10 percent. An impact that is 

significantly different from zero at a given significance leveló1 percent, for exampleóis also 

significantly different at any higher leveló5 and 10 percent. When an impact is significantly different 

from zero at the 1 percent level, we are 100 ñ 1 = 99 percent ìconfidentî that the impact is significantly 

different from zero. Thus, although the true impact might be smaller (or bigger) than we have 

estimated, we are fairly certain that it does not equal zero, even allowing for the potential error in our 

estimate. 

A word of caution about the interpretation of results from statistical tests is that ìstatistical 

significanceî does not imply ìpolicy importance.î An impact that is significantly different from zero 

might still be small. Conversely, because the evaluation sample from the QOP demonstration has fairly 

small numbers of youth from each school, an estimated impact might be large from a policy perspective 

but based on insufficient data for us to conclude that it is significantly different from zero. A related 

consideration in that case is that while there is a chanceóan unacceptable chance by conventional 
23 

standardsóthat the true impact is close to zero, leading us to conclude that the impact is not significant, 

there is also a chance that the true impact might be much greater than we have estimated. Finally, we 

note that when we derive many impact estimates and conduct many significance tests, some impacts will 

turn out to be significant just by chance, even though the true impact is close to zero. To address this 

issue, we look in subsequent chapters for consistent patterns of impacts across, for example, related 

outcomes. We have also assessed the sensitivity of our results to alternative estimation procedures. 

Appendix F presents results from our sensitivity analyses. 

Estimating Impacts for Subgroups 

Impact estimates for the full evaluation sample might conceal important differences in impacts 

across subgroups of youth. If impacts do exist overall, they might be heavily concentrated in or could 

be much larger for some subgroups. Conversely, if impacts do not exist overall, they might still exist for 

some subgroups. Thus, estimates of subgroup impacts can help policymakers identify the youth for 

whom a program is most effective and thereby better target a program or better tailor its services. 

In Chapter VII, we present impact estimates for subgroups defined by two classification schemes. 

The first scheme classified youth according to baseline characteristics, and the second scheme classified 

youth according to location. Applying the first scheme, we derived impact estimates for the following 

subgroups: 

• Males and females 

• Youth who entered the ninth grade when they were over the age of 14 and youth who 

entered ninth grade when they were 14 or younger 

• Youth in the lower third of the eighth-grade GPA distribution for the evaluation sample, 

youth in the middle third, and youth in the top third 

Applying the second subgroup classification scheme, we derived impact estimates for the following 

subgroups: 

• Enrollees from Cleveland, enrollees from the District of Columbia, enrollees from Fort 

Worth, enrollees from Houston, enrollees from Memphis, enrollees from Philadelphia, and 

enrollees from Yakima 

• Enrollees from the sites funded by the Department of Labor and enrollees from the sites 

funded by The Ford Foundation 

Because all of the subgroups were defined by characteristics that were fixed at baseline, we derived 

impacts and performed statistical tests by using the same methods that we used for the entire evaluation 

sample. Random assignment ensures that any differences in baseline characteristics between QOP 

enrollees and control-group youth for the whole sample and by subgroup were due entirely to chance. 

If we were to define a subgroup by a characteristic that was not fixed at baseline, the enrollees and 

the control-group members in that subgroup might have differed systematically at baseline. Then, any 

subsequent differences in outcomes might have been attributable to those baseline differences rather 

than to QOP. For this reason, we did not estimate impacts for outcomes that were conditional on other 

outcomes and, thus, may have been influenced by QOP. 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
25 

C HAPTER IV 

P ARTICIPATION IN QOP 

 QOP sought to foster active participation by all eligible youth, not just those who were strongly 

motivated to participate. To assess whether QOP was successful, we analyzed enrollee participation to 

determine, for example, whether most enrollees engaged in most activities or whether many enrollees 

participated only occasionally. 

 Our analysis of participation was based on data on the hours that each enrollee spent engaged in 

each of the three types of program activitiesóeducational, developmental, and community serviceóas 

determined from the management information system (MIS) used by each CBO to monitor and manage 

program operations. Case managers and site coordinators recorded MIS data for the purpose of 

computing periodic stipend payments and accrual account contributions for each enrollee. Hours were 

entered into the MIS for each type of activity each month for each enrollee. 

Enrollees spent an average of 174 hours per year on QOP activities, 23 percent of the annual goal of

750 hours. 

Enrollees spent an average of 72 hours per year on education (29 percent of the 250 hour goal), 76 

hours on developmental activities (30 percent of the 250 hour goal), and 26 hours on community

service (11 percent of the 250 hour goal). 

The average time spent on QOP activities fell steadily from 247 hours in the first year of the 

demonstration to 89 hours in the fourth year. 

Enrollees who attended many QOP activities tended to have higher grades at baseline, be age 14 or 

younger upon entering the ninth grade, be in families receiving welfare, and be in families headed by

a single parent. On the other hand, enrollees who attended few QOP activities tended to have a 

lower baseline GPA, be male, speak a language other than English at home, and be over 14 years of

age upon entering the ninth grade. 

Enrollees who attended few QOP activities reported being uninterested in those activities, having

other after-school activities, such as playing a sport, working, or caring for other family members. 
26 

 MIS data have four limitations stemming from the fact that the data were recorded for 

administrative, rather than research, purposes: 

• Given that mentoring time did not count toward stipends or accrual account contributions, 

data on time spent being mentored were not recorded. 

• In some sites, when an enrollee achieved a significant milestone, such as earning a B 

average or higher on his or her report card, the CBO would record extra hours (50 extra 

hours, for example) in the MIS for the enrollee. The extra hours resulted in an increased 

stipend payment and accrual account contribution. Unfortunately, bonus hours could not 

be distinguished from regular hours in the MIS data and thus result in overestimates of the 

amount of time spent on program activities for some enrollees. 

• Some sites were unable to provide MIS data for the final months of the demonstration. 

Results for the fifth and final year represent two sites (Cleveland and Fort Worth), results 

for the fourth period represent six sites (Cleveland; Fort Worth; Houston; Philadelphia; 

Washington, D.C.; and Yakima), whereas results for other time periods represent all seven 

sites. (Periods are defined below.) 

• The MIS data contain outliers. According to the MIS data, a few enrollees spent more than 

2,000 hours in a year on QOP, which, if true, would indicate that these enrollees spent 40-

hour weeks on program activities all year long. Because such extensive participation is 

implausible, we truncated recorded values at 36 hours per component per month, which 

seemed like a reasonable upper bound on participation. 

 Since QOP delivered services for five years, we might expect that the amount of time that enrollees 

spent on program activities changed over the course of the demonstration. Enrollees entered the 

program near the beginning of their adolescence and could remain active until early adulthood. To 

observe the changing participation patterns as enrollees aged, we divided the five-year demonstration 

period into five annual time periods (with one exception):12

• Period 1óThe first full year of program operations (February 1996 through January 1997) 

• Period 2óThe second full year of program operations (February 1997 through January 

1998) 

• Period 3óThe third full year of program operations (February 1998 through January 1999) 

• Period 4óA seven-month period from the end of Period 3 to the beginning of the fifth 

academic year (February 1999 through August 1999) 

• Period 5óThe final full year of program operations (September 1999 through August 

2000) 

 12 All periods are defined one year later for the Washington, D.C., site. 
27 

 During Periods 1 through 4, the large majority of enrollees were attending high school. Those who 

were not attending high school had officially or informally dropped out, and some were attending GED 

preparation classes. In Period 5, the majority of enrollees were engaged in one or more of a broad array 

of postsecondary activities, including college, employment, apprenticeship, trade school, and service in 

the armed forces. QOP services for such enrollees were limited to mentoring. In contrast, QOP 

offered the full range of services to the enrollees who were still in high school during Period 5. 

PARTICIPATION IN QOP 

 Figures IV.1 through IV.4 are based on MIS data and display the frequency distributions of hours 

spent on program activities per period. In each figure, there is a vertical bar for each of the five time 

periods, and each bar shows the distribution of all QOP enrollees across categories of hours. A 

segment on a bar shows the percentage of enrollees whose participation was in a particular category (for 

example, 101 to 200 hours). Across all of the segments in a bar, the percentages add to 100.13 Figure 

IV.1 shows time spent on all recorded program activities combinedóeducational, developmental, and 

community serviceówhere the program goal was 750 hours per year. Figures IV.2 through IV.4 show 

time spent on each of these three component activities, where the annual goal was 250 hours for each 

component. 

 The large majority of enrollees spent a small fraction of the target number of hours on program 

activities. On average, enrollees spent 174 hours on QOP activities per year, 23 percent of the goal. 

Enrollees spent an average of 72 hours per year on education, 76 hours on developmental activities, and 

26 hours on community service. Thus, enrollees spent roughly equal amounts of time on educational 

and developmental activities. In contrast, they spent substantially less time on community service. By 

Period 4, community service ceased to be an active program component for all but a small group of 

dedicated enrollees. 

 Throughout the demonstration, the mean amount of time spent on QOP activities fell steadily. 

The average enrollee spent 247 hours on QOP in Period 1 and 89 hours in Period 4.14 The percentage 

of enrollees spending no time at all on QOP activities increased steadily from 1 percent in Period 1 to 

36 percent in Period 4. The proportion of enrollees in the middle range of the distribution (201 to 300 

hours per year) declined steadily from 17 percent in Period 1 to 5 percent in the Period 4. 

 13 Since Period 4 was only seven months long, the hours for Period 4 in Figures IV.1-IV.4 are inflated by a factor of 

12/7 to be comparable to the hours of the other periods. 

14 Because QOP services in Period 5 differed substantially from those of the first four periods, we report trends over 

the first four periods. In Period 5, QOP offered enrollees who had graduated from high school only mentoring services, and 

hours spent being mentored were not recorded. 
28 

FIGURE IV.1 

DISTRIBUTION OF HOURS BY PERIOD 





    









    





    













    











    









0%

20%

40%

60%

80%

100%

Period 1 Period 2 Period 3 Period 4 Period 5

Percentage of Enrollees in Hours Category



500+ hours

401 - 500 hours



301 - 400 hours



201 - 300 hours

101 - 200 hours

 1 - 100 hours

0 hours

SOURCE: MIS data. 

FIGURE IV.2 

DISTRIBUTION OF EDUCATION HOURS BY PERIOD 





    









    





    









 







0%

20%

40%

60%

80%

100%

Period 1 Period 2 Period 3 Period 4 Period 5

Percentage of Enrollees in Hours Category

400+ hours

 301 - 400 hours

 201 - 300 hours

101 - 200 hours

 1 - 100 hours

0 hours

SOURCE: MIS data. 
29 

FIGURE IV.3 

DISTRIBUTION OF DEVELOPMENT HOURS BY PERIOD 





 









    





    









  









0%

20%

40%

60%

80%

100%

Period 1 Period 2 Period 3 Period 4 Period 5

Percentage of Enrollees in Hours Category

400+ hours



301 - 400 hours



201 - 300 hours

101 - 200 hours



1 - 100 hours

0 hours

SOURCE: MIS data. 

FIGURE IV.4 

DISTRIBUTION OF COMMUNITY SERVICE HOURS BY PERIOD 





    











 













0%

20%

40%

60%

80%

100%

Period 1 Period 2 Period 3 Period 4 Period 5

Percentage of Enrollees in Hours Category

400+ hours



301 - 400 hours



201 - 300 hours

101 - 200 hours



1 - 100 hours

0 hours

SOURCE: MIS data. 
30 

PROFILES OF DEDICATED ENROLLEES AND DISENCHANTED ENROLLEES

 In this section, we profile the ìdedicatedî enrollees who attended many QOP activities, and the 

ìdisenchantedî enrollees who attended few QOP activities. Dedicated enrollees spent 1,300 or more 

hours on QOP during the demonstration, and disenchanted enrollees spent 100 or fewer hours on 

QOP. According to the first row in Table IV.1, 19 percent of all QOP enrollees were dedicated, and 12 

percent were disenchanted. The second row in Table IV.1 shows that 17 percent of male enrollees were 

dedicated, while 15 percent were disenchantedóa figure that is significantly differently from the 12 

percent figure for all enrollees. 

 We find that QOP enrollees were more likely to be dedicated and spend large amounts of time on 

program activities if they: 

• Had a GPA in the top third of the baseline grade distribution. This finding suggests that 

more successful youth spent more time on program activities. 

• Were age 14 or younger when they entered the ninth grade. This finding may have two 

explanations. First, youth who were younger when they entered the ninth grade were less 

likely to have previously repeated a grade and thus may have been relatively successful 

students. Second, youth in their early adolescence may have been more receptive to QOP 

than were older youth. 

• Were members of families who received cash welfare or food stamps.15 Eligibility for 

QOP was not restricted to youth from low-income families, although the large majority of 

QOP enrollees lived in low-income neighborhoods. 

• Were members of families with a single biological or step-parent.16 

The profile of the disenchanted group indicates that enrollees were more likely to spend very little 

time on QOP activities if they: 

• Had a GPA in the bottom third of the baseline grade distribution. 

• Were male. 

• Did not usually speak English at home. 

• Were over 14 years of age upon entering the ninth grade. 

• Were not members of families who received cash welfare or food stamps. 

 15 Receipt of welfare and food stamps was measured only near the end of the demonstration period. While there is a 

possibility that receipt of assistance was influenced by participation in QOP, we believe that the risk of bias resulting from 

this is small for the following reason. Receipt of assistance was determined largely by the youthís parents, and we expect that 

the potential indirect effect of QOP on the youthís parents was small. 

16 Family composition was measured only near the end of the demonstration period. However, it seems unlikely that 

QOP would have a strong influence on family composition. 
31 

TABLE IV.1 

PROFILES OF ENROLLEES ATTENDING MANY QOP ACTIVITIES AND ENROLLEES ATTENDING FEW 

QOP ACTIVITIES 

 Percentage of enrollees with the indicated characteristic 

who were 

Characteristic Dedicated Disenchanted 

All enrollees 19 12 

Male 17 15ÜÜÜ

Female 20 9 

GPA in top third of baseline distribution 28ÜÜÜ 6ÜÜÜ

GPA in middle third of baseline distribution 18 12 

GPA in bottom third of baseline distribution 13ÜÜÜ 18ÜÜÜ

Does not usually speak English at home 18 25ÜÜÜ

Usually speaks English at home 19 8 

14 or younger when entering ninth grade 22ÜÜÜ 7ÜÜÜ

Older than 14 when entering ninth grade 13 22 

Someone in the household receives cash welfare or food 

stamps 

31ÜÜÜ 5Ü

No one in the household receives cash welfare or food 

stamps 

17 10 

No adult in household 7ÜÜÜ 10 

One or more adults, not biological or step-parent 14 8 

Only one biological or step-parent 25ÜÜ 6Ü

More than one biological or step-parent 22 11 

SOURCE: MIS data and telephone survey. 

Ü Significantly different from the percentage for all other enrollees at the 90% confidence level, two-tailed test 

ÜÜ Significantly different from the percentage for all other enrollees at the 95% confidence level, two-tailed test 

ÜÜÜ Significantly different from the percentage for all other enrollees at the 99% confidence level, two-tailed test 
32 

REASONS FOR LOW PARTICIPATION

 For the in-person survey during the fourth year of the demonstration, we asked enrollees whether 

they felt that the time they spent on QOP was limited, and if so, what factors limited their participation. 

We also asked what aspects of the program they found most and least helpful. The results, presented in 

Figure IV.5, show that: 

• Twenty-four percent reported that their lack of interest in QOP activities was the reason for 

not participating more. 

• Fifteen percent reported that their time commitment to a job was the reason for not 

participating more. 

For the 151 enrollees who participated early in the demonstration and then stopped participating 

before the end of the fourth academic year of the demonstration, Figure IV.6 presents the reasons for 

stopping participation. It shows that: 

• Nearly 3 in 10 stopped participating because they left high school as a result of either 

graduating early or dropping out. 

• About 1 in 5 stopped because of a job. 

• About 1 in 6 stopped because of a move out of the area or a transfer to another school. 

Half of those enrollees who reported that they participated in some QOP activity at some time 

during the demonstration (88 percent of all QOP enrollees) felt that the amount of time they spent on 

QOP activities was limited by some factor. Among that half, Figure IV.7 shows that: 

• The participation of nearly 2 in 5 was limited by a job. This is consistent with the fact that 

most enrollees were 17 or 18 years old at the time of the interview. 

• The participation of 1 in 5 was limited by responsibilites for caring for own children or 

other family members. 

• The participation of over 1 in 8 was limited by after-school activities, such as sports. 
33 

FIGURE IV.5 

REASONS FOR LIMITED PARTICIPATION BY DISENCHANTED ENROLLEES 

1

1

4

5

5

15

24

0 15 30 45

Own health problems

Sports/other school activities

Transportation problems

Did not like people at QOP

Taking care of family/child

Job

Not interested in QOP

Percentage Giving Reason

SOURCE: Telephone survey. 

FIGURE IV.6 

REASONS FOR STOPPING PARTICIPATION 

1

2

3

4

4

17

22

28

0 15 30 45

Did not like people at QOP

Transportation problems

Taking care of family/child

Sports/other school activities

N ot interested

M oved out of area/transferred

to another school

Job

Dropped out of school or

graduated early

Percentage Giving Reason

SOURCE: Telephone survey. 
34 

FIGURE IV.7 

REASONS FOR LIMITED PARTICIPATION BY ALL ENROLLEES WHO LIMITED THEIR PARTICIPATION 

2

2

3

8

13

20

38

0 15 30 45

Own health problems

Not interested

Did not like people at QOP

Transportation problems

Sports/other school activities

Taking care of family/child

Job

Percentage Giving Reason

SOURCE: Telephone survey. 
35 

C HAPTER V 

S HORT - T ERM I MPACTS ON H IGH S CHOOL 

C OMPLETION , H IGH S CHOOL P ERFORMANCE , 

AND P OSTSECONDARY A CTIVITIES

 

HIGH SCHOOL COMPLETION AND PERFORMANCE

The first of the two main goals of QOP was to increase the likelihood that enrollees graduated 

from high school. QOP raised the graduation rateóthe percentage of enrollees receiving a diplomaó

by seven percentage points. This impact was statistically significant. Table V.1 presents the impact 

QOP significantly increased by seven percentage points the likelihood that enrollees 

graduated from high school with a diploma. 

QOP increased the likelihood of engaging in postsecondary education or training, although 

the size and significance of the impact depends on how this outcome was measured and 

how the impact was estimated. 

• QOP significantly increased by six percentage points the likelihood of engaging in 

postsecondary education or training when education or training was defined to include 

college attendance, vocational or technical school attendance, apprenticeship 

enrollment, and armed forces enlistment. The impact became smaller and insignificant 

when this measure was either narrowed to include only college attendance or 

broadened to include employment. It also became smaller and insignificant when we 

used alternative approaches to estimating the impact. 

• When we included acceptance into collegeóin addition to current attendance at 

collegeóin the definition of postsecondary education or training, QOP significantly 

increased the likelihood of engaging in postsecondary education or training by six to 

nine percentage points for all but one measure of postsecondary activity. 

QOP did not significantly improve enrollee performance while in high school. 

• QOP did not significantly raise reading or mathematics achievement test scores or high 

school grades. 

• QOP did not significantly increase the number of credits earned by enrollees or reduce 

disciplinary actions taken against enrollees in high school. 
36 

estimates. (All tables appear at the end of the chapter.) The left-hand column lists several outcomes 

pertaining to high school completion. For a given outcome, the next three columns show, respectively, 

the mean outcome for QOP enrollees, the mean outcome for control-group youth, and the impact. The 

impact is the mean outcome for the QOP group minus the mean outcome for the control group. The 

first row of Table V.1 reveals that 46 percent of QOP enrollees received diplomas and that 40 percent 

of control-group youth received diplomas; the impact is an increase of 7 percentage points (when 

calculated from unrounded means for the QOP and control groups). In all of the tables in this report, 

any impact that is significantly different from zero at the 90, 95, and 99 percent confidence levels is 

marked with one, two, and three asterisks, respectively. 

In view of the fact that our data were collected before the end of the demonstration when about 16 

percent of sample members were still attending high school, we used four measures of high school 

completion. The measures begin with the narrowestógraduating from high school with a diploma (the 

top row of the table)óand end with the broadestógraduating from high school with a diploma, earning 

a GED certificate, attending high school, or attending a GED class (the bottom row of the table). 

Table V.1 shows that QOP had significant positive impacts on both the narrowest and the broadest 

measures of high school completion, but insignificant impacts on the two intermediate measures. This 

pattern indicates that QOP improved the likelihood that enrollees earned a diploma and suggests that 

QOP increased the likelihood that enrollees who dropped out of high school attended a GED class. It 

also suggests that QOP did not improve either the likelihood that enrollees earned a GED during the 

period covered by the survey or the likelihood that enrollees who did not graduate on time stayed in 

high school for a fifth year. This pattern of short-term impacts also indicates that the final size of 

QOPís impact on high school completion will depend on whether the sample members still attending 

high school when we conducted our survey eventually earn diplomas or GED certificates and whether 

those attending GED classes eventually earn GED certificates. This will be measured in the next survey 

of sample members. 

Beyond the significant impacts on high school graduation and completion that we have already 

discussed, we find in Table V.2 that QOP did not significantly improve achievement test scores, grades, 

or credits earned, and it did not significantly reduce disciplinary actions. Although QOP might not have 

raised grades if QOP enrollees were taking more challenging courses than the youth in the control 

group, we would have expected QOP to increase standardized test scores if it had an impact on 

achievement. 

POSTSECONDARY ACTIVITIES

The second of the two primary goals of QOP was to increase the likelihood that enrollees engaged 

in postsecondary education or training by attending a college or a vocational or technical school, 

enrolling in an apprenticeship program, or enlisting in the armed forces. According to data from our 

telephone survey, QOP significantly increased the percentage of youth undertaking such postsecondary 

education or training by 6 percentage points, from 26 to 32 percent. These estimated means appear in 

the first two columns of figures in Table V.3, and the estimated impact is in the first column of figures 

in Table V.4. 

When we consider narrower definitions of postsecondary education or training that count only 

college attendance or broader definitions of postsecondary activity that count not only education or 
37 

training but also employment, the estimated impacts in the first column of figures in Table V.4 are 

smaller and not significant. However, they are consistently positive. A comparison of the impact on 

college attendance with the impact on all four postsecondary education or training activities indicates 

that about half of the impact on the latter was attributable to increased college attendance while the 

other half was attributable to increased vocational and technical school attendance, apprenticeship 

training, and armed forces enlistment. 

 It is clear from Table V.4 that the size and significance of QOPís impact on postsecondary 

education or training depended on how we measured such activity. The size and significance of the 

impact also depended on how we estimated it. As reported in Appendix F, we found that the impact on 

the likelihood of engaging in postsecondary education or training was smaller and insignificant when we 

used an alternative approach to weighting site-level impacts and when we used regression methods to 

adjust for random differences between the baseline characteristics of the QOP group and the control 

group. 

Recognizing that some youth might have needed more than the six or fewer months between high 

school completion and our telephone survey to begin postsecondary education or training, we defined 

outcomes that count acceptance by a college as well as attending a college as forms of postsecondary 

education or training. The impacts of QOP on these outcomesóin the second column of figures in 

Table V.4ówere positive and significant with only one exception (attending or acceptance by a fouryear college). QOP increased by eight points the percentage of enrollees who had been accepted by a 

college, were attending a college, or were engaged in one of the other three types of postsecondary 

education or training (vocational/technical school, apprenticeship, and armed forces). 

Comparing estimates in the first two rows of Table V.4 reveals that the higher impact estimates 

obtained when we count acceptance by a college in addition to attendance are attributable to the higher 

acceptance rates for QOP enrollees than for control-group youth by two-year colleges. In fact, the 

impact on four-year college training is the same when acceptances are counted as when they are not. 

Table V.4 also shows that the impacts estimated when we count college applications as well as 

acceptances and attendance were roughly the same as the impacts estimated when we count only 

acceptances and attendance. 

The broadest measures of activity in Tables V.3 and V.4 include preparation for postsecondary 

education or training, namely, attending high school or a GED class. The generally smaller impacts for 

these outcomes suggest that the control-group youth might at least partially ìcatch upî to the QOP 

enrollees in obtaining postsecondary education or training if those control-group members who were 

attending high school or a GED class at the time of our telephone survey successfully complete high 

school. The additional follow-up data collected in the next two surveys will allow us to determine how 

many youth completed high school from among the many youthóboth QOP enrollees and controlgroup youthówho were attending high school or a GED class when the first telephone survey was 

conducted. The new data will also reveal whether the youth who had been accepted by colleges 

subsequently enrolled. If they did, the longer-term impacts on postsecondary education or training 

might be fairly close to the short-term impacts that count both college attendance and acceptance. If 

not, the longer-term impacts might be close to the short-term impacts that count only college 

attendance. Of course, neither of these two sets of short-term impacts might accurately reflect longerterm impacts if, for example, the patterns of postsecondary education or training for youth completing 

high school in five years are substantially different from the patterns for youth completing high school 

in four years. 
38 

TABLE V.1 

SHORT-TERM IMPACTS ON HIGH SCHOOL COMPLETION 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

Earned diploma 46 40 7*

Earned diploma or GED certificate 54 49 5 

Earned diploma or GED certificate or attending high 

school 68 66 3 

Earned diploma or GED certificate or attending high 

school or a GED class 79 72 7**

SOURCE: Telephone survey and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
39 

TABLE V.2 

SHORT-TERM IMPACTS ON HIGH SCHOOL PERFORMANCE 

Outcomea

QOP-Group 

Mean 

Control-Group 

Mean Impact 

Mathematics achievement test score (percentile) 40.9 40.5 0.4 

Reading achievement test score (percentile) 43.2 42.7 0.5 

Cumulative GPA (four-point scale) 2.13 2.19 -0.06 

Mathematics/science GPA (four-point scale) 1.81 1.85 -0.03 

Total credits (Carnegie units) 16.2 15.8 0.5 

Core academic credits (Carnegie units) 10.7 10.2 0.6 

Mathematics/science/English credits (Carnegie units) 7.2 6.9 0.3 

Ever suspended 44% 45% -1 

Ever expelled 8% 7% 0 

Suspended or expelled in past 12 months 34% 38% -4 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. Credits are expressed in Carnegie 

units that standardize for in-class time. One Carnegie unit corresponds to a class that meets for 45 to 60 minutes every day of the week for an entire 

academic year. Core academic credits are the credits earned in mathematics, science, English, social studies, and foreign language classes. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
40 

TABLE V.3 

GROUP MEANS FOR POSTSECONDARY ACTIVITIES 

(Percentages) 

 Alternative Definitions of College Training 

 Attending Attending or Accepted 

Attending, Accepted, or 

Applied 

Outcome a

QOP 

Group 

Control 

Group 

QOP 

Group 

Control 

Group 

QOP 

Group 

Control 

Group 

Four-year college 11 8 15 12 ó b ó b

Two- or four-year college 21 18 31 25 34 28 

College, vocational/technical school, 

apprenticeship, armed forces 32 26 42 34 47 38 

Postsecondary training or good job 48 43 56 47 61 52 

Postsecondary training or any job 66 61 70 63 75 67 

Postsecondary training or high school 47 43 57 51 62 55 

Postsecondary training or high school or 

GED class 54 48 64 55 69 59 

Postsecondary training or high school or 

GED class or good job 68 65 76 69 81 73 

Postsecondary training or high school or 

GED class or any job 84 80 88 82 93 87 

SOURCE: Telephone survey. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls. 

a

 In the last seven rows of the table, ìcollegeî means either a two-year or a four-year college. ìPostsecondary trainingî means college, apprenticeship, 

vocational/technical school, or armed forces. A ìgoodî job offers employer-sponsored health insurance. 

b There are no estimates in the last two columns of the first row because the survey question about applying to college did not differentiate between twoand four-year colleges. 
41 

TABLE V.4 

SHORT-TERM IMPACTS ON POSTSECONDARY ACTIVITIES 

(Percentage points) 

 Alternative Definitions of College Training 

Outcome a Attending 

Attending or 

Accepted 

Attending, 

Accepted, or 

Applied 

Four-year college 3 3 ó b

Two- or four-year college 3 6* 6*

College, vocational/technical school, apprenticeship, armed 

forces 6* 8** 9**

Postsecondary training or good job 5 9** 9**

Postsecondary training or any job 5 7** 8**

Postsecondary training or high school 4 6* 7*

Postsecondary training or high school or GED class 6* 9** 9***

Postsecondary training or high school or GED class or good 

job 3 7** 7**

Postsecondary training or high school or GED class or any 

job 3 6** 6***

SOURCE: Telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed in Table V.3. The evaluation sample had 580 QOP enrollees and 

489 controls. 

a

 In the last seven rows of the table, ìcollegeî means either a two-year or a four-year college. ìPostsecondary trainingî means college, apprenticeship, 

vocational/technical school, or armed forces. A ìgoodî job offers employer-sponsored health insurance. 

b There is no estimate in the last column of the first row because the survey question about applying to college did not differentiate between two- and fouryear colleges. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
43 

C HAPTER VI 

S HORT - T ERM I MPACTS ON

R ISKY B EHAVIORS AND R ESILIENCY F ACTORS

 QOP did not significantly reduce risky behaviors. 

 

• QOPís impacts on gang-related activity and crime were zero or detrimental, but 

insignificant. Its impacts on the likelihood of being arrested or charged and the 

likelihood of having a child were beneficial, but also insignificant. 

• According to data from the in-person survey, QOP significantly increased by seven 

percentage points the fraction of enrollees who had a drink and the fraction who used 

an illegal drug in the 30 days before the survey. However, some evidence suggests that 

there were differences between QOP enrollees and control-group youth in the accuracy 

with which they reported risky behaviors. Those differences might have contributed 

substantially to the estimated detrimental impacts on drinking and drug use. That QOP 

might not have increased drinking and drug use is also suggested by data from the 

telephone survey. According to those data, QOP had beneficialóbut not significantó

impacts on drinking and drug use. 

QOP significantly increased one resiliency factor. 

• QOP significantly increased by 31 percentage points the fraction of enrollees reporting 

participation in a special program that helped them. Nevertheless, slightly less than half 

(47 percent) of QOP enrollees reported participating in ìspecial programs other than 

your normal high school classesÖ[that try] to help students stay in school, make good 

grades, stay away from drugs, prepare for work or college, and make good decisions in 

life.î This might reflect the fact that participation in QOP activities fell substantially 

short of the programís goal, especially by the fourth year of the demonstration when we 

asked the youth in the evaluation sample about their participation in special programs. 

• QOP did not significantly increase the likelihood that an enrollee perceived himself or 

herself as being positively influenced by a caring adult. It also did not significantly 

improve resiliency factors such as having an optimistic outlook on the future or 

believing that risky behaviors are wrong. 
44 

RISKY BEHAVIORS

QOP emphasized mentoring and offered developmental activities, in part, to reduce the likelihood 

that enrollees would engage in risky behaviors such as substance abuse, crime, and teenage childbearing. 

 We found that QOP generally did not achieve this objective. 

In both the in-person and telephone surveys, we obtained data on the incidence of risky behaviors 

among QOP enrollees and control-group youth. Table VI.1 presents the impacts of QOP on substance 

abuse estimated from in-person survey data. We found that QOP increased the fraction of youth who 

had a drink in the 30 days before the survey by seven percentage points. It increased by the same 

amount the fraction of youth who had used an illegal drug in the 30 days before the survey. Both of 

these impacts were statistically significant. QOP had a detrimental but insignificant impact on binge 

drinking (defined as consuming five or more drinks in a row). According to the impact estimates 

derived from telephone survey data and presented in Table VI.2, QOP had insignificant beneficial 

impacts on binge drinking and the use of any drug. A comparison of the estimated means in Tables VI.1 

and VI.2 suggests that the incidence of drug use was likely underreported in the telephone survey. 

Furthermore, as discussed in Chapter VII, some of the available data suggest that there were differences 

between QOP enrollees and control-group youth in the accuracy with which they reported risky 

behaviors, and those differences might have contributed substantially to the significant detrimental 

impacts estimated from the in-person survey data. An alternative explanation is that the results were 

due to purely random baseline differences between the two groups for which we could not statistically 

adjust because the differences were not associated with any of the very limited number of baseline 

characteristics that could be measured. These considerations and the estimates obtained from the 

telephone survey data suggest that while QOP might not have increased substance abuse, it also did not 

decrease substance abuse. 

Table VI.3 presents the impacts on gang activity, crime, and involvement with the criminal justice 

system that we have estimated from in-person survey data. The impacts on gang activity and crime were 

zero or detrimental, but not significant. The impact on the likelihood of ever being arrested or charged 

was beneficial, but also insignificant. The reference period for crimes was the year before the survey. 

Neither of the beneficial impacts on crime or involvement with the criminal justice system that we 

estimated from telephone survey data was significant, as shown in Table VI.4. 

In Table VI.5, we present the impacts of QOP on several measures of sexual activity. Although 

QOP reduced the fraction of youth who had ever had sex and the fraction who had ever had a child, the 

impacts were not significant. 

RISK AND RESILIENCY FACTORS

QOPís efforts to influence risky behaviors may be viewed from the perspective of the juvenile 

justice literature as attempts to mitigate the risk factors in enrolleesí social environments and strengthen 

the resiliency factors (U.S. Department of Justice 1995). The concepts of risk and resiliency factors are 

based on the belief that although youth are inherently inclined toward socially useful and productive 

behaviors, they can be led to crime or other risky behaviors by individuals in their homes, peer groups, 
45 

or neighborhoods. Such individuals might include a parent who is a substance abuser, a friend who 

invites the youth to participate in a criminal endeavor, and a neighborhood drug dealer. These 

individuals are risk factors. However, youth are not defenseless in their encounters with negative 

influences. Some youth are protected from negative influences by their relatives, friends, and adult 

mentors. Such individuals are resiliency factors. 

As an adult mentor, the QOP case manager might have been a resiliency factor. The QOP 

program as a whole might also have been a resiliency factor. From that perspective, the results in 

Chapter V and, especially, the first part of this chapter provide an assessment of QOPís effectiveness as 

a resiliency factor. Although QOP was not successful in reducing risky behaviors, another approach to 

assessing QOPís effectiveness as a resiliency factor was to ask youth about whether they had a caring 

adult or a special program that helped them resist negative influences and about how the adult or 

program helped them. 

QOP increased by an insignificant 3 percentage points, from 69 to 72 percent, the fraction of youth 

who had an ìadult, besides a family member, who positively influenced your life in some significant way, 

for example, a teacher, counselor, coach, or minister.î Although a substantial fraction (69 percent) of 

control-group youth had mentorsódespite their not being in QOPóit is conceivable that their 

mentoring relationships were not as close and long-lasting and therefore as effective as the mentoring 

relationships for QOP enrollees. However, Table VI.6 shows that QOP enrollees were generally not 

significantly more likely to have a caring adult who helped them in specific ways. All of the impacts are 

positive, but only one is significantóQOP increased by seven percentage points the likelihood of 

having a caring adult who helped the youth ìtake advantage of opportunities to get aheadî in life. 

QOP was a more effective resiliency factor as a social program than as a provider of a caring adult. 

 As show in Table VI.7, QOP significantly increased by 31 percentage points, from 16 to 47 percent, the 

fraction of youth reporting participation in a helpful social program. QOP also increased significantly 

the likelihood that a youth reported participating in a social program that had a specific positive 

influence. Across the influences listed in Table VI.7, QOPís impacts ranged from 12 to 25 percentage 

points. 

It is striking that despite these large impacts, slightly less than half (47 percent) of QOP enrollees 

reported participating in ìspecial programs other than your normal high school classesÖ[that try] to 

help students stay in school, make good grades, stay away from drugs, prepare for work or college, and 

make good decisions in life.î The discrepancies between the subjective impacts measured in Table VI.7 

and the more objective impacts measured in other tables in this report are also striking. Perhaps these 

findings reflect the fact, documented in Chapter IV, that participation by enrollees in QOP activities fell 

substantially short of the programís goal. The shortfall was especially pronounced by the fourth year of 

the demonstration when we asked the youth in the evaluation sample about their participation in special 

programs. We note that although we asked about participation ìsince beginning the ninth grade,î some 

youth might have reported about their current or recent participation status in responding to our inperson survey. As reported in Chapter IV, 36 percent of QOP enrollees were spending no time at all on 

QOP activities in the fourth year of the demonstration according to the MIS data. 

When we assess whether QOP, as an external resiliency factor, produced internal resiliency factors 

in enrollees by fostering, for example, an optimistic outlook on the future and a clear sense of right and 

wrong, we found that QOP did not significantly improve internal resiliency factors (Table VI.8). The 

only significant impact was detrimentalóa five-percentage-point increase in the fraction of youth 

dissatisfied with their physical appearance. 
46 

TABLE VI.1 

SHORT-TERM IMPACTS ON SUBSTANCE ABUSE, IN-PERSON SURVEY 

Outcomea

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

Drinking in the past 30 days 40 33 7**

Frequent drinking in the past 30 days 11 11 0 

Binge drinking in the past 30 days 24 20 4 

Frequent binge drinking in the past 30 days 7 5 2 

Drunk or high at school in the past 12 months 20 20 0 

Used any illegal drug in the past 30 days 34 28 7**

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a ìBingeî drinking means five or more drinks in a row. Drinking or binge drinking was classified as ìfrequentî if it occurred on at least eight out of the past 

30 days. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 

TABLE VI.2 

SHORT-TERM IMPACTS ON SUBSTANCE ABUSE, TELEPHONE SURVEY 

Outcomea

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Binge drinking in the past 30 days 19 23 -4 

Frequent binge drinking in the past 30 days 5 4 0 

Used any illegal drug in the past 30 days 16 19 -3 

SOURCE: Telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a ìBingeî drinking means five or more drinks in a row. Drinking or binge drinking ass classified as ìfrequentî if it occurred on at least eight out of the past 

30 days. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
47 

TABLE VI.3 

SHORT-TERM IMPACTS ON GANG ACTIVITY, CRIME, AND INVOLVEMENT 

WITH THE CRIMINAL JUSTICE SYSTEM, IN-PERSON SURVEY 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Involved in gang fight in the past 12 months 16 14 2 

Ever a gang member 13 13 0 

Currently a gang member 6 4 2 

Committed any crime in the past 12 months 31 28 3 

Ever arrested or charged 25 29 -5 

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 

TABLE VI.4 

SHORT-TERM IMPACTS ON CRIME AND INVOLVEMENT 

WITH THE CRIMINAL JUSTICE SYSTEM, TELEPHONE SURVEY 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Committed any crime in the past 3 months 10 11 -1 

Arrested or charged in the past 3 months 5 6 -1 

SOURCE: Telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
48 

TABLE VI.5 

SHORT-TERM IMPACTS ON SEXUAL ACTIVITY 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

Ever had sex 78 83 -5 

Did not use condom last time 29 28 0 

Taught about HIV/AIDS 93 94 0 

Ever pregnant or get anyone pregnant 33 33 0 

Have had a child 23 26 -3 

SOURCE: In-person survey and telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
49 

TABLE VI.6 

SHORT-TERM IMPACTS ON HOW AN ADULT POSITIVELY INFLUENCED 

THE YOUTHíS LIFE 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

There was an influential adult in my life 72 69 3 

There was an influential adult who: a 

Respected my ideas and feelings 41 38 3 

Helped me learn things that helped me do well in life 43 41 2 

Helped me take advantage of opportunities to get ahead 44 37 7**

Recognized and appreciated the things I did well in my life 43 40 2 

Had clear expectations about what I do with my life 37 36 1 

Showed me that fighting is not a good way to solve problems 39 38 1 

Showed me that breaking the law does not help me achieve 

goals 51 50 1 

Showed me that using drugs or alcohol is not a good way of 

solving problems 51 49 2 

Had any of these positive influences 66 64 1 

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Respondents rated each statement as being definitely true, mostly true, mostly not true, or definitely not true. Percentages represent those who reported 

that the statement was definitely true. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
50 

TABLE VI.7 

SHORT-TERM IMPACTS ON PARTICIPATION IN SPECIAL PROGRAMS TO HELP 

STUDENTS 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

Participated in such a program 47 16 31***

Program helped in the following ways: a 

Improve my grades 19 7 12***

Stay away from drugs or get off drugs 29 11 19***

Stay out of trouble 27 11 16***

Deal with police and courts 19 6 12***

Prepare for college 33 11 22***

Earn and save money 33 9 25***

Program helped in any of these ways 43 14 29*** 

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Respondents were asked whether the program helped a lot, a little, or not at all along each of the listed dimensions. Percentages represent those who 

reported that the program helped a lot. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
51 

TABLE VI.8 

SHORT-TERM IMPACTS ON ATTITUDES TOWARD RISKY BEHAVIORS AND 

OUTLOOK ON LIFE 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Thought that the following activity is always wrong: a 

Using drugs or alcohol frequently 57 57 0 

Committing crimes 80 83 -3 

Having a baby while a teenager 36 35 1 

Dropping out of school 75 74 0 

Thought that all of these activities are always wrong 22 22 0 

 

Disagreed with the following statements: b 

Bad things happen to people like me 83 80 3 

I'm afraid my life will be unhappy 81 80 1 

I do not like the way I look 83 88 -5*

I'll probably die before I'm 30 90 89 1 

Disagreed with all of these statements 61 57 3 

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact 

might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Respondents were asked whether the listed behavior was always wrong, sometimes wrong, or not wrong. Percentages represent those who reported that 

the activity was always wrong. 

b Respondents were asked whether they strongly disagree, disagree, agree, or strongly agree with the statement. Percentages represent those who reported 

that they strongly disagreed or disagreed with the statement. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
53 

C HAPTER VII 

S HORT - T ERM I MPACTS ON S UBGROUPS

 

Some of QOPís impacts on females and some of its impacts on males were significantly different

from zero. Although the significant impacts were beneficial for females and detrimental for males, 

QOPís impact on females was significantly different from its impact on males for only one key

outcome, the likelihood of engaging in postsecondary education or training, attending high school or

a GED class, or working. 

QOP had significant beneficial impacts on both older and younger enrollees, and it did not

consistently benefit one age group more than the other across a range of key outcomes. There was 

just one outcome for which the impacts were significantly different. QOP decreased by nine 

percentage points the fraction of younger enrollees who had a child. This impact was significantly

different from both zero and the (insignificant) six-percentage-point increase in the fraction of older

enrollees who had a child. 

QOP had several significant impacts on enrollees in the middle third of the baseline grade 

distribution, and all of those impacts were beneficial. They included a 14-percentage-point increase 

in the likelihood of receiving a diploma, a 13-percentage-point increase in the likelihood of college 

attendance or acceptance, and an 8-percentage-point decrease in the likelihood of having a child. 

QOP had both significant beneficial and detrimental impacts on enrollees in the bottom third of the 

distribution, and it had only one significant impactóa detrimental impactóon enrollees in the top 

third of the distribution. 

QOPís impacts varied from site to site. The Cleveland site had significant beneficial impacts and no 

significant detrimental impacts. The Washington, D.C.; Houston; and Memphis sites had significant

detrimental impacts and no significant beneficial impacts. The Philadelphia site had both significant

beneficial and significant detrimental impacts. The Fort Worth site had no significant impacts, and 

the Yakima site had two impactsóone beneficial and one detrimentalóthat were significantly

different from the impacts for the other six sites. Some of the detrimental impacts for sites might

not have been attributable to QOP. 

The impacts for the whole QOP demonstration were substantiallyóbut not entirelyóattributable to 

the impacts of the Philadelphia site alone or the Philadelphia and Yakima sitesóthe Ford-funded 

sitesótogether. The five DOL-funded sites had just one significant impactóthey increased by

seven percentage points the likelihood that a QOP enrollee graduated from high school. This impact

on one of QOPís primary outcomes was not significantly different from the impact for the two 

Ford-funded sites. The Ford-funded sites had four significant beneficial impacts: a 2-percentilepoint increase in the mathematics achievement test score, a 14-percentage-point increase in the 

likelihood of engaging in postsecondary education or training, a 17-percentage-point increase in the 

likelihood of engaging in postsecondary education or training or working at a good job, and a 14-

percentage-point decrease in the likelihood of having a child. The Ford-funded sites also had three 

significant detrimental impacts: 17-, 14-, and 16-percentage-point increases in the likelihood of

engaging in binge drinking, using an illegal drug, and committing a crime, respectively. However, 

these detrimental impacts might not have been attributable to QOP. 
54 

SHORT-TERM IMPACTS BY SEX

Both of QOPís significant impacts on male enrollees were detrimental (Table VII.1). QOP 

significantly decreased high school GPAs and increased binge drinking. Although the impacts on these 

two outcomes for females were not significantly different from the impacts for males, they were 

sufficiently different that the impacts for males and females combined were not significantly different 

from zero, as reported in Table VII.1 and discussed in Chapters V and VI. 

In contrast to the significant impacts on male enrollees, both of QOPís significant impacts on 

female enrollees were beneficial. QOP significantly increased by nine percentage points the likelihood 

that a female enrollee graduated from high school. This impact was five percentage points higher but 

not significantly different from the impact on males. QOPís other significant impact on females was a 

nine-percentage-point increase in the likelihood of engaging in postsecondary education or training, 

attending high school or a GED class, or working. This impact was significantly different from the 

(insignificant) two-percentage-point decrease for males. Because of this difference in the impacts for 

males and females, the impact for all QOP enrollees was smaller than the impact for females and not 

significantly different from zero. As shown in Table VII.1, QOPís impact on females was significantly 

different from its impact on males for only this one key outcome. 

Although QOP had significant impacts on four outcomes for all enrollees, it had a significant 

impact on just one of the outcomes for either males or females. The main reason is that the sample size 

for each subgroup is substantially smaller (by about 50 percent) than the size of the entire evaluation 

sample, generally reducing the precision of impact estimates and making it more difficult to be confident 

that an estimated impact was significantly different from zero. 

SHORT-TERM IMPACTS BY AGE WHEN ENTERING NINTH GRADE

About one-third of the youth in the QOP demonstration were over age 14 when they entered ninth 

grade, and QOP had two significant impacts on these older enrollees (Table VII.2). QOP increased by 

10 percentage points the likelihood that an older enrollee engaged in postsecondary education or 

training, attended high school or a GED class, or worked. The program reduced by 11 percentage 

points the likelihood that an older enrollee had ever been arrested or charged with a crime. Although 

the impacts for younger enrollees were 9 and 8 percentage points smaller, respectively, than the impacts 

for older enrollees, the differences were not statistically significant. 

 

For younger enrollees, QOP significantly increased by 12 percentage points the likelihood of 

earning a high school diploma. This impact was not significantly different from the impact for older 

enrollees. QOP significantly increased by 7 percentage points the likelihood of engaging in 

postsecondary education or training and the likelihood of college attendance or acceptance for younger 

enrollees. The impacts on these two outcomesólike the impact on high school graduationówere also 

significant for all enrollees (about two-thirds of whom are younger enrollees). 

There was one significant difference between the impacts for older and younger enrollees. QOP 

significantly decreased by nine percentage points the likelihood that a younger enrollee had a child. In 
55 

contrast, QOP increased by six percentage points the likelihood that an older enrollee had a child. 

Although this impact for older enrollees was not significantly different from zero, it was significantly 

different from the impact for younger enrollees. 

Among the younger enrollees, nearly 85 percent were age 14, and the rest were age 13. Among the 

older enrollees, nearly 85 percent were age 15, and almost all of the rest were age 16. 

SHORT-TERM IMPACTS BY RANK IN THE BASELINE GRADE DISTRIBUTION

When assessing impacts for the subgroups defined by rank in the baseline grade distribution, it is 

important to remember that to be eligible for QOP, a youth had to be in the bottom two-thirds of the 

grade distribution based on grades from the eighth grade. Thus, youth in the bottom third of the 

baseline grade distribution for QOP eligibles were at or below the 22nd percentile in the distribution for 

all youth, including those who were not eligible for QOP based on their grades. Likewise, the youth in 

the middle and top thirds of the baseline grade distribution for QOP eligibles were between the 22nd and 

the 44th percentiles and between the 44th and the 66th percentiles, respectively, in the grade distribution 

for all youth. 

Across these three subgroups of enrollees, QOPís impacts varied (Table VII.3). The program was 

more successful for enrollees in the middle of the distribution than for enrollees at the top or bottom of 

the distribution. 

 

QOP had just one significant impact on enrollees in the top third of the baseline grade distribution. 

 It increased by eight percentage points the likelihood of binge drinking. The impacts on other 

outcomes pertaining to risky behaviors were also detrimental, but those impacts were not significant. 

For enrollees in the bottom third of the baseline grade distribution, QOP increased by 9 percentage 

points the likelihood of engaging in postsecondary education or training and decreased by 11 percentage 

points the likelihood of ever being arrested or charged with a crime. In contrast to these significant 

beneficial impacts, QOP increased by 14 percentage points the likelihood of using an illegal drug. This 

detrimental impact was significantly different from zero and from the impact on enrollees in the top 

two-thirds of the grade distribution. For enrollees in the bottom third of the grade distribution, QOPís 

detrimental impact on attending or being accepted into a college was also significantly different from the 

impact for other enrollees, although it was not significantly different from zero. 

All of the significant impacts on enrollees in the middle third of the baseline grade distribution were 

beneficial. The program increased both high school graduation and completion rates. It increased by 14 

percentage points the likelihood of earning a diploma and by 11 percentage points the likelihood of 

earning a diploma or a GED certificate. QOP increased by 13 percentage points the likelihood of 

attending or being accepted into college, and it decreased by 8 percentage points the likelihood of 

having a child. All four of these impacts are significantly different from zero. They are larger by at least 

several percentage points but not significantly different from the impacts for other enrollees. In 

contrast, the impacts on binge drinking and drug use by enrollees in the middle third of the grade 

distribution are significantly different from the impacts for other enrollees. However, the reductions of 

four percentage points in binge drinking and two percentage points in drug use are not significantly 

different from zero. 
56 

SHORT-TERM IMPACTS BY SITE

The impacts for the whole QOP demonstration were substantiallyóbut not entirelyóattributable 

to the impacts of the Philadelphia site alone or, as we discuss in the next section, the two Ford-funded 

sites, Philadelphia and Yakima. Although many of the seemingly large differences among the impacts 

shown in Table VII.4 are not significant because the sample sizes for each site are fairly small, 

Philadelphiaís impact was the largest or about equal to the largest for all of the outcomes pertaining to 

postsecondary activities and for four of the five outcomes pertaining to risky behaviors. Nevertheless, 

according to estimates for the other six sites combined (not shown in Table VII.4), those six sites had 

two significant impacts, and both were beneficial. The six sites increased by seven percentage points the 

likelihood of receiving a diploma and decreased by six percentage points the likelihood of ever being 

arrested or charged with a crime. For both of these outcomes, Philadelphiaís impact was not 

significantly different from the impact for the other six sites, as documented in Table VII.4. 

The Philadelphia site had two significant beneficial impacts and two significant detrimental impacts. 

The beneficial impacts were a 19-percentage-point increase in the likelihood of engaging in 

postsecondary education or training and a 22-percentage-point increase in the likelihood of engaging in 

postsecondary education or training, attending high school or a GED class, or working. The latter 

impact was significantly different from the impact for the other six sites combined. The two significant 

detrimental impacts for the Philadelphia site were also significantly different from the impacts for the 

other six sites combined. The Philadelphia site increased by 17 percentage points the likelihood of binge 

drinking and by 18 percentage points the likelihood of committing a crime. 

Examining the QOP-group and control-group means for some of the risky behaviors in Tables 

VII.5 and VII.6 suggests, however, that it is possibleóand maybe likelyóthat such detrimental impacts 

were not caused by QOP. The estimated means reveal that there might have been differences between 

QOP enrollees and control-group youth in the accuracy with which they reported risky behaviors, and 

those differences might have contributed substantially to the estimated detrimental impacts on risky 

behaviors for some sites and the demonstration as a whole. Specifically, some of the control-group 

means pertaining to drinking, drug use, and crime were unusually and, perhaps, implausibly low. In the 

Philadelphia site, only 3 percent and 12 percent of control-group youth reported having a drink or 

taking an illegal drug, respectively, in the 30 days before the in-person survey. In the other six sites 

combined, the rates of drinking and drug use among control-group youth were substantially higheró38 

percent and 30 percent, respectively. QOPís impact on drinking was a significant 30-percentage-point 

increase in the Philadelphia site and an insignificant 3-percentage-point increase in the other six sites 

combined. For drug use, the respective impacts were insignificant 13- and 5-percentage-point increases. 

 In addition to the relatively low rates of drinking and drug use among control-group youth in the 

Philadelphia site, 14 percent of those youth reported committing a crime in the year before the inperson survey, while the fraction was much higheró31 percentófor control-group youth in the other 

six sites combined. While QOPís impact on crime was a significant 18-percentage-point increase in the 

Philadelphia site, it was an insignificant 1-percentage-point increase in the other six sites combined. 

Finally, while the rates of binge drinking among control-group youth were 2 percent and 6 percent in 

the Philadelphia and Memphis sites, respectively, the rate for the other five sites combined was 27 

percent. Like the Philadelphia site, the Memphis site had a significant detrimental impact on binge 

drinking according to the in-person survey data. 

 
57 

As suggested above, these results might have been attributable to differences in the accuracy with 

which QOP enrollees and control-group youth reported risky behaviors. An alternative explanation is 

that the results were due to purely random baseline differences between the two groups for which we 

could not statistically adjust because the differences were not associated with any of the very limited 

number of baseline characteristics that could be measured. Regression-adjusted impact estimates for the 

whole demonstration are presented in Appendix F. 

The patterns of differences in control-group means just discussed and the Philadelphia and 

Memphis sitesí significant detrimental impacts on some risky behaviors were observed in data from the 

in-person survey. According to the telephone survey data, neither the Philadelphia site nor the 

Memphis site had significant detrimental impacts on risky behaviors (Table VII.7). However, neither 

site significantly reduced such behaviors. Also, while drug use increased in the Houston site by an 

insignificant 10 percentage points according to the in-person survey data, it increased by a significant 15 

percentage points according to the telephone survey data. 

According to Table VII.4, the Cleveland site had three significant impacts on enrollees, and all three 

were beneficial. The program in Cleveland increased by 13 percentage points both the likelihood of 

earning a diploma and the likelihood of college attendance or acceptance. It decreased by 16 percentage 

points the likelihood of binge drinking among enrollees. 

The Fort Worth; Washington, D.C.; Houston; and Memphis sites did not have any beneficial 

impacts that were significantly different from zero, although the 13-percentage-point decrease in the 

likelihood of binge drinking by enrollees in Fort Worth was significantly different from the impact on 

binge drinking for the other six sites combined. The Washington, D.C.; Houston; and Memphis sites 

had one significant detrimental impact. In the Washington, D.C., site, the detrimental impact was a 

decrease of over 2 percentile points in the average mathematics achievement test score. In the Houston 

site, the detrimental impact was a one-quarter-point decrease in the average GPA. And, in the Memphis 

site, QOP youth were 18 percentage points more likely than the control-group youth to engage in binge 

drinking, although this impact might not be attributable to QOP, as noted above. The Yakima site had 

one impact that was significantly different from zero, and it was beneficialóan increase of over 3 

percentile points in the average mathematics achievement test score. However, the decrease of 13 

percentage points in the likelihood that Yakima enrollees engaged in postsecondary education or 

training, attended high school or a GED class, or worked is significantly different from the impact on 

this outcome for the other six sites combined. 

SHORT-TERM IMPACTS BY FUNDING SOURCE

The five DOL-funded QOP sites collectively had just one significant impact, and it was beneficial 

(Table VII.8). The sites increased by seven percentage points the likelihood that an enrollee graduated 

from high school. This estimated impact on one of QOPís primary outcomes was not significantly 

different from the estimated impact for the two Ford-funded sites, which was also an increase of seven 

percentage points. All of the estimated impacts on postsecondary activities and risky behaviors were 

within five percentage points of zero for the DOL-funded sites. 

In contrast, the Ford-funded QOP sites had seven significant impacts. Four of those impacts were 

beneficial, and three were detrimental. Five of the seven impacts were significantly different from the 

impacts for the DOL-funded sites. 
58 

The Ford-funded sites increased by nearly 2 percentile points the average mathematics achievement 

test score. They also increased by 14 percentage points the likelihood of engaging in postsecondary 

education or training and by 17 percentage points the likelihood of engaging in postsecondary education 

or training or working at a good job, that is, a job offering employer-sponsored health insurance. 

The Ford-funded sites had significant impacts on four of the five risky behaviors listed in Table 

VII.8, and one of those impacts was beneficial. The two sites together decreased by 14 percentage 

points the likelihood that an enrollee had a child. In contrast, they increased by 17, 14, and 16 

percentage points the likelihood of binge drinking, using an illegal drug, and committing a crime, 

respectively. As discussed above, these detrimental impacts might be partly attributable to unusually low 

control-group means in the Philadelphia site and not to QOP. However, the detrimental impacts in the 

Yakima site were about the same size as the detrimental impacts in the Philadelphia site, although the 

impacts in the Yakima site were not statistically significant. 
59 

TABLE VII.1 

SHORT-TERM IMPACTS BY SEX 

(Percentage points except where noted) 

Impacts 

Outcomea Male Female Total Sample 

Mathematics test score (percentile) 0.72 0.08 0.38 

Reading test score (percentile) 1.10 0.06 0.50 

GPA (four-point scale) -0.13** -0.02 -0.06 

Earned high school diploma 4 9* 7* 

Earned diploma or GED certificate 0 7 5 

Attending college 3 2 3 

Attending postsecondary training 7 4 6*

Postsecondary training or good job 4 6 5 

Postsecondary training or high school or GED 

class or any job -2Ü 9Ü,** 3 

Attending or accepted into college 4 5 6*

Binge drinking in past 30 days 7* 0 4 

Used any illegal drug in past 30 days 7 7 7**

Committed any crime in past 12 months 5 2 3 

Ever arrested or charged -8 0 -5 

Have one or more own children -3 -5 -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP enrollees 

and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year 

or a four-year college. ìPostsecondary trainingî means college, vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers 

employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 

Ü Significantly different from the impact on all other youth at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
60 

TABLE VII.2 

SHORT-TERM IMPACTS BY AGE WHEN ENTERING NINTH GRADE 

(Percentage points except where noted) 

Impacts 

Outcomea Age > 14 Age ≤ 14 Total Sample 

Mathematics test score (percentile) 0.55 0.72 0.38 

Reading test score (percentile) 0.96 0.67 0.50 

GPA (four-point scale) -0.06 -0.02 -0.06 

Earned high school diploma 0 12*** 7* 

Earned diploma or GED certificate 6 7 5 

Attending college 3 5 3 

Attending postsecondary training 7 7* 6*

Postsecondary training or good job 8 6 5 

Postsecondary training or high school or GED 

class or any job 10* 1 3 

Attending or accepted into college 5 7* 6*

Binge drinking in past 30 days 5 4 4 

Used any illegal drug in past 30 days 8 5 7**

Committed any crime in past 12 months -4 5 3 

Ever arrested or charged -11* -3 -5 

Have one or more own children 6Ü -9Ü,** -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP enrollees 

and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year 

or a four-year college. ìPostsecondary trainingî means college, vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers 

employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 

Ü Significantly different from the impact on all other youth at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
61 

TABLE VII.3 

SHORT-TERM IMPACTS BY RANK IN THE BASELINE GRADE DISTRIBUTION 

(Percentage points except where noted) 

 Impacts 

Outcomea Bottom Third Middle Third Top Third 

Total 

Sample 

Mathematics test score (percentile) 0.06 0.28 0.04 0.38 

Reading test score (percentile) 0.27 -0.17 0.91 0.50 

GPA (four-point scale) -0.13 -0.06 0.03 -0.06 

Earned high school diploma 3 14** 4 7* 

Earned diploma or GED certificate 1 11* 3 5 

Attending college -2 7 4 3 

Attending postsecondary training 9* 3 4 6*

Postsecondary training or good job 7 9 -4Ü 5 

Postsecondary training or high school or GED 

class or any job 8 1 0 3 

Attending or accepted into college -3Ü 13** 9 6*

Binge drinking in past 30 days 8 -4Ü 8* 4 

Used any illegal drug in past 30 days 14Ü, ** -2Ü 7 7**

Committed any crime in past 12 months 2 4 8 3 

Ever arrested or charged -11* 0 1 -5 

Have one or more own children -4 -8* 3 -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP enrollees 

and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year 

or a four-year college. ìPostsecondary trainingî means college, vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers 

employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 

Ü Significantly different from the impact on all other youth at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
62 

TABLE VII.4 

SHORT-TERM IMPACTS BY SITE 

(Percentage points except where noted) 

Impacts 

Outcomea Fort Worth Cleveland D.C. Houston Memphis Philadelphia Mathematics test score (percentile) 0.16 0.39 -2.14Ü, ** 0.58 -0.21 0.69 Reading test score (percentile) -0.46 -0.43 1.85 0.47 -0.39 1.25 GPA (four-point scale) 0.04 -0.10 -0.15 -0.25Ü, ** 0.07 -0.13 

Earned high school diploma 2 13* 12 3 1 10 

Earned diploma or GED certificate 6 7 6 -2 4 10 Attending college 3 8 -9Ü 3 2 13 

Attending postsecondary training 6 2 -2 1 6 19* 

Postsecondary training or good job -1 -3 4 -3 2 17 Postsecondary training or high school or 

GED class or any job 9 -4 12 4 -6Ü 22Ü, ** 

Attending or accepted into college 2 13* -2 2 4 17 

Binge drinking in past 30 days -13Ü -16Ü, * -3 9 18Ü, *** 17Ü, ** Used any illegal drug in past 30 days 8 -6 -1 10 8 13 

Committed a crime in past 12 months -5 -7 6 2 -7 18Ü, * 

Ever arrested or charged -4 1 -12 -1 -11 3 Have one or more own children -3 3 -4 4 6 -15 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact might not equal the differen

displayed in Tables VII.5 and VII.6. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year or a four-year college. ì

vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. Ü Significantly different from the impact for all other sites at the 90% confidence level, two-tailed test * Estimate significantly different from zero at the 90% confidence level, two-tailed test ** Estimate significantly different from zero at the 95% confidence level, two-tailed test *** Estimate significantly different from zero at the 99% confidence level, two-tailed test
63 

TABLE VII.5 

QOP-GROUP MEANS BY SITE 

(Percentages except where noted) 

Means 

Outcomea Fort Worth Cleveland D.C. Houston Memphis Philadelphia Mathematics test score (percentile) 44.5 40.6 38.5 39.3 38.2 37.0 Reading test score (percentile) 45.6 43.1 43.4 41.4 41.9 41.0 GPA (four-point scale) 2.65 1.86 1.79 2.16 2.09 2.04 Earned high school diploma 67 36 35 32 50 57 Earned diploma or GED certificate 74 42 41 39 60 57 Attending college 25 15 19 15 19 31 Attending postsecondary training 35 24 28 22 30 42 Postsecondary training or good job 58 40 41 39 53 49 Postsecondary training or high school or 

GED class or any job 91 84 84 83 83 85 

Attending or accepted into college 31 26 33 21 32 43 Binge drinking in past 30 days 26 17 12 25 24 19 Used any illegal drug in past 30 days 29 34 36 39 34 25 Committed a crime in past 12 months 30 34 28 29 29 32 Ever arrested or charged 21 35 20 23 21 30 Have one or more own children 12 27 18 31 30 29 SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year or a four-year college. ì

vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 
64 

TABLE VII.6 

CONTROL-GROUP MEANS BY SITE 

(Percentages except where noted) 

Means 

Outcomea Fort Worth Cleveland D.C. Houston Memphis Philadelphia Mathematics test score (percentile) 44.4 40.2 40.7 38.7 38.5 36.4 Reading test score (percentile) 46.0 43.5 41.5 41.0 42.3 39.8 GPA (four-point scale) 2.61 1.96 1.94 2.41 2.02 2.17 Earned high school diploma 64 23 22 29 49 47 Earned diploma or GED certificate 67 34 35 41 56 47 Attending college 21 7 28 12 17 18 Attending postsecondary training 30 22 31 20 24 23 Postsecondary training or good job 58 43 36 42 51 32 Postsecondary training or high school or 

GED class or any job 82 88 71 80 89 64 

Attending or accepted into college 29 13 35 19 29 25 Binge drinking in past 30 days 39 34 16 17 6 2 Used any illegal drug in past 30 days 21 41 37 29 26 12 Committed a crime in past 12 months 35 41 23 27 35 14 Ever arrested or charged 25 34 31 24 31 28 Have one or more own children 15 24 23 27 24 44 SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year or a four-year college. ì

vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 
65 

TABLE VII.7 

SHORT-TERM IMPACTS ON RISKY BEHAVIORS BY SITE, TELEPHONE SURVEY 

Outcomea Fort Worth Cleveland D.C. Houston Memphis Philadelphia 

QOP-Group Mean (percentage) 

Binge drinking in past 30 days 26 11 9 29 16 5 Used any illegal drug in past 30 days 8 24 18 21 14 12 Committed a crime in past 3 months 8 13 6 12 13 5 Arrested or charged in past 3 months 2 11 3 9 1 2 

Control-Group Mean (percentage) 

Binge drinking in past 30 days 27 32 13 28 19 8 Used any illegal drug in past 30 days 15 31 32 6 20 13 Committed a crime in past 3 months 10 15 10 7 8 13 Arrested or charged in past 3 months 3 14 4 5 8 5 

Impact (percentile points) 

Binge drinking in past 30 days -1 -21Ü, *** -4 1 -2 -3 Used any illegal drug in past 30 days -7 -8 -14 15Ü, *** -6 -1 

Committed a crime in past 3 months -2 -1 -3 5 5 -8 Arrested or charged in past 3 months -1 -4 -1 4 -7Ü, ** -3 

SOURCE: Telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an impact might not equal the differen

displayed. The evaluation sample had 580 QOP enrollees and 489 controls. 

a ìBingeî drinking means five or more drinks in a row. 

Ü Significantly different from the impact for all other sites at the 90% confidence level, two-tailed test * Estimate significantly different from zero at the 90% confidence level, two-tailed test ** Estimate significantly different from zero at the 95% confidence level, two-tailed test *** Estimate significantly different from zero at the 99% confidence level, two-tailed test
66 

TABLE VII.8 

SHORT-TERM IMPACTS BY FUNDING SOURCE 

(Percentage points except where noted) 

Impacts 

Outcomea Ford-Funded Sites DOL-Funded Sites Total Sample 

Mathematics test score (percentile) 1.96Ü, * -0.24Ü 0.38 

Reading test score (percentile) 1.27 0.21 0.50 

GPA (four-point scale) -0.02 -0.08 -0.06 

Earned high school diploma 7 7* 7*

Earned diploma or GED certificate 6 4 5 

Attending college 8 1 3 

Attending postsecondary training 14* 3 6*

Postsecondary training or good job 17Ü, ** 0Ü 5 

Postsecondary training or high school or GED class or any 

job 4 3 3 

Attending or accepted into college 11 3 6*

Binge drinking in past 30 days 17Ü, *** -1Ü 4 

Used any illegal drug in past 30 days 14** 4 7**

Committed any crime in past 12 months 16Ü, ** -2Ü 3 

Ever arrested or charged -3 -5 -5 

Have one or more own children -14Ü, ** 1Ü -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP enrollees 

and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. ìCollegeî means either a two-year 

or a four-year college. ìPostsecondary trainingî means college, vocational/technical school, apprenticeship, or armed forces. A ìgoodî job offers 

employer-sponsored health insurance. ìBingeî drinking means five or more drinks in a row. 

Ü Significantly different from the impact for sites with the other funding source at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
67 

R EFERENCES

Aquilino, William S. ìInterview Mode Effects in Surveys of Drug and Alcohol Use.î Public Opinion 

Quarterly, vol. 58, 1994, pp. 210ñ240. 

Bloom, Howard. ìAccounting for No-Shows in Experimental Evaluation Designs.î Evaluation Review, 

vol. 8, 1984. 

Carlson, Barbara, Brenda Cox, and Linda Bandeh. ìSAS Macros Useful in Imputing Missing Survey 

Data.î Proceedings of the Twentieth Annual SAS Users Group International Conference. Cary, NC: SAS 

Institute, Inc., 1995. 

Maxfield, Myles, Laura Castner, Vida Maralani, and Mary Vencill. ìThe Quantum Opportunity Program 

Demonstration: Implementation Findings.î Washington, DC: Mathematica Policy Research, Inc., 

2003a. 

Maxfield, Myles, Allen Schirm, and Nuria Rodriguez-Planas. ìThe Quantum Opportunity Program 

Demonstration: Implementation and Short-Term Impacts.î Washington, DC: Mathematica Policy 

Research, Inc., 2003b. 

Schochet, Peter Z., John Burghardt, and Steven Glazerman. ìNational Job Corps Study: The Impacts 

of Job Corps on Participantsí Employment and Related Outcomes.î Princeton, NJ: Mathematica 

Policy Research, Inc., 2001. 

 . ìNational Job Corps Study: The Short-Term Impacts of Job Corps on Participantsí 

Employment and Related Outcomes.î Princeton, NJ: Mathematica Policy Research, Inc., 2000. 

U.S. Department of Justice. ìGuide for Implementing the Comprehensive Strategy for Serious, Violent, 

and Chronic Juvenile Offenders.î Washington, DC: U.S. Department of Justice, 1995. 

U.S. General Accounting Office. ìAt-Risk and Delinquent Youths: Fiscal Year 1998 Programs.î 

Report no. GAO/HEHS-99-88R. Washington, DC: U.S. General Accounting Office, March 

1999. 
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
APPENDIX A

OBTAINING AN EVALUATION SAMPLE AND CONDUCTING RANDOM ASSIGNMENT
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
A-3

Four steps led up to and concluded with random assignment: (1) developing lists of eligibles, 

(2) initial sampling, (3) obtaining consent, and (4) random assignment. These steps needed to be

completed to obtain an evaluation sample for the QOP demonstration.

To implement the four steps in the seven sites, we developed an individualized Student

Selection Plan (SSP) for each site by customizing a generic plan to accommodate local

circumstances. Exhibit A.1 displays the generic plan. As it turned out, few accommodations to

local circumstances were required; therefore, all of the SSPs were similar. The main differences in 

the sites’ SSPs concerned the number of QOP schools, how QOP slots were allocated across

schools, and the dates of sampling and random assignment. In the three sites with more than one 

QOP school, the QOP CBO was responsible for determining how slots would be allocated across 

the schools.

Although random assignment was successfully implemented in the seven demonstration sites,

the sites encountered three main problems in implementing the evaluation design: (1) developing

accurate lists of eligibles, (2) contacting students, and (3) collecting completed forms. In the

remainder of this appendix, we discuss these implementation problems in the context of the four

steps listed earlier. Although we present examples from individual sites, the examples usually

illustrate experiences common to most or all sites.

______________________________________________________________________________

DEVELOPING LISTS OF ELIGIBLES

______________________________________________________________________________

As shown in the model SSP, the generic instruction to each site was as follows:

Each school should compile a list of students who have entered the 9th grade for the first time in the 

current academic year and send the list to MPR. For every student, the list should include at least 

two pieces of identifying information and the students’ 8th grade GPA.

Fulfillment of this instruction completed the site’s responsibility. Then:

For each school, MPR will rank students—from highest to lowest—according to their GPAs from 

the 8th grade. The students in the bottom two-thirds of the GPA distribution for their school are 

eligible.

Although seemingly straightforward, these first two steps in implementing the evaluation design 

were probably the most difficult. They might also prove to be among the more difficult steps in

implementing an ongoing QOP program. There were two main problems in developing an accurate 

list of eligibles for a school: (1) determining current enrollment and (2) calculating GPAs.

Determining Current Enrollment

As a rule, most QOP schools did not know precisely which students were enrolled in the ninth 

grade. The explanation was sporadic attendance by many students combined with high turnover,

both from year to year and within a year, as students’ families moved frequently.
A-4

EXHIBIT A.1

Quantum Opportunity Program Student Selection Plan

This plan outlines the steps for selecting students for the Quantum Opportunity Program

(Quantum). For each step, we have listed the responsibilities of local Quantum staff (including staff 

of the participating high schools) and the responsibilities of Mathematica Policy Research (MPR)

staff.

1. Submitting Lists of Students.

Quantum. Each school should compile a list of students who have entered the 9th grade 

for the first time in the current academic year and send the list to MPR. For every student, the 

list should include at least two pieces of identifying information and the student’s 8th grade

GPA.

2. Identifying Eligible Students.

MPR. For each school, MPR will rank students—from highest to lowest—according to 

their grade point averages (GPAs) from the 8th grade. The students in the bottom two-thirds 

of the GPA distribution for their school are eligible for Quantum.

3. Selecting a Group of Eligible Students to Receive Quantum Information and Consent

Packets.

MPR. MPR will randomly select a group of 132 eligible students from ABC High School 

and 88 eligible students from XYZ High School. MPR will send the list of selected students to 

Quantum staff on [date]. If permission is obtained from their parents, these students will be 

the study group. Only some (about half) of the students in the study group will later be

selected, at random, to participate in the Quantum program.

4. Distributing Quantum Information and Consent Packets.

Quantum. Quantum staff should distribute Quantum information and consent packets to 

all 220 students in the prospective study group. The packet will contain a cover letter from the 

student’s school, a brochure describing the Quantum program and the Quantum study, a

consent form seeking parental permission for the student to participate in the study, and a

locator form. Quantum staff should make copies of the cover letter (on school letterhead) and 

copies of the consent and locator forms and assemble the packets.

MPR. MPR will draft all materials for the Quantum information and consent packet. 

MPR will also make copies of the brochures and send these to Quantum staff.
A-5

EXHIBIT A.1 (continued)

5. Collecting Completed Consent and Locator Forms.

Quantum. Quantum staff should collect completed consent and locator forms. When a

student returns completed forms, Quantum staff should attach preprinted labels for that

student to the forms. It is important that completed consent and locator forms be obtained for 

all 132 students at ABC High School and 88 students at XYZ High School so that every

interested student will have an opportunity to be considered for participation in the Quantum 

program. Quantum staff will be responsible for purchasing an incentive item and distributing it 

to students who promptly return completed consent and locator forms.

MPR. MPR will provide two preprinted labels for each student, one label for the consent 

form and one label for the locator form. MPR will pay for the incentive.

6. Submitting Consent and Locator Forms.

Quantum. Completed consent and locator forms should be sent to MPR at least weekly.

7. Selecting Students for the Quantum Program.

MPR. MPR will compile a list of all students for whom affirmative consent and a

completed locator form have been obtained. The list will be sent to Quantum staff for

verification.

Quantum. After verifying that the list of students with affirmative consent and completed 

locator forms is correct, Quantum staff should sign the list and send it to MPR.

MPR. From the list of students with affirmative consent and completed locator forms,

MPR will randomly select 60 students from ABC High School and 40 students from XYZ High 

School to participate in the Quantum program. Students who are not randomly selected for the 

Quantum program will be assigned to the control group for the study. On [date], MPR will

send lists of Quantum group students and control group students to Quantum staff.

Quantum. Quantum staff should notify all students about their group assignments 

(Quantum or control), and should inform MPR when all students have been notified. After

notifying Quantum students of their selection, Quantum staff should begin recruiting them for 

participation in the Quantum program. Only students randomly selected for the Quantum

group may participate in the Quantum program. Students assigned to the control group and

students who did not receive or did not complete consent and locator forms cannot participate 

in the Quantum program. All students in the Quantum and control groups are part of the

Quantum study.
A-6

EXHIBIT A.1 (continued)

8. Submitting Lists of Quantum Participants.

Quantum. To provide data for analyses of Quantum participation patterns, Quantum

staff should send to MPR a list of all students participating in the Quantum program on the

following dates: .... After [date], a list of Quantum participants should be submitted every

twelve weeks.

If this plan meets with your approval, please sign below and return to MPR. If you have any 

questions concerning this plan or any other issues related to the study, please call [MPR site liaison] 

at [phone number]. Thank you for your assistance in developing this plan.

_______________________________ _______________________

Quantum Coordinator Date
A-7

Although we considered requesting first-day-of-school enrollment lists, we learned from school 

and district staff that such lists would be unreliable.1 Many students expected to enroll in a school

do not do so, and many unexpected students enroll. Moreover, some students do not attend school 

for the first few weeks of the year, especially if school starts before Labor Day.

In lieu of a first-day-of-school enrollment list, we accepted the first properly constructed list

(with grades) that a school could produce. Such a list typically became available a month or more

after school started. 2

Even several weeks into the school year, however, students continued to transfer from school 

to school, and some students had attended classes on only a few days. For example, five weeks into 

the school year, one QOP school constructed a list of ninth graders who were not repeating the

ninth grade and were not ineligible because of a disability. The school constructed a second list of 

such students two weeks later. Nearly one out of every six students on the first list was not on the 

second list. However, out of every five students dropped, one was replaced by a new student. We 

suspect that many of the students that were dropped had left the school before the construction of 

the first list and that school record keeping was just catching up to student movements. 

Nevertheless, reports by school and QOP staff suggested that some dropped students and some

added students probably had moved during the two-week period between lists.

Once a school had a list of currently enrolled ninth graders, “categorically ineligible” students—

students repeating the ninth grade and disabled students for whom QOP would have been

inappropriate in the school’s judgment—had to be dropped from the list. Although a couple of

schools neglected to drop a category of ineligible students in a first attempt to develop a list of

eligibles, none of the schools in the demonstration appeared to have any significant difficulties in

identifying categorically ineligible students.

Calculating GPAs

After developing a list of currently enrolled ninth graders and dropping from the list

categorically ineligible students, a school attempted to calculate an eighth-grade GPA for each

remaining student on the list. Initial conversations with school staff revealed confusion about what 

would constitute an acceptable GPA. Some thought that GPA means a credit-weighted average on 

a four-point scale. We were told, for example, that it would not be possible to obtain GPAs for one 

school because only “grade averages” (on a 100-point scale) were available. Such confusion was

easily eliminated by distributing a brief memorandum discussing the calculation of GPAs and other 

issues pertaining to eligibility.3

 

1Even if first-day-of-school lists had been more reliable, schools generally were not prepared to produce them

because doing so would have interfered with regular school activities.

2If an ongoing QOP program were to start delivering services very near the beginning of the school year, the

proportion of students selected for QOP who turned out to have transferred to other schools would be much higher 

than in this demonstration, in which service delivery started almost half way through ninth grade. Also, many (if not 

most) students new to the school district or coming from middle schools within the district that are not traditionally 

feeder schools for the QOP high school would effectively be ineligible for QOP. As we discuss later, however, even 

when lists are constructed several weeks into the school year, many students new to a district are ineligible for QOP

because no grades are available for them.

3Some school staff were also confused about how to rank students based on grades. One school initially had a

separate ranking for each middle school that fed students to the high school. We eliminated the confusion by having 
A-8

Although it might be more serious if QOP were a permanent rather than a demonstration

program, another minor problem was that two schools did not have the resources to calculate

GPAs. For one school, QOP staff calculated GPAs from students’ eighth-grade transcripts. For

the other school, we performed the necessary calculations.4

The most serious problem that arose in attempting to calculate GPAs was obtaining eighthgrade transcripts for students who were new to the local public school system after transferring from 

other school systems or private schools. The typical procedure for calculating GPAs involved two 

steps. First, district and school staff obtained GPAs for as many students as possible from a

computerized database. That database rarely included grades for students new to the system. 

Second, if the database contained no grades, QOP staff searched a student’s paper files for an

eighth-grade transcript. If a transcript were available, QOP staff calculated a GPA by hand.5 More 

often than not, however, no transcript appeared in a student’s file.

For one QOP school, no grades were available in the district’s database for nearly 17 percent of 

students. QOP staff were able to locate an eighth-grade transcript for only 20 percent of those

students. So, overall, GPAs could be calculated for just 87 percent of the school’s categorically

eligible students.6

The consequences of this problem were borne by students. Because there was no basis for

ranking students for whom a GPA could not be calculated, such students were ineligible for QOP. 

Thus, potentially many students who were experiencing the difficulties of entering a new school

system had no opportunity to enroll in QOP because their transcripts were less mobile then they

were.

While problems arose in determining enrollment and calculating GPAs, we should note that in 

some sites accomplishing both of those tasks seemed more than twice as difficult as accomplishing 

either one of them. The problem was that information in a school system was dispersed. The QOP 

school had more accurate enrollment data than the central district office but much less easy access 

(if any access) to computerized records of grades.7 Moreover, there was rarely one person who had 

a good working knowledge of each data source. This problem was made worse by the fact that the 

most knowledgeable person generally did not have the authority to take direction from a third party 

(us or the CBO) or to make judgments such as whether a particular special education student should 

be eligible for QOP. Yet another obstacle was that schools often had little experience in responding 

 

(continued)

each QOP school send us a list with names and grades for all categorically eligible students. Then, we ranked students 

and identified the (fully) eligible students, that is, the students in the bottom two-thirds of the grade distribution.

4For another school, QOP staff entered GPAs from students’ transcripts into a database.

5The main difficulty in this case was making sure that the GPA was comparable to other students’ GPAs—that it 

was, for example, on the same scale.

6In two other schools, GPAs could be calculated for 88 and 65 percent of categorically eligible students. For the 

first school, QOP staff had to track down GPAs for about one in six students for whom GPAs could be calculated. For 

the other school, it was two in five.

7For one QOP school, an enrollment list prepared by the central district office missed three-fifths of the students 

on the school’s own enrollment list. At the same time, over one-quarter of the students on the district-prepared list were 

no longer enrolled according to the school’s list. For another school, the differences were less extreme, but still large. 

The district’s list missed one-quarter of the students on the school’s list, while about one-sixth of the students on the 

district’s list were not on the school’s list.
A-9

to information requests such as those that we made. A final obstacle was that despite enthusiasm 

for QOP and a cooperative spirit on the part of school and district staff, determining which students 

were eligible for QOP was generally not a high priority. Thus, the resources needed to do the job 

accurately were not always available.8

We discovered many errors in some lists submitted to us and returned the lists to the schools 

for corrections.9 Nevertheless, because little information was available for assessing the accuracy of 

the lists, we are certain that the final lists contained errors, some of which were discovered later in 

the process of obtaining an evaluation sample. Only by requesting more data and further burdening 

the schools could the numbers of errors have been determined and reduced substantially.

______________________________________________________________________________

INITIAL SAMPLING

______________________________________________________________________________

In all but two QOP schools, we drew a simple random sample (without replacement) from the 

list of students eligible for QOP. The selected students were eligible for random assignment if

consent was obtained for them to participate in the evaluation. The students who were not selected 

for the initial sample were not eligible for random assignment and therefore no longer had an

opportunity to participate in QOP. We did not draw random samples for two schools because the 

number of eligible students was less than the target sample size. We conducted sampling

independently for each school.

The initial sampling of eligibles had two purposes: (1) to minimize the impact of the evaluation 

on students and (2) to minimize the burden on QOP staff. Although such concerns about impact 

and burden arise in every random assignment evaluation, they were heightened in the QOP

demonstration because in several of the QOP schools, the number of eligible students was

substantially greater than the target size of the evaluation sample (100 in the Ford-funded sites and 

200 in all but one of the DOL-funded sites). Thus, there were many extra students who would not 

be selected for the limited number of QOP slots (50 in the Ford-funded sites and 100 in all but one 

of the DOL-funded sites) and were not needed to form a control group for the evaluation. Locating 

those students, telling them and their parents about QOP and the evaluation, and obtaining consent 

for them to participate in the evaluation would have substantially increased the workload of QOP

staff. Moreover, many more students than necessary would have had their hopes raised, only to be 

disappointed later. Sampling limited the number of disappointed students.

Once we decided to sample eligible students, we had to determine the size of the sample. We 

wanted to obtain a control group for each school that was the same size as the QOP group,

implying a target sample size that was twice the number of available QOP slots. However, if we had 

 

8For school staff, the highest priority was running the school. When attention was given to QOP, the highest

priority of school and QOP staff was, understandably, serving students. Promoting fairness by ensuring the accuracy of 

the list of eligible students, most of whom would not be served by QOP, was a lower priority.

9The most common errors were excluding students new to the school system and including repeaters. On lists

submitted by one site, for example, we discovered that new students had been excluded. We discovered this by

observing that not a single student had attended eighth grade in a school outside of the city. For one school, which had 

grades 9 through 12, we noticed that several students had attended that school the previous year, suggesting that

repeaters had been included.
A-10

drawn a sample with as many students as the target size of the evaluation sample, we would have 

had no surplus to allow for students who left the QOP school between development of the school 

enrollment roster and sampling (because they transferred, dropped out, or were expelled); for

students who simply could not be located (or, if located, could not be contacted); and for students

for whom consent was denied (explicitly or, by nonresponse, implicitly). Losing those students and 

dropping below the target size for the evaluation sample because we had no surplus would have

reduced the precision of impact estimates. On the other hand, if we had a generous surplus, we 

would have disappointed more students than necessary and excessively burdened QOP staff.

After weighing these considerations, we drew for each school a sample of eligible students that 

was 10 percent larger than the target size for the evaluation sample. Accordingly, if a CBO in a

DOL-funded site with two QOP schools specified that one school would have 60 QOP slots while 

the other would have 40, we drew a sample of 132 (= 60 H 2 H 1.1) students for the first school and 

88 (= 40 H 2 H 1.1) students for the second school.10 There were two exceptions to this rule for

setting the sample size. First, if (the number of QOP slots H 2 H 1.1) was greater than the number 

of eligible students in a school, we selected all of the eligible students. Second, last-minute changes 

in the allocation of QOP slots across the Memphis schools caused minor deviations from the

formula.11

Our sample size choice was a compromise between the ideal of randomly selecting a sample of 

eligible students and getting consent for every one of them and the reality that it could not be done. 

To emphasize the importance of reaching out to every eligible student—regardless of the student’s

initial interest in QOP—as a fundamental principle of the program, we instructed QOP staff to

make every reasonable effort to obtain a completed consent form for each student in the sample. In 

addition, as we discuss later, we imposed safeguards to ensure that such efforts were undertaken.

______________________________________________________________________________

OBTAINING CONSENT

______________________________________________________________________________

After selection of the initial sample for each school, QOP staff attempted to distribute

information packets to each selected student. The packets contained a cover letter from the

student’s school (usually signed by the principal), a brochure describing the program and the

evaluation, and a parental consent form for the evaluation.12 In addition to collecting completed

consent forms, QOP staff were responsible for having students and their parents complete a

“locator” form that would provide tracking information to enable us to contact students for followup data collection. All but one site chose to include the locator form in the packet with the other 

materials.

 

10The factor of 2 in the mathematical expressions reflects the fact that we wanted to obtain a control group that

was the same size as the QOP group. The QOP group had as many students as there were QOP slots.

11To avoid any further delays in enrolling students in QOP, we did not draw a supplemental sample if the 10

percent surplus in the original sample turned out to be too small. Instead, we allowed the control group to be smaller 

than the QOP group.

12 Spanish language materials were available. 
A-11

Although sites varied in how they distributed and collected completed consent and locator

forms, a typical approach involved the following four steps: (1) hold an in-school assembly to speak 

with students and distribute packets; (2) try to find at the school the students who did not attend the 

assembly; (3) request that students return completed forms to a specified location (usually an office 

in the school); and (4) follow up with telephone calls and, more often, home visits to meet with

parents and obtain completed forms. QOP staff carried out these steps, sometimes with limited

assistance from school staff.13

The home visits were especially important in obtaining completed forms from as many students 

in the sample as possible. First, a home visit was the first contact with a nontrivial fraction of

students who attended school sporadically. Second, it was often the most reliable means of getting 

forms delivered to a parent, completed, and returned to QOP staff.

To expedite the process of obtaining consent, all sites offered a nominal incentive, such as

movie theater or grocery store gift certificates, for returning completed forms promptly. 

Nevertheless, obtaining consent was difficult and time-consuming. For the median student, one

month elapsed between the time when the student was selected for the sample of students who

could receive information packets and the time when we received a completed consent form for that 

student.14 For 17 percent of students, more than seven weeks elapsed.

Two of the implementation problems mentioned earlier arose in the process of obtaining

consent and explain why the process was so difficult and time-consuming. These problems were (1) 

contacting students and (2) collecting completed forms.

Every site encountered difficulties in locating and contacting a substantial fraction of students. 

The main reason was that the students’ families moved frequently, which was an explanation noted 

earlier for why schools had trouble in determining their current enrollment. For some of these 

students, QOP staff learned that after the school constructed the enrollment list used for identifying 

eligibles and drawing the initial sample, the students quit attending the QOP school, often because 

they had moved and transferred to another school. For students still thought to be living nearby

and enrolled in the QOP school, QOP staff often discovered that the contact information contained 

in school records was badly out of date. Sometimes, the information was current but inaccurate,

referring, for example, to a nonexistent address. Using various means, such as talking with a

student’s friends, QOP staff were often able to determine where a student lived. However, it was 

still difficult to contact some students’ families because there was no telephone in the home, no

adult was at home much of the time, or a convenient meeting time could not be arranged.

Problems did not end when contact was made with a student. An information packet given to a 

student often was not delivered to the student’s parents, and sometimes completed forms were not 

 

13The Yakima site deviated most dramatically from the approach outlined. Confidentiality restrictions severely

limited the role of QOP staff until parental consent was obtained. Therefore, school staff were responsible for locating 

students, distributing materials, and collecting completed consent forms.

14This figure overstates the time required for a site to obtain a completed form. First, a day or two—sometimes 

more—elapsed between sample selection and the first attempt to contact a student. Second, a site typically waited until 

it had received completed forms for several students before shipping the forms to us. Therefore, some forms may have 

been in a site’s possession for a few days before being shipped. Even considering these two factors and the time

required for shipping, we figure that it took, on average, two to three weeks to contact a student and collect a completed 

consent form.
A-12

returned to school. In other instances, parents did not read the materials or complete the forms. 

Sometimes, the seeming lack of reliability was attributable, in fact, to an initial lack of interest in

QOP, concern about the time commitment required, or suspicions about government programs. 

QOP staff discussed these issues at length with students and parents. To address concerns about 

time commitments, for example, QOP staff explained that students were not obligated to participate 

in QOP if selected and could refuse to answer survey questions or take evaluation achievement tests.

Generally, when less intrusive approaches had failed in getting forms completed, the most

effective strategy seemed to be for QOP staff to visit parents in the students’ homes and wait there 

while the parents completed the forms. In contrast, telephone calls to parents achieved only limited 

success when previous contact with the student alone had failed.

The only other problem in obtaining completed forms pertained to how they were completed—

specifically, ensuring that the consent form was properly marked and signed and that the most

important items on the locator form were provided. Although about 40 percent of locator forms

(and 1 percent of consent forms) had deficiencies and were returned to sites, correcting the

deficiencies was usually straightforward and caused only minor delays in random assignment.

______________________________________________________________________________

RANDOM ASSIGNMENT

______________________________________________________________________________

After three to four months of developing a list of eligibles and obtaining completed consent

and locator forms for students, the final activities required to complete random assignment took

about one day. The main activities were a series of checks designed to ensure that random

assignment was conducted properly and fairly.

To be eligible for random assignment, a student had to (1) be eligible for QOP (some students 

were found to be ineligible after selection of a school’s initial sample), (2) have parental permission 

to participate in the evaluation, and (3) have a completed locator form. Before we proceeded to

random assignment of the eligible students to QOP and control groups, we required QOP

coordinators to:

C Verify that the list of students eligible for random assignment was accurate.

C Verify the planned allocation of QOP slots across schools (if there was more than one 

school).

C Verify that QOP staff had made good-faith efforts to locate, contact, and obtain

completed forms for students who were not eligible for random assignment.

Typically, the last verification involved a student-by-student review of the actions taken by

QOP staff and the outcome (e.g., QOP staff discovered that the student moved to another state

three months earlier). Sites had to establish that parental permission and a completed locator form 

were highly unlikely to be forthcoming in the near future.
A-13

After the verifications were completed, we randomly assigned students eligible for random

assignment to QOP and control groups. One student was assigned to each available QOP slot

regardless of how many students were eligible for random assignment. We conducted random

assignment independently for each school.

After completing random assignment for a site, we sent the QOP coordinator the list of QOPgroup students and the list of control-group students. QOP staff were responsible for notifying all 

students about the outcome of random assignment. To maintain the integrity of random

assignment, we imposed two rules: (1) a student in the control group could not participate in QOP 

and (2) a student who was not eligible for random assignment could not participate in QOP. To our 

knowledge based on several monitoring activities, these rules were not violated.

______________________________________________________________________________

SCHOOL-BY-SCHOOL SUMMARY OF SAMPLE DEVELOPMENT

______________________________________________________________________________

Table A.1 shows how the evaluation sample was developed for each school. The first row

shows the number of slots allocated to each school. The second row in the table—headed “GPA 

Eligibles”—shows the number of students in each school who were attending the school, were

entering ninth grade for the first time, were appropriate for QOP in accordance with applicable laws 

and regulations, and were in the bottom two-thirds of the grade distribution based on grades from 

the eighth grade (among students satisfying the first three criteria). The number of eligible students 

ranged from 82 to 523 across the QOP schools. Using the procedures described in detail earlier, we 

selected from the list of GPA Eligibles an “Initial Sample” consisting of the number of students

shown in the third row. Then, we instructed QOP staff to obtain consent for participation in the 

evaluation for all students in the initial sample.

As discussed in the main text, about five percent of the students in the initial sample—the 

students in the row headed “Ineligibles”—were determined to be ineligible for QOP based, in most 

instances, on evidence from school records indicating that a student had never attended the QOP

school or had left the school early in the school year before QOP eligibility was determined. The 

parents/guardians of about another seven percent of the students in the initial sample never

responded to QOP staff’s attempts to obtain consent. As we noted before, there was strongly

suggestive evidence from school staff or other sources—but not definitive evidence from school

records—that many of these students were, in fact, ineligible. However, in some instances, the

failure to respond probably was a passive denial of consent. Parents/guardians actively denied

consent for another two percent of the initial QOP sample. Before we would conduct random

assignment for a school, QOP staff had to verify that they had made substantial efforts to contact 

and obtain consent from the nonrespondents.15

The “Consenters” row in Table A.1 gives the number of students who were eligible for random 

assignment and therefore constitute our evaluation sample. From among these students, we filled 

the available QOP slots independently for each school by simple random sampling without

 

15 The nonresponse and active denial of consent percentages are the same when the base for the percentages is the 

number of students in the “Net Eligible Sample” rather than the initial sample.
A-14

replacement. Students who were selected for QOP became QOP enrollees. Students who were not 

selected for QOP became the control group.16

 

16One seemingly minor limitation of the group of consenters as a representative sample of the population of

students who satisfy the QOP eligibility criteria is that a few implicit and explicit denials of consent might not have

occurred in the absence of the evaluation. However, it seems unlikely that more than a trivial number of students would 

have accepted a 100 percent chance to participate in QOP but rejected a 50 percent chance that was essentially costless.
A-15

TABLE A.1

DEVELOPMENT OF THE EVALUATION SAMPLE

Cleveland Washington, D.C.

Fort

Worth Houston Memphis Philadelphia Yakima

All

Sites

Collinwood Anacostia Eastern Total Paschal Austin Yates Total Carver Hamilton Hillcrest Total Franklin Davis Total

QOP Slots 100 40 40 80 100 50 50 100 35 27 38 100 50 50 580

GPA Eligibles 175 130 165 295 398 523 305 828 82 225 108 415 210 229 2550

Initial Sample 175 88 88 176 220 110 110 220 82 58 88 228 110 110 1239

– Ineligibles 9 11 4 15 18 5 7 12 0 0 1 1 9 0 64

Net Eligible Sample 166 77 84 161 202 105 103 208 82 58 87 227 101 110 1175

Consenters 158 72 82 154 177 92 94 186 70 54 75 199 95 100 1069

Denied Consent 1 1 0 1 8 5 4 9 0 0 3 3 2 0 24

Did Not Respond 7 4 2 6 17 8 5 13 12 4 9 25 4 10 82

Consent Probabilitya 95 94 98 96 88 88 91 89 85 93 86 88 94 91 91

QOP Enrollees 100 40 40 80 100 50 50 100 35 27 38 100 50 50 580

Controls 58 32 42 74 77 42 44 86 35 27 37 99 45 50 489

QOP Probabilityb 63 56 49 52 56 54 53 54 50 50 51 50 53 50 54

a100 × Consenters/Net Eligible Sample

b100 × QOP Enrollees/Consenters
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
APPENDIX B

THE BASELINE DATA
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
B-3

Baseline data represent sample members’ characteristics that were unaffected by QOP, either

because they were determined prior to the demonstration or because—like age—they cannot be

affected by a social program. We used baseline characteristics to:

• Correct for nonresponse bias in the impact estimates (Appendix E).

• Correct for random differences between the QOP group and the control group

(Appendix F).

• Estimate impacts on subgroups of enrollees (Chapter VII).

Using data that are unaffected by QOP is necessary for each of these procedures to produce

unbiased results.

In this appendix, we describe the sources of our baseline data. Then, we examine the

prevalence of missing values, and discuss the methods used to impute for missing values. 

______________________________________________________________________________

DATA SOURCES FOR THE BASELINE DATABASE

____________________________________________________________________________

The baseline database contains information on sex, date of birth, race, ethnicity (Hispanic

origin), and eighth-grade grade point average (GPA). Because DOL elected not to conduct a

baseline survey, data on these characteristics were collected from four other sources: (1) the database 

used to determine eligibility for QOP; (2) the telephone survey administered during the fall and

winter of the fifth year after sample members entered the ninth grade; (3) high school transcripts; 

and (4) QOP case managers. The eligibility database included eighth-grade GPA and the name of

the school attended at the beginning of ninth grade. It also often included date of birth, and for

some schools, it included sex, race, or ethnicity. 

______________________________________________________________________________

DEVELOPMENT OF THE BASELINE DATABASE

______________________________________________________________________________

To develop the baseline database, we used the four data sources hierarchically in the order listed 

above. If a value needed for the baseline database was available from the eligibility database, no

other sources were consulted. Thus, the final source, QOP case managers, was used only when the 

value was not available from the first three sources. Table B.1 displays the proportion of sample

members who were missing baseline data items from both the eligibility files and the telephone

survey.
B-4

TABLE B.1

MISSING DATA RATES FOR BASELINE ITEMS IN THE ELIGIBILITY DATABASE AND TELEPHONE SURVEY,

BY QOP/CONTROL STATUS AND SCHOOL

(Percentages)

Fort Worth Cleveland Washington, D.C. Houston Memphis Philadelphia Yakima All Sites

Paschal Collinwood Eastern Anacostia Yates Austin Hillcrest Hamilton Carver Franklin Davis Total

Sex

Overall 16 14 15 31 19 5 15 24 16 18 17 17

QOP 15 14 5 15 12 6 8 26 11 16 18 13

Control 18 14 24 50 27 5 22 22 20 20 16 20

Ethnicity

Overall 16 16 16 32 23 5 15 24 17 20 19 18

QOP 15 14 8 18 18 6 8 26 11 16 22 14

Control 18 19 24 50 30 5 22 22 23 24 16 22

Race

Overall 59 14 15 31 20 70 15 26 16 20 60 34

QOP 57 14 5 15 14 64 8 30 11 16 62 30

Control 62 14 24 50 27 76 22 22 20 24 58 38

Date of Birth

Overall 0 1 5 0 0 0 0 0 0 1 1 1

QOP 0 1 5 0 0 0 0 0 0 2 2 1

Control 0 2 5 0 0 0 0 0 0 0 0 1

Eighth grade GPA

Overall 0 0 0 0 0 0 0 0 0 0 0 0

QOP 0 0 0 0 0 0 0 0 0 0 0 0

Control 0 0 0 0 0 0 0 0 0 0 0 0

SOURCE: Eligibility database and telephone survey.
B-5

Information regarding GPA in eighth grade and school attended was complete for all sample 

members because these items were required to determine eligibility. Missing data rates were nearly 

as low for date of birth—one percent for both the QOP and control groups. The telephone survey 

was the main source of data on sex, ethnicity, and race. The missing data rates for these three

characteristics were higher than for the other items: 17 to 18 percent for sex and ethnicity and 34

percent for race for the seven sites combined. Almost all of the students for whom data on sex and 

ethnicity were missing had not completed the survey at all, usually because the student was not

located. This was also an important reason why data on race were missing. In addition, we found 

that, for schools with relatively many Hispanic students, students who reported Hispanic ethnicity 

often did not respond to the question on race. The missing data rates for race for Paschal, Austin, 

and Davis were 59, 70, and 60 percent, respectively. The highest missing data rate at one of the

other eight schools was 31 percent.

Table B.2 displays the missing data rates for the items in the baseline database after using

school transcripts and using the youth’s first name to impute sex. After these steps, there were no 

missing values for sex, and only five missing values for date of birth. The overall missing data rate 

for ethnicity was 15 percent, down from 18 percent, and the overall missing data rate for race was 26 

percent, down from 34 percent. 

______________________________________________________________________________

STATISTICAL IMPUTATION FOR MISSING VALUES

______________________________________________________________________________

We imputed for the remaining missing values using a sequential hot deck procedure (Carlson et 

al. 1995). For a given data item, students in the evaluation sample were classified as donors and

imputees based on whether information on that particular data item was available (donor) or not

(imputee). The objective of hot deck imputation was to impute a value from a donor to an imputee. 

To achieve this objective, we categorized students into homogeneous classes, using variables

thought to be strongly associated with the data item being imputed. For example, because different 

schools might have different age distributions as a result of student performance and school policies, 

each QOP school was a separate imputation class for imputing date of birth. 

Because the ethnicity of an individual student is strongly associated with the ethnic composition 

of the student’s school and the origin of the student’s surname, we imputed ethnicity by assigning 

students to 22 imputation classes formed by cross-classifying the 11 schools and an indicator for

whether the student had a Spanish surname. Additionally, for better control over the imputations, 

we sorted the students within imputation classes by an indicator for being over age 14 when entering 

ninth grade and by eighth-grade GPA. Thus, the students adjacent to one another in the sorted data 

file were similar with respect to these variables.1 Finally, we performed imputations by donating to 

an imputee a nonmissing value from the previous student or the subsequent student (in an

 

1

 Serpentine sorting was used to ensure that adjacent students were as similar as possible. With serpentine sorting, 

the sort order is reversed as boundaries are crossed for higher level sorting variables. For example, suppose two

variables, age and GPA, are used for sorting. With conventional sorting, the resulting sort order is such that while most 

adjacent students differ by only one level of the GPA variable, when the age variable changes values, the students on 

either side of this boundary differ by only one level of the age variable, but their values for the GPA variable are as

different as they can be. With serpentine sorting, when the age variable changes values, the GPA variable goes in reverse 

order from before, so that the students on either side of the age boundary differ by only the age variable and have

identical (or nearly identical) values for the GPA variable.
B-6

TABLE B.2

MISSING DATA RATES FOR BASELINE ITEMS AFTER OBTAINING SOME DATA FROM TRANSCRIPTS AND USING FIRST NAMES TO IMPUTE SEX,

BY QOP/CONTROL STATUS AND SCHOOL

(Percentages)

Fort Worth Cleveland Washington, D.C. Houston Memphis Philadelphia Yakima All Sites

Paschal Collinwood Eastern Anacostia Yates Austin Hillcrest Hamilton Carver Franklin Davis Total

Sex

Overall 0 0 0 0 0 0 0 0 0 0 0 0

QOP 0 0 0 0 0 0 0 0 0 0 0 0

Control 0 0 0 0 0 0 0 0 0 0 0 0

Ethnicity

Overall 12 16 16 32 21 0 12 20 13 17 16 15

QOP 9 14 8 18 16 0 3 19 3 10 16 11

Control 17 19 24 50 27 0 22 22 23 24 16 21

Race

Overall 53 1 12 25 3 67 7 15 11 9 56 26

QOP 48 2 0 5 4 62 3 15 3 4 54 21

Control 58 0 24 50 2 74 11 15 20 16 58 31

Date of Birth

Overall 0 1 2 0 0 0 0 0 0 0 0 3

QOP 0 1 0 0 0 0 0 0 0 0 0 0

Control 0 2 5 0 0 0 0 0 0 0 0 1

Eighth grade GPA

Overall 0 0 0 0 0 0 0 0 0 0 0 0

QOP 0 0 0 0 0 0 0 0 0 0 0 0

Control 0 0 0 0 0 0 0 0 0 0 0 0

SOURCE: Eligibility database, telephone survey, and transcripts.
B-7

 

alternating manner). We combined the QOP-group and control-group members to perform the

imputations. Table B.3 displays the number of students available as donors and the number of

students requiring imputation of Hispanic ethnicity in each of the 22 classes.

To impute race, we assigned students to 22 classes, formed by cross-classifying the 11 schools 

and an indicator for whether the youth was of Hispanic ethnicity (as reported or imputed). Within 

each class, youth were sorted by an indicator for being over age 14 when entering ninth grade and by 

eighth-grade GPA. Table B.4 displays the number of students available as donors and the number 

of students requiring imputation of race in each of the 22 classes. Because Yates and Hamilton had 

no Hispanic donors, the race imputation was modified for these two schools by eliminating from the 

imputation process the classification based on ethnicity. Thus, for Yates and Hamilton, there were 

only two classes (one for each school), and within each class, sample members were sorted by the

age indicator and eighth-grade GPA. Table B.5 presents the frequency distribution of the number 

of times a given donor was used by data item.
B-8

TABLE B.3

NUMBER OF DONORS AND IMPUTEES FOR ETHNICITY, BY SPANISH SURNAME AND SCHOOL

Fort Worth Cleveland Washington, D.C. Houston Memphis Philadelphia Yakima All Sites

Paschal Collinwood Eastern Anacostia Yates Austin Hillcrest Hamilton Carver Franklin Davis Total

Total

Overall 177 158 82 72 94 92 75 54 70 95 100 1,069

Donors 155 133 69 49 74 92 66 43 61 79 84 905

Imputees 22 25 13 23 20 0 9 11 9 16 16 164

Spanish surname

Overall 105 0 0 1 2 83 0 0 1 1 57 250

Donors 93 0 0 1 2 83 0 0 1 1 47 228

Imputees 12 0 0 0 0 0 0 0 0 0 10 22

Non-Spanish surname

Overall 72 158 82 71 92 9 75 54 69 94 43 819

Donors 62 133 69 48 72 9 66 43 60 78 37 677

Imputees 10 25 13 23 20 0 9 11 9 16 6 142

SOURCE: Eligibility database, telephone survey, and transcripts.
B-9

TABLE B.4

NUMBER OF DONORS AND IMPUTEES FOR RACE, BY ETHNICITY AND SCHOOL

Fort Worth Cleveland Washington, D.C. Houston Memphis Philadelphia Yakima All Sites

Paschal Collinwood Eastern Anacostia Yates Austin Hillcrest Hamilton Carver Franklin Davis Total

Total

Overall 177 158 82 72 94 92 75 54 70 95 100 1,069

Hispanic

Overall 115 1 0 2 3 90 0 1 0 3 62 277

Donors 26 1 0 2 0 28 0 0 0 2 11 70

Imputees 89 0 0 0 3 62 0 1 0 1 51 207

Non-Hispanic

Overall 62 157 82 70 91 2 75 53 70 92 38 792

Donors 58 155 72 52 91 2 70 46 62 84 33 725

Imputees 4 2 10 18 0 0 5 7 8 8 5 67

SOURCE: Eligibility database, telephone survey, and transcripts.
B-10

TABLE B.5

NUMBER OF DONORS BY THE NUMBER OF TIMES USED, BY DATA ITEM

Data Item

Number of Times Used Date of Birth Ethnicity Race

1 4 114 66

2 19 18

3 4 5

4 4

5 3

6 8

7 2

8 3

9 2

10 1

11 0

12 1

SOURCE: Eligibility database, telephone survey, and transcripts.
APPENDIX C

FOLLOW-UP DATA
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
C-3

Data on outcomes were obtained from: 

1. An in-person survey administered during the spring of the fourth year after sample

members entered the ninth grade.

2. Achievement tests in mathematics and reading administered immediately before the inperson survey.

3. A telephone survey administered during the fall and the winter of the fifth year after

sample members entered the ninth grade.

4. High school transcripts from all the high schools the sample members attended during

the demonstration.

In this appendix, we describe the instruments and fielding procedures for the in-person survey, 

the achievement tests, the telephone survey, and the collection of transcripts. Then, after discussing 

the response rates to the surveys, achievement tests, and transcripts, we examine the prevalence of

missing values for outcomes, that is, item nonresponse.

______________________________________________________________________________

INSTRUMENTS AND FIELDING PROCEDURES

______________________________________________________________________________

Table C.1 lists the sites and schools that participated in the QOP demonstration. Table C.2

presents the dates within which each survey was conducted. The program in the Washington, D.C., 

site started one year later than the other programs; hence, data collection activities in this site

occurred one year later than in the other sites. 

The In-Person Survey and Achievement Tests

The achievement tests are presented in Exhibit C.1, and the in-person survey questionnaire that 

was used in all sites except Washington, D.C., is displayed in Exhibit C.2. The questionnaire that was 

used in the Washington, D.C., site was identical to the questionnaire that was used in the other sites 

except for the date and year references in some questions.

Four MPR staff traveled to each site to administer the survey and tests. Based on previous

telephone calls to locate each sample member, sample members who were still enrolled at the QOP 

schools were assigned to in-school sessions, whereas sample members who had transferred or

dropped out were assigned to sessions outside of the QOP schools. The sessions outside of QOP 

schools were held at conveniently located private schools, universities, public facilities with meeting 

spaces, and, in one instance, a church social hall.

Before the sessions, MPR sent parents or guardians of sample members a letter describing the 

purpose of the study and encouraging their child’s participation. In addition, the letter informed

them about the sensitive nature of the survey questions. Sample members were called approximately 

one week before the session and asked to attend. In addition, they received a reminder call either

the night before or the morning of their assigned session. 
C-4

TABLE C.1

QOP SITES AND SCHOOLS

QOP Site Schools

Fort Worth, TX Paschal High School

Cleveland, OH Collinwood High School

Washington, DC Anacostia High School

Eastern High School

Houston, TX Austin High School

Yates High School

Memphis, TN Carver High School

Hamilton High School

Hillcrest High School

Philadelphia, PA Ben Franklin High School

Yakima, WA Davis High School

TABLE C.2

DATA COLLECTION FIELDING DATES

Instrument Fielding Dates

Non-DC In-Person Survey/Achievement Tests February - April 1999

DC In-Person Survey/Achievement Tests April 2000

Non-DC Telephone Survey November 1999 - June 2000

DC Telephone Survey November 2000 - April 2001

Non-DC School Records September 1999 - December 2000

DC School Records December 2000 - April 2001
C-5

The achievement tests were administered before the survey. These tests were developed by the 

Educational Testing Service (ETS) from National Education Longitudinal Study (NELS)

achievement tests. ETS developed the NELS tests and administered them to a national probability 

sample of tenth graders in 1990. ETS used the national data to place each QOP evaluation sample 

member within the national distributions of reading and mathematics scores. The position in one of 

these distributions is expressed as a percentile. For example, a QOP evaluation sample member at 

the 47th percentile had a test score that was higher than 47 percent of the national population of

tenth graders. NELS is a survey program of the National Center for Educational Statistics (NCES) 

of the U.S. Department of Education that follows a representative cohort of students from the

eighth grade through their teenage years into early adulthood. NELS includes achievement tests in 

reading, mathematics, and other subjects. The QOP achievement tests contained a subset of the

questions in the NELS tests.

We administered four timed tests for the QOP evaluation. First, sample members completed a 

Phase I mathematics test. When they were finished, they returned it to the session proctors and

were given a Phase I reading test. The proctors scored the Phase I mathematics test while the Phase 

I reading test was under way. Based on their score on the Phase I mathematics test, sample

members were given a low-, medium-, or high-level Phase II mathematics test. The Phase I reading 

test was scored while the Phase II mathematics test was under way. Based on their score on the

Phase I reading test, sample members were given a low-, medium-, or high-level Phase II reading

test. 

The sample members were given a 5 to 10 minute break after the tests. At the end of the break, 

the proctors gave a questionnaire and a plain brown envelope to each sample member. The sample 

members were instructed to complete the questionnaire and seal it in the envelope before returning 

it to the proctors. In addition, each sample member was asked to read and sign a consent form

giving permission for MPR to collect his or her school records. At the end of the session, each

sample member was given a $30 check. Pizza and sodas were also provided.

Survey staff interviewed individually the sample members who did not participate in a group

session. Overall, 62 percent of the sample members who completed the tests and survey did so in a 

group session and the remaining 38 percent did so during field follow-up.

The Telephone Survey

We conducted a telephone survey with each sample member 7 to 10 months after the in-person 

survey and achievement tests. Interviews were administered using computer-assisted telephone

interviewing (CATI) and took about 20 minutes to complete. Exhibit C.3 presents the

questionnaire.

Each sample member received a letter about one week before the start of interviewing. The

letter indicated that we would call for an important follow-up study and encouraged him or her to 

participate. In addition, the letter indicated that we would pay $10 for completing the interview.

We interviewed in person those who did not respond to the telephone survey. The same field 

interviewers used in the in-person individual interviews followed up with nonrespondents to the

telephone survey. Overall, 92 percent of the sample members who responded did so via telephone 

and the remaining 8 percent responded during in-person follow-up.
C-6

School Transcript Collection

The last data collection activity completed to date involved collecting and processing transcripts 

from all high schools the sample members attended since the beginning of the demonstration. In

the telephone survey, sample members identified 194 schools—the 11 QOP schools and 183

schools to which the youth had transferred. From those schools, we requested 1,487 transcripts. In 

addition to the transcripts, schools were asked to provide a course list or course catalog and a

description of the school’s grading system. 

Each school was mailed a request packet (Exhibit C.4) that included the following items: a

cover letter explaining the purpose of the study; a checklist of instructions for providing the

transcripts and associated information; forms for identifying the youth and recording their

information; copies of the youths’ signed consent forms; disclosure notices to be placed in the

youths’ files indicating the purpose for which the school records were released; a form to request

reimbursement for transcript preparation; and an addressed, postage-paid envelope for returning the 

materials. We remailed the packet to the schools that did not respond to the initial request within 

three weeks. We also telephoned such schools a few days later. A third nonrespondent mailing was 

conducted three weeks after the second mailing. Telephone follow-up with nonresponding schools 

continued throughout the transcript collection. 

______________________________________________________________________________

RESPONSE RATES

______________________________________________________________________________

Table C.3 displays the unit response rates for each data collection activity. The figures are

presented separately for QOP- and control-group members and are presented for the full sample 

and by school. 

The In-Person Survey and Achievement Tests

The overall response rate to the in-person survey was 84 percent. The difference in response 

rates between the QOP and control groups was 8 percentage points overall (88 percent for the QOP 

group and 80 percent for the control group). However, as shown in Table C.3, this differential

varied widely across schools. The largest differences in response rates between the QOP and control 

groups were for Yates (17 percentage points) and Hillcrest (14 percentage points). 

Only a few youth completed the in-person survey but not the achievement tests or vice versa. 

Thus, the response rates to the achievement tests were very close to those of the in-person survey. 

For the reading achievement test, the overall response rate was 84 percent (88 percent for the QOP 

group and 80 percent for the control group). For the mathematics achievement test, the overall

response rate was 84 percent (87 percent for the QOP group and 80 percent for the control group).
C-7

TABLE C.3

RESPONSE RATES FOR DATA COLLECTION ACTIVITIES

(Percentages, except for sample sizes)

Fort Worth Cleveland Washington, D.C. Houston Memphis Philadelphia Yakima All Sites

Paschal Collinwood Eastern Anacostia Total Yates Austin Total Hillcrest Hamilton Carver Total Franklin Davis Total

Sample size

Overall 177 158 82 72 154 94 92 186 75 54 70 199 95 100 1069

QOP 100 100 40 40 80 50 50 100 38 27 35 100 50 50 580

Control 77 58 42 32 74 44 42 86 37 27 35 99 45 50 489

In-person survey

Overall 83 82 87 82 84 82 93 88 85 80 86 84 89 76 84

QOP 88 84 92 88 90 90 94 92 92 78 83 85 92 82 88

Control 77 79 81 75 78 73 93 83 78 81 89 83 87 70 80

Achievement tests

Reading

Overall 82 83 85 82 84 82 96 89 85 80 86 84 89 75 84

QOP 87 85 92 88 90 90 96 93 92 78 83 85 92 80 88

Control 77 79 79 75 77 73 95 84 78 81 89 83 87 70 80

Mathematics

Overall 81 83 85 82 84 82 96 89 85 80 86 84 89 75 84

QOP 86 85 92 88 90 90 96 93 92 78 83 85 92 80 87

Control 75 79 79 75 77 73 95 84 78 81 89 83 87 70 80

Telephone survey

Overall 84 86 85 69 78 81 95 88 85 76 84 82 82 83 83

QOP 85 86 95 85 90 88 94 91 92 74 89 86 84 82 87

Control 82 86 76 50 65 73 95 84 78 78 80 79 80 84 80

Transcripts

Overall 87 70 93 85 89 83 96 89 83 63 83 77 79 79 82

QOP 93 68 98 92 95 92 98 95 87 67 86 81 82 88 86

Control 79 72 88 75 82 73 93 83 78 59 80 74 76 70 77

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts.
C-8

There were several reasons for nonresponse to the in-person survey and achievement tests, and 

they were about equally important. Some sample members were not located or refused to answer

the survey. Others could not be contacted for an interview or did not meet with the field

interviewer when scheduled, and some had moved out of the survey area. Table C.4 displays the

final disposition report for the in-person survey and achievement tests. This report shows, for

example, how many youth responded and, by reason for nonresponse, how many did not respond.

The Telephone Survey

The overall response rate to the telephone survey was 83 percent. The difference in response 

rates between the QOP and control groups was 7 percentage points overall (87 percent for the QOP 

group and 80 percent for the control group). As with the in-person survey and achievement tests,

this differential varied widely across schools. The largest differences in response rates between the

QOP and control groups were for Anacostia (35 percentage points), Eastern (19 percentage points), 

Yates (15 percentage points), and Hillcrest (14 percentage points).

Most nonrespondents to the telephone survey were youth who could not be located. Table C.5 

displays the final disposition report for the telephone survey.

School Transcripts

The response rate to the school transcripts data collection was 82 percent. The difference in 

response rates between the QOP and control groups was 9 percentage points (86 percent for the

QOP group and 77 percent for the control group). Among schools, the largest difference in

response rates between the QOP and control groups was for Davis (18 percentage points). 

We collected complete academic records—all courses and grades from the beginning of ninth

grade until the end of high school (by graduation or not)—for 74 percent of the youth, and we

obtained no transcript at all for 18 percent. For the remaining 8 percent, we obtained incomplete

transcript data. For most of these cases, youth attended multiple schools, but we received

transcripts from only some of the schools attended. 

______________________________________________________________________________

MISSING VALUES

______________________________________________________________________________

Item nonresponse was uncommon for most outcome measures used in the impact analysis (see 

Table C.6). For example, item nonresponse was typically less than one percent for indicators of

high school completion, postsecondary activities, and childbearing. Item nonresponse for indicators 

of substance abuse and criminal activity was slightly higher, but remained under four percent. In

general, item nonresponse did not differ much between the QOP and control groups. 
C-9

TABLE C.4

IN-PERSON SURVEY AND ACHIEVEMENT TESTS DISPOSITIONS, BY SITE AND QOP/CONTROL STATUS

(Numbers of respondents and percentages)

Fort Worth Cleveland Washington, D.C. Houston

Disposition

Total

N=177

QOP

N=100

Control

N=77

Total

N=158

QOP

N=100

Control 

N=58

Total

N=154

QOP 

N=80

Control 

N=74

Total

N=186

QOP 

N=100

Control

N=86

Completed

Total 147 88 59 131 85 46 130 72 58 165 93 72

(83%) (88%) (77%) (83%) (85%) (79%) (84%) (90%) (78%) (89%) (93%) (84%)

Group 85 59 26 74 50 24 64 41 23 90 55 35

(48%) (59%) (34%) (47%) (50%) (41%) (42%) (51%) (31%) (48%) (55%) (41%)

Field 59 26 33 56 34 22 66 31 35 73 37 36

(33%) (26%) (43) (35%) (34%) (38%) (43%) (39%) (47%) (39%) (37%) (42%)

Partial 3 3 0 1 1 0 0 0 0 2 1 1

(2%) (3%) (0%) (1%) (1%) (0%) (0%) (0%) (0%) (1%) (1%) (1%)

Completed, by Instrument

Survey 147 88 59 130 84 46 127 70 57 163 92 71

(83%) (88%) (77%) (82%) (84%) (79%) (82%) (88%) (77%) (88%) (92%) (83%)

Reading Phase I 146 87 59 131 85 46 129 72 57 165 93 72

(82%) (87%) (77%) (83%) (85%) (79%) (84%) (90%) (77%) (89%) (93%) (84%)

Reading Phase II 145 86 59 130 84 46 129 72 57 165 93 72

(82%) (86%) (77%) (82%) (84%) (79%) (84%) (90%) (77%) (89%) (93%) (84%)

Math Phase I 145 86 59 131 85 46 129 72 57 165 93 72

(82%) (86%) (77%) (83%) (85%) (79%) (84%) (90%) (77%) (89%) (93%) (84%)

Math Phase II 144 85 59 130 84 46 129 72 57 165 93 72

(81%) (85%) (77%) (82%) (84%) (79%) (84%) (90%) (77%) (89%) (93%) (84%)

Not Completed

Total 30 12 18 26 14 12 24 8 16 20 7 13

(17%) (12%) (23%) (16%) (14%) (21%) (16%) (10%) (22%) (11%) (7%) (15%)

Located, not intervieweda 12 6 6 6 4 2 8 4 4 7 4 3

(7%) (6%) (8%) (4%) (4%) (3%) (5%) (5%) (5%) (4%) (4%) (3%)

Not located 6 2 4 8 3 5 11 3 8 1 0 1

(3%) (2%) (5%) (5%) (3%) (9%) (7%) (4%) (11%) (1%) (0%) (1%)

Out of area 9 4 5 4 2 2 2 1 1 11 3 8

(5%) (4%) (6%) (3%) (2%) (3%) (1%) (1%) (1%) (6%) (3%) (9%)

Refused 3 0 3 8 5 3 3 0 3 1 0 1

(2%) (0%) (4%) (5%) (5%) (5%) (2%) (0%) (4%) (1%) (0%) (1%)

Deceased 0 0 0 1 1 0 0 0 0 1 0 1

(0%) (0%) (0%) (1%) (1%) (0%) (0%) (0%) (0%) (1%) (0%) (1%)
C-10

TABLE C.4 (continued)

Memphis Philadelphia Yakima All Sites

Disposition

Total

N=199

QOP

N=100

Control 

N=99

Total

N=95

QOP

N=50

Control

N=45

Total

N=100

QOP

N=50

Control

N=50

Total

N=1,069

QOP

N=580

Control

N=489

Complete

Total 167 85 82 85 46 39 77 41 36 902 510 392

(84%) (85%) (83%) (89%) (92%) (87%) (77%) (82%) (72%) (84%) (88%) (80%)

Group 103 65 38 64 37 27 54 34 20 534 341 193

(52%) (65%) (38%) (67%) (74%) (60%) (54%) (68%) (40%) (50%) (59%) (39%)

Field 63 20 43 21 9 12 20 6 14 358 163 195

(32%) (20%) (43%) (22%) (18%) (27%) (20%) (12%) (28%) (33%) (28%) (40%)

Partial 1 0 1 0 0 0 3 1 2 10 6 4

(1%) (0%) (1%) (0%) (0%) (0%) (3%) (2%) (4%) (1%) (1%) (1%)

Completed, by Instrument

Survey 167 85 82 85 46 39 77 41 36 896 506 390

(84%) (85%) (83%) (89%) (92%) (87%) (77%) (82%) (72%) (84%) (87%) (80%)

Reading Phase I 167 85 82 85 46 39 76 41 35 899 509 390

(84%) (85%) (83%) (89%) (92%) (87%) (76%) (82%) (70%) (84%) (88%) (80%)

Reading Phase II 167 85 82 85 46 39 76 41 35 897 507 390

(84%) (85%) (83%) (89%) (92%) (87%) (76%) (82%) (70%) (84%) (87%) (80%)

Math Phase I 166 85 81 85 46 39 76 41 35 897 508 389

(84%) (85%) (82%) (89%) (92%) (87%) (76%) (82%) (70%) (84%) (88%) (80%)

Math Phase II 167 85 82 85 46 39 74 40 34 894 505 389

(84%) (85%) (83%) (89%) (92%) (87%) (74%) (80%) (68%) (84%) (87%) (80%)

Not Completed

Total 31 14 17 10 4 6 22 9 13 163 68 95

(16%) (14%) (17%) (11%) (8%) (13%) (22%) (18%) (26%) (15%) (12%) (19%)

Located, not intervieweda 16 7 9 4 1 3 4 0 4 57 26 31

(8%) (7%) (9%) (4%) (2%) (7%) (4%) (0%) (8%) (5%) (4%) (6%)

Not located 0 0 0 3 1 2 4 1 3 33 10 23

(0%) (0%) (0%) (3%) (2%) (4%) (4%) (2%) (6%) (3%) (2%) (5%)

Out of area 7 5 2 2 1 1 11 7 4 46 23 23

(4%) (5%) (2%) (2%) (2%) (2%) (11%) (14%) (8%) (4%) (4%) (5%)

Refused 8 2 6 1 1 0 3 1 2 27 9 18

(4%) (2%) (6%) (1%) (2%) (0%) (3%) (2%) (4%) (3%) (2%) (4%)

Deceased 1 1 0 0 0 0 1 0 1 4 2 2

(1%) (1%) (0%) (0%) (0%) (0%) (1%) (0%) (2%) (<1%) (<1%) (<1%)

SOURCE: In-person survey and achievement tests.

aIncludes cases who were located, but could not be contacted for an interview or who did not meet with the field interviewer when scheduled (passive refusals).
C-11

TABLE C.5

TELEPHONE SURVEY DISPOSITIONS, BY SITE AND QOP/CONTROL STATUS

Fort Worth Cleveland Washington, D.C. Houston

Disposition

Total

N=177

QOP

N=100

Control

N=77

Total

N=158

QOP

N=100

Control

N=58

Total

N=154

QOP

N=80

Control

N=74

Total

N=186

QOP

N=100

Control

N=86

Complete 148 85 63 136 86 50 121 72 49 163 91 72

(84%) (85%) (82%) (86%) (86%) (86%) (79%) (90%) (66%) (88%) (91%) (84%)

Telephone 141 82 59 128 86 42 112 66 46 131 73 58

(80%) (82%) (77%) (81%) (86%) (72%) (73%) (83%) (62%) (70%) (73%) (67%)

Mail 2 2 0 0 0 0 0 0 0 3 0 3

(1%) (2%) (0%) (0%) (0%) (0%) (0%) (0%) (0%) (2%) (0%) (3%)

Field 5 1 4 8 0 8 9 6 3 29 18 11

(3%) (1%) (5%) (5%) (0%) (14%) (6%) (8%) (4%) (16%) (18%) (13%)

Not Complete 29 15 14 21 13 8 33 8 25 21 9 12

(16%) (15%) (18%) (13%) (13%) (14%) (21%) (10%) (34%) (11%) (9%) (14%)

Not locateda 17 11 6 14 8 6 25 5 20 16 6 10

(10%) (11%) (8%) (9%) (8%) (10%) (16%) (6%) (27%) (9%) (6%) (12%)

Located, not interviewedb 8 3 5 4 2 2 6 3 3 3 2 1

(5%) (3%) (6%) (3%) (2%) (3%) (4%) (4%) (4%) (2%) (2%) (1%)

Refused 4 1 3 3 3 0 2 0 2 2 1 1

(2%) (1%) (4%) (2%) (3%) (0%) (1%) (0%) (3%) (1%) (1%) (1%)

Deceased 0 0 0 1 1 0 0 0 0 2 0 2

(0%) (0%) (0%) (1%) (1%) (0%) (0%) (0%) (0%) (1%) (0%) (2%)
C-12

TABLE C.5 (continued)

Memphis Philadelphia Yakima All Sites

Disposition

Total

N=199

QOP

N=100

Control

N=99

Total

N=95

QOP

N=50

Control

N=45

Total

N=100

QOP

N=50

Control

N=50

Total

N=1,069

QOP

N=580

Control

N=489

Complete 164 86 78 78 42 36 83 41 42 893 503 390

(82%) (86%) (79%) (82%) (84%) (80%) (83%) (82%) (84%) (84%) (87%) (80%)

Telephone 158 86 72 76 41 35 77 40 37 823 474 349

(79%) (86%) (73%) (80%) (82%) (78%) (77%) (80%) (74%) (77%) (82%) (71%)

Mail 1 0 1 2 1 1 1 1 0 9 4 5

(1%) (0%) (1%) (2%) (2%) (2%) (1%) (2%) (0%) (1%) (1%) (1%)

Field 5 0 5 0 0 0 5 0 5 63 25 36

(3%) (0%) (5%) (0%) (0%) (0%) (5%) (0%) (10%) (6%) (4%) (7%)

Not Complete 34 13 21 17 8 9 15 8 7 170 74 96

(17%) (13%) (21%) (18%) (16%) (20%) (15%) (16%) (14%) (16%) (13%) (20%)

Not locateda 25 13 12 11 5 6 12 7 5 120 55 65

(13%) (13%) (12%) (12%) (10%) (13%) (12%) (14%) (10%) (11%) (9%) (13%)

Located, not interviewedb 4 0 4 4 2 2 3 1 2 32 13 19

(2%) (0%) (4%) (4%) (4%) (4%) (3%) (2%) (4%) (3%) (2%) (4%)

Refused 5 0 5 2 1 1 0 0 0 18 6 12

(3%) (0%) (5%) (2%) (2%) (2%) (0%) (0%) (0%) (2%) (1%) (2%)

Deceased 1 1 0 0 0 0 2 1 1 6 3 3

(1%) (1%) (0%) (0%) (0%) (0%) (2%) (2%) (2%) (1%) (1%) (1%)

SOURCE: Telephone survey.

aIncludes cases who moved out of the range of the field interviewers and cases for whom we did not have a valid address.

bIncludes cases who were located, but could not be contacted for an interview or who did not meet with the field interviewer when scheduled (passive refusals).
C-13

TABLE C.6

ITEM RESPONSE RATES FOR KEY OUTCOMES

(Percentages)

Outcomea

QOP

Group

Control

Group

Total

Sample

Mathematics test score (percentile) (Achievement tests) 99.4 99.5 99.4

Reading test score (percentile) (Achievement tests) 99.6 99.7 99.7

GPA (four-point scale) (transcripts) 94.3 94.7 94.5

Earned high school diploma (Telephone survey and transcripts) 99.4 99.7 99.6

Earned diploma or GED certificate (Telephone survey and transcripts) 99.4 99.7 99.6

Attending college (Telephone survey) 99.6 99.5 99.6

Attending postsecondary training (Telephone survey) 99.2 99.0 99.1

Postsecondary training or good job (Telephone survey) 98.6 97.2 98.0

Postsecondary training or high school or GED class, or any job 

(Telephone survey) 99.8 100.0 99.9

Attending or accepted into college (Telephone survey) 99.6 99.7 99.7

Binge drinking in past 30 days (In-person survey) 95.7 98.0 96.7

Used any illegal drug in past 30 days (In-person survey) 96.9 97.4 97.1

Committed any crime in past 12 months (In-person survey) 96.5 96.4 96.4

Ever arrested or charged (In-person survey) 96.9 98.2 97.4

Have one or more own children (Telephone survey) 100.0 99.7 99.9

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts.

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” 

means either a two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, 

or armed forces. A “good” job offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row.
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
	






Page Is Intentionally Left Blank to Allow for Double-Sided Copying 





Page Is Intentionally Left Blank to Allow for Double-Sided Copying 





Page Is Intentionally Left Blank to Allow for Double-Sided Copying 






























	






Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
















	









Page Is Intentionally Left Blank to Allow for Double-Sided Copying 





























Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
	














APPENDIX D

OUTCOMES AND SUBGROUPS
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
D-3

This appendix describes the outcomes and subgroups of enrollees for which we estimated

impacts.

______________________________________________________________________________

OUTCOMES

______________________________________________________________________________

The outcomes fall into five broad categories:

1. High School Performance. The outcomes in this category include reading and mathematics

achievement test scores, grade point average (GPA), credits earned, and suspensions or 

expulsions from high school.

2. High School Completion. The outcomes in this category measure receipt of a high school

diploma or receipt of a general educational development (GED) certificate. They also

measure whether a youth was still attending high school.

3. Postsecondary Activity. The outcomes in this category measure engagement in

postsecondary education and training activities through college, vocational/technical

schools, certified apprenticeship programs, and the armed forces. They also measure 

employment.

4. Risky Behaviors. The outcomes in this category measure substance abuse, gang activity,

criminal activity, involvement with the criminal justice system, sexual activity, and

childbearing.

5. Risk and Resiliency Factors. The outcomes in this category measure whether there are

factors in a youth’s social environment that increase the likelihood that the youth will

achieve the goals of QOP. Having a caring adult mentor is an example of a factor that 

may improve a youth’s resiliency to negative influences. 

Table D.1 displays the complete list of outcomes by category, and indicates the source of data 

for each outcome. Most of the outcomes are self-explanatory, although several require additional

explanation, which is presented below. In several categories, outcomes are listed in order from

narrowest to broadest. For example, outcomes in the high school completion category begin with 

earning a high school diploma as the narrowest and most difficult to achieve. The second outcome 

broadens the first outcome to include earning a GED certificate. The final outcome in the category 

is the most inclusive and the easiest to achieve.

High School Performance

Credits reflect the number of courses completed (with a passing grade) and the amount of class

time required to complete courses. Credits are expressed in Carnegie units. One Carnegie unit

corresponds to a class that meets for 45 to 60 minutes every day of the week for a whole academic 

year. Core academic credits are the number of Carnegie units in mathematics, science, English,

social studies, and foreign language classes.
D-4

TABLE D.1

OUTCOMES

Category

High School Completion

From Telephone Survey and High School Transcripts

Received high school diplomaa

Received high school diploma or GEDa

Received high school diploma or GED or was still in high schoola

Received high school diploma or GED or was still in high school or was taking a GED coursea

High School Performance

From Achievement Tests

Math achievement test score, percentilea

Reading achievement test score, percentilea

From High School Transcripts

Cumulative GPA, on a scale from 0 to 4a

Math/Science GPA, on a scale from 0 to 4a

Total Credits, Carnegie unitsa

Core Academic Credits, Carnegie unitsa

Math/Science/English Credits, Carnegie units

From Telephone Survey

Ever suspended for disciplinary reasons 

Ever expelled for disciplinary reasons 

From In-Person Survey

Suspended or expelled in the last 12 months 

Postsecondary Activities

From Telephone Survey

Attending a four-year college

Attending a two- or four-year college

Attending college or enrolled in vocational/technical school or apprenticeship or armed forces ( = postsecondary training)

Postsecondary training or working at a good joba

Postsecondary training or working at any job

Postsecondary training or attending high school 

Postsecondary training or attending high school or a GED course

Postsecondary training or attending high school or a GED course or working at a good joba
TABLE D.1 (continued)

D-5

Category

Postsecondary training or attending high school or a GED course or working at any job

Attending or accepted into a four-year college

Attending or accepted into a two- or four-year college

Attending or accepted into college or enrolled in vocational/technical school or apprenticeship or armed forces ( = postsecondary training)

Postsecondary training (includes attending or accepted into college)or working at a good joba

Postsecondary training (includes attending or accepted into college)or working at any job

Postsecondary training (includes attending or accepted into college)or attending high school 

Postsecondary training (includes attending or accepted into college)or attending high school or a GED course

Postsecondary training (includes attending or accepted into college)or attending high school or a GED course or working at a good joba

Postsecondary training (includes attending or accepted into college)or attending high school or a GED course or working at any job

Attending, accepted, or applied to a college

Attending, accepted, or applied to a college or enrolled in vocational/technical school or apprenticeship or armed forces ( = postsecondary training)

Postsecondary training (includes attending, accepted, or applied to a college)or working at a good joba

Postsecondary training (includes attending, accepted, or applied to a college)or working at any job

Postsecondary training (includes attending, accepted, or applied to a college)or attending high school 

Postsecondary training (includes attending, accepted, or applied to a college)or attending high school or a GED course

Postsecondary training (includes attending, accepted, or applied to a college)or attending high school or a GED course or working at a good joba

Postsecondary training (includes attending, accepted, or applied to a college)or attending high school or a GED course or working at any job

Risky Behaviors

Substance Abuse

From In-Person Survey

Drinking in the past 30 daysa

Frequent drinking in the past 30 daysa

Binge drinking in the past 30 daysa

Frequent binge drinking in the past 30 daysa

Drunk or high in school in the past 12 months

Used any illegal drug in the past 30 daysa

From Telephone Survey

Binge drinking in past 30 daysa

Frequent drinking in the past 30 daysa

Used any illegal drug in the past 30 daysa

Gang Activity

From In-Person Survey

Involved in gang fight in prior year

Ever a member of a gang 

Currently a member of a gang
TABLE D.1 (continued)

D-6

Category

Criminal Activity 

From In-Person Survey

Committed any crime in the past 12 monthsa

From Telephone Survey

Committed any crime in the past 3 monthsa

Involvement with the Criminal Justice System

From In-Person Survey

Ever arrested or charged with a crime or parole violation

From Telephone Survey

Arrested or charged with a crime or parole violation in the past 3 months 

Sexual Activity and Childbearing

From In-Person Survey

Ever had sex 

Did not use condom last time had sex

Taught about HIV/AIDS 

From Telephone Survey

Ever pregnant or get anyone pregnant

Have had a child 

Resiliency Factors

Influential Adult

From In-Person Survey

There is an influential adult in my lifea

Influential adult pays attention to and respects my ideas and feelings about things in my lifea

Influential adult has helped me learn things that have helped me do well in my lifea

Influential adult has helped me take advantage of opportunities to get ahead in my lifea

Influential adult has recognized and appreciated the things I have done well in my lifea

Influential adult has clear expectations and standards about what I do with my lifea

Influential adult has shown me that fighting is not a good way of solving problems in my lifea

Influential adult has shown me that breaking the law does not help me achieve my goalsa

Influential adult has shown me that using drugs or alcohol is not a good way of solving problems in my lifea

Influential adult has had any of these influencesa

Special Programs

From In-Person Survey

Participated in a special program to help studentsa

Program helped me improve my gradesa

Program helped me stay away from drugs or get off drugsa
TABLE D.1 (continued)

D-7

Category

Program helped me stay out of troublea

Program helped me deal with the police and courtsa

Program helped me prepare for collegea

Program helped me earn and save moneya

Program helped in any of these waysa

Attitudes Toward Risky Behaviors

From In-Person Survey

Using drugs or alcohol frequently is always wronga

Committing crimes is always wronga

Having a baby while a teenager is always wronga

Dropping out of school is always wronga

All of the above activities are always wronga

Outlook on Life

From In-Person Survey

Disagree that bad things happen to people like mea

Disagree that my life will be unhappya

Disagree that I do not like the way I looka

Disagree that I will probably die before I’m thirtya

Disagree with all of the above four statementsa

a

 A more detailed explanation of how this outcome was measured can be found in the text of the appendix.
D-8

The reading and mathematics achievement test outcomes are expressed as percentiles in the

distribution of scores among a national sample of tenth graders in 1990. For example, a sample

member at the 47th percentile had a test score that was higher than 47 percent of the national

population of tenth graders in 1990 and lower than 52 percent of tenth graders. QOP sample

members were tested in the spring of the fourth academic year of the demonstration, when they 

should have been in the second semester of the twelfth grade. Percentiles were computed in terms 

of the population of tenth graders, rather than of twelfth graders. The Educational Testing Service 

(ETS) scored both achievement tests.

High School Completion

Data on high school completion were from the telephone survey and from transcripts. In the 

telephone survey, we asked sample members whether they had completed a high school degree or a 

GED. Those who answered yes to this question were then asked whether they had earned a high 

school diploma. In collecting transcripts, we discovered that most of the 11 QOP schools recorded 

high school graduation status on students’ transcripts. The exceptions were Eastern High School in 

Washington, D.C., which did not report graduation status at all, and Benjamin Franklin High School 

in Philadelphia, Pennsylvania, which reported the minimum number of credits required to graduate 

rather than the graduation status for each student. 

Using data from both the telephone survey and the transcripts, we constructed outcomes

measuring high school graduation status at the time of the survey. Of the 892 respondents to the 

telephone survey:

• 659 reported a high school graduation status that was consistent with their transcript.

• 126 reported a high school graduation status, but we were unable to obtain their

transcripts.

• 4 did not answer the survey question on high school graduation status, but graduation

status was recorded on their transcripts.

• 103 reported a high school graduation status that was inconsistent with their transcript. 

Of these:

- 84 reported that they had graduated, but their transcripts did not indicate that

they had graduated.

- 19 reported that they had not graduated, but their transcripts indicated that they

had graduated.

In 64 of the 103 cases with inconsistencies, a close examination of the transcripts and the credits

earned enabled us to reconcile the inconsistency. For the remaining 39 cases, the evidence from 

the transcript was not conclusive, and we gave the youth the benefit of the doubt, using the selfreported graduation status when constructing the outcomes pertaining to high school completion.

Postsecondary Activity

A “good” job offers employer-sponsored health insurance. 
D-9

Risky Behaviors 

Substance Abuse. “Binge” drinking means drinking five or more drinks in a row. Drinking or 

binge drinking is classified as “frequent” if it occurs on at least 8 out of the past 30 days. The

outcome “used any drug in the past 30 days” indicates that the respondent reported using at least 

one of the following illegal drugs or types of illegal drugs: marijuana or hashish; cocaine or crack

cocaine; heroin, opium, or methadone; stimulants; depressants; inhalants; or hallucinogens. Because 

the rates at which sample members were using most of the individual drugs were low and the

evaluation samples for schools and sites were small, impacts could not be reliably estimated by

calculating differences of means. Thus, we do not present impact estimates for individual drugs.

Criminal Activity. The outcomes “committed any crime in the past 12 months” and

“committed any crime in the past 3 months” indicate that the respondent reported committing at 

least one of the following seven crimes: (1) sold illegal drugs, (2) stole a motor vehicle, (3) stole

something other than a motor vehicle, (4) attacked and seriously hurt or killed someone, (5) carried a 

hand gun, (6) committed a sexual assault, or (7) committed any other illegal activity. Because the

rates at which sample members were committing most of the individual crimes were low and the

evaluation samples for schools and sites were small, impacts could not be reliably estimated by

calculating differences of means. Thus, we do not present impact estimates for individual crimes.

Resiliency Factors

Positive Influence of a Caring Adult. We defined a caring adult as “any adult, besides a

family member, who positively influenced your life in some significant way, for example, a teacher, 

counselor, coach or minister.” We asked those who indicated that they were influenced by such a 

person whether each of the following statements were definitely true, mostly true, mostly not true, 

or definitely not true:

• The person pays attention to and respects my ideas and feeling about things in my life.

• The person has helped me learn things that have helped me do well in my life.

• The person has helped me take advantage of opportunities to get ahead in my life.

• The person has recognized and appreciated the things I have done well in my life.

• The person has clear expectations and standards about what I do with my life.

• The person has shown me that fighting is not a good way of solving problems in my life

• The person has shown me that breaking the law does not help me achieve my goals.

• The person has shown me that using drugs or alcohol is not a good way of solving

problems in my life.

We calculated the percentages of QOP-group and control-group members who reported that each 

statement was definitely true. The base of these percentages was the entire QOP group or the entire 

control group, including those who were not influenced by a caring adult.
D-10

Positive Influence of a Social Program. We asked sample members whether they

participated in a social program that had a positive influence on their lives. The interviewer first

announced that she was going to ask about “social programs other than your normal high school

classes. These programs try to help students stay in school, make good grades, stay away from

drugs, prepare for work or college, and make good decisions in life.” Those who said they

participated in such a program since the ninth grade were asked, “How much did the program help 

you in the following ways?” Responses included “a lot”, “a little,” or “not at all.”

• Improve your grades

• Stay away from drugs or get off drugs

• Stay out of trouble

• Deal with the police and courts

• Prepare for college 

• Earn and save money

We calculated the percentages in the QOP and control groups who reported participating in such a 

program and the percentages being helped a lot by such a program in each of the listed dimensions. 

The base of these percentages was the entire QOP group or the entire control group, including

those who did not participate in such a program.

Attitudes Toward Risky Behaviors and Outlook on Life. We measured sample members’ 

attitudes toward risky behaviors by asking whether the youth thought each of the following was

always wrong, sometimes wrong, or not wrong:

• Using drugs or alcohol frequently

• Committing crimes such as stealing, assaulting someone, or selling drugs

• Having a baby while you are a teenager

• Dropping out of school

We calculated the percentages of the QOP and control groups reporting that the risky behaviors

were always wrong.

We measured sample members’ outlook on life by asking whether the youth agreed with each 

of the following statements. Responses included “strongly disagree,” “disagree,” “agree,” and

“strongly agree.”

• Bad things happen to people like me.

• I’m afraid my life will be unhappy.

• I like the way I look.
D-11

• I’ll probably die before I’m thirty.

We calculated the percentages of the QOP and control groups who disagreed or strongly disagreed 

with the first, second, and fourth statements and the percentages who agreed or strongly agreed with 

the third statement.

______________________________________________________________________________

SUBGROUPS

______________________________________________________________________________

In Chapter VII, we present impacts for subgroups defined by two classification schemes. The 

first scheme classified sample members by baseline characteristics—sex, age, and GPA. The second 

scheme classified them by location—each of the seven demonstration sites and Ford-funded sites

versus DOL-funded sites. Table D.2 lists the subgroups.

When assessing impacts for the subgroups defined by rank in the baseline grade distribution, it 

is important to remember that to be eligible for QOP, a youth had to be in the bottom two-thirds of 

the grade distribution based on grades from the eighth grade. Thus, youth in the bottom third of

the baseline grade distribution for QOP eligibles were at or below the 22nd percentile in the

distribution for all youth, including those who were not eligible for QOP based on their grades. 

Likewise, the youth in the middle and top thirds of the baseline grade distribution for QOP eligibles 

were between the 22nd and 44th percentiles and between the 44th and 66th percentiles, respectively, in 

the grade distribution for all youth. 
D-12

TABLE D.2

 SUBGROUPS

Subgroup

Classification According to Baseline Characteristics

Sex

Males

Females

Age when entered ninth grade

14 or younger

Over 14

Rank in baseline grade distribution (based on eighth-grade GPA)

In the bottom third of the grade distribution

In the middle third of the grade distribution

In the top third of the grade distribution

Classification According to Location

Site

Fort Worth

Cleveland

Washington, D.C.

Houston

Memphis

Philadelphia

Yakima

Funding Source

DOL

The Ford Foundation
APPENDIX E

WEIGHTING, IMPACT ESTIMATION, AND VARIANCE ESTIMATION
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
E-3

______________________________________________________________________________

WEIGHTING TO ADJUST FOR NONRESPONSE

______________________________________________________________________________

We developed person-level weights to adjust for the potential effects of unit nonresponse. Unit 

nonresponse occurred, for example, when a sample member did not respond at all to the telephone 

survey, that is, the sample member did not answer any questions on the survey.1 About 16 percent 

of sample members did not respond to the in-person survey or take the achievement tests. About 

17 percent did not respond to the telephone survey, and no transcript data at all were obtained for 

18 percent of sample members. For each data collection activity, the unit nonresponse rate for the 

control group was higher than the overall rate, while the unit nonresponse rate for enrollees was

lower than the overall rate. The difference between the unit nonresponse rates for the two groups 

was about seven or eight percentage points for each data collection activity.

As documented below, respondents and nonrespondents did not have identical baseline

characteristics. Such differences could potentially cause differences between the outcomes of

respondents and those of nonrespondents. In such a circumstance, an impact estimated using data 

from respondents only (since there are no outcome data from nonrespondents) would be a biased 

estimate of the impact that we seek, which is the impact on all sample members, respondents and

nonrespondents. The size of the bias is not estimable.

To adjust for the effects of nonresponse and reduce potential nonresponse bias, we assigned 

weights to respondents. We assigned larger weights to the respondents who more closely resembled 

the nonrespondents in terms of baseline characteristics and smaller weights to the respondents who 

less closely resembled the nonrespondents.2 Although differential weighting of respondents tended 

to increase the variances of impact estimates (by measurable amounts), we accepted small increases 

in variances to enhance our confidence that we controlled nonresponse bias to the extent possible.

Baseline Differences Between Respondents and Nonrespondents

We analyzed differences between respondents and nonrespondents separately for each of the

data collection activities. Because only a very few sample members completed the in-person survey 

but not the achievement tests or vice versa, we have treated those two activities as a single activity 

for the purposes of analyzing patterns of nonresponse and developing weights to adjust for

nonresponse.3

 

1

 In contrast, item nonresponse occurred when a sample member did not provide a valid answer to a question that 

was asked even though he or she answered other questions on the survey. As shown in Appendix C, item nonresponse 

rates were typically very low.

2

 As described in detail below, we evaluated resemblance using response propensity scores.

3

 We treated a sample member as an in-person survey/achievement tests unit nonrespondent only if the sample 

member did not complete the survey and the tests. If a sample member completed either the survey or the tests, the 

sample member was treated as a respondent with, potentially, item nonresponse that was as extensive as the whole

survey or both tests (in just a few instances).
E-4

Table E.1 compares the baseline characteristics of respondents to the in-person survey and

achievement tests with the baseline characteristics of all sample members—respondents and

nonrespondents combined—for the control group and for the QOP group. We have indicated in

the table which differences in baseline characteristics were statistically significant.4 Tables E.2 and 

E.3 compare respondents to the telephone survey and school transcripts, respectively, with all

sample members.

According to Table E.1, 50 percent of QOP enrollees who responded to the in-person survey 

and achievement tests were male. In contrast, 52 percent of all QOP enrollees were male. The

difference was statistically significant. Table E.2 shows a similar pattern for the telephone survey, 

implying that males were less likely than females to respond to either of our surveys. According to 

Tables E.1 and E.2, older youth and youth in the bottom third of the grade distribution generally 

had lower response rates to our surveys than younger youth and youth with higher grades, although 

differences were not always significant.

Table E.3 indicates that the patterns of nonresponse for transcripts were broadly similar to the 

patterns of nonresponse for the surveys.5 However, for transcripts, there were more significant

differences pertaining to sample members’ schools, reflecting the direct role of schools in providing 

transcripts. 

Weights

We corrected for the possibility of nonresponse bias by weighting each sample member. To

derive weights, we estimated three separate logit regression models to predict the probabilities that 

each sample member was a respondent. We repeated this process for (1) the in-person survey and 

achievement tests, (2) the telephone survey, and (3) the school transcripts. Using the three estimated 

models, we obtained three sets of weights. Then, we estimated the impact on an outcome using the 

weight associated with the source of data on that outcome. For example, the impact on an outcome 

measured in the telephone survey was based on weights that adjusted for telephone survey

nonresponse.

We derived each set of weights by carrying out the following four steps:

1. We estimated a “best” logit model for predicting response propensity scores

(probabilities). The best regression model included 22 predictors that we “forced”

into the model and additional predictors that we selected using an automated forward

selection procedure with a liberal inclusion criterion.6 The predictors forced into the

 

4

 The test statistic pertaining to a difference between respondents and all sample members is the same as the test 

statistic pertaining to a difference between respondents and nonrespondents.

5

 A sample member was classified as a unit nonrespondent for transcripts if we received no transcript data at all for 

that sample member. Otherwise, the sample member was a respondent with, potentially, item nonresponse. The

patterns of nonresponse for surveys and transcripts were similar because we had to use the telephone survey to obtain a 

sample member’s permission to request his or her transcript if the sample member was at least 18 years old, as most

sample members were when transcripts were collected.

6

 Our model selection procedure first estimated coefficients for the predictors forced into the model. Then, the 

procedure determined which excluded predictor had the largest adjusted chi-squared statistic for inclusion in the model. 

If the statistic was significant at the 75 percent confidence level, the procedure added the predictor to the model. The 

procedure never removed a predictor from the model. The procedure continued evaluating and adding excluded

predictors until there was no excluded predictor that satisfied the criterion for inclusion.
E-5

TABLE E.1

THE BASELINE CHARACTERISTICS OF RESPONDENTS TO THE IN-PERSON SURVEY AND 

ACHIEVEMENT TESTS COMPARED WITH THE BASELINE CHARACTERISTICS

OF ALL SAMPLE MEMBERS

(Percentages)

QOP Group Control Group

Baseline Characteristic Respondents

Respondents and 

Nonrespondents Respondents

Respondents and 

Nonrespondents

Male 50††† 52 54 56

Age when entering ninth grade

< 14 12††† 11 12 11

14 55 †† 53 59† 57

> 14 33††† 36 29 †† 31

Black 68 68 69 68

Hispanic 26 26 26 26

Rank based on eighth-grade GPA

Bottom third 37 37 32 †† 34

Middle third 31 31 35 36

Top third 32 32 34††† 30

School

Paschal (Fort Worth) 17 17 15 16

Collinwood (Cleveland) 17 17 12 12

Eastern (Washington, D.C.) 7 7 9 9

Anacostia (Washington, D.C.) 7 7 6 7

Yates (Houston) 9 9 8 9

Austin (Houston) 7 9 10 9

Hillcrest (Memphis) 7 7 7 8

Hamilton (Memphis) 4 5 6 6

Carver (Memphis) 6 6 8 7

Franklin (Philadelphia) 9 9 10 9

Davis (Yakima) 8 9 9 10

Sample Size 510 580 391 489

SOURCE: Baseline database, in-person survey, and achievement tests.

† Significantly different from nonrespondents at the 90% confidence level, two-tailed test

†† Significantly different from nonrespondents at the 95% confidence level, two-tailed test

††† Significantly different from nonrespondents at the 99% confidence level, two-tailed test
E-6

TABLE E.2

THE BASELINE CHARACTERISTICS OF RESPONDENTS TO THE TELEPHONE SURVEY COMPARED 

WITH THE BASELINE CHARACTERISTICS OF ALL SAMPLE MEMBERS

(Percentages)

QOP Group Control Group

Baseline Characteristic Respondents

Respondents and 

Nonrespondents Respondents

Respondents and 

Nonrespondents

Male 51† 52 55 56

Age when entering ninth grade

< 14 12†† 11 11 11

14 54 53 59 57

> 14 35† 36 30 31

Black 69 68 65††† 68

Hispanic 25 26 29††† 26

Rank based on eighth-grade GPA

Bottom third 37 37 31††† 34

Middle third 31 31 37 36

Top third 32 32 32††† 30

School

Paschal (Fort Worth) 17 17 16 16

Collinwood (Cleveland) 17 17 13 12

Eastern (Washington, D.C.) 8

†† 7 8 9

Anacostia (Washington, D.C.) 7 7 4

††† 7

Yates (Houston) 9 9 8 9

Austin (Houston) 9 9 10 9

Hillcrest (Memphis) 7 7 7 8

Hamilton (Memphis) 4 5 5 6

Carver (Memphis) 6 6 7 7

Franklin (Philadelphia) 8 9 9 9

Davis (Yakima) 8 9 11 10

Sample Size 503 580 389 489

SOURCE: Baseline database and telephone survey.

† Significantly different from nonrespondents at the 90% confidence level, two-tailed test

†† Significantly different from nonrespondents at the 95% confidence level, two-tailed test

††† Significantly different from nonrespondents at the 99% confidence level, two-tailed test
E-7

TABLE E.3

THE BASELINE CHARACTERISTICS OF RESPONDENTS TO THE SCHOOL

RECORDS DATA COLLECTION COMPARED WITH THE BASELINE CHARACTERISTICS OF ALL 

SAMPLE MEMBERS

(Percentages)

QOP Group Control Group

Baseline Characteristic Respondents

Respondents and 

Nonrespondents Respondents

Respondents and 

Nonrespondents

Male 50†† 52 53†† 56

Age when entering ninth grade

< 14 11 11 12† 11

14 56††† 53 60†† 57

> 14 33††† 36 27††† 31

Black 66†† 68 69 68

Hispanic 28††† 26 27 26

Rank based on eighth-grade GPA

Bottom third 36 37 30††† 34

Middle third 31 31 36 36

Top third 33† 32 34

††† 30

School

Paschal (Fort Worth) 19†† 17 16 16

Collinwood (Cleveland) 14††† 17 11 12

Eastern (Washington, D.C.) 8

††† 7 10†† 9

Anacostia (Washington, D.C.) 7

† 7 6 7

Yates (Houston) 9

† 9 8 9

Austin (Houston) 10 9 10 9

Hillcrest (Memphis) 6 7 8 8

Hamilton (Memphis) 4

†† 5 4† 6

Carver (Memphis) 6 6 7 7

Franklin (Philadelphia) 8 9 9 9

Davis (Yakima) 9 9 9 10

Sample Size 495 580 376 489

SOURCE: Baseline database and transcripts.

† Significantly different from nonrespondents at the 90% confidence level, two-tailed test

†† Significantly different from nonrespondents at the 95% confidence level, two-tailed test

††† Significantly different from nonrespondents at the 99% confidence level, two-tailed test
E-8

model were an intercept, 10 school indicators, a QOP/control indicator, and the 10

interactions between the QOP/control indicator and the 10 school indicators. 

Additional potential predictors included the following baseline characteristics: an

indicator for being male, an indicator for being black, an indicator for being Hispanic,

two indicators for age when entering ninth grade (one for under 14 and one for over 14), 

two indicators for rank based on eighth-grade GPA (one for middle third and one for

top third), percentile rank based on eighth-grade GPA, and the percentile rank squared. 

Other potential predictors included the interactions between any two baseline

characteristics (except for interactions between the predictors based on eighth-grade 

GPA), the interactions between any baseline characteristic and the QOP/control

indicator, and the interactions between any baseline characteristic and any school

indicator. An exception to this was that the indicators for race and Hispanic origin were 

interacted with the school indicators for only the two schools with substantial diversity 

by race or Hispanic origin (Paschal in Fort Worth and Davis in Yakima).

2. We derived predicted response propensity scores based on sample members’

characteristics using the best logit model.

3. We assigned a weight to a respondent equal to the inverse of the respondent’s

propensity score. We assigned a weight equal to zero to each nonrespondent. To

reduce the variability in weights and the resulting increase in variance of impact

estimates, we trimmed weights to 2.5, that is, any weight greater than 2.5 was set equal to 

2.5.7 Weighting respondents based on inverse propensity scores ensured that we

assigned a relatively large weight to a respondent who had a relatively low response

propensity and, thus, resembled a nonrespondent.

4. We ratio adjusted weights to sum to the number of respondents in each of the 22 

weighting classes defined by the cross-classification of school (11 categories) and 

experimental group (2 categories—QOP and control).

Having developed these weights, we estimated an impact as the difference between the weighted

QOP-group mean and the weighted control-group mean. The details of the estimation of impacts and 

the variances of those impacts are presented in the next section.8

______________________________________________________________________________

ESTIMATING IMPACTS AND VARIANCES OF IMPACTS

______________________________________________________________________________

Impacts for Schools

We estimated the impact for a school according to:

school Q,school C,school

impact = − X X

 

7

 This resulted in the trimming of none of the weights that adjusted for in-person survey/achievement tests

nonresponse, five of the weights that adjusted for telephone survey nonresponse, and nine of the weights that adjusted 

for transcript nonresponse.

8

 Comparing impacts derived from unweighted means (not presented in this report) with the impacts derived from 

weighted means revealed that the estimated impacts and our conclusions based on them are not sensitive to whether

weights are used.
E-9

where X is the outcome of interest, Q and C denote the QOP and control groups, and, for example: 

,

,

i i

i Q school

Q,school

i

i Q school

w X

X

w

∈

∈

=

∑

∑

wiis the weight for youth i.

9

 In other words, we estimated the impact for a school by subtracting

the mean outcome among youth in the control group for that school from the mean outcome

among QOP enrollees for that school, as described in the main text. Each youth remained a

member of the group to which he or she was originally assigned, regardless of subsequent behavior, 

such as transferring to another school, dropping out of school, or (for enrollees) dropping out of

QOP. 

Treating the QOP group and the control group as independent samples from a

superpopulation, we estimated the variance (the standard error squared) of the school-level impact

according to:10

var ( ) var( ) var( ) var( ) school Q,school C,school Q,school C,schoolimpact = X − XXX = +

We estimated the variance of the QOP-group mean for a given school according to:11

( ) ( )

2

, 2

2 , ,

,

,

,

1

var

1

Q school

Q school i i Q school

i Q school Q school

i

i Q school

n

X w X X

n

w

∈

∈

 

= −  

    −

     

∑

∑

where nQ,school is the number of responding youth in the school’s QOP group. When calculating any 

of the sums needed for estimating a mean or a variance, we included only those youth with valid 

(nonmissing) data for the outcome under consideration.12

 

9

 Because all of the youth from a school had the same probability of assignment to the QOP group, the only

purpose of weighting is to adjust for unit nonresponse. We described earlier in this appendix how we derived weights.

10 The basic idea is that we are not really interested in just the small population of youth who were eligible for

random assignment. Rather, we would like to generalize to a “superpopulation” that includes other youth, including 

those who met the four QOP eligibility criteria (but were not selected for the initial sample) and those who would have 

been eligible in prior or subsequent academic years. If the group of youth eligible for random assignment were our

population of interest, the QOP and control means would be correlated (because the control group is the complement 

of the QOP group). However, that correlation is not estimable—without some simplifying assumption—because we

observe each youth in only one experimental state, that is, as either a QOP enrollee or a control. One simplifying

assumption is that the impact of QOP is additive and fixed (the same for all youth). This assumption and the

superpopulation approach lead to the same statistical procedure.

11 A similar expression pertains for the variance of the control-group mean.

12 Sample members who did not respond at all to the survey (or other data collection activity) from which data

were obtained for the outcome under consideration were excluded because their weights were equal to zero. Sample

members who responded to the survey but did not answer the question or questions relevant to the outcome were

excluded from only those calculations for which they were missing data. The former group was substantially larger than 

the latter group for all of the outcomes that we considered. We sought to compensate for the loss of the former group 

by weighting respondents, as described previously.
E-10

Impacts for Sites

We estimated the impact for a site according to: 

 =

site school school

school Hsite

impact W impact

∈

∑

where Hsite is the set of schools from which youth were selected for the QOP program operated at 

the site in question.13 In other words, we estimated the impact for a site by taking a weighted 

average of the impacts for the schools at that site. We based the school-level weights (the W) on the 

allocation of slots observed in the demonstration. In fact, Wschool was the fraction of the site’s QOP 

slots allocated to the particular school. Thus, Wschool was 1.00 for Collinwood (Cleveland), Paschal 

(Fort Worth), Franklin (Philadelphia), and Davis (Yakima); 0.50 for Anacostia (Washington, D.C.),

Eastern (Washington, D.C.), Austin (Houston), and Yates (Houston); 0.35 for Carver (Memphis);

0.27 for Hamilton (Memphis); and 0.38 for Hillcrest (Memphis). This was our best estimate of how 

slots would have been allocated had the sites been part of an ongoing, national program. In such a 

program, as in the demonstration, CBOs in some sites would work with just one school, while

CBOs in other sites would have the same number of slots, but work with two or three schools. In 

the latter case, the CBOs would likely allocate slots in the same way that the CBOs in the

demonstration did. Note that for each site: 

1

school

school Hsite

W

∈

∑ =

This approach to weighting schools when calculating an impact estimate for a site implied, for

example, that: 

0.35 0.27 0.38

Memphis Carver Hamilton Hillcrest

impact = ×impact + ×impact + ×impact

Treating the allocation of QOP slots across schools within a site as fixed, we estimated the

variance of the site-level impact according to: 

( ) ( )

2

var = var

site school school

school Hsite

impact W impact

∈

∑

This expression reflects the fact that for each site, we had the full population of schools from which 

youth were selected and the fact that random assignment was carried out independently in each

school.

 

13 Hsite consists of one school for Cleveland, Fort Worth, Philadelphia, and Yakima; two schools for Washington, 

D.C., and Houston; and three schools for Memphis.
E-11

Impacts for the Whole Demonstration

We estimated the impact for the whole QOP demonstration according to: 

7

 = 1

 = 

demo site site

site

impact ∑ A impact

where 

7

 = 1

 1

site

site

∑ A =

Except when we assessed the sensitivity of our demonstration-level impact estimates to how we

weighted sites (see Appendix F), we assumed that Asite = 1/7 for all sites. Thus, we estimated the 

impact for the whole demonstration by taking the simple average of the seven site impacts. As

noted in the main text, our equal weighting of sites was based on the belief—or best guess—that if 

QOP were implemented as an ongoing, national program, CBOs would have roughly equal numbers 

of QOP slots. The relatively small sizes of the Washington, D.C., and Ford-funded programs in the 

demonstration were due to circumstances that we do not think would be replicated in a regular

program.14

For deriving all of the estimates presented in this report, we assumed that the collection of sites 

in the QOP demonstration was a fixed set, that is, a population. Thus, we estimated the variance of 

the demonstration-level impact according to: 

( ) ( )

7

2

 = 1

var = var

demo site site

site

impact ∑ A impact

Although the sites in the demonstration were not really a population, they were also not a

probability sample. Nevertheless, if statistically significant impact estimates from the demonstration 

are to be useful for informing policy, the demonstration sites must be approximately representative 

of a population of potential sites. Then, we would want to treat the demonstration sites as a random 

sample (of size seven), and estimate the “total” variance of an impact estimate. The total variance 

has both a within-site component and a between-site component. The within-site component

reflects the sampling error from selecting samples of youth in each site, and is captured by the

expression already given for the variance of the demonstration-level impact. The between-site 

component reflects the differences among the impacts for the different sites. Although we might 

have preferred to obtain estimates of total variances, we were not able to estimate total variances

very precisely because there were only seven sites in the demonstration. In fact, we discovered that 

for a large majority of impacts, the estimated total variance was smaller—often substantially

smaller—than the estimated within-site component of variance. Because we preferred a wellestimated within-site component of variance to a poorly estimated total variance, we have presented 

the former as our variance estimates.

 

14 The Ford Foundation wanted to fund two sites, but at only half of the size of DOL-funded sites, an outcome 

that would be unlikely to occur in a program that is fully funded by the federal government. The Washington, D.C., site 

was allocated 100 QOP slots, but given the short duration of the demonstration and the one-year delay in beginning

program operations in the site, efforts to identify eligible youth were halted at a third QOP school that would have had 

20 slots. This decision was not made early enough to increase the number of slots at the two remaining QOP schools.
E-12

As discussed in the main text, we conducted t-tests to determine whether estimated impacts

were significantly different from zero. For a t-test, we calculated a t-statistic by dividing an impact 

estimate by its standard error. The standard error is the square root of the variance, and the variance 

was estimated according to the relevant expression given in this appendix.
APPENDIX F

SENSITIVITY ANALYSES
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
F-3

In this appendix, we test the sensitivity of the impact estimates to alternative estimation

approaches. In particular, we test the sensitivity to:

• Alternative approaches to weighting site-level impacts. The impact estimates for the whole

demonstration that are presented in this report were derived by weighting site-level 

impacts as though each site had the same number of QOP slots and enrollees. The

alternative approach considered in this appendix weights site-level impacts in proportion 

to the actual number of slots and enrollees that sites had in the demonstration. 

• Using regression methods to adjust the impact estimates for random differences between the QOP group 

and the control group. Although the difference-of-means estimates presented in this report 

are unbiased, they may have been affected by purely random differences between the

baseline characteristics of QOP enrollees and the baseline characteristics of youth in the 

control group. Therefore, we adjusted for such differences using regression methods.

In each case, we determined whether our conclusions would have been different had they been 

based on estimates derived using the alternative approach. We found that our conclusions are

generally robust to alternative approaches.

ALTERNATIVE APPROACHES TO WEIGHTING SITE-LEVEL IMPACTS

Table F.1 presents the demonstration-level impact estimates—that is, estimates for all sites

combined—that we obtained using two approaches to weighting site-level impacts: (1) weighting 

site-level impacts equally (the approach used everywhere else in this report) and (2) weighting sitelevel impacts in proportion to the number of QOP slots in each demonstration site. The latter

approach assigned the following weights: 100/580 (. 0.172) for the Cleveland, Fort Worth,

Houston, and Memphis sites; 80/580 (. 0.138) for the Washington, D.C., site; and 50/580 (. 0.086) 

for the Philadelphia and Yakima sites. We estimated impacts and variances according to the

expressions given in Appendix E, which allow for both equal and unequal weighting.

In Chapter VII, we found that impacts for the two Ford-funded sites were often much larger in 

absolute value than the impacts for the five DOL-funded sites (although many differences were not 

significant). Therefore, because the approach that weights sites unequally gives less weight to the

Ford-funded sites than the approach that weights sites equally, we were not surprised to find that 

the demonstration-level impacts obtained when we weighted sites unequally are somewhat closer to 

zero than the impacts obtained when we weighted sites equally, if there is any difference at all.

Three of the four significant impacts obtained when we weighted sites equally are still

significant—but smaller by one percentage point—when we weighted sites unequally. The fourth

significant impact pertained to the likelihood of engaging in postsecondary education or training. 

When we weighted sites unequally, the impact on this outcome was one percentage point smaller—

five rather than six percentage points—and not significantly different from zero. Thus, our

conclusion that QOP significantly increased the likelihood of engaging in postsecondary education 

or training is sensitive to how we weighted site-level impacts. However, our conclusions about the 

significance of other impacts are not sensitive to how we weighted site-level impacts.
F-4

TABLE F.1

SHORT-TERM IMPACTS WHEN SITES ARE WEIGHTED EQUALLY VERSUS SHORT-TERM IMPACTS 

WHEN SITES ARE WEIGHTED UNEQUALLY

(Percentage points except where noted)

Impacts

Outcomea

Sites

Weighted

Equally

Sites Weighted 

Proportional to Number 

of QOP Slots

Mathematics test score (percentile) 0.38 0.20

Reading test score (percentile) 0.50 0.33

GPA (four-point scale) -0.06 -0.06

Earned high school diploma 7

* 6**

Earned diploma or GED certificate 5 4

Attending college 3 3

Attending postsecondary training 6

* 5

Postsecondary training or good job 5 3

Postsecondary training or high school or GED class or any 

job 3 3

Attending or accepted into college 6

* 5*

Binge drinking in past 30 days 4 2

Used any illegal drug in past 30 days 7

** 6*

Committed any crime in past 12 months 3 1

Ever arrested or charged -5 -5

Have one or more own children -3 -1

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts.

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP

enrollees and 489 controls.

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row.

* Estimate significantly different from zero at the 90% confidence level, two-tailed test

** Estimate significantly different from zero at the 95% confidence level, two-tailed test

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
F-5

REGRESSION-ADJUSTED IMPACTS

Baseline Differences Between the QOP and Control Groups

According to Table F.2, there was just one statistically significant difference between the means 

of baseline characteristics for the QOP and control groups for the whole demonstration. Compared 

with the control group, the QOP group had fewer youth in the middle third of the eighth-grade 

GPA distribution. When we examined QOP- and control-group means for schools—the level at 

which we conducted random assignment—we found only a few significant differences (not shown 

in Table F.2). For example, compared with the school’s control group, the QOP group from Austin 

High School (Houston) had more youth over age 14 and fewer youth from the top third of the

grade distribution; the QOP group from Yates High School (Houston) had fewer youth from the

bottom third and more youth from the top third of the grade distribution; and the QOP group from 

Hillcrest High School (Memphis) had more youth from the middle third of the grade distribution.

Deriving Regression-Adjusted Impact Estimates

We estimated a separate regression model for each of the key outcomes. Each regression

model included 26 variables that we forced into the model and additional variables that were

selected using an automated forward selection procedure. The variables that we forced into the

model (and thus the variables common to all models) were 11 school indicators, 22 interactions

between a QOP/control indicator and the 11 school indicators, an indicator for being male, an

indicator for being over age 14 when entering ninth grade, an indicator for being in the middle third 

of the eighth-grade GPA distribution, and an indicator for being in the top third of the eight-grade 

GPA distribution. The variables that our forward selection procedure potentially added to the

model were the 44 variables obtained by interacting the last four variables that were forced into the 

model with the 11 school indicators.1

For three outcomes—the mathematics test score, the reading test score, and high school

GPA—we used linear regression methods to estimate the model chosen by our selection procedure. 

For the other outcomes, we used logit regression methods to estimate the chosen model.2

 

1

 Our model selection procedure first estimated coefficients for the variables forced into the model. Then, the

procedure determined which excluded variable had the largest F-statistic for inclusion in the model. If the statistic was 

significant at the 85 percent confidence level, the procedure added the variable to the model. The procedure never

removed a variable from the model. The procedure continued evaluating and adding excluded variables until there was 

no excluded variable that satisfied the criterion for inclusion.

2

 We used logit regression methods because the outcomes are binary, e.g. the youth either graduated or did not. 

Although we estimated the chosen models using logit regression methods, we used linear regression methods when

implementing the forward selection procedure to choose a model. We found that regression-adjusted impact estimates 

obtained from logit models differ very little from estimates obtained from linear models. We report only the former.
F-6

TABLE F.2

GROUP MEANS FOR BASELINE CHARACTERISTICS

(Percentages)

Means

Baseline Characteristic QOP Group Control Group

Male 52 56

Age when entering ninth grade

< 14 11 11

14 53 57

> 14 36 31

Hispanic 26 26

Black 68 68

Rank based on eighth-grade GPA

Bottom Third 37 34

Middle Third 31† 36†

Top Third 32 30

SOURCE: Baseline database. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls.

† Significantly different from the mean for the other group at the 90% confidence level, two-tailed test

†† Significantly different from the mean for the other group at the 95% confidence level, two-tailed test

††† Significantly different from the mean for the other group at the 99% confidence level, two-tailed test
F-7

When we used linear regression methods, our estimate of the impact for a school was the

estimated coefficient for the interaction between that school’s indicator and the QOP/control

indicator. Our estimate of the variance of the impact was the variance for the estimated coefficient. 

When we used logit regression methods, we obtained impact and variance estimates for schools

from probabilities predicted by the logit model for the outcome under consideration.3 After

deriving school-level estimates, we derived site- and demonstration-level estimates using the

expressions in Appendix E.

Regression-Adjusted Impact Estimates

Table F.3 presents difference-of-means and regression-adjusted impact estimates, and Table F.4 

presents standard errors and p-values for the impact estimates. Comparing the alternative impact

estimates suggests that regression adjustment might affect only one of our conclusions—that QOP 

significantly increased the likelihood that an enrollee engaged in postsecondary education or training. 

The other three impacts that were significant according to difference-of-means estimates are also

significant according to regression-adjusted estimates.

Table F.3 shows that regression adjustment reduced by one percentage point—from six to five 

percentage points—the impact on the likelihood of engaging in postsecondary education or training. 

Table F.4 shows that regression adjustment had a negligible effect on the precision of the impact

estimate for this outcome (and most of the other outcomes). Because of the change in the impact 

estimate, the p-value for the t-statistic rose from 7.4 to 12.1 percent, crossing the 10 percent

threshold. Thus, a significant impact became insignificant. The same impact became smaller and 

insignificant when we weighted site impacts unequally rather than equally, as discussed earlier in this 

appendix.

 

3

 Suppose that we are estimating a regression-adjusted impact on high school graduation. Then, for every youth 

from a given school, we used the estimated logit model for high school graduation to obtain four predicted probabilities 

while ignoring the youth’s actual QOP/control status: (1) the probability of graduation if the youth is a control, (2) the 

probability of graduation if the youth is a QOP enrollee and the effect of QOP is measured by the coefficient on the 

interaction between the indicator for the youth’s school and the QOP/control indicator, (3) the probability of 

graduation if the youth is a QOP enrollee and the effect of QOP is measured by the coefficient on the interaction

between the indicator for the youth’s school and the QOP/control indicator plus the standard error of the coefficient, 

and (4) the probability of graduation if the youth is a QOP enrollee and the effect of QOP is measured by the coefficient 

on the interaction between the indicator for the youth’s school and the QOP/control indicator minus the standard error 

of the coefficient. Next, we calculated the mean for each of these probabilities across all of the youth from the school. 

The second mean minus the first mean was our impact estimate for the school. We estimated the variance of the impact 

by squaring the difference between the third and fourth means and dividing by four.
F-8

TABLE F.3

DIFFERENCE-OF-MEANS IMPACT ESTIMATES VERSUS REGRESSION-ADJUSTED IMPACT ESTIMATES

(Percentage points except where noted)

Impacts

Outcomea Difference of Means Regression Adjusted

Mathematics test score (percentile) 0.38 0.52

Reading test score (percentile) 0.50 0.57

GPA (four-point scale) -0.06 -0.04

Earned high school diploma 7

* 7**

Earned diploma or GED certificate 5 5

Attending college 3 3

Attending postsecondary training 6

* 5

Postsecondary training or good job 5 5

Postsecondary training or high school or GED class or any 

job 3 3

Attending or accepted into college 6

* 6*

Binge drinking in past 30 days 4 3

Used any illegal drug in past 30 days 7

** 7**

Committed any crime in past 12 months 3 4

Ever arrested or charged -5 -3

Have one or more own children -3 -4

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls.

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row.

* Estimate significantly different from zero at the 90% confidence level, two-tailed test

** Estimate significantly different from zero at the 95% confidence level, two-tailed test

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
F-9

TABLE F.4

STANDARD ERRORS AND P-VALUES FOR IMPACT ESTIMATES

(Percentage points except where noted)

Difference-of-Means Estimator Regression-Adjusted Estimator

Outcomea

Standard 

Error p-value

Standard 

Error p-value

Mathematics test score (percentile) 0.488 42.9 0.460 26.2

Reading test score (percentile) 0.528 33.2 0.514 26.4

GPA (four-point scale) 0.045 18.4 0.043 30.1

Earned high school diploma 3.4 5.0 3.3 4.5

Earned diploma or GED certificate 3.5 18.7 3.2 10.7

Attending college 2.8 22.9 3.0 29.0

Attending postsecondary training 3.2 7.4 3.3 12.1

Postsecondary training or good job 3.6 18.6 3.5 13.6

Postsecondary training or high school or GED 

class or any job 2.8 22.1 2.8 21.9

Attending or accepted into college 3.2 7.6 3.2 8.0

Binge drinking in past 30 days 3.0 16.9 3.6 42.8

Used any illegal drug in past 30 days 3.3 4.7 3.5 3.9

Committed any crime in past 12 months 3.3 35.8 3.4 29.4

Ever arrested or charged 3.3 14.3 2.9 31.4

Have one or more own children 3.1 29.5 2.8 17.3

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls.

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row.
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
APPENDIX G

SHORT-TERM IMPACTS ON PARTICIPANTS
Page Is Intentionally Left Blank to Allow for Double-Sided Copying 
G-3

In this appendix, we present impacts for participants. Participants in the QOP demonstration 

were the enrollees who showed up and participated in program activities. No-shows were the

enrollees who never showed up. Together, the participants and no-shows constituted the full group 

of QOP enrollees. All of the other impacts in this report pertain to enrollees.

We determined which enrollees showed up and thus were participants using data from the

QOP management information system (MIS), which was used by program staff to calculate stipends 

and accrual account contributions. According to the MIS data, which were described in more detail 

in Chapter IV, only 5 (out of 580) enrollees had exactly zero hours of participation—the level of

participation that corresponds to the literal definition of not showing up. Our operational definition 

of not showing up was participating for no more than 100 hours during the whole demonstration. 

Although this particular threshold is necessarily arbitrary, we believe that such a level of participation 

would usually result in a negligible impact on an enrollee. Most enrollees with so few hours

participated in a few, generally recreational activities at the beginning of the demonstration and did 

not participate meaningfully after that. By our operational definition of not showing up, about 12 

percent of QOP enrollees were no-shows. Thus, the show-up rate was 88 percent. Across the 11 

QOP schools, show-up rates ranged from 77 to 96 percent.

To estimate impacts for participants, we assumed that QOP had no impact on no-shows. 

Then, if the impact per enrollee and the show-up rate for a school were impactE,school and showschool, 

respectively, the impact per participant was (Bloom, 1984):

,

,

E school

P school

school

impact

impact

show

=

where showschool is expressed as a proportion rather than a percentage. In other words, showschool is 

equal to the number of participants divided by the number of enrollees. To take account of the

error in the estimated impact per enrollee, the error in the estimated show rate, and the correlation 

between those errors, we estimated the variance of the impact per participant according to:

( )

( ) ( )

( )

2

, ,

, 2 2

, ,

2

var var

var

2 cov ,

 

E school P school school

P school

school school

P school E school school

school

impact impact show

impact

show show

impact impact show

show

= +

−

We derived the impact per enrollee and its variance according to the expressions in Appendix E,

while obtaining the variance of the show rate according to:

( )

( )

,

1

var

1

school school

school

Q school

show show

show

N

−

=

−
G-4

where NQ,school is the number of QOP enrollees from the school. Then, noting that a control-group 

mean is uncorrelated with the show rate for enrollees, we estimated the covariance between the

impact per enrollee and the show rate according to:

( )( )

, ,

, 2

2 ,

,

,

,

cov( , ) cov( , )

1

 

1

E school school Q school school

Q school

i i Q school i school

i Q school Q school

i

i Q school

impact show X show

n

w X X S S

n

w

∈

∈

=

 

=   − −

    −

     

∑

∑

where Siis an indicator for whether enrollee i showed up and S school is the weighted (by wi) mean 

of Si over the nQ,school QOP enrollees with nonmissing values of X. We derived wi and the mean of 

Xi as described in Appendix E. After we estimated the impact per participant and the variance of

that impact for each school, we derived impact and variance estimates for each site and for the

whole demonstration (all sites combined) according to the expressions in Appendix E pertaining to 

site- and demonstration-level impacts.

Because an impact on participants for a school is obtained by dividing the impact on enrollees 

by a number that is less than or equal to one—that is, the show rate expressed as a proportion—the 

impact on participants is at least as large as the impact on enrollees. Also, the standard error for an 

impact on participants tends to be larger than the standard error for the impact on enrollees by the 

same proportion by which the impact on participants is larger than the impact on enrollees. Thus, 

the t-statistics pertaining to impacts on participants and impacts on enrollees tend to be about equal, 

and significance tests usually lead to the same conclusions.

Table G.1 presents the impacts on participants and the previously discussed impacts on

enrollees for the whole demonstration. Many of the impacts on participants rounded to the same

values as the impacts on enrollees, and the rest of the impacts on participants are only slightly larger 

than the impacts on enrollees. All of the significance tests led to the same conclusions for

participants and enrollees. 

 
G-5

TABLE G.1

SHORT-TERM IMPACTS ON QOP PARTICIPANTS AND QOP ENROLLEES

(Percentage points except where noted)

Impacts

Outcomea Participants Enrollees

Mathematics test score (percentile) 0.42 0.38

Reading test score (percentile) 0.55 0.50

GPA (four-point scale) -0.08 -0.06

Earned high school diploma 8

** 7*

Earned diploma or GED certificate 5 5

Attending college 4 3

Attending postsecondary training 6

* 6*

Postsecondary training or good job 5 5

Postsecondary training or high school or GED class or any job 4 3

Attending or accepted into college 6

* 6*

Binge drinking in past 30 days 4 4

Used any illegal drug in past 30 days 7

* 7**

Committed any crime in past 12 months 3 3

Ever arrested or charged -5 -5

Have one or more own children -3 -3

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts.

NOTE: The evaluation sample had 580 QOP enrollees and 489 controls.

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row.

* Estimate significantly different from zero at the 90% confidence level, two-tailed test

** Estimate significantly different from zero at the 95% confidence level, two-tailed test

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test