Contract No.: K-5547-5-00-80-30 

MPR Reference No.: 8279-093 

The Quantum 

Opportunity Program 

Demonstration: 

Implementation and 

Short-Term Impacts

August 2003 

Myles Maxfield 

Allen Schirm 

Nuria Rodriguez-Planas 

Submitted to: 

U.S. Department of Labor 

Employment and Training Administration 

200 Constitution Avenue, N.W. 

Room N-5637 

Washington, DC 20210 

Project Officer: Eileen Pederson 

Submitted by: 

Mathematica Policy Research, Inc. 

600 Maryland Avenue, S.W. 

Suite 550 

Washington, DC 20024 

(202) 484-9220 

Project Director: Allen Schirm 
 ii 

D ISCLAIMER

This report was prepared under Contract No. K-5547-5-00-80-30 from the U.S. Department of 

Labor. The views expressed herein do not necessarily reflect the policies or opinions of the U.S. 

Department of Labor. 

A CKNOWLEDGMENTS

The authors would like to acknowledge the contributions of David Lah, Eileen Pederson, 

Daniel Ryan, and James Woods of the U.S. Department of Labor. Several members of the 

Mathematica Policy Research staff also contributed to this report. Susan Mitchell and Julie Young 

directed the data collection. John Burghardt, Walter Corson, Stuart Kerachsky, David Myers, Peter 

Schochet, and Michael Sinclair of MPR, as well as Alan Zaslavsky of Harvard University, provided 

valuable technical guidance and comments on draft sections of the report. Alfreda Holmes prepared 

the manuscript. Errors and omissions are the sole responsibility of the authors. 
 iii 

C ONTENTS

Section Page 

Executive Summary..........................................................................................................................................iv

Introduction........................................................................................................................................................1

The QOP Target Group and Program Model ..............................................................................................2

How Does QOP Compare to Other Youth Programs? ..............................................................................4

How Well Was QOP Implemented?...............................................................................................................6

How Much Did QOP Cost? ............................................................................................................................8

How Much Did Enrollees Participate in QOP?............................................................................................8

What Were the Short-Term Impacts of QOP?.............................................................................................9

Summary............................................................................................................................................................23

References.........................................................................................................................................................25 
iv 

E XECUTIVE S UMMARY

From July 1995 through September 2001, the U.S. Department of Labor (DOL) and The Ford 

Foundation (Ford) operated a demonstration of the Quantum Opportunity Program (QOP). QOP 

offered intensive and comprehensive services to help at-risk youth graduate from high school and 

enroll in postsecondary education or training. The QOP demonstration included several features of 

Workforce Investment Act (WIA) youth programs, and findings from the demonstration might 

provide some insight about the implementation challenges that such WIA programs will encounter 

and the potential effectiveness of those programs. 

The QOP demonstration targeted youth with low grades entering high schools with high 

dropout rates. Randomly selected eligible youth were enrolled in QOP and served even if they 

transferred to other schools, dropped out of school, became incarcerated, or became inactive in 

QOP for a long time. QOP’s primary goals were to increase the rates of high school graduation and 

enrollment in postsecondary education or training. Its secondary goals were to improve high school 

grades and achievement test scores and to reduce risky behaviors, such as substance abuse, crime, 

and teen parenting. 

QOP was mainly an after-school program providing case management and mentoring, 

supplemental education, developmental activities, community service activities, supportive services, 

and financial incentives. These services were provided year-round for five years to enrollees who 

had not graduated from high school, and were designed to be comprehensive enough to address all 

barriers to success and to be intensive. The program model specified roughly 15 to 25 enrollees per 

case manager, and it prescribed an annual participation goal of 750 hours for each enrollee who had 

not graduated. From graduation to the end of the demonstration, enrollees who had graduated 

received limited services—some mentoring and assistance with enrolling in postsecondary education 

or training. 

Community-based organizations (CBOs) in seven sites operated QOP demonstration programs. 

Five sites (Cleveland, Fort Worth, Houston, Memphis, and Washington, D.C.) were funded by 

DOL. Four of the five served 100 youth each, and the Washington, D.C., site served 80 youth. The 

other two sites (Philadelphia and Yakima) served 50 youth each with funding from Ford. DOL has 

also funded an evaluation of the QOP demonstration. 

Evaluation Design 

Through annual site visits, annual QOP conferences, and conference calls with QOP staff, we 

assessed how well the CBOs in the QOP demonstration implemented the program model. From 

information provided by QOP staff, we also measured how much QOP cost and how much 

enrollees participated in QOP. To estimate QOP’s impacts on high school performance and 

graduation, postsecondary education or training, and risky behaviors, we conducted two surveys, 

administered achievement tests in reading and mathematics, and collected high school transcripts for 

a group of youth who were enrolled in QOP and a group of statistically identical youth—the control 

group—who were not allowed to participate in QOP. We formed the QOP and control groups at 
v 

the start of the demonstration by randomly assigning each of the nearly 1,100 youth eligible for the 

program to one group or the other. 

 

How Well Was QOP Implemented? 

Every site implemented a version of QOP. However, two sites implemented a version of QOP 

that deviated substantially from the program model, and the other five sites implemented versions 

that deviated moderately from the model. With the exception of the Philadelphia site where the 

program was operated by the CBO that helped to design the QOP model, local CBOs found 

implementing QOP difficult, primarily because QOP was substantially more comprehensive, 

intensive, and complex than their traditional programs. Other implementation findings include the 

following: 

• Most sites implemented the mentoring component as prescribed by the QOP model. 

Case managers developed deep personal relationships with the 40 to 60 percent of 

enrollees who attended some program activities regularly and addressed a wide range of 

barriers facing those youth. QOP’s policy of providing access to services regardless of 

the enrollee’s behaviors (that is, becoming incarcerated, moving to another community, 

dropping out of high school) was well implemented. 

• Few sites implemented the education component as prescribed. In most sites, tutors 

with education backgrounds were not provided on a long-term, consistent basis. Few 

DOL-funded sites implemented computer-assisted instruction as prescribed. While 

several sites assessed the educational achievement of enrollees, few sites translated those 

assessments into meaningful individualized education plans. 

• All sites implemented the developmental component, although with a greater emphasis 

on recreational activities than suggested by the QOP model. 

• No site implemented the community service component as prescribed. Sites scheduled 

only a small fraction of the prescribed number of community service activities. 

• All sites implemented stipends successfully. All sites implemented accrual accounts, 

although DOL-funded sites did not provide regular account statements to enrollees. 

Several sites provided bonuses to enrollees who completed major program activities. 

Most sites provided adequate food and, eventually, transportation services, but few sites 

provided adequate child care services or screening and referral for the physical and mental health 

needs of enrollees. 

How Much Did Enrollees Participate? 

Most enrollees attended relatively few program activities. Enrollees spent an average of 174 

hours per year on QOP activities—23 percent of the annual goal of 750 hours—through the first 

four years of the demonstration. The average fell steadily from 247 hours in the first year to 89 

hours in the fourth year, while the fraction of enrollees spending no time at all on QOP activities 

rose steadily from 1 percent to 36 percent. 
vi 

Participation varied substantially from site to site. The average annual participation in the two 

Ford-funded sites was 294 hours—more than twice the average annual participation of 126 hours in 

the five DOL-funded sites. 

How Much Did QOP Cost? 

The total cost per enrollee over the full five-year demonstration period was $18,000 to $22,000 

for DOL-funded sites, $23,000 for the Yakima site, and $49,000 for the Philadelphia site. 

What Were QOP’s Short-Term Impacts? 

Primary Outcomes 

• QOP increased the likelihood of graduation. QOP increased by a statistically 

significant seven percentage points the likelihood that enrollees graduated from high 

school with a diploma.

• QOP increased the likelihood of engaging in postsecondary education or 

training. The size and statistical significance of the impact, however, depends on how 

this outcome was measured and how the impact was estimated. 

Secondary Outcomes

• QOP did not improve grades or achievement test scores. 

• QOP did not reduce risky behaviors. QOP did not significantly reduce any risky 

behavior, including gang activity, crime, and teen parenting, and according to data from 

one of the evaluation surveys, QOP significantly increased the fraction of enrollees who 

had a drink and the fraction who used an illegal drug in the 30 days before the survey. 

However, some evidence suggests that there were differences between QOP enrollees 

and control-group youth in the accuracy with which they reported risky behaviors. 

Those differences might have contributed substantially to the estimated detrimental 

impacts on drinking and drug use. That QOP might not have increased drinking and 

drug use is also suggested by data from a second evaluation survey. According to those 

data, QOP had beneficial—but not significant—impacts on drinking and drug use. 

Subgroup and Site Impacts

• QOP helped some enrollees more than others. For example, QOP was more 

beneficial for enrollees in the middle of the eligible grade distribution than for enrollees 

at the top or bottom of the distribution. For enrollees in the middle of the distribution, 

QOP significantly increased the likelihood of graduating from high school, the 

likelihood of graduating or earning a GED, and the likelihood of attending or being 

accepted by a college. QOP significantly reduced for these enrollees the likelihood of 

having a child. 
vii 

• QOP’s impacts varied from site to site. And, only one of the seven sites—the 

Cleveland site—had significant beneficial impacts and no significant detrimental 

impacts. The Cleveland site significantly increased the likelihood of graduating from 

high school, significantly increased the likelihood of attending or being accepted by a 

college, and significantly decreased the likelihood of binge drinking. 

• Impacts for the whole demonstration were substantially—but not entirely—

attributable to the impacts of the Philadelphia site alone or the Philadelphia and 

Yakima sites, the Ford-funded sites, together. The Ford-funded sites significantly 

increased mathematics achievement and the likelihood of engaging in postsecondary 

education or training, significantly increased the combined likelihood of engaging in 

postsecondary education or training or having a good job, and significantly reduced the 

likelihood of having a child. However, these sites also significantly increased the 

likelihood of binge drinking, using an illegal drug, and committing a crime, although 

these detrimental impacts might not have been attributable to QOP for the reasons 

noted above. The DOL-funded sites significantly increased the likelihood of graduating 

from high school, one of QOP’s primary objectives. QOP had no other significant 

impacts in the DOL-funded sites.

These impacts are short-term impacts because we estimated them from data collected during 

the fourth and fifth years of the demonstration, that is, before the demonstration was over and when 

many youth were either still attending high school or had only recently graduated. Longer-term 

impacts, which may be a more appropriate basis for policy decisions, might be more or less 

beneficial than the short-term impacts presented in this report. To measure longer-term impacts, 

DOL is having us collect data in fall 2002 and fall 2004. The fall 2002 data collection is roughly 

seven years after the youth in the demonstration sample entered the ninth grade and two years after 

the end of the QOP demonstration.
1 

INTRODUCTION

This report summarizes the short-term impacts of the Quantum Opportunity Program (QOP)1

demonstration. From July 1995 through September 2001, the U.S. Department of Labor (DOL) 

and The Ford Foundation (Ford) operated a demonstration of QOP designed to help at-risk2 highschool-age youth graduate from high school and enroll in postsecondary education or training. 

QOP was an intensive case management and mentoring program that emphasized after-school 

supplemental academic education, developmental activities, and community service. The QOP 

demonstration included several features of Workforce Investment Act (WIA) youth programs, and 

findings from the demonstration might provide some insight about the implementation challenges 

that such WIA programs will encounter and the potential effectiveness of those programs. 

QOP is one of several approaches to assisting at-risk youth evaluated in recent years by DOL 

and the Department of Education (ED), including Job Corps, Job Training Partnership Act (JTPA) 

youth programs, Career Academies, Center for Employment Training (CET), Upward Bound, and 

Talent Search. As employers demanded more advanced technical skills and work-readiness skills in 

entry-level employees, DOL and ED became concerned that some youth are not effectively 

prepared to meet these rising standards. Such youth are at increased risk of unemployment, welfare 

dependency, substance abuse, criminal activity, and teenage childbearing. Finding effective 

approaches to assisting these youth in achieving economic self-sufficiency is critical to avoiding the 

personal losses resulting from such life events and to reducing the costs of providing assistance 

through Unemployment Insurance, WIA, Temporary Assistance for Needy Families (TANF), 

Medicaid, and other public programs. 

The motivation for DOL and Ford to conduct the QOP demonstration arose from QOP’s 

history. In the late 1980s, three organizations—Opportunities Industrialization Centers of America 

(OICA) in Philadelphia; Ford; and Remediation and Training Institute in Alexandria, Virginia—

developed the QOP model. From 1989 through 1993, Ford funded and OICA operated a smallscale pilot of QOP in six sites. The Center for Human Resources at Brandeis University evaluated 

the pilot, obtaining some findings that were encouraging to DOL and Ford. In early 1995, DOL 

and Ford agreed to test QOP on a larger scale via a demonstration with two sites under private 

management and administration and five sites under federal management and administration, 

specifically, under the demonstration title of JTPA. 

The QOP demonstration served a single cohort of youth from the beginning of the ninth grade 

in the fall of 1995 through the fall of 2000. 3 A local community-based organization (CBO) in each 

of six inner-city communities and one rural community implemented and operated a QOP program. 

Each CBO teamed with from one to three high schools and had 50, 80, or 100 youth enrolled in the 

program. By the end of the demonstration, enrollees were in a variety of statuses, including 

 1 The acronym QOP is customarily pronounced kwäp. 

2 At-risk youth are at a greater risk of substance abuse, criminal activity, teenage childbearing, not completing high 

school, or not enrolling in a postsecondary education or training program, compared to the average high-school-age 

youth in the United States. 

3 All events occurred one year later in the Washington, D.C., site. 
2 

attending college or another postsecondary training program, still attending high school, attending a 

general educational development (GED) certification program, working after finishing high school, 

and working or unemployed after dropping out of high school. 

The primary objectives of the demonstration were to increase the likelihood of high school 

completion and to increase enrollment in postsecondary education or training. Its secondary 

objectives were to increase academic achievement while in high school and to reduce risky 

behaviors, such as substance abuse, crime, and teenage childbearing. Under contract to DOL, 

Mathematica Policy Research is evaluating the QOP demonstration, and has assessed the program’s 

implementation and short-term impacts. Detailed findings obtained to date are presented in Schirm 

et al. (2003) and Maxfield et al. (2003). The program impacts reported herein are short-term in that 

they are based on data collected during the fourth and fifth years of the demonstration, that is, while 

sites were still providing services to enrollees and when many youth were either still attending high 

school or had only recently graduated. Future reports will present longer-term program impacts. 

This report summarizes the findings obtained to date and answers the following questions: 

• How well was the QOP program model implemented in the demonstration sites? 

• How much did QOP cost? 

• How much time did enrollees spend on program activities? 

• How did QOP affect enrollees in the short run? 

The report also seeks to determine whether the implementation, cost, and participation findings 

suggest why QOP had the short-term impacts that it did. Before summarizing the evaluation 

findings, we briefly describe the QOP target group and program model in the next section. 

THE QOP TARGET GROUP AND PROGRAM MODEL

The target group in the QOP demonstration was youth entering the ninth grade in fall 1995 

who met the following criteria: 

• Began the ninth grade at a high school selected for the QOP demonstration. Each high 

school had a dropout rate of 40 percent or more. 

• Were not repeating the ninth grade. 

• Were not so physically disabled or learning disabled that participation in the program 

would not be appropriate, as determined by the school. 

• Had a grade point average (GPA) below the 67th percentile among the students meeting 

the first three requirements. (The GPA was calculated from grades received in the 

eighth grade.) 
3 

The QOP model consisted of four primary components: case management and mentoring, 

education, developmental activities, and community service. Secondary aspects of the program 

model included financial incentives—stipends, accrual accounts, enrollee bonuses, and staff 

bonuses—and supportive services—snacks, transportation assistance, and other services as needed. 

Compared to the models for most other youth programs, the QOP model required more 

intensive case management and mentoring in four ways: 

(1) Enrollees were to have greater access to case managers and were to be involved in more 

program activities for longer periods of time. Each case manager was to have a caseload 

of approximately 15 to 25 enrollees. The QOP model set a target of 250 hours per year 

for activities in each of three service components—education, developmental activities, 

and community service—for a total of 750 hours per year until an enrollee graduated 

from high school. Enrollees who took full advantage of QOP received services for five 

years.4

 Most case managers were available during off hours for enrollees to call in 

emergencies.5

(2) Enrollees were to interact with case managers for longer periods of time because 

program eligibility was not contingent on enrollee behavior. Youth continued to be 

enrolled in QOP even if they transferred to another school, dropped out of school, 

became incarcerated, or became inactive in QOP for a long time. In contrast to some 

other youth programs, QOP did not accept or retain only those youth who were 

sufficiently motivated to apply and actively participate. QOP’s approach of enrolling all 

randomly selected eligible youth reflected the program’s philosophy that the leastmotivated youth might benefit the most from receiving help. 

(3) Enrollees were to receive more comprehensive services because the scope of case 

management called for addressing all barriers that enrolled youth faced. Case managers 

either addressed a barrier directly—by arranging transportation to program activities, for 

example—or referred the enrollee to another community resource, such as a substance 

abuse treatment program. 

(4) Enrollees were to receive services throughout school vacations and the summer. 

Enrollees who failed a class during the school year were encouraged to attend summer 

school. Case managers assisted other enrollees who were age 16 or older to find summer 

jobs. Developmental and community service activities continued throughout the 

summer for all enrollees. 

 4 Enrollees who had graduated from high school received some mentoring and assistance in enrolling in 

postsecondary education or training between graduation and the end of the fifth year of the demonstration. 

5 Our assessment of how well these and other features of the QOP model were implemented in the demonstration 

sites is presented below. 
4 

Each of the other three components of the QOP model was geared toward achieving a specific 

program goal. 

• Educational activities were intended to improve academic achievement, increase the 

likelihood of completing high school, and increase the likelihood of going on to college 

or some other postsecondary training program. After an academic assessment, which 

formed the basis of an individualized education plan, educational services were to 

consist of one-on-one tutoring and computer-assisted instruction in specific coursework 

as well as in basic reading and mathematics. Educational services also included visiting 

nearby college campuses and other activities designed to promote awareness of and 

planning for college or other postsecondary training. 

• Developmental activities were designed to reduce risky behaviors. They also 

promoted cultural awareness and provided recreation. 

• Community service activities, such as visiting the residents of a local nursing home or 

volunteering at a local food bank, were designed to help youth develop a sense of 

responsibility for the quality of life of others in their neighborhood. 

The QOP model addressed numerous barriers to success by specifying that supportive services 

were to be provided either directly or indirectly through referrals to other resources in the 

community. QOP case managers referred enrollees to community health and mental health services; 

summer jobs programs; and local agencies that provide housing, food, income support, or child care. 

In addition to supportive services, QOP provided youth with three types of financial incentives 

to attend program activities. The first was a stipend of approximately $1.25 for every hour devoted 

to educational activities, developmental activities that were not purely recreational, and community 

service. A matching amount was deposited in an accrual account and promised to the enrollee when 

he or she earned a high school diploma or GED certificate and enrolled in college, a certified 

apprenticeship program, an accredited vocational/technical training program, or the armed forces. 

Enrollees in some sites also received bonuses for completing major program activities. 

QOP also provided financial incentives to program staff. The two Ford-funded sites 

compensated staff entirely through incentive payments based on the time enrollees spent on 

program activities, while some DOL-funded sites provided bonuses to staff based at least partly on 

enrollee participation. 

HOW DOES QOP COMPARE TO OTHER YOUTH PROGRAMS? 

The goals of QOP were similar to those of many other youth programs or demonstrations 

sponsored by DOL and ED, such as Job Corps, WIA youth programs, Career Academies, the CET 

demonstration, School-to-Work programs, Upward Bound, and Talent Search. However QOP’s 

approach to achieving these goals differed from the approaches taken by these other programs. The 

differences included: 
5 

• QOP was more intensive than most other federal youth programs or demonstrations. 

QOP provided a mentor/case manager for roughly every 15 to 25 enrollees, provided 

services to each enrollee for five years, set a goal of 750 hours of participation per year 

until graduation, and cost about $25,000 per enrollee, on average. 

• QOP was more comprehensive than most other federal youth programs and 

demonstrations. It provided services related directly and indirectly to academic skills; 

college planning and applications (including financial aid); physical and mental health; 

substance abuse; conflict resolution; family planning; cultural and ethnic awareness; 

career awareness and planning; issues related to gang membership and involvement in 

the criminal justice system; coping with dysfunctional, abusive, or unsupportive family 

environments; summer jobs; transportation; nutrition; and paying bills in family 

emergencies. 

• QOP had a more prominent basic education component, as opposed to an occupational 

skills component, than Job Corps and CET do. It shared this orientation with Upward 

Bound and Talent Search. 

• QOP targeted younger youth than Job Corps and CET do. It targeted approximately 

the same age group as do School-to-Work programs, Upward Bound, and Talent 

Search. 

• QOP enrolled less motivated youth than most programs do because it did not limit 

enrollment to those youth who were sufficiently motivated to apply to and remain 

active in the program. 

• QOP explicitly targeted youth with lower grades than Upward Bound and Talent Search 

do. 

• QOP included out-of-school youth, unlike Career Academies, School-to-Work, Upward 

Bound, and Talent Search. QOP included in-school youth, unlike Job Corps and CET. 

• QOP had a substantially greater emphasis on mentoring than do other federal youth 

programs and demonstrations. 

Although QOP differed substantially from several other youth programs, it had many 

similarities with WIA youth programs. In contrast to JTPA youth programs, WIA youth programs 

and QOP provide services that are comprehensive and long term, including: 

• Case management and mentoring by a caring adult 

• Tutoring in basic education and study skills as well as close collaboration with local high 

schools and school districts to improve enrollees’ educational achievement 

• Community service and leadership training 

• Year-round services, including a summer jobs program that is integrated into the 

educational component of the program 
6 

• A broad array of supportive services, including transportation, child care, food, and 

emergency financial assistance 

• Technical assistance to local service providers in recognition of the fact that the model 

is unfamiliar and difficult to implement 

These similarities between QOP and WIA youth programs suggest that the findings from the 

evaluation of the QOP demonstration might reveal some of the implementation challenges that 

WIA youth programs might encounter and indicate whether WIA youth programs are likely to be 

effective. 

HOW WELL WAS QOP IMPLEMENTED? 

Assessing how faithfully demonstration sites implemented the QOP model is important for two 

reasons. First, it indicates, in part, what services were actually provided to enrollees, as opposed to 

what services were supposed to be provided. Second, it indicates how we might expect QOP-like 

WIA youth programs to be implemented by CBOs nationally. 

Every site implemented a version of QOP. However, as we learned from annual site visits, 

annual QOP conferences, and conference calls with QOP staff, two sites implemented a version of 

QOP that deviated substantially from the program model, and the other five sites implemented 

versions that deviated moderately from the model. (See Maxfield et al. (2003) for a detailed 

description of how the program was implemented in each site.) 

There were two main reasons why the QOP programs implemented by the demonstration 

CBOs did not closely adhere to the QOP model. First, with the exception of the Philadelphia site 

where the program was operated by the CBO that helped to design the QOP model, local CBOs 

found implementing QOP to be difficult, primarily because QOP was substantially more 

comprehensive, intensive, and complex than their traditional programs. Second, neither DOL nor 

Ford required the local CBOs to be faithful to the QOP model. 

By some measures, most sites implemented QOP with the prescribed intensity. All sites 

implemented the prescribed ratio of about 15 to 25 enrollees per case manager. Case managers 

developed deep personal relationships with the 40 to 60 percent of enrollees who attended some 

program activities regularly and addressed a wide range of barriers facing those youth. Most case 

managers stayed with the program for several years, and many stayed for the entire five years of the 

demonstration. QOP’s policy of providing access to services regardless of the enrollees’ behaviors 

(such as becoming incarcerated, moving to another community, or dropping out of high school) was 

well implemented. 

By other measures, however, the demonstration CBOs did not implement QOP with the 

prescribed intensity. Only two sites offered the prescribed number of hours of educational, 

developmental, and community service activities. The other sites offered fewer than the prescribed 

number of hours for at least one program component, frequently the community service 

component. Further, the demonstration revealed the practical limitation of QOP’s policy of case 
7 

managers being on duty or on call for large numbers of hours each week. Such a policy is limited by 

the case managers’ personal lives, the physical difficulties of providing services to enrollees who 

moved far away, and the legal limits on case manager overtime under the Fair Labor Standards Act. 

Most sites did not implement the education component effectively. In particular, few sites 

regularly assessed academic performance via achievement tests, no site developed individualized 

education plans based on assessment results, no site implemented a sustained program of coursebased tutoring, and only three sites successfully implemented computer-assisted instruction. 

The developmental component was relatively well implemented. Sites offered many different 

activities. Although developmental activities were intended to focus on life skills that would enable 

the youth to avoid risky behaviors, this component included many recreational activities at most 

sites. Nevertheless, participants found recreational activities to be fun, and case managers found 

them to be useful for fostering program participation. 

The community service component at most sites did not follow the program model. The most 

common reasons for deviations were the enrollees’ lack of interest and the case managers’ belief that 

enrollees needed other QOP services more. Most sites decided to reallocate their resources away 

from community service to mentoring, case management, and educational activities. 

Most sites operated QOP throughout school-year vacations and the summer months. Several 

sites subsidized the fee for summer school for enrollees who needed it. One site developed its own 

summer school during a summer in which the local public school district did not operate summer 

school. Case managers reported that many enrollees needed both summer school, because of failing 

a course during the school year, and a summer job, because of being a member of a low-income 

family. 

Enrollee stipends were well implemented and appeared to be an effective way to attract the 

enrollees to program activities in the first year or two of the demonstration. As enrollees aged, case 

managers found that other incentives, such as recognition, attention, and prizes, could replace the 

stipends. 

JTPA accounting regulations prohibited DOL-funded CBOs from establishing accrual accounts 

for enrollees. Instead, these CBOs kept informal records of accrual account balances and paid those 

balances to qualifying enrollees at the end of the demonstration. According to case managers, the 

resulting absence of periodic account statements limited the effectiveness of accrual accounts in 

increasing program participation. Nonetheless, the accounts enabled many enrollees to save for 

postsecondary education or training. Account balances at the end of the demonstration ranged 

from a few hundred dollars to nearly $10,000, with most being in the range of $1,000 to $3,000. 

Most sites supplied many of the most commonly needed supportive services, including 

afternoon snacks and transportation to program activities. On the other hand, most sites did not 

meet their enrollees’ needs for child care, health and mental health services, substance abuse 

treatment, and family counseling. In fact, QOP proved to be more a prevention program than a 

remediation program. The most well developed aspects of QOP were designed to prevent youth 

from engaging in risky behaviors. QOP was less well developed for providing services to youth 

facing the consequences of the risky behaviors in which they had already engaged. 
8 

QOP’s philosophy of addressing every barrier a participating youth faces means that a wide 

range of supportive services expenditures should be permitted. However, the title of JTPA through 

which DOL would have funded an operational version of QOP was in several ways inconsistent 

with such a comprehensive and long-term program. For example, the performance measures used 

for JTPA programs generally focused on short-term employment goals rather than on postsecondary 

education or training goals following a four- to five-year service delivery period. In fact, the cost 

accounting regulations for JTPA did not permit CBOs to be reimbursed for contributions to QOP 

accrual accounts. Such inconsistencies between the QOP model and the federal statute pertaining to 

DOL youth programs have been substantially addressed in WIA programs, which replaced JTPA 

programs in 1999. 

HOW MUCH DID QOP COST? 

The total QOP expenditure per enrollee averaged $25,000 for the full five years of the 

demonstration. The five-year expenditure per enrollee for the DOL-funded sites ranged from 

$18,000 to $22,000. For the two Ford-funded sites, the expenditure per enrollee was $23,000 in 

Yakima and $49,000 in Philadelphia. Thus, Philadelphia had a much higher expenditure per enrollee 

than any other site. 

Annual expenditures at most sites varied over the five years of the demonstration. Spending 

typically increased each year during the first four years and decreased during the fifth year. QOP 

coordinators reported that they developed a better understanding of what they could do with the 

money and where they needed to spend it after the first year or two of the demonstration. 

These cost figures cover program operations and management, but exclude the cost of technical 

assistance provided by OICA. Because of the anticipated need for technical assistance and OICA’s 

experience in helping to design the QOP model, Ford awarded a grant to OICA to provide technical 

assistance for the QOP demonstration. Technical assistance included helping sites set up 

management information software, funding annual week-long training conferences for all QOP 

staff, and answering questions as needed. OICA provided technical assistance for the demonstration 

at a cost of $1,125,000, or $38,000 per year per site (not counting the Philadelphia site itself). In 

addition to providing technical assistance, OICA operated the Philadelphia site throughout the 

demonstration. 

HOW MUCH DID ENROLLEES PARTICIPATE IN QOP? 

Most enrollees did not attend most program activities. According to the QOP participation 

data for the first four years of the demonstration, enrollees spent an average of 174 hours per year 

on QOP activities—23 percent of the annual goal of 750 hours. Enrollees spent an average of 72 

hours per year on education (29 percent of the goal), 76 hours on developmental activities (30 

percent of the goal), and 26 hours on community service (11 percent of the goal). The average time 

spent on QOP activities fell steadily from 247 hours in the first year of the demonstration to 89 
9 

hours in the fourth year. The percentage of enrollees spending no time at all on QOP activities 

increased steadily from 1 percent in the first year to 36 percent in the fourth year. This is 

disappointing for a program based on the belief that youth programs must be intensive to be 

effective.6

 The roughly 12 percent of enrollees who spent 100 or fewer hours on QOP activities 

during the entire demonstration reported being uninterested in those activities or having other afterschool activities, such as playing a sport, working, or caring for other family members. 

Participation varied substantially from site to site. Participation ranged from highs of 345 hours 

per year per enrollee in the Yakima site and 244 hours in the Philadelphia site to a low of 68 hours in 

the Fort Worth site. The average annual participation for the two Ford-funded sites was 294 hours, 

and the average annual participation in DOL-funded sites was 126 hours. Thus, the number of 

hours spent on program activities by enrollees at Ford-funded sites was 2.3 times that of enrollees at 

DOL-funded sites. 

WHAT WERE THE SHORT-TERM IMPACTS OF QOP? 

To estimate the short-term impacts of QOP, we translated each program goal, such as high 

school graduation, into a quantifiable outcome, such as whether a youth graduated from high school. 

We measured each outcome for a group of youth enrolled in QOP and a group of statistically 

identical youth, called the control group. We formed the QOP group and the control group at the 

start of the demonstration by randomly assigning each youth eligible for the program to one group 

or the other. All members of the QOP group were enrolled in QOP. Members of the control group 

were not allowed to participate in QOP and, thus, show what would have happened to the enrollees 

had they not been enrolled. 

We interviewed enrollees and control-group members in-person in the spring of the fourth 

academic year of the demonstration. The survey collected data on risky behaviors and factors that 

assist a youth in resisting negative influences in his or her social environment. At the same time, we 

administered achievement tests in reading and mathematics. Seven to ten months later, we 

conducted a telephone survey covering high school graduation, postsecondary activities, risky 

behaviors, and (for the enrollee group) attitudes toward QOP. Shortly thereafter, we requested 

transcripts from the high schools that sample members had attended since the beginning of the 

demonstration. 

We measured the impact of QOP on an outcome by subtracting the mean outcome for the 

control group from the mean outcome for the QOP group. Because the available data were 

collected before the end of the demonstration and when many youth were either still attending high 

school or had only recently graduated, the impacts estimated from those data and presented in this 

report should be interpreted as short-term impacts for many of the outcomes considered. To 

measure longer-term impacts, DOL is having us collect data in fall 2002 and fall 2004. The fall 2002 

data collection is roughly seven years after the youth in the demonstration sample entered the ninth 

 6 We will estimate how program impacts vary by level of participation once we have collected longer-term followup data. 
10 

grade and two years after the end of the demonstration. After all of the youth in the demonstration 

sample have left high school—with or without graduating—we might find that QOP’s impacts on 

high school completion and postsecondary activities, for example, are different from the impacts 

presented in this report. The longer-term impacts, which may be a more appropriate basis for policy 

decisions, might be higher or lower than the short-term impacts. The importance of estimating 

longer-term impacts is illustrated by the National Job Corps Study, which obtained longer-term 

impacts that were substantially more favorable than the short-term impacts (Schochet et al. 2000, 

2001). 

Primary Outcomes: High School Completion and Postsecondary Activities 

QOP significantly increased by seven percentage points the likelihood that enrollees 

graduated from high school with a diploma (Table 1).7 Because about 16 percent of sample 

members were still attending high school when we collected the data, we examined alternative ways 

to measure high school completion. QOP had significant positive impacts on both the narrowest 

and the broadest measures, but insignificant impacts on the two intermediate measures. This 

pattern indicates that QOP improved the likelihood that enrollees earned a diploma and suggests 

that QOP increased the likelihood that enrollees who dropped out of high school attended a GED 

class. It also suggests that QOP did not improve either the likelihood that enrollees earned a GED 

or the likelihood that enrollees who did not graduate on time stayed in high school for a fifth year. 

This pattern of short-term impacts also indicates that the final size of QOP’s impact on high school 

completion will depend on whether the sample members still attending high school when we 

conducted our last survey eventually earn diplomas or GED certificates and whether those attending 

GED classes eventually earn GED certificates. This will be measured in the next survey. 

TABLE 1 

SHORT-TERM IMPACTS ON HIGH SCHOOL COMPLETION 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage points) 

Earned diploma 46 40 7*

Earned diploma or GED certificate 54 49 5 

Earned diploma or GED certificate or attending high 

school 68 66 3 

Earned diploma or GED certificate or attending high 

school or a GED class 79 72 7**

SOURCE: Telephone survey and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an 

impact might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 

489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 

 7 Throughout this report, we use the statistical definition of “significant.” Under that definition, an estimated 

impact is significant if according to the available data, it is very likely that the impact is different from zero. That an 

impact is significant does not imply that it is, for example, big or substantively important. 
11 

QOP increased the likelihood of engaging in postsecondary education or training, 

although the size and significance of the impact depends on how this outcome was 

measured and how the impact was estimated (Table 2). QOP significantly increased by six 

percentage points the likelihood of engaging in postsecondary education or training when education 

or training was defined to include college attendance, vocational or technical school attendance, 

apprenticeship enrollment, and armed forces enlistment. About half of this impact was attributable 

to increased college attendance. The impact became smaller and insignificant when the measure of 

postsecondary education or training was either narrowed to include only college attendance or 

broadened to include employment. The impact also became smaller and insignificant when we used 

regression methods to adjust for random differences between the baseline characteristics of the 

QOP group and the control group (Schirm et al. 2003). 

TABLE 2 

SHORT-TERM IMPACTS ON POSTSECONDARY ACTIVITIES 

 Alternative Definitions of College Training 

Attending Attending or Accepted 

Outcomea

QOP-Group 

Mean 

(percentage) 

ControlGroup 

Mean 

(percentage) 

Impact 

(percentage 

points) 

QOP-Group 

Mean 

(percentage) 

ControlGroup 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Four-year college 11 8 3 15 12 3 

Two- or four-year college 21 18 3 31 25 6*

College, vocational/technical school, 

apprenticeship, armed forces 32 26 6* 42 34 8**

Postsecondary training or good job 48 43 5 56 47 9**

Postsecondary training or any job 66 61 5 70 63 7**

Postsecondary training or high school 47 43 4 57 51 6*

Postsecondary training or high school 

or GED class 54 48 6* 64 55 9**

Postsecondary training or high school 

or GED class or good job 68 65 3 76 69 7**

Postsecondary training or high school 

or GED class or any job 84 80 3 88 82 6**

SOURCE: Telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an 

impact might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 

489 controls. 

a In the last seven rows of the table, “college” means either a two-year or a four-year college. “Postsecondary training” means college, 

vocational/technical school, apprenticeship, or armed forces. A “good” job offers employer-sponsored health insurance. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
12 

Recognizing that some youth might have needed more than the six or fewer months between 

high school completion and our telephone survey to begin postsecondary education or training, we 

defined outcomes that count acceptance by a college as well as attending a college as forms of 

postsecondary education or training. When we included acceptance into college—in addition to 

current attendance at college—in the definition of postsecondary education or training, QOP 

significantly increased the likelihood of engaging in postsecondary education or training by six to 

nine percentage points for all but one measure of postsecondary activity. The higher impact 

estimates obtained when we count college acceptance are attributable to the higher acceptance rates 

for QOP enrollees than for control-group youth by two-year colleges. The longer-term impacts of 

QOP on postsecondary education or training will depend on whether the youth who had been 

accepted by colleges subsequently enrolled and whether the youth who were still in high school later 

completed high school and engaged in further education or training. We will measure such behavior 

in the next two follow-up surveys. 

That QOP had greater short-term impacts on high school graduation and enrollment in 

postsecondary education or training than did other JTPA youth programs (Orr et al. 1996) is 

consistent with QOP being more comprehensive and intensive than those other programs. On 

average, QOP spent more than 10 times as much per enrollee as the typical JTPA youth program, 

and addressed a much broader array of barriers in the lives of enrollees for a much longer period of 

time. 

Even though the impacts on high school graduation and enrollment in postsecondary education 

or training were statistically significant, their magnitudes were modest. This finding is consistent 

with the amount of enrollee participation. Although QOP enrollees spent substantially more time in 

program activities than did participants in the typical JTPA youth program, the number of hours 

spent on program activities by the average QOP enrollee was substantially less than the program 

goal during the first year of the program. Then, hours of participation fell steadily for the average 

enrollee, while the proportion of enrollees with little connection to the program grew steadily, as 

discussed above and in Schirm et al. (2003).8

Secondary Outcomes: High School Performance 

QOP did not significantly improve enrollee performance while in high school (Table 3). 

It did not significantly improve achievement test scores, grades, or credits earned, and it did not 

significantly reduce disciplinary actions. Although QOP might not have raised grades if QOP 

enrollees were taking more challenging courses than the youth in the control group, we would have 

expected QOP to increase standardized test scores if it had an impact on achievement. 

 8 Considering the impact findings in the light of the implementation, cost, and participation findings enables us to 

speculate about whether the average size of an impact and its variation across sites, for example, are broadly consistent 

with the average level of participation and variations in participation. However, the implementation, cost, and 

participation findings cannot generally explain—in a causal sense—the impact findings. The reason is that the QOP 

demonstration was not designed to assess the effects of variations in implementation, costs, and participation. Instead, 

the only factor that was experimentally controlled was whether a student was enrolled in QOP. Even quasi-experimental 

methods that use statistical modeling cannot help very much in the search for explanations because the demonstration 

had only a small number of sites, and they differed in so many ways that we cannot disentangle the effects of their 

differences. 
13 

TABLE 3 

SHORT-TERM IMPACTS ON HIGH SCHOOL PERFORMANCE 

Outcomea

QOP-Group 

Mean 

Control-Group 

Mean Impact 

Mathematics achievement test score (percentile) 40.9 40.5 0.4 

Reading achievement test score (percentile) 43.2 42.7 0.5 

Cumulative GPA (four-point scale) 2.13 2.19 -0.06 

Mathematics/science GPA (four-point scale) 1.81 1.85 -0.03 

Total credits (Carnegie units) 16.2 15.8 0.5 

Core academic credits (Carnegie units) 10.7 10.2 0.6 

Mathematics/science/English credits (Carnegie units) 7.2 6.9 0.3 

Ever suspended 44% 45% -1 

Ever expelled 8% 7% 0 

Suspended or expelled in past 12 months 34% 38% -4 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an 

impact might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 

489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. Credits are expressed in 

Carnegie units that standardize for in-class time. One Carnegie unit corresponds to a class that meets for 45 to 60 minutes every day of the week for 

an entire academic year. Core academic credits are the credits earned in mathematics, science, English, social studies, and foreign language classes. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 

That QOP did not improve any of the secondary indicators of academic performance while 

modestly improving high school graduation and enrollment in postsecondary education or training is 

consistent with the generally successful implementation of intensive mentoring and case 

management and the less successful implementation of the education component. A main objective 

of QOP’s mentoring and case management was to keep enrollees focused on, and overcoming 

barriers to, the twin goals of graduation and enrollment in postsecondary education or training. 

Case managers attempted to prevent each enrollee from giving up on school, advocated on behalf of 

the enrollee with the school, and tried to protect the enrollee from outside distractions and 

responsibilities that would divert the enrollee’s attention from school. In contrast, computerassisted instruction was designed primarily to improve the enrollee’s basic skills in reading and 

mathematics, and course-based tutoring was designed to improve the enrollee’s grades. Tutoring 

was poorly implemented by all sites, and computer-assisted instruction was poorly implemented by 

most sites. 
14 

Secondary Outcomes: Risky Behaviors 

QOP did not significantly reduce any risky behaviors, and according to data from one 

of the evaluation surveys, it significantly increased some risky behaviors (Table 4). QOP 

significantly increased by seven percentage points the fraction of enrollees who had a drink and the 

fraction of enrollees who used an illegal drug in the 30 days before the in-person survey. QOP did 

not have a significant impact on other risky behaviors. The impacts on gang-related activity and 

crime were zero or detrimental but insignificant. The impacts on the likelihood of being arrested or 

charged and the likelihood of having a child were beneficial but were also insignificant. 

What might explain these effects? By paying stipends and bonuses for participation, QOP 

might have provided some enrollees with the money to buy alcohol and drugs. By bringing 

enrollees together through program activities, QOP might have introduced some negative peer 

effects and facilitated the spread of drinking and drug use. Thus, although the design of the QOP 

demonstration does not allow either of these explanations to be proved true (or false), there were 

mechanisms through which QOP might have had significant detrimental effects on some risky 

behaviors. 

It is also possible—and maybe likely—that the detrimental effects were not caused by QOP. 

The available data suggest that there might have been differences between QOP enrollees and 

control-group youth in the accuracy with which they reported risky behaviors, and those differences 

might have contributed substantially to the estimated detrimental impacts on risky behaviors (Schirm 

et al. 2003). Specifically, some of the control-group means pertaining to drinking and drug use were 

unusually and, perhaps, implausibly low. In the Philadelphia site, only 3 percent and 12 percent of 

control-group youth reported having a drink or taking an illegal drug, respectively, in the 30 days 

before the in-person survey. In the other six sites combined, the rates of drinking and drug use 

among control-group youth were substantially higher—38 percent and 30 percent, respectively. 

QOP’s impact on drinking was a significant 30-percentage-point increase in the Philadelphia site and 

an insignificant 3-percentage-point increase in the other six sites combined. For drug use, the 

respective impacts were insignificant 13- and 5-percentage-point increases. As suggested above, 

these results might have been attributable to differences in the accuracy with which QOP enrollees 

and control-group youth reported risky behaviors. An alternative explanation is that the results were 

due to purely random baseline differences between the two groups for which we could not 

statistically adjust because the differences were not associated with any of the very limited number of 

baseline characteristics that could be measured.9

 

Other data also suggest that QOP might not have had significant detrimental effects on risky 

behaviors. According to the data collected in the second evaluation survey (the telephone survey), 

QOP had beneficial—but not significant—impacts on drinking and drug use (Schirm et al. 2003). 

Although a self-administered in-person survey like our in-person survey would tend to obtain more 

accurate responses to sensitive questions about risky behaviors than would a telephone survey, as 

discussed in Schirm et al. (2003), the impact estimates based on data from the telephone survey 

 9 Schirm et al. (2003) present regression-adjusted impact estimates for a set of key outcomes. The adjusted 

estimate pertaining to drug use was the same as the unadjusted estimate. 
15 

suggest that QOP might not have significantly increased risky behaviors, while confirming that QOP 

did not significantly reduce such behaviors. 

Secondary Outcomes: Resiliency Factors 

QOP’s efforts to influence risky behaviors may be viewed from the perspective of the juvenile 

justice literature as attempts to mitigate the risk factors in enrollees’ social environments and 

strengthen the resiliency factors (U.S. Department of Justice 1995). The concepts of risk and 

TABLE 4 

SHORT-TERM IMPACTS ON RISKY BEHAVIORS 

Outcome 

QOP-Group 

Mean 

(percentage) 

Control-Group 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Substance abusea 

Drinking in the past 30 days 40 33 7**

Frequent drinking in the past 30 days 11 11 0 

Binge drinking in the past 30 days 24 20 4 

Frequent binge drinking in the past 30 days 7 5 2 

Drunk or high at school in the past 12 months 20 20 0 

Used any illegal drug in the past 30 days 34 28 7**

Gang activity, crime, arrests 

Involved in gang fight in the past 12 months 16 14 2 

Ever a gang member 13 13 0 

Currently a gang member 6 4 2 

Committed any crime in the past 12 months 31 28 3 

Ever arrested or charged 25 29 -5 

Sexual activity 

Ever had sex 78 83 -5 

Did not use condom last time 29 28 0 

Taught about HIV/AIDS 93 94 0 

Ever pregnant or get anyone pregnant 33 33 0 

Have had a child 23 26 -3 

SOURCE: In-person survey (for all but the last two outcomes) and telephone survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an 

impact might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 

489 controls. 

a “Binge” drinking means five or more drinks in a row. Drinking or binge drinking was classified as “frequent” if it occurred on at least eight out of 

the past 30 days. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
16 

resiliency factors are based on the belief that although youth are inherently inclined toward socially 

useful and productive behaviors, they can be led to crime or other risky behaviors by individuals in 

their homes, peer groups, or neighborhoods. Such individuals might include a parent who is a 

substance abuser, a friend who invites the youth to participate in a criminal endeavor, and a 

neighborhood drug dealer. These individuals are risk factors. However, youth are not defenseless in 

their encounters with negative influences. Some youth are protected from negative influences by 

their relatives, friends, and adult mentors. Such individuals are resiliency factors. 

From this perspective, the QOP program and the QOP case manager may be viewed as 

resiliency factors, and we found that QOP significantly increased one resiliency factor (Table 5). 

QOP significantly increased by 31 percentage points the fraction of enrollees reporting participation 

in “special programs other than your normal high school classes ...[that try] to help students stay in 

school, make good grades, stay away from illegal drugs, prepare for work or college, and make good 

decisions in life.” Despite this significant positive impact, four other results suggest that QOP was 

not an effective resiliency factor. First, more than half (53 percent) of QOP enrollees failed to 

report participating in such a program. This finding might reflect the fact that participation in QOP 

TABLE 5 

SHORT-TERM IMPACTS ON RESILIENCY FACTORS 

Outcome 

QOPGroup 

Mean 

(percentage) 

ControlGroup 

Mean 

(percentage) 

Impact 

(percentage 

points) 

Participated in a program that helps students stay in school, make 

good grades, stay away from illegal drugs, prepare for work or 

college, and make good decisions in life 47 16 31***

There was an influential adult who positively influenced the 

youth’s life in some significant way 72 69 3 

Thought that all of the following activities are always wrong: 22 22 0 

Using illegal drugs or alcohol frequently 

Committing crimes 

Having a baby while a teenager 

Dropping out of school 

 

Disagreed with all of the following statements: 61 57 3 

Bad things happen to people like me 

I'm afraid my life will be unhappy 

I do not like the way I look 

I'll probably die before I'm 30 

SOURCE: In-person survey. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean prior to rounding those means; thus, an 

impact might not equal the difference between the rounded means that are displayed. The evaluation sample had 580 QOP enrollees and 

489 controls. 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
17 

activities fell substantially short of the program’s goals, especially by the fourth year of the 

demonstration when we asked the youth in the evaluation sample about their participation in special 

programs.10 Second, QOP did not significantly increase the likelihood that an enrollee perceived 

himself or herself as being positively influenced by a caring adult despite the program’s emphasis on 

mentoring. Third, QOP did not significantly improve resiliency factors such as having an optimistic 

outlook on the future or believing that risky behaviors are wrong. Fourth, QOP did not significantly 

reduce any risky behavior. 

Short-Term Impacts on Subgroups11

QOP had several significant impacts on enrollees in the middle third of the baseline 

(eighth grade) grade distribution for eligible youth, and all of those impacts were beneficial

(Table 6).12 They included a 14-percentage-point increase in the likelihood of receiving a diploma, a 

13-percentage-point increase in the likelihood of college attendance or acceptance, and an 8-

percentage-point decrease in the likelihood of having a child. For enrollees in the bottom third of 

the baseline grade distribution, QOP had significant beneficial impacts on the likelihood of 

attending postsecondary education or training and the likelihood of being arrested or charged with a 

crime, but it also had a significant detrimental impact on drug use. For enrollees in the top third of 

the baseline grade distribution, QOP had only one significant impact—a detrimental impact on 

binge drinking.13

This pattern of impacts across the three subgroups is consistent with reports from case 

managers that their caseloads included both youth who were doing well enough in school that they 

had little need for QOP services and youth who faced so many barriers to academic success that 

case managers referred them to Job Corps. Case managers felt that some members of the latter 

group were sufficiently alienated from school and involved in risky behaviors that they would be 

more receptive to program services if the youth were removed from their neighborhoods to the 

residential setting of Job Corps. 

While some of QOP’s impacts on females and some of its impacts on males were significantly 

different from zero, QOP’s impact on females was significantly different from its impact on males 

for only one key outcome. QOP significantly increased by nine percentage points the likelihood of a 

female enrollee engaging in postsecondary education or training, attending high school or a GED 

 10 Although we asked about participation “since beginning the ninth grade,” some youth may have reported about 

their current or recent participation status in responding to our in-person survey. According to a different source of 

information, the administrative data on QOP participation, 36 percent of QOP enrollees were spending no time at all on 

QOP activities in the fourth year of the demonstration, as noted above in our discussion of QOP participation. 

11 The QOP demonstration was designed primarily to estimate demonstration-wide impacts. Thus, the sample for 

a subgroup or individual site is small, which makes estimated impacts relatively imprecise. 

12 Tables 6, 7, and 8 present two types of significance tests. One test is whether the impact is significantly different 

from zero, indicated by asterisks. The other test is whether the impact for one subgroup, site, or group of sites is 

different from the impact for all of the other subgroups or sites combined, indicated by a dagger (†). The conclusions 

presented in the text are based on whether the impacts are significantly different from zero, unless otherwise noted. 

13 We defined these subgroups by dividing each QOP school’s evaluation sample into thirds. A school’s evaluation 

sample excludes youth who were above the 67th percentile in the entering ninth grade class and, therefore, ineligible for 

QOP. Thus, for example, the youth in the middle third of the evaluation sample were between roughly the 22nd and 44th

percentiles in the grade distribution for all entering ninth graders. 
18 

class, or working. This impact was significantly different from the (insignificant) two-percentagepoint decrease for males. The one other impact on females that was significantly different from 

zero was a nine-percentage-point increase in the likelihood of graduating from high school. Both of 

QOP’s significant impacts on male enrollees were detrimental—QOP significantly decreased high 

school GPAs and increased binge drinking. 

TABLE 6 

SHORT-TERM IMPACTS BY RANK IN THE BASELINE GRADE DISTRIBUTION 

(Percentage points except where noted) 

 Impacts 

Outcomea Bottom Third Middle Third Top Third 

Total 

Sample 

Mathematics test score (percentile) 0.06 0.28 0.04 0.38 

Reading test score (percentile) 0.27 -0.17 0.91 0.50 

GPA (four-point scale) -0.13 -0.06 0.03 -0.06 

Earned high school diploma 3 14** 4 7* 

Earned diploma or GED certificate 1 11* 3 5 

Attending college -2 7 4 3 

Attending postsecondary training 9* 3 4 6*

Postsecondary training or good job 7 9 -4† 5 

Postsecondary training or high school or GED 

class or any job 8 1 0 3 

Attending or accepted into college -3† 13** 9 6*

Binge drinking in past 30 days 8 -4† 8* 4 

Used any illegal drug in past 30 days 14†, ** -2† 7 7**

Committed any crime in past 12 months 2 4 8 3 

Ever arrested or charged -11* 0 1 -5 

Have one or more own children -4 -8* 3 -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP 

enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row. 

† Significantly different from the impact on all other youth at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test
19 

QOP did not consistently benefit younger enrollees more than older enrollees. (The older 

enrollees—about a third of all enrollees—were over age 14 then they entered the ninth grade, 

whereas the younger enrollees were age 14 or younger.) The impact on younger enrollees was 

significantly different from the impact on older enrollees for just one outcome. QOP decreased by 

nine percentage points the fraction of younger enrollees who had a child. This impact was 

significantly different from both zero and the (insignificant) six-percentage-point increase in the 

fraction of older enrollees who had a child. In addition to the beneficial impact on the likelihood 

that a younger enrollee had a child, QOP significantly increased by 12 percentage points the 

likelihood of graduating from high school, by 7 percentage points the likelihood of engaging in 

postsecondary education or training, and by 7 percentage points the likelihood of college attendance 

or acceptance for younger enrollees. For older enrollees, QOP had two significant beneficial 

impacts—a 10-percentage-point increase in the likelihood of engaging in postsecondary education or 

training, attending high school or a GED class, or working and an 11-percentage-point decrease in 

the likelihood of ever being arrested for or charged with a crime. 

Short-Term Impacts by Site 

Impacts varied substantially from site to site (Table 7). One site—Cleveland—had 

significant beneficial impacts and no significant detrimental impacts. Three sites—Washington, 

D.C.; Houston; and Memphis—had significant detrimental impacts and no significant beneficial 

impacts. The other three sites—Fort Worth, Philadelphia, and Yakima—had no significant impacts 

or both significant beneficial and significant detrimental impacts. 

• The Cleveland site had significant beneficial impacts on earning a high school diploma, 

attending or being accepted into college, and binge drinking. 

• The Washington, D.C., site had a significant detrimental impact on mathematics 

achievement test scores. 

• The Houston site had a significant detrimental impact on GPAs. 

• The Memphis site had a significant detrimental impact on binge drinking. 

• The Fort Worth site had no significant impacts. 

• The Philadelphia site had a significant beneficial impact on attending postsecondary 

education or training. It also had a significant beneficial impact on attending 

postsecondary education or training, attending high school or a GED class, or working. 

It had significant detrimental impacts on binge drinking and crime. 

• The Yakima site had a significant beneficial impact on mathematics achievement test 

scores. Its detrimental impact on attending postsecondary education or training, 

attending high school or a GED class, or working was significantly different from the 

impact for the other six sites combined. 
20 

TABLE 7 

SHORT-TERM IMPACTS BY SITE 

(Percentage points except where noted) 

Impacts 

Outcomea Fort Worth Cleveland D.C. Houston Memphis PhiladelMathematics test score (percentile) 0.16 0.39 -2.14†, ** 0.58 -0.21 0.69 Reading test score (percentile) -0.46 -0.43 1.85 0.47 -0.39 1.25 GPA (four-point scale) 0.04 -0.10 -0.15 -0.25†, ** 0.07 -0.13 

Earned high school diploma 2 13* 12 3 1 10 

Earned diploma or GED certificate 6 7 6 -2 4 10 Attending college 3 8 -9† 3 2 13 

Attending postsecondary training 6 2 -2 1 6 19

Postsecondary training or good job -1 -3 4 -3 2 17 Postsecondary training or high school or 

GED class or any job 9 -4 12 4 -6† 22

Attending or accepted into college 2 13* -2 2 4 17 

Binge drinking in past 30 days -13† -16†, * -3 9 18†, *** 17

Used any illegal drug in past 30 days 8 -6 -1 10 8 13 

Committed a crime in past 12 months -5 -7 6 2 -7 18

Ever arrested or charged -4 1 -12 -1 -11 3 Have one or more own children -3 3 -4 4 6 -15 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a two-year or a four-year 

means college, vocational/technical school, apprenticeship, or armed forces. A “good” job offers employer-sponsored health insurance. “Binge” drinking means five or more † Significantly different from the impact for all other sites at the 90% confidence level, two-tailed test * Estimate significantly different from zero at the 90% confidence level, two-tailed test ** Estimate significantly different from zero at the 95% confidence level, two-tailed test *** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
21 

As discussed above, the significant detrimental impacts on risky behaviors might have been due 

to differences between QOP enrollees and control-group youth in the accuracy with which they 

reported risky behaviors or to purely random differences in baseline characteristics for which we 

could not adjust. In addition to the relatively low rates of drinking and drug use among controlgroup youth in the Philadelphia site that were noted above, 14 percent of those youth reported 

committing a crime in the year before the in-person survey, while the fraction was much higher—31 

percent—for control-group youth in the other six sites combined. Similarly, the rates of binge 

drinking among control-group youth were 2 percent and 6 percent in Philadelphia and Memphis, 

respectively, whereas the rate for the other five sites combined was 27 percent. 

These patterns and the Philadelphia and Memphis sites’ significant detrimental impacts on some 

risky behaviors were observed in data from the in-person survey. According to the telephone survey 

data, neither Philadelphia nor Memphis had significant detrimental impacts on risky behaviors 

(Schirm et al. 2003). However, neither site significantly reduced such behaviors.14 

The impacts for the whole QOP demonstration were substantially—but not entirely—

attributable to the impacts of the Philadelphia site alone or the Philadelphia and Yakima 

sites—the Ford-funded sites—together (Table 8). Across a set of 15 key outcomes, the five 

DOL-funded sites had one significant impact—they increased by seven percentage points the 

likelihood that a QOP enrollee graduated from high school. This impact was not significantly 

different from the impact for the two Ford-funded sites, which was also an increase of seven 

percentage points (but not significant). In contrast, the Ford-funded sites had seven significant 

impacts, and five of the seven were significantly different from the impacts for the DOL-funded 

sites. The Ford-funded sites had four significant beneficial impacts: a 2-percentile-point increase in 

the mathematics achievement test score, a 14-percentage-point increase in the likelihood of engaging 

in postsecondary education or training, a 17-percentage-point increase in the likelihood of engaging 

in postsecondary education or training or working at a good job, and a 14-percentage-point decrease 

in the likelihood of having a child. The Ford-funded sites also had three significant detrimental 

impacts: 17-, 14-, and 16-percentage-point increases in the likelihood of engaging in binge drinking, 

using an illegal drug, and committing a crime, respectively. As discussed above, however, these 

detrimental impacts on risky behaviors might not have been attributable to QOP. 

That the Philadelphia site had relatively large impacts is consistent with several of the site’s 

characteristics. One such characteristic is that the QOP staff in Philadelphia understood the 

complex and nontraditional QOP model, especially the education component, from the beginning 

of the demonstration, and they were able to implement the QOP model more effectively and quickly 

than staff in other sites. Many QOP staff in other sites regarded QOP as substantially different 

from other programs operated by their CBOs. They reported that they needed at least one year, two 

training conferences, and ongoing technical assistance to understand the model and how to 

implement it. The technical assistance was provided by the staff in the Philadelphia site, including 

one of the original designers of the QOP model. 

 14 Drug use increased significantly by 15 percentage points in the Houston site according to the data from the 

telephone survey. The impact estimated from the in-person survey was 10 percentage points but not statistically 

significant. 
22 

The prominent role of the Philadelphia CBO in designing and, later, marketing the QOP model 

might have given it a substantial stake in the success of the demonstration and led its management 

to invest greater resources than were invested in other sites. As documented above, the Philadelphia 

site spent more than twice as much per enrollee as did any other site. Most of the additional 

spending was for staff compensation. A case manager in the Philadelphia site received about twice 

the compensation of a case manager in any other site. The higher level of compensation in the 

Philadelphia site might have produced more effective case management by, for example, 

encouraging case managers to devote extra time to QOP activities. 

TABLE 8 

SHORT-TERM IMPACTS BY FUNDING SOURCE 

(Percentage points except where noted) 

Impacts 

Outcomea Ford-Funded Sites DOL-Funded Sites Total Sample 

Mathematics test score (percentile) 1.96†, * -0.24† 0.38 

Reading test score (percentile) 1.27 0.21 0.50 

GPA (four-point scale) -0.02 -0.08 -0.06 

High school diploma 7 7* 7*

Diploma or GED certificate 6 4 5 

Attending college 8 1 3 

Attending postsecondary training 14* 3 6*

Postsecondary training or good job 17†, ** 0† 5 

Postsecondary training or high school or GED class or 

any job 4 3 3 

Attending or accepted into college 11 3 6*

Binge drinking in past 30 days 17†, *** -1† 4 

Used any drug in past 30 days 14** 4 7**

Committed any crime in past 12 months 16†, ** -2† 3 

Ever arrested or charged -3 -5 -5 

Have one or more own children -14†, ** 1† -3 

SOURCE: In-person survey, achievement tests, telephone survey, and transcripts. 

NOTE: Each impact was derived by subtracting the control-group mean from the QOP-group mean. The evaluation sample had 580 QOP 

enrollees and 489 controls. 

a Achievement test scores are expressed as percentiles in the distribution of scores for tenth graders in the United States. “College” means either a 

two-year or a four-year college. “Postsecondary training” means college, vocational/technical school, apprenticeship, or armed forces. A “good” job 

offers employer-sponsored health insurance. “Binge” drinking means five or more drinks in a row. 

† Significantly different from the impact for sites with the other funding source at the 90% confidence level, two-tailed test 

* Estimate significantly different from zero at the 90% confidence level, two-tailed test 

** Estimate significantly different from zero at the 95% confidence level, two-tailed test 

*** Estimate significantly different from zero at the 99% confidence level, two-tailed test 
23 

A distinguishing characteristic of not only the Philadelphia site but also the other Ford-funded 

site—Yakima—was that enrollee participation was much higher than at other sites, perhaps because 

case manager compensation was based entirely on enrollee participation. Compared with the 

average enrollee in a DOL-funded site, the average enrollee in Philadelphia and Yakima spent 1.9 

and 2.7 times as many hours on QOP activities during the demonstration. Although the 

Philadelphia site had relatively large impacts, the Yakima site had several relatively large—albeit 

mostly insignificant—impacts and some relatively small impacts. Whether, across all sites, QOP’s 

impacts became more beneficial as an enrollee’s hours of participation rose will be assessed when 

longer-run data become available. 

SUMMARY

Several broad findings emerge from our analyses of short-term impacts, program 

implementation, program costs, and enrollee participation: 

• QOP achieved some short-term success in meeting its two primary goals of raising rates 

of high school completion and enrollment in postsecondary education or training. It 

had statistically significant beneficial impacts of modest size on at least some measures 

of both outcomes. 

• QOP was not successful in meeting its secondary goals of improving grades and 

achievement test scores and reducing risky behaviors. 

• QOP was more beneficial in the short-run for enrollees in the middle of the eligible 

grade distribution than for enrollees at the top or bottom of the distribution. 

• QOP’s impacts varied from site to site, and the impacts for the whole QOP 

demonstration were substantially, but not entirely, attributable to the impacts of the 

Philadelphia site alone or the Philadelphia and Yakima sites (the Ford-funded sites) 

together. The DOL-funded sites significantly increased the likelihood of graduating 

from high school, one of QOP’s primary goals, but had no other statistically significant 

impacts. 

• Two sites implemented a version of QOP that deviated substantially from the program 

model, and the other five sites implemented versions that deviated moderately from the 

model. 

• The total cost per enrollee over the full five-year demonstration period was $18,000 to 

$22,000 for DOL-funded sites, $23,000 for the Yakima site, and $49,000 for the 

Philadelphia site. 
24 

• Most enrollees attended relatively few program activities. Enrollees spent an average of 

174 hours per year on QOP activities—23 percent of the annual goal of 750 hours—

through the first four years of the demonstration. The average fell steadily from 247 

hours in the first year to 89 hours in the fourth year, while the fraction of enrollees 

spending no time at all on QOP activities rose steadily from 1 percent to 36 percent. 

The average annual participation in the two Ford-funded sites was 294 hours—more 

than twice the average annual participation of 126 hours in the five DOL-funded sites. 

As noted, the impacts presented in this report are short-term impacts that we estimated from 

data collected during the fourth and fifth years of the demonstration, that is, before the 

demonstration was over and when many youth were either still attending high school or had only 

recently graduated. Longer-term impacts, which may be a more appropriate basis for policy 

decisions, might be more or less favorable than the short-term impacts. To measure longer-term 

impacts, DOL is having us collect data in fall 2002 and fall 2004. The fall 2002 data collection is 

roughly seven years after the youth in the demonstration sample entered the ninth grade and two 

years after the end of the QOP demonstration. 
25 

R EFERENCES

Maxfield, Myles, Laura Castner, Vida Maralani, and Mary Vencill. “The Quantum Opportunity 

Program Demonstration: Implementation Findings.” Washington, DC: Mathematica Policy 

Research, Inc., 2003. 

Orr, Larry L., Howard S. Bloom, Stephen H. Bell, Fred Doolittle, Winston Lin, and George Cave. 

Does Training for the Disadvantaged Work? Evidence from the National JTPA Study. Washington, DC: 

The Urban Institute Press, 1996. 

Schirm, Allen, Nuria Rodriguez-Planas, Myles Maxfield, and Christina Tuttle. “The Quantum 

Opportunity Program Demonstration: Short-Term Impacts.” Washington, DC: Mathematica 

Policy Research, Inc., 2003. 

Schochet, Peter Z., John Burghardt, and Steven Glazerman. “National Job Corps Study: The 

Impacts of Job Corps on Participants’ Employment and Related Outcomes.” Princeton, NJ: 

Mathematica Policy Research, Inc., 2001. 

 . “National Job Corps Study: The Short-Term Impacts of Job Corps on Participants’ 

Employment and Related Outcomes.” Princeton, NJ: Mathematica Policy Research, Inc., 2000. 

U.S. Department of Justice. “Guide for Implementing the Comprehensive Strategy for Serious, 

Violent, and Chronic Juvenile Offenders.” Washington, DC: U.S. Department of Justice, 1995.