Office of Inspector General—Office of Audit 

 

EMPLOYMENT AND TRAINING 

ADMINISTRATION 

JOB CORPS NEEDS TO IMPROVE RELIABILITY 

OF PERFORMANCE METRICS AND RESULTS 

Date Issued: September 30, 2011 

Report Number: 26-11-004-03-370 
 

 

 

 

 

U.S. Department of Labor 

Office of Inspector General 

Office of Audit 

BRIEFLY... 

Highlights of Report Number 26-11-004-03-370, 

issued to the Assistant Secretary, Employment and 

Training Administration. 

WHY READ THE REPORT 

The Department of Labor, Employment and Training 

Administration (ETA), oversees the Office of Job Corps 

(Job Corps), which administers and manages the Job 

Corps program. The program’s mission is to help at-risk 

youth become more employable, responsible, and 

productive citizens. 

Prior audit reports and evaluations conducted over the 

last 15 years cited concerns about the reliability of the 

metrics covering two key performance areas – job 

training match, which is used to measure how 

effectively participants are placed in jobs that match the 

training they received in Job Corps, and cost efficiency, 

which is used to measure program costs. Our audit 

confirmed these long-standing deficiencies persist. 

Job Corps’ budget for program years (PY) 2009 and 

2010 totaled approximately $3.39 billion ($1.68 billion 

for PY 2009 and $1.71 billion for PY 2010). 

WHY OIG CONDUCTED THE AUDIT 

We conducted the audit to address the following 

question: 

To what extent does Job Corps have metrics in place 

to assess the program’s performance? 

Our scope covered Job Corps’ performance metrics 

and outcomes for PY 2009 and the month of October 

2010, and policies, procedures, and processes through 

March 7, 2011. 

READ THE FULL REPORT 

To view the report, including the scope, methodology, 

and full agency response, go to: 

http://www.oig.dol.gov/public/reports/oa/2011/26-11-

004-03-370.pdf. 

SEPTEMBER 2011 

JOB CORPS NEEDS TO IMPROVE RELIABILITY OF 

PERFORMANCE METRICS AND RESULTS 

WHAT OIG FOUND 

Our audit identified deficiencies with 22 of the 58 

performance metrics. These deficiencies included 

reporting inaccurate results (9 metrics); not reporting 

results and/or establishing targets (4 metrics); and not 

publicly publishing required results as Job Corps 

asserted it would in response to our prior work on noncompliance with the Workforce Investment Act of 1998 

(WIA) reporting requirements (19 metrics). For 

example, we found Job Corps overstated 42.3 percent 

(7,517) of 17,787 job training match placements it 

reported to comply with WIA for the periods reviewed. 

These overstatements included 3,226 (18.1 percent) 

matches where the jobs did not relate or poorly related 

to the students’ training and 3,778 (21.2 percent) 

matches where students were enrolled in postsecondary education or training rather than jobs. The 

job training matches also included 1,569 placements in 

jobs that required little or no previous work-related 

skills, knowledge, or experience, such as fast food 

cooks and dishwashers that potentially could have been 

obtained without Job Corps training. Thus, we believe 

that if Job Corps improves oversight to better recruit, 

train, and place these students, an estimated $61.18 

million would be put to better use. 

There were also problems with Job Corps’ approach for 

calculating its cost efficiency metric, or cost per 

participant ($26,551 for PY 2009). For example, the 

metric did not effectively measure performance. 

Additionally, our analysis of available Job Corps data 

showed alternate cost efficiency metrics, such as cost 

per student training slot utilized ($37,880), if all slots are 

fully utilized) or job placement ($76,574), could provide 

decision-makers with more reliable information to 

measure and manage the program’s performance and 

costs. 

WHAT OIG RECOMMENDED 

The OIG recommended that ETA require Job Corps 

review and improve its performance metrics to provide 

decision-makers with useful and reliable information 

regarding the program’s performance and costs; 

improve oversight of its service providers to increase 

the number of students who find employment that relate 

to and utilize the vocational training received; and 

develop a process to ensure work contracted for and 

conducted by consultants is managed appropriately. 

The Assistant Secretary for Employment and Training 

did not completely agree with our conclusions, but fully 

concurred with one recommendation and partially 

concurred with two recommendations. 
U. S. Department of Labor – Office of Inspector General 

 PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Table of Contents 

Assistant Inspector General’s Report ......................................................................... 1
 

Results In Brief.............................................................................................................. 2
 

Objective 1 — To what extent does Job Corps have metrics in place to assess the 

program’s performance?..................................................................... 5
 

Job Corps overstated the program’s job training match accomplishments and 

cost efficiency. ........................................................................................................... 5
 

Finding — Job Corps’ performance metrics did not always provide an 

accurate assessment of the program’s performance.......................... 5
 

Recommendations ...................................................................................................... 20
 

Exhibits 

Exhibit 1 Analysis of Job Corps’ 58 Performance Metrics .................................. 23
 

Exhibit 2 Job Corps Expenses for Prior and On-going Performance 

Measurement Evaluations.................................................................... 33
 

Appendices 

Appendix A Background ..................................................................................... 37
 

Appendix B Objectives, Scope, Methodology, and Criteria ................................ 39
 

Appendix C Terminology .................................................................................... 45
 

Appendix D Acronyms and Abbreviations .......................................................... 51
 

Appendix E ETA Response to Draft Report........................................................ 53
 

Appendix F Acknowledgements ......................................................................... 61
 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

U.S. Department of Labor Office of Inspector General 

Washington, D.C. 20210 

September 30, 2011 

Assistant Inspector General’s Report 

Ms. Jane Oates 

Assistant Secretary 

for Employment and Training 

200 Constitution Avenue, NW 

Washington, D.C. 20210 

The Department of Labor (DOL), Employment and Training Administration (ETA), 

oversees the Office of Job Corps (Job Corps), which administers and manages the Job 

Corps program. This program’s mission is to help at-risk youth become more 

employable, responsible, and productive citizens. The typical Job Corps student 

entering the program has not completed high school, reads slightly below the 8th grade 

level, has never had a full-time job, is between 16 and 24 years old, and is from an 

economically disadvantaged family. Given the significant academic, employability, and 

vocational barriers many Job Corps students face, Job Corps takes a holistic approach 

to education and training, assessing each student’s needs and designing an 

individualized program or intervention to address those needs. 

Job Corps’ budget for program years (PY) 2009 and 2010 totaled approximately $3.39 

billion ($1.68 billion for PY 2009 and $1.71 billion for PY 2010). Job Corps relies on a 

complex performance management system consisting of 58 performance metrics to 

assess the program’s effectiveness and meet mandated and internal accountability 

priorities. Over the last 15 years, the Government Accountability Office (GAO), the 

Office of Management and Budget (OMB), and a consultant hired by Job Corps itself 

have expressed long-standing concerns about the reliability of the metrics covering two 

key performance areas – job training match, which is used to measure how effectively 

participants are placed in jobs that match the training they received in Job Corps, and 

cost efficiency, which is used to measure program costs. Since 2007, DOL has spent 

$1.46 million on consultants and an advisory committee to improve its performance 

metrics and outcomes, as well as other aspects of the program. 

Our audit objective was to answer the following question: 

To what extent does Job Corps have metrics in place to assess the program’s 

performance? 

Our scope covered Job Corps’ performance metrics and outcomes for PY 2009 and the 

month of October 2010, and policies, procedures, and processes through 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

1 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

March 7, 2011. Our work was generally conducted at Job Corps headquarters in 

Washington, D.C. 

To accomplish our objective, we interviewed Job Corps management officials and staff; 

interviewed program examiners at OMB to gain their feedback on Job Corps’ 

compliance with certain OMB requirements; reviewed related GAO guidance and 

consulted with GAO management and senior analysts to gain GAO’s perspective on 

performance measurement; reviewed applicable mandated and internal requirements; 

reviewed prior Office of Inspector General (OIG) and GAO audit reports and evaluations 

conducted by DOL consultants and Job Corps staff for prior deficiencies; assessed 

corrective actions and determined the impact on select performance results; analyzed 

Job Corps’ performance metrics and corresponding results against key attributes for 

successful performance measures; determined related data applicable to our scope was 

sufficiently reliable to accomplish our audit objectives; and performed walkthroughs of 

select metrics and results. 

We conducted this performance audit in accordance with generally accepted 

government auditing standards. Those standards require that we plan and perform the 

audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that the evidence 

obtained provides a reasonable basis for our findings and conclusions based on our 

audit objectives. Our objective, scope, methodology and criteria are detailed in 

Appendix B. 

RESULTS IN BRIEF 

While Job Corps had 58 performance metrics in place (refer to Exhibit 1), these metrics 

did not always provide a clear and accurate assessment of the program’s performance. 

In aggregate, we found reporting deficiencies with 22 of the 58 metrics; including 10 

with multiple deficiencies. Specifically, Job Corps reported inaccurate results (9 

metrics); did not report results and/or establish performance targets (4 metrics); or did 

not publish results and make them publicly available as asserted in its response to our 

March 2009 report on non-compliance with the Workforce Investment Act of 1998 (WIA) 

reporting requirements (19 metrics). Thus, Job Corps and other decision-makers (e.g., 

Congress, ETA) did not always have complete and accurate performance information 

on which to base effective decision-making. 

Five of the nine metrics reported inaccurate accomplishments related to two key areas 

of performance – job training match (four) and cost efficiency (one). Prior audit reports 

and evaluations over the last 15 years expressed serious concerns about the reliability 

of the metrics covering these two areas. While we noted changes that supported an 

overall trend toward improvement, such as reducing the number of allowable broad 

placement categories (e.g., cashier, retail salespersons) that were considered matches 

for several vocations, reliability problems persist. For example, problems with how Job 

Corps calculated its job training matches led to an overstatement of 7,517 (42.3 

percent) of its 17,787 matches reported to comply with WIA for the periods reviewed. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

2 Report No. 26-11-004-03-370 
 

 

U. S. Department of Labor – Office of Inspector General 

Specifically, 3,226 of these matches either did not relate or poorly related to the 

vocational training received (e.g., students trained in office administration placed in fast 

food restaurants); and another 4,291 matches that were enrollments in post-secondary 

education and training (3,778) and military enlistments (513), regardless of the students’ 

vocational training and assigned duties. 

The aggregate job training matches also included 1,569 placements in jobs that 

required little or no previous work-related skills, knowledge, or experience, such as fast 

food cooks and cashiers, janitors, and dishwashers that potentially could have been 

obtained without Job Corps training. While Job Corps may have helped these students 

become more employable and responsible, the final outcome – placements in jobs the 

students potentially could have obtained without Job Corps training – was not the best 

result the program could have achieved. 

In addition, there were numerous problems with Job Corps’ approach for calculating its 

cost efficiency metric, or cost per participant ($26,551 for PY 2009), which Job Corps 

derived by dividing a portion of its appropriated expenses by the number of new 

participants over the course of a program year. The metric did not reflect outcomes 

such as job placement or training completion; excluded program administration 

expenses; and could inaccurately show acceptable or improved performance when 

program performance is actually declining. For example, increases in new participants 

due to a higher number of drop-outs would inaccurately reflect improvements in Job 

Corps’ cost efficiency results because costs per participant would decrease. In 

response to concerns raised by OMB in 2007, Job Corps acknowledged this metric was 

inadequate because it was not linked to performance outcomes and indicated it would 

adopt efficiency metrics by September 2009 that would link to performance outcomes 

and account for all costs. However, Job Corps has not modified its approach and 

continues to use cost per participant to measure efficiency. Job Corps also told us it did 

not use this metric to make performance or outcome decisions, despite being provided 

to Congress in Job Corps’ Congressional Budget Justifications for fiscal years 2009 – 

2012. Our analysis of available Job Corps data showed many alternate cost efficiency 

metrics based on planned enrollment or outcomes, such as cost per training slot utilized 

($37,880, if all slots are fully utilized), successful outcome ($42,952), or job placement 

($76,574), could provide decision-makers with more reliable information to measure and 

manage the program’s performance and costs.1 

Since 2007, DOL has spent $1.46 million on consultants and an advisory committee to 

improve Job Corps’ performance metrics and outcomes, as well as other aspects of the 

program. However, many of these concerns persist because Job Corps did not place 

sufficient emphasis to implement the necessary improvements to more effectively 

assess the program’s performance. For example, Job Corps did not accept its 

consultant’s cost efficiency recommendations, which would have addressed some of 

our related concerns. 

1

Calculations based on Job Corps data for PY 2009. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

3 Report No. 26-11-004-03-370 
 

 

U. S. Department of Labor – Office of Inspector General 

In total, we identified $61.18 million in funds that would be put to better use if Job Corps 

improves oversight to more effectively recruit, train, and place these students. This 

included $1.65 million Job Corps would pay the contracted service providers to place 

students in jobs that relate to the vocational training received2 and $59.53 million Job 

Corps would spend on students obtaining jobs as a result of the recruiting, training, and 

placement efforts provided by Job Corps.3 

We recommend the Assistant Secretary for Employment and Training require Job Corps 

to (1) review and improve its performance metrics to provide decision-makers with 

useful and reliable information regarding the program’s performance and costs, which 

includes ensuring metrics are complete and accurate, comply with WIA, and have 

reported results and established targets; (2) improve oversight of its service providers to 

increase the number of students who find employment that relate to and utilize the 

vocational training received; and (3) develop a process to ensure the scope of work 

contracted for and conducted by consultants is managed appropriately. 

ETA RESPONSE 

In response to our draft report, ETA stated it will take responsibility for the job training 

matches found to be invalid and will assess liquidated damages to the responsible 

contractors. However, ETA did not agree with our conclusions related to automatically 

counting military enlistments and educational enrollments as job training matches; 

placing students in jobs that required little or no training; developing and using alternate 

cost efficiency metrics; managing its consultants more appropriately; and overall 

calculation of funds put to better use. 

OIG CONCLUSION 

We revised our estimate of funds put to better use for students placed in jobs that 

required little or no training as recommended by Job Corps. Our estimate is based on 

the liquidated damages amount of ($750) specified in Job Corps’ PRH for invalid 

placements. With the exception of this revision, ETA did not provide any new 

information that changed the conclusions stated in the report. 

2

Calculation of $1.65 million did not include overstated match placements in jobs that required little or no previous 

work-related skills, knowledge, or experience. See footnotes 14 and 15 for details. 3

Calculation of $59.53 million (1,569 x $37,941) derived by multiplying 1,569 placements by the cost per training slot 

utilized ($37,941 if all slots are fully utilized). See footnote 17 for additional details.
 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

4 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

RESULTS AND FINDINGS 

Objective 1 — To what extent does Job Corps have metrics in place to assess the 

program’s performance? 

Job Corps overstated the program’s job training match accomplishments and 

cost efficiency. 

Finding — Job Corps’ performance metrics did not always provide an accurate 

assessment of the program’s performance. 

Our audit identified deficiencies with 22 of the 58 performance metrics. These 

deficiencies included reporting inaccurate results (9 metrics); not reporting results 

and/or establishing targets (4 metrics); and not publicly publishing WIA-required results 

as Job Corps asserted it would in response to our prior work (19 metrics). In addition, 

prior audit reports and evaluations conducted over the last 15 years cited concerns 

about the reliability of the metrics covering two key performance areas – job training 

match and cost efficiency. Our audit confirmed these long-standing deficiencies persist. 

Job Corps established and uses a complex performance management system to meet 

mandated and internal accountability priorities and to assess the program’s 

effectiveness. The system consists of 58 metrics developed to address multiple 

performance reporting requirements mandated by WIA, the Government Performance 

and Results Act of 1993 (GPRA), OMB, the Common Measures, DOL, and Job Corps. 

These various requirements and corresponding metrics provide the basis for assessing 

the program’s performance and effectiveness, and establish accountability for achieving 

results, such as placements in employment, cost efficiency, and attainment of a high 

school diploma (HSD) or general educational development (GED) certificate. See 

Exhibit 1 and Appendix A for details. Job Corps uses metric results to assess the 

performance of Job Corps center operators and contracted service providers, which 

impact decisions about contract awards and renewals, incentive fees, and performance 

bonuses. 

We analyzed Job Corps’ 58 performance metrics to determine if they met certain 

attributes that GAO cited as key to successful performance measures. The attributes 

were: 

	 Linkage to Goals. Metric was aligned with agency and program mission and 

goals.4 

	 Complete and Accurate Results. Metric, by design, was consistent with 

mandated or internal performance measure requirements and produced the 

appropriate results. To address prior concerns related to Job Corps’ job training 

4

GAO, GAO 03-143 (Washington, D.C.: November 2002). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

5 Report No. 26-11-004-03-370 
 

 

 

U. S. Department of Labor – Office of Inspector General 

match and cost efficiency metrics, we also tested the data reliability of select 

performance outcomes related to these metrics.5 

	 Target and Reported Results. Metric had a specific target and reported results 

that were quantifiable and measurable to fully evaluate progress toward expected 

performance.6 

Of the 58 metrics, 25 were required by WIA. We tested these 25 WIA-required metrics 

to determine whether the performance results were published and publicly available as 

asserted by Job Corps in response to our prior work.7 

22 of 58 Performance Metrics Had Deficiencies 

In aggregate, we found all 58 performance metrics were linked or aligned with DOL and 

Job Corps’ mission and goals. However, 22 metrics had reporting deficiencies, which 

included 9 metrics with inaccurate results. These nine metrics had inaccurate results 

because the metrics, by design, did not measure the required performance areas as 

mandated by WIA or OMB and therefore, did not produce the appropriate results. For 

metrics to be effective, they must be properly defined and correctly measured. None of 

these nine metrics met all of these criteria. See Exhibit 1 for details. Five of these nine 

metrics related to job training match (four) and cost efficiency (one). 

Known Deficiencies Concerning the Reliability of Job Corps’ Job Training Match and 

Cost Efficiency Metrics Persisted for Years 

Over the last 15 years, GAO, OMB, and a consultant hired by Job Corps have 

expressed long-standing concerns about the reliability of the metrics covering two key 

performance areas – job training match and cost efficiency. For job training match, WIA 

requires Job Corps to measure and report how effectively participants are placed in jobs 

that match the vocational training they received in Job Corps. In 1997, GAO reported 

the program’s job training match metrics were flawed because performance results 

included jobs that (1) did not relate, or poorly related, to the training received or (2) 

required relatively little training to perform ranging from a short demonstration to one 

month of instruction.8

 In 1998, GAO also questioned Job Corps’ policy for counting any 

participant that enlisted in the military as a successful job training match, regardless of 

the student’s vocational training and assigned duties.9 

In addition, since 2007, OMB,10 a contractor hired by Job Corps,11 and GAO12 raised 

concerns about the limitations of Job Corps use of cost per participant to measure cost 

5

GAO, GAO/GGD-96-118 (Washington, D.C.: June 1996). 

6

GAO, GAO 03-143 (Washington, D.C.: November 2002). 

7

DOL OIG, Report No. 04-09-003-01-370 (Washington, D.C.: March 2009). 8

GAO, GAO/HEHS-98-1 (Washington, D.C.: October 1997).

9

GAO, GAO/HEHS-99-15 (Washington, D.C.: November 1998). 

10OMB, Program Assessment Rating Tool (PART): Job Corps (2007). 11HeiTech Services, Inc., Job Corps Cost Measure: Selecting a Cost Measure to Assess Program Results, Contract 

No. DOLJ079625486, (December 2008). 

12GAO, GAO-10-394 (Washington, D.C.: May 2010). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

6 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

efficiency and recommended the use of cost efficiency metrics that link performance 

outcomes (e.g., placements in employment or post-secondary education or training) to 

actual, total program costs rather than select appropriated funding. While Job Corps has 

taken steps to improve its job training matches and explore alternate cost efficiency 

metrics, our work found these concerns persist, and Job Corps continues to use cost 

per participant to measure the program’s cost efficiency. 

18.1 Percent of Job Training Matches Included Jobs That Did Not Relate or Poorly 

Related to Training 

To address long-standing concerns raised by GAO, Job Corps changed several policies 

and procedures over the years to improve its job training match crosswalk – criteria 

used to determine training-related placements – and enhance its overall credibility. 

These changes included improving the linkage between Job Corps’ crosswalk and the 

Occupational Information Network’s Standard Occupational Classification systems 

(O*NET-SOC), a database of occupational requirements and worker attributes, 

maintained by ETA. O*NET-SOC is the recognized standard for occupational 

information and describes occupations in terms of the skills and knowledge required; 

how the work is performed; and typical work settings. Between 2008 and 2010, Job 

Corps worked in consort with ETA to improve the linkage between training, placements, 

and O*NET-SOC, and issued a revised crosswalk in October 2010. According to Job 

Corps, the 2010 crosswalk substantially improved the fundamental linkage between 

training and placement by (1) categorizing training and job activities and (2) recreating 

the entire crosswalk from the ground up. In addition, Job Corps told us the 2010 

crosswalk fixed the broad placement categories, such as cashier and retail 

salespersons, that were considered matches for several unrelated vocations. 

As part of our testing, we reviewed Job Corps’ crosswalk and related policies, metrics, 

and outcomes for PY 2009 and the month of October 2010 to determine whether Job 

Corps had taken the necessary corrective actions to address prior audit deficiencies 

and the validity of its job training match placements reported for these periods. See 

Appendix B for a detailed description of our audit methodology. While we noted 

crosswalk changes that supported an overall trend toward improvement, the 

improvements were less significant than Job Corps described. For example, Job Corps 

reduced the number of broad placement categories in its 2010 crosswalk, but continued 

to allow placements as stock clerks (sales floor) or parts salespersons to be broadly 

used for any vocation, including culinary and nursing. Job Corps acknowledged the data 

integrity risks associated with these two broad job placement categories and told us it 

was developing software enhancements to limit the likelihood of inaccurate matches. In 

addition, Job Corps continued to allow matches that were questioned 15 years ago by 

GAO such as auto repair students placed as car detailers or car washers and 

hotel/motel service industry students placed as ticket sellers at movie theaters. Other 

matches considered acceptable in the 2009 and 2010 crosswalks included: 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

7 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

	 Culinary students placed as pest control workers, funeral attendants, baggage 

porters, concierges, tour guides, telemarketers, cashiers, telephone operators, 

financial examiners, accounting clerks, and file clerks (PY 2009). 

	 Nurse assistant and pharmacy technician students placed as cashiers, telephone 

operators, tax preparers, real estate agents, couriers, file clerks, and reservation 

and transportation ticket agents (PY 2009). 

	 Solar energy and weatherization students placed as janitors and pest control 

workers (October 2010). 

In verifying the data reliability of Job Corps job training match outcomes, we reviewed 

the database records for all 17,787 reported matches for PY 2009 (16,234) and October 

2010 (1,553). We determined whether each student’s vocational training appropriately 

related to their job placements in accordance with WIA. We based our determination on 

the student’s place of employment and job position, and duties (when available), 

regardless of whether the matches were considered acceptable by Job Corps’ 

crosswalks. For example, we considered nursing assistant students employed as 

medical assistants, home health aides, or registered nurses appropriate job training 

matches (included in the 2009 and 2010 crosswalks). However, we questioned the 

matches such as the ones described above. As a result, we found 3,226 (18.1 percent) 

overstated matches for both periods where the jobs did not relate or poorly related to 

the students’ training.13 These included: 

	 Non-culinary students, including office administration, medical office, and 

construction students, placed at fast food employers, such as McDonald’s, 

Burger King, and Wendy’s as crew persons, cashiers, or customer service. This 

accounted for 326 (PY 2009) and 20 (October 2010) matches. 

	 Non-retail students, including office administration, culinary, automobile 

technician, and nurse assistant students, placed as sales persons, stockers, or 

laborers at Walmart. This accounted for 113 (PY 2009) and 28 (October 2010) 

matches. 

	 Various students, including culinary, nurse assistant and medical office support, 

placed at the United States Census as enumerators, clerks, and crew. This 

accounted for 17 matches (PY 2009). 

13These placements were based upon our representative simple random sample that was estimated with 95 percent 

confidence and assuming the highest variance in the populations. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

8 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Table 1 below summarizes our test results and the specific training areas impacted. 

Table 1: Number of Job Placements That Did Not Relate or Poorly Related to Training 

PY 2009 October 2010 Total 

Reported Job Training Matches 16,234 1,553 17,787 

Vocational Training Received Not Related or Poorly Related Placements 

1) Office Administration 907 83 990 

2) Constructiona

 782 70 852 

3) Culinary Arts 269 15 284 

4) Medical Office Support 321 25 346 

5) Nurse Assistant 139 16 155 

6) Automobile Techniciana

 101 6 107 

7) Material Handling and Distribution 83 3 86 

8) Retail Sales 58 1 59 

9) Pharmacy Technician 54 11 65 

10) Hotel and Lodging 53 6 59 

11) Accounting 50 1 51 

12) Heavy Equipment Operationsa 49 12 61 

13) Facilities Maintenance 36 15 51 

14) Security and Protective 21 9 30 

15) Forestry Conservationa 9 0 9 

16) Landscaping 5 0 5 

17) Dentist Assistant 4 1 5 

18) Computer Technician 4 7 11 

Totals 2,945 281 3,226 

Percentage of Job Placements That Did 18.1 18.1 18.1 

Not Relate or Poorly Related to Training 

Source: OIG analysis of Job Corps job training match data. 

a

For purposes of this table: 

- construction included asphalt paving, bricklayer, carpentry, cement masonry, electrical, floor covering, overhead 

line construction, painter, plaster, plumber, sign and billboard, tile setting, and welding students; 

- automobile technician included auto parts, collision repair and general service technician students; 

- heavy equipment operations included heavy truck driving, machine repair and roustabout students; and 

- forestry conservation included urban forestry and conservation firefighting students. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

9 Report No. 26-11-004-03-370 
 

 

 

U. S. Department of Labor – Office of Inspector General 

On average, Job Corps spent an estimated $1,196 per student on career transition 

services (CTS) to help these students find and retain jobs that matched their training.14 

Given the results, we believe the $1.65 million spent on these efforts would be put to 

better use if Job Corps improves oversight of its CTS to more effectively place the 

students.15 

Job Corps indicated its job training match metrics were in compliance and consistent 

with WIA, but acknowledged that job training matches may have been overstated, just 

not to the extent OIG identified. In response, Job Corps told us it recently installed other 

system mechanisms to hold career transition services providers more accountable for 

genuine job training match placement, such as requiring them to attest, in writing, to the 

validity of each job training match credit. Job Corps said it would convene a workgroup 

to further improve the career transition services system. Job Corps indicated it will take 

responsibility for any invalid matches and take steps to correct them, assess liquidated 

damages to applicable career transition services agencies, and prevent future 

occurrences. 

Automatic Inclusions for Military Enlistments and Post-Secondary Education/Training 

Enrollments Overstated Job Training Match Placements by 3.9 Percent and 21.2 

Percent Respectively 

In addition, Job Corps continued to automatically count all military enlistments as job 

training matches without regard to the students’ vocational training and assigned duties, 

even though GAO questioned this practice in 1998. According to Job Corps, military 

enlistments were captured as automatic matches as early as 1992. In 2007, Job Corps 

further expanded the job training match metrics to automatically count non-jobs (i.e., 

enrollments in post-secondary education or training) in response to recommendations 

from the Secretary’s Advisory Committee on Job Corps, an advisory committee formed 

by the Secretary of Labor to provide advice and recommendations related to policies, 

legislation, and regulations related to Job Corps. Concerned that the job training match 

metrics created a disincentive for some centers to place students forward to advanced 

training or other more beneficial placements, the committee recommended Job Corps 

incentivize job training match to include rewards for advanced training, post-secondary 

placement or additional training that might lead to higher wage employment. Job Corps 

indicated that the inclusion of both military and educational placements is consistent 

with Job Corps’ overall teaching and learning approach because these placements 

present students with a broad range of career, not just job, options that lead to long14Calculation of $1,196 per student ($53.38 million/44,636) is the result of dividing the total CTS expenses in the 

amount of $53.38 million by the total number of students in Job Corps placement pool for PY 2009 (44,636). 

Expenses and number eligible graduates and former enrollees for October 2010 were not available. Numbers may 

not add up exactly due to rounding. 

15Job Corps indicated that the estimate of $1,196 did not account for all of the services provided beyond placements, 

such as identifying housing and other forms of assistance. As recommended by Job Corps, we used the liquidated 

damages amount of ($750) to calculate the funds put to better use as specified in Job Corps’ Policy and 

Requirements Handbook (PRH) for invalid placements. While we do not consider these placements to be invalid, we 

agreed to use this lower amount to calculate the funds put to better use. The calculation of $1.65 million (2,194 x 

$750) did not include overstated match placements in jobs that required little or no previous work-related skills, 

knowledge, or experience (1,032) and is the product of 2,194 (3,226 – 1,032) multiplied by $750. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

10 Report No. 26-11-004-03-370 
 

 

 

 

 

U. S. Department of Labor – Office of Inspector General 

term, stable employment with self-sustaining wages. While Job Corps’ broad 

interpretations of job training matches acknowledge the various pathways to gaining 

employment and the depth of the program’s training, it strays from WIA by automatically 

counting military enlistments and non-jobs without showing how these placements 

match the training the students received. Moreover, Job Corps has other metrics which 

include placements in military and education. Including these placements in the job 

training match metrics overstates Job Corps’ performance and dilutes the value of the 

metrics. These deficiencies were found in Job Corps’ crosswalk, policies, and related 

outcomes for both of the periods reviewed and accounted an aggregate of 513 (3.9 

percent) military enlistments and 3,778 (21.2 percent) placements in non-jobs (postsecondary education and training). 

Overall, we found Job Corps overstated 42.3 percent (7,517) of its 17,787 job training 

matches reported to comply with WIA for both PY 2009 and October 2010. See Table 2 

below. 

Table 2: Job Training Matches Overstated by 42.3 Percent 

PY 2009 

Reported Job Training Matches 16,234 

October 

2010 

1,553 

Totals (%) 

17,787 (100) 

Overstated Job Training Matches 

1) Job placements that did not relate or 

poorly related to training received.............. 

2) Automatic placements 

2,945 281 3,226 (18.1) 

Military enlistments............................. 469 44 513 (3.9) 

Post-secondary education and training 

enrollments....................................... 3,368 410 3,778 (21.2) 

Subtotals 

Totals 

3,837 

6,782 

454 

735 

4,291 (24.1) 

7,517 (42.3) 

Source: OIG analysis of Job Corps job training match data. 

Placements in Jobs Requiring Relatively Little Training to Perform Resulted in Poor Job 

Training Matches 

Job Corps continued to report training-related matches for job placements that required 

relatively little training to perform even though this practice was questioned by GAO 15 

years ago. Job Corps criteria and related outcomes included 42 occupations that 

O*NET-SOC, upon which the crosswalk was based, identified as needing little or no 

previous work-related skill, knowledge, or experience to perform. According to O*NETSOC, the average training time for employees in these occupations ranged from a few 

days (short demonstration) to three months.16 In comparison, graduates, who account 

16The occupational categories GAO questioned in 1997 required up to one month of training. Similar categories used 

in the crosswalks we reviewed indicated up to three months of training. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

11 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

for all job training matches, stay an average of 12 months at a cost of approximately 

$37,941 per student training slot, if fully utilized.17 This raises the concern that a number 

of Job Corps students potentially could have obtained some of these jobs without Job 

Corps training. 

According to Job Corps, these classifications were written with the average American 

worker in mind, not those from disadvantage backgrounds. The typical Job Corps 

student entering the program has not completed high school, reads slightly below the 8th 

grade level, has never had a full-time job, is between 16 and 24 years old, and is from 

an economically disadvantaged family. Given the significant academic, employability, 

and vocational barriers many Job Corps students face, we reduced the 42 occupations 

classified by O*NET-SOC to 15 occupations we determined needed the least amount of 

training to perform. These 15 occupations included fast food cooks, janitors, vehicle 

cleaners, dishwashers, and parking lot attendants. In aggregate, 1,569 students were 

placed in these jobs that potentially could have been obtained without Job Corps 

training.18 Incidentally, jobs at fast food restaurants accounted for over one-third or 37.3 

percent of these placements.19 The specific occupations and the impact on placements 

for PY 2009 and October 2010 are summarized in Table 3 on the following page. 

17The average cost per training slot for both periods totaled $37,941 (($37,880 + $38,002)/2). The cost per training 

slot totaled $37,880 ($1,683,938,000 budgeted appropriations/44,455 budgeted training slots) for PY 2009 and 

$38,002 ($1,708,205,000/44,950 budgeted training slots) for October 2010. 18These placements were based upon our representative simple random sample that was estimated with 95 percent 

confidence and assuming the highest variance in the populations. 

19This amount was calculated by dividing the total number of fast food cooks (476) and fast food cashiers (110) by 

these placements (1,569). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

12 Report No. 26-11-004-03-370 
 

 

 

 

U. S. Department of Labor – Office of Inspector General 

Source: OIG analysis of Job Corps job training match data. 

a

These placements also include 1,032 placements that did not relate or poorly related to the training received and are 

accounted for in Tables 1 and 2 above. 

b

For purposes of this table: 

- office administration also included accounting services, computer networking, computer technician and visual 

communication students; 

- construction also included bricklaying, carpentry, cement masonry, electrical, floor covering, heating ventilation 

and air conditioning, landscaping, painting, plasterer, plumbing, tile setting, urban forestry and welding students; 

- medical office also included clinical medical assistant, dentist assistant and pharmacy technician students; and 

- heavy equipment operations includes students completing vocational training of heavy truck driving, manufacturing 

and machining. 

c

For purposes of our testing, we broke out two categories – fast food cook and counter attendant – to account for 

related cashier placements (e.g., cashier at a fast food restaurant). 

Job Corps acknowledged these were not ideal placements, but stated that achieving 

and maintaining a basic entry level job at the end of the Job Corps experience is a 

crowning achievement for most students, particularly in a labor market of high 

unemployment as has been the case for the past several years. While we recognize 

that Job Corps efforts may have helped these students become more employable and 

responsible, and in some instances placed students in jobs related to the training 

received (e.g., culinary students at fast food restaurants), the final outcome – 

placements in jobs the students potentially could have obtained without Job Corps 

training – was not the best result the program could have achieved. Thus, we believe 

the funds expended for these students’ training slots, which in aggregate cost an 

estimated $59.53 million (1,569 x $37,941) would be put to better use if Job Corps 

improves oversight to more effectively recruit, train, and place these students. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

13 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

As such, the job training match metric did not provide Job Corps and other decisionmakers (e.g., Congress, ETA) with reliable performance information to determine how 

effectively Job Corps placed students in jobs that matched the vocational training they 

received and to determine where Job Corps needs to improve to achieve desired 

outcomes. Additionally, these results are used to assess and rank the performance of 

Job Corps center operators and contracted service providers on performance report 

cards which impact incentive fee, performance bonus, and contract renewal decisions. 

Job training match performance accounts for 10 percent of the overall report card rating 

for contracted service providers that place students and 5 percent for center operators. 

Cost Efficiency Metric Did Not Reflect Outcomes and Overstated Cost Efficiency 

There were numerous problems with Job Corps’ approach for calculating its cost 

efficiency metric, or cost per participant ($26,551 for PY 2009), which Job Corps derived 

by dividing a portion of its appropriated funds by the number of new enrollees over the 

course of a program year. The metric did not reflect outcomes (e.g., job placement, 

HSD or GED attainment, training completion); excluded program administration 

expenses ($28.66 million for PY 2009); and could inaccurately show acceptable or 

improved performance when program performance is actually declining. For example, 

increases in new enrollees due to a higher number of drop-outs would inaccurately 

reflect improvements in Job Corps’ cost efficiency results and lower costs per 

participant. Even though the metric is consistent with ETA’s guidance for reporting cost 

efficiency, OMB, consultants hired by Job Corps and ETA, and GAO have all cited the 

limitations of using cost per participant to measure cost efficiency as early as 2007 and 

recommended the use of alternate metrics that link performance outcomes to actual 

program costs. Given some of these limitations, Job Corps told us the metric, while 

used for resource, contract management and budget decisions, was not used to make 

performance or outcome decisions, despite being provided to Congress in Job Corps’ 

Congressional Budget Justifications for fiscal years 2009 – 2012. For FYs 2010 and 

2011, Job Corps reported PY 2009 cost efficiency, or cost per participant, totaled 

$26,551. These results were based on appropriated operation and construction funds of 

$1.66 billion and 62,344 estimated new enrollees for PY 2009.20 The results did not 

include Job Corps $28.66 million in appropriated program administration funds that 

were used to pay for Federal staff in Job Corps National Office that are responsible for 

program accountability and assessment and monitoring of contractor performance, 

including data integrity and financial accountability. 

In 2007, OMB reported that Job Corps’ efficiency metric was inadequate because it was 

not linked to performance outcomes. Job Corps concurred and indicated it would adopt 

efficiency metrics that link to performance outcomes, account for all costs, and facilitate 

comparisons across DOL training and employment programs by September 2009. To 

address these concerns, Job Corps contracted with a technical expert who 

recommended Job Corps use a cost per successful outcome efficiency metric based on 

the program’s positive performance outcomes, such as graduation, job placement, and 

20Congressional Budget Justification, ETA, Job Corps (FY 2010 and 2011). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

14 Report No. 26-11-004-03-370 
 

 

U. S. Department of Labor – Office of Inspector General 

educational attainment; and actual operating, construction, and direct administration 

costs (rather than appropriations or obligations). Additionally, an ETA-commissioned 

study to help develop outcome-based efficiency metrics for 11 ETA-administered 

employment and training programs, excluding Job Corps, similarly noted the limited 

usefulness of cost per participant efficiency metrics and recommended the 

implementation of outcome-based efficiency metrics that reflect Common Measures 

performance outcomes and actual program costs in 2010.21 Among the efficiency 

metrics recommended for consideration were cost per placement in employment or 

education, cost per retained employment, and cost per post-program (average) 

earnings.22 In 2010, GAO indicated that appropriations or obligations may not represent 

the resources actually used to produce a program’s outcomes in that given year and 

cited Job Corps as an example of a program that used incomplete cost data in its 

efficiency measure. 

While both Job Corps and ETA’s consultants recommended alternate cost efficiency 

metrics, they also cautioned against hasty implementation, as well as against 

comparisons with other employment and training programs because of varying 

objectives, different target populations, and different cost structures. We also recognize 

the challenge of developing efficiency metrics to measure and improve program 

performance. As such, Job Corps had several options for alternate cost efficiency 

metrics to better measure and manage the program’s performance and costs. Many of 

the options considered enrollment, outcome, and cost data that were already being 

collected. Examples using Job Corps PY 2009 data included:23 

	 Cost per student training slot utilized ($37,880). Student training slots represent 

the program’s planned capacity and the maximum number of students that can 

be enrolled in Job Corps at any given time during the program year. Throughout 

the year, multiple students can occupy one slot. However, only one student can 

occupy a slot at a time. This metric provides an assessment of cost relative to 

Job Corps ability to operate at full capacity. When Job Corps’ enrollment efforts 

result in decreasing slot utilization, the cost per student training slot utilized 

increases, indicating a drop in performance. For PY 2009, an estimated 62,344 

students were enrolled, but only 44,455 training slots were available, meaning 

each slot was occupied by 1.4 students. Total program funding equaled $1.68 

billion. As a result, the cost per slot for PY 2009, if fully utilized, averaged 

$37,880 ($1.68 billion/44,455 slots).24 

	 Cost per successful performance outcome ($42,952). Cost per successful, 

outcome-based measure was an option suggested by the commissioned studies 

21Job Corps was not included because it was not part of ETA during the period of the study. However, four other DOL 

WIA programs were included. 

22Capital Research Corporation and Johns Hopkins University Institute for Policy Studies, Implementing Efficiency 

Measures for Employment and Training Programs, Contract No. DOLJ061A20363 (May 2010). 23The breakdown of Job Corps actual operations, construction, and administration costs for PY 2009 was not 

available during our fieldwork. As a result, we used Job Corps appropriated funding for PY 2009 for each of these 

areas as provided in Job Corps’ Congressional Budget Justifications (FY 2010 and 2011). 24Calculation was based on PY 2009 data provided in Job Corps’ Congressional Budget Justifications for FY 2010 

and 2011. Numbers do not add exactly due to rounding. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

15 Report No. 26-11-004-03-370 
 

 

U. S. Department of Labor – Office of Inspector General 

cited above that considered and linked certain Common Measure outcomes, 

such as former enrollees who were placed in a job; the military; or postsecondary education or training and total program graduates to total program 

costs.25 For PY 2009, there were 5,036 former enrollee placements and 34,169 

graduates. Total program funding equaled $1.68 billion. As a result, the cost per 

these successful outcomes averaged $42,952 ($1.68 billion/(5,036 former 

enrollee placements + 34,169 graduates)).26 

	 Cost per single Common Measure performance outcome ($76,574). A third 

option links a single Common Measure performance outcome, such as job 

placement, regardless of job training match, to total program costs. For PY 2009, 

there were 21,991 job placements. Total program funding equaled $1.68 billion. 

As a result, the cost per job placement averaged $76,574 ($1.68 billion/21,991 

job placements).27 

Despite the conclusions and recommendations cited by OMB, GAO, and consultants 

hired by both Job Corps and ETA, Job Corps has not modified its approach for 

calculating cost efficiency and continues to use the cost per participant metric. Job 

Corps did not agree that alternate metrics would provide a better indication of the 

program’s ability to manage costs. Job Corps asserted that its cost per participant 

metric was designed to express the level of resources dedicated to providing each 

student with academic, career technical training, and post-enrollment services; and that 

using a cost per utilized training slot metric would distort cost efficiency. In addition, Job 

Corps indicated it appropriately excluded program administration costs and used 

appropriations in its metric calculations because administration funding was ancillary to 

the delivery of services to students and 100 percent of appropriations were inevitably 

obligated and outlayed. Furthermore, Job Corps asserted the use of performance data 

in cost efficiency metrics significantly distorts the program’s use of resources. For 

example, a cost per successful outcome metric would only consider the costs and not 

the actual services, such as job search assistance that Job Corps is legislatively 

mandated to provide students who are not placed in a job or post-secondary education. 

While we acknowledge such services would not be reflected in this metric, the 

program’s effectiveness is currently assessed based on its ability to achieve certain 

performance outcomes as defined by WIA, the Common Measures, and OMB (e.g., 

employment, education, and job training match placements). Therefore, we affirm the 

value in knowing the costs associated with these outcomes. As such, Job Corps did not 

provide any additional information to cause us to change our conclusions. 

25WIA defined former enrollees as individuals who enrolled in Job Corps, but left before completing their vocational 

training, or attaining a HSD or GED, as a result of participation in the Job Corps program. Graduates are defined as 

individuals who enrolled in Job Corps and completed their vocational training, or attained a HSD or GED, as a result 

of participation in the Job Corps program. Job Corps limited the WIA definition to include only those individuals who 

met these requirements; stayed in the program 60 or more days; and did not separate for violating Job Corps’ Zero 

Tolerance (ZT) conduct policy. 

26Calculation was based on PY 2009 data provided in Job Corps’ MPO-36 management report and Job Corps’ 

Congressional Budget Justifications for FY 2010 and 2011. Numbers do not add exactly due to rounding. 

27Ibid. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

16 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Four Metrics Lacked Reported Results and/or Established Targets 

With regards to targets and results, three metrics did not have reported results and 

established targets (graduate attainment of job readiness and employability skills, 

graduate average wage at 12 months after initial placement, and graduate placement in 

unsubsidized employment of less than 20 hours per week) and one metric did not have 

a target (average length of stay). Measureable targets and reported results are essential 

feedback mechanisms to identify performance gaps, set improvement goals, and 

improve results. Job Corps agreed that three of its metrics did not have reported results 

and established targets. Job Corps did not agree that it needed to establish a target for 

the fourth metric – average length of stay – even though Job Corps told us it has taken 

steps in recent years to increase student length of stay in the program, as this leads to 

more positive outcomes for students. Specifically, the longer a student stays in the 

program, the more likely students will obtain a HSD or GED certificate, or complete their 

vocational training. As such, we affirm our conclusion regarding the need for 

performance targets to improve performance for this metric. See Exhibit 1 for details. 

Performance Results for 19 of 25 WIA Metrics Were Not Published and Publicly 

Available as Job Corps Asserted 

In 2009, our work showed Job Corps did not annually report the related performance 

results for each Job Corps center to Congress as stipulated by WIA. While Job Corps 

acknowledged the results were not submitted to Congress, Job Corps asserted that all 

of the required information on the program’s performance and outcomes were published 

and publicly available. In addition to reviewing the key attributes of successful 

performance measures, we reviewed each of Job Corps’ 25 WIA mandated 

performance metrics to determine whether the corresponding results were published 

and publicly available as asserted. 

We found 19 of 25 WIA mandated performance results were not published or publicly 

available. See Exhibit 1 for details. For example, special data runs were required to 

extract the results for the WIA mandated performance metrics of number of graduates 

who entered unsubsidized employment and were retained in the unsubsidized 

employment 6 months and 12 months after the first day of enrollment. These results 

were not published or publicly available, but were embedded in other metrics designed 

to measure 6 and 12 month retention in all placements including post-secondary 

education and training. Job Corps partially agreed that 19 metrics were not publicly 

reported as asserted in response to our prior audit report because some of the required 

performance data was rolled up into other reported performance metrics. In its 

response, Job Corps told us it is developing an annual report for submission to the 

Congress that will contain all of the required WIA performance results for PY 2010. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

17 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Long-Standing Concerns Persist Despite Spending $1.46 Million on Consultants and an 

Advisory Committee 

Concerns over Job Corps’ metrics persist because Job Corps has not placed significant 

emphasis on addressing the known deficiencies our audit confirmed and implementing 

the necessary improvements so that its metrics effectively assess the program’s 

performance. For example, Job Corps did not accept its contractor’s recommendations 

to develop a cost efficiency metric that reflected cost per successful outcome from a 

report issued in December 2008 because it ultimately found their findings to be 

inappropriate or too costly to implement. Had Job Corps acted on the consultant’s 

recommendations, some of our concerns would have been addressed. Since 2007, 

DOL has spent $1.46 million on consultants and an advisory committee to improve its 

performance metrics and outcomes, as well as other aspects of the program. In 2010, 

DOL awarded contracts to more consultants to evaluate the program, including its 

performance measurement system, at a cost of $748,846.28 See Exhibit 2 for details. 

Job Corps disagreed with our conclusion and stated that the work conducted by these 

entities is beneficial, and worth paying for, even if the final recommendations are not 

always implemented. We believe the scope of work contracted for and conducted by 

consultants should be managed appropriately to maximize value, ensure such 

investments work as planned, and result in meaningful improvements. 

In total, we identified $61.18 million in funds put to better use if Job Corps improves 

oversight of its service providers to increase the number of students who find 

employment that relate to and utilize the vocational training received. This includes 

$1.65 million Job Corps would spend to more effectively place students in jobs that 

relate to the vocational training received and $59.53 million Job Corps would spend on 

students obtaining jobs as a result of the recruiting, training, and placement efforts 

provided by Job Corps. 

ETA Response to Draft Report 

In response to our draft report, ETA stated it will take responsibility for the job training 

matches found to be invalid and will assess liquidated damages to the responsible 

contractors. However, ETA incorrectly concluded that the majority of our concerns were 

related to Job Corps’ PY 2009 crosswalk and were resolved by its October 2010 

crosswalk. As indicated in our report, our testing identified problems with the matches 

considered acceptable by Job Corps in the crosswalks for both periods. To determine 

whether Job Corps had taken the necessary actions to address prior deficiencies, we 

reviewed Job Corps outcomes for PY 2009 and the month of October 2010 against the 

applicable crosswalks and policies. We found the exception rates for both PY 2009 

(18.1 percent) and October 2010 (18.1 percent) remained unchanged. For example, Job 

Corps’ PY 2009 and revised October 2010 crosswalks continued to allow matches such 

as hotel/motel service industry students placed as ticket sellers at movie theaters that 

were questioned 15 years ago by GAO. 

28During the time of the audit, DOL paid $374,423 of these contract awards. This amount was included in the 

calculation of $1.46 million spent since 2007. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

18 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

ETA disagreed its job training matches were overstated because of military (3.9 

percent) and educational (21.2 percent) placements. With regards to military 

placements, ETA indicated it had a process to identify suitable positions in the military, 

prior to a graduate’s enlistment that included the use of a “clock hours” form that was 

developed by ETA and the military to validate Job Corps training. While we 

acknowledge this process can be used to help identify suitable positions in the military, 

this process did not include verifying whether students were actually placed in positions 

and performed duties that related to their vocational training after they joined the 

military. In addition, ETA continued to assert that placements in non-jobs, such as 

educational and training enrollments, should be included in its job training match results 

because it was recommended by the Secretary’s Advisory Committee on Job Corps. 

ETA did not agree that placements in jobs that required little or no training was a poor 

use of resources. ETA again cited the limitations of the O*NET classifications used by 

Job Corps, the many barriers Job Corps students face, and the economic recession. As 

previously noted, we acknowledged the limitations of the O*NET classifications and 

significant barriers many students face and adjusted our testing accordingly. ETA also 

indicated that it did not state that “achieving and maintaining a basic entry level job at 

the end of the Job Corps experience is a crowning achievement for most students.” This 

statement was provided in writing to us in response to our discussion draft report. 

This report includes a revised estimate of funds put to better use for students placed in 

jobs that required little or no training, as recommended by Job Corps. The estimate is 

based on the liquidated damages amount of ($750) specified in Job Corps’ PRH for 

invalid placements. While we do not consider these placements to be invalid, we agreed 

to use this lower amount to calculate the funds put to better use. 

ETA also incorrectly stated that we had concluded in Exhibit 1 that metrics 49 and 50 

were unpublished and not publicly available even though these metrics can be found on 

Job Corps’ public website. We did not test or conclude on whether metrics 26 through 

58 were published or publicly available [refer to Exhibit 1, footnote b.] 

With regard to cost efficiency, ETA asserted that we incorrectly reported: 1) Job Corps 

uses a cost efficiency metric, 2) it is entitled “cost per participant,” and 3) it is in line with 

ETA’s efficiency measure guidance for reporting cost efficiency. We disagree. Job 

Corps represented in correspondence to us that it reports its annual cost per participant 

to Congress as the average cost per new enrollee and that this metric was in line with 

ETA’s efficiency measure guidance as outlined in ETA Training and Guidance Letter 15-

03.29 In addition, Job Corps used cost efficiency, cost per participant, and cost per new 

enrollee interchangeably in Job Corps’ Congressional Budget Justifications for 2009-

2011. Further, Job Corps acknowledged its cost efficiency metric was inadequate in 

response to OMB’s assessment in 2007 and contracted with a technical expert to 

propose modifications to its cost efficiency metric. To clarify, we replaced the term “new 

participant” used in the report with “new enrollee.” We acknowledge and have cited the 

29ETA Training and Employment Guidance Letter No. 15-03, page 14 (December 10, 2003). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

19 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

challenges of developing efficiency metrics. The development and implementation of 

alternate efficiency metrics is a management decision. However, our work confirmed 

that use of efficiency metrics that reflect outcomes and total program costs could 

provide decision-makers with information to better measure and manage the program’s 

performance and costs. 

ETA did not agree the work conducted by its consultants and the Secretary’s Advisory 

Committee on Job Corps should have been managed more appropriately. ETA did not 

provide any new information to change our conclusions. As such, we affirm our 

conclusion that the work contracted for and conducted by consultants should have been 

managed more appropriately. 

With the exception of the revision related the calculation of funds put better use, ETA 

did not provide any new information that would change the conclusions stated in the 

report. 

RECOMMENDATIONS 

We recommend that the Assistant Secretary for Employment and Training require Job 

Corps to: 

1. Review and improve its performance metrics to provide decision-makers with useful 

and reliable information to make informed decisions regarding the program’s 

performance and costs. This includes ensuring metrics are complete and accurate, 

comply with WIA, and have reported results and established targets. 

2. Improve oversight of its service providers to increase the number of students who 

find employment that relate to and utilize the vocational training received. 

3. Develop a process to ensure the scope of work contracted for and conducted by 

consultants is managed appropriately to maximize value, ensure such investments 

work as planned, and result in meaningful improvements. 

We appreciate the cooperation and courtesies that Job Corps personnel extended to 

the Office of Inspector General during this audit. OIG personnel who made major 

contributions to this report are listed in Appendix [F]. 

Elliot P. Lewis 

Assistant Inspector General 

for Audit 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

20 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Exhibits 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

21 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

22 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Exhibit 1 

Analysis of Job Corps’ 58 Performance Metrics 

Job Corps relied on 58 performance measures to address the program’s multiple 

performance reporting requirements mandated by WIA, the Common Measures, the 

GPRA, OMB, DOL, and Job Corps. Exhibit 1 shows the extent to which these measures 

met the following attributes: 

	 Linkage to Goals. Metric was aligned with agency and program mission and 

goals. 

	 Complete and Accurate Results. Metric, by design, was consistent with 

mandated or internal performance measure requirements and produced the 

appropriate results. 

	 Target and Reported Results. Metric had a specific target and reported results 

that were quantifiable and measurable to fully evaluate progress toward expected 

performance. 

Of the 58 metrics, 25 were required by WIA. We tested these 25 WIA-required metrics 

to determine whether the performance results were published and publicly available as 

asserted by Job Corps in response to our prior work. 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported 

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

1. Number of graduates X X X X 

and the rate of such 

graduation, analyzed by 

type of vocational training 

received through the Job 

Corps program and by 

whether the vocational 

training was provided by 

a local or national service 

provider [WIA § 

159(c)(1)A]. 

2. Number of graduates X X Job Corps’ metric, by 

who entered design, incorrectly 

unsubsidized included jobs that did 

employment related to not or poorly related 

the vocational training to the training 

received through the Job received, as well as 

Corps program and by military enlistments, 

whether the placement in non-jobs (education 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

23 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Complete Target Published 

Linkage and and and 

Performance Metric to Accurate Reported Publicly Description of 

[Source] Goals Results Resultsa Availableb Audit Deficiencies 

the employment was 

conducted by a local or 

national service provider 

[WIA § 159(c)(1)B]. 

and training), and 

volunteer service, 

without regard to the 

students’ training and 

assigned duties. 

Results were not 

published and 

publicly available as 

asserted. 

3. Number of graduates 

who entered 

unsubsidized 

employment not related 

to the vocational training 

received through the Job 

Corps program and by 

whether the placement in 

the employment was 

conducted by a local or 

national service provider 

[WIA § 159(c)(1)B]. 

X X This metric is 

inaccurate and 

incomplete by default 

for the inclusion of job 

placements which do 

not or poorly relate to 

training in No. 2 

described above. 

Results were not 

published and 

publicly available as 

asserted. 

4. Average wage 

received by graduates 

who entered 

unsubsidized 

employment related to 

the vocational training 

received through the Job 

Corps program [WIA § 

159(c)(1)C]. 

X X Job Corps metric, by 

design, incorrectly 

included jobs that did 

not or poorly related 

to the training 

received, as well as 

military enlistments 

without regard to the 

students’ training and 

assigned duties. 

Results were not 

published and 

publicly available as 

asserted. 

5. Average wage 

received by graduates 

who entered 

unsubsidized 

employment not related 

to the vocational training 

received through the Job 

Corps program [WIA § 

X X This metric is 

inaccurate and 

incomplete by default 

for the inclusion of job 

placements which do 

not or poorly relate to 

training in No. 4 

described above. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

24 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

159(c)(1)C]. Results were not 

published and 

publicly available as 

asserted. 

6. Average wage 

received by graduates 

placed in unsubsidized 

employment after 

completion of the Job 

Corps program on the 

first day of the 

employment analyzed by 

type of vocational training 

received through the Job 

Corps program [WIA § 

159(c)(1)D(i)]. 

X X X X 

7. Average wage 

received by graduates 

placed in unsubsidized 

employment after 

completion of the Job 

Corps program 6 months 

after the first day of the 

employment analyzed by 

type of vocational training 

received through the Job 

Corps program [WIA § 

159(c)(1)D(ii)]. 

X X X X 

8. Average wage 

received by graduates 

placed in unsubsidized 

employment after 

completion of the Job 

Corps program 12 

months after the first day 

of the employment, 

analyzed by type of 

vocational training 

received through the Job 

Corps program [WIA § 

159(c)(1)D(iii)]. 

X X Metric did not have a 

result or an 

established target. 

Results were not 

published and 

publicly available as 

asserted. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

25 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

9. Number of graduates 

who entered 

unsubsidized 

employment and were 

retained in the 

unsubsidized 

employment 6 months 

after the first day of the 

employment [WIA § 

159(c)(1)E(i)]. 

X X X Results were not 

published and 

publicly available as 

asserted, but were 

obtained through a 

special data run. 

10. Number of graduates 

who entered 

unsubsidized 

employment and were 

retained in the 

unsubsidized 

employment 12 months 

after the first day of the 

employment [WIA § 

159(c)(1)E(ii)]. 

X X X Results were not 

published and 

publicly available as 

asserted, but were 

obtained through a 

special data run. 

11. Number of graduates 

who entered 

unsubsidized 

employment for 32 hours 

per week or more [WIA § 

159(c)(1)F(i)]. 

X X X X 

12. Number of graduates 

who entered 

unsubsidized 

employment for not less 

than 20 but less than 32 

hours per week [WIA § 

159(c)(1)F(ii)]. 

X X X Results were not 

published and 

publicly available as 

asserted, but were 

obtained through a 

special data run. 

13. Number of graduates 

who entered 

unsubsidized 

employment for less than 

20 hours per week [WIA 

§ 159(c)(1)F(iii)]. 

X X Metric did not have 

result or an 

established target. 

Results were not 

published and 

publicly available as 

asserted. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

26 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

14. Number of graduates 

who entered 

postsecondary education 

or advanced training 

programs, including 

apprenticeship programs, 

as appropriate [WIA § 

159(c)(1)G]. 

X 

15. Number of graduates 

who attained job 

readiness and 

employment skills [WIA § 

159(c)(1)H]. 

X 

Complete 

and 

Accurate 

Results 

X 

X 

Target 

and 

Reported

Resultsa 

X 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

Results were not 

published and 

publicly available as 

asserted, but were 

obtained through a 

special data run. 

Metric did not have a 

result or an 

established target. 

Results were not 

published and 

publicly available as 

asserted. 

16. Number of enrollees 

retained in the Job Corps 

program for 30 days after 

initial placement in the 

program [WIA § 

159(c)(2)]. 

17. Number of enrollees 

retained in the Job Corps 

program for 60 days after 

initial placement in the 

program [WIA § 

159(c)(2)]. 

18. Number of enrollees 

served [WIA § 159(d)(1)]. 

19. Average level of 

learning gains for 

graduates and former 

enrollees [WIA § 

159(d)(2)]. 

20. Number of former 

enrollees and graduates 

who entered the armed 

forces [WIA § 159(d)(3)]. 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Results were not 

published and 

publicly available as 

asserted, but were 

included in an internal 

management report. 

Results were not 

published and 

publicly available as 

asserted, but were 

included in an internal 

management report. 

Results were not 

published and 

publicly available as 

asserted, but were 

included in an internal 

management report. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

27 Report No. 26-11-004-03-370 
 

 

 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

21. Number of former 

enrollees who entered 

postsecondary education 

[WIA § 159(d)(4)]. 

X X X Results were not 

published and 

publicly available as 

asserted, but were 

obtained through a 

special data run. 

22. Number of former 

enrollees who entered 

unsubsidized 

employment related to 

the vocational training 

received through the Job 

Corps program [WIA § 

159(d)(5)]. 

X X Metric, by design, 

does not provide 

accurate and 

complete results 

because former 

enrollees have not 

completed vocational 

training. Results were 

not published and 

publicly available as 

asserted. 

23. Number of former 

enrollees who entered 

unsubsidized 

employment not related 

to the vocational training 

received through the Job 

Corps program [WIA § 

159(d)(5)]. 

X X Metric, by design, 

does not provide 

accurate and 

complete results 

because former 

enrollees have not 

completed vocational 

training. Results were 

not published and 

publicly available as 

asserted. 

24. Number of former 

enrollees and graduates 

who obtained a 

secondary school 

diploma or its recognized 

equivalent [WIA § 

159(d)(6)]. 

X X Metric, by design, 

incorrectly includes 

former enrollees who 

have not attained 

their HSD or GED 

certificate through 

Job Corps. Results 

were not published 

and publicly available 

as asserted. 

25. Number and 

percentage of dropouts 

from the Job Corps 

program including the 

number dismissed under 

X X X Results were not 

published and 

publicly available as 

asserted, but were 

included in an internal 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

28 Report No. 26-11-004-03-370 
 

 

 

 

 

 

 

 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

the zero tolerance policy 

described in section 

management report. 

152(b) [WIA § 159(d)(7)]. 

26. Student attainment of X X X 

a degree, GED, or 

certificate rate [Common 

Measure]. 

27. Student placement in 

employment or education 

[Common Measure]. 

X X X 

28. Percentage of 

students who achieve 

X X X 

literacy or numeracy 

gains of one adult basic 

education level [Common 

Measure]. 

29. Cost per new enrollee 

[Common Measure]. 

X X Inaccurate and 

incomplete for not 

reflecting outcomes, 

excluding program 

administrative 

appropriations, and 

using appropriated 

funds in place of 

actual costs. 

30. Percent of Job Corps 

students who attain 

X X X 

industry recognized 

credentials [Job Corps’ 

PRH Appendix 501(a)].c 

31. Average wage of 

participants at initial 

placement [FY 2010 

Performance Agreement]. 

X X X 

32. CTT completion rate 

[PRH Appendix 501(a)]. 

X X X 

33. CTT completer 

placement rate [PRH 

Appendix 501(d)]. 

X X X 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

29 Report No. 26-11-004-03-370 
 

 

 

 

 

 

 

 

 

 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

34. CTT completer 

average hourly wage at 

placement [PRH 

Appendix 501(d)]. 

X X X 

35. CTT completer fulltime job placement rate 

[PRH Appendix 501(d)].c 

X X X 

36. CTT completer 

industry recognized 

credential attainment rate 

X X X 

[PRH Appendix 501(d)].c 

37. CTT completer job 

training match average 

wage [PRH Appendix 

501(d)]. 

X X X 

38. CTT completer job 

training match/post 

secondary education 

placement rate [PRH 

Appendix 501(c)]. 

X X Job Corps’ metric, by 

design, incorrectly 

included jobs that did 

not or poorly related 

to the training 

received, as well as 

military enlistments 

and volunteer service, 

without regard to the 

students’ training and 

assigned duties. 

39. CTT completer 6 

month follow-up 

placement rate [PRH 

Appendix 501(d)]. 

X X X 

40. CTT completer 12 

month follow-up 

placement rate [PRH 

Appendix 501(d)]. 

X X X 

41. CTT completer 6 

month follow-up average 

weekly wage of 

placements [PRH 

Appendix 501(d)]. 

X X X 

42. Combination X X X 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

30 Report No. 26-11-004-03-370 
 

 

 

 

 

 

 

 

 

 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

HSD/GED/CTT 

attainment rate [PRH 

Appendix 501(a)]. c 

43. Female arrival rate 

[PRH Appendix 501(b)]. 

X X X 

44. Total arrival rate 

[PRH Appendix 501(b)]. 

X X X 

45. Arrivals with Level 1 

ZT non-separation rate 

[PRH Appendix 501(b)]. 

X X X 

46. Arrivals with 90 day 

commitment [PRH 

Appendix 501(b)]. 

X X X 

47. Graduate rate [PRH 

Appendix 501(b)]. c 

X X X 

48. Graduate initial 

placement rate [PRH 

Appendix 501(a)]. 

X X X 

49. Student on-board 

strength [PRH Appendix 

501(a)]. 

X X X 

50. Average length of 

stay [Job Corps Quarterly 

Highlights Report]. 

X X Metric did not have 

an established target. 

While the metric is 

not required to have 

an established target, 

OIG analysis found 

having a reported 

result and established 

target is a key 

attribute of successful 

performance 

measures. 

51. Center quality rating 

[PRH Appendix 501(a)]. 

X X X 

52. Former enrollee initial 

placement [PRH 

Appendix 501(c)].c 

X X X 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

31 Report No. 26-11-004-03-370 
 

 

 

 

 

 

 

U. S. Department of Labor – Office of Inspector General 

Performance Metric 

[Source] 

Linkage 

to 

Goals 

Complete 

and 

Accurate 

Results 

Target 

and 

Reported

Resultsa 

Published 

and 

Publicly 

Availableb 

Description of 

Audit Deficiencies 

53. Average literacy gain 

[PRH Appendix 501(a)]. 

X X X 

54. Average numeracy 

gain [PRH Appendix 

501(a)]. 

X X X 

55. HSD / GED 

attainment rate [PRH 

Appendix 501(a)]. 

X X X 

56. CTT completion rate 

[PRH Appendix 501(d)]. 

X X X 

57. Graduates of green 

training programs [FY 

2010 Performance 

Agreement]. 

X X X 

58. Post-enrollment 

placement rate [PRH 

Appendix 501(a)]. 

X X X 

Totals 58 49 54 6 

Summary Analysis 

 58 of 58 metrics linked to goals 

 9 of 58 metrics did not have complete and accurate results. See metric nos. 2, 3, 4, 5, 22, 

23, 24, 29, and 38. Five of these nine metrics related to job training match (four) [nos. 2, 4, 

22, and 38] and cost-efficiency (one) [no. 29]. 

 4 of 58 Metrics did not have reported results and/or established targets. See metric nos. 8, 

13, 15, and 50. 

 19 of 25 WIA metrics did not have results published and publicly available. See metric nos. 

2, 3, 4, 5, 8, 9, 10, 12, 13, 14, 15, 16, 17, 20, 21, 22, 23, 24, and 25. 

 In aggregate, 22 of 58 metrics had reporting deficiencies. See nos. 2, 3, 4, 5, 8, 9, 10, 12, 

13, 14, 15, 16, 17, 20, 21, 22, 23, 24, 25, 29, 38, and 50. 

 10 of 58 metrics had multiple reporting deficiencies. See metric nos. 2, 3, 4, 5, 8, 13, 15, 22, 

23, and 24. 

Source: OIG analysis of Job Corps’ performance metrics. 

X Indicates metric met the attribute. 

a

WIA required Job Corps establish 17 core performance metrics (see nos. 1-17) and collect data on 8 additional 

performance areas (see nos. 18-25). Since WIA did not require metrics for these additional performance areas, we
 

did not determine whether quantifiable targets were established. Source: WIA §159(C)(3)(d). 

b

This attribute is only applicable for the 25 WIA metrics (see metrics 1 – 25). 

c

Job Corps started reporting this metric in PY 10.
 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

32 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

 Exhibit 2 

Job Corps Expenses for Prior and On-going Performance Measurement 

Evaluations 

Since 2007, DOL has spent $1.46 million on consultants and an advisory committee to 

improve its performance metrics and outcomes. In 2010, DOL awarded contracts to 

more consultants to evaluate the program’s performance measurement system at a cost 

of $748,846.b See details below. 

Consultant Scope of Work Cost 

Advisory Committee on In August 2007, the Secretary of Labor formed an $630,994 

Job Corps advisory committee to provide advice and 

recommendations related to policies, legislation, and 

regulations related to Job Corps. This included 

evaluating the program’s efficiency and performance 

measures. The committee included representatives 

from Job Corps contractors, Job Corps staff, organized 

labor, the Departments of Education and Health and 

Human Services, and the Deputy Secretaries of Labor, 

Agriculture, and Interior. 

HeiTech Services, Inc. Contract was awarded in 2007 to identify a cost 

efficiency metric based on successful program 

outcomes to measure cost and quality. 450,000a 

HeiTech Services, Inc. Contract was awarded in 2007 to perform a 

comprehensive and objective review of Job Corps 

performance management system, including extensive 

analysis of data and reports generated through the 

Mathematica Policy 

outcome management system. 

Contract was awarded in 2010 to determine the 374,423b 

Research, Inc. association between center-level performance and 

center-level impacts, when accounting for participant 

characteristics, using previously collected performance 

data. 

Impaq International, Contract was awarded in 2010 to determine the factors 

LLC that contribute to or impede the successful operation of 

the program. 

Total Costs $1,455,417 

Source: OIG analysis of Job Corps contracts and expenses. 

a The award amount for both contracts totaled $450,000. 

b

The award amount for both the Mathematica Policy Research, Inc. contract ($49,946) and Impaq International, LLC 

($698,900) totaled $748,846. During the time of the audit, DOL paid $374,423 of these contract awards. This amount 

was included in the calculation of $1.46 million in funds spent since 2007. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

33 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

34 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Appendices 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

35 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

36 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Appendix A 

Background 

DOL, ETA oversees Job Corps, which administers and manages the Job Corps 

program. This program’s mission is to help at-risk youth become more employable, 

responsible, and productive citizens. The program annually provides education, 

vocational training, and support services to approximately 60,000 students at 124 

centers, residential and non-residential, located nationwide. Job Corps’ budget for PY 

2009 and 2010 totaled approximately $3.39 billion ($1.68 billion for PY 2009 and $1.71 

billion for PY 2010). 

Job Corps established and uses a complex performance management system to meet 

mandated and internal accountability priorities and assess the program’s effectiveness. 

The system is comprised of 58 performance metrics developed to address the 

program’s multiple performance reporting areas required by GPRA, WIA, President’s 

Management Agenda (PMA), PART, the Common Measures, DOL, and Job Corps. See 

Exhibit 1 for details. These various requirements and corresponding metrics provide the 

basis for assessing the program’s performance and effectiveness, and establish 

accountability for achieving results, such as placements in employment or postsecondary education or training; cost efficiency; and attainment of a HSD or GED 

certificate. Many of the results, which are shown in four report cards, are used to assess 

the performance of Job Corps center operators and contracted service providers and 

impact decisions about contract awards and renewals, incentive fees, and performance 

bonuses: 1) outreach and admissions, 2) center, 3) career technical training, and 4) 

career transition services. Job Corps intentionally distributes responsibility and 

accountability across the various contracted service providers involved, as a means to 

drive performance in meeting national reporting measures and holding these providers 

accountable for their contractual obligations. Incentive fee and performance bonus 

amounts are determined based on center operator and service providers’ aggregate 

performance for each of the weighted measures on a report card exceeding 

performance levels established by Job Corps. 

In 1993, Congress passed GPRA to improve federal government management and 

emphasize the achievement of federal program results rather than activities and 

processes. GPRA required the development of performance plans with annual goals 

and measures, as well as performance reports on prior year performance. These 

performance plans included results-oriented annual goals linked to program activities 

and indicators the agencies used to measure performance against the results-oriented 

goals. Enacted after GPRA, the 1998 WIA increased focus on accountability and 

contained core indicators of performance for Job Corps concerning recruitment, 

education and placement rates, wages, and long-term outcomes of graduates after 

initial placement. The PMA, initiated in 2001, and the PART, created in 2002, 

emphasized specific requirements to develop program-level efficiency measures and 

show annual improvements in efficiency. Under the PMA, Job Corps began the process 

of moving toward reporting with one set of definitions, measures, and procedures 

(Common Measures). Specifically, these measures included placement in employment 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

37 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

or higher education, attainment of a degree or certificate, literacy and numeracy gains, 

and cost efficiency. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

38 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

 Appendix B 

Objectives, Scope, Methodology, and Criteria 

Objective 

Our audit objective was to answer the following question: 

To what extent does Job Corps have metrics in place to assess the program’s 

performance? 

Scope 

This report reflects the audit work that was generally conducted at Job Corps 

headquarters in Washington, D.C. Our scope covered Job Corps’ performance metrics 

and outcomes for PY 2009 and the month of October 2010, and policies, procedures 

and processes through March 7, 2011. Job Corps revised its job training match 

crosswalk – criteria used to determine training-related placements – in October 2010. 

We covered October 2010 to account for the impact on the related metrics and 

outcomes. 

The relationship between the populations and the testing performed and the kinds and 

sources of evidence are fully described in the Methodology section below. 

We conducted this performance audit in accordance with generally accepted 

government auditing standards. Those standards require that we plan and perform the 

audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our 

findings and conclusions based on our audit objectives. We believe that the evidence 

obtained provides a reasonable basis for our findings and conclusions based on our 

audit objectives. 

Methodology 

To accomplish our audit objective, we interviewed Job Corps management and staff at 

Job Corps headquarters in Washington, D.C. and reviewed applicable laws, regulations, 

and policies, including GPRA, WIA, PMA, PART, Common Measures, the PRH, and 

Job Corps’ performance agreements, to gain an understanding of Job Corps’ 

requirements and processes for measuring and reporting program performance. We 

also reviewed DOL’s strategic plan (2011-2016), financial report (2010), performance 

report (2010), and performance and accountability report (2009). Based on our 

understanding, we identified 58 performance metrics that were required and/or used by 

Job Corps during the scope of our audit. See Exhibit 1. 

In addition, we reviewed GAO guidance and reports regarding government agencies’ 

performance metrics and consulted with GAO management and senior analysts to gain 

GAO’s perspective on performance measurement, including the key attributes for 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

39 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

successfully measuring performance. GAO’s work identified a number of key attributes. 

We selected three of these attributes to test. 

Table A: Key Attributes of Successful Performance Measures 

Attribute Definition 

Potentially adverse 

consequences of not 

meeting attribute 

1. Linkage to goals Measure is aligned with agency and program Behaviors and incentives 

goals and mission. created by measure do not 

support achieving agency or 

program goals or mission. 

2. Complete and Metric, by design, was consistent with Decision makers lack 

Accurate Results mandated or internal performance measure reliable and consistent data 

requirements and produced the appropriate to effectively measure and 

results. manage performance. 

3. Target and Metric had a specific target and reported Decision makers lack 

Reported Results. results that were quantifiable and measurable essential feedback 

to fully evaluate progress toward expected mechanisms to identify 

performance. performance gaps, set 

improvement goals and set 

results. 

Source: OIG analysis of GAO’s key attributes for successfully measuring performance. 

We also reviewed prior OIG and GAO audit reports, evaluations conducted by DOL 

consultants and Job Corps staff, and OMB’s PART assessment of Job Corps for prior 

deficiencies; interviewed Job Corps, ETA, DOL Office of Policy Development and 

Research officials, and OMB staff; and analyzed Job Corps data to identify relevant 

concerns. We found long-standing concerns about the reliability of the metrics covering 

two key performance areas – job training match, which is used to measure how 

effectively participants are placed in jobs that match the training they received in Job 

Corps, and cost efficiency, which is used to measure program costs. As a result, we 

assessed corrective actions and determined the impact on the related performance 

metrics. 

Of the 58 metrics, 25 were required by WIA. We tested these 25 WIA-required metrics 

to determine whether the performance results were published and publicly available as 

asserted by Job Corps in response to our March 2009 audit report. 

At Job Corps headquarters, we identified and evaluated Job Corps internal controls 

over its performance metric system. We assessed risks related to inaccurate results and 

evaluated the corresponding control environment. We assessed the reliability of related 

data for the applicable audit period and determined the data was sufficiently reliable to 

accomplish our audit objective. We used a statistical sampling approach to select 

certain metric results. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

40 Report No. 26-11-004-03-370 
 

 

U. S. Department of Labor – Office of Inspector General 

Attribute Testing 

We evaluated each of Job Corps 58 performance metrics to determine whether they 

met the following attributes: linkage, complete and accurate results, and measurable 

target and results. 

	 Linkage. We reviewed for linkage between each metric and Job Corps and 

DOL’s goals and mission. 

	 Complete and accurate results. We reviewed each metric to determine 

whether the metric was designed to collect data or calculate the results in 

accordance (or consistently) with mandated or internal performance measure 

requirements. If the metric did not capture these results, we concluded the 

metric’s results were incomplete and/or inaccurate. For example, WIA required 

Job Corps to measure how effectively the program placed students in jobs that 

related to the training received. If the metric, by design, did not capture this 

information or the results contained information for unrelated jobs or non-jobs, we 

considered the results incomplete and inaccurate. 

	 Target and reported results. Where appropriate, we determined whether each 

metric had quantifiable targets and reported results. Some of the WIA required 

performance metrics did not require targets. We considered instances where 

multiple performance metrics were embedded in a single metric acceptable, if the 

required results could be appropriately extracted. We also considered results 

generated through special data runs that were not available in published reports 

acceptable. 

Job Training Match 

WIA required Job Corps to measure and report how effectively participants are placed 

in jobs that match the vocational training they received in the program. In 1997, GAO 

reported the program’s job training match metrics were flawed because performance 

results included jobs that (1) did not relate, or poorly related, to the training received or 

(2) required relatively little training. In 1998, GAO also questioned Job Corps’ policy for 

counting any participant that enlisted in the military as a successful job training match, 

regardless of the student’s vocational training and assigned duties. 

We found the 2009 and 2010 crosswalks continued to include jobs that (1) did not 

relate, or poorly related, to the training received and/or (2) required relatively little 

training to perform. The crosswalks for both periods also continued to provide automatic 

job training matches for all military enlistments, as well as enrollments in postsecondary education or training and volunteer service with AmeriCorps without regard 

to the vocational training received or the actual duties performed. As a result, we 

performed additional audit procedures to determine the impact of these deficiencies on 

Job Corps’ job training match outcomes for PY 2009 and October 2010. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

41 Report No. 26-11-004-03-370 
 

U. S. Department of Labor – Office of Inspector General 

According to Job Corps’ records (MPO-36 reports, internally generated management 

reports), Job Corps reported an aggregate of 17,787 job training matches for PY 2009 

(16,234) and October 2010 (1,553). We obtained the databases Job Corps provided to 

support these matches, which included each student’s vocational training, placement 

employer, placement position title, and O*NET-SOC code used to make placement. To 

address these long-standing concerns, we performed the following steps: 

	 Reviewed the supporting databases to determine whether vocational training 

appropriately related to each of the 17,787 job placements in accordance with 

WIA, based on the student’s place of employment, job position, and O*NET-SOC 

code, regardless of whether the matches were considered acceptable by Job 

Corps’ crosswalks. For example, we considered nursing assistant students 

employed as medical assistants, home health aides, or registered nurses 

appropriate job training matches. However, we questioned the validity of matches 

such as nursing assistant students placed as telephone operators, tax preparers, 

and transportation ticket agents, which were considered acceptable matches in 

the 2009 crosswalk. This review also included placements in the military, postsecondary education and training, and volunteer service in AmeriCorps. 

	 Reviewed the crosswalks for both periods and found 42 occupations that O*NETSOC identified as needing little or no previous work-related skill, knowledge, or 

experience to perform. Given the significant academic, employability, and 

vocational barriers many Job Corps students face, we reduced these 42 

occupations to 15 occupations we determined needed the least amount of 

training to perform. These 15 occupations are detailed in Table 3. 

In aggregate, we initially identified 2,380 job placements that did not relate or poorly 

related to training received; 4,291 placements in the military, post-secondary education 

and training, and volunteer service in AmeriCorps; and 1,468 placements in jobs 

needing little or no previous work-related skill, knowledge, or experience to perform. To 

reduce the number of overlapping results, our initial review did not include any jobs 

needing little or no previous work-related skill, knowledge, or experience that also did 

not relate or poorly related to the training received in our calculation of 2,380 job 

placements. For example, if a construction student was employed at a fast food 

restaurant or as a parking lot attendant, we only accounted for this deficiency as an 

exception for jobs needing little or no previous work-related skill, knowledge, or 

experience (1,468). 

To address concerns raised by Job Corps that our database testing was not sufficient to 

support some of these conclusions, we subsequently selected statistical, random 

samples of 247 of 2,380 job placements that did not relate or poorly related to training 

received and 276 of 1,468 placements in jobs needing little or no previous work-related 

skill, knowledge, or experience and performed additional audit procedures to verify the 

reliability of the data used to make these initial determinations. In response, Job Corps 

provided required and supplemental documentation to substantiate the vocational 

training the students received matched these job placements. Detailed documentation 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

42 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

reviewed included Job Corps’ placement assistance records used by contracted CTS 

(i.e., placement) providers to record placement information including employer, job title, 

and wage; third-party employment verification forms completed by employers at the 

request of placement contractors; placement provider case notes; and other information 

(e.g., written correspondence, job descriptions, job postings, and resumes), when 

available. In aggregate, we found that the databases used to support our initial 

determinations for our sample of 490 of 523 (247 + 276) job placements, or 94 percent, 

were sufficiently reliable. We adjusted our results to account for the 33 job placements 

that were considered valid. 

Subsequent to our sample testing, we accounted for job placements that had 

overlapping exceptions in both exception categories. Our sample testing also resulted in 

a reassessment of a number of exceptions. For example, we initially considered an 

office administration student placed as cashier as exception for placements that did not 

relate or poorly related to the training received. However, if the additional 

documentation indicated the student was placed as a cashier at a fast food restaurant, 

we subsequently accounted for this exception in the exception category for jobs needing 

little or no previous work-related skill, knowledge, or experience as well. 

As a result, we found, in aggregate, 3,226 job placements did not relate or poorly 

related to training received and 1,569 placements were in jobs needing little or no 

previous work-related skill, knowledge, or experience to perform. These placements 

were based upon our representative simple random samples that were estimated with 

95 percent confidence level and assuming highest variance in the populations. 

Cost Efficiency 

Since 2007, OMB, a contractor hired by Job Corps, and GAO raised concerns about the 

limitations of Job Corps use of cost per participant to measure cost efficiency and 

recommended the use of cost efficiency metrics that link performance outcomes to 

actual, total program costs rather than select appropriated costs. While Job Corps has 

taken steps to improve its job training matches and explore alternate cost efficiency 

metrics, we found these concerns persist, and Job Corps continues to use cost per 

participant to measure the program’s cost efficiency. As a result, we performed 

additional audit procedures to determine the impact of this deficiency on related 

outcomes. 

We obtained Job Corps’ appropriated funding (operation and construction funds of 

$1.66 billion and administration funds of $28.66 million), estimated new enrollments 

(62,344), and student training slots (44,455) from Job Corps’ FY 2011 Congressional 

Budget Justification. We also obtained select performance outcomes by center for PY 

2009 and October 2010 reported on Job Corps’ MPO-36 management reports, as we 

were told by management this was the most comprehensive source for data as of our 

cut off date and the data was relied upon by management, and reconciled these 

outcomes to published year-end and monthly reports. As such, we determined the data 

was sufficiently reliable to accomplish our objective. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

43 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Based upon prior recommendations and the data we had available, we computed 

alternate cost efficiency metrics that reflected the costs of successful outcomes (e.g., 

GED/HSD attainment, vocational training completion, job placement), as well as dropouts and compared the results to Job Corps’ cost per participant. 

WIA Reporting 

In response to our March 2009 report, Job Corps asserted that all of its 25 WIA 

mandated performance metrics and outcomes were published and publicly available. 

We reviewed all relevant, published Job Corps reports to determine whether the 

corresponding results were included and publicly available. We considered instances 

where multiple performance metrics were embedded in a single metric unacceptable, 

even if the required results could be appropriately extracted. We also considered results 

generated through special data runs that were not available in published reports 

unacceptable. 

Job Corps Expenses for Prior and On-going Performance Measurement Evaluations 

We requested and obtained all costs associated with prior (since 2007) and on-going 

DOL evaluations of Job Corps’ performance measurement system. 

Criteria 

We used the following criteria to perform this audit: 

 WIA, 

 Common Measures, 

 GPRA, 

 PMA 

 PART, 

 PRH, 

 DOL and Job Corps’ goals and mission, 

 O*NET-SOC taxonomy, 

 GAO, GAO/GGD-96-118 (Washington, D.C.: June 1996), and 

 GAO, GAO 03-143 (Washington, D.C.: November 2002). 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

44 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

 Appendix C 

Terminology 

Term Definition Source 

Enrollee An individual who has voluntarily applied for, WIA Section 142(3) 

been selected for, and enrolled in the Job 

Corps program, and remains with the 

program, but has not yet become a graduate. 

20 Code of Federal 

Regulations (CFR) § 

670.120 

Enrollees are also referred 

to as ‘‘students’’ in this 

part. 

Former Enrollee An individual who has voluntarily applied for, 

been selected for, and enrolled in the Job 

Corps program, but left the program before 

completing the requirements of a vocational 

training program, or receiving a secondary 

school diploma or recognized equivalent, as 

a result of participation in the Job Corps 

program. 

WIA Section 142(4) 

One who has completed 60 or more days, 

has not attained graduate status, and whose 

separation is for reasons other than a ZT 

“Level One” infraction. 

PRH Chapter 4.2 R1.b 

Graduate An individual who has voluntarily applied, 

been selected, and enrolled in the Job Corps 

program and has completed the requirements 

of a vocational training program, or received 

a secondary school diploma or recognized 

equivalent, as a result of participation in the 

Job Corps program. 

WIA Section 142(5) 

An enrollee who has: 

(1) Completed the requirements of a 

vocational training program, or received a 

secondary school diploma or its equivalent as 

a result of participating in the Job Corps 

program; and 

(2) Achieved job readiness and employment 

skills as a result of participating in the Job 

Corps program. 

20 CFR § 670.120 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

45 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Term Definition Source 

One who has completed the requirements of 

career technical training or earned a high 

school diploma or its equivalent (GED) while 

enrolled in Job Corps, or who completes 

both, and has completed 60 or more days 

enrollment. Students who have separated for 

“Level One” ZT infractions do not qualify. 

PRH Chapter 4.2 R1.a 

Participant An individual who has been determined to be 

eligible to participate in and who is receiving 

services (except follow-up services 

authorized under this title) under a program 

authorized by this title. Participation shall be 

deemed to commence on the first day, 

following determination of eligibility, on which 

the individual began receiving subsidized 

employment, training, or other services 

provided under this title. 

WIA Section 101(34) 

An individual who has registered under 20 

CFR 663.105 or 664.215 and has been 

determined to be eligible to participate in and 

who is receiving services (except for follow up 

services) under a program authorized by WIA 

title I. Participation commences on the first 

day, following determination of eligibility, on 

which the individual begins receiving core, 

intensive, training or other services provided 

under WIA title I. 

20 CFR § 660.300 

Placements Student employment, entry into the Armed 

Forces, or enrollment in other training or 

education programs following separation from 

Job Corps. 

20 CFR § 670.120 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

46 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Term Definition Source 

Educational 

Placements 

1. High school: no less than 20 hours in class 

per week for an expected duration of one 

semester or trimester or quarter; or 

2. Postsecondary career technical training or 

technical education program: no less than 20 

hours in class per week for an expected 

duration of at least 90 calendar days; or 

3. University/College: registered for no less 

than 9 credit hours per quarter or semester at 

a 2-year or 4-year university/college 

accredited by an agency recognized by the 

U.S. Department of Education or the Council 

for Higher Education Accreditation as an 

accrediting agency for higher education and 

that offers, at minimum, an associate’s 

degree; or 

4. Online University/College: registered for no 

less than 9 credit hours per quarter or 

semester at a 2-year or 4-year 

university/college accredited by an agency 

recognized by the U.S. Department of 

Education as an accrediting agency for 

distance education and that offers, at 

minimum, an associate’s degree; or 

5. On-the-job-training or other subsidized 

employment: no less than 20 hours per week; 

or 

6. Other training program: no less than 20 

hours in class per week for an expected 

duration of at least 90 calendar days. 

PRH Exhibit 4-1 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

47 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Term Definition Source 

Job Placements Full-Time: 1. 32 hours or more in one or two 

jobs in a 7-consecutive-day period; or 

2. An apprenticeship job registered by the 

Bureau of Apprenticeship and Training or a 

state Apprenticeship Council, that combines 

supervised, structured on-the-job training with 

related theoretical instruction leading to 

defined levels of skill and career 

advancement, and where the student 

receives a wage; or 

3. Armed Forces: 40 hours minimum per 

week active duty. This includes initial 

Reserve Forces and National Guard training 

but does not include weekend and summer 

training sessions. Pre-enlistments are not 

placements. The first 40- hour week must be 

completed prior to 12 months after separation 

Part-Time: 20 hours or more, but less than 

32, in one or two jobs in a 7- consecutive-day 

period. 

PRH Exhibit 4-1 

Combination Full-Time: 1. A combination of work and PRH Exhibit 4-1 

Job/College university/college: a minimum of 16 hours 

Placements work at one job in a 7- consecutive-day 

period and a minimum 6 credit hours per 

quarter or semester at a 2-year or 4-year 

university/college accredited by an agency 

recognized by the U.S. Department of 

Education or the Council for Higher 

Education Accreditation as an accrediting 

agency for higher education and that offers, 

at minimum, an associate’s degree; or 

2. A combination of work and online 

university/college: a minimum of 16 hours 

work at one job in a 7- consecutive-day 

period and a minimum 6 credit hours per 

quarter or semester at a 2-year or 4-year 

college accredited by an agency recognized 

by the U.S. Department of Education as an 

accrediting agency for distance education 

and that offers, at minimum, an associate’s 

degree. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

48 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Term Definition Source 

Part-Time: University/college enrollment for a 

minimum of 6 credit hours and a minimum of 

10 hours work at one job per week. 

School Dropout An individual who is no longer attending any 

school and who has not received a 

secondary school diploma or its recognized 

equivalent. 

WIA section 101(39) 

Separation Means the action by which an individual 

ceases to be a student in the Job Corps 

program, either voluntarily or involuntarily. 

20 CFR § 670.120 

Student An individual enrolled in the Job Corps. 20 CFR § 670.120 

Uncommitted 

Students 

One who has remained in Job Corps less 

than 60 days (regardless of achievement), or 

who has separated for a ZT “Level One” 

infraction, per Chapter 3, Exhibit 3.1 

(Infraction Levels and Appropriate Center 

Actions). 

PRH Chapter 4.2 R1.c 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

49 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Term Definition Source 

Zero Tolerance Students who exit due to Level 1 ZT PRH Appendix 501 

infractions within 30/45 days are not included 

in the Center Report Card. Students who exit 

due to Level 1 ZT infractions after 30/45 

days, however, are included in all pools for 

on-center measures and credit will be given 

for academic and CTT credentials earned 

prior to exit. 

However, since all students who exit due to 

Level 1 ZT infractions, regardless of timing 

(within or after 30/45 days), are not 

considered former enrollees or graduates, 

they are ineligible for post-center services 

and are therefore excluded from all postcenter pools in both the Center and CTS 

Report Cards. 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

50 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

 Appendix D 

Acronyms and Abbreviations 

CFR Code of Federal Regulations 

CTS Career Transition Services 

CTT Career Technical Training 

DOL Department of Labor 

ETA Employment and Training Administration 

GAO Government Accountability Office 

GED General Educational Development 

GPRA Government Performance and Results Act of 1993 

HSD High School Diploma 

Job Corps Office of Job Corps 

OIG Office of Inspector General 

OMB Office of Management and Budget 

O*NET-SOC Occupational Information Network’s Standard Occupational 

Classification System 

PART Program Assessment Rating Tool 

PMA President’s Management Agenda 

PRH Job Corps’ Policy and Requirements Handbook 

PY Program Year 

WIA Workforce Investment of 1988 

ZT Zero Tolerance 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

51 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

52 Report No. 26-11-004-03-370 
U.S. Department of Labor 

SEP 29 2011 

Assistant Secretary for 

Employment and Training 

Washington. D.C. 20210 

MEMORANDUM FOR ELLIOT P. LEWIS 

Assistant Inspector General 

Assistant 

JANE OATES 

Secretary for 

(r'- .. ~ 

Employment and Training 

DIG Audit of Job Corps Needs to Improve Reliability ofPerforrnance 

Metrics and Results 

Draft Report #26-11-004-03-370 

This memorandum responds to the subject draft audit report, dated September 23,2011 , Job 

Corps Needs to Improve Reliability ofPerfonnance Metrics and Results. We appreciate the 

opportunity to provide input to this draft audit report as well as to the recent discussion draft 

report and "statement of facts." The Employment and Training Administration (ETA), which 

manages and administers the Office of Job Corps (OIC), appreciates the Office of Inspector 

General's (OiG's) efforts to identify areas in which the program's perfonnance metrics can be 

improved. 

First, ETA is committed to maintaining a sound performance metrics system for all of its 

programs, including Job Corps. We take great pride in the work we do on behalf of youth and 

adults across the nation, and our efforts fully support the Department's priorities to achieve 

"Good Jobs for E"eryone." We recognize that performance metrics are criticaJ in reflecting both 

program results and worthwhile taxpayer investment. The Job Corps program, in particular, 

serves youth in targeted. at-risk populations. For over four decades, Job Corps has made a 

difference in young people's lives by helping them gain academic and career technical training 

credentials, complemented by placement in employment, education, the military, and 

apprenticeship, with defined career paths that will lead them to economic self-sufficiency. 

As such, ETA takes responsibility for any invalid Job Training Matches (JTMs) identified by the 

DIG, which constitute the primary finding of this report. The OJC, in coordination with the ETA 

Office of Contracts Management (OCM), will assess liquidated damages to applicable 

contractors that have misreported these placements. However, ETA believes the vast majority of 

the DIG audit report misrepresents OJC's perfonnance management system, disregards the 

changes implemented to ensure reliability in perfonnance data (as the OIG audit was conducted 

using outdated policies and procedures), and, in some areas, is inaccurate. ETA officials, 

including representatives from the Offices of Job Corps, Policy, Development and Research, and 

Workforce Investment, met with the OIG and provided information on multiple occasions to 

substantiate and clarify ETA's and OJC's performance metrics systems. 

U. S. Department of Labor – Office of Inspector General 

Appendix E 

ETA Response to Draft Report 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

53 Report No. 26-11-004-03-370 
ETA's response is divided into four sections, to align with the items addressed in the report: Job 

Training Match, Performance Metrics, Cost EfficieDcy Measures, aDd EvaluatioD. 

Job Training Match 

The OIG expressed three concerns with Job Corps' Job Training Match Metric: I) Invalid JTM 

placements; 2) military and educational placements as JTMs; and 3) the placement of Job Corps 

students in entry-level positions as a poor use of resources. 

Invalid Job Training Matches 

ETA takes responsibility for those JTMs that were found to be invalid and will assess liquidated 

damages to the responsible contractors. However, ETA would like to reiterate that we have 

taken steps to address this concern. As a result, over 90% of the placements audited by the OIG 

were matched to training using policies that are no longer in place. The time period of the 

majority of the OIG's JTM findings (PY 2009 and October 2010) covers the previous, nowdefunct JTM policy. As noted in the report, a new JTM Crosswalk was introduced to the Job 

Corps system in October 20ID, approximately 30 days before the OIG audit began. Thus, some 

of the PY 2009 JTM placements that were valid at that time would not have been misreported in 

PY 2010. The OIG was made aware of this transition at the entrance meeting. 

OJC and ETA have worked collaboratively for years to improve the crosswalk, which has had 

updated versions since 2002. ETA had already identified weaknesses with the PY 09 version 

and asked OJC and ETA's Office of Workforce lnvesUllent to better align Job Corps' crosswalk 

with the Occupational Information Network's Standard Occupational Classification System 

(O·NET-SOC). This newly-developed crosswalk substantially improves the fundamental 

linkage between training and placement in two ways: 1) through its introduction of Training 

Program Areas (TP As) to categorize both training and job activities; and 2) in its re-creation of 

the entire crosswalk "from the ground up." 

Many of the OIG's concerns regarding JTM matches have been resolved by the new crosswalk. 

Certain codes, such as "cashier," are no longer used broadly, and greater levels of detail between 

occupations are provided (i.e., carpenters are now treated separately from brick1ayers and 

painters, as opposed to being grouped into "construction''). Additionally, current policy requires 

that career transition services contractors attest to the vaJidity of each JTM credit in the 

program's electronic career transition system. This enhancement is another safeguard to ensure 

the data integrity of JTM placements. 

Finally, the methodology used to calculate the cost of $1,196 to provide each student with 

placement services is flawed. While dividing the tetaJ cost of career transition services by the 

number of students receiving career transition services may result in a per student cost, not all of 

those expenses are tied to a placement. Career transition specialists provide an array of services, 

beyond placement, that assist students in transitioning into the workforce or higher education. 

They assist in identifying housing and transportation resources, student financial aid and other 

forms of assistance, and providing employability skills training, such as job interviewing skills 

and resume building. 

2 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

54 Report No. 26-11-004-03-370 
Military and Educational Placements as Job Training Matches 

We disagree with the OIG's finding that JTMs are overstated by 25.2% because of military and 

educational placements. In our opinion, these are valid training matches. In accordance with 

recommendations from the Secretary of Labor's Advisory Committee on Job Corps, which 

submitted its final report to the Secretary in April 2008, mc began including placements in the 

post-secondary institutions as JTMs. In addition to federal staff, the Secretary's Advisory 

Committee consisted of members of industry, academia. labor unions, and non-profit 

organizations. Ultimately, the Committee recommended expanding training matches beyond 

traditional employment to incentivize placements that would lead Job Corps graduates to higher 

wage employment 

OJC worked with the military to develop a "clock hours" fonn that is accepted by all branches of 

the Anned Forces to validate Job Corps training. This process includes reviewing Job Corps' 

Training Achievement Records (TARs), which list the skills, knowledge, and industryrecognized credentials a graduate gained upon completion of a training program, to identify 

suitable positions in the military, prior to a graduate's enlistment. Although the OIG was 

provided with both the fonn and sample TARs, the OIG disagrees that these are valid matches. 

ETA also agrees with the Secretary's Advisory Committee that post-secondary placements are a 

vaJid match to the training Job Corps graduates receive. Job Corps graduates' entry into postsecondary institutions, especially community colleges, continues to rise, and should be 

incentivized. In teaching students the values oflifelong learning and long-term career 

advancement, counselors, teachers, and many other center staff emphasize the importance of 

continuing education. 

Given ETA's position on this issue, these placements are not considered an overstatement of 

JTMs. Instead, ETA and the OIG have a difference of opinion on the policy defining JTMs. 

Under current Job Corps policy, what the OIG considers "overstatements" are actually valid 

training matches. 

The Placement of Job Corps Students in Entry-Level Positions 

We disagree with the OIG's finding that the placement of students in jobs that require little or no 

training was a poor use of resources, because Job Corps was not a necessary intervention for 

these placements. This finding dismisses key relevant points and discounts the value of the skills 

achieved by our students. In fact, these students completed valid career technicaJ training 

programs and received career transition readiness and employability skills instruction. Further, 

this fmding fails to recognize: 

I. the needs and background of the typical Job Corps student in attaining and retaining 

employment are different than those of the "typical worker," which is the target audience 

for this broad O*NET definition; 

2. that entry-level positions are appropriate placements for workers first entering the 

workforce and are the initial step in a career pathway; and 

3. the adverse impact of the economic recession on the employment prospects of Job Cotps' 

demographic population. 

3 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

55 Report No. 26-11-004-03-370 
A comprehensive review of Job Corps placement must begin with a solid undentanding of the 

students served. The typical Job Corps student entering the program has not completed high 

school, reads slightly below the Sib grade level, has never had a full·time job, is between 16 and 

24 years old, and is from an economically disadvantaged family. Many students have significant 

basic skills and behavioral deficiencies, lacking the skills and knowledge to gain and retain 

employment. As. a result, Job Corps takes a holistic approach to education and training, 

assessing each student's needs and designing essentially an individualized program or 

intervention to address those needs. 

For example, in the OIG sample, over 300 students had a documented disability. The lowest 

entry level math score was a grade level equivalent (GLE) of 1.9 (less than second grade), with 

the student achieving a final math assessment of 4.9 (less than fifth grade). Similar scenarios 

were identified in the sample students' reading scores. mc asserts that these students certainly 

did benefit from the program's training, and that funds were not misspent to place them in entrylevel jobs. FW1her, the 1,569 placements represent less than 10% of JTM placements and 5.9"10 

of all PY 2009 graduate placements. Given the population that Job Corps serves, thls statistic is 

more a testament of Job Corps' success than its shortcomings. The majority of graduates in this 

sample maintained retention in employment or education 12 months after the initial placement. 

Further, the economic recession in recent years impacted minority youth to a much greater 

degree than other sectors of the population. As of August 2011 , the unemployment rate for 

youth, ages 16-24, was over twice that of the national average of 9.1 percent. For AfricanAmerican males in thls age group, the unemployment rate jumps to almost 50%. This economic 

downturn has resulted in Job Corps youth competing in today's workforce with adults of all skill 

levels. Under these conditions, ETA disagrees with the OIG that vaJid placements in entry-level 

jobs equates to wasteful spending. 

Finally, ETA did not state that "achieving and maintaining a basic entry level job at the end of 

the Job Corps experience is a crowning achievement for most students," but that "entering the 

workforce and maintaining employment is a first step in a student's career pathway." 

PerformaDce Metrics 

ETA agrees with the OIG that 100% of Job Corps' perfonnance metrics are aJigned with the 

Secretary's goaJs and the agency's mission. Beyond this, though, ETA has concerns about the 

reliability and analysis of the perfonnance metrics, especially as outlined in Exhibit 1: AnaJysis 

of Job Corps' 5S Performance Metrics. This analysis appears to be based on incorrect data. mc 

has on more than one occasion provided data to revise Exhibit I, yet the final venion does not 

include the corrected data. For example, metrics 49' and 50Z are identified as unpUblished and 

not publicly available, though mc has shared with the OIG the URLs where these memcs can 

be found on Job Corps' public website. 

I Metric 50, Student on·boud strength. can be found here: 

http;l/www.jobcorps.goyILjbrarieslpdUjob C9!pS cemer invetllory.sf1b 

i The most recent release ofmelric 50, Avcrage Length of Stay, can be found here: 

hUD jllwww. jobcorps.govlLibrariesJpdU10 3rd gtr.sflb 

4 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

56 Report No. 26-11-004-03-370 
Further, ETA maintains that Job Corps' performance metrics meet the intent of the Workforce 

Investment Act, Government Perfonnance and Results Act, and Departmental priorities. Job 

Corps' metrics are derived from these requirements. Rather than burden the system with 

individual metrics, some metrics have been streamlined and assigned a target. For example, the 

graduate placement metric meets the WIA requirement to set targets fo r graduates placed in 

employment related and not related to training. as well as the number of hours worked. This 

metric also fulfills the requirement to collect infonnation on the number of graduates who eoter 

education programs, including apprenticeship, and enter the military. 

Cost Emc:icDCY 

We take issue with the discussion draft's statements that Job Corps has a cost efficiency metric, 

that it is entitled. "cost per participant," and that it is in compliance with ETA's guidance for 

reporting cost efficiency. 

Job Corps, like all ETA programs, does not use a cost efficiency metric. Rather, Job Corps 

tracks and report cost per new enrollee as part of its annual budget submission. It is used by 

management for historical and comparative purposes and to infonn management decisions. It is 

not, however, used to drive perfonnance and outcome decisions, as this would lead to unintended 

consequences. For example, Job Corps has taken steps in r«:ent years to increase student length 

of stay in the program, as this leads to more positive outcomes for students. However, an 

increased student length of stay would increase the cost per new enrollee, as fewer students 

would move through the system. Therefore, using cost per new enrollee to drive perfonnance 

management, instead of as an infonnation tool to make longitudinal comparisons, would mean 

reducing per student costs at the expense of student achievement. 

At this time, none of ETA's programs are using a cost efficiency measure, nor has ETA 

developed guidance to calculate cost efficiency. The 010 has cited the May 201 0 study, 

"Implementing Efficiency Measures for Employment and Training Programs," using language 

that would indicate that Job Corps is out of sync with other ETA programs. This is not the case 

as ETA programs are not using cost efficiency metrics. In addition, the study cites significant 

challenges to establishing cost efficiency metrics that were ignored by the 010. 

Finally, the OIO states that Administration costs should be included when calculating Job Corps' 

cost per new enrollee. This practice would set Job Corps apart from all other ETA programs, 

which do not include their administration funding in participant cost calculations. 

Administration funding impacts only the federal workforce that oversees and monitors the 

program, and does not affect the direct delivery of services to students. 

Throughout the course of the audit, the 010 provided a wide range of alternative cost metrics for 

ETA's consideration. Some of the calculations for OIO's proposed alternative measures 

disregarded large subsections of Job Corps' population, which the program is legislatively 

mandated to serve. For example, the calculation the OIG proposed for the "cost per success " 

metric did not account for any services provided to fonner enrollees, who receive post. 

enrollment services outlined in the Workforce Investment Act.. Others disregarded core services 

of the program, such as cost per high school diploma completions, which overlooked career 

technical training and other core services provided by the program. 

5 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

57 Report No. 26-11-004-03-370 
Evaluations 

Job Corps disagrees that the Secretary's Advisory Committee and consultants hired to provide 

assessments of the program should have been managed more appropriately. This OIG finding 

stems not from the management of the work, but from ETA's reluctance to adopt specific 

recommendations from these consultants. ETA has reiterated to the GIG that the work 

conducted by these entities is beneficial, even if the final recommendations are not completely 

implemented. In fact, ETA and OJC believe that its efforts to address management concerns by 

commissioning studies demonstrate sound management technlques. 

ETA values the research conducted to improve its programs and services. and comprehensive 

analysis is conducted to review results and consider implementation strategies. To the extent 

feasible, ETA adopts recommendations that arise from these studies, but does not consider 

evaluations invaluable if they do not lead to changes in the program. In some cases, 

recommendations may not be cost effective. or aligned with Departmenta1 priorities and 

authorizing legislation. This does not negate ETA's commitment to explore multiple avenues to 

improve its programs, or the valuable impact of research and evaluation studies on all aspects of 

public and private sector programs, services, and initiatives. 

Recommendations 

ETA is committed to improving the transparency and availability of Job Corps' performance 

metrics. As stated previously, ETA has already released a new JTM crosswalk that a1igns Job 

Corps training with O·NET·SOC occupational codes, which. will help Job Corps make great 

strides in addressing the OIG recommendations. Our responses to the draft report's 

reconunendations follow: 

OIG Recommendation 1. Review and improve its performance metria to provide decision· 

makers with useful and reliable information to make informed decisions regarding the program's 

performance and costs. This includes ensuring metrics are complete and accurate, comply with 

WIA, and have reported results and established targets. 

Response: Management partially accepts this recommendation. Job Corps currently provides 

decision makers with reliable data to inform management and programmatic decisions. In Fiscal 

Year 2012, ETA will make performance outcomes more transparent to stakeholders and the 

public by publishing additional performance metrics as well as an annual report on WIA metrics. 

OIG Recommendation 2. Improve oversight of its service providers to increase the number of 

students who find employment that relate to and utilize the vocational training received. 

Response: Management accepts this recommendation. Based on the current policies and 

procedures, during the 1'1 quarter ofFY 2012, the National Director of Job Corps will issue a 

memorandwn to Regional Offices to reiterate policies and procedures regarding oversight 

responsibilities, to include audit sampling during compliance assessments. 

6 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

58 Report No. 26-11-004-03-370 
Job Corps has also created a new safeguard in its electronic system that will require all career 

transition services contractors to validate the relevance of a student's placements to the training. 

'This new feature will allow for targeted internal audits and reporting centered on high-risk 

placement codes, such as "cashier," which have previously held the potential for excessive use. 

The National Director of Job Corps will also issue a memorandum through the Regional Offices 

to direct career transition services contractors to strengthen policies and procedures to ensure 

compliance with Job Corps' policies for determining job training match placements. 

Job Corps, in coordination with the ETA Office of Contracts Management, will evaluate the 

questionable job training match placements, and seek documentation from the applicable career 

transition setVices contractors. In the event that a contractor cannot support the questioned costs, 

liquidated damages will be assessed. Appropriate information to close this recommendation will 

be forwarded to the OIG. 

We consider this recommendation resolved. 

01G Reco",melldiztit", 3. Develop a process to ensure the scope of work contracted for and 

conducted by consultants is managed appropriately to maximize value, ensure sllch investments 

work as planned, and result in meaningful improvements. 

RespolIse: Management partially accepts this recommendation. ETA conducts its procurement 

and contracting activities in accordance with Federal Acquisition Regulations (FAR) 

requirements. Further, Contracting Officer's Representative responsibilities are identified in the 

FAR. and are strictly enforced. Thus, processes are already in place to execute the OIG's 

recommendations. ETA will continue to closely monitor consultants' and committees' work in 

assessing Job Corps, and will consider all recommendations. 

We consider this recommendation resolved. 

Based on the foregoing responses, we anticipate that the audit report's recommendations will be 

resolved and can be closed upon completion of the corrective actions. 

cc: Roberta Gassman, ETA 

Edna Primrose, Job Corps 

7 

U. S. Department of Labor – Office of Inspector General 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

59 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

PAGE INTENTIONALLY LEFT BLANK 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

60 Report No. 26-11-004-03-370 
U. S. Department of Labor – Office of Inspector General 

Appendix F 

Acknowledgements 

Key contributors to this report were Ray Armada (Audit Director), Jon K. Ling (Audit 

Manager), Steven Grubb, Kathleen Mitomi, Catherine Christian, and Patrick Trager. 

Additional support for this report was provided by Steve Witherspoon (Reviewer) and 

Ajit Buttar (Statistician), 

Job Corps Needs to Improve Reliability of Performance Metrics and Results 

61 Report No. 26-11-004-03-370 
 

TO REPORT FRAUD, WASTE, OR ABUSE, PLEASE CONTACT: 

Online: http://www.oig.dol.gov/hotlineform.htm 

Email: hotline@oig.dol.gov 

Telephone:	 1-800-347-3756 

202-693-6999 

Fax: 202-693-7020 

Address: Office of Inspector General 

U.S. Department of Labor 

200 Constitution Avenue, N.W. 

 Room S-5506 

Washington, D.C. 20210