United States General Accounting Office

GAO Report to Congressional Requesters

March 1996 JOB TRAINING

PARTNERSHIP ACT

Long-Term Earnings

and Employment

Outcomes

GAO/HEHS-96-40

GAO United States

General Accounting Office

Washington, D.C. 20548

Health, Education, and

Human Services Division

B-256749 

March 4, 1996

The Honorable Paul Simon

United States Senate

Dear Senator Simon:

Although the federal government spends billions of dollars annually to

support employment training programs, little is known about their

long-term effects on participants’ earnings and employment rates.1

 Few

training programs have been rigorously evaluated to assess their net

impact, and, for those that have, the research results have often been

inconclusive. Furthermore, most research on training programs has

focused on short-term effects—the year or two immediately following

completion of training.

Enacted in 1982, title II of the Job Training Partnership Act (JTPA) has been

the cornerstone of federal employment training programs, providing block

grants to state and local governments to administer these federally funded

programs. JTPA supports job training for individuals facing barriers to

employment and needing special training to obtain productive

employment. Under recent legislative proposals2

 to consolidate multiple

federally funded training programs, states would have the flexibility to

design and implement a statewide approach to job training based on the

concept of one-stop career centers. As states design and implement their

approaches to job training, lessons learned from JTPA can help in

reallocating training dollars and in setting performance standards.

Because of concerns about the long-term impact of job training programs,

you asked us to (1) ascertain the long-term earnings of participants in

JTPA-sponsored programs and to compare their earnings with those of

nonparticipants and (2) calculate the long-term employment rates for

these JTPA participants and to compare their employment rates with those

of nonparticipants. For this study, we defined long-term earnings and

employment rates as the annual earnings and employment rates achieved

in the fifth year after applying for JTPA training.

1

Multiple Employment Training Programs: Most Federal Agencies Do Not Know If Their Programs Are

Working Effectively (GAO/HEHS-94-88, Mar. 2, 1994).

2

Both the Senate and the House have passed bills that would consolidate over 90 federal education,

employment, and job training programs. The Senate’s Workforce Development Act of 1995 (S.

143) would replace the programs with one block grant to each state, while the House’s Consolidated

and Reformed Education, Employment, and Rehabilitation Systems Act (CAREERS Act, H.R.

1617) would authorize three consolidation grants to each state.

Page 1 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

To develop this information, we merged data from the National JTPA Study3

 (NJS) with annual earnings records from the Social Security

Administration (SSA). Participants in the NJS were randomly assigned to

either the treatment group (allowed to enroll in JTPA training) or the

control group (not allowed to enroll in JTPA training for 18 months). We

calculated the average earnings and employment rates4

 of four target

groups (adult men, adult women, male youths, and female youths5

) for 5

years after their acceptance into the study. We considered a 5-year period

sufficient to provide meaningful data on the long-term effects of JTPA

training.

Because of the NJS’ inherent design problems, we cannot unequivocally use

our findings to draw conclusions about JTPA’s effectiveness. For example,

the participating areas were neither randomly selected nor necessarily

representative of JTPA training nationally. Moreover, not everyone assigned

to the treatment group enrolled in or completed JTPA training. Also, many

control group members received some training services from other

sources. Nevertheless, the NJS had some design advantages, such as

random assignment of applicants for JTPA training, that made the data

appropriate for our study.

Appendix I has further information on the NJS and SSA data sets and our

statistical results. We conducted our work between April 1994 and

January 1996 in accordance with generally accepted government

accounting standards.

Results in Brief Although our statistical analysis showed some positive effects of JTPA in

the years immediately following training, we found no significant6

 effect of

JTPA on earnings or employment rates after 5 years. In some earlier years,

adult men and women who received training—but not male or female

3

The Department of Labor commissioned Abt Associates to conduct the National JTPA Study in 1986

to evaluate the impact of JTPA on adults and youths.

4

We defined employment on the basis of SSA earnings records. If a person’s SSA earnings record

showed positive earnings in a given calendar year, we considered that person to be employed. If 78 out

of 100 people in a group showed positive earnings in their SSA earnings records, the employment rate

for that group would be 78 percent.

5

Adults were defined as those 22 years old and older. Youths were defined as those aged 16 to 21 who

no longer attended school when applying for JTPA training. This included both high school dropouts

and graduates.

6

“Significance” refers to statistical significance at the 5-percent level. This significance means that we

can be 95-percent confident that the observed difference between groups is not due to chance or

random variation. Our convention is to calculate significance at the 5-percent level. We use this

convention both in reference to other reports and in presenting our own findings.

Page 2 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

youths—had earnings or employment rates significantly higher than those

of the control group. By the fifth year, each of the four treatment groups

had earnings and employment rates that were nominally higher than those

of the control group. Because none of the fifth-year differences were

statistically significant, however, we could not attribute the higher

earnings to JTPA training rather than to chance alone.

Background Enacted in 1982, JTPA is the largest federal employment training program,

with titles II-A and II-C intended to prepare economically disadvantaged

adults and youths, respectively, for entry into the labor force.7 JTPA

emphasizes state and local government responsibility for administering

federally funded job training programs. In fiscal year 1995, JTPA title II-A

and II-C programs received approximately $1.6 billion in funding.

JTPA training programs annually provide employment training for specific

occupations and services, such as job search assistance and remedial

education, to roughly one million economically disadvantaged individuals.

Training is provided in local service delivery areas (SDA) through service

providers, such as vocational-technical high schools, community colleges,

proprietary schools, and community-based organizations. The program

objectives are to increase earnings and employment and to reduce welfare

dependence for participants of all ages. During the NJS, participation in

JTPA involved roughly 3 to 4 months of training at an average cost of about

$2,400 per participant.

In 1986, Labor commissioned the NJS to evaluate the impact of JTPA on

adults and youths because previous findings on the effects of job training

programs had been hampered by poor data and statistical problems. The

NJS randomly assigned persons who sought JTPA services, and were eligible

for them, to a treatment group or a control group. The treatment group

was offered JTPA training, and the control group was not. The study was

intended to ensure that the two groups would not differ systematically in

any way except access to the program, so any subsequent differences in

outcomes could be attributed solely to JTPA.

The study included over 20,000 eligible participants who applied for JTPA

services between November 1987 and September 1989 in 16 local SDAs. The

7

JTPA title II-A was originally targeted to both adults and youths. The 1992 JTPA Amendments split

title II-A into two components: title II-A for adults and title II-C for youths.

Page 3 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

study followed up on a sample of people in the treatment and control

groups 18 months after assignment8

 and then again at 30 months.9

The NJS showed mixed results on the impact of JTPA programs.10 Adult

women assigned to JTPA training had significantly higher earnings than the

control group of adult women after 18 and 30 months, but the treatment

group of adult men, as well as of both male and female youths, did not

have significantly higher earnings than its respective control groups.

Participant Earnings

Not Significantly

Greater Than Control

Group Earnings After

5 Years

Participants assigned to receive JTPA training did not have significantly

greater earnings than control group members 5 years after their

assignment. For some of the four targeted worker categories—adult men,

adult women, male youths, and female youths—treatment group earnings

exceeded those of the control group in some of the intervening years, but

any statistically significant effects disappeared by the fifth year.11

Earnings Outcomes for

Adult Men

Annual earnings of adult men increased in each year following assignment

for both the treatment and control groups. As shown in figure 1, in the first

year after assignment, the average annual earnings of adult men in the

treatment group grew from about $4,400 to about $6,900. This group’s

earnings continued to rise in the subsequent years, reaching

approximately $8,700 in the fifth year after assignment to receive JTPA

training. The earnings of adult men in the control group, which did not

receive JTPA training, also rose following assignment, but this group’s

earnings were less than those of the treatment group for each of the 5

years.

8

See Howard S. Bloom and others, The National JTPA Study: Title II-A Impacts on Earnings and

Employment at 18 Months (Bethesda, Md.: Abt Associates, Inc., Jan. 1993).

9

See Larry L. Orr and others, The National JTPA Study: Impacts, Benefits, and Costs of Title II-A

(Bethesda, Md.: Abt Associates, Inc., Mar. 1994) for the results at 30 months.

10Bloom and others, The National JTPA Study: Title II-A Impacts on Earnings and Employment at 18

Months and Orr and others, The National JTPA Study: Impacts, Benefits, and Costs of Title II-A.

11Generally, our results matched the findings of the NJS for the 30 months immediately following

assignment.

Page 4 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 1: Earnings of Adult Men Before

and After Assignment Annual Earnings

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

After 5 years, the difference between earnings of the treatment and control

groups was not statistically significant. Five years after assignment, the

treatment group’s earnings had exceeded those of the control group by

approximately $300 to $500 annually, but only in the first 3 years were

these differences statistically significant.

Earnings Outcomes for

Adult Women

Earnings of adult women showed a pattern similar to those of adult men,

increasing in each year after assignment. Figure 2 shows that the annual

earnings of adult women assigned to the treatment group increased from

approximately $2,800 in the year of assignment to approximately $4,700 in

the first year following assignment. This group’s earnings continued to

climb, reaching approximately $6,600 in the fifth year. Earnings of adult

women in the control group followed a similar pattern, but this group’s

earnings were lower than those of the treatment group in each year,

reaching approximately $6,200 during the fifth year.

Page 5 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 2: Earnings of Adult Women

Before and After Assignment Annual Earnings

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

As with the earnings of adult men, 5 years after assignment the difference

between the treatment and control groups’ annual earnings was not

statistically significant. However, during the first 4 years after assignment,

the differences between the treatment and the control groups’ earnings

were statistically significant in each year, with treatment group earnings

approximately $300 to $600 higher than control group earnings annually.

Earnings Outcomes for

Male Youths

The earnings of male youths in the control group, like those of adult men

and adult women, increased in each year following assignment. Figure 3

shows that the earnings of male youths in the treatment group increased

from approximately $2,900 in the year of assignment to approximately

$4,600 in the first year after assignment. This group’s earnings continued

to grow during the 5-year period, reaching a high of approximately $7,600

in the fifth year. The earnings of male youths in the control group also rose

during the 5-year period following assignment, climbing from

approximately $4,800 in the first year to approximately $6,800 in the fifth

year.

Page 6 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 3: Earnings of Male Youths

Before and After Assignment Annual Earnings

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

We found no significant difference between the treatment and control

groups’ annual earnings 5 years after assignment. Although the control

group’s earnings were higher than the treatment group’s during the first 3

years following assignment, the differences, which ranged from

approximately $200 to $400 each year, were not statistically significant.

During the fourth and fifth years, the treatment group had higher earnings

than the control group, but these differences too were not statistically

significant.

Earnings Outcomes for

Female Youths

Earnings of female youths showed a pattern similar to that of male youths,

growing in each year following assignment. Earnings of female youths in

the treatment group rose from approximately $2,000 during the year of

assignment to approximately $3,300 in the first year following assignment

(see fig. 4). This group’s earnings continued to climb, reaching

approximately $5,400 in the fifth year following assignment. The earnings

of female youths in the control group also rose during the 5-year period,

Page 7 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

climbing from approximately $3,400 in the first year to a high of

approximately $5,200 in the fifth year.

Figure 4: Earnings of Female Youths

Before and After Assignment Annual Earnings

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

We found no significant differences between the treatment and control

groups’ annual earnings 5 years after receiving their assignments. During

the first 2 years following assignment, the control group’s earnings were

higher than the treatment group’s, but the differences of less than $100

annually were not statistically significant. In the fourth and fifth years

following assignment, the treatment group had earnings of approximately

$100 to $300 higher than the control group, but these differences also were

not statistically significant.

Page 8 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Employment Rates of

Participants Not

Significantly Greater

Than Control Group

Employment Rates

After 5 Years

As with earnings, employment rates of those assigned to receive JTPA

training were not significantly greater than employment rates of control

group members 5 years after assignment. For some of the four targeted

worker categories, treatment group employment rates were higher than

those of the control group in some years, but any statistically significant

effects disappeared by the fifth year.

Employment Rates of

Adult Men

The employment rates of both treatment and control group adult men

peaked during the calendar year of assignment and then declined in

subsequent years, eventually reaching levels lower than those of the men

before entering the study (see fig. 5). For example, the employment rate

for adult men in the treatment group was 87 percent in the year of

assignment. The percent employed declined in the following years,

reaching 72 percent by the fifth year following assignment, which was

lower than the group’s employment rate of 79 percent in the year before

entering the study. The adult men in the control group showed a similar

pattern—their employment rate was 87 percent in the year of assignment

but dropped to 71 percent in the fifth year after assignment.

Page 9 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 5: Employment Rates of Adult

Men Before and After Assignment Percent Employed

40

45

50

55

60

65

70

75

80

85

90

95

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

After 5 years, the difference between the treatment and control groups’

employment rates was not statistically significant. The treatment group’s

employment rates were higher than the control group’s in each year

following assignment, although the differences in the employment rates

were statistically significant only in the fourth year following assignment.

Employment Rates of

Adult Women

The pattern of employment rates of adult women was somewhat similar to

that of adult men. The employment rates of adult women were highest

during the calendar year following assignment, with 80 percent of the

treatment group and 77 percent of the control group employed (see fig. 6).

After the first year, however, the employment rates for both the treatment

and control groups fell, reaching 69 percent and 67 percent, respectively,

in the fifth year following assignment. These rates in the fifth year were

also lower than each group’s employment rate in the year before

assignment.

Page 10 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 6: Employment Rates of Adult

Women Before and After Assignment Percent Employed

40

45

50

55

60

65

70

75

80

85

90

95

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

We found no significant differences between the treatment and control

groups’ employment rates 5 years after assignment. The treatment group’s

employment rates exceeded the control group’s in all 5 years following

assignment, usually by about 2 to 3 percent, but only in the first 3 years

were these differences statistically significant.

Employment Rates of Male

Youths

The pattern of employment rates of male youths was somewhat similar to

that of adult men and women: the male youths’ employment rates peaked

during the calendar year following assignment—reaching nearly

91 percent for the treatment group and over 92 percent for the control

group—but then declined (see fig. 7). However, in contrast to the

employment rates of adults, those of male youths were slightly higher 5

years after assignment than before assignment, reaching 81 percent for the

treatment group in the fifth year, compared with 80 percent in the year

before assignment.

Page 11 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 7: Employment Rates of Male

Youths Before and After Assignment Percent Employed

40

45

50

55

60

65

70

75

80

85

90

95

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

We found no significant differences between the treatment and control

groups’ employment rates 5 years after assignment. While the employment

rates for the control group actually exceeded those for the treatment

group in the year of assignment and the first and third years following

assignment, none of the differences were statistically significant.

Employment Rates of

Female Youths

The employment rates of female youths in both the treatment and control

groups peaked during the calendar year of assignment, declined somewhat

over the next 4 years, and then slightly increased in the fifth year (see fig.

8). As with those of male youths, the employment rates of female youths

were slightly higher 5 years after assignment than before assignment. The

employment rates of female youths were 74 percent for the treatment

group and 73 percent for the control group in the fifth year following

assignment, compared with 71 and 73 percent, respectively, in the year

before assignment.

Page 12 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

Figure 8: Employment Rates of Female

Youths Before and After Assignment Percent Employed

40

45

50

55

60

65

70

75

80

85

90

95

-3 -2 -1 0 1 2 3 4 5

Years Before and After Random Assignment

Treatment Group

Control Group

We found no significant differences between the treatment and control

groups’ employment rates 5 years after assignment. Employment rates for

the treatment group exceeded those for the control group in 4 of the 5

years following assignment, but none of the differences in employment

rates were statistically significant.

Conclusions Though both long-term earnings and employment rates for NJS treatment

groups surpassed those for their respective control groups, the differences

did not meet our test for statistical significance. Five years after

expressing an interest in JTPA-sponsored job training, individuals assigned

to participate in the program did not have earnings or employment rates

significantly higher than individuals not assigned to participate.

Agency Comments In commenting on a draft of this report, Labor expressed several concerns.

It took exception to what it characterized as unwarranted negative

conclusions that are not consistent with the report findings. Labor also

Page 13 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

took issue with the importance the report places on tests of statistical

significance applied to earnings of an individual group in a given year,

preferring to emphasize other evidence of the positive effect of JTPA on

participant earnings over the 5-year period. Labor also expressed concerns

that the report findings have limited relevance to current job training

programs.

We believe that our conclusions are well supported by our findings. On

several occasions where appropriate, we have noted comparisons

favorable to the JTPA treatment groups, including in the “Results in Brief”

and “Conclusions” sections. Although other evidence covering the 5-year

period might be found to better highlight the positive effects of JTPA

training, our research focused on the earnings and employment rates of

each target group in the fifth year after applying for JTPA training. Also, we

do not believe that current or proposed job training programs sufficiently

differ from JTPA training at the time of the NJS to limit the relevance of our

report findings.

In its response, Labor enclosed an attachment with specific comments on

the report and additional information. This attachment and our evaluation

of the comments appear in appendix III. Labor also provided us with

technical comments, which we have incorporated in the report where

appropriate.

As agreed with your office, unless you publicly announce its contents

earlier, we plan no further distribution of this report until 30 days from the

date of this letter. At that time, we will send copies to interested parties

and make copies available to others upon request.

Page 14 GAO/HEHS-96-40 JTPA Outcomes
B-256749 

This report was prepared under the direction of Wayne B. Upshaw,

Assistant Director, who may be reached on (202) 512-7006 if you or your

staff have any questions. Gene Kuehneman, Senior Economist,

(202) 512-4091, Jill Schamberger, Senior Evaluator, and Thomas L.

Hungerford, Senior Economist, were major contributors to this report.

Sincerely yours,

Cornelia M. Blanchette

Associate Director, Education

 and Employment Issues

Page 15 GAO/HEHS-96-40 JTPA Outcomes
Contents

Letter 1

Appendix I 

Data Sources and

Methodology

18

National JTPA Study 18

Social Security Earnings Records 19

Data Analysis 19

Appendix II 

Data and Statistics for

Figures

20

Appendix III 

Comments From the

Department of Labor

and Our Evaluation

26

GAO Comments 33

Tables Table II.1: Data Points for Figure 1—Earnings of Adult Men

Before and After Assignment

20

Table II.2: Data Points for Figure 2—Earnings of Adult Women

Before and After Assignment

20

Table II.3: Data Points for Figure 3—Earnings of Male Youths

Before and After Assignment

21

Table II.4: Data Points for Figure 4—Earnings of Female Youths

Before and After Assignment

21

Table II.5: Data Points for Figure 5—Employment Rates of Adult

Men Before and After Assignment

22

Table II.6: Data Points for Figure 6—Employment Rates of Adult

Women Before and After Assignment

22

Table II.7: Data Points for Figure 7—Employment Rates of Male

Youths Before and After Assignment

23

Table II.8: Data Points for Figure 8—Employment Rates of

Female Youths Before and After Assignment

23

Table II.9: Standard Errors Associated With Earnings of Adult

Men Before and After Assignment

24

Table II.10: Standard Errors Associated With Earnings of Adult

Women Before and After Assignment

24

Table II.11: Standard Errors Associated With Earnings of Male

Youths Before and After Assignment

25

Page 16 GAO/HEHS-96-40 JTPA Outcomes
Contents

Table II.12: Standard Errors Associated With Earnings of Female

Youths Before and After Assignment

25

Figures Figure 1: Earnings of Adult Men Before and After Assignment 5

Figure 2: Earnings of Adult Women Before and After Assignment 6

Figure 3: Earnings of Male Youths Before and After Assignment 7

Figure 4: Earnings of Female Youths Before and After

Assignment

8

Figure 5: Employment Rates of Adult Men Before and After

Assignment

10

Figure 6: Employment Rates of Adult Women Before and After

Assignment

11

Figure 7: Employment Rates of Male Youths Before and After

Assignment

12

Figure 8: Employment Rates of Female Youths Before and After

Assignment

13

Abbreviations

BLS Bureau of Labor Statistics

CPS Current Population Survey

JTPA Job Training Partnership Act

NJS National JTPA Study

SDA service delivery area

SSA Social Security Administration

Page 17 GAO/HEHS-96-40 JTPA Outcomes
Appendix I 

Data Sources and Methodology

To select our sample of individuals assigned to receive training under JTPA

and our control group, we used participation information from the

National JTPA Study (NJS). We then obtained long-term earnings and

employment information for these individuals from SSA. Our analysis

compared earnings and employment levels of individuals in the treatment

and control groups to determine whether differences between these

groups were statistically discernable.

National JTPA Study The original NJS data set contained demographic and program information

on 20,601 people who applied for JTPA services between November 1987

and September 1989 in 16 local service delivery areas. Program applicants

were recruited, screened to determine their eligibility, assessed to

determine their service needs and wants, and recommended for services.

NJS participants were then randomly assigned to either the treatment

group, which was allowed to participate in JTPA title II-A programs, or the

control group, which was not allowed to participate in these programs for

18 months. Approximately two-thirds of the applicants were assigned to

the treatment group and one-third to the control group. The control and

treatment groups were closely matched in demographic variables such as

age, race, and education, which typically allows a meaningful comparison

of average outcomes between the two groups.

However, two factors intervened to make such a comparison problematic.

First, not all members of the treatment group participated in JTPA

programs. For example, about two-thirds of the adult treatment group

members enrolled in JTPA, but the other one-third either found jobs on

their own or decided not to participate in the program. Second, a

substantial minority of the control group members chose to participate in

some alternative, non-JTPA training programs. These complications

preclude attributing earnings differences between the two groups solely to

JTPA training. Therefore, our findings refer to differences between the

treatment and control groups rather than between individuals who did or

did not receive JTPA training. Furthermore, we do not know which of the

control and treatment group members chose to receive training later than

18 months after assignment.

The NJS was not designed to track treatment or control group members

beyond 30 months. Therefore, to calculate and compare longer term

earnings and employment outcomes for these groups, we needed

information from another source.

Page 18 GAO/HEHS-96-40 JTPA Outcomes
Appendix I 

Data Sources and Methodology

Social Security

Earnings Records

We obtained annual earnings records from SSA for the individuals in the NJS

treatment and control groups. SSA maintains information on annual

earnings of individuals contributing to either Social Security or Medicare.

We assumed that an individual was employed if his or her SSA records

showed positive earnings for a given year. We adjusted data for what we

assumed were data entry or processing errors, and we also rounded

reported negative earnings to zero.

Data Analysis We analyzed the NJS and SSA earnings records of 13,699 NJS participants12 to

determine their annual earnings and employment outcomes for the 3 years

before assignment to the treatment or control group, the year of

assignment, and 5 years following assignment. The 3 years of prior

earnings and employment data served to demonstrate the prior

comparability of treatment and control groups. The 5 years of

postassignment data effectively doubled the 30-month follow-up period for

the NJS. The treatment group had 9,275 individuals, and the control group

had 4,424 individuals.

We used individual earnings data and calculated means and variances for

each of the four target groups—adult men, adult women, male youths, and

female youths—to compare the treatment groups’ earnings and

employment outcomes with those of the control groups. We tested for

differences in earnings and employment outcomes at the 5-percent

significance level.

We calculated annual earnings amounts using SSA information on Social

Security-covered earnings for nonfederal workers and on earnings covered

by Medicare for federal workers. We calculated employment rates as the

percentage of each group with positive covered earnings in a calendar

year. Individuals with unreported earnings may have had their earnings

and employment understated in our analysis. Individuals whose earnings

exceeded the Social Security withholding ceiling may also have had their

earnings understated in our analysis.13 These limitations applied to both

the treatment and control groups, and we do not believe they affected the

two groups differently.

12We did not analyze earnings and employment information for the NJS participants who were

assigned after 1988 because we had only 4 years of postassignment data for them. We also excluded

the NJS participants whose SSA records did not adequately match information collected for the NJS,

such as name or birth month and year.

13Only a certain amount of workers’ earnings—for example $55,500 in 1992—were subject to the Social

Security tax.

Page 19 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.1: Data Points for Figure

1—Earnings of Adult Men Before and

After Assignment Annual earnings

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before $5,883 $5,924 no

2 years before 5,680 5,894 no

1 year before 5,106 5,246 no

Assignment 4,439 4,242 no

1 year after 6,901 6,410 yes

2 years after 7,792 7,254 yes

3 years after 7,936 7,363 yes

4 years after 8,282 7,725 no

5 years after 8,651 8,326 no

Table II.2: Data Points for Figure

2—Earnings of Adult Women Before

and After Assignment Annual earnings

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before $3,262 $3,020 no

2 years before 3,377 3,215 no

1 year before 3,230 3,048 no

Assignment 2,823 2,703 no

1 year after 4,702 4,323 yes

2 years after 5,705 5,047 yes

3 years after 5,902 5,319 yes

4 years after 6,367 5,811 yes

5 years after 6,556 6,154 no

Page 20 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.3: Data Points for Figure

3—Earnings of Male Youths Before

and After Assignment Annual earnings

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before $860 $828 no

2 years before 1,456 1,575 no

1 year before 2,179 2,303 no

Assignment 2,894 3,014 no

1 year after 4,612 4,792 no

2 years after 5,620 5,963 no

3 years after 6,130 6,497 no

4 years after 6,687 6,425 no

5 years after 7,554 6,778 no

Table II.4: Data Points for Figure

4—Earnings of Female Youths Before

and After Assignment Annual earnings

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before $629 $663 no

2 years before 1,069 1,090 no

1 year before 1,529 1,707 no

Assignment 1,974 2,098 no

1 year after 3,339 3,389 no

2 years after 4,045 4,125 no

3 years after 4,393 4,383 no

4 years after 4,934 4,610 no

5 years after 5,433 5,209 no

Page 21 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.5: Data Points for Figure

5—Employment Rates of Adult Men

Before and After Assignment Employment rates

(percent employed)

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before 79.2 78.8 no

2 years before 77.6 79.0 no

1 year before 78.3 80.6 no

Assignment 87.0 87.4 no

1 year after 86.7 86.4 no

2 years after 81.3 80.7 no

3 years after 76.3 73.9 no

4 years after 73.7 70.0 yes

5 years after 72.3 71.1 no

Table II.6: Data Points for Figure

6—Employment Rates of Adult Women

Before and After Assignment Employment rate

(percent employed)

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before 65.8 65.8 no

2 years before 67.1 66.2 no

1 year before 69.4 68.4 no

Assignment 76.7 75.7 no

1 year after 80.3 77.4 yes

2 years after 76.3 73.7 yes

3 years after 71.2 68.1 yes

4 years after 69.7 67.7 no

5 years after 68.7 67.4 no

Page 22 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.7: Data Points for Figure

7—Employment Rates of Male Youths

Before and After Assignment Employment rate

(percent employed)

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before 46.5 48.3 no

2 years before 63.2 66.4 no

1 year before 79.6 79.6 no

Assignment 89.2 91.8 no

1 year after 90.5 92.1 no

2 years after 88.4 87.8 no

3 years after 82.2 82.6 no

4 years after 80.4 79.4 no

5 years after 81.1 77.5 no

Table II.8: Data Points for Figure

8—Employment Rates of Female

Youths Before and After Assignment Employment rate

(percent employed)

Time period Treatment group Control group

Statistically

significant

difference at 5%

level?

3 years before 41.2 43.6 no

2 years before 57.6 60.5 no

1 year before 70.7 72.8 no

Assignment 82.0 81.8 no

1 year after 82.0 79.6 no

2 years after 79.7 78.2 no

3 years after 73.8 75.1 no

4 years after 71.7 70.7 no

5 years after 73.9 73.0 no

Page 23 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.9: Standard Errors Associated

With Earnings of Adult Men Before and

After Assignment

Standard error

Time period Treatment Control Pooled error T-ratio

3 years before 6,740 6,983 221 –0.19

2 years before 6,547 6,702 213 –1.00

1 year before 5,818 5,779 188 –0.74

Assignment 4,724 4,405 149 1.32

1 year after 6,825 6,406 216 2.27

2 years after 8,029 7,516 254 2.12

3 years after 8,753 8,335 279 2.06

4 years after 9,446 9,117 302 1.84

5 years after 10,134 9,611 322 1.01

Subgroup size 2,874 1,435

Table II.10: Standard Errors

Associated With Earnings of Adult

Women Before and After Assignment

Standard error

Time period Treatment Control Pooled error T-ratio

3 years before 4,638 4,412 134 1.81

2 years before 4,714 4,664 137 1.18

1 year before 4,315 4,296 126 1.44

Assignment 3,297 3,128 95 1.27

1 year after 5,017 4,828 145 2.62

2 years after 6,242 5,707 178 3.70

3 years after 6,829 6,414 196 2.98

4 years after 7,646 7,039 218 2.55

5 years after 7,797 7,480 225 1.79

Subgroup size 3,631 1,723

Page 24 GAO/HEHS-96-40 JTPA Outcomes
Appendix II 

Data and Statistics for Figures

Table II.11: Standard Errors

Associated With Earnings of Male

Youths Before and After Assignment

Standard error

Time period Treatment Control Pooled error T-ratio

3 years before 1,689 1,687 87 0.36

2 years before 2,339 2,604 125 –0.96

1 year before 2,762 2,778 142 –0.88

Assignment 3,048 2,902 154 –0.78

1 year after 4,799 4,603 243 –0.74

2 years after 5,807 5,703 297 –1.16

3 years after 6,561 6,646 338 –1.08

4 years after 7,445 7,186 378 0.69

5 years after 8,150 7,807 413 1.88

Subgroup size 1,177 559

Table II.12: Standard Errors

Associated With Earnings of Female

Youths Before and After Assignment

Standard error

Time period Treatment Control Pooled error T-ratio

3 years before 1,327 1,392 61 –0.56

2 years before 1,885 1,797 84 –0.25

1 year before 2,113 2,352 99 –1.80

Assignment 2,194 2,307 101 –1.23

1 year after 3,828 3,767 172 –0.29

2 years after 4,612 4,663 209 –0.38

3 years after 5,271 5,388 240 0.04

4 years after 6,053 5,803 270 1.20

5 years after 6,494 6,309 291 0.77

Subgroup size 1,593a 707

aSubgroup size for the treatment group 3 years after assignment is 1,592.

Page 25 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

See comment 17.

Page 26 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

Page 27 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

See comment 1.

See comment 1.

See comment 2.

See comment 2.

See comment 3.

See comment 4.

Page 28 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

See comment 5.

See comment 5.

See comment 6.

See comment 7.

See comment 8.

See comment 9.

See comment 10.

See comment 7.

See comment 11.

See comment 11.

Page 29 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

See comment 12.

See comment 13.

Page 30 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

Page 31 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

See comment 14.

See comment 14.

See comment 15.

See comment 16.

Page 32 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

The following are GAO’s comments on the Department of Labor’s letter

dated November 30, 1995.

GAO Comments 1. Labor comments that our report understates the gross returns to JTPA

training. Furthermore, Labor implies that these gross returns calculations

compare favorably with the returns to college education.

Our objective, as clearly stated in the report, was not to evaluate the

cost-effectiveness of JTPA training, but rather to determine and compare

the long-term effects of JTPA training. In fact, because we did not calculate

the gross return to JTPA participants the report cannot have understated or

overstated the values. Such calculations were not within the scope of this

report. While it may be true that these returns are favorable, we have no

basis to judge the favorability of the gross returns to training.

2. Labor states that the report does not acknowledge favorable aspects of

this study. Specifically, Labor cited that (1) all four target groups had

higher earnings in the fifth year after assignment; (2) both adult treatment

groups had higher earnings than their respective control groups in each of

the 5 years following assignment; and (3) for male youths, a positive trend

exists, and the fifth-year earnings exceed those of the control group by

over 10 percent.

Contrary to Labor’s comment, we did note many of these favorable

program outcomes in our report. We stated that adult male treatment

group members had higher earnings than adult male control group

members and presented similar findings for the other three target groups.

We further stated that adult male treatment group earnings exceeded

control group earnings in each of the 5 years and reported similar

information for adult women. Also, we noted the positive trend for

earnings of male youths. We did not note the percentage difference for

male youths in the fifth year because we did not report percentage

comparisons for any of the target groups.

3. Labor states that if the training impacts are accumulated over time

during the 5-year follow-up period, the net benefits outweigh the costs.

As we stated in comment 1, our objective was not to evaluate the

cost-effectiveness of JTPA training, but rather to determine and compare

the long-term effects of JTPA training. While it may be true that the net

Page 33 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

benefits outweigh the costs, we have no basis to judge this because such

calculations were beyond the scope of this report.

4. Labor states that the increase in standard errors is primarily responsible

for the decline in statistical significance of the estimated impacts.

While Labor is correct in stating that the standard errors were greater in

the fifth year, it is not accurate to attribute a decline in statistical

significance to either the estimated training effects or to the standard

errors. The test statistics used for our significance tests are determined by

the ratio of the estimates to their standard error, and attributing the lack of

significance solely to either component of these ratios is inappropriate.

5. Labor states that our conclusion requires assessing the total impact of

JTPA and the overall cost and benefits. The Department states that we

overemphasize the importance of year-by-year significance tests in

questioning the program’s usefulness in improving participants’ long-term

earnings prospects by stressing the insignificant effect of JTPA in the fifth

year.

We agree with Labor that year-by-year significance tests have limited value

in assessing the total impact of JTPA and the overall cost and benefits.

Furthermore, our year-by-year significance tests provide statistical

evidence that adult treatment group members achieved higher earnings for

several years following assignment to JTPA training. While other evidence

covering the 5-year period might be found to better highlight the positive

effects of JTPA training, we chose to address the question of whether the

fifth-year earnings of those assigned to participate in JTPA differed

significantly from the fifth-year earnings of those not assigned to

participate in JTPA.

6. While acknowledging that the observed earnings differences between

the four target groups were not statistically significant in the fifth year,

Labor asserts that the odds that all four differences would be positive

purely by chance is 6.25 percent.14

This implies that an accumulation of not statistically significant

observations provides more compelling empirical evidence than the actual

14For example, if training truly had no effect on earnings, then the probability that the estimated

difference would be positive for any one target group is 50 percent—that is, half of the time the

estimate would be positive and the other half it would be negative. Thus, the chance that all four

groups would show positive estimated earnings effects, if no true effect existed for any group, would

be 50 percent to the fourth power or 6.25 percent.

Page 34 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

significance test for any one group. While the probability (not odds) that

all four not significant fifth-year earnings differences would be positive

purely by chance might be low, our research question is whether a

significant earnings difference occurred for each target group.

7. Labor comments that the report does not report the standard errors.

Labor states that the report should include confidence intervals for the

estimates, sample sizes, and standard errors and specify significance levels

for the estimates.

We have made several additions to tables in appendix II in response to this

comment. We have added the sample size, the size of the treatment and

control groups, the standard errors, and a reminder that the significance

level chosen is 5 percent for the tables in this appendix. Since we have not

presented point estimates of the earnings effects, we did not calculate

confidence intervals for these estimates. Technical readers of our report

can construct such estimates and the associated intervals from the

information in the appendix II tables.

8. Labor claims that the report treats figures that are not significant as

zero.

We do not report any training effect as zero. The magnitude of the earnings

differences, whether significant or not, is discussed in the report and is

easily calculated from the tables in appendix II.

9. Labor states that statistical significance is not a knife edge of yes or no

but a continuum.

The level used for tests of statistical significance may be chosen from a

broad range (or continuum) of values. Although different researchers may

choose to use different values for the significance level, choosing a

significance level before analyzing any data is common practice. Once this

level has been chosen, statistical hypothesis testing very much involves a

yes or no decision. Either the data reject the null hypothesis of no training

effect at the set significance level or not. We follow these commonly

accepted procedures for hypothesis testing, and our convention is to set

the significance level for such tests at 5 percent.

10. Labor also states that the report should discuss the results, the

probability values, and changes in the significance levels.

Page 35 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

Our report does discuss the results as well as whether the earnings and

employment effects were significant and whether this significance

changed over time. Although we do not present the probability values,

technical readers of our report can calculate them using the information in

the appendix II tables.

11. Labor takes issue with our statement that complications (not all

treatment group members received training and some control group

members did receive training) precluded our attributing earnings

differences solely to JTPA training. It claims that these complications led us

to understate the effect of training, implying that the earnings differences

observed, along with perhaps some further overlooked earnings effects,

can be attributed to JTPA.

We clearly state that these complications preclude solely attributing the

earnings effects to JTPA training. However, we have no evidence that these

factors led to an understatement of the effect of JTPA training. In the first

place, a short delay can occur before an assignee can begin a training

program. In some of these cases, individuals find and accept employment

instead of reporting for training. To the extent that these individuals are

more fully employed and may earn more than they might have if they had

attended JTPA training, our estimate may actually overstate the effect of

training. Second and more importantly, if those who attend training are in

some way more motivated than those who do not attend, it would be

difficult to separate any increase in earnings due to training from the

increase in earnings due to this motivation. At a minimum, we would need

to identify which control group members were motivated to attend

training to draw such inferences.

12. Labor recommends adjusting the comparison by effectively removing

treatment group members who did not enroll in training.

We chose to compare only those assigned to JTPA training with those not

assigned to training to take full advantage of the original random

assignment design. As we stated in comment 11, we would have needed to

identify which control group members were motivated to attend training

to justify removing the treatment group members who did not attend

training. Since we could not take all the necessary steps to fully implement

Labor’s suggestion, we chose not to make that or any other adjustments.

Page 36 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

13. Labor comments that we will be providing the Department’s contractor

with access to Social Security data for additional analysis, including

examining the results for subgroups.

We would like to clarify the details of this arrangement in light of the

sensitive nature and confidentiality of individual earnings records. When

we began our work, Labor was also planning to evaluate the long-term

impact of JTPA training on earnings. Both our and Labor’s evaluation

(contracted out to Westat, Inc.) planned to use Social Security earnings

records to supplement the information collected through the NJS. In the

spirit of cooperation, Labor requested and we agreed to provide

aggregated earnings data to the Department, which will submit computer

programs to us; we will in turn run the programs and provide the output to

Labor. Only aggregated information, such as means and standards

deviations, will be reported. No data will be released that could be traced

to individuals, nor will we provide Labor or its contractor with individual

earnings records.

14. Labor suggests that we include earnings and employment information

for the third of the sample for whom only 4 years of follow-up data were

available.

We did not include this group in our analysis since we could not report on

their earnings or employment 5 years after training. As such, any

additional information provided would not address the question of

whether JTPA had a long-term effect on the earnings or employment

outcomes of the treatment group.

15. Labor states that we should explain that our employment measure is

not the definition that is generally reported in government statistics and is

not comparable to figures reported in Current Population Survey (CPS) and

Bureau of Labor Statistics (BLS) reports.

Our employment rate differs from measures reported in CPS and BLS

reports but is appropriate for our purposes. Our employment rate is the

number employed divided by all who were in the treatment or control

group, which includes those workers who may have dropped out of the

labor force.15 Since these workers presumably applied for training because

15The CPS counts people at any given time as (1) employed, (2) unemployed, or (3) not in the labor

force—that is, not employed and not seeking employment. The unemployment rate in the CPS is the

number of unemployed divided by the sum of the employed plus unemployed. If an employment rate

were reported, it would be one minus the unemployment rate, or, equivalently, the number employed

divided by the sum of the employed plus unemployed.

Page 37 GAO/HEHS-96-40 JTPA Outcomes
Appendix III 

Comments From the Department of Labor

and Our Evaluation

they intended to keep working, we believe that all workers should be

included in the denominator of the measure. Our measure also counts as

employed everyone who worked during the year, even if they might have

been unemployed for some portion of the year. As such, our measure may

overstate the instantaneous employment outcomes of both the treatment

and control groups relative to figures reported in CPS and BLS reports.

16. Labor states that Social Security data are subject to considerable

revisions in the first year of availability. It believes this calls into doubt

fifth-year estimates for adult men, adult women, and male youths.

While data are often subject to revision, we have no reason to suspect that

the data for those assigned to training in 1988 are materially less reliable

than for those assigned to training in 1987. The fifth year (1993) of

earnings data for those assigned to training in 1988 was extracted from SSA

records in March 1995. An SSA official responsible for updates and

revisions to SSA earnings data said that we could expect the accuracy and

completeness of our extract to exceed 99 percent.

17. Labor states that we fail to recognize the limited relevance of our

findings to current job programs.

While we agree that our evaluation has limitations, we disagree that it has

little relevance to current job programs. We make it quite clear that our

analysis is not nationally representative of JTPA training. Additionally, we

cite many flaws associated with the design and implementation of the

original NJS that limit our analysis. However, no evidence exists to suggest

that job training funded by JTPA and administered at the state and local

level has changed so dramatically since 1989 that our findings are not

relevant to the current program.

(104778) Page 38 GAO/HEHS-96-40 JTPA Outcomes
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100