Job Training Partnership Act: Long-Term Earnings and Employment Outcomes
Released: 1996-04-03T00:00:00

Letter

Background

Enacted in 1982, JTPA is the largest federal employment training program, with titles II-A and II-C intended to prepare economically disadvantaged adults and youths, respectively, for entry into the labor force. JTPA emphasizes state and local government responsibility for administering federally funded job training programs. In fiscal year 1995, JTPA title II-A and II-C programs received approximately $1.6 billion in funding.

JTPA training programs annually provide employment training for specific occupations and services, such as job search assistance and remedial education, to roughly one million economically disadvantaged individuals. Training is provided in local service delivery areas (SDA) through service providers, such as vocational-technical high schools, community colleges, proprietary schools, and community-based organizations. The program objectives are to increase earnings and employment and to reduce welfare dependence for participants of all ages. During the NJS, participation in JTPA involved roughly 3 to 4 months of training at an average cost of about $2,400 per participant.

In 1986, Labor commissioned the NJS to evaluate the impact of JTPA on adults and youths because previous findings on the effects of job training programs had been hampered by poor data and statistical problems. The NJS randomly assigned persons who sought JTPA services, and were eligible for them, to a treatment group or a control group. The treatment group was offered JTPA training, and the control group was not. The study was intended to ensure that the two groups would not differ systematically in any way except access to the program, so any subsequent differences in outcomes could be attributed solely to JTPA.

The study included over 20,000 eligible participants who applied for JTPA services between November 1987 and September 1989 in 16 local SDAs. The study followed up on a sample of people in the treatment and control groups 18 months after assignment and then again at 30 months.

The NJS showed mixed results on the impact of JTPA programs. Adult women assigned to JTPA training had significantly higher earnings than the control group of adult women after 18 and 30 months, but the treatment group of adult men, as well as of both male and female youths, did not have significantly higher earnings than its respective control groups.

Participant Earnings Not Significantly Greater Than Control Group Earnings After 5 Years

Participants assigned to receive JTPA training did not have significantly greater earnings than control group members 5 years after their assignment. For some of the four targeted worker categories—adult men, adult women, male youths, and female youths—treatment group earnings exceeded those of the control group in some of the intervening years, but any statistically significant effects disappeared by the fifth year.

Earnings Outcomes for Adult Men

Annual earnings of adult men increased in each year following assignment for both the treatment and control groups. As shown in figure 1, in the first year after assignment, the average annual earnings of adult men in the treatment group grew from about $4,400 to about $6,900. This group’s earnings continued to rise in the subsequent years, reaching approximately $8,700 in the fifth year after assignment to receive JTPA training. The earnings of adult men in the control group, which did not receive JTPA training, also rose following assignment, but this group’s earnings were less than those of the treatment group for each of the 5 years.

After 5 years, the difference between earnings of the treatment and control groups was not statistically significant. Five years after assignment, the treatment group’s earnings had exceeded those of the control group by approximately $300 to $500 annually, but only in the first 3 years were these differences statistically significant.

Earnings Outcomes for Adult Women

Earnings of adult women showed a pattern similar to those of adult men, increasing in each year after assignment. Figure 2 shows that the annual earnings of adult women assigned to the treatment group increased from approximately $2,800 in the year of assignment to approximately $4,700 in the first year following assignment. This group’s earnings continued to climb, reaching approximately $6,600 in the fifth year. Earnings of adult women in the control group followed a similar pattern, but this group’s earnings were lower than those of the treatment group in each year, reaching approximately $6,200 during the fifth year.

As with the earnings of adult men, 5 years after assignment the difference between the treatment and control groups’ annual earnings was not statistically significant. However, during the first 4 years after assignment, the differences between the treatment and the control groups’ earnings were statistically significant in each year, with treatment group earnings approximately $300 to $600 higher than control group earnings annually.

Earnings Outcomes for Male Youths

The earnings of male youths in the control group, like those of adult men and adult women, increased in each year following assignment. Figure 3 shows that the earnings of male youths in the treatment group increased from approximately $2,900 in the year of assignment to approximately $4,600 in the first year after assignment. This group’s earnings continued to grow during the 5-year period, reaching a high of approximately $7,600 in the fifth year. The earnings of male youths in the control group also rose during the 5-year period following assignment, climbing from approximately $4,800 in the first year to approximately $6,800 in the fifth year.

We found no significant difference between the treatment and control groups’ annual earnings 5 years after assignment. Although the control group’s earnings were higher than the treatment group’s during the first 3 years following assignment, the differences, which ranged from approximately $200 to $400 each year, were not statistically significant. During the fourth and fifth years, the treatment group had higher earnings than the control group, but these differences too were not statistically significant.

Earnings Outcomes for Female Youths

Earnings of female youths showed a pattern similar to that of male youths, growing in each year following assignment. Earnings of female youths in the treatment group rose from approximately $2,000 during the year of assignment to approximately $3,300 in the first year following assignment (see fig. 4). This group’s earnings continued to climb, reaching approximately $5,400 in the fifth year following assignment. The earnings of female youths in the control group also rose during the 5-year period, climbing from approximately $3,400 in the first year to a high of approximately $5,200 in the fifth year.

We found no significant differences between the treatment and control groups’ annual earnings 5 years after receiving their assignments. During the first 2 years following assignment, the control group’s earnings were higher than the treatment group’s, but the differences of less than $100 annually were not statistically significant. In the fourth and fifth years following assignment, the treatment group had earnings of approximately $100 to $300 higher than the control group, but these differences also were not statistically significant.

Employment Rates of Participants Not Significantly Greater Than Control Group Employment Rates After 5 Years

As with earnings, employment rates of those assigned to receive JTPA training were not significantly greater than employment rates of control group members 5 years after assignment. For some of the four targeted worker categories, treatment group employment rates were higher than those of the control group in some years, but any statistically significant effects disappeared by the fifth year.

Employment Rates of Adult Men

The employment rates of both treatment and control group adult men peaked during the calendar year of assignment and then declined in subsequent years, eventually reaching levels lower than those of the men before entering the study (see fig. 5). For example, the employment rate for adult men in the treatment group was 87 percent in the year of assignment. The percent employed declined in the following years, reaching 72 percent by the fifth year following assignment, which was lower than the group’s employment rate of 79 percent in the year before entering the study. The adult men in the control group showed a similar pattern—their employment rate was 87 percent in the year of assignment but dropped to 71 percent in the fifth year after assignment.

After 5 years, the difference between the treatment and control groups’ employment rates was not statistically significant. The treatment group’s employment rates were higher than the control group’s in each year following assignment, although the differences in the employment rates were statistically significant only in the fourth year following assignment.

Employment Rates of Adult Women

The pattern of employment rates of adult women was somewhat similar to that of adult men. The employment rates of adult women were highest during the calendar year following assignment, with 80 percent of the treatment group and 77 percent of the control group employed (see fig. 6). After the first year, however, the employment rates for both the treatment and control groups fell, reaching 69 percent and 67 percent, respectively, in the fifth year following assignment. These rates in the fifth year were also lower than each group’s employment rate in the year before assignment.

We found no significant differences between the treatment and control groups’ employment rates 5 years after assignment. The treatment group’s employment rates exceeded the control group’s in all 5 years following assignment, usually by about 2 to 3 percent, but only in the first 3 years were these differences statistically significant.

Employment Rates of Male Youths

The pattern of employment rates of male youths was somewhat similar to that of adult men and women: the male youths’ employment rates peaked during the calendar year following assignment—reaching nearly 91 percent for the treatment group and over 92 percent for the control group—but then declined (see fig. 7). However, in contrast to the employment rates of adults, those of male youths were slightly higher 5 years after assignment than before assignment, reaching 81 percent for the treatment group in the fifth year, compared with 80 percent in the year before assignment.

We found no significant differences between the treatment and control groups’ employment rates 5 years after assignment. While the employment rates for the control group actually exceeded those for the treatment group in the year of assignment and the first and third years following assignment, none of the differences were statistically significant.

Employment Rates of Female Youths

The employment rates of female youths in both the treatment and control groups peaked during the calendar year of assignment, declined somewhat over the next 4 years, and then slightly increased in the fifth year (see fig. 8). As with those of male youths, the employment rates of female youths were slightly higher 5 years after assignment than before assignment. The employment rates of female youths were 74 percent for the treatment group and 73 percent for the control group in the fifth year following assignment, compared with 71 and 73 percent, respectively, in the year before assignment.

We found no significant differences between the treatment and control groups’ employment rates 5 years after assignment. Employment rates for the treatment group exceeded those for the control group in 4 of the 5 years following assignment, but none of the differences in employment rates were statistically significant.

Conclusions

Though both long-term earnings and employment rates for NJS treatment groups surpassed those for their respective control groups, the differences did not meet our test for statistical significance. Five years after expressing an interest in JTPA-sponsored job training, individuals assigned to participate in the program did not have earnings or employment rates significantly higher than individuals not assigned to participate.

Agency Comments

In commenting on a draft of this report, Labor expressed several concerns. It took exception to what it characterized as unwarranted negative conclusions that are not consistent with the report findings. Labor also took issue with the importance the report places on tests of statistical significance applied to earnings of an individual group in a given year, preferring to emphasize other evidence of the positive effect of JTPA on participant earnings over the 5-year period. Labor also expressed concerns that the report findings have limited relevance to current job training programs.

We believe that our conclusions are well supported by our findings. On several occasions where appropriate, we have noted comparisons favorable to the JTPA treatment groups, including in the “Results in Brief” and “Conclusions” sections. Although other evidence covering the 5-year period might be found to better highlight the positive effects of JTPA training, our research focused on the earnings and employment rates of each target group in the fifth year after applying for JTPA training. Also, we do not believe that current or proposed job training programs sufficiently differ from JTPA training at the time of the NJS to limit the relevance of our report findings.

In its response, Labor enclosed an attachment with specific comments on the report and additional information. This attachment and our evaluation of the comments appear in appendix III. Labor also provided us with technical comments, which we have incorporated in the report where appropriate.

As agreed with your office, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days from the date of this letter. At that time, we will send copies to interested parties and make copies available to others upon request.

This report was prepared under the direction of Wayne B. Upshaw, Assistant Director, who may be reached on (202) 512-7006 if you or your staff have any questions. Gene Kuehneman, Senior Economist, (202) 512-4091, Jill Schamberger, Senior Evaluator, and Thomas L. Hungerford, Senior Economist, were major contributors to this report.

Data Sources and Methodology

To select our sample of individuals assigned to receive training under JTPA and our control group, we used participation information from the National JTPA Study (NJS). We then obtained long-term earnings and employment information for these individuals from SSA. Our analysis compared earnings and employment levels of individuals in the treatment and control groups to determine whether differences between these groups were statistically discernable.

National JTPA Study

The original NJS data set contained demographic and program information on 20,601 people who applied for JTPA services between November 1987 and September 1989 in 16 local service delivery areas. Program applicants were recruited, screened to determine their eligibility, assessed to determine their service needs and wants, and recommended for services. NJS participants were then randomly assigned to either the treatment group, which was allowed to participate in JTPA title II-A programs, or the control group, which was not allowed to participate in these programs for 18 months. Approximately two-thirds of the applicants were assigned to the treatment group and one-third to the control group. The control and treatment groups were closely matched in demographic variables such as age, race, and education, which typically allows a meaningful comparison of average outcomes between the two groups.

However, two factors intervened to make such a comparison problematic. First, not all members of the treatment group participated in JTPA programs. For example, about two-thirds of the adult treatment group members enrolled in JTPA, but the other one-third either found jobs on their own or decided not to participate in the program. Second, a substantial minority of the control group members chose to participate in some alternative, non-JTPA training programs. These complications preclude attributing earnings differences between the two groups solely to JTPA training. Therefore, our findings refer to differences between the treatment and control groups rather than between individuals who did or did not receive JTPA training. Furthermore, we do not know which of the control and treatment group members chose to receive training later than 18 months after assignment.

The NJS was not designed to track treatment or control group members beyond 30 months. Therefore, to calculate and compare longer term earnings and employment outcomes for these groups, we needed information from another source.

Social Security Earnings Records

We obtained annual earnings records from SSA for the individuals in the NJS treatment and control groups. SSA maintains information on annual earnings of individuals contributing to either Social Security or Medicare. We assumed that an individual was employed if his or her SSA records showed positive earnings for a given year. We adjusted data for what we assumed were data entry or processing errors, and we also rounded reported negative earnings to zero.

Data Analysis

We analyzed the NJS and SSA earnings records of 13,699 NJS participants to determine their annual earnings and employment outcomes for the 3 years before assignment to the treatment or control group, the year of assignment, and 5 years following assignment. The 3 years of prior earnings and employment data served to demonstrate the prior comparability of treatment and control groups. The 5 years of postassignment data effectively doubled the 30-month follow-up period for the NJS. The treatment group had 9,275 individuals, and the control group had 4,424 individuals.

We used individual earnings data and calculated means and variances for each of the four target groups—adult men, adult women, male youths, and female youths—to compare the treatment groups’ earnings and employment outcomes with those of the control groups. We tested for differences in earnings and employment outcomes at the 5-percent significance level.

We calculated annual earnings amounts using SSA information on Social Security-covered earnings for nonfederal workers and on earnings covered by Medicare for federal workers. We calculated employment rates as the percentage of each group with positive covered earnings in a calendar year. Individuals with unreported earnings may have had their earnings and employment understated in our analysis. Individuals whose earnings exceeded the Social Security withholding ceiling may also have had their earnings understated in our analysis. These limitations applied to both the treatment and control groups, and we do not believe they affected the two groups differently.

Data and Statistics for Figures

Statistically significant difference at 5% level?

Statistically significant difference at 5% level? no Statistically significant difference at 5% level?

Employment rates (percent employed)

Statistically significant difference at 5% level? (percent employed)

Statistically significant difference at 5% level? no (percent employed)

Statistically significant difference at 5% level? (percent employed)

Comments From the Department of Labor and Our Evaluation

The following are GAO’s comments on the Department of Labor’s letter dated November 30, 1995.

GAO Comments

1. Labor comments that our report understates the gross returns to JTPA training. Furthermore, Labor implies that these gross returns calculations compare favorably with the returns to college education.

Our objective, as clearly stated in the report, was not to evaluate the cost-effectiveness of JTPA training, but rather to determine and compare the long-term effects of JTPA training. In fact, because we did not calculate the gross return to JTPA participants the report cannot have understated or overstated the values. Such calculations were not within the scope of this report. While it may be true that these returns are favorable, we have no basis to judge the favorability of the gross returns to training. 2. Labor states that the report does not acknowledge favorable aspects of this study. Specifically, Labor cited that (1) all four target groups had higher earnings in the fifth year after assignment; (2) both adult treatment groups had higher earnings than their respective control groups in each of the 5 years following assignment; and (3) for male youths, a positive trend exists, and the fifth-year earnings exceed those of the control group by over 10 percent.

Contrary to Labor’s comment, we did note many of these favorable program outcomes in our report. We stated that adult male treatment group members had higher earnings than adult male control group members and presented similar findings for the other three target groups. We further stated that adult male treatment group earnings exceeded control group earnings in each of the 5 years and reported similar information for adult women. Also, we noted the positive trend for earnings of male youths. We did not note the percentage difference for male youths in the fifth year because we did not report percentage comparisons for any of the target groups. 3. Labor states that if the training impacts are accumulated over time during the 5-year follow-up period, the net benefits outweigh the costs.

As we stated in comment 1, our objective was not to evaluate the cost-effectiveness of JTPA training, but rather to determine and compare the long-term effects of JTPA training. While it may be true that the net benefits outweigh the costs, we have no basis to judge this because such calculations were beyond the scope of this report. 4. Labor states that the increase in standard errors is primarily responsible for the decline in statistical significance of the estimated impacts.

While Labor is correct in stating that the standard errors were greater in the fifth year, it is not accurate to attribute a decline in statistical significance to either the estimated training effects or to the standard errors. The test statistics used for our significance tests are determined by the ratio of the estimates to their standard error, and attributing the lack of significance solely to either component of these ratios is inappropriate. 5. Labor states that our conclusion requires assessing the total impact of JTPA and the overall cost and benefits. The Department states that we overemphasize the importance of year-by-year significance tests in questioning the program’s usefulness in improving participants’ long-term earnings prospects by stressing the insignificant effect of JTPA in the fifth year.

We agree with Labor that year-by-year significance tests have limited value in assessing the total impact of JTPA and the overall cost and benefits. Furthermore, our year-by-year significance tests provide statistical evidence that adult treatment group members achieved higher earnings for several years following assignment to JTPA training. While other evidence covering the 5-year period might be found to better highlight the positive effects of JTPA training, we chose to address the question of whether the fifth-year earnings of those assigned to participate in JTPA differed significantly from the fifth-year earnings of those not assigned to participate in JTPA. 6. While acknowledging that the observed earnings differences between the four target groups were not statistically significant in the fifth year, Labor asserts that the odds that all four differences would be positive purely by chance is 6.25 percent.

This implies that an accumulation of not statistically significant observations provides more compelling empirical evidence than the actual significance test for any one group. While the probability (not odds) that all four not significant fifth-year earnings differences would be positive purely by chance might be low, our research question is whether a significant earnings difference occurred for each target group. 7. Labor comments that the report does not report the standard errors. Labor states that the report should include confidence intervals for the estimates, sample sizes, and standard errors and specify significance levels for the estimates.

We have made several additions to tables in appendix II in response to this comment. We have added the sample size, the size of the treatment and control groups, the standard errors, and a reminder that the significance level chosen is 5 percent for the tables in this appendix. Since we have not presented point estimates of the earnings effects, we did not calculate confidence intervals for these estimates. Technical readers of our report can construct such estimates and the associated intervals from the information in the appendix II tables. 8. Labor claims that the report treats figures that are not significant as zero.

We do not report any training effect as zero. The magnitude of the earnings differences, whether significant or not, is discussed in the report and is easily calculated from the tables in appendix II. 9. Labor states that statistical significance is not a knife edge of yes or no but a continuum.

The level used for tests of statistical significance may be chosen from a broad range (or continuum) of values. Although different researchers may choose to use different values for the significance level, choosing a significance level before analyzing any data is common practice. Once this level has been chosen, statistical hypothesis testing very much involves a yes or no decision. Either the data reject the null hypothesis of no training effect at the set significance level or not. We follow these commonly accepted procedures for hypothesis testing, and our convention is to set the significance level for such tests at 5 percent. 10. Labor also states that the report should discuss the results, the probability values, and changes in the significance levels.

Our report does discuss the results as well as whether the earnings and employment effects were significant and whether this significance changed over time. Although we do not present the probability values, technical readers of our report can calculate them using the information in the appendix II tables. 11. Labor takes issue with our statement that complications (not all treatment group members received training and some control group members did receive training) precluded our attributing earnings differences solely to JTPA training. It claims that these complications led us to understate the effect of training, implying that the earnings differences observed, along with perhaps some further overlooked earnings effects, can be attributed to JTPA.

We clearly state that these complications preclude solely attributing the earnings effects to JTPA training. However, we have no evidence that these factors led to an understatement of the effect of JTPA training. In the first place, a short delay can occur before an assignee can begin a training program. In some of these cases, individuals find and accept employment instead of reporting for training. To the extent that these individuals are more fully employed and may earn more than they might have if they had attended JTPA training, our estimate may actually overstate the effect of training. Second and more importantly, if those who attend training are in some way more motivated than those who do not attend, it would be difficult to separate any increase in earnings due to training from the increase in earnings due to this motivation. At a minimum, we would need to identify which control group members were motivated to attend training to draw such inferences. 12. Labor recommends adjusting the comparison by effectively removing treatment group members who did not enroll in training.

We chose to compare only those assigned to JTPA training with those not assigned to training to take full advantage of the original random assignment design. As we stated in comment 11, we would have needed to identify which control group members were motivated to attend training to justify removing the treatment group members who did not attend training. Since we could not take all the necessary steps to fully implement Labor’s suggestion, we chose not to make that or any other adjustments. 13. Labor comments that we will be providing the Department’s contractor with access to Social Security data for additional analysis, including examining the results for subgroups.

We would like to clarify the details of this arrangement in light of the sensitive nature and confidentiality of individual earnings records. When we began our work, Labor was also planning to evaluate the long-term impact of JTPA training on earnings. Both our and Labor’s evaluation (contracted out to Westat, Inc.) planned to use Social Security earnings records to supplement the information collected through the NJS. In the spirit of cooperation, Labor requested and we agreed to provide aggregated earnings data to the Department, which will submit computer programs to us; we will in turn run the programs and provide the output to Labor. Only aggregated information, such as means and standards deviations, will be reported. No data will be released that could be traced to individuals, nor will we provide Labor or its contractor with individual earnings records. 14. Labor suggests that we include earnings and employment information for the third of the sample for whom only 4 years of follow-up data were available.

We did not include this group in our analysis since we could not report on their earnings or employment 5 years after training. As such, any additional information provided would not address the question of whether JTPA had a long-term effect on the earnings or employment outcomes of the treatment group. 15. Labor states that we should explain that our employment measure is not the definition that is generally reported in government statistics and is not comparable to figures reported in Current Population Survey (CPS) and Bureau of Labor Statistics (BLS) reports.

Our employment rate differs from measures reported in CPS and BLS reports but is appropriate for our purposes. Our employment rate is the number employed divided by all who were in the treatment or control group, which includes those workers who may have dropped out of the labor force. Since these workers presumably applied for training because they intended to keep working, we believe that all workers should be included in the denominator of the measure. Our measure also counts as employed everyone who worked during the year, even if they might have been unemployed for some portion of the year. As such, our measure may overstate the instantaneous employment outcomes of both the treatment and control groups relative to figures reported in CPS and BLS reports. 16. Labor states that Social Security data are subject to considerable revisions in the first year of availability. It believes this calls into doubt fifth-year estimates for adult men, adult women, and male youths.

While data are often subject to revision, we have no reason to suspect that the data for those assigned to training in 1988 are materially less reliable than for those assigned to training in 1987. The fifth year (1993) of earnings data for those assigned to training in 1988 was extracted from SSA records in March 1995. An SSA official responsible for updates and revisions to SSA earnings data said that we could expect the accuracy and completeness of our extract to exceed 99 percent. 17. Labor states that we fail to recognize the limited relevance of our findings to current job programs.

While we agree that our evaluation has limitations, we disagree that it has little relevance to current job programs. We make it quite clear that our analysis is not nationally representative of JTPA training. Additionally, we cite many flaws associated with the design and implementation of the original NJS that limit our analysis. However, no evidence exists to suggest that job training funded by JTPA and administered at the state and local level has changed so dramatically since 1989 that our findings are not relevant to the current program.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.