United States General Accounting Office

GAO Report to the Ranking Minority Member,

Committee on Finance, U.S. Senate

April 1995 WELFARE TO WORK

Measuring Outcomes

for JOBS Participants

GAO/HEHS-95-86

GAO United States

General Accounting Office

Washington, D.C. 20548

Health, Education, and

Human Services Division

B-256125 

April 17, 1995

The Honorable Daniel P. Moynihan

Ranking Minority Member

Committee on Finance

United States Senate

Dear Senator Moynihan:

Between 1989 and 1994, federal and state governments spent about

8 billion dollars on the Job Opportunities and Basic Skills Training (JOBS)

program. The program helps recipients of Aid to Families with Dependent

Children (AFDC) obtain the education, training, and services necessary for

employment. For program managers and policymakers to determine

whether this investment has helped achieve the objective of reducing

welfare dependency requires information on JOBS participants’ outcomes,

such as whether they are becoming employed and leaving AFDC. In working

toward welfare reform, information on the extent to which JOBS is

achieving its objectives is more important than ever for the Congress.

This report responds to your request that we study the use of outcome

measurement by the Department of Health and Human Services (HHS) and

states in determining whether JOBS participants are finding employment

and leaving AFDC. Specifically, we addressed the following questions:

(1) What progress has HHS made in measuring the employment and AFDC

status of JOBS participants at the national level and setting national goals

against which program performance will be measured? (2) To what extent

are states measuring participant outcomes and setting performance goals?

(3) What major issues should be considered in establishing a national

approach to measuring JOBS participant outcomes and setting performance

goals?

To assess the progress that HHS has made in establishing outcome

indicators and goals, we interviewed officials from HHS and various welfare

research and interest groups. We also reviewed JOBS regulations and

reports. Using a mail questionnaire, we surveyed JOBS administrators in the

50 states and the District of Columbia to determine state performance

monitoring practices. To determine the issues that should be considered in

developing indicators and goals, we reviewed HHS’ proposed approach to

developing outcome indicators, examined the literature on the

development of performance monitoring systems, and interviewed experts

Page 1 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

in the field. See appendix I for further details on our scope and

methodology.

Results in Brief HHS does not know whether JOBS is reducing welfare dependency because

it does not gather enough information on critical program outcomes, such

as the number of participants entering employment and leaving AFDC

annually. In addition, states are held accountable for the number and type

of participants enrolled in education and training but not for outcomes,

such as the number of participants finding employment. While the current

approach to monitoring performance provides important information on

the activities of JOBS participants, state JOBS directors are concerned that

the approach provides little incentive for states to focus on moving

participants off AFDC and into jobs.

While little progress has been made in monitoring JOBS outcomes at the

federal level, the picture is better at the state level. Nearly all states use

some information on participant outcomes to manage their individual

programs, although the extent to which states monitor outcomes varies

widely. At least in part to demonstrate to their state legislatures that

program objectives are being achieved, a majority of states monitor the

number of JOBS participants entering employment and hourly wages at

hire. In addition, over one-half of the states have established annual

outcome goals. Although many states gather some JOBS outcome data,

without a standard federal approach, few states could provide us with

comparable data. However, our survey of JOBS directors found that 27

states could provide annual data on individuals entering employment. In

these states, about 21 percent of JOBS participants found jobs in 1993.

The current national interest in making welfare more employment

focused, as well as requirements in the Government Performance and

Results Act (GPRA) that performance monitoring become more outcome

oriented governmentwide, indicate a need for HHS to move decisively to

ensure that it meets its current schedule for developing outcome measures

and goals for JOBS. HHS has reported to the Congress that it plans to finalize

JOBS outcome measures by October 1996 and outcome goals by

October 1998. A critical first step in developing performance goals will be

working with the states and other concerned parties to resolve differences

regarding whether the primary objective of JOBS is to help participants

(1) obtain employment quickly or (2) get the education and training

needed for better-paying jobs. Congress is considering whether AFDC and

JOBS should be replaced with a welfare-to-work block grant program that

Page 2 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

includes some JOBS’ objectives and activities. However this issue is

resolved, the need for federal accountability would be well served by

clearly defined program objectives and outcome goals.

Background JOBS, created by the Family Support Act (FSA) of 1988, is designed to help

families avoid long-term welfare dependency. The act requires all states to

establish JOBS programs that make available to AFDC recipients the

education, training, and support services they need to prepare for, accept,

and retain employment. States can provide these services either directly or

through local service providers. Both the states and the federal

government share in the costs of the program. States were required to

begin their JOBS programs by October 1, 1990, and be in full statewide

operation by October 1, 1992. HHS’ Administration for Children and

Families (ACF) is responsible for managing JOBS at the federal level.

To understand whether a human services program, such as JOBS, is

achieving its objectives or in need of improvement requires a system for

gathering information about program performance. Evaluating a program

by regularly collecting and analyzing performance information is known as

performance monitoring. Performance monitoring systems include two

key elements: (1) indicators—which define what performance information

will be gathered and (2) goals—a target level of performance against

which actual program performance will be gauged.

There are two basic types of indicators: process and outcome. Process

indicators for JOBS would provide information about program activities,

such as the number of AFDC recipients participating in JOBS, the number of

JOBS participants receiving training, and the amount of money being spent

on teenage participants. Outcome indicators for JOBS, on the other hand,

would capture what happens to people after participating in program

activities, such as the number of people who begin working, the number

who leave AFDC, and the number still employed after 6 months.

Goals establish the levels of performance that programs are expected to

achieve. For example, the goal for participants starting work could be

“25 percent of those participating in JOBS each year will enter full-time

employment.” Goals can be established for outcomes, such as the

percentage of participants finding employment, and processes, such as the

percentage of participants involved in jobs skills training, for example.

Goals are often accompanied by financial incentives for meeting or

Page 3 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

penalties for not meeting goals whether they are related to outcomes or

processes.

Because the overall goal of JOBS was defined very broadly, states have had

the flexibility to focus on a variety of different objectives in an effort to

achieve the broader purpose. These varying objectives can result in

different approaches for providing JOBS services and different program

results. For example, programs with the objective of quickly increasing

welfare recipients’ earnings may emphasize helping participants find any

job; whereas, pursuing the objective of long-term self-sufficiency may lead

to more of an emphasis on education and training activities with the hope

of placing participants in employment that allows them to move off and

stay off AFDC. If there is no agreement on program objectives, reaching

agreement on the outcome indicators and goals needed to monitor

achievement of the objectives will be very difficult.

FSA mandated that HHS develop outcome goals (known as standards in the

act) for JOBS outcomes over time and established goals for certain

processes. It initially required HHS by October 1993 to recommend JOBS

goals based on specific outcome indicators, such as the number of

participants who obtained jobs and moved off welfare. This requirement

was later amended to allow HHS until October 1994 to develop criteria for

outcome goals for JOBS. Through its funding formula, FSA, in effect, set

minimum goals for two JOBS process indicators: rate of program

participation and target group1

 expenditures. FSA specified that (1) at a

minimum, 20 percent of nonexempt2

 adult AFDC recipients participate in

JOBS in fiscal year (FY) 1995 and (2) 55 percent of JOBS program funds be

spent in each FY on specified target groups. States are held accountable for

meeting both of these process goals and can lose a portion of their federal

funding if they fail.

Recent legislation reinforces the expectation, originally articulated in FSA,

that HHS develop outcome indicators and goals for JOBS. GPRA seeks, among

other objectives, to transform the focus of federal agencies from what they

are doing to what results they are accomplishing. To accomplish this

1

JOBS target group members include AFDC recipients or applicants who have received AFDC for at

least 36 months out of the past 5 years; are under 24 years old and have not completed nor are enrolled

in high school or have little or no work experience for the preceding year; or are a member of a family

in which the youngest child is within 2 years of being ineligible for AFDC because of age.

2

Subject to the availability of state resources, AFDC recipients 16 through 59 years old must participate

in JOBS unless they are exempt. Reasons for exemption include illness or incapacity, working 30 hours

or more per week, attending high school, or caring for children under 3 years old. However, teenage

parents who have not completed high school and have children under 3 years old are not exempt.

Page 4 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

purpose, the act requires agencies to develop 5 year strategic plans

beginning in FY 1998 and annual performance plans beginning in FY 1999.

The strategic plans need to include comprehensive mission statements and

general goals and objectives for the agencies’ major functions. The annual

performance plans, which are based on the strategic plans, should set

specific performance goals for the year. Performance indicators will then

be used to monitor progress toward meeting the goals. By adopting a focus

on outcomes, agency effectiveness and congressional decision-making are

expected to improve.

Outcome indicators are useful to program managers and policymakers in

assessing the status of program operations, identifying areas needing

improvement, and ensuring accountability for end results. Indicators

alone, however, do not show the extent to which the program accounts for

an observed outcome. For example, suppose 25 percent of JOBS

participants become employed in a certain time period. JOBS activities as

well as events outside the program, such as participants’ independent

efforts to find work or an upsurge in the economy, could account for

participants finding employment. Determining the extent to which the

program contributed to the observed outcome involves studies that use

experimental designs to estimate what would have happened without the

program. In this example, to estimate the program’s impact, such studies

might compare the percentage of JOBS participants becoming employed

with the percentage of comparable AFDC recipients becoming employed

without the program. To measure the impact of JOBS, FSA authorized

studies using experimental designs to isolate the actual impact of the

program. Because such evaluations are usually costly, they are done

infrequently and often involve only select locations.

HHS Has Made Little

Progress in

Establishing an

Outcome-Focused

JOBS Performance

Monitoring System

Six years after passage of FSA, HHS only holds state JOBS programs

accountable for participation, not employment. As a result, very limited

national data are available regarding the outcomes of JOBS participants. In

addition, the current approach to performance monitoring does not assist

states in determining whether they are meeting program goals related to

employment and independence from welfare. According to HHS, a

combination of technical and environmental factors has impeded the

development of outcome indicators for JOBS.

Page 5 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

HHS Data Gathering for

JOBS Focuses on

Participation, Not

Outcomes

HHS focuses its JOBS data collection primarily on indicators of participation.

It collects information from all states on the numbers of program

participants, expenditures on target group members, and the activities

individuals are participating in on a monthly basis. States are accountable

for meeting process goals; if they fail to meet these goals the rates at

which state expenditures are matched by federal dollars may be reduced.3

Although HHS has established some outcome indicators, data on these

indicators present a very incomplete picture of JOBS outcomes. HHS data on

job entry show that on an average monthly basis in FY 1993, 8 percent of

JOBS participants entered employment.4 In addition to job entry, HHS

gathers data on hourly wages and whether an individual stopped receiving

AFDC due to increased income from working. An ACF official told us that

because states are not held accountable for outcomes, neither the states

nor HHS pays much attention to the monthly outcome data submitted.

The usefulness of HHS’ outcome indicators as tools to help manage the

program is limited for a number of reasons. Because of this approach to

gathering information, HHS cannot answer important questions regarding

whether participants are becoming self-sufficient. To measure

participants’ activities, including education, training, job search, and

employment, HHS gathers data each month on a sample of JOBS participants

who (1) took part in any JOBS-sponsored activity in that month or

(2) became employed in the sample or preceding month. The sample,

therefore, excludes anyone who has been employed for more than 2

months or did not participate in a JOBS activity in the sample month. This

approach to sampling is designed to measure participants’ current

employment-related activities, not outcomes related to whether

participants remain employed and move off AFDC as a result of their

earnings. To measure such outcomes, HHS would need to track individual

participants across time. In addition, current measurement approaches do

not yield annual statistics—a common indicator of program

performance—on the percentage of JOBS participants who became

employed. HHS also believes that the quality of some of the data is poor.

For example, an ACF official told us that the data on hourly wages are

unreliable because they are missing in many cases and often entered

incorrectly.

3

For example, if a state failed to meet the JOBS participation goal of 15 percent of adult nonexempt

AFDC recipients in FY 1994, the federal matching rate in FY 1995 could be reduced from 90 percent to

50 percent (for expenditures up to an amount equal to the state’s Work Incentive (WIN) program

allotment for FY 1987).

4

HHS began collecting data on job entry in FY 1992. The most recent year for which data are available

is FY 1993.

Page 6 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

Current Monitoring

Approach for JOBS Does

Not Support an Emphasis

on Employment and

Reducing Welfare

Dependence

Our 1994 survey found that state JOBS directors believe that little progress

has been made in establishing a performance monitoring system that

supports achieving program goals related to employment and

independence from welfare. Eighty-two percent of JOBS directors we

surveyed believe that HHS has not sufficiently moved to focus JOBS on

outcome measurement. Our survey also found that over one-half of state

JOBS directors believe that the data gathered on participation rates and

target group expenditures are of little or no use in helping states achieve

their employment and training program goals. Over one-half of the states

believed that the nature of federal reporting requirements actually

hindered their abilities to collect data on outcome indicators. In addition,

ACF reported in May 1994 that while it spends a significant amount of time

and resources on monitoring for JOBS and other programs, performance

monitoring at ACF is in a “state of crisis,” in part because the system does

not provide a means for determining if programs are producing the desired

outcomes.5

These beliefs echo concerns about the absence of outcome data voiced in

1989 in response to HHS’ notice of proposed rule making (NPRM) for JOBS. In

its NPRM for JOBS, HHS originally included no outcome indicators. Numerous

commenters on the NPRM, however, favored the addition of outcome data.

Some of them believed that an excessive emphasis on participation would

drive program operations toward meeting goals not necessarily related to

achieving independence from welfare through employment. In response to

the NPRM comments, HHS amended its proposed regulations to include

some outcome data related to job entry stating that these data should be

included

“since employment as a means to self-sufficiency and independence from welfare is the

objective of the JOBS program.”

However, HHS chose not to include additional outcome measures at that

time, in part, because it agreed with one commenter’s concern that

outcome data do not show the extent to which the program accounts for

the observed outcomes.

HHS’ lack of an outcome-focused performance monitoring system also

limits the possibilities for information sharing and coordination with other

employment and training programs, such as the Job Training Partnership

Act (JTPA). At the local level, JOBS’ and JTPA’s services are often combined

5

Report of the Administration for Children and Families’ Monitoring Team, Administration for Children

and Families, Department of Health and Human Services (May 1994).

Page 7 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

to meet the education, training, and support service needs of AFDC

recipients. Close coordination is necessary between the programs to

facilitate effective service delivery. In 1992, the National Governors’

Association reported that a majority of the JTPA administrators they

surveyed believed that one barrier to effective coordination was the lack

of consistency between JTPA’s outcome goals and JOBS’ process reporting

requirements.6

HHS Has Encountered

Barriers to Developing an

Outcome-Focused

Performance Monitoring

System

While FSA set process goals, it also required HHS to develop and submit

recommendations for outcome goals to the Congress by October 1993.

These recommendations were to include goals for increased earnings and

reduced welfare dependency. HHS missed the October 1993 deadline, but

submitted its report on September 30, 1994.7

 This report identified

problems in developing an outcome-focused performance monitoring

system and provided a detailed plan and schedule for developing outcome

indicators and goals.

In its report to the Congress, HHS identified several technical and

environmental factors that contributed to delays in the issuance of

recommendations for outcome goals by the October 1993 deadline. HHS

reported that appropriate outcome indicators had proven difficult to

define in part due to disagreements among key stakeholders, such as

researchers; congressional staff; and federal, state, and local officials,

regarding the primary objectives of the JOBS program. In addition, setting

goals was complicated by possible unintended program effects, such as

programs focusing on the most employment-ready individuals in order to

meet goals. HHS also reported that turbulence in the welfare system—for

example, funding shortfalls and caseload growth—made it difficult to

focus the necessary attention and resources on developing outcome goals.

HHS also wanted to ensure that proposed goals were compatible with

welfare reform plans being developed by the new administration.

6

JTPA and JOBS: Coordination and Other Issues, National Governors’ Association (Washington, D.C.:

October 1992)

7

In legislation passed in late October 1994, the reporting requirement was amended to allow HHS to

submit a report to the Congress by October 1, 1994, on criteria for the development of outcome goals

for JOBS.

Page 8 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

Many States Active in

Monitoring JOBS

Program Outcomes,

but Have Mixed Views

About Setting

Nationwide Goals

In contrast to the relatively slow progress at the federal level, many states

have been active in developing outcome indicators to monitor JOBS

participant outcomes. To a large extent, this activity has grown out of each

state program’s efforts to demonstrate its effectiveness and garner support

for additional state funding. According to our survey results, states use a

variety of outcome indicators, relying most often on the number of JOBS

participants entering employment and less frequently on job retention rate

and reductions in AFDC payments. Over one-half of the states have also

established goals for their outcome indicators. Appendix II contains a

copy of our survey questionnaire and results for selected questions.

Although many states use their own outcome goals and support

establishing national goals, they have concerns about how these goals will

be set and used. They maintain that HHS may not be able to adequately

control for differences across states in local economic conditions and

client characteristics that can affect how successful programs are in

placing participants in jobs. They are also concerned that certain outcome

indicators will automatically favor certain state programs and unduly

influence program design decisions, which they believe should be

maintained at the state level.

States Use Various

Indicators to Monitor JOBS

Outcomes

Despite the absence of a federal approach to collecting JOBS outcome data,

many state programs have been active in tracking JOBS outcomes at the

state level. Our survey of state JOBS directors showed that states use a

variety of indicators to measure outcomes. The two indicators that states

use most often are the number of JOBS participants entering employment

and their hourly wages at hire (see table 1).

Table 1: Outcome Indicators Used by

States

Indicator

Number of states using

indicatora

Participants entering employment 49

Hourly wages at hire 42

Participants no longer receiving AFDC due to employment 33

Job retention rate 26

Participants with reductions in AFDC due to employment 24

Educational/training achievement 24

aBase is 50 and includes 49 states who responded to our survey and the District of Columbia.

Page 9 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

As shown in figure 1, almost all states reported that they collected data on

the number of participants entering employment during FY 1993 and

responded that this is one of the most important indicators to use to

monitor JOBS outcomes. A relatively large number of states also collected

data on hourly wages at hire. However, slightly fewer states favored using

hourly wages at hire as an outcome indicator. Several states expressed

concerns that hourly wages at hire were more a reflection of local

economic conditions than an outcome of the JOBS program. One state

official also noted that measuring hourly wages at hire could discourage

programs from placing participants in low-wage entry-level positions,

which often serve as stepping stones to higher paying positions.

Figure 1: Percent of States Using and

Favoring Various Outcome Indicators Number of States

0

10

20

30

40

50

Entered employment

Hourly wages at hire

No AFDC due to employment

Job retention rate

Reduced AFDC due to employment

Educational/training achievement

Performance Indicators

Indicators Used by the States

Indicators Favored by the States

Page 10 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

In comparison to employment entry and wages, only two-thirds of states

collect data on the number of participants no longer receiving AFDC due to

employment. However, as shown in figure 1, a relatively large number of

states reported that they believe this indicator should be used to monitor

program outcomes but do not collect the information because of the

difficulty and resources required to obtain it.

Three other outcome indicators—job retention rate, the number of

participants with reductions in AFDC due to employment, and educational

achievement—are used by almost one-half of the states. Many more states,

however, favored using these three indicators. While the survey results

show that state officials believe that job retention rate and changes in AFDC

benefits are particularly useful outcomes to monitor, tracking AFDC

recipients after case closure is difficult, time-consuming, and expensive,

according to several state officials. Staffing limitations and the inability to

locate ex-AFDC clients were two reasons they cited for not pursuing these

indicators more aggressively. Several states also explained that resources

allocated to data collection are limited and that they are bound to first

comply with federal reporting requirements. One state official doubted the

program would be given the authority or resources to gather additional

data without a federal mandate to do so.

States also have relied on different approaches to measure outcomes. For

example, to measure the employment rate of JOBS participants, some states

tracked JOBS participants over time and maintained a count of individual

JOBS participants obtaining or retaining employment over a year. Other

states performed periodic studies to determine how many JOBS

participants were working and for how long. Other states did no more than

collect the monthly caseload data required by HHS.

Given the variation in how states measure employment rates, determining

the rate at which JOBS participants are finding employment on a national

basis is difficult. However, as part of our survey of state JOBS program

directors, we asked states to provide the number of JOBS participants who

had obtained employment during FY 1993. Based on the responses from the

27 states able to provide data in the format requested, approximately

21 percent of JOBS participants entered employment during FY 1993.8

 We

also asked states to provide data on the number of JOBS participants

8

Of the 50 states, 27 provided us with total number of JOBS participants and the number who had

entered employment during the year. These 27 states represent 54 percent of the average monthly

participants in the JOBS program. Because we do not have a national count of the total number of

JOBS participants during the year, we could not determine the percent of the total number of

participants in a year represented by these states.

Page 11 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

retaining their jobs for 3 months. For the nine states reporting this

information, 33 percent of JOBS participants who entered employment

retained their jobs for at least 3 months.9

While states have taken different approaches to measuring outcomes, their

interest in outcome measurement appears high. As mentioned earlier,

according to our survey, 82 percent of states indicated HHS has not done

enough to establish outcome indicators for JOBS. Officials in several states

emphasized that they need to establish outcome indicators to provide their

state legislatures with information about JOBS participants outcomes. Some

states were also disappointed that HHS had not introduced outcome

indicators earlier when states were implementing their JOBS data collection

systems so that they would not have to modify their systems later to meet

federal reporting requirements.

Majority of States Support

Establishing Outcome

Goals, but Want

Substantial Control Over

Their Development and

Use

While states recognize the need and value of outcome indicators in

managing the JOBS program, their views on establishing outcome goals are

mixed. According to our survey, 29 states had established at least one goal

at the state level for fiscal year 1993 (see fig. 2). Of these, five states

reported that they formally adjusted the performance goals to account for

local differences in client characteristics or the availability of employment.

9

These nine states represent 16 percent of the average monthly number of JOBS participants.

Page 12 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

Figure 2: States With Outcome Goals

State reported not setting outcome goals, except Iowa which did not respond to survey.

State reported establishing at least one outcome goal.

State reported formally adjusting outcome goals for local differences.

Oregon

Tennessee

Maryland

Connecticut 

Vermont

As shown in table 2, of the 29 states with established goals, 27 reported

that they did so for the number of participants entering employment. A

substantial number of these states also had established goals for hourly

wages earned at the time of hire, job retention rate, and

educational/training achievement.

Page 13 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

Table 2: Outcome Goals Used by

States

Outcome indicator

Number of states with

goalsa

Participants entering employment 27

Hourly wages at hire 16

Educational/training achievement 14

Job retention rate 12

Participants no longer receiving AFDC due to employment 9

aBase is 29 states that reported establishing at least one goal.

Several states have been particularly active in developing and using

state-level outcome goals. For example, North Carolina has established

goals for several outcome indicators, including (1) the percentage of JOBS

participants obtaining employment, (2) AFDC closures or reductions due to

earnings, and (3) the percentage of JOBS participants returning to the AFDC

rolls. New Mexico has recently started funding programs based on

performance; its JOBS program will receive state funding based on how

well it does in meeting established goals. Similarly, California has recently

undertaken an initiative to allocate to counties a portion of state funding

based on performance against designated outcome goals. A recent study

by the American Public Welfare Association’s (APWA) Institute for Family

Self-Sufficiency also confirmed that more states are establishing outcome

goals as mechanisms for managing and improving their programs.10

A majority of states also support the establishment of nationwide outcome

goals, although many states are concerned about how goals will be set and

used at the federal level. Over 90 percent of the states responding to our

survey indicated that they would like the flexibility to establish their own

goals or to choose their goals from a menu established by the federal

government. During follow-up interviews, state officials emphasized that

they believed certain outcome indicators would favor particular state

approaches to implementing the JOBS program. For example, programs

that invest more in education and training would benefit from being

judged on education and training achievement or hourly wages at hire,

while programs focusing on early initial job search would probably fare

10APWA’s report, Measuring Client Success: Six States Report on Efforts to Assess What Happens to

Clients After They Receive JOBS Services is based on case studies of Kansas, Maryland, North

Carolina, Oregon, Pennsylvania, and Wisconsin. APWA’s results were similar to GAO’s. APWA found

that each of the six states used job placement as well as a variety of other performance goals to

manage its programs. In all six states, performance goals were used to publicize the achievements of

the JOBS program, hold contractors accountable to specific goals, and facilitate improvements in

service delivery. Three states also used goals to determine funding for local programs.

Page 14 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

better being judged on job entry rate. In either case, states agreed that they

wanted to retain the flexibility to design their own JOBS programs.

State views were mixed on whether federal funds should be linked to

meeting national outcome goals. Over 40 percent of the states were against

linking outcome goals to federal funding, while nearly 30 percent were in

favor of doing so. Several state officials did not believe that federal

funding should be tied to national goals because they doubted that HHS

could sufficiently control for differences in economic conditions and client

characteristics across various geographical regions.

Establishing Outcome

Indicators and Goals

Is Critical to Making

JOBS More

Employment-Focused

Establishing effective outcome indicators and goals is critical to

sharpening JOBS’ focus on the ultimate goals of employment and

independence from welfare, whether JOBS remains the same or is replaced

with a welfare-to-work block grant program that includes some JOBS’

objectives and activities, as has been proposed. Current congressional and

public interest in welfare reform as well as GPRA requirements indicate a

need for HHS to move decisively to establish national leadership regarding

outcome measurement for JOBS. Before effective outcome indicators and

goals can be established, important differences among stakeholders

regarding the objectives of JOBS will have to be resolved.

HHS Plans to Add

Outcome Indicators to the

JOBS Performance

Monitoring System and

Modify the System’s

Process Indicators

In its September 1994 report to the Congress, HHS acknowledged the value

of and affirmed its commitment to using outcome measurement in its

performance monitoring system for JOBS. Specifically, HHS plans to develop

outcome indicators and goals and refine existing process indicators and

goals. In addition, HHS plans to modify the AFDC Quality Control system11 by

adding key process indicators, such as participation rates.

In developing outcome indicators and goals, HHS faces a complex and

difficult task. HHS recognizes that it must ensure that indicators and goals

help the program achieve its objectives and meet the needs of numerous

stakeholders, including local service providers, state and federal

managers, and policymakers. In its 1994 report to the Congress, HHS

identified and laid the groundwork for addressing a number of critical

design and implementation issues that must be addressed to ensure that

indicators and goals support program objectives, are fair to all states, and

avoid unintended program consequences. These issues include

11The states and HHS use the Quality Control system to evaluate whether AFDC payments are made

accurately and to determine how well states comply with regulations to prevent waste, fraud, and

abuse.

Page 15 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

(1) developing a process to ensure states are given an equal opportunity to

meet standards by adjusting for differences among states (that is, levelling

the playing field), (2) designing strategies to discourage states from

serving only the easiest-to-serve clients, and (3) selecting data collection

approaches that are feasible and cost-effective.

To develop outcome indicators, HHS plans to convene a working group

composed of representatives from the Congress, HHS, the Department of

Labor, the states, AFDC recipients, community-based organizations, and

others. This group will convene by April 1995 and make recommendations

to HHS on specific outcome indicators and methods for data collection by

January 1, 1996. Proposed indicators are to be published in the Federal

Register no later than April 1996 and finalized by October 1, 1996. Using a

similar process, current plans call for outcome goals to be developed and

finalized by October 1, 1998.

HHS supports continued use of existing JOBS process indicators but believes

that changes are needed to process goals to make them more effective. In

addition, in its September 1994 report to the Congress, HHS proposed

changing the process goal from the current 20 percent rate to a rate

between 45 and 55 percent. HHS also suggested changing target-group goals

to achieve higher levels of participation among the youngest AFDC parents.

Congressional and

National Interest and

GPRA Add Urgency to the

Need for HHS to Establish

Outcome Indicators

Recently, there has been strong congressional and national interest in AFDC

becoming more focused on helping recipients become employed and leave

AFDC in a limited time period. Numerous bills to reform AFDC and JOBS were

introduced in the 103rd Congress and more are likely to be introduced in

the 104th. Several recent welfare reform bills would replace AFDC and JOBS

with a welfare-to-work block grant program. To the extent that JOBS

objectives and activities are retained in the block grant, outcome

indicators and goals for JOBS would be useful in ensuring accountability

and improving congressional oversight.

While FSA, which originally required HHS to develop outcome goals only for

JOBS, was limited to one program, GPRA requires all federal agencies to

develop strategic plans, annual performance plans, and annual

performance reports. Strategic plans articulate the agency’s essential

mission, long-term general goals and objectives, and a plan of action for

achieving the objectives. The annual performance plans, by establishing a

set of performance indicators and goals, provide a link between the

agency’s longer-term objectives and what managers and staff must

Page 16 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

accomplish on a daily basis to achieve those objectives. GPRA requires that

agencies must submit to Congress 5-year strategic plans beginning in FY

1998 and annual performance plans beginning in FY 1999. The performance

plans are expected to cover each program activity set forth in the agency’s

budget.

Currently, over 70 agencies and programs are involved in pilot projects for

GPRA. The National Academy of Public Administration (NAPA) recently

reviewed these pilots and found that some agencies had made limited

progress in developing their plans due to insufficient preparation time.

NAPA recommended that all programs in an agency, not only those

currently operating as pilots, begin as soon as possible to develop strategic

and performance plans so that the FY 1998 and FY 1999 deadlines could be

met.

If JOBS activities are retained within an AFDC block grant, a system of

accountability for the end results of these activities would still be needed.

GPRA and the administration’s National Performance Review (NPR) promote

the use of outcome indicators for all programs, including those funded

through block grants. To date, however, data collected by states under

most block grants have focused on process indicators such as the number

of clients served. In addition, past block grant programs have not often

gathered consistent information on program activities and outcomes to

support congressional oversight. In reviewing data collection under block

grants, we found that, among other things, national leadership in directing

the development of model data-gathering criteria could increase data

comparability and, as a result, oversight.

Clarifying Program

Objectives Is First Step in

Establishing Indicators and

Goals

HHS and performance monitoring system experts agree that the first critical

step in developing outcome indicators and performance goals is to reach

agreement among stakeholders, such as the Congress; researchers; and

federal, state, and local officials, regarding the objectives of the program.

According to HHS, disagreements among stakeholders about the objectives

of the JOBS program have been a major obstacle to developing JOBS

outcome indicators. Difficulty clarifying JOBS objectives may again prove

to be one of the biggest obstacles in the effort to establish outcome

indicators and goals.

In our survey of state JOBS directors, we found some disagreement

regarding the programs’ overriding objectives. Eighty percent of the

directors responded that the overriding objective was to prepare and place

Page 17 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

participants in employment that allows them to move off and stay off AFDC.

The other 20 percent stated that their objective was to get participants

employed in any job, part- or full-time, even if the job might not allow

them to move off AFDC. These two objectives, although consistent with the

overall objectives of FSA, would likely produce differently designed JOBS

programs and different short-term results, thus making the establishment

of appropriate outcome goals difficult.

Conclusion Program managers, policymakers and other stakeholders need to know

whether JOBS participants are finding employment and leaving AFDC. Very

little is known nationally about the outcomes of JOBS participants because

HHS has not moved aggressively on developing an outcome monitoring

system. Many believe that establishing effective outcome indicators and

goals is critical to refocusing JOBS on the ultimate goals of employment and

independence from welfare. Effective indicators and goals would also help

ensure accountability for achieving these critical outcomes should AFDC

and JOBS be replaced with a welfare-to-work block grant program that

includes some JOBS’ objectives and activities. The states have a strong

interest in outcome measurement and are aggressively pursuing a variety

of approaches in this area. State efforts provide a rich well of experience

that can be drawn on in developing a national approach to measuring JOBS

outcomes. To meet the requirements of FSA and GPRA, HHS must move

decisively to establish national leadership regarding outcome

measurement for JOBS.

Agency Comments In its March 28, 1995, comments on our draft report, HHS generally agreed

with our conclusion that its data are incomplete and focused on

process-oriented goals. However, HHS believes that we did not

(1) adequately portray the difficulty of developing an outcome-focused

performance measurement system or (2) give adequate weight to the

importance of certain technical issues or the progress that HHS has made in

addressing those issues (see app. III). We added language in the report to

more explicitly recognize the difficulty of the task and HHS’ progress in

identifying important technical issues (see p. 15). HHS also suggested minor

technical revisions to the draft, which we incorporated as appropriate.

Page 18 GAO/HEHS-95-86 JOBS Outcomes
B-256125 

Our work was performed between January 1994 and February 1995, in

accordance with generally accepted government auditing standards, with

the exception that we did not check the accuracy of outcome data

reported by HHS and the states.

We are providing copies of this report to the Secretary of Health and

Human Services, other HHS officials, and state JOBS program

administrators. We will also make copies available to other interested

parties upon request. Major contributors to this report are listed in

appendix IV. If you have questions about this report, please call Robert

MacLafferty on (415) 904-2000.

Sincerely yours,

Jane L. Ross

Director, Income Security Issues

Page 19 GAO/HEHS-95-86 JOBS Outcomes
Contents

Letter 1

Appendix I 

Scope and

Methodology

22

Appendix II 

GAO Questionnaire

Regarding JOBS

Program

Characteristics

24

Appendix III 

Comments From the

Department of Health

and Human Services

40

Appendix IV 

Major Contributors to

This Report

43

Tables Table 1: Outcome Indicators Used by States 9

Table 2: Outcome Goals Used by States 14

Figures Figure 1: Percent of States Using and Favoring Various Outcome

Indicators

10

Figure 2: States With Outcome Goals 13

Page 20 GAO/HEHS-95-86 JOBS Outcomes
Contents

Abbreviations

ACF Administration for Children and Families

AFDC Aid to Families With Dependent Children

APWA American Public Welfare Association

FSA Family Support Act

FY fiscal year

GPRA Government Performance and Results Act

HHS Department of Health and Human Services

JOBS Job Opportunities and Basic Skills Training

JTPA Job Training Partnership Act

NAPA National Academy of Public Administration

NPR National Performance Review

NPRM notice of proposed rule making

WIN Work Incentive program

Page 21 GAO/HEHS-95-86 JOBS Outcomes
Appendix I 

Scope and Methodology

To determine the progress HHS had made in establishing outcome

indicators and goals and the issues relevant to their development, we

interviewed officials from HHS’ Administration for Children and Families,

which is responsible for the JOBS program at the federal level. We also

reviewed (1) the data-reporting procedures for the JOBS program, (2) HHS

reports that summarize the outcome data collected at the federal level, and

(3) various reports on the status within ACF of monitoring and developing

outcome measures. We did not verify the accuracy of federal data, but

were told by HHS officials that the data were not complete or accurate. In

addition, we reviewed the welfare-to-work and performance measurement

literature and spoke with officials from various welfare research and

interest groups, including APWA, the National Governors’ Association, and

the Manpower Demonstration Research Corporation.

To determine state performance measurement practices, in May 1994, we

surveyed JOBS program administrators in the 50 states and the District of

Columbia. We also examined 1993 and 1994 reviews of state JOBS programs

conducted by ACF’s regional offices and obtained available reports on

outcome data produced by the states.

In our survey, we used a mail questionnaire to collect information on FY

1993 general program characteristics, the use of performance indicators

and goals at the state level, and state preferences regarding the

development of nationwide indicators and goals. We also requested

information for selected outcome data elements for FY 1993. We received

survey responses from the District of Columbia and all 50 states except

Iowa. However, no respondents could provide complete responses to our

request for annual, unduplicated outcome data for FY 1993, even though

many states reported that they monitor some outcome measures.

To obtain additional information and determine why states did not provide

requested outcome data, we conducted follow-up telephone interviews

with officials in 10 states: Alaska, Arkansas, California, Idaho, Louisiana,

New Jersey, New Mexico, Nevada, New York, and Washington. We first

identified states that reported that they monitor participants entering

employment but were unable to provide an annual, unduplicated count of

JOBS participants entering employment for FY 1993 (as we had requested in

our survey). Among these states, we then selected 10 to contact, which

included states with large, medium, and small caseloads. Based on our

follow-up work, we determined that six states could not provide annual,

unduplicated data because the data were not available. In the other four

Page 22 GAO/HEHS-95-86 JOBS Outcomes
Appendix I 

Scope and Methodology

states, the data were available but reporting them would have required

significant time or resources.

Page 23 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

Our survey, sent to state JOBS administrators, contained questions on

general program characteristics, outcome indicators, outcome goals, and

participant outcome data. This appendix includes the full text of the

survey and the aggregate responses for the first three sections of the

survey. The number cited next to each response category is the number of

states that responsed.

The appendix does not include the responses regarding participant

outcome data. Many states were not able to provide the total number of

individuals served by their programs in fiscal year 1993 and even fewer

states could answer questions about participant outcomes in the format

that we requested. As a result, we have not annotated the participant

outcome data section of the survey with the incomplete data reported to

us.

Page 24 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

U.S. GENERAL ACCOUNTING OFFICE

Survey of JOBS Program Directors Regarding

Performance Standards and Participant Outcomes

INTRODUCTION

The U. S. General Accounting Office is conducting a study

of performance standards and participant outcomes related to

the Family Support Act of 1988. Specifically, we are

interested in JOBS performance monitoring practices in your

state and participant outcome data from your JOBS program.

We are asking you to complete this questionnaire as part of

a survey of all 50 state JOBS programs. The survey data

will be used to provide a national picture of outcomes for

the JOBS program. We will not use the data to compare

performance among states.

INSTRUCTIONS

This questionnaire should be completed by the person who is

most knowledgeable about performance monitoring and

participant outcomes in your JOBS Program.

Please respond to the following questions for federal fiscal

year 1993 (October 1, 1992 through September 30, 1993),

unless otherwise noted. If your records are not organized by

federal fiscal year, please respond for your state’s fiscal year

1993.

Because some terms and their usage may vary across

institutions, we have provided a glossary of terms that we

will be using in the questionnaire. For example, the

glossary defines outcome data and performance indicators.

For your convenience, the glossary, listing the terms in

alphabetical order, is on the inside cover of this

questionnaire.

If you have any questions about this questionnaire, please

call Steve Secrist in our San Francisco office, at

(415) 904-2236. He will be glad to help you.

Please return the questionnaire in the enclosed pre-addressed

business reply envelope within 14 days of receipt. If the

envelope is misplaced, please return your questionnaire to:

Steve Secrist

U.S. General Accounting Office

301 Howard St., Suite 1200

San Francisco, CA 94105

Before mailing your completed questionnaire, please make a

copy that you can refer to should we call to ask for

additional information.

Thank you for your assistance.

Please provide the information below about the person

whom we may contact regarding the completion of this

questionnaire.

Name: ___________________________________

Title: __________________________________

Agency: __________________________________

Phone

Number: _________________________________

24

Page 25 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

GENERAL PROGRAM CHARACTERISTICS

1. In FY 93, was your JOBS program state-administered

or county (locally) administered? (CHECK ONE)

1. [ 39 ] State-administered

2. [ 11 ] County (locally) administered

2. In FY 93, at what point in your JOBS program were

most participants required to engage in a job search?

(CHECK ALL THAT APPLY)

By participants we mean individuals who participated

in at least one hour of approved JOBS component

activities. Approved JOBS activities are those which

states can provide according to federal regulations.

1. [ 5 ] Upon first involvement with JOBS

2. [ 36 ] After completion of formal assessment, if

determined job ready

3. [ 18 ] After completion of an education activity,

but before completion of the participant’s

employability plan

4. [ 21 ] After completion of a training activity, but

before completion of the participant’s

employability plan

5. [ 19 ] After completion of a work-related activity,

but before completion of the participant’s

employability plan

6. [ 25 ] After completion of all activities in the

participant’s employability plan

7. [ 11 ] After a determination that an individual is

not making satisfactory progress in an

education/training component

8. [ 1 ] Never required

9. [ 12 ] Other (PLEASE SPECIFY)

________________________________

10. [ 0 ] Don’t know

3. In FY 93, had most county (local) JOBS programs in

your state established a minimum education level at

which they considered participants ready for

employment? (CHECK ONE)

1. [27 ] Yes (CONTINUE)

2. [ 23 ] No (GO TO QUESTION 5)

3. [ 0 ] Don’t know (GO TO QUESTION 5)

4. What is the minimum educational level at which the

largest number of your county (local) JOBS programs

considered participants ready for employment?

(CHECK ONE)

1. [ 10 ] Literacy level at grade 8.9

2. [ 13 ] High school diploma or GED equivalent

3. [ 4 ] Other (Please specify)

________________________________

5. In general, when your state prepared JOBS participants

for employment in FY 93, which of the following was

more similar to your program’s overriding goal?

(CHECK ONE)

1. [ 10 ] To get participants employed in any job,

part-or full-time, even if the job might not

allow them to move off AFDC

2. [ 40 ] To prepare and place participants in

employment that allows them to move off

and stay off AFDC

25

Page 26 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

6. In most of the county (local) JOBS programs in your

state, what, if anything, happened to mandatory

participants who refused to participate in a component

activity in FY 93? (CHECK ONE)

The initial refusal ...

1. [ 39 ] started the sanction and reconciliation

process

2. [ 6 ] had no immediate consequences, but

subsequent refusals started the sanction and

reconciliation process

3. [ 2 ] had no consequences

4. [ 3 ] Other (PLEASE SPECIFY)

__________________________________

7. Consider the services your state provided to JOBS

participants in FY 93. During the course of providing

these services, how often, if ever did your state

communicate to its JOBS participants that the ultimate

goal of the JOBS program is employment? (CHECK

ONE)

1. [ 18 ] Very often

2. [ 24 ] Often

3. [ 3 ] Sometimes

4. [ 1 ] Rarely, if ever

5. [ 4 ] Don’t know/No basis to judge

8. In FY 93, to what extent, if any, did the JOBS program

in your state emphasize to JOBS workers that their role

was to prepare JOBS participants for employment?

(CHECK ONE)

1. [ 15 ] To a very great extent

2. [ 23 ] To a great extent

3. [ 8 ] To a moderate extent

4. [ 4 ] To some extent

5. [ 0 ] To little or no extent

6. [ 0 ] Don’t know

9. In FY 93, to what extent, if any, did the JOBS program

in your state emphasize to JOBS participants that they

were obliged to participate in JOBS program activities?

(CHECK ONE)

1. [ 21 ] To a very great extent

2. [ 22 ] To a great extent

3. [ 5 ] To a moderate extent

4. [ 1 ] To some extent

5. [ 1 ] To little or no extent

6. [ 0 ] Don’t know

10. To what extent, if any, has the JOBS program in your

state moved the welfare system from a focus on

income maintenance to one promoting work and selfsufficiency? (CHECK ONE)

1. [ 11 ] To a very great extent

2. [ 6 ] To a great extent

3. [ 17 ] To a moderate extent

4. [ 14 ] To some extent

5. [ 2 ] To little or no extent

6. [ 0 ] Don’t know

26

Page 27 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

PERFORMANCE INDICATORS

11. In FY 93, did your state collect data on each of the

following performance indicators to monitor the

outcomes of your state’s JOBS program? (CHECK

YES OR NO FOR EACH)

Performance indicators are tools to monitor client

outcomes in the JOBS program. By monitoring we

mean that data are used by managers to assess program

results.

PERFORMANCE

INDICATORS

Yes

(1)

No

(2)

1. Educational/training achievement, such

as graduation rates

24 26

2. Number of participants with reductions

in AFDC grants due to employment

24 26

3. Number of participants no longer

receiving AFDC due to employment

33 17

4. Number of participants entering

employment

49 1

5. Number of participants entering

employment that provides health

insurance coverage

16 34

6. Hourly wages earned at time of hire 42 8

7. Expected weekly earnings at time of

hire

11 39

8. Hourly wages earned at a specified

time period after initial hire

11 39

9. Job retention rate after a specified

length of time

26 24

10. Rate of return to AFDC, after a

specified length of time, for those who

left AFDC due to employment

14 36

11. Other (PLEASE SPECIFY)

_________________________

_________________________

8 42

12. What are the most important performance indicators

that should be used to monitor JOBS participant

outcomes at the national level? (CHECK ALL THAT

APPLY)

1. [ 36 ] Educational/training achievement, such as

graduation rates

2. [ 39 ] Number of participants with reductions in

AFDC grants due to employment

3. [ 44 ] Number of participants no longer receiving

AFDC due to employment

4. [ 46 ] Number of participants entering

employment

5. [ 29 ] Number of participants entering

employment that provides health insurance

coverage

6. [ 35 ] Hourly wages earned at time of hire

7. [ 13 ] Expected weekly earnings at time of hire

8. [ 20 ] Hourly wages earned at a specified time

period after initial hire

9. [ 40 ] Job retention rate after a specified length of

time

10. [ 41 ] Rate of return to AFDC, after a specified

length of time, for those who left AFDC

due to employment

11. [ 7 ] Other (PLEASE SPECIFY)

___________________________________

___________________________________

27

Page 28 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

13. Which of the following performance indicators, if any,

does your state believe should not be used to monitor

JOBS participant outcomes at the national level?

(CHECK ALL THAT APPLY)

1. [ 5 ] Educational/training achievement, such as

graduation rates

2. [ 4 ] Number of participants with reductions in

AFDC grants due to employment

3. [ 2 ] Number of participants no longer receiving

AFDC due to employment

4. [ 1 ] Number of participants entering

employment

5. [ 9 ] Number of participants entering

employment that provides health insurance

coverage

6. [ 9 ] Hourly wages earned at time of hire

7. [ 18 ] Expected weekly earnings at time of hire

8 [ 15 ] Hourly wages earned at a specified time

period after initial hire

9. [ 4 ] Job retention rate after a specified length of

time

10. [ 3 ] Rate of return to AFDC, after a specified

length of time, for those who left AFDC

due to employment

11. [ 3 ] Other (PLEASE SPECIFY)

___________________________________

___________________________________

14. For each of the performance indicators that you

checked in question 13, please briefly describe the

reason why your state believes it should not be used to

monitor JOBS participant outcomes at the national

level.

15. Which one of the following methods for establishing

nationwide JOBS performance indicators would your

state prefer? (CHECK ONE)

1. [ 0 ] Federal government, in consultation with

the states, establishes the indicators for all

states

2. [ 34 ] States have flexibility to choose indicators

from a menu established by the federal

government in consultation with the states

3. [ 12 ] States have complete flexibility to establish

their own indicators

4. [ 4 ] Other (PLEASE SPECIFY)

__________________________________

28

Page 29 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

16. How much do you think HHS’ Administration for

Children and Families (ACF) has moved the JOBS

program to focus on performance indicators related to

participant outcomes-- more than enough, enough, or

less than enough? (CHECK ONE)

1. [ 0 ] Much more than enough

2. [ 1 ] More than enough

3. [ 7 ] Enough

4. [ 25 ] Less than enough

5. [ 16 ] Much less than enough

[ 1 ] No answer provided

17. During FY 93, how easy or difficult was it for your

state to collect data on outcome-related performance

indicators? (CHECK ONE)

1. [ 0 ] Very easy

2. [ 5 ] Somewhat easy

3. [ 5 ] Neither easy nor difficult

4. [ 23 ] Somewhat difficult

5. [ 17 ] Very difficult

6. [ 0 ] No basis to judge

__________________________________________________________________________________________________________

18. During FY 93, to what extent, if any, did each of the following factors hinder your program’s ability to collect data on

outcome-related performance indicators? (CHECK ONE FOR EACH REASON)

PERFORMANCE INDICATORS

Little or No

Extent

(1)

Some

Extent

(2)

Moderate

Extent

(3)

Great

Extent

(4)

Very

Great

Extent

(5)

1. Availability of funds 6 16 10 13 5

2. Availability of staff 3 6 9 20 12

3. Level of automation and data

systems

5 9 10 10 16

4. Nature of current federal data

gathering requirements

5 3 15 12 15

5. Amount of guidance provided

by ACF on JOBS

performance indicators

related to outcomes

18 6 12 7 7

6. Others (PLEASE SPECIFY)

_______________________

________________________

29

Page 30 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

PERFORMANCE STANDARDS

19. In FY 93, did your JOBS program have state-wide

performance standards for any of the outcomes you

monitor?

Performance standards are benchmarks for given

performance indicators. For example, a program may

collect data on the performance indicator "number of

participants entering employment." Its performance

standard may be "25 percent of participants will gain

full-time employment each year." (CHECK ONE)

1. [ 29 ] Yes (CONTINUE)

2. [ 21 ] No (GO TO QUESTION 22)

20. For which JOBS participant outcomes did your state

have state-wide performance standards in FY 93?

(CHECK ALL THAT APPLY)

1. [ 14 ] Educational/training achievement, such as

graduation rates

2. [ 3 ] Number of participants with reductions in

AFDC grants due to employment

3. [ 9 ] Number of participants no longer receiving

AFDC due to employment

4. [ 27 ] Number of participants entering

employment

5. [ 3 ] Number of participants entering

employment that provides health insurance

coverage

6. [ 16 ] Hourly wages earned at time of hire

7. [ 2 ] Expected weekly earnings at time of hire

8. [ 1 ] Hourly wages earned at a specified time

period after initial hire

9. [ 2 ] Job retention rate after a specified length of

time

10. [ 3 ] Rate of return to AFDC, after a specified

length of time, for those who left AFDC

due to employment

11. [ 0 ] Other (PLEASE SPECIFY) _________

21. In FY 93, did your state make any formal adjustments

in its performance standards to account for local

differences in either client characteristics or the

availability of jobs? (CHECK ONE)

1. [ 5 ] Yes

2. [ 24 ] No

22. In FY 93, how many county (local) JOBS programs in

your state had established countywide performance

standards related to participant outcomes? (CHECK

ONE)

1. [ 16 ] All

2. [ 2 ] Most

3. [ 0 ] About half

4. [ 7 ] Some

5. [ 25 ] None

23. In FY 93, how many counties in your state had

established performance standards related to participant

outcomes for county (local) JOBS program managers

or staff? (CHECK ONE)

1. [ 7 ] All

2. [ 3 ] Most

3. [ 0 ] About half

4. [ 3 ] Some

5. [ 30 ] None

[ 1 ] No answer provided

30

Page 31 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

24. In your view, should nationwide performance standards

related to participant outcomes be established for the

JOBS program? (CHECK ONE)

1. [ 9 ] Definitely yes

2. [ 17 ] Probably yes

3. [ 8 ] As much yes as no (undecided)

4. [ 10 ] Probably no

5. [ 6 ] Definitely no

25. If the federal government established nationwide

performance standards related to participant outcomes

for JOBS, which one of the following methods for

establishing standards would your state prefer?

(CHECK ONE)

1. [ 1 ] Federal government, in consultation with

the states, establishes the standards for all

states

2. [ 33 ] States have flexibility to choose standards

from a menu established by the federal

government in consultation with the states

3. [ 13 ] States have complete flexibility to establish

their own standards

4. [ 3 ] Other (PLEASE SPECIFY) ____________

26. If the federal government established nationwide

performance standards related to participant outcomes

for JOBS, should amounts of federal funding to state

JOBS programs be linked to meeting these standards?

(CHECK ONE)

1. [ 3 ] Definitely yes

2. [ 11 ] Probably yes

3. [ 15 ] As much yes as no (undecided)

4. [ 8 ] Probably no

5. [ 13 ] Definitely no

6. [ 0 ] No opinion

27. Consider the data you were required to collect and

report to meet federal reporting requirements on JOBS

participation rates in FY 93. How easy or difficult

was it for your state to gather the data required to meet

these reporting requirements? (CHECK ONE)

1. [ 0 ] Very easy

2. [ 3 ] Somewhat easy

3. [ 2 ] Neither easy nor difficult

4. [ 24 ] Somewhat difficult

5. [ 21 ] Very difficult

6. [ 0 ] No opinion

28. When your state collects federally required JOBS

participation rate data, how useful, if at all, are these

data in helping your state achieve its JOBS

employment and training goals? (CHECK ONE)

1. [ 1 ] Very useful

2. [ 2 ] Moderately useful

3. [ 20 ] Somewhat useful

4. [ 26 ] Of little or no use

5. [ 1 ] No Opinion

29. Consider the data you were required to collect and

report to meet federal reporting requirements on JOBS

target groups in FY 93. How easy or difficult was it

for your state to gather the data required to meet these

reporting requirements? (CHECK ONE)

1. [ 3 ] Very easy

2. [ 9 ] Somewhat easy

3. [ 9 ] Neither easy nor difficult

4. [ 18 ] Somewhat difficult

5. [ 10 ] Very difficult

6. [ 1 ] No opinion

31

Page 32 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

30. When your state collects federally required data on

JOBS target groups, how useful, if at all, are these

data in helping your state achieve its JOBS

employment and training goals? (CHECK ONE)

1. [ 2 ] Very useful

2. [ 5 ] Moderately useful

3. [ 12 ] Somewhat useful

4. [ 30 ] Of little or no use

5. [ 1 ] No Opinion

JOBS PARTICIPANT OUTCOME DATA

In this section of the questionnaire we are interested in data

that are representative of your entire state. If your state

collects participant outcome data on a sample basis, please

generalize to the entire JOBS caseload in answering the

participant outcome questions. If you do not believe your

sample data are representative of the entire state, please

check the "data are not available" box.

31. Are you providing information in this section of the

questionnaire by federal fiscal year or state fiscal year?

(CHECK ONE)

1. [ ] Federal fiscal year (10/1/92 - 9/30/93)

2. [ ] State fiscal year --> What is the beginning

and ending date for your state’s fiscal year

1993?

__________ to __________

Month/Year Month/Year

32. Of your total AFDC and AFDC-UP recipients in FY

93, how many participated for at least one hour, at any

time during the year, in an approved JOBS activity?

An approved JOBS activity is one which states can

provide according to federal regulations. (ENTER

UNDUPLICATED NUMBER)

____________ Number participating in an approved

JOBS activity

[ ] Data are not available--> (GO TO QUESTION

51 ON PAGE 12)

Employment Entry and Retention

33. Of those in an approved JOBS activity in your state in

FY 93, how many entered subsidized or unsubsidized

employment at some point during the year? (ENTER

NUMBER; IF NONE, ENTER ’0’)

1. ____________ Entered subsidized employment

[ ] Data are not available

2. ____________ Entered unsubsidized employment

[ ] Data are not available--> (GO TO

QUESTION

46 ON

PAGE 11)

34. Of those who entered unsubsidized employment in FY

93, how many had high school diplomas or GEDs?

(ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Had high school diplomas or GEDs

[ ] Data are not available

35. Of those who did not enter unsubsidized employment

in FY 93, how many had high school diplomas or

GEDs? (ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Had high school diplomas or GEDs

[ ] Data are not available

36. Of those who entered unsubsidized employment in FY

93, how many had any work experience in the previous

12 months? (ENTER NUMBER; IF NONE, ENTER

’0’)

____________ Had work experience in the previous 12

months

[ ] Data are not available

32

Page 33 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

37. Of those who did not enter unsubsidized employment

in FY 93, how many had any work experience in the

previous 12 months? (ENTER NUMBER; IF NONE,

ENTER ’0’)

____________ Had work experience in the previous 12

months

[ ] Data are not available

38. Of those who entered unsubsidized employment in FY

93, how many, if any, took (1) full time jobs (working

30 hours or more per week) and (2) part-time jobs

(working less than 30 hours per week)? (ENTER

NUMBERS; IF NONE, ENTER ’0’)

1. _______________ Took full-time jobs

[ ] Data are not available

2. _______________ Took part-time jobs

[ ] Data are not available

39. Of those who entered unsubsidized employment in FY

93, how many took jobs which provided them with

health insurance? (ENTER NUMBER; IF NONE,

ENTER ’0’)

___________ Took jobs with health insurance

[ ] Data are not available

40. For those who entered unsubsidized employment in FY

93, on average how many months were they in an

approved JOBS activity prior to entering employment?

(ENTER NUMBER; IF NONE ENTER ’0’)

___________ Months in an approved JOBS activity

[ ] Data are not available

41. Consider the individuals in an approved JOBS activity

who entered unsubsidized employment in your state

during FY 93. Were follow-up efforts made to

determine if they are still employed? (CHECK ONE)

1. [ ] Yes (CONTINUE)

2. [ ] No (GO TO QUESTION 43)

42. Consider again individuals in an approved JOBS

activity who entered unsubsidized employment during

FY 93. Please indicate the time period(s) during which

you followed-up on these individuals, and for each

applicable time period, provide the number of

individuals who remained employed. (CHECK YES

OR NO FOR EACH; IF YES, ENTER NUMBER)

YES

(1)

NO

(2)

If

yes

-->

NUMBER

REMAINED

EMPLOYED

1. After 3

months

______

2. After 6

months

_______

3. After other

time period

(SPECIFY)

___________ _______

Changes in AFDC Receipt for Those Who Entered

Employment

43. For those in an approved JOBS activity who entered

unsubsidized employment in FY 93, how many were in

the following categories at time of job entry? (ENTER

NUMBERS; IF NONE, ENTER ’0’)

1. Remained on AFDC

with no grant reduction __________

[ ] Data are not available

2. Remained on AFDC

with a reduced cash grant __________

[ ] Data are not available

3. Stopped receiving

a cash grant __________

[ ] Data are not available --> (GO TO QUESTION

46)

33

Page 34 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

44. Consider the individuals in an approved JOBS activity

who stopped receiving AFDC due to employment in

FY 93. Were follow-up efforts made to determine if

they remained off AFDC? (CHECK ONE)

1. [ ] Yes (CONTINUE)

2. [ ] No (GO TO QUESTION 46)

45. Consider again the individuals in an approved JOBS

activity who stopped receiving AFDC due to

employment in FY 93. Please indicate the time period

during which you followed-up on these individuals, and

for each applicable time period, provide the number of

individuals who remained off AFDC due to

employment. (CHECK YES OR NO FOR EACH; IF

YES, ENTER NUMBER)

YES

(1)

NO

(2)

If

yes

-->

NUMBER

REMAINED OFF

AFDC

1. After 3 months ______

2. After 6 months _______

3. After other time

period

(SPECIFY)

___________ _______

Education Component

46. Of those in an approved JOBS activity in your state

during FY 93, how many participated in an education

component, that is adult basic education, English-as-aSecond Language, high school, GED, or post-secondary

education? (ENTER UNDUPLICATED NUMBER;

IF NONE, ENTER ’0’)

_______________ Number in an education

component

[ ] Data are not available--> (GO TO

QUESTION 51)

47. Of those in your state’s JOBS program who

participated in an education component, how many

successfully completed that component in FY 93?

(ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Successfully completed education

component

[ ] Data are not available

48. Of those in your state’s JOBS program who

successfully completed an education component during

FY 93, how many entered unsubsidized employment in

FY 93? (ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Entered unsubsidized employment

[ ] Data are not available

49. Of those in your state’s JOBS program in FY 93 who

successfully completed an education component, how

many did each of the following. (ENTER NUMBERS;

IF NONE, ENTER ’0’)

1. Obtained high school diploma

or GED ____________

[ ] Data are not available

2. Obtained post-secondary degree ____________

[ ] Data are not available

3. Other (PLEASE SPECIFY)

___________________ ___________

50. Did your state test those who completed an educational

component in FY 93 to determine whether they made

any measurable educational gains as a result of their

participation? (CHECK ONE)

1. [ ] Yes

2. [ ] No

34

Page 35 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

Training Component

51. Of those in an approved JOBS activity in your state

during FY 93, how many participated in a training

component (program), that is, one which provides

vocational training in technical job skills and/or helps

develop knowledge and abilities in a specific

occupational area? (ENTER UNDUPLICATED

NUMBER; IF NONE, ENTER ’0’)

_______________ Number in a training component

[ ] Data are not available --> (GO TO QUESTION

55)

52. Of those in your state’s JOBS program who were in a

training component during FY 93, how many

successfully completed the training component in FY

93? (ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Successfully completed training

component

[ ] Data are not available

53. Of those in your state’s JOBS program who

successfully completed a training component during FY

93, how many entered unsubsidized employment in FY

93? (ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Entered unsubsidized employment

[ ] Data are not available

54. Of those in your state’s JOBS program who entered

unsubsidized employment in FY 93 after completing a

training component, how many, if any, entered an

occupation for which they had been trained by the

JOBS program? (ENTER NUMBER; IF NONE,

ENTER ’0’)

____________ Entered occupation for which they were

trained

[ ] Data are not available

Teen Parents

55. How many teen parents participated in an approved

activity in your state’s JOBS program during FY 93?

(ENTER NUMBER; IF NONE, ENTER ’0’)

______________ Teen parents in approved JOBS

activity

[ ] Data are not available --> (GO TO

QUESTION 60)

56. Of the teen parents participating in an approved JOBS

activity, during FY 93 how many were in a high

school, GED, basic education, or similar education

program? (ENTER NUMBER; IF NONE, ENTER ’0’)

____________ Teen parents in education programs

[ ] Data are not available

57. Of the teen parents in an education program during FY

93, how many completed their high school degree or

equivalent education program in FY 93? (ENTER

NUMBER; IF NONE, ENTER ’0’)

____________ Teen parents completed high

school or equivalent

[ ] Data are not available

58. Of those teen parents who completed a high school

degree or equivalent education program, how many

entered employment in FY 93? (ENTER NUMBER; IF

NONE, ENTER ’0’)

_____________ Teen parents who completed a

high school degree or equivalent

and entered employment

[ ] Data are not available

35

Page 36 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

59. Of those teen parents without a high school degree or

equivalent, how many entered employment in FY 93?

(ENTER NUMBER; IF NONE, ENTER ’0’)

_____________ Teen parents without a high

school degree or GED and entered

employment

[ ] Data are not available

PARTICIPATION, COST AND PROGRAM

EVALUATIONS

60. Of your total AFDC and AFDC-UP recipients in

January, 1994, how many participated in an approved

JOBS activity during the month? (ENTER

UNDUPLICATED NUMBER; IF NONE, ENTER ’0’)

______________ Number participating in an

approved JOBS activity

61. Of those in an approved activity in your state’s JOBS

program in January, 1994, how many had participated

in JOBS for a total of 2 years or more? (ENTER

NUMBER; IF NONE, ENTER ’0’.)

_______________ Number participating in JOBS for

2 years or more

[ ] Data are not available

62. What is the average length of stay in months for a

single spell on AFDC, for all AFDC recipients in your

state and for those in an approved JOBS activity?

(ENTER NUMBERS)

1. ___________ Months for all AFDC recipients

[ ] Data are not available

2. ___________ Months for those in an approved

JOBS activity

[ ] Data are not available

63. What was the total annual cost (federal and state) of

your JOBS program during FY 93? Please exclude

child care costs and services obtained from other

programs and not charged to the JOBS budget.

(ENTER AMOUNT)

$ _________________

[ ] Data are not available

64. Have the statewide JOBS program, a component of the

statewide program, or a county (local) program ever

been the subject of a program evaluation which used an

experimental design to compare the outcomes of those

who participate in JOBS with the outcomes of those

who do not participate in JOBS. (CHECK ONE)

1. [ ] Yes --> Please provide the contact person,

name and telephone number of

the organization conducting the

evaluation.

Contact

person: ____________________________

Name of

Organization: _______________________

__________________________________

Telephone

Number:___________________________

2. [ ] No

36

Page 37 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

65. In your opinion, which county (or other local

jurisdiction) within your state operates the most

effective JOBS program? By effective we mean a

program that achieves state program goals and can

document its success with outcome-based performance

data or program evaluation results?

Name of local jurisdiction: _____________________

____________________________________________

Please provide the name and phone number of the

county JOBS director for the county named above.

Name _________________________________

Phone Number ( )

66. In the space below, please briefly describe your

reason(s) for selecting the above program.

COMMENTS

67. In this questionnaire, we ask about statewide

performance data related to employment, job retention,

AFDC status, and education and training. Have you

collected any other data on JOBS outcomes in your

state-- for example, from a special program, a pilot

program, or a demonstration project? (CHECK ONE)

1. [ ] Yes --> PLEASE PROVIDE YOUR

DATA

2. [ ] No

68. Please provide below any comments that you might

have about this questionnaire, performance indicators

and standards, or client outcomes in the JOBS program.

37

Page 38 GAO/HEHS-95-86 JOBS Outcomes
Appendix II 

GAO Questionnaire Regarding JOBS

Program Characteristics

GLOSSARY

AFDC Recipient -- As used in this questionnaire, AFDC

recipient refers to the parent or parents receiving AFDC

benefits and does not include dependent children.

Approved JOBS Activity -- A JOBS activity which can be

provided by states according to federal regulations.

Education -- Education includes programs such as adult

basic education, English-as-a-Second Language, high school,

GED, and post-secondary degrees.

Outcome Data -- Information about the status of individuals

enrolled in JOBS at a given point in time. Examples of

outcome data include the number of individuals who find a

job, the number of individuals who are still employed after 1

year, and the number of individuals who leave AFDC due to

wages.

Participants -- Individuals who participate in at least one

hour of approved JOBS component activities.

Performance Indicator -- A performance indicator is a tool

to monitor client outcomes in the JOBS program. For

example, one possible performance indicator for the JOBS

program could be the percent of those enrolled in the

program who find jobs annually.

Performance Standard -- A performance standard is a

benchmark or quantitative target for a given performance

indicator. For example, 75 percent of teen parents enrolled

in JOBS each year will complete their high school

education.

Subsidized Employment -- Subsidized employment requires

full or partial payment of the wages from government funds.

Examples of this include on-the-job training and work

supplementation.

Training -- Training programs provide vocational training in

technical job skills and/or help develop knowledge and

abilities in a specific occupational area.

Unsubsidized employment -- In unsubsidized employment,

the wages of the employee are paid solely by the employer.

For the purposes of this questionnaire, this includes

employment for which an employer receives the Targeted

Jobs Tax Credit.

Page 39 GAO/HEHS-95-86 JOBS Outcomes
Appendix III 

Comments From the Department of Health

and Human Services

Page 40 GAO/HEHS-95-86 JOBS Outcomes
Appendix III 

Comments From the Department of Health

and Human Services

Page 41 GAO/HEHS-95-86 JOBS Outcomes
Appendix III 

Comments From the Department of Health

and Human Services

Page 42 GAO/HEHS-95-86 JOBS Outcomes
Appendix IV 

Major Contributors to This Report

Robert L. MacLafferty, Assistant Director, (415) 904-2000

Stephen D. Secrist, Evaluator-in-Charge, (415) 904-2000

Susan E. Arnold

Kay E. Brown

LuAnn M. Moy

Ann T. Walker

Christina L. Warren

Karen D. Wright

(105818) Page 43 GAO/HEHS-95-86 JOBS Outcomes
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. Orders for 100 or more copies to be mailed to a

single address are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (301) 258-4097 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Mail

Postage & Fees Paid

GAO

Permit No. G100
GAO/HEHS-95-86 JOBS Outcomes