On identifying and collecting data on performance indicators, the most frequently
mentioned response was for ACF to sponsor conferences or provide information on
what other States are doing (12). Again, respondents from 10 States indicated they
are not interested in any technical assistance from ACF. Advance notice on what
ACF is thinking was requested by respondents from eight States. How to develop and
assess an Outcomes system was mentioned by respondents from five States. Four
States are reportedly interested in a packaged program for the data system. One
respondent noted "the system [requirements] is constantly changing and 50 States are
going crazy trying to develop their own system." Other kinds of assistance mentioned
include interface with other automated systems (3), feedback on how the State’s are
doing (2), family-based outcome measures (2), and how to set standards (2). Six
States’ respondents indicated "anything."

FINDING #5: WHILE NO CONSENSUS EMERGED ON INDICATORS
APPROPRIATE FOR FEDERAL PERFORMANCE STANDARDS, MOST
STATES WANT FLEXIBILITY IN BOTH THEIR DEVELOPMENT AND USE

No Indicator Was Identified By The Majority Of States As A Basis For Standards. Of
Those Mentioned, Job Placement, Earnings, And Educational Improvements Or
Credentials Had The Greatest Support As Potential Federal Performance Standards

When asked about performance indicators they believe are most indicative of success
in the context of the Federal performance standards, only five indicators are
mentioned by more that 10 States. Those include job placement, earnings, educational
improvements or credentials, and AFDC case closures, It is interesting that
respondents from 15 States mentioned a category, long-term self-sufficiency measures,
rather than a specific indicator. Although it is unclear what this means, it may reflect
general discomfort with sending a message to Federal policymakers that is too specific
in the absence of information or strong views. Table 2 on the following page shows
the range of responses provided and frequency of mentions.

States Are Concerned That The Federal Performance Standards Will Not Consider State
Variation And That The Focus Will Be Too Narrow

While there is significant variation in the concerns conveyed by respondents about the
Federal performance standards, the most frequently mentioned concern was that they
will not give States flexibility or consider State variations (mentioned by 18 of 51
State’s respondents). For example, one respondent said "we’re worried about
comparisons among States when programs differ so greatly." Another common
concern is that the focus will be too narrow (16). A number of respondents expressed
concerns that the performance standards will be developed without State input (10).
One respondent stated "they have to include State and local representatives, not just
academicians and researchers...MDRC [the Manpower Demonstration Research
Corporation] doesn’t speak for States...they have valuable information, but they have a
different perspective."

13


FINDINGS

FINDING 1: STATES APPEAR TO BE SUPPORTIVE OF THE NEED TO
MEASURE PERFORMANCE IN THE JOBS PROGRAM. HOWEVER, THEY
DISAGREE ON THE BEST INDICATORS TO USE TO ACCOMPLISH THIS

Most respondents indicated an interest and conveyed ongoing State initiatives relating
to identifying and using performance indicators. Many had strong opinions on which
indicators should be developed and readily identified issues relating to performance
measurement. Respondents indicate that almost all States have in place a system for
collecting and using data on output.and outcome indicators for purposes other than
meeting federal reporting requirements.

Five Output And Outcome Indicators Were Identified By A Majority Of States As
Important In Understanding And Monitoring JOBS Program Performance

We asked respondents to identify the output and outcome indicators their State
believes are the key measures that should be used to understand and track JOBS
program performance.

The five indicators suggested by a majority of respondents include: earnings or wage
at placement (mentioned by 85% or 43 of 51 State respondents), entered employment
or placement (65%), job retention or number of months employed and educational
credentials (both 59%), and number served or participation rates (52%). However, 13
States objected to the use of job placement as an indicator. Responses explaining this
aversion include "we need to show progress, not just the end goal of employment," and
"employment is the easiest measure, but not the best indicator that the program is
successful...the program isn’t structured to focus on job placement - its focus is on
skills and long-term outcomes."

States Mentioned 57 Other Output And Outcome Indicators As Important In Assessing
JOBS Program Performance.

While most respondents suggested five indicators for both outputs and outcomes, the
range of responses reflects considerable divergence of opinion regarding which
indicators are most appropriate. One respondent listed 11 indicators, while another
was unable to recommend any output indicators. Of the 62 indicators suggested, 46
indicators were mentioned by less than 10 States. The appendix shows the overall
responses, using a schematic to categorize responses. The 24 indicators mentioned by
one State only have been collapsed into an "other" category.

AFDC grant reductions and case closures, job placement with medical benefits
provided, educational gains, and JOBS completion ratios were all mentioned by close
to half of the States.


FINDINGS

‘States appear to be supportive of the need to measure performance in the JOBS program.
However, they disagree on the best indicators to use to accomplish this.

Most respondents indicated interest and conveyed ongoing State initiatives relating to
identifying and using performance indicators. Many had strong opinions on which
indicators should be developed and readily identified issues relating to performance
measurement. Five output and outcome indicators were identified by a majority of
States as important in understanding and monitoring JOBS program performance.
These include: earnings or wage at placement, job placement, job retention/number
of months employed, and number served or participation rates. However, States
mentioned 57 other output and outcome indicators as important.

States have undertaken a number of initiatives to develop and use performance indicators.

We found a number of surprisingly varied and sophisticated initiatives, given the
newness of the JOBS program and its administrative complexity. Most States are
tracking JOBS participants after they leave the program. Twelve States are using
some form of family-based outcome measures to evaluate the impact of JOBS on
families. Twenty-two States have developed performance standards relating to key
indicators, and fifteen have developed a system for rewarding and/or sanctioning
program managers or staff based on the performance standards.

Many States currently have systems that provide some data to support performance
tracking, but identify barriers to the development of comprehensive systems.

We found that almost all States have an automated system for collecting data on
JOBS participants, but a small number still collect data manually. Fifty percent of the
States use an additional data system to track JOBS participants who have left the
program; 25 percent use more than one other data system for tracking. States
identified data systems, time, and cost as the greatest barriers to collecting and using
data on performance indicators. While tracking and follow-up are particularly
challenging, respondents have or anticipate having difficulty collecting data on a broad
range of indicators.

States are looking to the Federal Government to act as a clearinghouse for information
on what other States are doing.

Interests in technical assistance from the Administration for Children and Families are
mixed, but most States want a clearinghouse function.


Other concerns mentioned include that the standards won’t reflect the long-term focus
of the program (9). For example, one respondent is concerned that "the standards will
be generic, easy to measure things, like departure rates and wage rates, without
looking at the demographics of clients and the political economy."

The most frequent suggestion for how the Department can structure its
recommendations to make the standards more fair to States was that they be flexible
and account for State and/or local variation, offered by 66 percent of the States’
respondents (32 of 47 respondents). The second most common response was that they
include State input (13), closely followed by the suggestion that they be weighted
based on population or demographics (11). Respondents from six States suggested
both that they should be established as sets, or some combination of indicators, and
that States should be allowed to get waivers to set their own goals.

Table 2: Performance Indicators Most Indicative Of Success For
Federal Performance Standards, According To Respondents

Response Total # States Mentioned
(N=49)

Job Placement 20
Earnings 17
Educational Improvements/Credentials 16
Long-Term Self-Sufficiency Measures 15
AFDC Case Closures W
Retention

Incremental Measures

Skills Training/Competency Improve/Certificate
Recidivism Reduction

AFDC Grant Savings

Literacy

Improved Life Skills

Time in Program/On Assistance

Placement with Benefits

Completion of Training

Repeat Pregnancies

High School Dropout Reduction

Good Progress

Removal of Barriers

States That Did Not Respond

Other*

ONNNNNWWAAATANWWO

* "Other" includes all responses provided only once. Most respondents suggested three
or four indicators, with a one to seven variance.

14


that might affect JOBS program outcomes in their data analysis, respondents indicate
that 11 States are attempting to control for economic conditions.

We asked if the system was capable of sharing data with Adult Education and
Employment and Training (JTPA) data systems. Respondents from the 45 States with
an automated data system indicated that while most States cannot share data with
either system (23 States or 51 percent of those with automated systems), a number of
States at least have or are working towards the capacity to do so. Six States share
data with both systems. Nine States share data with JTPA only, while two share data
with JTPA and have a limited exchange with Adult Education. Most of these States
have some system interface, with four States indicating their systems are integrated.
The most frequently mentioned mechanisms for this interface were tape matches and
exchanges (7). Respondents from four States indicated they were capable of
exchanging data but were not yet doing so.

Most States Use _A Number Of Data Systems To Track Participants

Respondents indicated that half of the States are using data systems to track JOBS
participants who have left the program. While ten States are using only one system,
11 States are using two systems and four States are using three systems. The most
frequently mentioned response was a combined use of the unemployment insurance
wage data and IV-A data systems, reportedly used by seven States. Four States are
using IV-A data only. Three States are using employment services/unemployment
insurance data. Two States are using IV-A, Food Stamps, and Medicaid data systems,
while two States are using transitional child care data systems.

States Identify Data Systems, Time, And Cost As The Greatest Barriers To Collecting And
Using Data On Performance Indicators

The single most frequently mentioned barrier was time and the labor intensive nature
of collecting and using data (18). However, 60 percent of the respondents (30 States)
listed some aspect of the data collection system as a barrier. Respondents from 16
States listed the data system, without any elaboration. The other significant response
was data collection and concern about the lack of consistency and quality of the data,
mentioned by respondents from 12 States. No statewide or automated system was
mentioned by six States’ respondents, and competing system priorities was mentioned
by two. Administrative costs and resource limitations were mentioned by respondents
from 14 States, with a budget and/or staffing crisis mentioned by another eight. One
respondent observed that her State currently spends more on data collection than on
serving clients.

States have varied responses to these barriers. The most frequently mentioned
Tesponse was developing an automated system or implementing FAMIS or JAS (JOBS
automated system). Some States are providing technical assistance or training to
counties exhibiting problems with data collection. Other initiatives mentioned less

11


accountability. Analysis techniques for performance data may include identifying
trends, examining variance between expected and actual performance, comparisons
between entities, and identifying factors causing changes.

One important dimension of performance indicators is the presumed ability to convey
"good" and "bad" performance. While a performance standard may take several forms,
it essentially provides a benchmark or quantitative target to measure a program’s
progress towards certain goals. A performance standard may provide an absolute
standard, or specified level of performance that must be attained in order to receive
rewards or avoid sanctions. Alternatively, a standard may provide a relative basis for
comparing current performance with that of previous years or with targets set at the
beginning of the year. In this relative context, performance relative to a standard may
serve to trigger technical assistance to programs not achieving the standard. Efforts to
mitigate issues relating to different units, geographical areas, or "case-mix" of services
rely on relative standards or provide an optional adjustment model to modify national
standards to more accurately reflect local circumstances.

While the utility of performance indicators is persuasive, the process of reaching
consensus on appropriate indicators can be quite challenging. It requires agreement
between policymakers and various levels of program management on appropriate
indicators of service effort, quantity of services delivered, service quality, and program
outcomes. Given multiple and conflicting program goals held by policymakers,
program managers, staff, and interest groups, an appropriate performance monitoring
system typically requires multiple performance indicators. Additionally, any set of
program performance measures requires balancing the value of specific program
information against the feasibility and cost of data collection.

Several Government-wide initiatives reflect the increasing interest in establishing
performance monitoring systems. Most significant on the Federal level is the Chief
Financial Officers Act of 1990. This statute, currently implemented on a pilot basis,
establishes a new financial management system for all executive agencies. Within the
context of an annual report, it requires operating divisions to develop and regularly
report performance indicators and measures as part of their financial statements.
Additionally, pending legislation in the U.S. Senate would require the development of
performance standards and goals in each Federal agency, with annual performance
reports to the President and the Congress. Further, the Government Accounting
Standards Board adopted a resolution that strongly encourages State and local
governmental entities to become familiar with service efforts and accomplishments
(SEA) measurement and reporting and to experiment in identifying and using these
indicators.

Performance Standards in Other Federal Employment and Training Programs
There is strong interest in federally-funded employment and training programs

operating under performance standards. The Job Training Partnership Act (JTPA),
established in 1982, has had outcome-based performance standards in effect since


TABLE OF CONTENTS

EXECUTIVE SUMMARY ...........0. ccc cee eee e cece eee e eect eens i
INTRODUCTION ..... 2... eee ccc eee cece eee eee eee 1
FINDINGS
General support for the concept ........ 2... cece eee ee eee eee eee eeee 7
State initiatives 2.0.0... cece cece eee eee tee eens 8
State systems 1... eee cette cette teenies 10
Federal role... 2... 6. eect eect e teen eee eeae 12
Performance standards ........... 0.2 eee eee eee eee eee e ee eee 13


While no consensus emerged on appropriate Federal performance standards, most States
want flexibility in their development and use.

No indicator was recommended by the majority of States as a potential basis for
Federal performance standards. However, of those mentioned, job placement,
earnings, and educational improvements or credentials had the greatest support. They
are concerned that the Federal performance standards will not consider State variation
and that the focus will be too narrow.

iii


frequently include further developing or simplifying their system, providing personal
computers, and coordinating with or developing cooperative agreements with other
agencies.

While Tracking And Follow-Up Are Particularly Challenging, Respondents Have Or
Anticipate Having Difficulty Collecting Data On A Broad Range Of Indicators

While respondents indicated that most States (42) have had or anticipate having
difficulty developing a reliable data collection system for certain indicators, there is
little agreement on which indicators are most troublesome. Twenty indicators were
mentioned, with a range of 16 State respondents mentioning one indicator, another 16
mentioning two indicators, and respondents from one State mentioning five.

Tracking and follow-up were mentioned by respondents from 18 States, followed by.
monitoring participation mentioned by respondents from 11 States. Other mentions
include family data or outcomes (6), satisfactory or incremental progress (5), welfare
savings (5), the FSA 108 (4), and 14 other indicators that were mentioned by one, two
or three States’ respondents.

FINDING #4: STATES ARE LOOKING TO THE FEDERAL GOVERNMENT
TO ACT AS A CLEARINGHOUSE FOR INFORMATION ON WHAT OTHER
STATES ARE DOING

Interests In Technical Assistance From ACF Are Mixed, But Most Want A Clearinghouse
Function

We asked States what kind of technical assistance they would like from the
Administration for Children and Families (ACF) on two areas: evaluating the JOBS
program, and identifying and collecting data on performance indicators.

On evaluating the JOBS program, 37 percent of those that responded (18 of 48
States) requested information on what other States are doing. Fourteen States are
interested in ACF serving as a clearinghouse on State information and effective
practices, and six are interested in getting together with other States (two mentioned
both). Among the suggestions offered are "make information available on what’s
working: how are other States meeting participation rates, what type of programs do
they have, how are they defining participation," and "bring States together that have
similar needs, not by region or convenience."

Twenty percent of the respondents are not interested in any technical assistance.
Typical of the comments these respondents offered is "they have no in-house expertise
to provide; the best they can do is be a facilitator for those that do have it and act as
a clearinghouse." Other responses included obtaining resources to fund the data
system (4), explain why they’re set the way they are (3), how to develop an automated
system (3), how to develop valid measures (3), how to market to and coordinate with
other agencies (2). Three States’ respondents indicated "anything."

12


While the Department ultimately is required to develop recommended performance
standards, the first step in this process will be reaching consensus on key indicators
that are appropriate for standards. Given the significant variation in JOBS program
characteristics and goals from State to State, and sometimes from county to county
within States, the process of reaching consensus on appropriate indicators will be
particularly challenging. Presently, the Department has convened a task force
composed of representatives from several States, program offices, interest groups,
representative and research organizations, and academics to identify indicators with
broad support, and to determine the nature of standards that would be most
appropriate.

Definition of Terms Used in This Report

Performance Indicator Variable or measure designated as key
program feature to monitor

Outputs Services provided through JOBS program,
including quantity of services, process
measures, and transitional support services

Intermediate Outcomes Early indicators that show progress toward
program goals by identifying changes in the
participant’s skills, competencies, and/or
abilities

Outcomes Status at termination or some time following
participation in JOBS program activities
relating to ultimate goals, including job
placement and IV-A (AFDC) program
effects

Performance Standard Benchmark or quantitative target on a given
indicator

Performance Indicators

Program managers and policymakers at all levels of government are increasingly
interested in establishing performance management systems that provide data on
service efforts and accomplishments, or performance indicators. The designation of
performance indicators occurs as part of a broader effort to develop a performance
monitoring system that will periodically provide data on the quantity and quality of
service delivery and the outcomes (results) achieved in a program. Such systems
include - but go beyond - the more typical measurements of program costs and
activities, though they stop short of identifying actual impact, or the extent to which
the program caused observed outcomes.

There are various uses for performance indicators. Fundamentally, they are used to
improve performance, ensure cost-effectiveness, and enhance credibility and


Department of Health and Human Services

OFFICE OF
INSPECTOR GENERAL

PERFORMANCE INDICATORS IN THE
JOBS PROGRAM



wage rate, and occupation. States must provide initial information on education and
literacy levels, and may provide information on follow-up levels, if available.

Caveats are in order, however, regarding the use of outcome measures in the JOBS
program, as with any job training program. While outcome measures, such as job
placement, capture the desired effect of the program, the correlation between these
indicators and net impacts may not be strong, Studies have found that many AFDC
recipients find jobs and leave welfare in the absence of training. Outcome measures
of performance in job training programs overstate true impact and cannot provide
meaningful comparisons of performance across programs. For example, some
programs’ high rates of job entry may result from their having a relatively "job-ready"
target population or a strong labor market, while those experiencing a lower rate of
job entry may stem from the reverse conditions.

METHODOLOGY

We collected data from respondents in the 50 States and the District of Columbia
through telephone interviews conducted in August - October of 1991. The initial
contact was a letter to the JOBS coordinator suggesting that both program and
tesearch/evaluation staff might be present for a conference call to collect information.
A copy of the discussion guide was sent along with this letter.

We had a total of 89 respondents. Six States had a single respondent, and most had
two or three present for a conference call. Most respondents were managers,
including JOBS directors, coordinators, or managers (23), managers of training and
employment services (22), and deputy directors or assistant managers (6). Other
respondents included policy staff (9), management information systems staff (5),
department of labor staff (4), and smaller numbers representing research, the
Commissioner’s office, AFDC staff, and field office staff.

In all cases, we are reporting information provided to us by respondents. We did not
independently verify this information.

Analysis

Despite this escalating interest in improving the management of government programs
by identifying and tracking key measures and indicators, the process is still in the
formative stages. There is still significant resistance to the process of identifying key
measures, as well as confusion about the terminology. The distinction between various
types of indicators is not always immediately apparent.

In our survey, there was some confusion about the classification of several types of
indicators. In a few cases, our schematic classifies indicators differently than
respondents. Responses have been categorized to reflect the flow of clients through
the program and their achievement of stated program goals, for the most part. As we


1983. Initially, the performance standards focused primarily on the outcomes attained
upon completion of training. Gradually, the Department of Labor added measures to
capture the longer-term effects of JTPA training. As of July 1990, the standards focus
exclusively on post-program outcomes based on a 13-week follow-up period, including
wages, job placement, and retention. States are encouraged to adjust the performance
standards to reflect the influence of client characteristics and local economic
conditions. States use the JTPA performance monitoring system to reward high
performance and penalize poor performance by local service delivery areas.

The Food Stamp Employment and Training program, mandated by the Hunger
Prevention Act of 1988, included specific instructions on the development of
performance standards relating to job placement rates, wage rates, job retention rates,
households ceasing to need food stamp benefits, improvements in household members’
educational levels, and the extent to which persons elect to participate. The proposed
regulations, issued in August 1991, offered two alternative approaches to measuring
outcomes for comment. Comments received were significant and negative.

In response to concern about the administrative burden of requiring States to
implement potentially varying performance standards for overlapping programs,
Congress imposed a moratorium on these performance standards until at least a year
after the JOBS program issues its regulations on performance standards.

Performance Indicators in the JOBS Program

Within the general parameters of the JOBS program, States have considerable
flexibility in program design. The statute establishes a structure of mandates and
optional JOBS program components that allow for significant State variation in the
emphasis of their JOBS services. Some States emphasize longer-term "human capital
investment" and stress basic skills education, job readiness training, and vocational
skills training that can take months to complete. Other States emphasize "labor force
attachment" and stress services that provide job search assistance that can be
accomplished in a few weeks. The performance indicators that make sense to a
particular State will reflect, for the most part, the program design and emphasis in
that State.

Agreement between policymakers and program managers on appropriate performance
indicators is the first step in this process of monitoring program performance. The
next step is collecting data on those indicators. The implementation of the JOBS
program included significant reporting requirements for States and extensive effort to
develop systems capable of capturing the required data. Currently, there are
standards relating to participation rates and targeting services on several groups of
those who are, or who are likely to become, long-term recipients. Initially, all data
collection requirements focused on output indicators. With the most recent data
collection initiative, effective October 1, 1991, States are providing some outcome data
in the required monthly reports. The required outcome data includes job entry, hourly


INTRODUCTION

PURPOSE

This report provides precursory information from States on the identification and use
of performance indicators in the JOBS program. It was requested by the Assistant
Secretary for Planning and Evaluation. It provides: 1) information on State
perspectives on performance indicators; 2) baseline information on State initiatives
and capacity to develop the data systems required to regularly collect and use
performance data; and 3) information on State perspectives on the development of
Federal performance standards.

BACKGROUND

The Family Support Act of 1988 created the Job Opportunities and Basic Skills
Training (JOBS) program for recipients of Aid to Families with Dependent Children
(AFDC). The JOBS program is designed to assist recipients become self-sufficient by
providing needed employment-related activities and support services. States must
provide educational activities, vocational classroom training, job readiness activities,
job development, and job placement. States must also provide at least two of the
following activities: group and individual job search, on-the-job training, work
supplementation, or community work experience.

The Department has a legislative mandate to develop performance standards for the
JOBS program and propose them to the appropriate Congressional committees by
October 1, 1993. A performance standard denotes a benchmark or level of
performance on a key indicator that is expected and thereby serves as the basis for
assessing how well a program is performing. Section 203 of the Family Support Act of
1988 requires the Secretary to recommend performance standards that provide specific
measures of outcomes and not solely levels of activity or participation. The
recommended standards are to be based on the degree of success that may be
reasonably expected of States in helping individuals to increase earnings, achieve self-
sufficiency, and reduce welfare dependency. The statute requires the Department to
consult with various representative organizations in developing these recommended
standards.

While the statutory language mandates standards relating to "specific measures of
outcome," and, conventionally, performance monitoring systems focus on short-term
and medium-term outcomes of program activities, the Department has expressed an
interest in associating impact with the performance standards, or the extent to which
programs caused observed outcomes. This report focuses on output and outcome
indicators, and performance monitoring, in general. We did not prompt respondents
to address the issue of impact, consequently impact is not addressed in this report.


Appendix: State Perspectives On Output & Outcome Indicators

Response Total Number States Mentioned

ip

Services JOBS Completion Ratios 2
Target Group Participation 13
Percent Sanctions 8
Number of Referrals 7
Number Receive Support Services 6
Effectiveness/Cost 5
Percent AFDC Served 3

Process Length of Stay/Hours 9
Measures Satisfactory Completion of JOBS 7
Client Flow 7
Satisfactory Progress 6

Transitional Benefits 5

OE cs tials
Educational Gains

Training Completion/Certification
Literacy Gains

Skills Competency

Barrier Removal/Reduction

Life Management Skills
intermediate Outcomes

Family Measures

He ee

Job Placement Earnings/Wage at Placeme
Enter Employ/Placement 33

Number Months Employ/Retention 30

Placement w/ Medical Benefits 22

Full-/Part-time Placement 1
6

4

3

3

3

3

2

4

Cc

Employ Related to Training
Promotional Opportunity

Quality of Placement

Target Group Success

Average Weekly Wages
Unsubsidized Employment
Short- or Long-Term Employment

Program Effects Grant Reduction/Welfare Savings 24
AFDC Case Closures 22
Recidivism 16
Cost Per Participant/Cost Benefit 9

Other* 24

*‘Other" includes all responses provided only once.

15


EXECUTIVE SUMMARY

PURPOSE

This report provides precursory information from States on the identification and use
of performance indicators in the JOBS program. It was requested by the Assistant
Secretary for Planning and Evaluation. It includes: 1) information on State
perspectives on performance indicators; 2) baseline information on State initiatives
and capacity to develop the data systems required to regularly collect and use
performance data; and 3) information on State perspectives on the development of
Federal performance standards.

BACKGROUND

The Department has a statutory mandate to propose recommended performance
standards for the Job Opportunities and Basic Skills Training (JOBS) program to the
appropriate Congressional committees by October 1, 1993. The first step in this
process will be reaching consensus on key indicators that are appropriate for
standards.

Given the significant variation in JOBS program characteristics and goals from State to
State, and sometimes from county to county within States, the process of reaching
consensus on appropriate indicators will be particularly challenging. Presently, the
Department has convened a task force composed of representatives from several
States, program offices, interest groups, representative and research organizations, and
academics to identify indicators with broad support, and to determine the nature of
standards that would be most appropriate.

While the statutory language mandates standards relating to "specific measures of
outcome," the Department has expressed an interest in associating impact with the
performance standards. This report focuses on output and outcome indicators, and
does not address the issue of impact.

The following terms are used throughout this report. A performance indicator is a
variable or measure designated as a key program feature to monitor; a performance
standard establishes a benchmark or quantitative target on a given indicator.

METHODOLOGY

We conducted telephone interviews with respondents from the 50 States and the
District of Columbia in August - October of 1991. We did not independently verify
information provided to us by respondents.


Negative views about indicators are also dispersed. While 80 percent of the States’
respondents indicated that there are output and outcome indicators that they have
negative views about, no single indicator stands out. Only three indicators were
identified by more than eight States: job placement or entered employment (13), the
federally defined participation rate (10), and welfare savings or grant reductions (8).

FINDING #2: STATES HAVE UNDERTAKEN A NUMBER OF INITIATIVES
TO DEVELOP AND USE PERFORMANCE INDICATORS

While the first finding explores respondents’ views about performance indicators, we
also asked about what States are doing to collect and use data on program
performance for their own purposes, beyond the Federal reporting requirements. We
found a number of surprisingly varied and sophisticated initiatives, given the newness
of the JOBS program and its administrative complexity.

Most States Are Tracking JOBS Participants After They Leave The Program

Respondents indicated that 35 States are tracking JOBS participants after they leave
the program. In half of these States (19 of 35 States), the case manager or IV-A
worker is responsible for tracking. Other States use a variety of mechanisms. Four
States contract out this function. Three mentioned it as part of a longitudinal study or
evaluation, and two others identified planning and evaluation staff as having this
responsibility. Other States have JTPA or the contracting agency conduct follow-up.

There is significant diversity in the frequency of contact and duration of tracking
among States. Of the 30 States that provided specific information, we found that 15
States track participants for 90 days or less, and the other 15 track participants for 6-
months to 2-years. The most frequently mentioned length of time was 10 States who
track participants for 12 months. On the frequency of contact with post-JOBS clients,
we found that most States have one or two contacts over the duration of the tracking
period. Respondents from the other five States report that the tracking is done
through an evaluation on a sample basis.

Twelve States Are Using Some Form Of Family Outcome Measures To Evaluate The
Impact Of JOBS On Families

While most States are not collecting this kind of data, respondents reported that
almost 25 percent of the States have underway a variety of initiatives to identify the
impact of the JOBS program on families. Four States are focusing on teens and
repeat pregnancies. Another five States have an evaluation or demonstration program
that includes family outcome data. Two States indicated that they are looking at
family outcome measures "holistically", and one State is looking at reduced substance
abuse, school retention, and reduced child abuse.


Fifteen States Have Developed A System For Rewarding/Sanctioning Program Managers
Or Staff

Respondents reported that about 75 percent of the States that have developed
standards have gone on to develop a system to reward or sanction program managers
or staff for meeting or failing to meet a performance standard. Twelve States are
reportedly using only one mechanism to reward or sanction program managers or
staff, while three States are using two mechanisms and one State is using three.

The most frequently mentioned system for rewarding or sanctioning managers or staff
was incentives and rewards, reportedly used by six States. For example, one State has
a $500,000 pot, $400,000 of which is provided as incentives and $100,000 is available
for technical assistance. Respondents indicated that five States distribute a quarterly
or yearly report that includes performance information by county or district.
Respondents from another five States considered rewards and sanctions as built into
the annual performance evaluation or contract review process. For example, one
State has county offices develop goals and notes that while there are no sanctions for
failing to achieve these goals, there is some effect on personal performance reviews.

Defunding or sanctioning managers and/or staff was listed by respondents from three
States. One example of this is a State that requires 22 percent placement by
providers. If the provider fails to meet this standard, 5 percent of funds may be
withheld and allocated on a pro-rated basis to those that do meet the standard. One
State reportedly uses data on meeting or failing to meet-a performance standard in its
distribution of JOBS funds. Another State provides "hands-on training" on data
collection to every county office that fails to meet its standards.

FINDING #3: MANY STATES CURRENTLY HAVE SYSTEMS THAT
PROVIDE SOME DATA TO SUPPORT PERFORMANCE TRACKING, BUT
IDENTIFY BARRIERS TO THE DEVELOPMENT OF COMPREHENSIVE
DATA COLLECTION SYSTEMS

Almost All States Have An Automated System For Collecting Data on JOBS Participants,
But A Small Number Still Collect Data Manually

We asked about State systems for collecting data in localities where JOBS operates.
According to respondents, 40 States have a common system among all localities, and
five States have a common system in some localities, but not all. Six States do not
have an automated data collection system. All six of these States have a State-
administered JOBS program.

Of the States with an automated system, respondents indicated that 16 States analyze
output or outcome data by demographic information on clients, such as AFDC history,
work history, age, literacy. Respondents indicated that 18 States analyze output and/or
outcome data by target groups. While most States are not adjusting for other factors

10


Department of Health and Human Services

OFFICE OF
INSPECTOR GENERAL

PERFORMANCE INDICATORS IN THE
JOBS PROGRAM

ey,

{ Richard P. Kusserow
7 & INSPECTOR GENERAL

OEI-12-91-01380



OFFICE OF INSPECTOR GENERAL

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, as
amended, is to protect the integrity of the Department of Health and Human Services’ (HHS)
programs as well as the health and welfare of beneficiaries served by those programs. This
statutory mission is carried out through a nationwide network of audits, investigations, and
inspections conducted by three OIG operating components: the Office of Audit Services, the
Office of Investigations, and the Office of Evaluation and Inspections. The OIG also informs
the Secretary of HHS of program, and management problems, and recommends courses to
correct them.

OFFICE OF AUDIT SERVICES

The OIG’s Office of Audit Services (OAS) provides all auditing services for HHS, either by
conducting audits with its own audit resources or by overseeing audit work done by others.
Audits examine the performance of HHS programs and/or its grantees and contractors in
carrying out their respective responsibilities and are intended to provide independent
assessments of HHS programs and operations in order to reduce waste, abuse, and
mismanagement and to promote economy and efficiency throughout the Department.

OFFICE OF INVESTIGATIONS

The OIG’s Office of Investigations (OI) conducts criminal, civil, and administrative
investigations of allegations of wrongdoing in HHS programs or to HHS beneficiaries and of
unjust enrichment by providers. The investigative efforts of OI lead to criminal convictions,
administrative sanctions, or civil money penalties. The OI also oversees State Medicaid fraud
control units which investigate and prosecute fraud and patient abuse in the Medicaid program.

OFFICE OF EVALUATION AND INSPECTIONS

The OIG’s Office of Evaluation and Inspections (OEI) conducts short-term management and
program evaluations (called inspections) that focus on issues of concern to the Department,
the Congress, and the public. The findings and recommendations contained in these inspection
reports generate rapid, accurate, and up-to-date information on the efficiency, vulnerability,
and effectiveness of departmental programs.

This report was prepared in Headquarters under the direction of Emilie Baebel, Chief, Public
Health and Human Services Branch, Office of Evaluation and Inspections. Participating in this
project were:

Ruth Folchman, Project Leader

Mark Krushat, Chief, Technical Support Staff

Brian Ritchie, Technical Support Staff

To obtain a copy of this report, call (202) 619-3394.


considered how to present this information, we assumed that the respondent was clear
on the particular element mentioned and not clear on the terminology we were using.

More importantly, we determined that an additional category of "intermediate
outcomes" would both better reflect the data we collected and the concern that many
respondents expressed about the importance of incremental gains, particularly given
the economic conditions and unemployment rate in many States. Some of the
elements in the categories we have identified may be subject to differences of opinion
about where they most appropriately belong.


Twenty-two States Have Developed Performance Standards Relating To Key Indicators

We asked if any States had developed performance standards, or a specified level of
activity on a particular indicator that local programs must achieve in order to consider
their program successful. The following table reflects the responses of the 22 States
that have developed standards.

Participation rate and placements are the most frequently mentioned standards. It
may be significant that the JOBS program currently has a participation rate standard,
and the JTPA program has performance standards on placement rates, retention, and
wages. The JTPA program serves many AFDC recipients, and coordination between
JTPA and JOBS is required and encouraged in the regulations implementing the
JOBS program. We did not get information on the level involved with these
standards. They may be higher than those standards established by JTPA and JOBS,
or may only reflect the State’s determination to meet existing Federal standards by
imposing a State-based incentive system.

Table 1: Performance Standards Developed by States

Response Total
(N=22)

Number Served/Participation Rate
Placements

Wages

Educational Achievement

Caseworker Caseload

Expenditures

Retention

Number of Hours/Minimum Wage Placement
Pre-/Post-Testing for Training

Number in Educational Component
Percent Target Group in Educat. Activity
Resolution Rate

Percent in Unsubsidized Employment
Sanctions

Penetration Rate

Hours of Participation

Number in Post-Secondary Education
Off AFDC

Number Client Contacts by Caseworker
Number of Referrals

Bae enn neni innnnwwanaoo

*Range: Eight States identified only one standard; three States identified five standards;
seven States identified three or more standards in place.