Department of Health and Human Services 
OFFICE OF 
INSPECTOR GENERAL 
Early Efforts by States to Monitor 
Outcomes of Welfare Reform 
JUNE GIBBS BROWN 
Inspector General 
JANUARY 1999 
OEI-05-98-00130  OFFICE OF INSPECTOR GENERAL 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 
protect the integrity of the Department of Health and Human Services programs as well as the 
health and welfare of beneficiaries served by them.  This statutory mission is carried out through a 
nationwide program of audits, investigations, inspections, sanctions, and fraud alerts.  The 
Inspector General informs the Secretary of program and management problems and recommends 
legislative, regulatory, and operational approaches to correct them. 
Office of Evaluation and Inspections 
The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 
Inspector General.  It conducts short-term management and program evaluations (called 
inspections) that focus on issues of concern to the Department, the Congress, and the public.  The 
inspection reports provide findings and recommendations on the efficiency, vulnerability, and 
effectiveness of departmental programs. 
OEI's Chicago regional office  prepared this report under the direction of William C. Moran, 
Regional Inspector General and Natalie Coen, Deputy Regional Inspector General.  Principal OEI 
staff included: 
REGION  HEADQUARTERS 
Thomas F. Komaniecki, Project Leader  Linda Hall, Program Specialist

Ann Maxwell  Ann OConnor

Suzanne Johnson

Victoria Jacobs

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-4124.

Reports are also available on the World Wide Web at our home page address:

http://www.dhhs.gov/progorg/oei
 E X E C U T I V E  S U M M A R Y

PURPOSE 
To examine State efforts in measuring outcomes of the Personal Responsibility and Work 
Opportunity Reconciliation Act. 
BACKGROUND 
The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) (P.L. 
104-193) replaced the Aid to Families with Dependent Children entitlement program with 
Temporary Assistance for Needy Families (TANF) block grants to the States.  This legislation 
also altered the framework for welfare policy by emphasizing program outcomes over process 
measures and by devolving the responsibilities of welfare administration from the Federal 
Government to the States. 
While PRWORA does not require States to develop client-based outcome measures, many States 
have begun developing outcome measures to understand the effects of their programs. With a 41 
percent drop in the welfare rolls, from 14 million recipients in January 1993 to 8.4 million 
recipients in June 1998, policy makers and the public want to know what is happening to families 
that go off the rolls. 
We asked State administrators how they were assessing outcomes of their TANF policies on the 
children and families in their State.  For purposes of final analysis, we defined an outcome 
indicator as any client-based outcome measure at the point of exit from TANF or beyond.  We 
also made a distinction between what States are tracking through administrative data versus more 
formal evaluations. 
This inspection is descriptive.  We made no attempts to assess the validity or reliability of the 
outcome indicators used by the States.  We also did not attempt to assess the methodology of 
State evaluations.  However, this inspection does begin to examine whether States will be able to 
measure outcomes of welfare reform on individuals and whether systems are being developed to 
accomplish the task of measuring outcomes.  These are questions that the Congress, State 
legislatures and the media have been consistently asking and writing about. 
FINDINGS 
Two-thirds of the States are beginning to use administrative data that track the short-term 
outcomes of former TANF clients. 
)))))))))))
i  All States are required by PRWORA to measure the participation in work activities by current 
TANF clients, however, 34 States are beginning to measure the outcomes of former TANF 
clients.  Most of these States are focusing on the areas of economic self-sufficiency (e.g. job 
retention, employment, earnings and job benefits) and welfare dependency (e.g. recidivism and use 
of other government programs).  Only a few are beginning to look at outcomes that measure 
areas such as child well-being and family well-being.  The outcomes measured by administrative 
data can be limited by the availability and quality of the data and the definitions of outcomes used 
by States. 
Almost all States are planning or beginning to conduct evaluations examining the outcomes 
of TANF families and children; however, evaluations vary widely in scope and 
methodological rigor and few results are currently available. 
Thirty-eight States are currently conducting some type of evaluation that examines the outcomes 
of children and families.  The majority of these evaluations are in the early implementation phase 
of the evaluation.  Typically, evaluations are designed to provide administrators with a more 
complex analysis of the effect of TANF and a broader range of outcome measures for a wider 
selection of the former caseload. 
The evaluations that States are conducting or planning range widely in scope and methodological 
rigor.  Their scope ranges from one-time surveys to longitudinal studies that track a random 
sample of clients.  Currently, only 12 States have evaluations that use an experimental design;  the 
majority of evaluations simply gather a broad range of client outcomes. 
Few States believe their current efforts to monitor outcomes of welfare reform are 
sufficient. 
Only 10 States believe their current efforts to monitor outcomes of their welfare reform programs 
to be sufficient.  Of these 10 States, 5 used both evaluations and administrative data to track client 
outcomes, while the remaining 5 States used either evaluations or administrative data to monitor 
outcomes of welfare reform.  There appears to be little relationship between the breadth of State 
efforts to monitor outcomes of welfare reform and respondents beliefs in the sufficiency of 
information they are currently collecting. 
ISSUES FOR FURTHER CONSIDERATION 
This inspection documents States efforts to assess the outcomes of their TANF programs at this 
early stage of implementation.  It is intended to provide a foundation for future research efforts. 
Issues for future consideration include: 
<  exploring the capacity of States to track outcome measures using their administrative 
databases, 
<  validating the data used for the outcome measures, 
)))))))))))
ii  <  reviewing the quality and scope of States evaluations, and,

<  assessing the breadth and depth of a States overall efforts to measure outcomes of

welfare reform. 
By answering these research questions, we will know if States have been successful, not only in 
cutting back the welfare rolls, but also in improving the lives of low-income children and families. 
As one of the agents in the Department responsible for overseeing program effectiveness and 
beneficiary protection, the Office of Inspector General is vitally interested in knowing if States 
have established appropriate client outcome measures. 
AGENCY COMMENTS 
We received comments from the Administration for Children and Families (ACF), the Assistant 
Secretary for Planning and Evaluation (ASPE) and the Assistant Secretary for Management and 
Budget (ASMB).  Their comments are included in Appendix A.  We made revisions to the report 
based on their comments. 
Both ACF and ASPE suggested that to ensure that people do not infer that States are conducting 
studies that can causally link the efforts of welfare reform and the outcomes of clients, we limit 
our use of the word impact.  In our draft report, we defined the word impact broadly to 
encompass all client-based outcomes associated with TANF intervention and not only those 
results which could be proven to be causally linked to TANF through the use of experimental 
methodology.  We agree that this could be confusing, so to limit that problem we have replaced 
the word impact with outcomes in our report. 
The ASMB pointed out that authority for the administration of welfare reform programs rests 
with the States and that they may be understandably reluctant to volunteer un-mandated data 
that is not yet universally accepted as reflecting necessary and appropriate outcomes.  However, 
noting that alternative measures are feasible and are being tested, ASMB suggested that the 
Department of Health and Human Services participation in program assessment may 
appropriately involve facilitating the collaboration of State and Federal stakeholders rather than 
the traditional forms of oversight.  We agree with this observation, and it is in that spirit that we 
offer this report. 
)))))))))))
iii  T A B L E  O F  C O N T E N T S

EXECUTIVE SUMMARY

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  1

FINDINGS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

Use of administrative data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  5

Launching of evaluations  . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Adequacy of current monitoring activities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

ISSUES FOR CONSIDERATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  11

AGENCY COMMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  11

APPENDIX

A:  Agency Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A-1
 I N T R O D U C T I O N

PURPOSE 
To examine State efforts in measuring outcomes of the Personal Responsibility and Work 
Opportunity Reconciliation Act. 
BACKGROUND 
The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) 
(P.L. 104-193) replaced the Aid to Families with Dependent Children (AFDC) entitlement 
program with Temporary Assistance for Needy Families (TANF) block grants to the States.  The 
TANF program requires States to meet increasing work participation goals and places a lifetime 
limit of 5 years on benefits.  This legislation also altered the framework for welfare policy by 
emphasizing program outcomes over process measures and by devolving the responsibility of 
welfare administration and tracking from the Federal Government to the States. 
Welfare Reform Outcome Measures 
Under AFDC, States were required to collect information on many aspects of the programs 
performance.  Most of the measures used by the Administration for Children and Families (ACF) 
to judge the compliance of States in administering AFDC emphasized process goals such as 
whether applications were processed timely or cash benefits were accurate.  The Government 
Performance and Results Act of 1993 (P.L. 103-62) shifted this departmental focus from process 
and output measures to outcomes and program effectiveness.  The PRWORA continued this 
emphasis on outcomes by establishing four basic goals.  The four goals outlined in the legislation 
are: 
<  providing assistance to families so that children may be cared for in their own homes 
or in the homes of relatives; 
<  ending dependence on government assistance by promoting job preparation, work and 
marriage; 
<  preventing and reducing out-of-wedlock births; and, 
<  encouraging the formation and maintenance of two parent families. 
Besides establishing these basic goals, PRWORA specified that States had to meet steadily rising 
work participation requirements for their clients along with establishing numeric goals for 
reducing illegitimacy. 
)))))))))))
1  Congress established a high performance bonus and an illegitimacy bonus to reinforce these 
legislative goals.  These bonuses offer financial rewards to top performing States.  Proposed 
regulations for the illegitimacy bonus, issued on March 2, 1998, outline a plan to distribute $100 
million a year amongst the States demonstrating the largest decreases in out-of-wedlock 
childbearing while simultaneously reducing their abortion rates.  Regulations have yet to be 
published regarding the high performance bonus.  However, ACF did issue guidance on the high 
performance bonus in March 1998.  Additional guidance on the high performance bonus was 
issued in August 1998.  For Fiscal Year 1998, the interim formula weighs job placement, job 
retention and wage progression. 
While PRWORA does not require States to develop client-based outcome measures, many States 
have begun developing outcome measures to understand outcomes of their programs.  However, 
States are at different stages of implementing these measures as well as their TANF programs. 
Currently, most of the public discussion regarding outcomes of welfare reform focuses on 
reducing welfare caseloads and getting clients into jobs.  However, many people are beginning to 
question whether these are the only outcomes that should be used to measure outcomes of 
welfare reform.  An article by a group of senior welfare administrators stated that these two 
criteria [reducing welfare caseloads and getting clients into jobs] alone are not sufficient to 
measure the success of welfare reform.  They suggest that States initial areas of concern should 
include welfare dependency, self-sufficiency, economic well-being and parental responsibility. 
They believe that ultimately States should measure child well-being, parent well-being, family 
well-being, community involvement, economic development and family formation in order to 
understand the broader context in which welfare reform occurs.  Many others also believe that 
more needs to be known to gauge outcomes of welfare reform. 
Federal Responsibilities 
The PRWORA also represents a major shift in the roles and responsibilities of the State and 
Federal Governments.  While the Federal Government retains some measure of oversight, much 
of the accountability for policy decisions now rests at the State level.  The new law limited 
Federal regulatory and approval authority to provide States with the authority and flexibility to set 
their own policies.  It did, however, leave the Department with some responsibility for tracking 
State performance. 
The oversight role of ACF changed considerably with the passage of PRWORA.  With the 
devolution of program design and responsibility to the States through the block granting of all 
funds to the State, States are now responsible for measuring outcomes of their programs.  The 
ACF, however, maintains oversight responsibilities and must present an Annual Report to 
Congress on TANF.  The first Annual Report to Congress was released on August 5, 1998. 
)))))))))))
2  In November 1997, ACF issued proposed regulations to cover the work, accountability and data 
collection and reporting provisions of the new TANF program.  These regulations detail how 
States will collect data, how penalties will be assessed and define how participation rates will be 
determined.  Currently, ACF is reviewing comments received on the proposed regulations.  In the 
interim, States are reporting on a basic set of data elements covered by the Emergency TANF 
Reporting Requirements. 
To encourage this development of State level accountability, the Department has funded 23 
evaluations in 20 States to continue or modify evaluations of welfare reform efforts begun under 
waivers.  In addition, both ACF and the Assistant Secretary for Planning and Evaluation (ASPE) 
are working with States to evaluate the impact of welfare reform on child well-being.  The ASPE 
is also funding numerous studies looking at outcomes of families that are being diverted from or 
leaving TANF. 
METHODOLOGY 
The goal of this study was to gather descriptive data on how States are measuring outcomes of 
their TANF programs at this early stage of implementation.  To do this, we conducted a telephone 
survey of all 50 States plus the District of  Columbia (hereafter referred to as the States) from 
March 1998 to June 1998.  The surveys were administered to a representative of the TANF 
agency appointed by the head of that agency.  We asked these administrators how they were 
assessing outcomes of their TANF policies on the welfare of the children and families in their 
State.  We asked for current and planned outcome indicators, as well as any other efforts the 
State was involved in, to gather information regarding outcomes of their programs. 
We also conducted onsite visits in Oregon, Mississippi, Ohio, Connecticut, Massachusetts and 
North  Dakota.  These sites were selected to represent a county run program, high and low levels 
of overall block grant per caseload and longest running TANF programs.  In the States we visited, 
we interviewed the head of the human services department, the head of the TANF agency, county 
directors, front-line workers and an advocate group to more fully understand the implementation 
and use of outcome indicators throughout the organization. 
In addition to our survey work, we gathered additional information on States through their web 
sites.  We also interviewed knowledgeable organizations such as the National Governors 
Association, American Public Human Services Association, Welfare Information Network, Center 
for Legal and Social Policy and the General Accounting Office. 
For purposes of final analysis, we defined an outcome indicator as any client-based outcome 
measure at the point of exit from TANF or beyond.  We also made a distinction between what 
States are tracking through administrative data versus more formal evaluations.  Tracking 
consisted of following an outcome measure on a routine basis through access to one or more 
administrative databases.  Evaluations were anything from a simple, one-time telephone survey to 
more complex studies looking at longitudinal outcomes of a cohort of clients. 
)))))))))))
3  SCOPE 
This study is descriptive.  We made no attempts to assess the validity or reliability of the outcome 
indicators used by the States.  We also did not attempt to assess the methodology of State 
evaluations.  In addition, we did not review whether States were measuring work participation by 
TANF clients.  However, this inspection does begin to examine whether States will be able to 
measure outcomes of welfare reform on individuals and whether systems are being developed to 
accomplish the task of measuring outcomes. 
We conducted our review in accordance with the Quality Standards for Inspections issued by the 
Presidents Council on Integrity and Efficiency. 
)))))))))))
4  F I N D I N G S

Two-thirds of the States are beginning to use administrative data that track the 
short-term outcomes of former TANF clients. 
All States are required by PRWORA to measure the participation in work activities by current 
TANF clients.  Thirty-four States are beginning to measure client outcomes at the point of exit 
from TANF and beyond.  In many cases, the measures used by the States to examine outcomes 
only look at short-term results and do not examine longer term effects.  Most of these States are 
focusing on the areas of client self-sufficiency and welfare dependency.  A few are beginning to 
look at outcomes that measure areas such as child well-being and family well-being.  In a majority 
of States measuring the outcomes of former TANF clients, their measures were developed 
internally by the agency responsible for implementing TANF. 
State use of client based outcomes 
All States are required by PRWORA to report on participation rates by TANF clients in work 
activities.  Although not required, almost all States measure some type of process or output 
measures such as:  number of new cases, percent of TANF cases receiving child support and 
payment accuracy.  In addition to process or 
output measures, 34 States reported 
measuring at least one client-based outcome 
Table 1 
measure. 
Client-Based Outcome Measures 
Many of the client-based outcome measures 
Type of Outcome Measure  # of States 
that States use are primarily focused on 
aspects of a clients self-sufficiency.  For  Economic Self-Sufficiency  25* 
example, 25 States measure some indicator of 
- Job Retention  12 
economic self-sufficiency (See Table 1). 
States measure these indicators anywhere 
- Employment  9 
from 30 days to 18 months after the client has 
left the TANF program.  The other  - Earnings & Wages  9 
predominate area measured through 
- Job Benefits  5 
administrative data are those indicators 
looking at welfare dependency.  Sixteen  Welfare Dependency  16 
States use their administrative data to measure 
- Recidivism  14 
welfare dependency (See Table 1).  Eleven 
States use at least one indicator measuring 
- Use of Government 
both economic self-sufficiency and welfare 
Programs (e.g. food 
dependency. 
stamps and Medicaid)  3 
*States can use more than one outcome measure
)))))))))))
5  Only a few States are beginning to use administrative data to look at outcome indicators that go 
beyond client self-sufficiency and welfare dependency.  Two such measures are child well-being 
(child health, education and social development) and family well-being (families ability to provide 
a healthy, supportive and safe environment).  One way in which these States are gathering 
information about these indicators is through the use of data matching with other administrative 
data systems.  States use Unemployment Insurance data to understand how much former TANF 
clients are earning.  Other systems that are beginning to be used include child welfare and the new 
hire database.  Through the use of these data matches, States are trying to get a universal picture 
of what is happening to their former clients. 
However, administrative data is only as good as the information initially entered and is limited to 
what is collected for the administration of the program for which the system was created.  In 
addition, as States report the results from their administrative data, the definitions used by States 
may differ. 
State development of outcome measures 
In 20 of the 34 States using client-based outcome measures, these measures were developed 
internally by the agency responsible for TANF or in consultation with the Governors office.  In 
some cases, the State agency used focus groups with stakeholders including advocacy groups to 
help define the outcome measures that would be used.  Another State brought in outside help 
from the local universities to help them define what outcomes they should measure. 
In the remaining 14 States, State legislatures were directly or indirectly involved in the 
development or use of client-based outcome measures.  When a State legislature was directly 
involved, this involvement consisted of mandating in the States welfare reform legislation what 
outcomes would be measured.  For example, Minnesotas State legislature required seven 
performance measures that included job retention and recidivism as well as caseload reduction and 
cost of job placement into an unsubsidized job.  Another State legislature mandated that the 
TANF agency assess child well-being in the homes of families terminated from TANF due to time 
limits. 
Where the legislatures were indirectly involved in setting the outcome measures, the legislature 
usually set up an advisory council or task force that developed goals and measures for impact.  In 
some cases their involvement was more general in that they mandated that the TANF agency 
develop measures that were focused on outcomes. 
)))))))))))
6  Almost all States are planning or beginning to conduct evaluations examining the 
outcomes of TANF families and children; however, evaluations vary widely in scope 
and methodological rigor and few results are currently available. 
Thirty-eight States are currently conducting some type of evaluation examining the outcomes of 
their TANF program on children and families.  Ten more States have plans to begin evaluations 
on client outcomes in the near future.  Of the 34 States using administrative data to track client 
outcomes, 27 are supplementing that information with evaluations.  Typically, evaluations are 
designed to provide administrators with a more sophisticated understanding of their programs 
impact.  They offer a more complex analysis of client outcomes, track a broader range of outcome 
measures and indicators, and collect information on a wider selection of former or diverted cases. 
While States have initiated numerous evaluation efforts, most are currently in the planning or early 
implementation phases.  Only a handful of State evaluations have issued findings, even on an 
interim basis.  Thus, the information generated from evaluations that TANF administrators 
currently have at their disposal is limited.  However, several States are planning on issuing interim 
reports based on their evaluations even for studies scheduled to last several years. 
The evaluations that States are conducting or planning range widely in scope.  They range from 
simple one-time surveys to assess outcomes of a specific policy or pinpoint a problem to highly 
complex, longitudinal studies that track a random sample of clients on a variety of outcomes. 
Louisiana is conducting a simple, one-time phone survey of the last 1,000 clients to leave the rolls 
in order to find out why clients are leaving the rolls and how they are doing.  On the other end of 
the spectrum, Illinois is leading a consortium of universities to craft a longitudinal study to assess 
outcomes of TANF on large versus small cities, urban versus rural populations, different racial 
groups, family structure, welfare cycling and other public systems. 
Evaluations also vary widely in their methodological rigorousness.  The majority of evaluations 
are designed to simply gather a broad range of client outcome information.  Only 12 States have 
evaluations that utilize an experimental design.  By establishing control groups, experimental 
methodologies attempt to establish causality from program intervention to client outcomes by 
holding extraneous factors constant. 
The impetus for evaluations usually arises out of the TANF agency or the broader human services 
department.  Program administrators (ultimately responsible for answering to the public, the press 
and State legislatures) have a vested interest in finding out outcomes of their programs.  On the 
other hand, in some States, the legislature has taken the initiative and mandated that an evaluation 
be undertaken.  In these instances, it is typical for the legislature to set up an advisory group to 
plan the evaluation or to outline for the TANF agency the broad areas of concern they would like 
addressed.  Generally, the State legislatures want a picture of what happens to clients once they 
go off the rolls, including such topics as job placement, job retention and child and family well-
being.  For example, the Arkansas legislature requires an independent evaluation with biannual 
)))))))))))
7  reports.  The law requires that the evaluation look at nine items including the effect on children, 
job training and business participation. 
Evaluations capture a wider selection of former TANF cases 
State administrators turn to evaluations to look at clients that cannot be tracked through 
administrative databases.  Administrative data is limited, with the exception of Unemployment 
Insurance data, to those clients still receiving services from a State agency.  Therefore, 
administrators turn to evaluations to gather information on former clients who no longer show up 
on government databases, clients who have been sanctioned off the program, applicants who 
applied but never entered the system and clients diverted by front-end services provided by the 
State. 
Over half the States are attempting to monitor outcomes by conducting longitudinal surveys that 
track a cohort of clients from 1 to 5 years.  For example, Mississippi is tracking a sample of 
closed cases over a three year period to ascertain the long-term employment experience of the 
adult members of the case.  This evaluation will also cover housing status, benefit history, 
educational status of adults and children and health status. 
In addition to longitudinal evaluations, eight States are currently surveying diverted, denied or 
sanctioned cases.  States assessing outcomes of sanctions typically utilize surveys to obtain 
information regarding the impediments that caused the sanctions, the clients understanding of 
State sanction policy and any financial hardships caused by the sanction.  Michigan is planning a 
more in-depth assessment of sanctioned cases with the help of the Department of Community 
Health.  Local health department nurses will conduct home visits to all sanctioned cases to assess 
child well-being indicators as well as help the children retain Medicaid benefits. 
Evaluations encompass a broader range of outcome measures 
States typically use their evaluation efforts to target client outcomes that go beyond the outcome 
measures that can be captured through administrative data.  While the routine tracking of 
outcome measures is timely and efficient, it is often limited by data elements designed to 
accommodate administrative needs.  Evaluations, in contrast, offer the ability to capture a broad 
range of information. 
Many evaluations include outcome measures that move beyond the basic economic security 
concerns of welfare dependency and self-sufficiency that States typically tracked via 
administrative databases.  Evaluations tend to use a broad range of outcome measures to examine 
outcomes of TANF on the overall physical and social well-being of children and families.  They 
strive to answer the harder question of is the family better off?. 
Typical of the outcome measures States are generating from these types of evaluations is child 
well-being.  This measure may include such indicators as:  child care arrangements, abuse and 
)))))))))))
8  neglect, foster care caseload, child custody information, school attendance, immunizations and 
developmental outcomes.  Family well-being is another measure often assessed through 
evaluations.  This measure may include such indicators as:  family structure and stability, housing 
arrangements, substance abuse and domestic violence.  Other issues States are addressing through 
evaluations include housing, health care and transportation. 
Evaluations provide a more sophisticated analysis of outcome measures 
Although most evaluations cannot establish a causal link between TANF programs and client 
outcomes, they do offer a more accurate and reliable picture of client outcomes by allowing for 
the analysis of several variables and their interrelationships.  Data matching between 
administrative databases is still in the early stages of implementation in most States.  In addition to 
the early stage, administrative databases are limited by the data collected.  Consequently, many 
States are turning to evaluations to answer not only complex questions regarding client outcome 
but also to add rigor to their analysis and collect measures that are not available through 
administrative data. 
Evaluations focus on such questions as:  what are the outcomes of time limits on clients, what are 
the outcomes of work first and post-employment services on job entry and job retention and how 
do clients who voluntarily leave the welfare rolls support themselves.  Iowa is planning an 
evaluation that will examine the effect of parents moving from unemployment to employment on 
children and families.  Alaska is planning to examine the long-term effects of their TANF program 
on substance abuse and domestic violence within families. 
Despite the complexity and sophistication of some States evaluation plans, evaluations are often 
compromised by low response rates and other technical considerations.  Tracking former welfare 
clients to assess the long-term effects of TANF is a difficult, time-consuming task.  It is often 
impossible to locate former clients for follow-up interviews due to frequently shifting housing 
arrangements.  The inability to reach families by telephone and competing resource demands also 
hamper States efforts to locate former clients.  Beyond the task of locating former clients, 
securing their cooperation to answer questions can also present a problem.  One State reported 
the outcomes of an evaluation with only 18 percent of former clients responding.  Other 
considerations include sampling bias and the validity of responses that are presented to be 
representative of the State. 
Few States believe their current efforts to monitor outcomes of welfare reform are 
sufficient. 
Only 10 States believe their current efforts to monitor outcomes of their welfare reform programs 
to be sufficient.  Thirty-seven State respondents thought their present assessment efforts were not 
sufficient to fully determine outcomes of their welfare reform programs.  However, many of these 
)))))))))))
9  States believe that their current, planned or envisioned evaluations, when completed, will provide 
them with sufficient information to judge outcomes of their programs. 
Of the 10 States that believe their current efforts to be sufficient, 5 are using both administrative 
data and evaluations to monitor outcomes of welfare reform.  The remaining five monitor impact 
either through outcome measures using their administrative data systems or through evaluations. 
There appears to be little relationship between the breadth of State efforts to monitor outcomes of 
welfare reform and respondents beliefs in the sufficiency of information they are currently 
collecting. 
Overall, the data believed to be essential to determine outcomes of welfare reform programs is 
varied.  In some cases, this belief is dependent on the capabilities of their data systems (e.g. inter-
and intra-departmental programs interface) or resource limitations.  Barring any such restrictions, 
many respondents (both those that believe their efforts to be sufficient and those that do not) wish 
to track welfare clients off assistance over extended periods of time, ranging from 1 to 10 years. 
These evaluations, often longitudinal, would focus primarily on self-sufficiency and economic, 
family and child well-being after clients have left cash assistance. 
)))))))))))
10  I S S U E S  F O R  C O N S I D E R A T I O N

As previously stated, the intent of this study was to provide descriptive information regarding 
State efforts to measure the effects of their TANF program.  With a 41 percent drop in the 
welfare rolls, from 14 million recipients in January 1993 to 8.4 million recipients in June 1998, 
policy makers and the public want to know what is happening to families that go off the rolls. 
This study, along with others,  takes the first step in answering that question by documenting the 
data gathering efforts States are involved in, at this early stage of implementation, to assess the 
outcomes of their TANF programs. 
To gain a more thorough understanding of States ability to track the outcomes of their programs, 
successive research efforts need to address a series of issues raised in this report.  Issues for future 
consideration include: 
<  exploring the capacity of States to track outcome measures using their administrative 
databases, 
<  validating the data used for the outcome measures, 
<  reviewing the quality and scope of States evaluations, and, 
<  assessing the breadth and depth of a States overall efforts to measure outcomes of 
welfare reform. 
By answering these research questions, we will know if States have been successful, not only in 
cutting back the welfare rolls, but also in improving the lives of low-income children.  As one of 
the agents in the Department responsible for overseeing program effectiveness and beneficiary 
protection, the Office of Inspector General is vitally interested in knowing if States have 
established appropriate client outcome measures. 
AGENCY COMMENTS 
We received comments from ACF, ASPE  and the Assistant Secretary for Management and 
Budget (ASMB).  Their comments are included in Appendix A.  We made revisions to the report 
based on their comments. 
Both ACF and ASPE suggested that to ensure that people do not infer that States are conducting 
studies that can causally link the efforts of welfare reform and the outcomes of clients, we limit 
our use of the word impact.  In our draft report, we defined the word impact broadly to 
encompass all client-based outcomes associated with TANF intervention and not only those 
results which could be proven to be causally linked to TANF through the use of experimental 
methodology.  We agree that this could be confusing, so to limit that problem we have replaced 
the word impact with outcomes in our report. 
The ASMB pointed out that authority for the administration of welfare reform programs rests 
with the States and that they may be understandably reluctant to volunteer un-mandated data 
)))))))))))
11  that is not yet universally accepted as reflecting necessary and appropriate outcomes.  However, 
noting that alternative measures are feasible and are being tested, ASMB suggested that the 
Department of Health and Human Services participation in program assessment may 
appropriately involve facilitating the collaboration of State and Federal stakeholders rather than 
the traditional forms of oversight.  We agree with this observation, and it is in that spirit that we 
offer this report. 
)))))))))))
12  A P P E N D I X  A

AGENCY

COMMENTS

)))))))))))
A - 1  )))))))))))
A - 2
 )))))))))))
A - 3
 )))))))))))
A - 4
 )))))))))))
A - 5
 )))))))))))
A - 6
 )))))))))))
A - 7