Department of Health and Human Services 

OFFICE OF 

INSPECTOR GENERAL 

JUNE GIBBS BROWN 

Inspector General 

JANUARY 1999 

OEI-05-98-00130 

Early Efforts by States to Monitor 

Outcomes of Welfare Reform 
OFFICE OF INSPECTOR GENERAL 

The mission of the Office of Inspector General (OIG), as mandated by Public Law 95-452, is to 

protect the integrity of the Department of Health and Human Services programs as well as the 

health and welfare of beneficiaries served by them. This statutory mission is carried out through a 

nationwide program of audits, investigations, inspections, sanctions, and fraud alerts. The 

Inspector General informs the Secretary of program and management problems and recommends 

legislative, regulatory, and operational approaches to correct them. 

Office of Evaluation and Inspections 

The Office of Evaluation and Inspections (OEI) is one of several components of the Office of 

Inspector General. It conducts short-term management and program evaluations (called 

inspections) that focus on issues of concern to the Department, the Congress, and the public. The 

inspection reports provide findings and recommendations on the efficiency, vulnerability, and 

effectiveness of departmental programs. 

OEI's Chicago regional office prepared this report under the direction of William C. Moran, 

Regional Inspector General and Natalie Coen, Deputy Regional Inspector General. Principal OEI 

staff included: 

REGION HEADQUARTERS 

Thomas F. Komaniecki, Project Leader Linda Hall, Program Specialist

Ann Maxwell Ann O’Connor

Suzanne Johnson

Victoria Jacobs

To obtain copies of this report, please call the Chicago Regional Office at (312) 353-4124.

Reports are also available on the World Wide Web at our home page address:

http://www.dhhs.gov/progorg/oei
EXECUTIVE SUMMARY

PURPOSE 

To examine State efforts in measuring outcomes of the Personal Responsibility and Work 

Opportunity Reconciliation Act. 

BACKGROUND 

The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) (P.L. 

104-193) replaced the Aid to Families with Dependent Children entitlement program with 

Temporary Assistance for Needy Families (TANF) block grants to the States. This legislation 

also altered the framework for welfare policy by emphasizing program outcomes over process 

measures and by devolving the responsibilities of welfare administration from the Federal 

Government to the States. 

While PRWORA does not require States to develop client-based outcome measures, many States 

have begun developing outcome measures to understand the effects of their programs. With a 41 

percent drop in the welfare rolls, from 14 million recipients in January 1993 to 8.4 million 

recipients in June 1998, policy makers and the public want to know what is happening to families 

that go off the rolls. 

We asked State administrators how they were assessing outcomes of their TANF policies on the 

children and families in their State. For purposes of final analysis, we defined an outcome 

indicator as any client-based outcome measure at the point of exit from TANF or beyond. We 

also made a distinction between what States are tracking through administrative data versus more 

formal evaluations. 

This inspection is descriptive. We made no attempts to assess the validity or reliability of the 

outcome indicators used by the States. We also did not attempt to assess the methodology of 

State evaluations. However, this inspection does begin to examine whether States will be able to 

measure outcomes of welfare reform on individuals and whether systems are being developed to 

accomplish the task of measuring outcomes. These are questions that the Congress, State 

legislatures and the media have been consistently asking and writing about. 

FINDINGS 

Two-thirds of the States are beginning to use administrative data that track the short-term 

outcomes of former TANF clients. 

))))))))))) i 
All States are required by PRWORA to measure the participation in work activities by current 

TANF clients, however, 34 States are beginning to measure the outcomes of former TANF 

clients. Most of these States are focusing on the areas of economic self-sufficiency (e.g. job 

retention, employment, earnings and job benefits) and welfare dependency (e.g. recidivism and use 

of other government programs). Only a few are beginning to look at outcomes that measure 

areas such as child well-being and family well-being. The outcomes measured by administrative 

data can be limited by the availability and quality of the data and the definitions of outcomes used 

by States. 

Almost all States are planning or beginning to conduct evaluations examining the outcomes 

of TANF families and children; however, evaluations vary widely in scope and 

methodological rigor and few results are currently available. 

Thirty-eight States are currently conducting some type of evaluation that examines the outcomes 

of children and families. The majority of these evaluations are in the early implementation phase 

of the evaluation. Typically, evaluations are designed to provide administrators with a more 

complex analysis of the effect of TANF and a broader range of outcome measures for a wider 

selection of the former caseload. 

The evaluations that States are conducting or planning range widely in scope and methodological 

rigor. Their scope ranges from one-time surveys to longitudinal studies that track a random 

sample of clients. Currently, only 12 States have evaluations that use an experimental design; the 

majority of evaluations simply gather a broad range of client outcomes. 

Few States believe their current efforts to monitor outcomes of welfare reform are 

sufficient. 

Only 10 States believe their current efforts to monitor outcomes of their welfare reform programs 

to be sufficient. Of these 10 States, 5 used both evaluations and administrative data to track client 

outcomes, while the remaining 5 States used either evaluations or administrative data to monitor 

outcomes of welfare reform. There appears to be little relationship between the breadth of State 

efforts to monitor outcomes of welfare reform and respondents’ beliefs in the sufficiency of 

information they are currently collecting. 

ISSUES FOR FURTHER CONSIDERATION 

This inspection documents States’ efforts to assess the outcomes of their TANF programs at this 

early stage of implementation. It is intended to provide a foundation for future research efforts. 

Issues for future consideration include: 

< exploring the capacity of States to track outcome measures using their administrative 

databases, 

< validating the data used for the outcome measures, 

))))))))))) ii 
< reviewing the quality and scope of States’ evaluations, and,

< assessing the breadth and depth of a State’s overall efforts to measure outcomes of

welfare reform. 

By answering these research questions, we will know if States have been successful, not only in 

cutting back the welfare rolls, but also in improving the lives of low-income children and families. 

As one of the agents in the Department responsible for overseeing program effectiveness and 

beneficiary protection, the Office of Inspector General is vitally interested in knowing if States 

have established appropriate client outcome measures. 

AGENCY COMMENTS 

We received comments from the Administration for Children and Families (ACF), the Assistant 

Secretary for Planning and Evaluation (ASPE) and the Assistant Secretary for Management and 

Budget (ASMB). Their comments are included in Appendix A. We made revisions to the report 

based on their comments. 

Both ACF and ASPE suggested that to ensure that people do not infer that States are conducting 

studies that can causally link the efforts of welfare reform and the outcomes of clients, we limit 

our use of the word “impact.” In our draft report, we defined the word “impact” broadly to 

encompass all client-based outcomes associated with TANF intervention and not only those 

results which could be proven to be causally linked to TANF through the use of experimental 

methodology. We agree that this could be confusing, so to limit that problem we have replaced 

the word “impact” with “outcomes” in our report. 

The ASMB pointed out that authority for the administration of welfare reform programs rests 

with the States and that they may be understandably reluctant to volunteer “un-mandated” data 

that is not yet universally accepted as reflecting necessary and appropriate outcomes. However, 

noting that alternative measures are feasible and are being tested, ASMB suggested that the 

Department of Health and Human Services’ participation in program assessment may 

appropriately involve facilitating the collaboration of State and Federal stakeholders rather than 

the traditional forms of oversight. We agree with this observation, and it is in that spirit that we 

offer this report. 

))))))))))) iii 
TABLE OF CONTENTS

EXECUTIVE SUMMARY

INTRODUCTION .......................................................... 1

FINDINGS ................................................................ 5

Use of administrative data .................................................... 5

Launching of evaluations .................................................... 7

Adequacy of current monitoring activities ........................................ 9

ISSUES FOR CONSIDERATION ............................................ 11

AGENCY COMMENTS .................................................... 11

APPENDIX

A: Agency Comments ................................................... A-1
INTRODUCTION

PURPOSE 

To examine State efforts in measuring outcomes of the Personal Responsibility and Work 

Opportunity Reconciliation Act. 

BACKGROUND 

The Personal Responsibility and Work Opportunity Reconciliation Act of 1996 (PRWORA) 

(P.L. 104-193) replaced the Aid to Families with Dependent Children (AFDC) entitlement 

program with Temporary Assistance for Needy Families (TANF) block grants to the States. The 

TANF program requires States to meet increasing work participation goals and places a lifetime 

limit of 5 years on benefits. This legislation also altered the framework for welfare policy by 

emphasizing program outcomes over process measures and by devolving the responsibility of 

welfare administration and tracking from the Federal Government to the States. 

Welfare Reform Outcome Measures 

Under AFDC, States were required to collect information on many aspects of the program’s 

performance. Most of the measures used by the Administration for Children and Families (ACF) 

to judge the compliance of States in administering AFDC emphasized process goals such as 

whether applications were processed timely or cash benefits were accurate. The Government 

Performance and Results Act of 1993 (P.L. 103-62) shifted this departmental focus from process 

and output measures to outcomes and program effectiveness. The PRWORA continued this 

emphasis on outcomes by establishing four basic goals. The four goals outlined in the legislation 

are: 

< providing assistance to families so that children may be cared for in their own homes 

or in the homes of relatives; 

< ending dependence on government assistance by promoting job preparation, work and 

marriage; 

< preventing and reducing out-of-wedlock births; and, 

< encouraging the formation and maintenance of two parent families. 

Besides establishing these basic goals, PRWORA specified that States had to meet steadily rising 

work participation requirements for their clients along with establishing numeric goals for 

reducing illegitimacy. 

))))))))))) 1 
Congress established a “high performance” bonus and an “illegitimacy” bonus to reinforce these 

legislative goals. These bonuses offer financial rewards to top performing States. Proposed 

regulations for the illegitimacy bonus, issued on March 2, 1998, outline a plan to distribute $100 

million a year amongst the States demonstrating the largest decreases in out-of-wedlock 

childbearing while simultaneously reducing their abortion rates. Regulations have yet to be 

published regarding the high performance bonus. However, ACF did issue guidance on the high 

performance bonus in March 1998. Additional guidance on the high performance bonus was 

issued in August 1998. For Fiscal Year 1998, the interim formula weighs job placement, job 

retention and wage progression. 

While PRWORA does not require States to develop client-based outcome measures, many States 

have begun developing outcome measures to understand outcomes of their programs. However, 

States are at different stages of implementing these measures as well as their TANF programs. 

Currently, most of the public discussion regarding outcomes of welfare reform focuses on 

reducing welfare caseloads and getting clients into jobs. However, many people are beginning to 

question whether these are the only outcomes that should be used to measure outcomes of 

welfare reform. An article by a group of senior welfare administrators stated that “these two 

criteria [reducing welfare caseloads and getting clients into jobs] alone are not sufficient to 

measure the success of welfare reform.” They suggest that States’ initial areas of concern should 

include welfare dependency, self-sufficiency, economic well-being and parental responsibility. 

They believe that ultimately States should measure child well-being, parent well-being, family 

well-being, community involvement, economic development and family formation in order to 

understand the broader context in which welfare reform occurs. Many others also believe that 

more needs to be known to gauge outcomes of welfare reform. 

Federal Responsibilities 

The PRWORA also represents a major shift in the roles and responsibilities of the State and 

Federal Governments. While the Federal Government retains some measure of oversight, much 

of the accountability for policy decisions now rests at the State level. The new law limited 

Federal regulatory and approval authority to provide States with the authority and flexibility to set 

their own policies. It did, however, leave the Department with some responsibility for tracking 

State performance. 

The oversight role of ACF changed considerably with the passage of PRWORA. With the 

devolution of program design and responsibility to the States through the block granting of all 

funds to the State, States are now responsible for measuring outcomes of their programs. The 

ACF, however, maintains oversight responsibilities and must present an Annual Report to 

Congress on TANF. The first Annual Report to Congress was released on August 5, 1998. 

))))))))))) 2 
In November 1997, ACF issued proposed regulations to cover the work, accountability and data 

collection and reporting provisions of the new TANF program. These regulations detail how 

States will collect data, how penalties will be assessed and define how participation rates will be 

determined. Currently, ACF is reviewing comments received on the proposed regulations. In the 

interim, States are reporting on a basic set of data elements covered by the Emergency TANF 

Reporting Requirements. 

To encourage this development of State level accountability, the Department has funded 23 

evaluations in 20 States to continue or modify evaluations of welfare reform efforts begun under 

waivers. In addition, both ACF and the Assistant Secretary for Planning and Evaluation (ASPE) 

are working with States to evaluate the impact of welfare reform on child well-being. The ASPE 

is also funding numerous studies looking at outcomes of families that are being diverted from or 

leaving TANF. 

METHODOLOGY 

The goal of this study was to gather descriptive data on how States are measuring outcomes of 

their TANF programs at this early stage of implementation. To do this, we conducted a telephone 

survey of all 50 States plus the District of Columbia (hereafter referred to as the States) from 

March 1998 to June 1998. The surveys were administered to a representative of the TANF 

agency appointed by the head of that agency. We asked these administrators how they were 

assessing outcomes of their TANF policies on the welfare of the children and families in their 

State. We asked for current and planned outcome indicators, as well as any other efforts the 

State was involved in, to gather information regarding outcomes of their programs. 

We also conducted onsite visits in Oregon, Mississippi, Ohio, Connecticut, Massachusetts and 

North Dakota. These sites were selected to represent a county run program, high and low levels 

of overall block grant per caseload and longest running TANF programs. In the States we visited, 

we interviewed the head of the human services department, the head of the TANF agency, county 

directors, front-line workers and an advocate group to more fully understand the implementation 

and use of outcome indicators throughout the organization. 

In addition to our survey work, we gathered additional information on States through their web 

sites. We also interviewed knowledgeable organizations such as the National Governors’ 

Association, American Public Human Services Association, Welfare Information Network, Center 

for Legal and Social Policy and the General Accounting Office. 

For purposes of final analysis, we defined an outcome indicator as any client-based outcome 

measure at the point of exit from TANF or beyond. We also made a distinction between what 

States are tracking through administrative data versus more formal evaluations. Tracking 

consisted of following an outcome measure on a routine basis through access to one or more 

administrative databases. Evaluations were anything from a simple, one-time telephone survey to 

more complex studies looking at longitudinal outcomes of a cohort of clients. 

))))))))))) 3 
SCOPE 

This study is descriptive. We made no attempts to assess the validity or reliability of the outcome 

indicators used by the States. We also did not attempt to assess the methodology of State 

evaluations. In addition, we did not review whether States were measuring work participation by 

TANF clients. However, this inspection does begin to examine whether States will be able to 

measure outcomes of welfare reform on individuals and whether systems are being developed to 

accomplish the task of measuring outcomes. 

We conducted our review in accordance with the Quality Standards for Inspections issued by the 

President’s Council on Integrity and Efficiency. 

))))))))))) 4 
FINDINGS

Two-thirds of the States are beginning to use administrative data that track the 

short-term outcomes of former TANF clients. 

All States are required by PRWORA to measure the participation in work activities by current 

TANF clients. Thirty-four States are beginning to measure client outcomes at the point of exit 

from TANF and beyond. In many cases, the measures used by the States to examine outcomes 

only look at short-term results and do not examine longer term effects. Most of these States are 

focusing on the areas of client self-sufficiency and welfare dependency. A few are beginning to 

look at outcomes that measure areas such as child well-being and family well-being. In a majority 

of States measuring the outcomes of former TANF clients, their measures were developed 

internally by the agency responsible for implementing TANF. 

State use of client based outcomes 

All States are required by PRWORA to report on participation rates by TANF clients in work 

activities. Although not required, almost all States measure some type of process or output 

measures such as: number of new cases, percent of TANF cases receiving child support and 

payment accuracy. In addition to process or 

output measures, 34 States reported 

measuring at least one client-based outcome 

measure. 

Many of the client-based outcome measures 

that States use are primarily focused on 

aspects of a client’s self-sufficiency. For 

example, 25 States measure some indicator of 

economic self-sufficiency (See Table 1). 

States measure these indicators anywhere 

from 30 days to 18 months after the client has 

left the TANF program. The other 

predominate area measured through 

administrative data are those indicators 

looking at welfare dependency. Sixteen 

States use their administrative data to measure 

welfare dependency (See Table 1). Eleven 

States use at least one indicator measuring 

both economic self-sufficiency and welfare 

dependency. 

Table 1 

*

States can use more than one outcome measure

Client-Based Outcome Measures 

Type of Outcome Measure # of States 

Economic Self-Sufficiency 25* 

- Job Retention 12 

- Employment 9 

- Earnings & Wages 9 

- Job Benefits 5 

Welfare Dependency 16 

- Recidivism 14 

- Use of Government 

Programs (e.g. food 

stamps and Medicaid) 3 

))))))))))) 5 
Only a few States are beginning to use administrative data to look at outcome indicators that go 

beyond client self-sufficiency and welfare dependency. Two such measures are child well-being 

(child health, education and social development) and family well-being (families’ ability to provide 

a healthy, supportive and safe environment). One way in which these States are gathering 

information about these indicators is through the use of data matching with other administrative 

data systems. States use Unemployment Insurance data to understand how much former TANF 

clients are earning. Other systems that are beginning to be used include child welfare and the new 

hire database. Through the use of these data matches, States are trying to get a universal picture 

of what is happening to their former clients. 

However, administrative data is only as good as the information initially entered and is limited to 

what is collected for the administration of the program for which the system was created. In 

addition, as States report the results from their administrative data, the definitions used by States 

may differ. 

State development of outcome measures 

In 20 of the 34 States using client-based outcome measures, these measures were developed 

internally by the agency responsible for TANF or in consultation with the Governor’s office. In 

some cases, the State agency used focus groups with stakeholders including advocacy groups to 

help define the outcome measures that would be used. Another State brought in outside help 

from the local universities to help them define what outcomes they should measure. 

In the remaining 14 States, State legislatures were directly or indirectly involved in the 

development or use of client-based outcome measures. When a State legislature was directly 

involved, this involvement consisted of mandating in the State’s welfare reform legislation what 

outcomes would be measured. For example, Minnesota’s State legislature required seven 

performance measures that included job retention and recidivism as well as caseload reduction and 

cost of job placement into an unsubsidized job. Another State legislature mandated that the 

TANF agency assess child well-being in the homes of families terminated from TANF due to time 

limits. 

Where the legislatures were indirectly involved in setting the outcome measures, the legislature 

usually set up an advisory council or task force that developed goals and measures for impact. In 

some cases their involvement was more general in that they mandated that the TANF agency 

develop measures that were focused on outcomes. 

)))))))))))

6 
Almost all States are planning or beginning to conduct evaluations examining the 

outcomes of TANF families and children; however, evaluations vary widely in scope 

and methodological rigor and few results are currently available. 

Thirty-eight States are currently conducting some type of evaluation examining the outcomes of 

their TANF program on children and families. Ten more States have plans to begin evaluations 

on client outcomes in the near future. Of the 34 States using administrative data to track client 

outcomes, 27 are supplementing that information with evaluations. Typically, evaluations are 

designed to provide administrators with a more sophisticated understanding of their program’s 

impact. They offer a more complex analysis of client outcomes, track a broader range of outcome 

measures and indicators, and collect information on a wider selection of former or diverted cases. 

While States have initiated numerous evaluation efforts, most are currently in the planning or early 

implementation phases. Only a handful of State evaluations have issued findings, even on an 

interim basis. Thus, the information generated from evaluations that TANF administrators 

currently have at their disposal is limited. However, several States are planning on issuing interim 

reports based on their evaluations even for studies scheduled to last several years. 

The evaluations that States are conducting or planning range widely in scope. They range from 

simple one-time surveys to assess outcomes of a specific policy or pinpoint a problem to highly 

complex, longitudinal studies that track a random sample of clients on a variety of outcomes. 

Louisiana is conducting a simple, one-time phone survey of the last 1,000 clients to leave the rolls 

in order to find out why clients are leaving the rolls and how they are doing. On the other end of 

the spectrum, Illinois is leading a consortium of universities to craft a longitudinal study to assess 

outcomes of TANF on large versus small cities, urban versus rural populations, different racial 

groups, family structure, welfare cycling and other public systems. 

Evaluations also vary widely in their methodological rigorousness. The majority of evaluations 

are designed to simply gather a broad range of client outcome information. Only 12 States have 

evaluations that utilize an experimental design. By establishing control groups, experimental 

methodologies attempt to establish causality from program intervention to client outcomes by 

holding extraneous factors constant. 

The impetus for evaluations usually arises out of the TANF agency or the broader human services 

department. Program administrators (ultimately responsible for answering to the public, the press 

and State legislatures) have a vested interest in finding out outcomes of their programs. On the 

other hand, in some States, the legislature has taken the initiative and mandated that an evaluation 

be undertaken. In these instances, it is typical for the legislature to set up an advisory group to 

plan the evaluation or to outline for the TANF agency the broad areas of concern they would like 

addressed. Generally, the State legislatures want a picture of what happens to clients once they 

go off the rolls, including such topics as job placement, job retention and child and family wellbeing. For example, the Arkansas legislature requires an independent evaluation with biannual 

)))))))))))

7 
reports. The law requires that the evaluation look at nine items including the effect on children, 

job training and business participation. 

Evaluations capture a wider selection of former TANF cases 

State administrators turn to evaluations to look at clients that cannot be tracked through 

administrative databases. Administrative data is limited, with the exception of Unemployment 

Insurance data, to those clients still receiving services from a State agency. Therefore, 

administrators turn to evaluations to gather information on former clients who no longer show up 

on government databases, clients who have been sanctioned off the program, applicants who 

applied but never entered the system and clients diverted by front-end services provided by the 

State. 

Over half the States are attempting to monitor outcomes by conducting longitudinal surveys that 

track a cohort of clients from 1 to 5 years. For example, Mississippi is tracking a sample of 

closed cases over a three year period to ascertain the long-term employment experience of the 

adult members of the case. This evaluation will also cover housing status, benefit history, 

educational status of adults and children and health status. 

In addition to longitudinal evaluations, eight States are currently surveying diverted, denied or 

sanctioned cases. States assessing outcomes of sanctions typically utilize surveys to obtain 

information regarding the impediments that caused the sanctions, the client’s understanding of 

State sanction policy and any financial hardships caused by the sanction. Michigan is planning a 

more in-depth assessment of sanctioned cases with the help of the Department of Community 

Health. Local health department nurses will conduct home visits to all sanctioned cases to assess 

child well-being indicators as well as help the children retain Medicaid benefits. 

Evaluations encompass a broader range of outcome measures 

States typically use their evaluation efforts to target client outcomes that go beyond the outcome 

measures that can be captured through administrative data. While the routine tracking of 

outcome measures is timely and efficient, it is often limited by data elements designed to 

accommodate administrative needs. Evaluations, in contrast, offer the ability to capture a broad 

range of information. 

Many evaluations include outcome measures that move beyond the basic economic security 

concerns of welfare dependency and self-sufficiency that States typically tracked via 

administrative databases. Evaluations tend to use a broad range of outcome measures to examine 

outcomes of TANF on the overall physical and social well-being of children and families. They 

strive to answer the harder question of “is the family better off?”. 

Typical of the outcome measures States are generating from these types of evaluations is child 

well-being. This measure may include such indicators as: child care arrangements, abuse and 

))))))))))) 8 
neglect, foster care caseload, child custody information, school attendance, immunizations and 

developmental outcomes. Family well-being is another measure often assessed through 

evaluations. This measure may include such indicators as: family structure and stability, housing 

arrangements, substance abuse and domestic violence. Other issues States are addressing through 

evaluations include housing, health care and transportation. 

Evaluations provide a more sophisticated analysis of outcome measures 

Although most evaluations cannot establish a causal link between TANF programs and client 

outcomes, they do offer a more accurate and reliable picture of client outcomes by allowing for 

the analysis of several variables and their interrelationships. Data matching between 

administrative databases is still in the early stages of implementation in most States. In addition to 

the early stage, administrative databases are limited by the data collected. Consequently, many 

States are turning to evaluations to answer not only complex questions regarding client outcome 

but also to add rigor to their analysis and collect measures that are not available through 

administrative data. 

Evaluations focus on such questions as: what are the outcomes of time limits on clients, what are 

the outcomes of work first and post-employment services on job entry and job retention and how 

do clients who voluntarily leave the welfare rolls support themselves. Iowa is planning an 

evaluation that will examine the effect of parents moving from unemployment to employment on 

children and families. Alaska is planning to examine the long-term effects of their TANF program 

on substance abuse and domestic violence within families. 

Despite the complexity and sophistication of some States’ evaluation plans, evaluations are often 

compromised by low response rates and other technical considerations. Tracking former welfare 

clients to assess the long-term effects of TANF is a difficult, time-consuming task. It is often 

impossible to locate former clients for follow-up interviews due to frequently shifting housing 

arrangements. The inability to reach families by telephone and competing resource demands also 

hamper States’ efforts to locate former clients. Beyond the task of locating former clients, 

securing their cooperation to answer questions can also present a problem. One State reported 

the outcomes of an evaluation with only 18 percent of former clients responding. Other 

considerations include sampling bias and the validity of responses that are presented to be 

representative of the State. 

Few States believe their current efforts to monitor outcomes of welfare reform are 

sufficient. 

Only 10 States believe their current efforts to monitor outcomes of their welfare reform programs 

to be sufficient. Thirty-seven State respondents thought their present assessment efforts were not 

sufficient to fully determine outcomes of their welfare reform programs. However, many of these 

))))))))))) 9 
States believe that their current, planned or envisioned evaluations, when completed, will provide 

them with sufficient information to judge outcomes of their programs. 

Of the 10 States that believe their current efforts to be sufficient, 5 are using both administrative 

data and evaluations to monitor outcomes of welfare reform. The remaining five monitor impact 

either through outcome measures using their administrative data systems or through evaluations. 

There appears to be little relationship between the breadth of State efforts to monitor outcomes of 

welfare reform and respondents’ beliefs in the sufficiency of information they are currently 

collecting. 

Overall, the data believed to be essential to determine outcomes of welfare reform programs is 

varied. In some cases, this belief is dependent on the capabilities of their data systems (e.g. interand intra-departmental programs interface) or resource limitations. Barring any such restrictions, 

many respondents (both those that believe their efforts to be sufficient and those that do not) wish 

to track welfare clients off assistance over extended periods of time, ranging from 1 to 10 years. 

These evaluations, often longitudinal, would focus primarily on self-sufficiency and economic, 

family and child well-being after clients have left cash assistance. 

))))))))))) 10 
ISSUES FOR CONSIDERATION

As previously stated, the intent of this study was to provide descriptive information regarding 

State efforts to measure the effects of their TANF program. With a 41 percent drop in the 

welfare rolls, from 14 million recipients in January 1993 to 8.4 million recipients in June 1998, 

policy makers and the public want to know what is happening to families that go off the rolls. 

This study, along with others, takes the first step in answering that question by documenting the 

data gathering efforts States are involved in, at this early stage of implementation, to assess the 

outcomes of their TANF programs. 

To gain a more thorough understanding of States’ ability to track the outcomes of their programs, 

successive research efforts need to address a series of issues raised in this report. Issues for future 

consideration include: 

< exploring the capacity of States to track outcome measures using their administrative 

databases, 

< validating the data used for the outcome measures, 

< reviewing the quality and scope of States’ evaluations, and, 

< assessing the breadth and depth of a State’s overall efforts to measure outcomes of 

welfare reform. 

By answering these research questions, we will know if States have been successful, not only in 

cutting back the welfare rolls, but also in improving the lives of low-income children. As one of 

the agents in the Department responsible for overseeing program effectiveness and beneficiary 

protection, the Office of Inspector General is vitally interested in knowing if States have 

established appropriate client outcome measures. 

AGENCY COMMENTS 

We received comments from ACF, ASPE and the Assistant Secretary for Management and 

Budget (ASMB). Their comments are included in Appendix A. We made revisions to the report 

based on their comments. 

Both ACF and ASPE suggested that to ensure that people do not infer that States are conducting 

studies that can causally link the efforts of welfare reform and the outcomes of clients, we limit 

our use of the word “impact.” In our draft report, we defined the word “impact” broadly to 

encompass all client-based outcomes associated with TANF intervention and not only those 

results which could be proven to be causally linked to TANF through the use of experimental 

methodology. We agree that this could be confusing, so to limit that problem we have replaced 

the word “impact” with “outcomes” in our report. 

The ASMB pointed out that authority for the administration of welfare reform programs rests 

with the States and that they may be understandably reluctant to volunteer “un-mandated” data 

))))))))))) 11 
that is not yet universally accepted as reflecting necessary and appropriate outcomes. However, 

noting that alternative measures are feasible and are being tested, ASMB suggested that the 

Department of Health and Human Services’ participation in program assessment may 

appropriately involve facilitating the collaboration of State and Federal stakeholders rather than 

the traditional forms of oversight. We agree with this observation, and it is in that spirit that we 

offer this report. 

))))))))))) 12 
APPENDIX A

AGENCY

COMMENTS

))))))))))) A - 1 
))))))))))) A - 2
))))))))))) A - 3
))))))))))) A - 4
))))))))))) A - 5
))))))))))) A - 6
))))))))))) A - 7