Welfare Reform: Data Available to Assess TANF's Progress
Released: 2001-03-19T00:00:00

Letter

Background

PRWORA overhauled the nation’s welfare system by abolishing the previous welfare program, AFDC, and creating the TANF block grant. PRWORA established four broad goals for TANF, which included (1) providing assistance to needy families so that children may be cared for in their own homes or in the homes of relatives; (2) ending dependence of needy parents on government benefits by promoting job preparation, work, and marriage; (3) preventing and reducing the incidence of out-of-wedlock pregnancies; and (4) encouraging the formation and maintenance of two- parent families. Unlike the previous program, TANF gives states great flexibility to design programs that meet these goals. However, while states have flexibility, the programs they design must meet several federal requirements that emphasize the importance of work and the temporary nature of TANF. For example, PRWORA requires that parents receiving assistance engage in work, as defined by the state, after receiving assistance for 24 months, or earlier, at state option. In exercising their option, 28 states require immediate participation in work, and 9 other states require participation in work within 6 months of receiving cash assistance, resulting in great interstate variation in program provisions. Further, despite the programmatic flexibility authorized by TANF, states must meet federal data reporting requirements by submitting quarterly reports that include information from administrative records about those receiving welfare and those terminated from assistance, as well as an annual report, to HHS. The annual report contains information about program characteristics, such as states’ activities used to prevent out-of- wedlock pregnancy.

In 1995, we reported that the block grants enacted as part of the Omnibus Budget Reconciliation Act of 1981 (OBRA) carried no uniform federal information requirements. We found that the program information states collected was designed to meet individual states’ needs and that, as a result, it was difficult to aggregate states’ experiences and speak from a national perspective on the block grant activities or their effects. Without uniform information definitions and collection methodologies, it was difficult for the Congress to compare state efforts or draw meaningful conclusions about the relative effectiveness of different strategies. In a second examination of federal block grant programs, we reported that problems in information and reporting under many block grants—the Education Block Grant, the Community Services Block Grant, and the Alcohol, Drug Abuse, and Mental Health Services Block Grant—have limited the Congress’ ability to evaluate them. However, for the TANF Block Grant, the regulations require that states submit the quarterly TANF Data Report and the TANF Financial Report or be subject to statutory penalties. For these reports, HHS provides data reporting specifications including timing, format, and definitions for such data topics as family composition, employment status, and earned and unearned income. These specifications facilitate the use of HHS’ TANF administrative data for welfare reform research by improving the data’s comparability from state to state. Several national surveys and data collected for state and local studies of welfare reform also are potential sources of data for an assessment of TANF.

National Data Sources on TANF

A number of national surveys that collect information about welfare receipt have been used in the past by researchers to analyze welfare reform or have been developed to assess current welfare reform. Four surveys—the Survey of Income and Program Participation (SIPP), the Current Population Survey (CPS), the National Longitudinal Survey of Youth (NLSY), and the Panel Study of Income Dynamics (PSID)—have been used in past research on the AFDC program and the low-income population in general. Both the SIPP and the PSID have updated their questionnaires to include questions that pertain to welfare reform specifically, including questions about the work participation requirements and penalties for not complying with these and other program rules. Moreover, two national surveys are designed specifically to answer questions about welfare reform. The U.S. Census Bureau, at the direction of the Congress, is conducting a longitudinal survey of a nationally representative sample of families, with emphasis on eligibility for and participation in welfare programs, employment, earnings, the incidence of out-of-wedlock births, and adult and child well-being. This survey, the Survey of Program Dynamics, was designed to help researchers understand the impact of welfare reform on the well-being of low-income families and children. Similarly, the Urban Institute has been conducting a multiyear project monitoring program changes and fiscal developments, along with changes in the well-being of children and families. Part of this project includes a nationally representative survey of 50,000 people called the National Survey of America’s Families (NSAF) that is collecting information on the well-being of adults and children as welfare reform is implemented.

With the change in the fundamental structure of the nation’s welfare program, there have been several efforts by private research organizations to document the policies states have adopted under TANF. The Center for Law and Social Policy and the Center on Budget and Policy Priorities, in collaboration, have created the State Policy Documentation Project to document policies in all 50 states and the District of Columbia. Available on the Web, the State Policy Documentation Project contains information about state policies contained in statutes, regulations, and caseworker manuals, but it does not describe state practices. In addition, the Urban Institute has developed and made available to the public a database that documents changes in state program rules since 1996.

State and Local Data From Welfare Reform Research

Prior to and since TANF’s implementation, a considerable body of research about the low-income population has been conducted to examine the circumstances of families affected by welfare reform, the effectiveness of welfare reform initiatives, and the implementation of TANF at the state level. HHS has played a major role in laying the foundation for this welfare reform research. During the early 1990s, HHS granted waivers to states that allowed them to test various welfare reform provisions. In return, states were required to evaluate the effectiveness of the waiver provisions by randomly assigning welfare recipients to either participate in the waiver program or not. With the passage of TANF, states were given the option to continue their waiver evaluations as originally designed or modify the evaluation design. Several states opted to continue with their original random assignment design, while others modified their evaluation designs to focus on examining the implementation of the waivers or describe participants’ employment, earnings, and well-being. Because some elements of the waivers granted to states were incorporated into many TANF programs, the waiver evaluations provide useful insights into issues and designs for research about TANF.

However, according to HHS, one aspect of waiver policies may mean that some waiver evaluations may not represent TANF requirements completely. TANF established work requirements for all adult recipients, but states could delay adhering to these requirements under their TANF program, in part or whole, if the requirements were inconsistent with state waiver policies. Under the Job Opportunities and Basic Skills Training (JOBS) program, work requirements were mandatory for a work-ready or able-bodied population, excluding a number of subgroups such as those caring for young children and the disabled. For the most part, states that continued the original random assignment design maintained some or all of the JOBS exemptions from work requirements and applied these exemptions in determining who was subject to time-limited assistance. Consequently, while these states’ waivers may incorporate other work policies prescribed under TANF, these policies would not be expected to affect the exempt population. In contrast, in states that do not claim JOBS exemptions from work requirements, all adults are subject to work requirements and time limits on assistance. Thus, while testing TANF-like policies, evaluations that continued the random assignment design may not fully reflect the experience, outcomes, or impacts of fully implemented TANF requirements.

In addition to the waiver evaluations, HHS, as well as private foundations, has provided funding for demonstration programs across the country. The demonstration programs are pilot projects designed to measure the effects of a particular strategy, rather than an entire program, on welfare recipients or those eligible to receive welfare. Many of these demonstration programs were intended to increase employment, decrease out-of-wedlock pregnancy, or promote marriage. For example, in the late 1980s, several demonstration programs aimed at decreasing teen pregnancy among welfare recipients were developed. One program, the New Chance Demonstration, randomly assigned teen mothers receiving welfare to participate in a program that offered education or training classes and other support services and then compared the accomplishments of these teen mothers with those of teen mothers who did not participate in the program.

Given states’ greater responsibility for welfare programs under PRWORA and the larger number of people leaving the welfare rolls, there has been general interest among program administrators and state and local policymakers about the condition of those who are no longer receiving TANF, otherwise known as “leavers.” In response to this concern, a growing body of research about leavers has been initiated at both the state and federal levels. Generally, researchers have found that once low-income families leave welfare, they become hard to keep track of. Moreover, we previously reported that studies of former TANF recipients’ status differ in important ways, including geographic scope, the time period covered, and the categories of families studied, which limits the comparability of the data across states. In order to facilitate cross-state study comparisons, the Office of the Assistant Secretary for Planning and Evaluation (ASPE) within HHS has issued guidance to states and the research community on developing comparable measures for commonly reported outcomes and defined these outcomes. In fiscal year 1998, ASPE awarded approximately $2.9 million in grants to 10 states and three large counties to study leavers, followed by additional grants in fiscal years 1999 and 2000. ASPE also has encouraged the researchers to use comparable measures.

Research is also being conducted to examine the effects of welfare reform in metropolitan areas or neighborhoods. This area of research is important because the caseload decline in urban areas has been substantially lower than in other areas of the country. Moreover, urban areas can have higher unemployment rates and a greater concentration of poverty than suburban or rural communities; thus, insights gathered from these studies will be useful in understanding the potential for the success of welfare reform in the event of an economic downturn. For example, one study—the Three City Study—will survey primarily low-income, single-mother families from poor and moderate-income areas in Boston, Chicago, and San Antonio, with half of those surveyed being TANF recipients. The survey will collect information on adult and family well-being, employment, and welfare receipt three times within 4 years.

Finally, a body of welfare reform research examines the implementation of TANF at the state and local levels. Since PRWORA has not only granted states greater responsibility for providing cash assistance but also changed the nature of cash assistance, it is important to learn how states and localities are coping with these changes. Much of the research about program implementation focuses on challenges faced by state, and in some cases local, administrators in implementing TANF. Typically, in this research qualitative data are collected by visiting state or local TANF agencies; reviewing program records; and interviewing agency officials, caseworkers, and clients. For example, the State Capacity Study conducted by the State University of New York, Rockefeller Institute of Government, is collecting data in 20 states about the implementation of TANF at the state level, such as the structure of government services and information systems used to track clients.

Framework for Assessing Data Availability

Because we expect much of the reauthorization debate to focus on TANF’s four legislative goals, the framework for our data assessment was based on those goals. To assess whether data exist to address the goals, we first created a list of “descriptive” and “effect” research questions relevant to each goal. Descriptive questions concern a low-income individual’s or family’s status or behavior, such as the receipt of TANF cash assistance or support services like transportation, housing, child care, or health services; an adult’s employment status and earnings; and a family’s reliance on non- TANF government benefits, such as Food Stamps, Medicaid, or the Earned Income Tax Credit. Effect questions concern the extent to which changes in an individual’s or family’s status or behavior, such as obtaining employment, earning income, avoiding out-of-wedlock births, or forming a two-parent family, are the result of the TANF program. These research questions represent the broad issues that the Congress will consider during TANF’s reauthorization.

To summarize our findings, we identified data categories associated with TANF’s goals, some of which are more narrowly focused than the research questions. The data categories represent combinations of topics we found in the data, such as employment and earnings or family and child well- being, that were associated with the research questions. Figure 1 shows the relationships among TANF’s goals, the research questions, and the data categories, several of which are associated with more than one question. We then compared the data categories with the HHS administrative data, the data collected by national surveys, and the data derived from existing and planned studies.

Our assessment of the data’s usefulness for determining TANF’s progress is based on the data’s strengths and weaknesses, the design of the survey or study for which the data were gathered, and the topics to which the data related. The criteria we used in assessing the strengths and weaknesses of survey data included survey sample size, the attrition rate of respondents from whom data were collected over time, and survey response rate. For administrative data, we examined the geographic scope and the comparability of the data among states. The design features examined included what the data collection method was, whether the data were collected at one point in time or at different points in time, and whether the data were used for descriptive analysis of TANF or AFDC program recipients and their families or analysis of the program’s effects. Data that can be used for descriptive analysis are useful for research that addresses questions in the descriptive column of figure 1, and data that can be used for analyses of effect are useful for questions in the effect column of the figure.

National, State, and Local Data Address Different TANF Goals

Together, national surveys, HHS administrative data, and data from state and local studies of welfare reform address TANF’s four legislative goals. The national data provide extensive information related to TANF’s goals of providing assistance to needy families and ending dependency on government benefits through job preparation, work, and marriage. State and local data not only address the same goals as the national data but in some cases also provide information related to the goals of preventing out- of-wedlock pregnancies and promoting family formation.

National Data Address Two TANF Goals but Provide Limited Information on Out- of-Wedlock Pregnancy Prevention and Family Formation Strategies

National data provide detailed descriptive information related to two of TANF’s goals, but limited information related to TANF’s goals of preventing out-of-wedlock pregnancies and promoting family formation. HHS administrative data and the six national surveys we examined—the CPS, NLSY, NSAF, PSID, SIPP, and SPD—provide descriptive information related to TANF’s goal of providing assistance to needy families, including information about the change in size and composition of the TANF caseload and the use of noncash assistance by current and former TANF recipients (see fig. 2).

National data also address TANF’s goal of ending dependence on government benefits by describing the circumstances of those receiving TANF and those who are no longer receiving TANF. HHS administrative records and national surveys provide descriptive information about TANF recipients’ participation in work activities, employment status, earnings, and other family well-being measures. HHS administrative records contain information only about whether a recipient is working and how much income that individual earns, while national surveys collect more detailed employment and earnings data, such as the types of jobs held and the hourly wage. National data are also available about family well-being measures, which provide information about how TANF’s focus on work and marriage may be changing the lives of low-income families. For instance, national surveys have information about the amount of personal income spent on health and housing, whether recipients or former recipients rent or own housing, and the well-being of children of welfare recipients. Several of the national surveys provide information about children’s school attendance or developmental status, while SIPP and SPD also collect data about the number of births to teenagers. SIPP is the only national survey we examined that contains information about whether parents have had to terminate their parental rights or give a child up for adoption.

National data related to the goals of preventing out-of-wedlock pregnancy and promoting family formation are limited. While all the national data sets include information about recipients’ and nonrecipients’ marital status, only HHS administrative records contain information about out-of-wedlock births among the TANF caseload. However, states did not begin reporting this information to HHS until fiscal year 2000.

Aside from information about welfare reform in general, national surveys and HHS collect information about several different groups of individuals affected by TANF, including those who remain on assistance, those who no longer receive TANF, those who are diverted from TANF, and those who are eligible but choose not to participate. HHS administrative data and all six national surveys collect data about current and former TANF recipients, but the type of information collected about these individuals differs. As figure 3 shows, only the NSAF and SIPP have data about those diverted from TANF, while the NLSY, NSAF, PSID, SIPP, and SPD have data about individuals who are eligible to receive TANF but do not.

State and Local Data Complement and Fill Gaps in the National Data

The state and local data we reviewed can be classified into four categories that complement and, in some cases, fill in gaps not covered by the national data. Waiver data come from evaluations that tested the effects of programs implemented by states under waivers approved by HHS prior to TANF. Demonstration data come from studies that tested the effectiveness of particular strategies aimed at individuals either receiving welfare or eligible to receive welfare. Leavers data come from administrative records and surveys that describe the circumstances of those who left welfare. Finally, metropolitan and community-based data come from studies that, in general, describe the circumstances of low- income families and TANF participants in specific metropolitan areas, neighborhoods, or communities.

Waiver data have been used to examine the effects of TANF-like provisions on welfare recipients’ employment status, birth rates, and marital status, as shown in figure 4. Several states have been evaluating the waiver provisions in their welfare programs by randomly assigning welfare recipients to either the waiver program or AFDC. Waiver programs require participants to follow provisions that later were required or permitted under TANF, such as being required to work or risk losing eligibility for benefits or being allowed to receive welfare for only a limited time. Most of the waiver program evaluations collected data used to analyze the effect of waivers on welfare receipt, employment, and income. Data from several of the evaluations have also been used to analyze the effects of waivers on out-of-wedlock pregnancy or family formation.

With the passage of PRWORA, several states incorporated their waiver provisions into their TANF program and have been collecting data about the experiences of participants in the program. Some of these states chose not to continue their evaluations as originally designed, instead conducting modified evaluations that typically involved studies that will provide information on the experience of implementing the program. For example, Montana is surveying TANF participants to collect data about the duration of their welfare receipt, the types of noncash assistance they use, and their employment.

Demonstration data provide information on topics that are similar to those addressed by waiver data and have also been used to analyze the effects of programs on their participants, but demonstration data differ in two key ways. First, most demonstration data, including all data related to pregnancy prevention and family formation, were collected before PRWORA was enacted. Second, demonstration data were collected for studies focused on how a particular approach affected program participants. In fact, many of the demonstration data we examined were used entirely to assess the effects of various strategies on participants’ employment status and earnings, which helps to distinguish the effects of particular provisions included in a program like TANF.

Leavers data provide descriptive information about those who have left welfare. This information includes the length of time an individual received TANF, reasons for leaving welfare, types of noncash assistance used, and employment and earnings information. In addition, some leavers data sets contain information about former recipients’ marital status, and a few have data about the number of pregnancies and births among former recipients.

Metropolitan and community-based data cover some of the same issues as the other data categories, including information about TANF work requirements and time limits. Although the same issues are addressed, the data are collected in large cities or neighborhoods in order to examine the circumstances of welfare recipients in areas that may have high concentrations of poverty or limited access to jobs. In addition, metropolitan and community-based data provide information about groups other than TANF recipients and former recipients—including individuals diverted from TANF and those who are eligible to participate in TANF but do not.

Strengths and Weaknesses of Existing Data Affect Usefulness for Assessing TANF’s Progress

Although existing data provide rich information about the lives of families who are receiving or have received TANF, the strengths and weaknesses of these data affect their usefulness for understanding welfare under the TANF block grant. National data can be analyzed to gain a descriptive picture of what has happened under TANF for the nation as a whole. However, of the seven national data sets we reviewed, only two can be used to describe the well-being of families receiving TANF within individual states. Although waiver and demonstration data can be analyzed to gain information about TANF’s effects, these analyses can be done within only a limited number of states and disparate localities. We examined nearly 40 data sets that could be analyzed for information about the circumstances of former recipients. However, only a subgroup of these data sets met criteria that allowed the sample to be generalized statewide. These data sets represented 15 states. In some cases, the value of survey data collected from those who left welfare was limited because few former recipients actually responded to the surveys: in some cases, former recipients could not be located, and in other cases they chose not to answer the questions posed to them. Metropolitan and community-based data can be analyzed to describe changes over time in the lives of welfare recipients in urban centers. Much of this data collection will continue beyond 2001.

National Survey Data Provide Descriptive Information About Nation, but Not Effects in Individual States

The strength of the national data is that they were collected from samples selected randomly from the nation’s population and include low-income families and TANF recipients in numbers sufficient to allow reliable estimates about these groups. In addition, most of the national data were collected for the same individuals over time, allowing changes in welfare recipients’ employment, earnings, and well-being to be tracked across programs implemented at different times. However, all the national surveys have participants who drop out of the survey sample over time, and this may limit how well the samples represent the nation’s welfare recipients.

National data are collected from random samples that contain low-income families and TANF recipients. Because samples from national surveys are selected randomly, they are, at the time of selection, representative of the population at large, including the welfare population. In addition, all the national data sets we reviewed have sample sizes large enough to allow reliable estimates about the nation’s low-income and TANF populations— as sample size increases, the degree of precision of the estimates made using that sample also increases (see table 1).

As shown in figure 5, two national data sources collect data on individuals at one point in time; others collect data on the same individuals across time. In both cases, the data can be used for comparisons between groups of individuals living under welfare provisions implemented at different time periods.

Five national surveys—the CPS, NLSY, PSID, SIPP, and SPD—collect data from the same individuals over time. For the SIPP, the Census Bureau, after a specified period, changes the group of individuals from whom data are collected. For example, the 1993 SIPP panel followed a group of individuals through 1996. In 1996, a new group was randomly selected and followed through 2000. Data collected over time could be analyzed to describe how people cycle on and off TANF, how their use of benefits changes over time, and how their family well-being changes. In addition, comparisons could be made between groups covered by different welfare provisions. For example, AFDC recipients included in the 1993-96 SIPP panel could be compared with TANF recipients who were part of the 1996-2000 SIPP panel. The NSAF, as well as HHS administrative records, has collected data from different samples of individuals in different years. For example, in 1997 one group of people completed the NSAF; another group completed the survey in 1999. In cases such as these, the samples from different years can be compared with each other to look for changes across time.

For those national surveys that collect information about changes in welfare across time, the likelihood that survey participants will drop out over time increases, potentially affecting how well the data actually represent all members of the nation’s low-income and TANF populations. In general, the greater the attrition rate, the less likely a sample is to be representative of the larger population from which it was drawn. Those who have continued participating in the survey may be different from those who stopped or dropped out. As surveys that collect data over time, the NLSY, PSID, SIPP, and SPD all have experienced sample loss, as shown in table 2. Concerns about attrition are especially significant for the SPD, because it was designed specifically to track welfare recipients from AFDC through TANF. Census has tried mathematically adjusting available responses to compensate for the survey’s sample loss, but this adjustment has not sufficiently remedied the problem, according to a Census official. Census will take steps to lessen attrition through intensive follow-up with survey dropouts to enlist their participation and through the use of monetary incentives for future respondents to the survey.

For national surveys, the response rate—the number of people in the survey sample who actually responded, compared with those who were asked to respond but did not—has been large enough to allow the survey results to be generalized beyond those who completed the survey, with the exception of the 1999 NSAF. Most practitioners of survey research, including GAO, require at least a 70- to 75-percent response rate before survey data can be generalized beyond those who completed the survey. As table 3 shows, the response rate for all the national surveys except the 1999 NSAF was at or above the 70-percent standard. Given the survey’s response rate, using the 1999 NSAF survey data would require determining whether patterns in who responded and who did not respond existed and what this means for how well respondents represent the original sample. For those surveys that collect data on the same individuals over time, response rates sometimes are considered in conjunction with rates of attrition.

The major limitation of most existing national data is that they cannot be used for state-level analyses. In general, national data sources have state sample sizes that are too small to allow reliable generalizations about TANF recipients within individual states. The NLSY, PSID, SIPP, and SPD collect data not from states per se, but from regions that, in some cases, include more than one state. Thus, while these data can be analyzed to provide a descriptive picture of TANF for the nation, they cannot be used within states for descriptive analyses or to analyze the effects of states’ TANF provisions. This does not mean that researchers do not use these data sources for state-level analyses. For example, some researchers combine several years of CPS data to obtain adequate sample sizes within states for state-level analyses. However, Census, which administers the CPS, SIPP, and SPD, does not recommend using data from these surveys for state-level analyses, because doing so when sample sizes are small may produce findings that are not reliable.

Two national data sources, HHS administrative records and the NSAF survey, can be used for state-level analysis, but with limitations. HHS administrative records provide data from all 50 states and the District of Columbia. However, the reporting requirements for these data are not completely standardized across states, so that how a variable is defined may vary among states. For example, each state may define the work or work-related activities in which TANF recipients participate as they think appropriate to the state program.

Like HHS administrative records, NSAF survey data can be used for state- level analyses. NSAF has samples large enough to allow state-level analyses in 13 states, representing 58 percent of the fiscal year 1999 national TANF caseload; this is not the case in the 37 remaining states. For example, the number of low-income children surveyed for the 1997 NSAF ranged from a low of 760 to a high of 1,813 in each of the 13 states where NSAF collected samples large enough to permit state-level analysis. However, the number of low-income children surveyed in the 37 remaining states averaged 35 per state, a number too small to allow reliable conclusions about the children of TANF recipients in any of these states.

Even if the issue of sample sizes within states were resolved, obstacles to using the national data to analyze TANF’s effects within states would still exist. The lack of information about the choices states have made about TANF policies and program rules has been identified as one of the challenges to using national data to analyze TANF’s effects. However, research organizations have collected this information. The Center for Law and Social Policy has worked with the Center on Budget and Policy Priorities to document policies in all 50 states and the District of Columbia, and the Urban Institute has developed a state database that documents state program rules. Yet, even with this information, using national data to measure state-level effects poses challenges.

The first challenge is deciding with whom TANF recipients should be compared. To test TANF’s effects, the employment, earnings, and well- being of individuals in the program must be compared with those of individuals who are not in the program. In the case of TANF, it would be difficult to determine what group should provide the point of comparison. Because waivers introduced TANF-like policies and program rules while AFDC was still in effect, it would be difficult to select a group of welfare recipients whose experiences with welfare were not influenced by TANF.

The second challenge is determining the effect of any single welfare provision given the multiple provisions that make up states’ TANF programs. For example, TANF recipients are required to work, and states must impose penalties or sanctions when recipients do not comply with work requirements. In such cases, it would be difficult to separate the combined effects of work requirements and any penalties or sanctions that were imposed into the individual effects of each.

A third challenge is detecting the long-term effects of state programs that have been recently implemented. Although PRWORA was enacted in 1996, states implemented their TANF programs at different points in time. Some states were still refining their TANF programs at the beginning of 1998. Consequently, the long-term effects of TANF may not yet be realized.

Finally, state-level analyses may not be the best way to measure TANF’s effects in every state. Some states have further devolved TANF to localities, and different localities may implement a state’s TANF provisions differently. In total, 17 states have given local governments responsibility for TANF program design and implementation.

Waiver and Demonstration Data Allow Assessment of TANF Effects, but Only for Localities

The strength of the waiver and demonstration data is that they can be used to analyze TANF’s effects, but with few exceptions these data were collected from city and county samples rather than statewide samples. (See app. II for the localities examined.) Most of the waiver and demonstration data were collected as part of experiments—studies that randomly assigned welfare recipients to groups that were subject to different welfare provisions. Experiments, when done correctly, are recognized as the most rigorous way of determining the extent to which an observed outcome can be attributed to the program itself, rather than to differences among the program participants. Over half of the waiver data sets and virtually all of the demonstration data sets we reviewed consisted of data from experiments. Of the waiver data sets, about half were collected from city and county samples, with the others being collected from statewide samples. All of the demonstration data sets were collected from city and county samples. Overall, 6 of the 54 waiver and demonstration data sets that could be used for analyses of effect were collected from statewide samples.

According to the project directors of two waiver evaluations, the high cost of conducting rigorous program evaluations may explain, in part, why data sets used to analyze TANF’s effects tend to use samples from cities and counties and not entire states. Given limited resources, researchers may choose to conduct rigorous evaluations in selected cities or counties rather than sacrifice rigor to evaluate a program statewide. Data sources we reviewed for both the Vermont and Iowa waiver evaluations mentioned budget constraints as a factor that led researchers to limit their data collection efforts.

Another limitation of the waiver and demonstration data is that most often they were collected prior to the implementation of TANF. This is not surprising given that in many cases the waiver provisions and the demonstration projects were intended to test provisions before they were adopted and implemented. However, the provisions tested may not have been those ultimately adopted by the state.

Finally, in almost all cases in which waiver evaluations and demonstration projects collected survey data, response rates were above the 70-percent standard (see table 4).

State-Level Analyses of Leavers Data Are Possible in 15 States; in Some Cases, Picture Is Incomplete

The strength of the leavers data is that in most cases, they were collected from statewide samples. However, in some cases, leavers data collected using surveys may not be representative of a state’s leaver population. Although we reviewed nearly 40 leavers data sets, on the basis of the type of data available, response rates, and the absence of significant differences between survey respondents and nonrespondents, we concluded that state- level analyses could be done for 15 states using the data sets we examined.

To be representative of a state’s leavers population, survey data need to meet the 70-percent standard for response rates, or, through a comparison of survey respondents with nonrespondents, show that the two groups do not differ significantly. When a state has both administrative data and survey data available, the administrative data could be used in place of survey data that are not representative. As figure 6 shows, Arkansas, Florida, Georgia, North Carolina, and South Carolina have either survey data that meet the standard for response rates or data from survey respondents who were not significantly different from nonrespondents. Arizona, Colorado, the District of Columbia, Illinois, Kansas, Missouri, Virginia, Washington, and Wisconsin have both administrative data and survey data. The response rate for the District of Columbia, Illinois, Kansas, Virginia, and Wisconsin was below 70 percent, but for Virginia, a comparison of respondents with nonrespondents revealed no significant differences between the two groups. Although New York has no survey data, its administrative data provide information about the state’s leavers. California, Massachusetts, and Texas are the three states for which, given the available data, state-level analyses of leavers cannot be done. We previously reported that eight leavers studies covering seven states had collected adequate information to allow the study findings to be generalized to the states’ welfare populations. Thus 4 states—Indiana, Maryland, Oklahoma, and Tennessee—can be added to the list of 15 states we identify in figure 6 as having data that can be generalized statewide. In appendix II we list all the sources we reviewed that provide data on those who have left welfare.

Some researchers may wish to compare those who left TANF with those who left AFDC on outcomes such as employment, earnings, and well-being. Contrasting outcomes for these two groups would require deciding which AFDC leavers provide the best point of comparison. Many factors specific to the year in which recipients left the welfare rolls would influence their employment prospects, wages, and well-being. For example, labor markets and economic conditions in a given year would influence former recipients’ employment opportunities. Historical influences such as these would complicate the issue of selecting a comparable group of AFDC leavers and TANF leavers.

Metropolitan and Community-Based Data Cover Urban Centers; Data Collection Will Continue Beyond 2001

The strength of the metropolitan and community-based data is that they can be used in descriptive analyses that provide information about how the lives of low-income families and TANF participants have changed over time. Because data collection is occurring over time, in some cases it has yet to be completed. For example, the Los Angeles Family and Neighborhood Study (LA FANS) is collecting data about participation in welfare programs from residents of 65 neighborhoods in Los Angeles County over a 4-year period. LA FANS began data collection in January 2000 and will continue data collection through 2004. Most of the materials we reviewed regarding metropolitan and community-based data sets did not report information about attrition rates. When response rates were reported, they were above the 70-percent standard. Figure 7 shows the time periods for which the data are or will be available for different metropolitan areas and communities.

Three of the metropolitan and community-based data sources have measures that can be used to analyze TANF’s effects, even though the data were not collected as part of an experiment. For example, data from the Fragile Families study can be used to examine TANF’s effects by drawing comparisons between the 3,675 unmarried parents and the 1,125 married parents who compose the survey sample in cities with populations over 200,000. Data collection for Fragile Families began in 1998 and will continue through 2004. The data have already been used to examine differences in relationship quality between married and unmarried couples, including whether a father gave money to or helped a mother in a nonmonetary way during pregnancy.

Existing Data Are More Adequate for State and Local Descriptive Analyses Than for Analyses of Effect

The current body of research on TANF addresses many issues of interest to the Congress but does not provide a comprehensive national picture of TANF. However, existing national data and data from state and local studies could be pieced together to develop a descriptive picture of what has happened to TANF participants in all 50 states. In addition, within a limited number of states and various cities and counties, existing data can be used to conduct analyses of TANF’s effects.

National survey data can be used with data from HHS administrative records for descriptive analyses of TANF’s progress nationwide. HHS administrative data can be used for analyses within each of the 50 states, and national survey data can be analyzed for national trends. These analyses could be compared to examine the extent to which the employment experiences, for example, of current and former TANF recipients in individual states conform with or depart from the experiences of such individuals identified with national survey data. This comparison could be extended to the individual states and localities covered by the NSAF data, waiver and demonstration data, leavers data, and metropolitan and community-based evaluation data.

While piecing the data together in this way would build on their strengths, each data type still has limitations. Specifically, national survey data provide national samples useful for comparing the lives of welfare recipients covered by welfare provisions implemented at different times.

However, attrition or low response rates may affect the degree to which these samples represent all members of the nation’s low-income and TANF population. Within each of the 50 states, HHS administrative data can be analyzed to gain insight into current recipients’ use of noncash benefits, among other things, but the lack of standardized reporting requirements would complicate comparisons across states. Supplemental descriptive analyses for individual states can be done using NSAF survey data, leavers data, waiver and demonstration data, and metropolitan and community- based data. In addition, like the national survey data, many of these data represent multiple measures over time. However, these analyses in many cases can be generalized only to cities and counties and not to entire states.

Existing data can also be analyzed to gain information about TANF’s effects. Although the 1997 and 1999 NSAF survey samples do not include pre-TANF welfare recipients, the samples do include other populations, such as low-income families who do not participate in TANF, whose employment, earnings, and well-being can be compared with those of TANF recipients, assuming adequate sample sizes for both groups. Moreover, because NSAF has sample sizes in 13 states large enough to allow state-level analyses, the employment, earnings, and well-being of TANF recipients in those states can be considered in relation to the state’s TANF programs and policies. However, using the NSAF data for such analyses would require resolving the challenges to analyzing effects described earlier in this report. Similarly, although most of the metropolitan and community-based evaluation samples do not include pre- TANF welfare recipients, other populations represented in the study samples could be compared with TANF recipients. Finally, waiver and demonstration data can be analyzed to gain information about TANF’s effects, keeping in mind that this information is about the effects of programs and provisions often implemented prior to TANF and implemented in cities and counties rather than entire states.

Concluding Observations

The data available for addressing TANF’s goals will provide useful information, but with some limitations. Given the costs, some limitations may be difficult to overcome. Our examination of the data raised three issues.

First, for a comprehensive assessment of TANF, it is important to have data for a representative sample of TANF recipients and nonrecipients that allow for analyses of effect at the state level. The federal government has made an investment in national surveys, which either in whole or in part are intended to gather information about the lives of TANF recipients. One of these, the SPD, was funded as a means to gather data about TANF recipients. For another, the SIPP, the Census Bureau added a special section of questions about welfare and reworded questions so that they would better capture respondents’ participation in state programs. However, even with these efforts, none of Census’ surveys currently being administered can be used for state-level analyses of TANF’s effects because of small sample sizes within individual states. In addition, the SPD has a high attrition rate. The Census Bureau plans to take steps to improve response to the SPD through intensive follow-up with survey dropouts to enlist their participation and through monetary incentives for future respondents to participate in the survey. However, the issue of small sample sizes at the state level will remain unresolved.

Second, HHS has encouraged state agencies to study the effects of their TANF programs through the AFDC waiver requirement for experimental studies and subsequent research initiatives. Moreover, our examination of data indicates that, because of the variability in TANF program provisions across states, analysis of TANF’s effects at the state and local levels can be done with the greatest confidence. However, even when conducted at the state and local levels, studies designed to examine TANF’s effects tend to be costly, time-consuming, and impractical to implement in every state. In some cases, conducting an evaluation for an entire state is determined to be so expensive that data collection is limited to a portion of the state. For example, the evaluation of Vermont’s waiver program focused on 6 of 12 welfare service districts. The evaluation’s 42-month follow-up survey was administered to only these 6 district offices and, owing to cost constraints, included a subset of the sample for whom administrative records, rather than survey responses, were collected. Policymakers, federal and state officials, and the welfare reform research community will need to seek ways to balance the need for information about TANF’s effects with the resource demands of rigorous studies.

Third, both qualitative and quantitative data may be needed to understand what has happened to former TANF recipients. Leavers are a difficult population to track, and, in some cases, using multiple methods of quantitative data collection has not necessarily increased the number of former recipients who could be located or who responded to surveys. In fact, in some of the studies we reviewed, the low rate of success in gathering data from these individuals makes the data’s usefulness questionable. Surveys that used only one mode of data collection, such as telephoning former TANF recipients, generally had the lowest response rates. Some leavers’ studies followed telephone surveys with personal interviews of those who could not be reached by phone or who did not respond. However, even the use of multiple modes of data collection did not always ensure high response rates. Given the difficulties inherent in collecting quantitative data from this group, other data collection strategies that use local communication networks to identify families as well as interviews of respondents in their homes may be needed to gain information about the lives of TANF leavers.

Agency Comments

In commenting on a draft of this report, HHS said that the report will be of help to the Congress and other interested parties. In its technical comments, HHS expressed concern that in highlighting the importance of statewide samples, we understated the value of data from local samples. In response to this concern, we have noted in the report not only that findings from local samples are important but also that, in some cases, they provide data only recently available from national surveys. We concur with HHS that a sample need not be statewide in order for findings to be useful. However, we have emphasized the value of data that can be generalized to the state level because of the Congress’ interest in a picture of TANF’s progress nationwide. HHS’ comments appear in appendix IV.

As agreed with your offices, unless you publicly announce its contents earlier, we plan no further distribution of this report until 30 days after its issue date. At that time, we will send copies to the Honorable Tommy G. Thompson, Secretary of Health and Human Services; appropriate congressional committees; and other interested parties. We also will make copies available to others on request.

If you or your staff have any questions about this report, please contact me on (202) 512-7202 or David D. Bellis on (415) 904-2272. Another GAO contact and staff acknowledgments are listed in appendix V.

Scope and Methodology

This appendix discusses in more detail our scope and methodology for identifying, selecting, and assessing studies and surveys that might provide data to help researchers as they seek to describe what has happened to recipients of Temporary Assistance for Needy Families (TANF) and to estimate the effect of welfare reform on them.

Sample Design

Because no comprehensive list of data sources for welfare reform research exists, we used a judgmental sampling method for our assessment of data resources. We began our work by examining six key critiques of welfare reform research that had been issued, in draft or final form, by the fall of 1999. The six critiques listed in figure 8 both gave us insight into issues that will probably arise in assessing TANF and identified studies that may be potential sources of data for an assessment of TANF.

We started the development of a list of data sources from three of the critiques—the Research Forum’s report and its related on-line database, the National Research Council’s interim report, and Peter Rossi’s paper. To ensure that this list was comprehensive, we consulted with officials at the Department of Health and Human Services (HHS) about important bodies of work in the welfare reform research field. We also conducted follow-up interviews with HHS project officers and experts in the welfare reform research community to ensure that we had identified the most relevant national surveys and studies, particularly those that might have data about family, marriage, and pregnancy issues. As a result of these discussions and an examination of the original list, we designed a judgmental sample of potential data sources for welfare reform research that included the following categories: national surveys and HHS’ TANF administrative data; studies that collected data about the major TANF subpopulations in three or more states or municipalities; studies of TANF leavers; HHS’ waiver evaluations; and studies listed on the websites of HHS’ Administration for Children and Families (ACF), HHS’ Office of the Assistant Secretary for Planning and Evaluation (ASPE), and the Welfare Information Network of the Finance Project.

We then began to develop lists of the surveys and studies in each of the sample’s categories.

The national surveys included in our list were the Current Population Survey (CPS), the National Longitudinal Survey of Youth (NLSY), the National Survey of America’s Families (NSAF), the Panel Study of Income Dynamics (PSID), the Survey of Income and Program Participation (SIPP), and the Survey of Program Dynamics (SPD). We used information from ASPE and from the National Conference of State Legislatures to identify leavers studies sponsored by HHS or states. Similarly, we used information from ACF to ensure that our list contained the body of research funded by ACF that focused on waivers implemented by state welfare agencies prior to TANF’s authorization. As we added items to the list, we continually checked to avoid any duplication. This comparison involved our judgment, as some lists were of projects or studies and others were of study reports. Because we relied on multiple reviews of the body of work undertaken in the welfare reform research community, we believe that the list of 443 entries we compiled included the key sources of data.

Sample Selection

We selected surveys and studies systematically from this list within each sample category. We were interested in surveys or studies that were as comprehensive as possible in geographic coverage and topics addressed. Thus, we selected all of the national surveys and the HHS administrative data. We also selected all studies on the original list that by their description appeared to have produced data concerning the major subpopulations affected by TANF in three or more states, municipalities, or counties. This resulted in 55 studies and surveys.

We then selected studies that pertained to individual states in the following way. First we selected all leavers studies financed by ASPE. Of the leavers studies listed by the National Conference of State Legislatures and those mentioned in an article authored by Brauner and Loprest, we included only those that had not been included in our previous report or were not from a state that already had an ASPE-funded study. In states that had issued multiple reports for their leavers studies for people who left welfare in different years, we selected the most recent study. When a state had no ASPE-funded study or any listed by the National Conference of State Legislatures or Brauner and Loprest, but did have a report available on its Web site, we selected the Web report. Waiver studies generally produced several reports. We selected for review the most recently issued waiver report because the data topics examined were similar in the initial and later reports.

After selecting these types of studies and surveys, we removed from our list studies that did not appear to contain data that could answer our research questions or that used data from one of the national surveys on our list. In summary, we excluded literature searches, reviews of research on state policies or programs, technical assistance projects focused on improving or evaluating information systems or databases, and studies based on data from a national survey that we had included in our list. A list of 239 studies remained. Finally, we obtained advice from five welfare experts about which of these 239 studies we should include. Ultimately, we selected 17 of these studies.

In all, we judgmentally selected 141 national surveys and studies that yielded 187 data sets to review. A complete list of the national surveys and studies that we examined for data is provided in appendix II.

Data Assessment

Identifying data resources for a comprehensive assessment of TANF required criteria that could be used to assess data sets. The first step in this process was to express each of TANF’s goals as a research question. In looking at the goals themselves, it is evident that some express expected results—for example, that work and marriage will improve the well-being of low-income families. Assessing TANF’s progress toward these expected results required, in part, questions about TANF’s effects. However, some of TANF’s goals focus on its general purpose—for example, providing assistance to needy families. In this case, assessing TANF’s progress required research questions that are descriptive, that is, questions that ask what public assistance looks like under TANF. To translate TANF’s goals into research questions, we considered the nature of each of TANF’s goals and formulated questions to represent key issues the Congress will consider at reauthorization. As shown in figure 1, we created corresponding questions that asked for descriptions of what has happened under TANF, the effects of TANF, or both.

We then specified the information, or data topics, necessary to address our research questions. We developed a data collection instrument that listed the data topics associated with each question and used the instrument to record the data topics found in each data set examined. It is important to note that what we identified as data topics were not equivalent to specific measures. In other words, our coding captured the fact that a certain data source collected measures on employment. It did not capture the specific manner in which employment was measured. In addition to data topics, we collected such pertinent information as the unit of analysis, population, sampling method, sample size, dates covered by data collection, and design of the study for which data were gathered. We recorded response rates and attrition rates when they were relevant given the method of data collection. We also looked to see if data had been or were being collected for a comparison or control group.

To summarize our findings, we identified data categories related to TANF’s goals, some of which represented the research questions and others of which were more narrowly focused. The narrowly focused data categories represented combinations of data topics, such as employment and earnings or family and child well-being, that were associated with the research questions. We took this approach for a variety of reasons. First, in making a judgment that data were available to address particular questions, we required that certain data topics be present in combination and, for effect questions, that the data were collected using control groups or comparison groups. However, a data source could provide relevant data topics, even though the data topic could not be used to address the particular question we had posed. Rather than discount the value of these data topics, we decided to note their availability. Second, in many cases, the same data topics and data categories were being used to address different questions. For example, as figure 1 shows, the data categories associated with employment were related to 5 of our 11 questions. Presenting our findings in terms of data categories allowed us to report on all of the data topics, including those that were not available in the combinations needed to address a research question.

Finally, to assess how the data might be used for an assessment of TANF, we considered three attributes of the data. We considered the geographic scope of the sample; the data topics included in the data set; and whether or not the data could be used for descriptive analyses or analyses of effect, given the design of the study. In determining the geographic scope of the sample, we looked at the sampling method and sample size, as well as at response rates and attrition rates, since both affect how well a sample represents a population. We relied on the design of the study, the data topics included in a data set, and how researchers had used the data to make a judgment about whether the data could be used for descriptive analyses or analyses of effect. We coded data as being useable for analyses of effect when they came from a study that made comparisons between groups, one of which served as the treatment group and the other as the absence of the treatment, or the comparison group. In deciding whether a study included a treatment and a comparison group, we recognized that such groups could be formed through experimental designs, quasi- experimental designs, or statistical modeling.

Study Limitations

Because this assessment is based on a judgmental sample and the data needs of an assessment of TANF’s progress are derived from TANF’s legislative objectives, several study limitations should be considered. First, while every attempt was made to be comprehensive in sample design and selection, some relevant data sources may have been omitted. Second, framing the data needs for an assessment of TANF’s progress around TANF’s objectives, which focus on the behavior and well-being of low- income children and families, excluded from consideration the bodies of welfare reform research concerned with institutions, including studies of TANF’s implementation at the state and local levels and descriptions of TANF program policies and practices. Third, the study’s focus on identification of quantitative data resulted in our eliminating data from most studies that used qualitative data collection methods. Fourth, because our bibliographic sources for surveys and studies included both existing and planned surveys and studies, complete documentation for data sets was not always available. Finally, because our coding focused on whether a certain data source collected measures on specific topics, but not on the precise measures used, we did not assess whether measures were comparable across studies.

Scope and Type of Analyses for Which Waiver and Demonstration Data Can Be Used

Jobs First: Implementation and Early Impacts of Connecticut’s Welfare Reform Initiative Administrative Survey

The Family Transition Program: Implementation and Three-Year Impact of Florida’s Initial Time-Limited Welfare Program Administrative Survey Iowa’s Family Investment Program: Impacts During the First 3-½ Years of Welfare Reform Administrative Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family Investment Program Administrative Surveys (2)

Comments From the Department of Health and Human Services

GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, the following individuals made important contributions to this report: Patrick DiBattista designed the data collection instrument used to assess the 187 data sets reviewed, oversaw data collection, and designed and conducted the analysis of the data's strengths and weaknesses; Andrea Sykes played a major role in data collection and developed the analysis of the data's availability to address TANF's goals; Stephen Langley III also played a major role in data collection, provided consultation on multivariate analysis issues, and prepared the report's methodology appendix; James Wright provided guidance on study design and measurement; and Gale Harris, Kathryn Larin, and Heather McCallum provided consultation on TANF policy and implementation issues.

Ordering Information

The first copy of each GAO report is free. Additional copies of reports are $2 each. A check or money order should be made out to the Superintendent of Documents. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders by phone: (202) 512-6000 fax: (202) 512-6061 TDD (202) 512-2537 Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.

To Report Fraud, Waste, or Abuse in Federal Programs

Web site: http://www.gao.gov/fraudnet/fraudnet.htm e-mail: fraudnet@gao.gov 1-800-424-5454 (automated answering system)