Welfare Reform: Data Available to Assess TANF's Progress
Published: 2001-02-28T00:00:00

Fast Facts:

Highlights:
section_title: What GAO Found
paragraphs: GAO commented on the federal government's ability to assess the goals of the Temporary Assistance for Needy Families (TANF) program using national, state, and local data. These data address the goals to differing degrees. National data, which includes data collected in national surveys and information that all states report to the Department of Health and Human Services (HHS), include extensive information on TANF's two goals of providing assistance to needy families and ending dependency on government benefits but have limited information on promoting family formation. The data pertain to such issues as changes in TANF workloads, recipients' participation in work activities, employment status and earnings, and family well-being. Although there are national data on the incidence of out-of-wedlock births and marriage among TANF recipients and other low-income families, these data include only very recently available information on states' strategies to prevent out-of-wedlock pregnancies or promote family formation. Studies of welfare reform at the state and local levels contain the same kind of information as national data, but they also include information about areas very recently covered by national data. Much of these data come from waiver evaluations--evaluations conducted in states that experimented with their welfare program, under a waiver from HHS, prior to TANF. The usefulness of existing data for assessing TANF's progress varies. In general, the need for information about TANF's progress will have to be balanced against the challenges of rigorous data collection from the low- income population.

section_title: Letter
paragraphs: Over 4 years have passed since federal welfare reform legislation was  enacted, replacing the legal entitlement to cash assistance under the Aid to  Families With Dependent Children (AFDC) program with the promotion of  work and personal responsibility. The policy and research community has  been asking what has happened to low-income families since the Personal  Responsibility and Work Opportunity Reconciliation Act of 1996  (PRWORA) created the Temporary Assistance for Needy Families (TANF)  block grant, replacing the federal entitlement to assistance and  institutionalizing reforms that had been initiated by many states. Overseen  by the Department of Health and Human Services (HHS) at the federal  level, TANF makes $16.8 billion in federal funds available to states each  year through 2002. While TANF delegates wide discretion to the states to  design and implement the program, it does specify four broad program  goals that focus on children and families:  providing assistance to needy families so that children may be cared for  in their own homes or in the homes of relatives;  ending the dependence of needy parents on government benefits by  promoting job preparation, work, and marriage;  preventing and reducing the incidence of out-of-wedlock pregnancies;   encouraging the formation and maintenance of two-parent families.
As TANF has been implemented by the states, the Congress has asked  questions about a broad range of issues concerning welfare reform’s  consequences. Several issues—for example, the circumstances of those  who have left TANF and the characteristics of those currently receiving  cash assistance—relate to what has happened to low-income families since  TANF has been in effect. The Congress also has asked about the extent to  which any changes that have occurred in low-income families’ life  circumstances, such as TANF recipients’ increases in employment and  earned income and decreases in births, can be attributed to TANF rather  than to other factors, such as changes in economic conditions. These  questions will be especially important when the program is considered for  reauthorization in 2001.
To determine what data will be available from planned and existing  research to address these and other questions, you asked us to examine (1)  the type of data that will be available from national surveys and studies of  welfare reform to help assess progress towards TANF’s goals and (2) the  usefulness of the data for assessing TANF’s progress nationwide. Our major  objective was to examine data already collected by national surveys of low- income families and for major studies of welfare reform to identify data  available for use in a comprehensive assessment of TANF for the  reauthorization debate. We did not examine study findings; rather, we  examined the data that have been collected or will be collected as part of  the considerable body of research about TANF that is accumulating. The  focus on data also addresses issues raised by several critiques of this work  that have emerged. The critiques that examine the methodology of this  research agree that, for welfare reform research, the quality and  comprehensiveness of the existing data are cause for concern. Two  critiques in particular question whether the major national survey designed  to provide information to assess TANF at the national level—the Survey of  Program Dynamics (SPD)—has “a sufficient number of observations” for  an analysis of TANF’s effects in individual states.
To determine what data will be available, we considered all of the national  surveys and welfare reform studies discussed in six major critiques of  welfare reform research, as well as surveys and studies available through  the Research Forum and the Welfare Information Network, the major  clearinghouses for welfare reform research. We also interviewed HHS  officials about their perspective on welfare reform data and asked welfare  experts to nominate key pieces of work in the field for inclusion in our  review. We selected a sample of surveys and studies that resulted in 187  individual data sets related to TANF reauthorization issues. For each data  set, we analyzed the content of the data to determine (1) the range of  TANF-related data topics covered; (2) the level of detail the data offer; (3)  the data’s quality, including, where relevant, the sampling method, sample  size, response rate, and attrition rate; and (4) how the data could be used to  provide the Congress with an understanding of what has happened under  TANF. Appendix I provides additional details about our methodology and  its limitations. We conducted our work between January 2000 and January  2001 in accordance with generally accepted government auditing  standards.
subsections: section_title: Background
paragraphs: PRWORA overhauled the nation’s welfare system by abolishing the  previous welfare program, AFDC, and creating the TANF block grant.  PRWORA established four broad goals for TANF, which included (1)  providing assistance to needy families so that children may be cared for in  their own homes or in the homes of relatives; (2) ending dependence of  needy parents on government benefits by promoting job preparation, work,  and marriage; (3) preventing and reducing the incidence of out-of-wedlock  pregnancies; and (4) encouraging the formation and maintenance of two- parent families. Unlike the previous program, TANF gives states great  flexibility to design programs that meet these goals. However, while states  have flexibility, the programs they design must meet several federal  requirements that emphasize the importance of work and the temporary  nature of TANF. For example, PRWORA requires that parents receiving  assistance engage in work, as defined by the state, after receiving  assistance for 24 months, or earlier, at state option. In exercising their  option, 28 states require immediate participation in work, and 9 other  states require participation in work within 6 months of receiving cash  assistance, resulting in great interstate variation in program provisions.  Further, despite the programmatic flexibility authorized by TANF, states  must meet federal data reporting requirements by submitting quarterly  reports that include information from administrative records about those  receiving welfare and those terminated from assistance, as well as an  annual report, to HHS. The annual report contains information about  program characteristics, such as states’ activities used to prevent out-of- wedlock pregnancy.
In 1995, we reported that the block grants enacted as part of the Omnibus  Budget Reconciliation Act of 1981 (OBRA) carried no uniform federal  information requirements. We found that the program information states  collected was designed to meet individual states’ needs and that, as a  result, it was difficult to aggregate states’ experiences and speak from a  national perspective on the block grant activities or their effects. Without  uniform information definitions and collection methodologies, it was  difficult for the Congress to compare state efforts or draw meaningful  conclusions about the relative effectiveness of different strategies. In a  second examination of federal block grant programs, we reported that  problems in information and reporting under many block grants—the  Education Block Grant, the Community Services Block Grant, and the  Alcohol, Drug Abuse, and Mental Health Services Block Grant—have  limited the Congress’ ability to evaluate them. However, for the TANF  Block Grant, the regulations require that states submit the quarterly TANF  Data Report and the TANF Financial Report or be subject to statutory  penalties. For these reports, HHS provides data reporting specifications  including timing, format, and definitions for such data topics as family  composition, employment status, and earned and unearned income. These  specifications facilitate the use of HHS’ TANF administrative data for  welfare reform research by improving the data’s comparability from state  to state. Several national surveys and data collected for state and local  studies of welfare reform also are potential sources of data for an  assessment of TANF.
subsections: section_title: National Data Sources on TANF
paragraphs: A number of national surveys that collect information about welfare receipt  have been used in the past by researchers to analyze welfare reform or  have been developed to assess current welfare reform. Four surveys—the  Survey of Income and Program Participation (SIPP), the Current  Population Survey (CPS), the National Longitudinal Survey of Youth  (NLSY), and the Panel Study of Income Dynamics (PSID)—have been used  in past research on the AFDC program and the low-income population in  general. Both the SIPP and the PSID have updated their questionnaires to  include questions that pertain to welfare reform specifically, including  questions about the work participation requirements and penalties for not  complying with these and other program rules. Moreover, two national  surveys are designed specifically to answer questions about welfare  reform. The U.S. Census Bureau, at the direction of the Congress, is  conducting a longitudinal survey of a nationally representative sample of  families, with emphasis on eligibility for and participation in welfare  programs, employment, earnings, the incidence of out-of-wedlock births,  and adult and child well-being. This survey, the Survey of Program  Dynamics, was designed to help researchers understand the impact of  welfare reform on the well-being of low-income families and children.  Similarly, the Urban Institute has been conducting a multiyear project  monitoring program changes and fiscal developments, along with changes  in the well-being of children and families. Part of this project includes a  nationally representative survey of 50,000 people called the National  Survey of America’s Families (NSAF) that is collecting information on the  well-being of adults and children as welfare reform is implemented.
With the change in the fundamental structure of the nation’s welfare  program, there have been several efforts by private research organizations  to document the policies states have adopted under TANF. The Center for  Law and Social Policy and the Center on Budget and Policy Priorities, in  collaboration, have created the State Policy Documentation Project to  document policies in all 50 states and the District of Columbia. Available on  the Web, the State Policy Documentation Project contains information  about state policies contained in statutes, regulations, and caseworker  manuals, but it does not describe state practices. In addition, the Urban  Institute has developed and made available to the public a database that  documents changes in state program rules since 1996.
section_title: State and Local Data From Welfare Reform Research
paragraphs: Prior to and since TANF’s implementation, a considerable body of research  about the low-income population has been conducted to examine the  circumstances of families affected by welfare reform, the effectiveness of  welfare reform initiatives, and the implementation of TANF at the state  level. HHS has played a major role in laying the foundation for this welfare  reform research. During the early 1990s, HHS granted waivers to states  that allowed them to test various welfare reform provisions. In return,  states were required to evaluate the effectiveness of the waiver provisions  by randomly assigning welfare recipients to either participate in the waiver  program or not. With the passage of TANF, states were given the option to  continue their waiver evaluations as originally designed or modify the  evaluation design. Several states opted to continue with their original  random assignment design, while others modified their evaluation designs  to focus on examining the implementation of the waivers or describe  participants’ employment, earnings, and well-being. Because some  elements of the waivers granted to states were incorporated into many  TANF programs, the waiver evaluations provide useful insights into issues  and designs for research about TANF.
However, according to HHS, one aspect of waiver policies may mean that  some waiver evaluations may not represent TANF requirements  completely. TANF established work requirements for all adult recipients,  but states could delay adhering to these requirements under their TANF  program, in part or whole, if the requirements were inconsistent with state  waiver policies. Under the Job Opportunities and Basic Skills Training  (JOBS) program, work requirements were mandatory for a work-ready or  able-bodied population, excluding a number of subgroups such as those  caring for young children and the disabled. For the most part, states that  continued the original random assignment design maintained some or all of  the JOBS exemptions from work requirements and applied these  exemptions in determining who was subject to time-limited assistance.  Consequently, while these states’ waivers may incorporate other work  policies prescribed under TANF, these policies would not be expected to  affect the exempt population. In contrast, in states that do not claim JOBS  exemptions from work requirements, all adults are subject to work  requirements and time limits on assistance. Thus, while testing TANF-like  policies, evaluations that continued the random assignment design may not  fully reflect the experience, outcomes, or impacts of fully implemented  TANF requirements.
In addition to the waiver evaluations, HHS, as well as private foundations,  has provided funding for demonstration programs across the country. The  demonstration programs are pilot projects designed to measure the effects  of a particular strategy, rather than an entire program, on welfare recipients  or those eligible to receive welfare. Many of these demonstration programs  were intended to increase employment, decrease out-of-wedlock  pregnancy, or promote marriage. For example, in the late 1980s, several  demonstration programs aimed at decreasing teen pregnancy among  welfare recipients were developed. One program, the New Chance  Demonstration, randomly assigned teen mothers receiving welfare to  participate in a program that offered education or training classes and  other support services and then compared the accomplishments of these  teen mothers with those of teen mothers who did not participate in the  program.
Given states’ greater responsibility for welfare programs under PRWORA  and the larger number of people leaving the welfare rolls, there has been  general interest among program administrators and state and local  policymakers about the condition of those who are no longer receiving  TANF, otherwise known as “leavers.” In response to this concern, a  growing body of research about leavers has been initiated at both the state  and federal levels. Generally, researchers have found that once low-income  families leave welfare, they become hard to keep track of. Moreover, we  previously reported that studies of former TANF recipients’ status differ in  important ways, including geographic scope, the time period covered, and  the categories of families studied, which limits the comparability of the  data across states. In order to facilitate cross-state study comparisons, the  Office of the Assistant Secretary for Planning and Evaluation (ASPE)  within HHS has issued guidance to states and the research community on  developing comparable measures for commonly reported outcomes and  defined these outcomes. In fiscal year 1998, ASPE awarded approximately  $2.9 million in grants to 10 states and three large counties to study leavers,  followed by additional grants in fiscal years 1999 and 2000. ASPE also has  encouraged the researchers to use comparable measures.
Research is also being conducted to examine the effects of welfare reform  in metropolitan areas or neighborhoods. This area of research is important  because the caseload decline in urban areas has been substantially lower  than in other areas of the country. Moreover, urban areas can have higher  unemployment rates and a greater concentration of poverty than suburban  or rural communities; thus, insights gathered from these studies will be  useful in understanding the potential for the success of welfare reform in  the event of an economic downturn. For example, one study—the Three  City Study—will survey primarily low-income, single-mother families from  poor and moderate-income areas in Boston, Chicago, and San Antonio,  with half of those surveyed being TANF recipients. The survey will collect  information on adult and family well-being, employment, and welfare  receipt three times within 4 years.
Finally, a body of welfare reform research examines the implementation of  TANF at the state and local levels. Since PRWORA has not only granted  states greater responsibility for providing cash assistance but also changed  the nature of cash assistance, it is important to learn how states and  localities are coping with these changes. Much of the research about  program implementation focuses on challenges faced by state, and in some  cases local, administrators in implementing TANF. Typically, in this  research qualitative data are collected by visiting state or local TANF  agencies; reviewing program records; and interviewing agency officials,  caseworkers, and clients. For example, the State Capacity Study conducted  by the State University of New York, Rockefeller Institute of Government,  is collecting data in 20 states about the implementation of TANF at the state  level, such as the structure of government services and information  systems used to track clients.
section_title: Framework for Assessing Data Availability
paragraphs: Because we expect much of the reauthorization debate to focus on TANF’s  four legislative goals, the framework for our data assessment was based on  those goals. To assess whether data exist to address the goals, we first  created a list of “descriptive” and “effect” research questions relevant to  each goal. Descriptive questions concern a low-income individual’s or  family’s status or behavior, such as the receipt of TANF cash assistance or  support services like transportation, housing, child care, or health services;  an adult’s employment status and earnings; and a family’s reliance on non- TANF government benefits, such as Food Stamps, Medicaid, or the Earned  Income Tax Credit. Effect questions concern the extent to which changes  in an individual’s or family’s status or behavior, such as obtaining  employment, earning income, avoiding out-of-wedlock births, or forming a  two-parent family, are the result of the TANF program. These research  questions represent the broad issues that the Congress will consider during  TANF’s reauthorization.
To summarize our findings, we identified data categories associated with  TANF’s goals, some of which are more narrowly focused than the research  questions. The data categories represent combinations of topics we found  in the data, such as employment and earnings or family and child well- being, that were associated with the research questions. Figure 1 shows the  relationships among TANF’s goals, the research questions, and the data  categories, several of which are associated with more than one question.  We then compared the data categories with the HHS administrative data,  the data collected by national surveys, and the data derived from existing  and planned studies.
Our assessment of the data’s usefulness for determining TANF’s progress is  based on the data’s strengths and weaknesses, the design of the survey or  study for which the data were gathered, and the topics to which the data  related. The criteria we used in assessing the strengths and weaknesses of  survey data included survey sample size, the attrition rate of respondents  from whom data were collected over time, and survey response rate. For  administrative data, we examined the geographic scope and the  comparability of the data among states. The design features examined  included what the data collection method was, whether the data were  collected at one point in time or at different points in time, and whether the  data were used for descriptive analysis of TANF or AFDC program  recipients and their families or analysis of the program’s effects. Data that  can be used for descriptive analysis are useful for research that addresses  questions in the descriptive column of figure 1, and data that can be used  for analyses of effect are useful for questions in the effect column of the  figure.
section_title: National, State, and Local Data Address Different TANF Goals
paragraphs: Together, national surveys, HHS administrative data, and data from state  and local studies of welfare reform address TANF’s four legislative goals.  The national data provide extensive information related to TANF’s goals of  providing assistance to needy families and ending dependency on  government benefits through job preparation, work, and marriage. State  and local data not only address the same goals as the national data but in  some cases also provide information related to the goals of preventing out- of-wedlock pregnancies and promoting family formation.
subsections: section_title: National Data Address Two TANF Goals but Provide Limited Information on Out- of-Wedlock Pregnancy Prevention and Family Formation Strategies
paragraphs: National data provide detailed descriptive information related to two of  TANF’s goals, but limited information related to TANF’s goals of preventing  out-of-wedlock pregnancies and promoting family formation. HHS  administrative data and the six national surveys we examined—the CPS,  NLSY, NSAF, PSID, SIPP, and SPD—provide descriptive information related  to TANF’s goal of providing assistance to needy families, including  information about the change in size and composition of the TANF  caseload and the use of noncash assistance by current and former TANF  recipients (see fig. 2).
National data also address TANF’s goal of ending dependence on  government benefits by describing the circumstances of those receiving  TANF and those who are no longer receiving TANF. HHS administrative  records and national surveys provide descriptive information about TANF  recipients’ participation in work activities, employment status, earnings,  and other family well-being measures. HHS administrative records contain  information only about whether a recipient is working and how much  income that individual earns, while national surveys collect more detailed  employment and earnings data, such as the types of jobs held and the  hourly wage. National data are also available about family well-being  measures, which provide information about how TANF’s focus on work  and marriage may be changing the lives of low-income families. For  instance, national surveys have information about the amount of personal  income spent on health and housing, whether recipients or former  recipients rent or own housing, and the well-being of children of welfare  recipients. Several of the national surveys provide information about  children’s school attendance or developmental status, while SIPP and SPD  also collect data about the number of births to teenagers. SIPP is the only  national survey we examined that contains information about whether  parents have had to terminate their parental rights or give a child up for  adoption.
National data related to the goals of preventing out-of-wedlock pregnancy  and promoting family formation are limited. While all the national data sets  include information about recipients’ and nonrecipients’ marital status,  only HHS administrative records contain information about out-of-wedlock  births among the TANF caseload. However, states did not begin reporting  this information to HHS until fiscal year 2000.
Aside from information about welfare reform in general, national surveys  and HHS collect information about several different groups of individuals  affected by TANF, including those who remain on assistance, those who no  longer receive TANF, those who are diverted from TANF, and those who  are eligible but choose not to participate. HHS administrative data and all  six national surveys collect data about current and former TANF recipients,  but the type of information collected about these individuals differs. As  figure 3 shows, only the NSAF and SIPP have data about those diverted  from TANF, while the NLSY, NSAF, PSID, SIPP, and SPD have data about  individuals who are eligible to receive TANF but do not.
section_title: State and Local Data Complement and Fill Gaps in the National Data
paragraphs: The state and local data we reviewed can be classified into four categories  that complement and, in some cases, fill in gaps not covered by the national  data. Waiver data come from evaluations that tested the effects of  programs implemented by states under waivers approved by HHS prior to  TANF. Demonstration data come from studies that tested the  effectiveness of particular strategies aimed at individuals either receiving  welfare or eligible to receive welfare. Leavers data come from  administrative records and surveys that describe the circumstances of  those who left welfare. Finally, metropolitan and community-based data  come from studies that, in general, describe the circumstances of low- income families and TANF participants in specific metropolitan areas,  neighborhoods, or communities.
Waiver data have been used to examine the effects of TANF-like provisions  on welfare recipients’ employment status, birth rates, and marital status, as  shown in figure 4. Several states have been evaluating the waiver  provisions in their welfare programs by randomly assigning welfare  recipients to either the waiver program or AFDC. Waiver programs require  participants to follow provisions that later were required or permitted  under TANF, such as being required to work or risk losing eligibility for  benefits or being allowed to receive welfare for only a limited time. Most of  the waiver program evaluations collected data used to analyze the effect of  waivers on welfare receipt, employment, and income. Data from several of  the evaluations have also been used to analyze the effects of waivers on  out-of-wedlock pregnancy or family formation.
With the passage of PRWORA, several states incorporated their waiver  provisions into their TANF program and have been collecting data about  the experiences of participants in the program. Some of these states chose  not to continue their evaluations as originally designed, instead conducting  modified evaluations that typically involved studies that will provide  information on the experience of implementing the program. For example,  Montana is surveying TANF participants to collect data about the duration  of their welfare receipt, the types of noncash assistance they use, and their  employment.
Demonstration data provide information on topics that are similar to those  addressed by waiver data and have also been used to analyze the effects of  programs on their participants, but demonstration data differ in two key  ways. First, most demonstration data, including all data related to  pregnancy prevention and family formation, were collected before  PRWORA was enacted. Second, demonstration data were collected for  studies focused on how a particular approach affected program  participants. In fact, many of the demonstration data we examined were  used entirely to assess the effects of various strategies on participants’  employment status and earnings, which helps to distinguish the effects of  particular provisions included in a program like TANF.
Leavers data provide descriptive information about those who have left  welfare. This information includes the length of time an individual received  TANF, reasons for leaving welfare, types of noncash assistance used, and  employment and earnings information. In addition, some leavers data sets  contain information about former recipients’ marital status, and a few have  data about the number of pregnancies and births among former recipients.
Metropolitan and community-based data cover some of the same issues as  the other data categories, including information about TANF work  requirements and time limits. Although the same issues are addressed, the  data are collected in large cities or neighborhoods in order to examine the  circumstances of welfare recipients in areas that may have high  concentrations of poverty or limited access to jobs. In addition,  metropolitan and community-based data provide information about groups  other than TANF recipients and former recipients—including individuals  diverted from TANF and those who are eligible to participate in TANF but  do not.
section_title: Strengths and Weaknesses of Existing Data Affect Usefulness for Assessing TANF’s Progress
paragraphs: Although existing data provide rich information about the lives of families  who are receiving or have received TANF, the strengths and weaknesses of  these data affect their usefulness for understanding welfare under the  TANF block grant. National data can be analyzed to gain a descriptive  picture of what has happened under TANF for the nation as a whole.  However, of the seven national data sets we reviewed, only two can be used  to describe the well-being of families receiving TANF within individual  states. Although waiver and demonstration data can be analyzed to gain  information about TANF’s effects, these analyses can be done within only a  limited number of states and disparate localities. We examined nearly 40  data sets that could be analyzed for information about the circumstances of  former recipients. However, only a subgroup of these data sets met criteria  that allowed the sample to be generalized statewide. These data sets  represented 15 states. In some cases, the value of survey data collected  from those who left welfare was limited because few former recipients  actually responded to the surveys: in some cases, former recipients could  not be located, and in other cases they chose not to answer the questions  posed to them. Metropolitan and community-based data can be analyzed to  describe changes over time in the lives of welfare recipients in urban  centers. Much of this data collection will continue beyond 2001.
subsections: section_title: National Survey Data Provide Descriptive Information About Nation, but Not Effects in Individual States
paragraphs: The strength of the national data is that they were collected from samples  selected randomly from the nation’s population and include low-income  families and TANF recipients in numbers sufficient to allow reliable  estimates about these groups. In addition, most of the national data were  collected for the same individuals over time, allowing changes in welfare  recipients’ employment, earnings, and well-being to be tracked across  programs implemented at different times. However, all the national surveys  have participants who drop out of the survey sample over time, and this  may limit how well the samples represent the nation’s welfare recipients.
National data are collected from random samples that contain low-income  families and TANF recipients. Because samples from national surveys are  selected randomly, they are, at the time of selection, representative of the  population at large, including the welfare population. In addition, all the  national data sets we reviewed have sample sizes large enough to allow  reliable estimates about the nation’s low-income and TANF populations— as sample size increases, the degree of precision of the estimates made  using that sample also increases (see table 1).
As shown in figure 5, two national data sources collect data on individuals  at one point in time; others collect data on the same individuals across  time. In both cases, the data can be used for comparisons between groups  of individuals living under welfare provisions implemented at different time  periods.
Five national surveys—the CPS, NLSY, PSID, SIPP, and SPD—collect data  from the same individuals over time. For the SIPP, the Census Bureau, after  a specified period, changes the group of individuals from whom data are  collected. For example, the 1993 SIPP panel followed a group of individuals  through 1996. In 1996, a new group was randomly selected and followed  through 2000. Data collected over time could be analyzed to describe how  people cycle on and off TANF, how their use of benefits changes over time,  and how their family well-being changes. In addition, comparisons could be  made between groups covered by different welfare provisions. For  example, AFDC recipients included in the 1993-96 SIPP panel could be  compared with TANF recipients who were part of the 1996-2000 SIPP  panel. The NSAF, as well as HHS administrative records, has collected data  from different samples of individuals in different years. For example, in  1997 one group of people completed the NSAF; another group completed  the survey in 1999. In cases such as these, the samples from different years  can be compared with each other to look for changes across time.
For those national surveys that collect information about changes in  welfare across time, the likelihood that survey participants will drop out  over time increases, potentially affecting how well the data actually  represent all members of the nation’s low-income and TANF populations.  In general, the greater the attrition rate, the less likely a sample is to be  representative of the larger population from which it was drawn. Those  who have continued participating in the survey may be different from those  who stopped or dropped out. As surveys that collect data over time, the  NLSY, PSID, SIPP, and SPD all have experienced sample loss, as shown in  table 2. Concerns about attrition are especially significant for the SPD,  because it was designed specifically to track welfare recipients from AFDC  through TANF. Census has tried mathematically adjusting available  responses to compensate for the survey’s sample loss, but this adjustment  has not sufficiently remedied the problem, according to a Census official.  Census will take steps to lessen attrition through intensive follow-up with  survey dropouts to enlist their participation and through the use of  monetary incentives for future respondents to the survey.
For national surveys, the response rate—the number of people in the  survey sample who actually responded, compared with those who were  asked to respond but did not—has been large enough to allow the survey  results to be generalized beyond those who completed the survey, with the  exception of the 1999 NSAF. Most practitioners of survey research,  including GAO, require at least a 70- to 75-percent response rate before  survey data can be generalized beyond those who completed the survey. As  table 3 shows, the response rate for all the national surveys except the 1999  NSAF was at or above the 70-percent standard. Given the survey’s  response rate, using the 1999 NSAF survey data would require determining  whether patterns in who responded and who did not respond existed and  what this means for how well respondents represent the original sample.  For those surveys that collect data on the same individuals over time,  response rates sometimes are considered in conjunction with rates of  attrition.
The major limitation of most existing national data is that they cannot be  used for state-level analyses. In general, national data sources have state  sample sizes that are too small to allow reliable generalizations about  TANF recipients within individual states. The NLSY, PSID, SIPP, and SPD  collect data not from states per se, but from regions that, in some cases,  include more than one state. Thus, while these data can be analyzed to  provide a descriptive picture of TANF for the nation, they cannot be used  within states for descriptive analyses or to analyze the effects of states’  TANF provisions. This does not mean that researchers do not use these  data sources for state-level analyses. For example, some researchers  combine several years of CPS data to obtain adequate sample sizes within  states for state-level analyses. However, Census, which administers the  CPS, SIPP, and SPD, does not recommend using data from these surveys  for state-level analyses, because doing so when sample sizes are small may  produce findings that are not reliable.
Two national data sources, HHS administrative records and the NSAF  survey, can be used for state-level analysis, but with limitations. HHS  administrative records provide data from all 50 states and the District of  Columbia. However, the reporting requirements for these data are not  completely standardized across states, so that how a variable is defined  may vary among states. For example, each state may define the work or  work-related activities in which TANF recipients participate as they think  appropriate to the state program.
Like HHS administrative records, NSAF survey data can be used for state- level analyses. NSAF has samples large enough to allow state-level analyses  in 13 states, representing 58 percent of the fiscal year 1999 national TANF  caseload; this is not the case in the 37 remaining states. For example, the  number of low-income children surveyed for the 1997 NSAF ranged from a  low of 760 to a high of 1,813 in each of the 13 states where NSAF collected  samples large enough to permit state-level analysis. However, the number  of low-income children surveyed in the 37 remaining states averaged 35 per  state, a number too small to allow reliable conclusions about the children  of TANF recipients in any of these states.
Even if the issue of sample sizes within states were resolved, obstacles to  using the national data to analyze TANF’s effects within states would still  exist. The lack of information about the choices states have made about  TANF policies and program rules has been identified as one of the  challenges to using national data to analyze TANF’s effects. However,  research organizations have collected this information. The Center for Law  and Social Policy has worked with the Center on Budget and Policy  Priorities to document policies in all 50 states and the District of Columbia,  and the Urban Institute has developed a state database that documents  state program rules. Yet, even with this information, using national data to  measure state-level effects poses challenges.
The first challenge is deciding with whom TANF recipients should be  compared. To test TANF’s effects, the employment, earnings, and well- being of individuals in the program must be compared with those of  individuals who are not in the program. In the case of TANF, it would be  difficult to determine what group should provide the point of comparison.  Because waivers introduced TANF-like policies and program rules while  AFDC was still in effect, it would be difficult to select a group of welfare  recipients whose experiences with welfare were not influenced by TANF.
The second challenge is determining the effect of any single welfare  provision given the multiple provisions that make up states’ TANF  programs. For example, TANF recipients are required to work, and states  must impose penalties or sanctions when recipients do not comply with  work requirements. In such cases, it would be difficult to separate the  combined effects of work requirements and any penalties or sanctions that  were imposed into the individual effects of each.
A third challenge is detecting the long-term effects of state programs that  have been recently implemented. Although PRWORA was enacted in 1996,  states implemented their TANF programs at different points in time. Some  states were still refining their TANF programs at the beginning of 1998.  Consequently, the long-term effects of TANF may not yet be realized.
Finally, state-level analyses may not be the best way to measure TANF’s  effects in every state. Some states have further devolved TANF to localities,  and different localities may implement a state’s TANF provisions  differently. In total, 17 states have given local governments responsibility  for TANF program design and implementation.
section_title: Waiver and Demonstration Data Allow Assessment of TANF Effects, but Only for Localities
paragraphs: The strength of the waiver and demonstration data is that they can be used  to analyze TANF’s effects, but with few exceptions these data were  collected from city and county samples rather than statewide samples. (See  app. II for the localities examined.) Most of the waiver and demonstration  data were collected as part of experiments—studies that randomly  assigned welfare recipients to groups that were subject to different welfare  provisions. Experiments, when done correctly, are recognized as the most  rigorous way of determining the extent to which an observed outcome can  be attributed to the program itself, rather than to differences among the  program participants. Over half of the waiver data sets and virtually all of  the demonstration data sets we reviewed consisted of data from  experiments. Of the waiver data sets, about half were collected from city  and county samples, with the others being collected from statewide  samples. All of the demonstration data sets were collected from city and  county samples. Overall, 6 of the 54 waiver and demonstration data sets  that could be used for analyses of effect were collected from statewide  samples.
According to the project directors of two waiver evaluations, the high cost  of conducting rigorous program evaluations may explain, in part, why data  sets used to analyze TANF’s effects tend to use samples from cities and  counties and not entire states. Given limited resources, researchers may  choose to conduct rigorous evaluations in selected cities or counties rather  than sacrifice rigor to evaluate a program statewide. Data sources we  reviewed for both the Vermont and Iowa waiver evaluations mentioned  budget constraints as a factor that led researchers to limit their data  collection efforts.
Another limitation of the waiver and demonstration data is that most often  they were collected prior to the implementation of TANF. This is not  surprising given that in many cases the waiver provisions and the  demonstration projects were intended to test provisions before they were  adopted and implemented. However, the provisions tested may not have  been those ultimately adopted by the state.
Finally, in almost all cases in which waiver evaluations and demonstration  projects collected survey data, response rates were above the 70-percent  standard (see table 4).
section_title: State-Level Analyses of Leavers Data Are Possible in 15 States; in Some Cases, Picture Is Incomplete
paragraphs: The strength of the leavers data is that in most cases, they were collected  from statewide samples. However, in some cases, leavers data collected  using surveys may not be representative of a state’s leaver population.  Although we reviewed nearly 40 leavers data sets, on the basis of the type  of data available, response rates, and the absence of significant differences  between survey respondents and nonrespondents, we concluded that state- level analyses could be done for 15 states using the data sets we examined.
To be representative of a state’s leavers population, survey data need to  meet the 70-percent standard for response rates, or, through a comparison  of survey respondents with nonrespondents, show that the two groups do  not differ significantly. When a state has both administrative data and  survey data available, the administrative data could be used in place of  survey data that are not representative. As figure 6 shows, Arkansas,  Florida, Georgia, North Carolina, and South Carolina have either survey  data that meet the standard for response rates or data from survey  respondents who were not significantly different from nonrespondents.  Arizona, Colorado, the District of Columbia, Illinois, Kansas, Missouri,  Virginia, Washington, and Wisconsin have both administrative data and  survey data. The response rate for the District of Columbia, Illinois, Kansas,  Virginia, and Wisconsin was below 70 percent, but for Virginia, a  comparison of respondents with nonrespondents revealed no significant  differences between the two groups. Although New York has no survey  data, its administrative data provide information about the state’s leavers.  California, Massachusetts, and Texas are the three states for which, given  the available data, state-level analyses of leavers cannot be done. We  previously reported that eight leavers studies covering seven states had  collected adequate information to allow the study findings to be  generalized to the states’ welfare populations. Thus 4 states—Indiana,  Maryland, Oklahoma, and Tennessee—can be added to the list of 15 states  we identify in figure 6 as having data that can be generalized statewide. In  appendix II we list all the sources we reviewed that provide data on those  who have left welfare.
Some researchers may wish to compare those who left TANF with those  who left AFDC on outcomes such as employment, earnings, and well-being.  Contrasting outcomes for these two groups would require deciding which  AFDC leavers provide the best point of comparison. Many factors specific  to the year in which recipients left the welfare rolls would influence their  employment prospects, wages, and well-being. For example, labor markets  and economic conditions in a given year would influence former recipients’  employment opportunities. Historical influences such as these would  complicate the issue of selecting a comparable group of AFDC leavers and  TANF leavers.
section_title: Metropolitan and Community-Based Data Cover Urban Centers; Data Collection Will Continue Beyond 2001
paragraphs: The strength of the metropolitan and community-based data is that they  can be used in descriptive analyses that provide information about how the  lives of low-income families and TANF participants have changed over  time. Because data collection is occurring over time, in some cases it has  yet to be completed. For example, the Los Angeles Family and  Neighborhood Study (LA FANS) is collecting data about participation in  welfare programs from residents of 65 neighborhoods in Los Angeles  County over a 4-year period. LA FANS began data collection in January  2000 and will continue data collection through 2004. Most of the materials  we reviewed regarding metropolitan and community-based data sets did  not report information about attrition rates. When response rates were  reported, they were above the 70-percent standard. Figure 7 shows the time  periods for which the data are or will be available for different  metropolitan areas and communities.
Three of the metropolitan and community-based data sources have  measures that can be used to analyze TANF’s effects, even though the data  were not collected as part of an experiment. For example, data from the  Fragile Families study can be used to examine TANF’s effects by drawing  comparisons between the 3,675 unmarried parents and the 1,125 married  parents who compose the survey sample in cities with populations over  200,000. Data collection for Fragile Families began in 1998 and will  continue through 2004. The data have already been used to examine  differences in relationship quality between married and unmarried couples,  including whether a father gave money to or helped a mother in a  nonmonetary way during pregnancy.
section_title: Existing Data Are More Adequate for State and Local Descriptive Analyses Than for Analyses of Effect
paragraphs: The current body of research on TANF addresses many issues of interest to  the Congress but does not provide a comprehensive national picture of  TANF. However, existing national data and data from state and local studies  could be pieced together to develop a descriptive picture of what has  happened to TANF participants in all 50 states. In addition, within a limited  number of states and various cities and counties, existing data can be used  to conduct analyses of TANF’s effects.
National survey data can be used with data from HHS administrative  records for descriptive analyses of TANF’s progress nationwide. HHS  administrative data can be used for analyses within each of the 50 states,  and national survey data can be analyzed for national trends. These  analyses could be compared to examine the extent to which the  employment experiences, for example, of current and former TANF  recipients in individual states conform with or depart from the experiences  of such individuals identified with national survey data. This comparison  could be extended to the individual states and localities covered by the  NSAF data, waiver and demonstration data, leavers data, and metropolitan  and community-based evaluation data.
While piecing the data together in this way would build on their strengths,  each data type still has limitations. Specifically, national survey data  provide national samples useful for comparing the lives of welfare  recipients covered by welfare provisions implemented at different times.
However, attrition or low response rates may affect the degree to which  these samples represent all members of the nation’s low-income and TANF  population. Within each of the 50 states, HHS administrative data can be  analyzed to gain insight into current recipients’ use of noncash benefits,  among other things, but the lack of standardized reporting requirements  would complicate comparisons across states. Supplemental descriptive  analyses for individual states can be done using NSAF survey data, leavers  data, waiver and demonstration data, and metropolitan and community- based data. In addition, like the national survey data, many of these data  represent multiple measures over time. However, these analyses in many  cases can be generalized only to cities and counties and not to entire states.
Existing data can also be analyzed to gain information about TANF’s  effects. Although the 1997 and 1999 NSAF survey samples do not include  pre-TANF welfare recipients, the samples do include other populations,  such as low-income families who do not participate in TANF, whose  employment, earnings, and well-being can be compared with those of  TANF recipients, assuming adequate sample sizes for both groups.  Moreover, because NSAF has sample sizes in 13 states large enough to  allow state-level analyses, the employment, earnings, and well-being of  TANF recipients in those states can be considered in relation to the state’s  TANF programs and policies. However, using the NSAF data for such  analyses would require resolving the challenges to analyzing effects  described earlier in this report. Similarly, although most of the  metropolitan and community-based evaluation samples do not include pre- TANF welfare recipients, other populations represented in the study  samples could be compared with TANF recipients. Finally, waiver and  demonstration data can be analyzed to gain information about TANF’s  effects, keeping in mind that this information is about the effects of  programs and provisions often implemented prior to TANF and  implemented in cities and counties rather than entire states.
section_title: Concluding Observations
paragraphs: The data available for addressing TANF’s goals will provide useful  information, but with some limitations. Given the costs, some limitations  may be difficult to overcome. Our examination of the data raised three  issues.
First, for a comprehensive assessment of TANF, it is important to have data  for a representative sample of TANF recipients and nonrecipients that  allow for analyses of effect at the state level. The federal government has  made an investment in national surveys, which either in whole or in part  are intended to gather information about the lives of TANF recipients. One  of these, the SPD, was funded as a means to gather data about TANF  recipients. For another, the SIPP, the Census Bureau added a special  section of questions about welfare and reworded questions so that they  would better capture respondents’ participation in state programs.  However, even with these efforts, none of Census’ surveys currently being  administered can be used for state-level analyses of TANF’s effects because  of small sample sizes within individual states. In addition, the SPD has a  high attrition rate. The Census Bureau plans to take steps to improve  response to the SPD through intensive follow-up with survey dropouts to  enlist their participation and through monetary incentives for future  respondents to participate in the survey. However, the issue of small  sample sizes at the state level will remain unresolved.
Second, HHS has encouraged state agencies to study the effects of their  TANF programs through the AFDC waiver requirement for experimental  studies and subsequent research initiatives. Moreover, our examination of  data indicates that, because of the variability in TANF program provisions  across states, analysis of TANF’s effects at the state and local levels can be  done with the greatest confidence. However, even when conducted at the  state and local levels, studies designed to examine TANF’s effects tend to  be costly, time-consuming, and impractical to implement in every state. In  some cases, conducting an evaluation for an entire state is determined to  be so expensive that data collection is limited to a portion of the state. For  example, the evaluation of Vermont’s waiver program focused on 6 of 12  welfare service districts. The evaluation’s 42-month follow-up survey was  administered to only these 6 district offices and, owing to cost constraints,  included a subset of the sample for whom administrative records, rather  than survey responses, were collected. Policymakers, federal and state  officials, and the welfare reform research community will need to seek  ways to balance the need for information about TANF’s effects with the  resource demands of rigorous studies.
Third, both qualitative and quantitative data may be needed to understand  what has happened to former TANF recipients. Leavers are a difficult  population to track, and, in some cases, using multiple methods of  quantitative data collection has not necessarily increased the number of  former recipients who could be located or who responded to surveys. In  fact, in some of the studies we reviewed, the low rate of success in  gathering data from these individuals makes the data’s usefulness  questionable. Surveys that used only one mode of data collection, such as  telephoning former TANF recipients, generally had the lowest response  rates. Some leavers’ studies followed telephone surveys with personal  interviews of those who could not be reached by phone or who did not  respond. However, even the use of multiple modes of data collection did  not always ensure high response rates. Given the difficulties inherent in  collecting quantitative data from this group, other data collection strategies  that use local communication networks to identify families as well as  interviews of respondents in their homes may be needed to gain  information about the lives of TANF leavers.
section_title: Agency Comments
paragraphs: In commenting on a draft of this report, HHS said that the report will be of  help to the Congress and other interested parties. In its technical  comments, HHS expressed concern that in highlighting the importance of  statewide samples, we understated the value of data from local samples. In  response to this concern, we have noted in the report not only that findings  from local samples are important but also that, in some cases, they provide  data only recently available from national surveys. We concur with HHS  that a sample need not be statewide in order for findings to be useful.  However, we have emphasized the value of data that can be generalized to  the state level because of the Congress’ interest in a picture of TANF’s  progress nationwide. HHS’ comments appear in appendix IV.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies to the Honorable Tommy G.  Thompson, Secretary of Health and Human Services; appropriate  congressional committees; and other interested parties. We also will make  copies available to others on request.
If you or your staff have any questions about this report, please contact me  on (202) 512-7202 or David D. Bellis on (415) 904-2272. Another GAO  contact and staff acknowledgments are listed in appendix V.
section_title: Scope and Methodology
paragraphs: This appendix discusses in more detail our scope and methodology for  identifying, selecting, and assessing studies and surveys that might provide  data to help researchers as they seek to describe what has happened to  recipients of Temporary Assistance for Needy Families (TANF) and to  estimate the effect of welfare reform on them.
subsections: section_title: Sample Design
paragraphs: Because no comprehensive list of data sources for welfare reform research  exists, we used a judgmental sampling method for our assessment of data  resources. We began our work by examining six key critiques of welfare  reform research that had been issued, in draft or final form, by the fall of  1999. The six critiques listed in figure 8 both gave us insight into issues that  will probably arise in assessing TANF and identified studies that may be  potential sources of data for an assessment of TANF.
We started the development of a list of data sources from three of the  critiques—the Research Forum’s report and its related on-line database,  the National Research Council’s interim report, and Peter Rossi’s paper. To  ensure that this list was comprehensive, we consulted with officials at the  Department of Health and Human Services (HHS) about important bodies  of work in the welfare reform research field. We also conducted follow-up  interviews with HHS project officers and experts in the welfare reform  research community to ensure that we had identified the most relevant  national surveys and studies, particularly those that might have data about  family, marriage, and pregnancy issues. As a result of these discussions and  an examination of the original list, we designed a judgmental sample of  potential data sources for welfare reform research that included the  following categories:  national surveys and HHS’ TANF administrative data;  studies that collected data about the major TANF subpopulations in  three or more states or municipalities;  studies of TANF leavers;  HHS’ waiver evaluations; and  studies listed on the websites of HHS’ Administration for Children and  Families (ACF), HHS’ Office of the Assistant Secretary for Planning and  Evaluation (ASPE), and the Welfare Information Network of the Finance  Project.
We then began to develop lists of the surveys and studies in each of the  sample’s categories.
The national surveys included in our list were the Current Population  Survey (CPS), the National Longitudinal Survey of Youth (NLSY), the  National Survey of America’s Families (NSAF), the Panel Study of Income  Dynamics (PSID), the Survey of Income and Program Participation (SIPP),  and the Survey of Program Dynamics (SPD). We used information from  ASPE and from the National Conference of State Legislatures to identify  leavers studies sponsored by HHS or states. Similarly, we used information  from ACF to ensure that our list contained the body of research funded by  ACF that focused on waivers implemented by state welfare agencies prior  to TANF’s authorization. As we added items to the list, we continually  checked to avoid any duplication. This comparison involved our judgment,  as some lists were of projects or studies and others were of study reports.  Because we relied on multiple reviews of the body of work undertaken in  the welfare reform research community, we believe that the list of 443  entries we compiled included the key sources of data.
section_title: Sample Selection
paragraphs: We selected surveys and studies systematically from this list within each  sample category. We were interested in surveys or studies that were as  comprehensive as possible in geographic coverage and topics addressed.  Thus, we selected all of the national surveys and the HHS administrative  data. We also selected all studies on the original list that by their  description appeared to have produced data concerning the major  subpopulations affected by TANF in three or more states, municipalities, or  counties. This resulted in 55 studies and surveys.
We then selected studies that pertained to individual states in the following  way. First we selected all leavers studies financed by ASPE. Of the leavers  studies listed by the National Conference of State Legislatures and those  mentioned in an article authored by Brauner and Loprest, we included  only those that had not been included in our previous report or were not  from a state that already had an ASPE-funded study. In states that had  issued multiple reports for their leavers studies for people who left welfare  in different years, we selected the most recent study. When a state had no  ASPE-funded study or any listed by the National Conference of State  Legislatures or Brauner and Loprest, but did have a report available on its  Web site, we selected the Web report. Waiver studies generally produced  several reports. We selected for review the most recently issued waiver  report because the data topics examined were similar in the initial and later  reports.
After selecting these types of studies and surveys, we removed from our list  studies that did not appear to contain data that could answer our research  questions or that used data from one of the national surveys on our list. In  summary, we excluded literature searches, reviews of research on state  policies or programs, technical assistance projects focused on improving  or evaluating information systems or databases, and studies based on data  from a national survey that we had included in our list. A list of 239 studies  remained. Finally, we obtained advice from five welfare experts about  which of these 239 studies we should include. Ultimately, we selected 17 of  these studies.
In all, we judgmentally selected 141 national surveys and studies that  yielded 187 data sets to review. A complete list of the national surveys and  studies that we examined for data is provided in appendix II.
section_title: Data Assessment
paragraphs: Identifying data resources for a comprehensive assessment of TANF  required criteria that could be used to assess data sets. The first step in this  process was to express each of TANF’s goals as a research question. In  looking at the goals themselves, it is evident that some express expected  results—for example, that work and marriage will improve the well-being  of low-income families. Assessing TANF’s progress toward these expected  results required, in part, questions about TANF’s effects. However, some of  TANF’s goals focus on its general purpose—for example, providing  assistance to needy families. In this case, assessing TANF’s progress  required research questions that are descriptive, that is, questions that ask  what public assistance looks like under TANF. To translate TANF’s goals  into research questions, we considered the nature of each of TANF’s goals  and formulated questions to represent key issues the Congress will  consider at reauthorization. As shown in figure 1, we created  corresponding questions that asked for descriptions of what has happened  under TANF, the effects of TANF, or both.
We then specified the information, or data topics, necessary to address our  research questions. We developed a data collection instrument that listed  the data topics associated with each question and used the instrument to  record the data topics found in each data set examined. It is important to  note that what we identified as data topics were not equivalent to specific  measures. In other words, our coding captured the fact that a certain data  source collected measures on employment. It did not capture the specific  manner in which employment was measured. In addition to data topics, we  collected such pertinent information as the unit of analysis, population,  sampling method, sample size, dates covered by data collection, and design  of the study for which data were gathered. We recorded response rates and  attrition rates when they were relevant given the method of data collection.  We also looked to see if data had been or were being collected for a  comparison or control group.
To summarize our findings, we identified data categories related to TANF’s  goals, some of which represented the research questions and others of  which were more narrowly focused. The narrowly focused data categories  represented combinations of data topics, such as employment and earnings  or family and child well-being, that were associated with the research  questions. We took this approach for a variety of reasons. First, in making a  judgment that data were available to address particular questions, we  required that certain data topics be present in combination and, for effect  questions, that the data were collected using control groups or comparison  groups. However, a data source could provide relevant data topics, even  though the data topic could not be used to address the particular question  we had posed. Rather than discount the value of these data topics, we  decided to note their availability. Second, in many cases, the same data  topics and data categories were being used to address different questions.  For example, as figure 1 shows, the data categories associated with  employment were related to 5 of our 11 questions. Presenting our findings  in terms of data categories allowed us to report on all of the data topics,  including those that were not available in the combinations needed to  address a research question.
Finally, to assess how the data might be used for an assessment of TANF,  we considered three attributes of the data. We considered the geographic  scope of the sample; the data topics included in the data set; and whether  or not the data could be used for descriptive analyses or analyses of effect,  given the design of the study. In determining the geographic scope of the  sample, we looked at the sampling method and sample size, as well as at  response rates and attrition rates, since both affect how well a sample  represents a population. We relied on the design of the study, the data  topics included in a data set, and how researchers had used the data to  make a judgment about whether the data could be used for descriptive  analyses or analyses of effect. We coded data as being useable for analyses  of effect when they came from a study that made comparisons between  groups, one of which served as the treatment group and the other as the  absence of the treatment, or the comparison group. In deciding whether a  study included a treatment and a comparison group, we recognized that  such groups could be formed through experimental designs, quasi- experimental designs, or statistical modeling.
section_title: Study Limitations
paragraphs: Because this assessment is based on a judgmental sample and the data  needs of an assessment of TANF’s progress are derived from TANF’s  legislative objectives, several study limitations should be considered. First,  while every attempt was made to be comprehensive in sample design and  selection, some relevant data sources may have been omitted. Second,  framing the data needs for an assessment of TANF’s progress around  TANF’s objectives, which focus on the behavior and well-being of low- income children and families, excluded from consideration the bodies of  welfare reform research concerned with institutions, including studies of  TANF’s implementation at the state and local levels and descriptions of  TANF program policies and practices. Third, the study’s focus on  identification of quantitative data resulted in our eliminating data from  most studies that used qualitative data collection methods. Fourth, because  our bibliographic sources for surveys and studies included both existing  and planned surveys and studies, complete documentation for data sets  was not always available. Finally, because our coding focused on whether a  certain data source collected measures on specific topics, but not on the  precise measures used, we did not assess whether measures were  comparable across studies.
section_title: Scope and Type of Analyses for Which Waiver and Demonstration Data Can Be Used
subsections: section_title: Jobs First: Implementation and Early Impacts of Connecticut’s Welfare Reform Initiative Administrative Survey
paragraphs: The Family Transition Program: Implementation and Three-Year Impact of Florida’s  Initial Time-Limited Welfare Program   Administrative  Survey Iowa’s Family Investment Program: Impacts During the First 3-½ Years of Welfare  Reform   Administrative Reforming Welfare and Rewarding Work: Final Report on the Minnesota Family  Investment Program   Administrative  Surveys (2)
section_title: Comments From the Department of Health and Human Services
section_title: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, the following individuals made  important contributions to this report: Patrick DiBattista designed the data  collection instrument used to assess the 187 data sets reviewed, oversaw  data collection, and designed and conducted the analysis of the data's  strengths and weaknesses; Andrea Sykes played a major role in data  collection and developed the analysis of the data's availability to address  TANF's goals; Stephen Langley III also played a major role in data  collection, provided consultation on multivariate analysis issues, and  prepared the report's methodology appendix; James Wright provided  guidance on study design and measurement; and Gale Harris, Kathryn  Larin, and Heather McCallum provided consultation on TANF policy and  implementation issues.
section_title: Ordering Information
paragraphs: The first copy of each GAO report is free. Additional copies of  reports are $2 each. A check or money order should be made out to  the Superintendent of Documents. VISA and MasterCard credit  cards are accepted, also.
Orders for 100 or more copies to be mailed to a single address are  discounted 25 percent.
Orders by mail: U.S. General Accounting Office P.O. Box 37050 Washington, DC  20013 Orders by visiting: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders by phone: (202) 512-6000 fax: (202) 512-6061 TDD (202) 512-2537 Each day, GAO issues a list of newly available reports and  testimony. To receive facsimile copies of the daily list or any list  from the past 30 days, please call (202) 512-6000 using a touchtone  phone. A recorded menu will provide information on how to obtain  these lists.
section_title: To Report Fraud, Waste, or Abuse in Federal Programs
paragraphs: Web site: http://www.gao.gov/fraudnet/fraudnet.htm  e-mail: fraudnet@gao.gov  1-800-424-5454 (automated answering system)