Child Welfare: Most States Are Developing Statewide Information Systems, but the Reliability of Child Welfare Data Could Be Improved
Published: 2003-07-31T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: To better monitor children and families served by state child welfare agencies, Congress authorized matching funds for the development of statewide automated child welfare information systems (SACWIS) and required that the Department of Health and Human Services (HHS) compile information on the children served by state agencies. This report reviews (1) states' experiences in developing child welfare information systems and HHS's role in assisting in their development, (2) factors that affect the reliability of data that states collect and report on children served by their child welfare agencies and HHS's role in ensuring the reliability of those data, and (3) practices that child welfare agencies use to overcome challenges associated with SACWIS development and data reliability.
section_title: What GAO Found
paragraphs: HHS reported that 47 states are developing or operating a SACWIS, but many continue to face challenges developing their systems. Most state officials said they recognize the benefit their state will achieve by developing SACWIS, such as contributing to the timeliness of child abuse and neglect investigations; however, despite the availability of federal funds since 1994, states reported a median delay of 2 and a half years beyond the timeframes they set for completion. States reported that they encountered some difficulties during SACWIS development, such as challenges receiving state funding and creating a system that reflected their work processes. In response to some of these challenges, HHS has provided technical assistance to help states develop their systems and conducted on-site reviews of SACWIS to verify that the systems meet federal requirements. Despite efforts to implement comprehensive information systems, several factors affect the states' ability to collect and report reliable adoption, foster care, and child abuse and neglect data. States responding to GAO's survey and officials in the 5 states GAO visited reported that insufficient caseworker training and inaccurate and incomplete data entry affect the quality of the data reported to HHS. In addition, states reported technical challenges reporting data. Despite HHS's assistance, many states report ongoing challenges, such as the lack of clear and documented guidance on how to report child welfare data. In addition, although states were mandated to begin reporting data to the Adoption and Foster Care Analysis and Reporting System (AFCARS) in 1995, few reviews of states' AFCARS reporting capabilities have been conducted to assist states in resolving some of their reporting challenges. Some states are using a variety of practices to address the challenges associated with developing SACWIS and improving data reliability. For example, 44 states included caseworkers and other system users in the design and testing of SACWIS, and 28 states reported using approaches to help caseworkers identify and better understand the data elements that are required for federal reporting.

section_title: Letter
paragraphs: Recent news reports of tragedies involving child welfare agencies across  the country highlight the long-standing problems states have had  monitoring children in their care. Given that hundreds of thousands of  children are found to be victims of abuse and neglect and are estimated to  spend some time in foster care each year, the Congress required that the  Department of Health and Human Services (HHS) compile information on  the children served by state agencies and authorized federal funds to  match those of states for use in the development of state child welfare  information systems. Since 1994, designated federal matching funds have  been available to states to develop and implement comprehensive case  management systems—Statewide Automated Child Welfare Information  Systems (SACWIS)—to manage their child welfare cases as well as to  report child abuse and neglect, foster care, and adoption information to  the federal government. States have the option to implement a SACWIS or  develop different information systems without using SACWIS funds to  support their child welfare agencies and collect information on their child  welfare cases. Regardless of the type of system a state develops, child  welfare caseworkers at the county or local level are the key personnel  who collect and document information on children and families served by  child welfare agencies, in addition to performing a wide range of services  to protect children—such as investigating child abuse or neglect reports or  providing support services to maintain the children in their homes.  Currently, HHS compiles state-reported child welfare data in two  databases: the Adoption and Foster Care Analysis and Reporting System  (AFCARS) and the National Child Abuse and Neglect Data System  (NCANDS). HHS relies on the information available in its databases to  analyze and track children’s experiences in the child welfare system, to  determine states’ performance on federal child welfare outcome measures,  and to report to Congress on children’s well being and child welfare  experiences.
Because of your interest in how states have developed systems to collect  and report information on the children they serve and the reliability of that  information, you asked us to determine (1) states’ experiences in  developing child welfare information systems and HHS’s role in assisting  in their development; (2) factors that affect the reliability of data that  states collect and report on children served by their child welfare  agencies, and HHS’s role in ensuring the reliability of those data; and   (3) practices that child welfare agencies use to overcome challenges  associated with SACWIS development and data reliability.
To conduct our work, we surveyed all 50 states and the District of  Columbia regarding their experiences in developing and using information  systems and their ability to report data to HHS. We received responses  from 49 states and the District of Columbia, although all states did not  respond to every question. We also reviewed a variety of HHS documents,  including the protocol and reports for its reviews of SACWIS systems and  states’ AFCARS reporting capabilities. In addition, we visited 5 states— Colorado, Iowa, New York, North Carolina, and Oklahoma—to obtain first- hand information on their experiences developing SACWIS and reporting  data to HHS. We selected these states to represent geographic diversity  and different stages of SACWIS implementation. Finally, we interviewed  HHS officials and child welfare and data experts and reviewed relevant  literature. We conducted our work between June 2002 and June 2003 in  accordance with generally accepted government auditing standards. A  more detailed discussion of our scope and methodology appears in  appendix I.
subsections: section_title: Background
paragraphs: ACF’s Children’s Bureau is responsible for the administration and  oversight of federal funding to states for child welfare services under  Titles IV-B and IV-E of the Social Security Act. However, the monitoring of  children served by state child welfare agencies is the responsibility of the  state agencies that provide the services to these children and their  families. Child welfare caseworkers at the county or local level are the key  personnel responsible for documenting the wide range of services offered  to children and families, such as investigations of abuse and neglect;  treatment services offered to families to keep them intact and prevent the  need for foster care; and arrangements made for permanent or adoptive  placements when children must be removed from their homes.  Caseworkers are supported by supervisors who typically assign new cases  to workers and monitor caseworkers’ progress in achieving desired  outcomes, analyzing and addressing problems, and making decisions  about cases.
A number of efforts at the national level have been taken to implement  comprehensive data systems that capture, report, and analyze the child  welfare information collected by the states (see table 1 for information on  national data systems as well as information on state systems).
To qualify for federal funding for SACWIS, states must prepare and submit  an advance planning document (APD) to ACF’s Children’s Bureau, in  which they describe the state’s plan for managing the design, development,  implementation, and operation of a SACWIS that meets federal  requirements and state needs in an efficient, comprehensive, and cost- effective manner. In addition, the state must establish SACWIS and  program performance goals in terms of projected costs and benefits in the  APD. States are required to submit separate APDs for the planning and  development phases, in addition to periodic updates.
Since the administration and structure of state child welfare agencies vary  across the nation, states can design their SACWIS to meet their state  needs, as long as states meet certain federal requirements. Federal funding  is available to states for SACWIS that    meet the requirements for reporting AFCARS data to HHS;  to the extent practicable, are capable of linking with the state data  collection system that collects information on child abuse and neglect;  to the extent practicable, are capable of linking with, and retrieving  information from, the state data collection system that collects  information on the eligibility of individuals under Title IV-A—Temporary  Assistance for Needy Families; and    provides for more efficient, economical, and effective administration of  the programs carried out under a state’s plans approved under Titles IV-B  and IV-E of the Social Security Act.
A SACWIS must operate uniformly as a single system in each state and  must encompass all entities that administer programs provided under  Titles IV-B and IV-E. In some cases, HHS will allow the statewide system  to link to another state system to perform required functions, such as  linking to financial systems to issue and reconcile payments to child  welfare service providers. The state’s APD must describe how its SACWIS  will link to other systems to meet the requirements in the SACWIS  regulations.
In addition to monitoring the APDs of the states that are developing  SACWIS, HHS reviews state information systems through formal SACWIS  assessment reviews and the Child and Family Services Reviews (CFSR)—a  federal review process to monitor states’ compliance with child welfare  laws and federal outcome measures. The formal SACWIS reviews are  conducted by ACF’s Children’s Bureau to determine if a state has  developed and implemented all components detailed in the state’s APD  and if the system adheres to federal requirements. The CFSR assesses  statewide information systems, along with other systemic factors, to  determine if the state is operating a system that can readily identify the  status, demographic characteristics, location, and goals for placement of  every child who is in foster care. This systemic factor is reviewed in all  states, regardless of whether the state is developing a SACWIS or the stage  of system development. According to results from the fiscal years 2001 and  2002 CFSRs, 4 of the 32 states in which HHS reviewed were not in  substantial conformity on the statewide information system indicator.  These 4 states must address how they will come into conformity with this  factor in a program improvement plan. HHS has also conducted SACWIS  reviews in 2 of these states.
section_title: Most States Are Developing SACWIS, but Challenges Remain Despite HHS’s Oversight and Technical Assistance
paragraphs: While 47 states are developing or operating a SACWIS, many challenges  remain despite HHS’s oversight and technical assistance. Since 1994,  states reported that they have spent approximately $2.4 billion in federal,  state, and local funding on SACWIS. While most state officials we  interviewed and those responding to our survey said that they recognize  the benefits their state will achieve by developing a statewide system,  many states reported that the development of their SACWIS is delayed  between 2 months and 8 years beyond the time frames the states set for  completion, with a median delay of 2-½ years. Most states responding to  our survey faced challenges, such as obtaining state funding and  developing a system that met the child welfare agency’s needs statewide.  In response to some of these challenges, HHS has provided technical  assistance to help states develop their systems and conducted on-site  SACWIS reviews to verify that the systems meet all federal requirements.
subsections: section_title: States Are Using Federal and State Funds and Various Participants to Develop MultiComponent SACWIS
paragraphs: Currently, 47 states are developing or operating a SACWIS and are in  various stages of development—ranging from planning to complete. The  states responding to our survey reported using approximately $1.3 billion  in federal funds and approximately $1.1 billion in state and local funds for  their SACWIS. However, HHS estimated that it allocated approximately  $821 million between fiscal years 1994 and 2001 in SACWIS developmental  funds and $173 million between fiscal years 1999 and 2001 in SACWIS  operational funds. The total amount of federal funding provided to states  for SACWIS is unknown because states claimed operational costs as a part  of their Title IV-E administrative expenses prior to 1999. Although the  federal government matched state funding at an enhanced rate of   75 percent beginning in 1994, many states did not apply for federal funding  or begin SACWIS development until 1996 and 1997 when more than   $467 million—the bulk of federal funds—were allocated. Most states were  still developing their SACWIS by the time enhanced funding expired in  1997, after which states could receive a 50 percent FFP for SACWIS  development and operation. Although 47 states are currently developing or  operating a SACWIS, all states except Hawaii received some federal  SACWIS funds. For example, according to figures provided by HHS, North  Carolina received approximately $9.6 million in developmental funds and  North Dakota received approximately $2.4 million in developmental funds  and $245,000 in operational funds for SACWIS, but both states  encountered difficulties that prevented them from completing their  systems. In these situations, HHS entered into negotiations with the states  about the amount of money that the states must return to the federal  government.
In order to track states’ SACWIS development, HHS places them in six  categories that identify their stage of development (see table 2). States are  required to submit APD updates periodically, which inform HHS of their  progress in developing SACWIS. See appendix II for a complete list of  states’ phases of development. Although most states continue to advance  in the development of their systems, some encounter problems that cause  HHS to recategorize them into a lower stage of development. In  Pennsylvania, for example, the child welfare agency encountered  difficulties, such as inadequate computer software to support a  comprehensive SACWIS, after attempting to implement its SACWIS in  2000. Due to these problems, the state is in the process of shutting down  the system and has resubmitted an APD for a new system to HHS for  review and approval for further federal funding. According to figures  provided by HHS, Pennsylvania has received approximately $9.7 million in  federal funding thus far. In addition, while HHS may classify a state system  as complete following an assessment of their SACWIS, a state may make  additional changes to the system since SACWIS, like other computer  systems, continually evolve as technology and child welfare practices  change. States can claim federal funding for these changes as operational  expenses. For example, Oklahoma’s SACWIS was the first system to be  determined complete, but it has made enhancements to its system since  HHS found the system in compliance with federal requirements in 1998. In  addition, Oklahoma is currently considering moving to a Web-based  system. An HHS official reported that such changes do not need prior  approval unless they are in excess of $5 million.
In developing a system, states have considerable flexibility in the design of  their SACWIS. According to HHS officials, a state should be using its  SACWIS as a case management tool that uses automation to support the  various aspects of state child welfare programs, such as recording child  protection, out of home care, and foster care and adoption services. To  further assist child welfare practice, states have designed their systems to  follow the natural flow of child welfare practice in their state and have  added design features to help track key events during a case. For example,  in Iowa child welfare work is divided between child abuse and neglect  investigations and ongoing case management for children brought into the  care of the child welfare agency. As a result, Iowa designed a SACWIS to  reflect this work process by linking two databases—one to record child  abuse and neglect information and one to record ongoing case records— that share information with one another. In Rhode Island, the SACWIS  was designed to alert caseworkers if an alleged perpetrator has been the  subject of three reports of abuse or neglect within 1 year. Regardless of  the findings of each report, this alert notifies the caseworker to initiate an  investigation when a third report is received.
Since many states are in different phases of SACWIS development, their  systems currently support to varying degrees a variety of child welfare and  administrative components (see table 3). According to HHS, while the  components represented in table 3 are required for a state’s SACWIS to be  considered compliant with federal guidance—either through an interface  or built within the system—some of the subcomponents, such as a  function that helps caseworkers manage their caseloads, are optional.  HHS has encouraged states to automate as many functions as possible in  the SACWIS in an effort to cut down on the additional paperwork or  duplicative steps inherent in manual data collection. One of these services,  tracking independent living, is becoming more important for states as HHS  decides how to implement the Foster Care Independence Act of 1999 and  considers the development of the NYTD. Some states have already started  collecting data on older youth and the services they receive. Currently,   27 states reported in our survey that they are at some stage of using their  SACWIS to track independent living services, and an additional 14 states  plan to include this component in their system in preparation for the  requirements. However, 21 of the 46 states reporting to our survey that  they are developing or operating a SACWIS reported that they would have  to make substantial changes to their SACWIS in order to capture this  information.
To assist with the design of their SACWIS, states relied on a number of  different participants including internal users, such as caseworkers and  managers, information technology (IT) staff, and contractors. Most states  found these participants to be extremely or very helpful in the process  (see table 4). In Oklahoma, for example, 150 child welfare staff from the  field worked closely with the contractor in intensive work group sessions  to design and test the system. To complement the caseworkers’ knowledge  of child welfare practice, 43 states relied on IT staff. In Colorado, for  example, IT staff said that during SACWIS design and development, they  shared office space with program staff that had been assigned to help with  SACWIS development. This co-location of staff aided in the exchange of  information pertaining to the development of the system. Finally, 42 states  reported that they hired private contractors to conduct a large part of  SACWIS design and development. The contractors helped states meet  federal requirements, designed the system with state specific options,  wrote the necessary software, tested and implemented the system, and  trained users.
At the time of our review, HHS reported that 4 states were not pursuing  SACWIS development and most of these states reported various reasons in  our survey for not developing a system. In Hawaii, the child welfare  agency chose not to pursue SACWIS because it already had a statewide  system in place that it believed was adequately meeting its needs and  which was collecting and reporting federal child welfare data. After an  attempt to develop a system, North Carolina cancelled its efforts because  it could not build consensus across its 100 counties on the design of a  uniform system. On our site visit to North Carolina, child welfare officials  reported that they are currently working on a statewide information  system that will encompass a number of social services, such as food  stamps and mental health services, but an HHS official reported that North  Carolina is not seeking federal SACWIS funding to support the  development of this system. Vermont officials reported that they did not  pursue SACWIS because the legislature declined to provide the matching  state funds. In retrospect, they believe that the choice not to develop  SACWIS was best for the state because they found the SACWIS  requirements too restrictive to enable the state to design a system to meet  its needs. Officials said that the state would not use a number of the  required SACWIS components, such as developing all the required  electronic links to other agencies’ systems, especially since the state has a  small child welfare population. Another state—North Dakota—did not  report in our survey the reason for stopping SACWIS development;  however, HHS officials reported that the state had attempted to develop a  SACWIS, but faced a variety of problems, such as receiving state funding.
section_title: States Accrue Benefits from Using SACWIS, but Several Issues Create Delays in Completing States’ Systems
paragraphs: While most state child welfare agency officials said they recognize the  benefits the state will achieve by developing SACWIS, such as enhancing  their ability to track the whereabouts of foster children, 31 state agencies  lag behind the time frames they set for completion, with 26 states  reporting delays ranging from 2 months to 8 years. State officials reported  in our survey and during site visits that SACWIS has contributed to more  efficient and effective agency functioning, which can improve states’  capabilities to manage their child welfare cases, including keeping track of  where the children are living and the services they are receiving. Child  welfare officials in Colorado reported that automation has improved  agency functioning by making child welfare case information available  statewide, which is especially helpful when families move from one county  to another. In Oklahoma, caseworkers and state officials noted that they  believe their children are safer since the implementation of SACWIS  simply because the information on the children is easily accessible to the  caseworkers and their supervisors. According to survey results, automated  systems provided easier access to data and allowed caseworkers to better  monitor children in their care, which may contribute to additional child  welfare and administrative benefits, such as decreased incidences of child  abuse and neglect, shortened length of time to achieve adoption,  timeliness of payments to foster families, and timeliness of payments to  foster facilities (see table 5). New Jersey reported in our survey that its  goal in developing a SACWIS is to integrate the more than 40 stand-alone  systems that currently capture information on the children served by their  child welfare agency. By pulling all of these systems together into a  uniform SACWIS, the state hopes to improve the recording of casework  activities in a timely manner and to develop a tool to better target  resources and services. Effectively integrating these systems will require  the state to use a disciplined IT management approach that includes   (1) detailed analyses of users’ needs and requirements, (2) a clearly  defined strategy for addressing information needs, and (3) sufficient  technical expertise and resources to support the effort.
Despite the benefits that many states have accrued with SACWIS, 31 states  reported in our survey that they have been delayed in system completion  beyond their initial deadline and identified a number of challenges that  have led to the delay (see table 6). Some of the common difficulties states  reported in developing SACWIS included receiving state funding approval,  reaching internal agreement on system development, and creating a  system that reflects child welfare work processes and is user friendly (see  table 7).
Forty-two states reported challenges receiving funding approval, and   32 states reported that insufficient state funding allocations for SACWIS  development were a challenge in developing a comprehensive system. For  example, Vermont officials reported that the state legislature declined to  provide the matching state funds needed to secure federal funding for  SACWIS. As a result, the state could not pursue development. In addition  to the development challenges reported in our survey, 2 of the 5 states we  visited reported that insufficient funding affected ongoing SACWIS  maintenance. In Colorado, state agencies have received a series of budget  cuts, which child welfare officials report have impacted their ability to  respond to child welfare caseworkers’ needs for system improvements. In  Iowa, child welfare officials reported that the state legislature  appropriated $17,000 for state fiscal year 2002 for all child welfare  automated systems activity, which they noted was an insufficient amount  to maintain and upgrade systems as well as to pay staff. They reported  that, as a result, the child welfare agency provided the information  systems department with approximately $1 million from other parts of the  agency’s budget.
Despite user involvement in system design, some states still faced  challenges trying to reach internal agreement among agency officials and  caseworkers on the design of a system, resulting in a delay in  development. In New York—a state where the counties are responsible for  administering child welfare services—the development of SACWIS was  stalled when significant frustration with the system’s design led  Commissioners from five large counties and New York City to request that  the state stop SACWIS development until a reassessment of the design and  plans for the implementation of the system was completed. After a  thorough evaluation of the project, the state made changes to the project  plan and developed statewide work groups to ensure all counties were in  agreement with the system design. In addition, they hired a contractor to  monitor system development and ensure that all users’ requirements are  seriously considered.
Similarly, despite states’ heavy reliance on contractors, many reported that  securing contractors with knowledge of child welfare practice was a  challenge for timely SACWIS development. Contractors are hired by the  state for their system development knowledge but often are unfamiliar  with child welfare policies and practices, especially since they vary from  state to state. Officials in Colorado, for example, said they encountered  difficulties with their contractors because of high turnover among the  contractor staff and their lack of knowledge of child welfare policies. A  contractor who has worked with 7 states to develop their SACWIS  reported that contractors are asked to learn the child welfare business  practices of a state in a short amount of time and that states cannot devote  many resources, such as caseworkers, to help in the design process  because caseworkers need to devote their time to providing services to  children and families. Therefore, contractors often have to acquire  knowledge on their own.
Many states reported that creating a system that reflects child welfare  work processes and is user friendly was a challenge in developing  SACWIS. These issues were also identified in the federal reviews of states’  SACWIS. For example, one state explained in the SACWIS review that it  had designed a system to meet the caseworkers’ needs and reflect the  nature of the child welfare work processes by developing a system that  required events to be documented as they occurred. However, this design  limited the SACWIS’s functionality because it did not allow the  caseworkers to go back and enter information after an event happened.  The state explained that caseworkers do not use the system in real time,  but provide services to the children and families and then record the  information in the system. The state had to redesign the system to correct  for this design flaw. In addition, the 14 states reporting that they have  adapted a system from another state have experienced some challenge  modifying the systems to reflect their work processes. While HHS advises  states to consider adapting another state’s system if it requires few  changes, states report that they and their contractors were not always  aware of the changes that would need to be made to adapt a system.  Although Colorado and New York decided to modify another state’s  SACWIS instead of designing a new system, child welfare officials in these  states reported that adapting a SACWIS from another state has created  more problems than anticipated. Colorado and New York adapted systems  from state-administered child welfare agencies, which required extensive  modifications to meet their needs as county-administered states. For  example, Colorado needed a system that supported its administrative  structure and could handle a larger number of cases.
section_title: HHS Provides Some Assistance to Help States Meet SACWIS Requirements
paragraphs: HHS has assisted states in a variety of ways in developing and completing  their SACWIS. As a part of its regulatory responsibilities, HHS must  review, assess, and inspect the planning, design, development, installation,  and operation of SACWIS. In addition to reviewing and monitoring states’  APDs, HHS conducts on-site SACWIS reviews to comply with these  responsibilities. HHS officials told us that these reviews are a detailed and  thorough assessment of state systems to ensure the systems’ compliance  with SACWIS requirements. In addition, officials reported that they  provide verbal technical assistance during the on-site review to help states  that do not fully conform with the applicable regulations and policies. At  the time of our review, HHS had conducted 26 SACWIS reviews—5 of  which were determined as meeting all the requirements and classified as  complete. HHS officials told us that since states have the flexibility to  build a SACWIS that meets their needs, a large portion of the formal  reviews concentrate on ensuring that the systems conform to state  business practices. For example, while SACWIS regulations require that a  state report all AFCARS data from their SACWIS, one state HHS reviewed  relied on a separate state system to report data on the children served by  the juvenile justice agency who are eligible for IV-E foster care funds. The  state proved it had developed an automated process to merge data from  both systems to compile a single AFCARS report that included children  captured in both their SACWIS and juvenile justice systems. Therefore,  HHS recognized that this process best met the state’s needs and  determined the SACWIS to be complete and meeting all requirements.
Few systems have been determined complete after an on-site review  because of unresolved issues, such as not being able to build links to other  state information systems or not implementing certain eligibility  determination functions. To help states address some of these  development challenges, the SACWIS review team provides the state with  recommendations for complying with SACWIS requirements. For example,  HHS observed during a review in one state that the SACWIS was available  statewide, but information collected in one county was not available to  caseworkers in other counties. The federal officials offered  recommendations to the state to meet the SACWIS requirement that all  information be available statewide. In addition, HHS officials reported that  once the draft report with the results of the SACWIS review are  completed, federal staff schedule a conference call with the state officials  to walk through the system’s deficiencies and offer guidance on how the  state can move forward.
HHS facilitates the sharing of information between states developing  SACWIS through an automated system users group that allows state and  federal officials to exchange information, ideas, and concerns. According  to some state and HHS officials, the trust level at these meetings is very  high, which promotes open discussions and also creates an atmosphere  for informal dialogue with HHS. The systems users group developed out of  another active group—the child welfare users group—when HHS solicited  state representatives to help HHS define a model child welfare information  system, which was later used as the basis for the SACWIS functional  requirements after the passage of the 1993 legislation authorizing  enhanced federal funding. State officials in Iowa and New York reported  that the systems users group continues to play an important role in  providing a forum for the honest exchange of information on SACWIS  development. For example, child welfare and technical officials in New  York said that the systems users group has been very beneficial because  they have learned from other states’ positive and negative experiences in  developing SACWIS, as well as the experiences unique to states with  county-administered agencies. In addition to the users group, HHS officials  also sponsor a listserv—an electronic mailing list—that allows state  officials to exchange information, and a monthly conference call with state  information technology directors. Iowa child welfare information  technology officials said that they find the monthly SACWIS telephone  conference call helpful because project managers discuss issues such as  promising practices and new regulations.
Technical assistance for SACWIS development is also available to states  through the National Resource Center for Information Technology in Child  Welfare (Resource Center). According to survey results, 9 states said they  used the Resource Center for assistance in developing SACWIS and   14 states reported using it for help with SACWIS maintenance and  improvements. According to Resource Center officials, they assist states  with SACWIS development by helping states understand the technology  that is available for use, providing information on the automation of child  welfare work and converting data, and reviewing the APD documentation.  For example, the Resource Center offered technical assistance to  Pennsylvania to help the state decide if it should continue development of  its current SACWIS, abandon the SACWIS project and allow the counties  to operate individual systems, or design a different SACWIS. The Resource  Center evaluated the current SACWIS to determine if it could capture  information based on the SACWIS regulations and if it was user friendly  for the caseworker. Following the Resource Center’s analysis,  Pennsylvania decided to discontinue the existing SACWIS and develop a  new SACWIS. When the Resource Center opened in 1999—5 years after  many states started developing SACWIS—staff were not very familiar with  many of the efforts states made during development. In an attempt to  remedy this lack of knowledge on states’ issues developing SACWIS,  Resource Center staff participated in some of the on-site SACWIS reviews  conducted by HHS. Both HHS and Resource Center officials believe this  exposure to the SACWIS systems enhanced the availability of technical  assistance resources and knowledge available to the states.
section_title: Several Factors Affect the States’ Ability to Ensure Reliable Data on Children’s Experiences, and Some of HHS’s Oversight and Assistance Is Problematic
paragraphs: Several factors affect states’ ability to collect and report reliable data on  children served by state child welfare agencies, and some problems exist,  such as a lack of clear and documented guidance, with HHS’s oversight  and technical assistance. Almost all of the states responding to our survey  reported that insufficient caseworker training and inaccurate and  incomplete data entry affect the quality of the data reported to HHS. In  addition, 36 of the 50 states that responded to our survey reported that  technical challenges, such as matching their state data element definitions  to HHS’s data categories, affected the quality of the data that they report to  the federal government. For example, North Carolina officials told us that  while state policy mandates that they count every location in which a child  resides, including hospital stays, AFCARS regulations say that hospital  stays and other short-term placements should not be included in the count  of foster care placements. In cases where state policy differs from federal  policy, state officials must carefully re-format their data in order to meet  federal reporting requirements. Similarly, during assessments of 6 states’  compliance with AFCARS reporting standards, HHS found that these  issues affect data reliability. Despite the assistance that HHS offers to  states, such as testing state data quality and providing the results to states  to aid them in resubmitting data, states report ongoing challenges  receiving clear and documented guidance and accessing technical  assistance.
subsections: section_title: Insufficient Caseworker Training and Inaccurate and Incomplete Data Entry Are the Most Common Factors That Affect Data Reliability
paragraphs: Almost every state responding to our survey and all the states we visited  reported that insufficient training for caseworkers and inaccurate and  incomplete data entry affect the quality of the data reported to AFCARS  and NCANDS (see fig. 1). Although most states reported these as separate  factors, HHS and the states we visited found that insufficient training and  inaccurate and incomplete data entry are often linked. For example, in  official reviews of states’ information systems capability to capture data  and report them to AFCARS, HHS advised states to offer additional  training to caseworkers on several AFCARS data elements, such as  recording the reasons for a child leaving foster care, to improve the  accuracy of the data submitted. Similarly, Oklahoma reported that the  state found that caseworkers were misinterpreting reports of policy  violations by foster parents and inaccurately recording them as abuse or  neglect allegations. However, state officials told us that training is typically  one of the first programs cut when states face tight budget restrictions.  For example, Iowa officials told us that training has been significantly  reduced in recent years because of budget cuts and new workers may wait  2 to 3 months before being trained how to enter data appropriately into  their SACWIS.
Inaccurate and incomplete data entry can also result from a number of  other factors, such as caseworkers’ hesitation to ask families for sensitive  information. For example, caseworkers in Oklahoma reported that they  did not feel comfortable asking if a child’s mother was married at the time  of birth or if a child is of Hispanic origin—both of which are required  AFCARS data elements. In commenting on a draft of this report, Oklahoma  added that caseworkers did not understand why the data elements were  required and how the federal government used the information. In  addition, Iowa state officials said that caseworkers may guess the racial  backgrounds of children in their care or record them as unknown,  especially when children come from mixed racial backgrounds, rather  than asking the family for the information. HHS noted similar issues in   5 states that have had an AFCARS review. Caseworkers were  inaccurately recording a child’s race as “unable to determine” even though  this option should be selected only if the child’s parents or relatives  cannot provide the information, such as when a child is abandoned.
Caseworkers, supervisors, and managers in the 5 states we visited  reported that additional factors, such as difficulties balancing data entry  with the time that they spend with the families and children, contributed  to inaccurate or incomplete data entry. In addition, our recent work on  caseworker recruitment and retention found that caseworkers struggle to  balance the time they spend with children and data entry, and reportedly  spend at least 50 percent of their time documenting case records.  Supervisors in Iowa explained that since caseworkers are responsible for  ensuring that children and their families receive the services they need, the  caseworkers tend to initially limit data entry to the information that is  necessary to ensure timely payment to foster care providers, and complete  all other data elements when the caseworkers have time. In addition,  caseworkers in Colorado said that they are between 30 and 60 days behind  in their data entry, so the information in the automated system may not  accurately reflect the current circumstances of children in care. The  caseworkers reported that they tend to concentrate only on entering data  that will allow them to open a case in their SACWIS. HHS’s Inspector  General recently issued a report in which more than two-thirds of the  states reported that caseworkers’ workloads, turnover, a lack of training,  and untimely and incomplete data entry affected the reporting of AFCARS  data.
section_title: Technical Challenges, such as Matching State Definitions to Federal Definitions, Affect Data Reliability
paragraphs: In addition to data quality being affected by caseworker issues, many  states experienced technical challenges reporting their data to HHS. The  problems reported by states are typically a result of challenges associated  with data “mapping”—matching state data elements to the federal data  elements. For example, 36 states reported in our survey that matching  their state-defined data to HHS’s definitions affected the quality of the data  reported to NCANDS and AFCARS. Similarly, 24 states reported that  matching the more detailed data options available in their states’  information systems to the federal data elements affected the quality of  the data reported to NCANDS. Twenty-nine states reported that this issue  created challenges in reporting data to AFCARS. For example, following  an AFCARS assessment, HHS instructed a state that collects detailed  information on children’s disabilities, such as Downs Syndrome, Attention  Deficit Disorder, and eating disorders, to map the information to the more  limited options in AFCARS, such as mental retardation and emotionally  disturbed. The Inspector General’s report found that states faced similar  challenges mapping their data to meet the AFCARS reporting  requirements.
In many cases, states have to balance state policy with federal  requirements to ensure that they are reporting accurate data to AFCARS  and NCANDS, but are not contradicting their state policies. For example,  Texas officials reported that although the findings of their AFCARS review  instructed them to modify their SACWIS to collect, map, and extract data  on guardianship placements, the state does not support guardianship  arrangements. In addition, a recent report from the Child Welfare League  of America (CWLA) found that when reporting the number of times  children move from one foster care placement to another, states varied in  the type of placements included in that count. For example, 29 percent of  the states responding to CWLA’s survey included respite, 25 percent  included runaways, and 16 percent included trial home visits when  reporting the number of placements a child had during the AFCARS report  period. According to federal guidance, the “number of placements”  element is meant to gather information on the number of times the child  welfare agency found it necessary to move a child while in foster care and  that by including runaways or trial home visits, a state is inflating the  number of moves a child experienced. However, North Carolina officials  told us that although the federal definition for placements instructs states  not to include such stays when counting the number of children’s foster  care placements, the state instructs them to count each time a child is  sleeping in a different place as a new placement. The Inspector General  reported that the placement definitions were the most commonly cited  source of confusion among the states surveyed.
In addition to the challenges reported in our survey, HHS reported that  transferring data from older data systems into SACWIS affects the quality  of the data reported to AFCARS and NCANDS. HHS officials reported that  they have observed that states experience the biggest change in data  quality when they begin reporting from their SACWIS. In general, the first  data submissions are of low quality because of the time it takes states to  transfer data or the system re-sets the information for data elements. For  example, in 1 state, 65 percent of the records reviewed by HHS during an  AFCARS assessment recorded the date the children were removed from  their homes as July 28, 1997—the date the SACWIS came on-line; however,  the actual dates of removal for these children ranged from 1988 to 1997.
section_title: Although HHS Has Taken Steps to Help States Improve Their Data, Some Problems with Its Efforts Exist
paragraphs: HHS provides technical assistance for AFCARS and NCANDS reporting  through a number of resources. HHS officials in the central office and  NCANDS contractor staff serve as the points of contact for states to ask  questions and seek guidance on reporting child welfare data. HHS officials  reported that assistance is offered in a number of ways, including  telephone and e-mail communication. The officials in 3 of the 5 states that  we visited said that the one-on-one focused technical assistance was  useful when provided in a timely fashion. Most state officials found the  NCANDS data easier to report, in part because more people were available  for consultation and they were more accessible and responsive. For  example, states have access to four NCANDS specialists and staff in the  contractor’s central office when they need assistance reporting child  abuse and neglect information. However, some of the states we visited  reported that only one or two staff in HHS’s central office are available to  assist with AFCARS reporting.
In addition, the Resource Center offers states assistance with improving  data quality; however, Resource Center staff reported that the assistance is  geared more towards improving the limited data used in the federal review  process to monitor states’ compliance with child welfare laws and federal  outcome measures—CFSR—rather than all the data reported to HHS. The  Resource Center also sponsors an annual information technology  conference during which sessions covering all data-related issues are held,  including practices for ensuring data quality and outcome evaluation in  child welfare.
In conjunction with the national data conference, the HHS officials and the  contractors that operate NCANDS hold an annual technical assistance  meeting for states to share ideas with one another, discuss data elements  that pose difficulties, and explore ways to address these problems. For  example, at a recent technical assistance meeting, approximately 43 state  representatives attended sessions on preparing the calendar year 2002  NCANDS data submissions and received a detailed explanation of how the  NCANDS staff test states’ data submissions for quality. In addition, an  NCANDS state advisory group meets annually to talk with HHS officials  about NCANDS data and their experiences reporting data. From these  meetings, the state advisory group proposes changes or improvements to  NCANDS. HHS and state officials reported that this partnership has helped  ease some of the challenges in reporting child abuse and neglect data.
In addition to the direct assistance through consultation with HHS officials  and the Resource Center, HHS has made available to states the software it  uses to examine states’ AFCARS and NCANDS submissions for  inconsistencies and invalid data. Officials in all the states we visited said  that they regularly use this software, and an HHS official said that nearly  every state has used the software at least once. When the data are  submitted to HHS, they are run through the same software, and HHS  notifies the states of areas where data are missing or inconsistent and  allows the states to resubmit the data after errors are corrected. For  example, HHS officials said that they worked with one state that was  trying to determine the source of data errors in reporting to AFCARS the  race or ethnicity of children in their care. The state was not able to  determine the source of the problem, so an HHS official examined the  state’s submissions and helped correct the data errors. The officials  reported that these tests help them to identify some data quality errors,  such as missing data, and said that they believe that, in general, data have  improved in recent years.  However, they indicated that the tests cannot  pinpoint the underlying problems contributing to these errors.   Furthermore, one official reported that no specific efforts have been  conducted to track the individual data elements and, therefore, HHS  cannot report on how data quality has changed over time.  The results of  these quality tests had been the basis for penalties levied against states  that submitted low quality AFCARS data before the penalties were  rescinded. HHS officials reported that the penalties served as an effective  motivation to states to correct their data. Although HHS was not able to  report how the lack of penalties might be affecting recent data quality, an  official reported that the agency plans to conduct this analysis in the  future.
In an attempt to help states comply with the reporting standards and  address some of the factors that contribute to data quality problems, HHS  performs comprehensive reviews of state information systems’ ability to  capture AFCARS data to identify problems associated with data collection  and reporting, and to ensure that the information in the automated system  correctly reflects children’s experiences in care. The assessments include  a technical review of the states’ computer code, a comparison of the data  from selected cases available in the information system to the case files,  and an improvement plan to resolve any errors. In addition, HHS officials  offer guidance to the states on improvements that can be made to the  information system and changes to program code used to report the  AFCARS data. HHS conducted pilot reviews in eight states between 1996  and 2000. By March 2003, HHS had conducted eight official reviews—even  though states began reporting to AFCARS in 1995. According to results  from six of the eight official AFCARS assessments we reviewed, no state  met the reporting requirements for all AFCARS data elements. Table 8  shows a selection of the data elements and the states’ ratings. The  problems noted in the reviews are similar to those we heard from states  responding to our survey and those we visited. For example, most states  received ratings of 2 or 3, indicating technical and/or data entry errors that  affect the AFCARS data quality.
State officials in these 6 states reported that they found the reviews useful  for improving their AFCARS data submissions. In particular, they valued  the thorough review by HHS officials of the computer code states use to  report the data. Some of these officials reported that if all states were  reviewed, the quality of data available in AFCARS would improve  tremendously. However, HHS officials reported that they are not  mandated to conduct the AFCARS reviews and that priority is placed on  other reviews, such as the CFSR and SACWIS reviews. In addition,  officials explained that the AFCARS reviews are not conducted in states  developing SACWIS until the systems are operational. HHS expects to  complete approximately four reviews each year depending on available  resources and has scheduled states through 2006. Similar to the SACWIS  reviews, HHS officials offer recommendations and technical assistance to  states during the review on how they can improve the quality of the data  reported to AFCARS.
Although the states we visited appreciated some of HHS’s efforts to assist  with improving state data quality, they and most states responding to our  survey agreed that the assistance is not always consistent or easily  accessible (see fig. 2). States reported similar information to the Inspector  General—AFCARS data elements were not clearly and consistently  defined and technical assistance is effective but difficult to access.
The primary concerns reported by the states we visited were delays in  receiving clear, written guidance on defining and reporting certain data  elements and the lack of state input in suggesting changes to AFCARS.  Despite the written guidance available to states in the form of regulations  and an on-line policy manual, states reported that the variation in state  policies and practices make it difficult to interpret how to apply the  general guidance. As a result, states consult with HHS to ensure they are  applying the regulations appropriately. However, in commenting on a draft  of this report, officials in Oklahoma told us that a common concern among  the states is the lack of timely response from HHS when seeking guidance  on how to report data. In addition, officials in New York explained they  have made it a practice to check the HHS Web site on a regular basis for  current guidance, but have not found it a useful tool, and may turn to other  states for guidance on AFCARS reporting. In commenting on a draft of this  report, HHS explained that it first refers states to its Web site for  information and believes that the available guidance addresses states’  concerns in most instances. In addition, the states that have had an  AFCARS review experienced delays in obtaining guidance on how to  proceed following the on-site review. Although they found the review to be  very helpful, some states reported that HHS officials are delayed in  responding to their questions. For example, Texas officials reported that  the state sought clarification on its improvement plan and submitted  additional questions to HHS following the review; however, when we  spoke with the state officials, they said that they had been waiting 3  months for a response on how to proceed. An HHS official told us that  since the review process is relatively new, the agency is still developing a  process to respond to the states and recognizes that it has not been  responsive to the states already reviewed. In addition, HHS is taking steps  to gather feedback from states and other users of AFCARS data to  determine how to improve the system to make the data more accurate and  useable. As a part of these efforts, HHS has published a Federal Register  notice soliciting comments and held focus group meetings at national  conferences.
The difficulties states face in receiving federal guidance and assistance, as  well as the other challenges they face in reporting data, may negatively  impact the reliability of the data available in AFCARS and NCANDS. As a  result, states are concerned that the national standards used in the CFSR  are based on unreliable data and should not be used as a basis of  comparison and potential financial penalty. The variation in states’  reporting practices may affect the validity of the measures and may place  some states at a disadvantage. For example, the CWLA and Inspector  General studies found that approximately half the states include the  juvenile justice population in their AFCARS reports, while the other states  do not. Child welfare experts and some state officials believe that the  states that include children served by the juvenile justice agency in their  AFCARS report may report a higher number of re-entries into the child  welfare system or a higher number of moves within the system when  compared to states that do not have IV-E agreements with their juvenile  justice systems. As a result, a state that includes such children in their  AFCARS report are likely to fare less favorably when compared to the  national standard than other states on two outcome measures—foster care  re-entries and stability of foster care placements—and may face financial  penalties associated with the CFSR.
section_title: States Are Using Various Practices to Overcome System Development Challenges and Improve Data on Children’s Experiences
paragraphs: Some states are using a variety of practices to address the challenges  associated with developing SACWIS and improving data reliability,  although no formal evaluations are available on their effectiveness. To  address the challenge of developing a system to meet statewide needs,  states relied on caseworkers and supervisors from local offices to assist in  the design and testing of the system. Few states reported in our survey  strategies to overcome the other key challenges, such as limited funding  and securing knowledgeable contractors, but some states we visited have  devised some useful approaches. For example, Oklahoma child welfare  officials—in order to maximize the limited state funding for maintaining  their SACWIS—reported saving $1 million each year by hiring some of the  contractors who developed their SACWIS as permanent staff. To improve  data reliability, the 5 states we visited routinely review their data to  identify data entry errors so that managers can ensure that the missing  data are entered appropriately. In addition, some states reported that  frequent use of the data, such as publishing periodic management reports  detailing local offices’ performance on outcome measures, helps  caseworkers understand the importance of entering timely information.
subsections: section_title: States Are Primarily Relying on SACWIS Users to Overcome Some of the Challenges to Completing Their Systems
paragraphs: To overcome development challenges, survey respondents emphasized the  importance of including system users in the various phases of completing  SACWIS—planning, design, development, testing, and implementation.  Past GAO work and other research efforts have determined similar  approaches as best practices in building information systems. Forty-four  of the 46 states responding to our survey that they are developing or  operating a SACWIS indicated that they relied on internal users, such as  caseworkers and supervisors, in the development of their systems and 34  of these states said that they were extremely helpful participants. The  extent to which the users were involved in development differed across  the states. For example, in Texas, caseworkers from all of their child  welfare regions were recruited to provide input on design and  development, as well as during initial testing, pilot testing, and  implementation of the system. Arkansas reported establishing a committee  made up of users to review the work plan and sign off on recommended  changes. In addition, states reported that their system users served a  number of purposes, including serving as experts on the different  specialties within child welfare, such as child abuse, foster care, or  adoption, and as representatives from local or county offices to assist in  identifying the diverse approaches to capturing information across the  state. For example, Indiana reported that caseworkers involved in SACWIS  development represented the unique needs of the different geographical  areas of the state and helped design a uniform statewide system to meet  the diverse needs of large, intermediate, and small local offices.
Ten states noted that user input should not be limited to frontline workers,  such as caseworkers, but should include representatives from other areas  of the agency, such as the financial staff, and other agencies that serve  children, such as child support enforcement. Since many SACWIS link  with other state information systems, states advised that developing a  collaborative relationship with other state agencies will help the  development of the system. While not one of the most common challenges  reported in our survey, New Hampshire reported that one of its challenges  with meeting its SACWIS timeframe was not working collaboratively with  other agencies, such as Temporary Assistance for Needy Families  (TANF) and child support enforcement, to develop the payment  component of SACWIS. Similarly, we previously reported that the  difficulty developing linkages between social services agencies limits the  effectiveness of all the programs to serve families. To attempt to  overcome this challenge, 26 of the 46 states responding to our survey that  they are developing or operating a SACWIS indicated that they included  external public agency users and 23 reported using representatives from  other state agencies that serve children in developing their SACWIS.  Indiana said that a task force made up of representatives from the TANF  and child support enforcement agencies was developed to design the  linkages between the systems. In addition, Colorado officials reported that  they are working with the Department of Youth Corrections—an agency  that shares the SACWIS with child welfare—to ensure that the shared  screens use the same definitions.
In addition to seeking input from caseworkers and other system users  while developing SACWIS, many states continue to include users as a part  of the implementation teams, to serve as contacts in the field and provide  ongoing assistance, and to provide input on system enhancements.  Alabama responded in our survey that the state had “mentors” in each  county to help caseworkers adjust to the new system. These mentors  continue to provide ongoing support now that the system is implemented.  Similarly, Oklahoma developed Field Implementation Teams consisting of  one contractor and one child welfare staff person. During system  implementation, the teams went to field offices to provide on-site  assistance with using SACWIS and becoming accustomed with the new  method of recording child welfare information. Furthermore, Oklahoma  recruits experienced child welfare field staff for its SACWIS help desk  because of their knowledge of the system and child welfare policy and  practice.
Although states faced other challenges in completing their SACWIS, few  reported implementing approaches to overcome the barriers. According to  survey results, a common problem states faced in developing SACWIS was  receiving insufficient state funding for development. However, in our  previous work on managing information technology, we found that the IT  products can become obsolete in a matter of months rather than years,  calling for more frequent investments in upgrades and enhancements. In  addition, officials in Iowa told us that maintaining systems takes just as  much money as building them. States did not report in our survey  approaches for obtaining more funding for developing SACWIS, and few  states reported developing strategies in an attempt to overcome the  challenges associated with tight budgets for maintaining their systems.   For example, Iowa officials engaged in careful planning with system users  to ensure that they addressed the highest priorities when enhancing the  system. In particular, the officials reported that maintaining tight control  over the development and maintenance processes helps them avoid  investing inordinate amounts of resources to make corrections to the  system. In Oklahoma, child welfare officials reported that they relied on  the contractors who developed their SACWIS to conduct ongoing  maintenance activities until the contract expired in 2001. At that time, the  agency hired some of the contract staff as full-time state employees to  continue with the maintenance activities. State officials explained that this  approach ensured continuity of service, in addition to saving the agency  approximately $1 million each year. Similarly, few states reported on  approaches to overcome the challenge of finding contractors with  knowledge of child welfare practice. However, Iowa officials explained  that once the contract staff are hired, they are required to attend the same  training as new caseworkers to ensure that they are familiar with the  state’s child welfare policies and to familiarize themselves with casework  practices.
section_title: States Use Strategies, such as Producing Reports That Identify Missing Data, in an Attempt to Improve the Reliability of the Data Reported to HHS
paragraphs: Twenty-eight states reported using approaches to help caseworkers  identify the data elements that are required for federal reporting and to  help them better understand the importance of entering timely and  accurate data. Ten states responding to our survey reported reviewing the  federal reporting requirements in training sessions as a promising  approach they use to improve data quality or as a lesson learned. For  example, Tennessee reported that the state added a component about  AFCARS to the initial and ongoing training workers receive about using  SACWIS. The curriculum addresses the AFCARS report in general and the  individual data elements to help the caseworkers better understand the  purpose of collecting the information. In Nebraska, a “desk aid” that  explains the data elements and where and why to enter them in the system  is available on the caseworkers’ computer desktops. In addition, New York  has developed a step-by-step guide explaining to workers how NCANDS  data should be entered, with references to the policy or statute requiring  the information.
To improve data reliability, some states have designed their information  systems with special features to encourage caseworkers to enter the  information. Four states responding to our survey and 3 states we visited  designed their SACWIS with color-coded fields to draw attention to the  data elements that caseworker are required to enter. For example, the  AFCARS data fields in Oklahoma’s system are coded red until the data are  entered, after which the fields change to blue. In addition, workers can  look at a single screen in the Oklahoma system to see what AFCARS data  elements need to be completed without having to scroll through the entire  case record. Colorado, Iowa, New York, and Oklahoma have built into  their systems alerts—also known as “ticklers”—to remind caseworkers  and supervisors of tasks that they need to complete. For example, in Iowa,  alerts are sent to supervisors if a caseworker fails to enter the data  necessary to complete a payment to a foster care provider. Whereas, in  Oklahoma, a stoplight icon on the caseworker’s computer desktop  reminds the worker when tasks are due. A green light indicates that  nothing is due within 5 days; a yellow light means that something is due  within 5 days; and a red light means that something is overdue.  Caseworkers and supervisors in the states we visited had mixed responses  about the usefulness and effectiveness of the alerts. Some caseworkers  found them to be a nuisance, while other caseworkers and supervisors  found them to be useful tools in managing workloads and prioritizing daily  tasks.
Six states reported that the best way to improve data quality was to use  the data in published reports and hold the caseworkers and supervisors  accountable for the outcomes of the children in their care. In addition, 6  states responding to our survey reported using the data available in their  information systems to measure state outcomes similar to the CFSR. State  officials reported that this approach is an effective way to get local offices  invested in the quality of the data. For example, North Carolina publishes  monthly reports for each county comparing their performance on state  data indicators, such as the length of time children spend in care, to  counties of similar size and the state as a whole. County officials reported  that these reports encourage workers to improve the quality of the data  collected and entered into the state system since their performance is  being widely published and compared to other counties.
In addition, all the states we visited reported that frequent review of their  data, such as using software from HHS to test their AFCARS and NCANDS  data to pin-point data entry errors prior to submitting them to HHS, has  helped improve data quality. When the states identify poor data, they alert  the caseworkers and supervisors of needed corrections and data entry  improvements. For example, Colorado runs these reports about 4 to 5  times a year, with one run occurring approximately 6 weeks before each  AFCARS submission. When the data specialists find errors, they notify the  caseworker to clean up the data. New York officials told us that they  incorporate the results from these tests in training if a consistent pattern  of errors is identified.
section_title: Conclusion
paragraphs: While most states are developing statewide information systems,  challenges with data reliability remain. Although SACWIS development is  delayed in many states, state officials recognize the benefits of having a  uniform system that enhances the states’ ability to monitor the services  provided and the outcomes for children in their care. Although states  began reporting to NCANDS in1990 and were mandated to begin reporting  to AFCARS in 1995, most states continue to face challenges providing  complete, accurate, and consistent data to HHS. In addition, the results of  more recent HHS efforts, such as conducting AFCARS-related focus  groups, are unknown. Reliable data are essential to the federal  government’s development of policies that address the needs of the  children served by state child welfare agencies and its ability to assist  states in improving child welfare system deficiencies. Without well- documented, clearer guidance and the completion of more comprehensive  reviews of states’ AFCARS reporting capabilities, states are limited in  overcoming challenges that affect data reliability. Because these  challenges still remain, HHS may be using some questionable data as the  foundation for national reports and national standards for the CFSR and  may not have a clear picture of how states meet the needs of children in  their care.
section_title: Recommendation to the Secretary of Health and Human Services
paragraphs: To improve the reliability of state-reported child welfare data, we are  recommending that the Secretary of HHS consider, in addition to HHS’s  recent efforts to improve AFCARS data, ways to enhance the guidance and  assistance offered to states to help them overcome the key challenges in  collecting and reporting child welfare data. These efforts could include a  stronger emphasis placed on conducting AFCARS reviews and more  timely follow-up to help states implement their improvement plans or  identifying a useful method to provide clear and consistent guidance on  AFCARS and NCANDS reporting.
section_title: Agency Comments
paragraphs: We obtained comments on a draft of this report from the Department of  Health and Human Services’ Administration for Children and Families  (ACF). These comments are reproduced in appendix III. ACF also  provided technical clarifications, which we incorporated when  appropriate.
ACF generally agreed with our findings and commented that the report  provides a useful perspective of the problems states face in collecting data  and of ACF’s effort to provide ongoing technical assistance to improve the  quality of child welfare data. In response to our recommendation, ACF  said that we categorized its efforts as “recent” and did not recognize the  long-term efforts to provide AFCARS and NCANDS related guidance to the  states. Although we did not discuss each effort in depth, we do mention  the agency’s ongoing efforts in our report. However, we refer to the recent  efforts in the recommendation in recognition of the agency’s current  activities to formally obtain, document, and incorporate feedback from the  states with regard to collecting and reporting adoption and foster care  data. ACF also noted in its comments that the data definitions need to be  updated and revised and said it is currently in the process of revising the  AFCARS regulations to further standardize the information states are to  report—which we acknowledge in our report. In addition to the steps HHS  is taking to further improve the AFCARS data, our recommendation  encourages HHS to consider ways to enhance the ongoing guidance and  assistance offered to states to help them overcome the key challenges in  collecting and reporting child welfare data. ACF requested specific  recommendations on approaches to overcome the difficulty of collecting  and merging information from multiple state and county programs into a  single national database. While there may be additional methodologies  that the agency could use to overcome such challenges, our  recommendation focuses on improving the guidance already offered to the  states as a step to helping them better comply with the reporting  requirements.
In addition, ACF added that although staff turnover in state child welfare  agencies is a significant contributor to data quality issues, we did not focus  on this as a significant factor. ACF also commented that it is firmly  committed to continue to support the states and to provide technical  assistance and other guidance as its resources will permit. However,  because we recently issued a detailed report on a variety of caseworker  issues, we primarily focused in this report on the key data entry  challenges caseworkers face and refer readers to our previous work for  additional information on challenges related to caseworker recruitment  and retention and their affect on child welfare agencies. In commenting on  our previous work prior to its release, HHS indicated that it does not have  the authority to require states to address factors that contribute to staff  turnover, such as high caseloads and said that it has limited resources to  assist the states in the area of staff recruitment and retention.
ACF commented that it provided increased funding to the National  Resource Centers in fiscal year 2003, which they believe will improve  ACF’s ability to provide assistance to the states. After receiving the draft  report for comment, HHS separately provided information on an  additional service the National Resource Center for Information  Technology in Child Welfare provides to states. In an effort to assist states  with improving the quality of their AFCARS data, the Resource Center will  review states’ programming code used for AFCARS data. As of June 2003,  HHS reported that the Resource Center provided this assistance to  Arkansas, Louisiana, Mississippi, North Carolina, Nevada, New Jersey, and  Rhode Island, and 3 states—Maryland, Michigan, and Wisconsin—and the  District of Columbia have requested the assistance.
In response to our survey methodology, ACF requested that we explain  why the territory of Puerto Rico was not included in the state survey.   Although Puerto Rico receives federal child welfare funds, we traditionally  focus on the states and therefore do not include the U.S. territories,  including American Samoa, the Commonwealth of the Northern Mariana  Islands, Guam, Puerto Rico, and the Virgin Islands, in the scope of our  reviews.
Finally, in response to our discussion of the AFCARS review process, ACF  provided a few clarifications. During the course of our review, an HHS  official characterized the AFCARS review process as relatively new and  explained that the agency is still developing a process to respond to the  states following the completion of the on-site review. When responding to  a draft of this report, ACF disagreed with this characterization. ACF  commented that the review process has been in place since 1996, pointing  to the pilot reviews as evidence that the agency has a defined process.   However, when we requested AFCARS reports for review, HHS explained  that the states undergoing pilot reviews would be re-reviewed and that the  official process was formalized in 2001 with the release of an AFCARS  review guide and the start of the official reviews. In addition, ACF  commented that SACWIS reviews do not take priority over AFCARS  reviews. However, officials had previously explained that although  SACWIS and AFCARS reviews can happen at the same time, in practice,  the AFCARS reviews are scheduled to occur in the states that are  developing SACWIS after they have participated in a SACWIS review.   Furthermore, ACF explained that states do not develop their improvement  plan following the conclusion of the AFCARS review. Instead, ACF  officials draft the plan for the state. Although state representatives had  described a challenge in receiving timely feedback on their improvement  plan, we have changed the language in the report to reflect ACF’s  comment.
We also provided a copy of our draft to child welfare officials in the 5  states we visited—Colorado, Iowa, North Carolina, New York, and  Oklahoma. Iowa and New York had technical clarifications, which we in  incorporated when appropriate. Oklahoma provided additional  information, which was incorporated. Colorado had no suggested  corrections or edits. North Carolina did not provide any comments.
As agreed with your offices, unless you publicly announce its contents  earlier, we plan no further distribution of this report until 30 days after its  issue date. At that time, we will send copies of this report to the Secretary  of Health and Human Services, state child welfare directors, and other  interested parties. We will make copies available to others on request. In  addition, the report will be available at no charge on GAO’s Web site at  http://www.gao.gov. If you or your staff have any questions or wish to  discuss this material further, please call me at (202) 512-8403 or Diana  Pietrowiak at (202) 512-6239.  Key contributors to this report are listed in  appendix IV.
section_title: Appendix I: Scope and Methodology
paragraphs: To determine the progress states have made in developing Statewide  Automated Child Welfare Information Systems (SACWIS), we surveyed all  50 states and the District of Columbia through a Web-based survey. We  pretested the survey instrument in Maryland and the District of Columbia.  We received responses from 49 states and the District of Columbia. The  state of Nevada did not respond to the survey. We discarded a question  that asked states to report the date their advance planning document  (APD) was approved by the Department of Health and Human Services  (HHS). Due to a technical error, the date was truncated and a valid answer  was not stored in the responses. Of the 50 survey responses, 46 were from  states that are pursuing SACWIS development. The 4 states not developing  SACWIS were asked to skip sections of the survey that asked about  SACWIS development, system modifications, and supported services and  links. We did not independently verify the survey responses. In addition,  we visited 5 states to obtain more detailed and qualitative information  regarding states’ experiences developing SACWIS. We conducted site  visits in Colorado, Iowa, New York, North Carolina, and Oklahoma. We  selected these states to represent a range of SACWIS development stages,  sizes of foster care populations, and geographic locations. During our site  visits, we interviewed state and local child welfare staff, state and local  staff that regularly exchange information with the child welfare agency,  and private contractors. We also spoke with HHS staff in the central and  regional offices, National Resource Center officials, contractors involved  in SACWIS development, and child welfare experts from the Child Welfare  League of America and the American Public Human Services Association.
To determine how states and HHS ensure reliable data exist on children  served by child welfare agencies we surveyed states using the above- mentioned survey instrument. In addition, we interviewed state and HHS  officials on their efforts to analyze and compile data and HHS’s role in  providing technical assistance to states. We spoke with state officials  during our site visits and HHS officials in the central and regional offices  and attended the 6th National Child Welfare Data Conference. We  obtained and reviewed available SACWIS and Adoption and Foster Care  Analysis and Reporting System (AFCARS) reports. At the time of our  review, HHS had conducted 26 SACWIS reviews. We obtained and  reviewed 23 reports. The remaining reports were not available for review  because HHS has not yet completed the report or shared the results with  the state. Most of the SACWIS reports were considered drafts since many  states are in the process of resolving issues with completing their systems.  We reviewed AFCARS assessment reports from 6 of the 8 states assessed  by HHS—Arkansas, Connecticut, New Mexico, Texas, Vermont, and  Wyoming. HHS conducted reviews in Delaware and West Virginia after we  completed our analysis. We did not review any of the eight pilot review  reports since these were not final reports and HHS plans to conduct  official reviews in these states. These AFCARS assessment reports were  analyzed to understand the breadth of on-site assistance HHS provides to  states during the review and to identify common data collection and  reporting difficulties among states. Finally, we talked with officials in 6 of  the 8 states that had an AFCARS review about their experiences during the  review and child welfare experts.
To identify practices state and local child welfare agencies are using to  help ensure the accuracy, timeliness, and completeness of child welfare  data we interviewed state and local child welfare officials on our site visits  and inquired about the practices they are employing. We also included  questions on practices and lessons learned in our survey. In addition, we  spoke with numerous child welfare experts, including individuals from the  National Resource Center for Information Technology in Child Welfare,  the Child Welfare League of America, and the American Public Human  Services Association.
section_title: Appendix II: State SACWIS Stages of Development
section_title: Appendix III: Comments from the Department of Health and Human Services
section_title: Appendix IV: GAO Contacts and Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, Leah DeWolf and Rachel Seid made key  contributions to this report. Avrum Ashery, Patrick DiBattista,   Barbara Johnson, Valerie Melvin, and Rebecca Shea also provided key  technical assistance.
section_title: Bibliography
paragraphs: The American Public Welfare Association. Statewide Automated Child  Welfare Information Systems: Survey of State Progress. Washington,  D.C., July 1997.
The American Public Welfare Association. Child Welfare Information  Systems: Some Concepts and Their Implications. Washington, D.C., July  1994.
The American Public Welfare Association. Survey of State Child Welfare  Information Systems: Status of AFCARS and SACWIS. Washington, D.C.,  April 1995.
Caliber Associates, Analysis of State Child Welfare Data: VCIS Survey  Data from 1990 through 1994, May 1998, Department of Health and  Human Services.
Center for Technology in Government, University of Albany, SUNY. Tying  a Sensible Knot: A Practical Guide to State-Local Information Systems.  Albany, N.Y., June 1997.
Child Welfare League of America. National Working Group Highlights,  “Child Maltreatment in Foster Care: Understanding the Data.” Washington,  D.C., October 2002.
Child Welfare League of America. National Working Group Highlights,  “Placement Stability Measure and Diverse Out-of-Home Care Populations.”  Washington, D.C., April 2002.
U.S. Department of Health and Human Services, Administration for  Children and Families, Administration on Children, Youth and Families,  Children’s Bureau. Child Maltreatment 2001. Washington, D.C., 2003.
U.S. Department of Health and Human Services, Administration for  Children and Families, Administration on Children, Youth and Families,  Children’s Bureau. Child Welfare Outcomes 1999: Annual Report.  Washington, D.C., n.d..
U.S. Department of Health and Human Services, Office of Inspector  General. Adoption and Foster Care Analysis and Reporting System  (AFCARS): Challenges and Limitations. Washington, D.C., March 2003.
section_title: Related GAO Products
paragraphs: Child Welfare and Juvenile Justice: Federal Agencies Could Play a  Stronger Role in Helping States Reduce the Number of Children Placed  Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.:  April 21, 2003.
Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare  Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March  31, 2003.
Human Services: Federal Approval and Funding Processes for States’  Information Systems. GAO-02-347T. Washington, D.C.: July 9, 2002.
Foster Care: Recent Legislation Helps States Focus on Finding  Permanent Homes for Children, but Long-Standing Barriers Remain.  GAO-02-585. Washington, D.C.: June 28, 2002.
Human Services Integration: Results of a GAO Cosponsored Conference  on Modernizing Information Systems. GAO-02-121. Washington, D.C.:  January 31, 2002.
District of Columbia Child Welfare: Long-Term Challenges to Ensuring  Children’s Well-Being. GAO-01-191. Washington, D.C.: December 29, 2000.
Child Welfare: New Financing and Service Strategies Hold Promise, but  Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.
Welfare Reform: Improving State Automated Systems Requires  Coordinated Federal Effort. GAO/HEHS-00-48. Washington, D.C.: April 27,  2000.
Foster Care: States’ Early Experiences Implementing the Adoption and  Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.
Foster Care: HHS Could Better Facilitate the Interjurisdictional  Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19,  1999.
Foster Care: Effectiveness of Independent Living Services Unknown.  GAO/HEHS-00-13. Washington, D.C.: November 5, 1999.
Foster Care: Kinship Care Quality and Permanency Issues. GAO/HEHS- 99-32. Washington, D.C.: May 6, 1999.
Juvenile Courts: Reforms Aim to Better Serve Maltreated Children.  GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.
Child Welfare: Early Experiences Implementing a Managed Care  Approach. GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.
Foster Care: Agencies Face Challenges Securing Stable Homes for  Children of Substance Abusers. GAO/ HEHS-98-182. Washington, D.C.:  September 30, 1998.
Managing Technology: Best Practices Can Improve Performance and  Produce Results. GAO/T-AIMD-97-38, January 31, 1997.
Child Welfare: HHS Begins to Assume Leadership to Implement National  and State Systems. GAO/AIMD-94-37. Washington, D.C.: June 8, 1994.
Executive Guide: Improving Mission Performance Through Strategic  Information Management and Technology. GAO/AIMD-94-115. May 1,  1994.