Child and Family Services Reviews: Better Use of Data and Improved Guidance Could Enhance HHS's Oversight of State Performance
Published: 2004-04-20T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In 2001, the Department of Health and Human Services' (HHS) Administration for Children and Families (ACF) implemented the Child and Family Services Reviews (CFSR) to increase states' accountability. The CFSR uses states' data profiles and statewide assessments, as well as interviews and an on-site case review, to measure state performance on 14 outcomes and systemic factors, including child well-being and the provision of caseworker training. The CFSR also requires progress on a program improvement plan (PIP); otherwise ACF may apply financial penalties. This report examines (1) ACF's and the states' experiences preparing for and conducting the statewide assessments and on-site reviews; (2) ACF's and the states' experiences developing, funding, and implementing items in PIPs; and (3) any additional efforts that ACF has taken beyond the CFSR to help ensure that all states meet federal goals related to children's safety, permanency, and well-being.
section_title: What GAO Found
paragraphs: ACF and many state officials perceive the CFSR as a valuable process and a substantial undertaking, but some data enhancements could improve its reliability. ACF staff in 8 of the 10 regions considered the CFSR a helpful tool to improve outcomes for children. Further, 26 of 36 states responding to a relevant question in our survey commented that they generally or completely agreed with the results of the final CFSR report, even though none of the 41 states with final CFSR reports released through 2003 has achieved substantial conformity on all 14 outcomes and systemic factors. Additionally, both ACF and the states have dedicated substantial financial and staff resources to the process. Nevertheless, several state officials and child welfare experts we interviewed questioned the accuracy of the data used in the review process. While ACF officials contend that stakeholder interviews and case reviews complement the data profiles, many state officials and experts reported that additional data from the statewide assessment could bolster the evaluation of state performance. Program improvement planning is under way, but uncertainties have affected the development, funding, and implementation of state PIPs. Officials from 3 of the 5 states we visited said ACF's PIP-related instructions were unclear, and at least 9 of the 25 states reporting on PIP implementation in our survey said that insufficient funding and staff were among the greatest challenges. While ACF has provided some guidance, ACF and state officials remain uncertain about PIP monitoring efforts and how ACF will apply financial penalties if states fail to achieve their stated PIP objectives. Since 2001, ACF's focus has been almost exclusively on the CFSRs and regional staff report limitations in providing assistance to states in helping them to meet key federal goals. While staff from half of ACF's regions told us they would like to provide more targeted assistance to states, and state officials in all 5 of the states we visited said that ACF's existing technical assistance efforts could be improved, ACF officials acknowledged that regional staff might still be adjusting to the new way ACF oversees child welfare programs.

section_title: Letter
paragraphs: Recent media reports have called into question a number of states’  abilities to protect the more than 800,000 children estimated to spend  some time in foster care each year. To help states investigate reports of  child abuse and neglect, provide placements to children outside their  homes, and deliver services to help keep families together, the federal  government provides approximately $7 billion dollars to states annually. In  addition, the federal Department of Health and Human Services (HHS)  monitors states’ compliance with key federal goals, specified in part by the  Adoption and Safe Families Act (ASFA) of 1997, to keep children safe and  ensure their placement in stable and permanent homes.
In 2001, HHS—through its Administration for Children and Families’  (ACF) Children’s Bureau—began implementing a new federal review  system, known as the Child and Family Services Reviews (CFSR), to hold  states accountable for improving child welfare outcomes. Unlike prior  federal reviews—which determined states’ adherence to certain process  measures—ACF designed the CFSRs as an outcome-oriented approach to  assess children’s safety; their timely placement in permanent homes; and  their mental, physical, and educational well-being. ACF also developed a  number of factors to assess states’ performance in systemic areas, such as  staff training and foster parent licensing. According to ACF, this CFSR  process combines statewide assessments, which the states complete using  a profile of their recent child welfare data; on-site reviews of child welfare  case files; and interviews with stakeholders to provide both qualitative  and quantitative information that helps describe and clarify complex child  welfare issues. Further, states are required to develop and implement  program improvement plans (PIP) to improve their child welfare practice  and capacity when they are found to be deficient. Pursuant to CFSR  regulations, ACF can withhold federal funds if states do not show  adequate progress implementing their PIPs.
To better understand the CFSR process, you asked us to examine   (1) ACF’s and the states’ experiences preparing for and conducting the  statewide assessments and on-site reviews; (2) ACF’s and the states’  experiences developing, funding, and implementing items in their PIPs;  and (3) additional efforts, if any, that ACF has taken beyond the CFSR to  help ensure that all states meet federal goals of safety, permanency, and  well-being for children.
To conduct our work, we surveyed all 50 states, the District of Columbia,  and Puerto Rico regarding their interactions with ACF, the resources and  staffing required to prepare for the review and develop and implement  their PIPs, and any assistance ACF provided throughout the process. We  achieved a 98 percent response rate from this survey, as Puerto Rico was  the only non-respondent. To supplement this information, we also engaged  in intensive post-survey follow-up phone calls with key states in each  phase of the CFSR process. Further, we visited 5 states—California,  Florida, New York, Oklahoma, and Wyoming—to obtain first-hand  information on states’ experiences. We selected these states for diversity  in their location, size, program administration, performance on the CFSR,  and the timing of their review. We also examined all 31 approved PIPs  available as of January 1, 2004, to categorize common strategies states  developed to improve their performance. In addition, we conducted  interviews with ACF’s senior officials and regional staff from all 10  regions, ACF contractors, directors and staff from all 10 national resource  centers, and key child welfare experts. We conducted our work between  May 2003 and February 2004 in accordance with generally accepted  government auditing standards. A more detailed discussion of our scope  and methodology appears in appendix I.
subsections: section_title: Background
paragraphs: ACF’s Children’s Bureau administers and oversees federal funding to  states for child welfare services under Titles IV-B and IV-E of the Social  Security Act, and states and counties provide these child welfare services,  either directly or indirectly through contracts with private agencies.  Among other activities, ACF staff are responsible for developing  appropriate policies and procedures for states to follow to obtain and use  federal child welfare funds and conduct administrative reviews of states’  case files to ensure that children served by the state meet statutory  eligibility requirements.
In 2001, ACF launched a new outcome-oriented process, known as the  Child and Family Services Reviews (CFSR), to improve its existing  monitoring efforts, which had once been criticized for focusing exclusively  on states’ compliance with regulations rather than on their performance  over a full range of child welfare services. Passage of the 1997 Adoption  and Safe Families Act (ASFA) helped spur the creation of the CFSR by  emphasizing the outcomes of safety, permanency, and well-being for  children. Subsequently, ACF consulted with state officials, child welfare  experts, and other interested parties, and conducted pilot CFSR reviews in  14 states. In January 2000, ACF released a notice of proposed rule making  and published final CFSR regulations. In March 2001, ACF conducted the  first of its state reviews in Delaware. By March of 2004, ACF had  completed an on-site review in all 50 states, the District of Columbia, and  Puerto Rico. Although ACF plans to initiate a second round of reviews, an  official start date for this process has not yet been determined.
As figure 1 indicates, the CFSR is a four-phase process that involves state  staff as well as central and regional ACF officials. This process begins with  a statewide assessment in the first phase.
The assessment of state performance continues in the second phase, most  commonly known as the on-site review, when ACF sends a team of  reviewers to three sites in the state for one week to assess state  performance. A list of all the outcomes and systemic factors and their  associated items appears in appendix II. In assessing performance, ACF  relies, in part, on its own data systems, known as the National Child Abuse  and Neglect Data System (NCANDS) and the Adoption and Foster Care  Analysis and Reporting System (AFCARS), which were designed prior to  CFSR implementation to capture, report, and analyze the child welfare  information collected by the states. Today, these systems provide the  national data necessary for ACF to calculate national standards for key  CFSR performance items with which all states’ data will be compared.
After the on-site review, ACF prepares a final report for the state— identifying areas needing improvement, as well as the outcomes and  systemic factors for which the state was determined not to be in  substantial conformity—and provides the state with an estimated financial  penalty. As a result, the state must develop a 2-year PIP with action steps  to address its noted deficiencies and performance benchmarks to measure  progress. Once ACF approves the PIP, states are required to submit  quarterly progress reports, which ACF uses to monitor improvement.  Pursuant to CFSR regulations, federal child welfare funds can be withheld  if states do not show adequate progress as a result of PIP implementation,  but these penalties are suspended during the 2-year implementation term.  As of January 2004, no financial penalties had been applied, but according  to data on the 41 states for which final CFSR reports have been released  through December 2003, potential penalties range from $91,492 for North  Dakota to $18,244,430 for California.
ACF staff in HHS’s 10 regional offices provide technical assistance to  states through all phases of the CFSR process, and they are also  responsible for reviewing state planning documents required by Title IV-B,  assisting with state data system reviews, and assessing states’ use of IV-E  funds. In addition to these efforts, ACF has established cooperative  agreements with 10 national resource centers to help states implement  federal legislation intended to ensure the safety, permanency, and well- being of children and families. ACF sets the resource centers’ areas of  focus, and although each center has a different area of expertise, such as  organizational improvement or information technology, all of them  conduct needs assessments, sponsor national conference calls with states,  collaborate with other resource centers and agencies, and provide on-site  training and technical assistance to states.
Members of the 108th Congress introduced a proposal to provide federal  incentive payments directly to states that demonstrate significant  improvements to their child welfare systems. At the time of publication,  the House of Representatives was considering H.R. 1534, the Child  Protective Services Improvement Act, which contains provisions to award  grants to states with approved PIPs and additional bonuses to states that  have made considerable progress in achieving their PIP goals for the  previous year.
section_title: The CFSR Is a Valuable yet Substantial Undertaking, but Data Enhancements Could Improve Its Reliability
paragraphs: ACF and many state officials perceive the CFSR as a valuable process— highlighting many areas needing improvement—and a substantial  undertaking, but some state officials and child welfare experts told us that  data enhancements could improve its reliability. ACF staff in 8 of the   10 regions considered the CFSR a helpful tool to improve outcomes for  children. Further, 26 of the 36 states responding to a relevant question in  our survey commented that they generally or completely agreed with the  results of the final CFSR report, even though none of the 41 states with  final CFSR reports released through 2003 has achieved substantial  conformity on all 14 outcomes and systemic factors. In addition, both ACF  and the states have dedicated substantial financial and staff resources to  the process. However, several state officials and child welfare experts we  interviewed questioned the accuracy of the data used to compile state  profiles and establish the national standards. While ACF officials in the  central office contend that stakeholder interviews and case reviews  compliment the data profiles, many state officials and experts reported  that additional data from the statewide assessment could bolster the  evaluation of state performance.
subsections: section_title: The CFSR Is a Valuable Process for ACF and the States
paragraphs: ACF and state officials support the objectives of the review, especially in  focusing on children’s outcomes and strengthening relationships with  stakeholders, and told us they perceive the process as valuable. ACF staff  in 8 of the 10 regions considered the CFSR a helpful tool to improve  outcomes for children. Also, ACF officials from 8 regional offices noted  that the CFSRs were more intensive and more comprehensive than the  other types of reviews they had conducted in the past, creating a valuable  tool for regional officials to monitor states’ performance. In addition, state  officials from every state we visited told us that the CFSR process helped  to improve collaboration with community stakeholders. For example, a  court official in New York said that the CFSR acted as a catalyst for  improved relations between the state agency and the courts, and he  believes that this has contributed to more timely child abuse and neglect  hearings. Additionally, a state official in Florida said that the CFSR  stimulated discussions among agency staff about measuring and improving  outcomes, particularly the data needed to examine outcomes and the  resources needed to improve them. Furthermore, state staff from 4 of the   5 states we visited told us the CFSR led to increased public and legislative  attention to critical issues in child welfare. For example, caseworkers in  Wyoming told us that without the CFSR they doubted whether their state  agency’s administration would have focused on needed reforms. They  added that the agency used the CFSR findings to request legislative  support for the hiring of additional caseworkers.
In addition to affirming the value associated with improved stakeholder  relations, the ACF officials we talked to and many state officials reported  that the process has been helpful in highlighting the outcomes and  systemic factors, as well as other key performance items that need  improvement. According to our survey, 26 of the 36 states that commented  on the findings of the final CFSR report indicated that they generally or  completely agreed with the findings, even though performance across the  states was low in certain key outcomes and performance items. For  example, not one of the 41 states with final reports released through 2003  was found to be in substantial conformity with either the outcome  measure that assesses the permanency and stability of children’s living  situations or with the outcome measure that assesses whether states had  enhanced families’ capacity to provide for their children’s needs.  Moreover, across all 14 outcomes and systemic factors, state performance  ranged from achieving substantial conformity on as few as 2 outcomes and  systemic factors to as many as 9. As figure 2 illustrates, the majority of  states were determined to be in substantial conformity with half or fewer  of the 14 outcomes and systemic factors assessed.
States’ performance on the outcomes related to safety, permanency, and  well-being—as well as the systemic factors—is determined by their  performance on an array of items, such as establishing permanency goals,  ensuring worker visits with parents and children, and providing accessible  services to families. The CFSR showed that many states need  improvement in the same areas, and table 1 illustrates the 10 items most  frequently rated as needing improvement across all 41 states reviewed  through 2003.
section_title: ACF and the States Report That Reviews Have Been a Substantial Undertaking
paragraphs: Given the value that ACF and the states have assigned to the CFSR  process, both have spent substantial financial resources and staff time to  prepare for and implement the reviews. In fiscal years 2001-2003, when  most reviews were scheduled, ACF budgeted an additional $300,000  annually for CFSR-related travel. In fiscal year 2004, when fewer reviews  were scheduled, ACF budgeted about $225,000. To further enhance its  capacity to conduct the reviews, and to obtain additional logistical and  technical assistance, ACF spent approximately $6.6 million annually to  hire contractors. Specifically, ACF has let three contracts to assist with  CFSR-related activities, including training reviewers to conduct the on-site  reviews, tracking final reports and PIP documents, and, as of 2002, writing  the CFSR final reports. Additionally, ACF hired 22 new staff to build  central and regional office capacity and dedicated 4 full-time staff and   2 state government staff temporarily on assignment with ACF to assist  with the CFSR process. To build a core group of staff with CFSR expertise,  ACF created the National Review Team, composed of central and regional  office staff with additional training in and experience with the review  process. In addition, to provide more technical assistance to the states,  ACF reordered the priorities of the national resource centers to focus their  efforts primarily on helping states with the review process.
Like ACF, states also spent financial resources on the review. While some  states did not track CFSR expenses—such as staff salaries, training, or  administrative costs—of the 25 states that reported such information in  our survey, the median expense to date was $60,550, although states  reported spending as little as $1,092 and as much as $1,000,000 on the  CFSR process. For example, California officials we visited told us that  they gave each of the state’s three review sites $50,000 to cover the salary  of one coordinator to manage the logistics of the on-site review. Although  ACF officials told us that states can use Title IV-E funds to pay for some of  their CFSR expenses, only one state official addressed the use of these  funds in our survey, commenting that it was not until after the on-site  review occurred that the state learned these funds could have been used  to offset states’ expenses. Furthermore, 18 of 48 states responding to a  relevant question in our survey commented that insufficient funding was a  challenge—to a great or very great extent—in preparing for the statewide  assessment, and 11 of 40 states responded that insufficient funding was  similarly challenging in preparing for the on-site review. Officials in other  states reported that because available financial resources were  insufficient, they obtained non-financial, in-kind donations to cover  expenses associated with the on-site review. For example, a local site  coordinator in Oklahoma informed us that while the state budgeted $7 per  reviewer each day of the on-site review for food, refreshments, and other  expenses, she needed to utilize a variety of resources to supplement the  state’s budget with donations of supplies and food from local companies  and agency staff.
States reported that they also dedicated staff time to prepare for the  statewide assessment and to conduct the on-site review, which sometimes  had a negative impact on some staffs’ regular duties. According to our  survey, 45 states reported dedicating up to 200 full-time staff equivalents  (FTEs), with an average of 47 FTEs, to the statewide assessment process.  Similarly, 42 states responded that they dedicated between 3 and   130 FTEs, with an average of 45 FTEs, to the on-site review process. To  prepare for their own reviews, officials in all 5 states we visited told us  that they sent staff for a week to participate as reviewers in other states.  Additionally, local site coordinators in 4 of the 5 states we visited reported  that planning for the CFSR on-site review was a full-time job involving  multiple staff over a period of months. An official in Florida also told us  that the extensive preparation for the CFSR dominated the work of the  quality assurance unit, which was about 20 people at that time. In addition  to preparing for the review, staff in all 5 of the states we visited, who  served as reviewers in their own states, reported that to meet their  responsibility as case reviewers they worked more than 12-hour days  during the week of the on-site review. For some caseworkers, dedicating  time to the CFSR meant that they were unable or limited in their ability to  manage their typical workload. For example, Wyoming caseworkers  whose case files were selected for the on-site review told us that they  needed to be available to answer reviewers’ questions all day every day  during the on-site review, which they said prevented them from  conducting necessary child abuse investigations or home visits.
Child welfare-related stakeholders—such as judges, lawyers, and foster  parents—also contributed time to the CFSR, but some states found it took  additional staff resources to initiate and maintain stakeholder involvement  over time. According to our survey, 46 states reported that an average of  about 277 stakeholders were involved with the statewide assessment, and  42 states responded that an average of about 126 stakeholders participated  in the on-site review. However, state officials told us that it was difficult  to recruit and maintain stakeholder involvement because long time lapses  sometimes occurred between CFSR phases. For example, more than a  year can elapse between completion of the statewide assessment and the  initiation of PIP development—both key points at which stakeholder  participation is critical. Nonetheless, all 5 of the states we visited tried to  counter this obstacle by conducting phone and in-person interviews,  holding focus groups, or giving presentations in local communities to  inform and recruit stakeholders to be involved in the CFSR process. As a  result, some stakeholders were involved throughout the process. For  example, a tribal representative in Oklahoma assisted with the  development of the statewide assessment, was interviewed during the on- site review, and provided guidance on the needs of Indian children during  PIP development.
section_title: States and Child Welfare Experts Report That Several Data Improvements Could Enhance CFSR Reliability
paragraphs: State officials in all 5 states, as well as child welfare experts, reported on  several data improvements that could enhance the reliability of CFSR  findings. In particular, they highlighted inaccuracies with the AFCARS and  NCANDS data that are used for establishing the national standards and  creating the statewide data profiles, which are then used to determine if  states are in substantial conformity. These concerns echoed the findings of  a prior GAO study on the reliability of these data sources, which found  that states are concerned that the national standards used in the CFSR are  based on unreliable information and should not be used as a basis for  comparison and potential financial penalty. Several of the state officials  we visited and surveyed also questioned the reliability of data given the  variation in states’ data-reporting practice, which they believe may  ultimately affect the validity of the measures and may place some states at  a disadvantage. Furthermore, many states needed to resubmit their  statewide data after finding errors in the data profiles ACF would have  used to measure compliance with the national standards. According to  our national survey, of the 37 states that reported on resubmitting data for  the statewide data profile, 23 needed to resubmit their statewide data at  least once, with 1 state needing to resubmit as many as five times to  accurately reflect revised data. Four states reported in our survey that they  did not resubmit their data profiles because they did not know they had  this option or they did not have enough time to resubmit before the  review.
In addition to expressing these data concerns, child welfare experts as  well as officials in all of the states we visited commented that existing  practices that benefit children might conflict with actions needed to attain  the national standards. Specifically, one child welfare expert noted that if  a state focused on preventing child abuse and neglect while the child was  still living at home, and only removed the most at-risk children, the state  may perform poorly on the CFSR reunification measure—assessing the  timeliness of children’s return to their homes from foster care  arrangements—because the children they did remove would be the  hardest to place or most difficult to reunite. Additionally, officials in New  York said that they recently implemented an initiative to facilitate  adoptions. Because these efforts focus on the backlog of children who  have been in foster care for several years, New York officials predict that  their performance on the national standard for adoption will be lower  since many of the children in the initiatives have already been in foster  care for more than 2 years.
These same state officials and experts also commented that they believe  the on-site review case sample of 50 cases is too small to provide an  accurate picture of statewide performance, although ACF officials stated  that the case sampling is supplemented with additional information. Of the  40 states that commented in our survey on the adequacy of the case  sample size for the on-site review, 17 states reported that 50 cases were  very or generally inadequate to represent their caseload. Oklahoma  officials we visited also commented that they felt the case sample size was  too small, especially since they annually assess more than 800 of their own  cases—using a procedure that models the federal CFSR—and obtain  higher performance results than the state received on its CFSR.  Furthermore, because not every case in the states’ sample is applicable to  each item measured in the on-site review, we found that sometimes as few  as one or two cases were being used to evaluate states’ performance on an  item. Specifically, an ACF contractor said that several of the CFSR items  measuring children’s permanency and stability in their living conditions  commonly have very few applicable cases. For example, Wyoming had  only two on-site review cases applicable for the item measuring the length  of time to achieve a permanency goal of adoption, but for one of these  cases, reviewers determined that appropriate and timely efforts had not  been taken to achieve finalized adoptions within 24 months, resulting in  the item being assigned a rating of area needing improvement. While ACF  officials acknowledged the insufficiency of the sample size, they contend  that the case sampling is augmented by stakeholder interviews for all  items and applicable statewide data for the five CFSR items with  corresponding national standards, therefore providing sufficient evidence  for determining states’ conformity.
All of the states we visited experienced discrepant findings between the  aggregate data from the statewide assessment and the information  obtained from the on-site review, which complicated ACF’s determination  of states’ performance. Each of the 5 states we visited had at least one, and  sometimes as many as three, discrepancies between its performance on  the national standards and the results of the case review. We also found  that in these 5 states, ACF had assigned an overall rating of area needing  improvement for 10 of the 11 instances where discrepancies occurred.  ACF officials acknowledged the challenge of resolving data discrepancies,  noting that such complications can delay the release of the final report and  increase or decrease the number of items that states must address in their  PIPs. While states have the opportunity to resolve discrepancies by  submitting additional information explaining the discrepancy or by  requesting an additional case review, only one state to date has decided to  pursue the additional case review. In addition, among the states we  visited, for example, one state acknowledged that it has not opted to  pursue the supplemental case review because doing so would place  additional strain on its already limited resources.
Several state officials and experts told us that additional data from the  statewide assessments—or other data sources compiled by the states— could bolster the evaluation of states’ performance, but they found this  information to be missing or insufficiently used in the final reports.  According to our survey, of the 34 states that commented on the adequacy  of the final report’s inclusion of the statewide assessment, 10 states  reported that too little emphasis was placed on the statewide assessment.  Specifically, 1 state reported that the final report would have presented a  more balanced picture of the state’s child welfare system if the statewide  assessment were used as a basis to compare and clarify the on-site review  findings. Further, child welfare experts and state officials from California  and New York—who are using alternative data sources to AFCARS and  NCANDS, such as longitudinal data that track children’s placements over  time—told us that the inclusion of this more detailed information would  provide a more accurate picture of states’ performance nationwide. North  Carolina officials also reported in our survey that they tried to submit  additional longitudinal data—which they use internally to conduct  statewide evaluations of performance—in their statewide assessment, but  HHS would not accept the alternative data for use in evaluating the state’s  outcomes. An HHS official told us that alternative data are used only to  assess state performance in situations where a state does not have  NCANDS data, since states are not mandated to have these systems.
Given their concerns with the data used in the review process, state  officials in 4 of the 5 states believed that the threshold for achieving  substantial conformity was difficult to achieve. One state official we  visited acknowledged that she believed child welfare agencies should be  pursuing high standards to improve performance, but she questioned the  level at which ACF established the thresholds. While an ACF official told  us that different thresholds for the national standards had been  considered, ACF policy makers ultimately concluded that a more rigorous  threshold would be used. ACF officials recognize that they have set a high  standard. However, they believe it is attainable and supportive of their  overall approach to move states to the standard through continuous  improvement.
In preparation for the next round of CFSRs, ACF officials have formed a  Consultation Work Group of ACF staff, child welfare administrators, data  experts, and researchers who will propose recommendations on the CFSR  measures and processes. The group began its meetings prior to our  publication, but no proposals were available.
section_title: Program Improvement Planning Under Way, but Uncertainties Challenge Plan Development, Implementation, and Monitoring
paragraphs: Forty-one states are engaged in program improvement planning, but many  uncertainties, such as those related to federal guidance and monitoring  and the availability of state resources, have affected the development,  implementation, and funding of the PIPs. State PIPs include strategies  such as revising or developing policies, training caseworkers, and  engaging stakeholders, and ACF has issued regulations and guidance to  help states develop and implement their plans. Nevertheless, states  reported uncertainty about how to develop their PIPs and commented on  the challenges they faced during implementation. For example, officials  from 2 of the states we visited told us that ACF had rejected their PIPs  before final approval, even though these officials said that the plans were  similar in the level of detail included in previously approved PIPs that  regional officials had provided. Further, at least 9 states responding to our  survey indicated that insufficient time, funding, and staff, as well as high  caseloads, were the greatest challenges to PIP implementation. As states  progress in PIP implementation, some ACF officials expressed a need for  more guidance on how to monitor state accomplishments, and both ACF  and state officials were uncertain about how the estimated financial  penalties would be applied if states fail to achieve the goals described in  their plans.
subsections: section_title: State Plans Include a Variety of Strategies to Address Identified Weaknesses
paragraphs: State plans include a variety of strategies to address weaknesses identified  in the CFSR review process. However, because most states had not  completed PIP implementation by the time of our analysis, the extent to  which states have improved outcomes for children has not been  determined. While state PIPs varied in their detail, design, and scope,  according to our analysis of 31 available PIPs, these state plans have  focused to some extent on revising or developing policies; reviewing and  reporting on agency performance; improving information systems; and  engaging stakeholders such as courts, advocates, foster parents, private  providers, or sister agencies in the public sector. Table 2 shows the  number of states that included each of the six categories and  subcategories of strategies we developed for the purposes of this study,  and appendix I details our methodology for this analysis.
Our analysis found that every state’s PIP has included policy revisions or  creation to improve programs and services. For example, to address  unsatisfactory performance in the prevention of repeat maltreatment,  California’s PIP includes a policy change, pending legislative approval,  granting more flexibility to counties in determining the length of time they  spend with families to ensure child safety and improve family functioning  before closing cases. Additionally, 30 of the plans included caseworker  training; 20 of the plans included requests for state legislative action; and  27 of the plans included requests for federal technical assistance from ACF  or the resource centers. For example, California planned to ask its  legislature to allow the agency to consolidate standards for foster care and  adoption home studies, believing this would facilitate the adoption of  children in the state. In addition, New York worked with its legislature to  secure additional funding to improve the accessibility of independent  living and adoption services for children and families.
Our analysis also showed that many states approached PIP development  by building on state initiatives in place prior to the on-site review. Of the  42 surveyed states reporting in our survey on this topic, 30 said that their  state identified strategies for the PIP by examining ongoing state  initiatives. For example, local officials in New York City and state officials  in California told us that state reform efforts—borne in part from legal  settlements—have become the foundation for the PIP. In New York, the  Marisol case was one factor in prompting changes to the state’s child  welfare financing structure. Subsequently, the state legislature  established a quality enhancement fund. Much of this money today  supports strategies in New York’s PIP, such as permanency mediation to  support family involvement in case planning and new tools to better assess  children’s behavioral and mental health needs.  California state officials  also informed us that state reform efforts initiated by the governor prior to  the CFSR, such as implementing a new system for receiving and  investigating reports of abuse and neglect and developing more early  intervention programs, became integral elements in the PIP.
section_title: Insufficient Guidance Hampered State Planning Efforts, but ACF Has Taken Steps to Clarify Expectations and Improve Technical Assistance
paragraphs: ACF has provided states with regulations and guidance to facilitate PIP  development, but some states believe the requirements have been unclear.  Some of the requirements for program improvement planning are outlined  in the following table.
Some states in our survey indicated that the guidance ACF had provided  did not clearly describe the steps required for PIP approval. In addition,  some state officials believe that even ACF’s more recent efforts to improve  PIP guidance have also been insufficient. Of the 21 states reporting on the  PIP approval process in our survey, 6 states—4 reviewed in 2001 and   2 reviewed in 2002—said that ACF did not clearly describe its approval  process, and another 8 states reported that ACF described the approval  process as clearly as it did unclearly. Further, several states commented in  our survey that several steps in the approval process were unclear to  them, such as how much detail and specificity the agency expects the plan  to include, what type of feedback states could expect to receive, when  states could expect to receive such feedback, and whether a specific  format was required. Officials in the states we visited echoed survey  respondents’ concerns with officials from 3 of the 5 states informing us  that ACF had given states different instructions regarding acceptable PIP  format and content. For example, California and Florida officials told us  that their program improvement plans had been rejected prior to final  approval, even though they were based on examples of approved plans  that regional officials had provided. In addition, California officials told us  that they did not originally know how much detail the regional office  expected in the PIP and believed that the level of detail the regional office  staff ultimately required was too high. Although some steps may be  duplicative, officials in California said that the version of their plan that  the region accepted included 2,932 action steps—a number these officials  believe is too high given their state’s limited resources and the 2-year time  frame to implement the PIP.
ACF officials have undertaken several steps to clarify their expectations  for states and to improve technical assistance, but state responses to this  assistance have been mixed. For example, in 2002, 2 years after ACF  released the CFSR regulations and a procedures manual, ACF offered  states additional guidance and provided a matrix format to help state  officials prepare their plans. ACF officials told us the agency is also  helping states through a team approach to providing on-site technical  assistance. Under this approach, when ACF determines a state is slow in  developing its PIP, the agency sends a team of staff from ACF and  resource centers to the state to provide intensive on-site technical  assistance. An official from West Virginia who had received this team  assistance reported that ACF’s support was very beneficial.  Further, ACF  has attempted to encourage state officials to start developing program  improvement plans before the final report is released. To do so, the agency  has provided training to state officials and stakeholders almost  immediately after the completion of the on-site review. ACF has sent staff  from the resource center for Organizational Improvement to provide such  training.  An official from Utah, however, reported that the resource center  training on PIP development had been general, and she wished the  resource center staff had better tailored its assistance and provided more  examples of strategies other states are pursuing to improve.
Analysis of state survey responses indicates that starting to develop  improvement plans early can make the 90-day time frame to prepare a PIP  seem adequate. Of 9 states reporting that they started developing their PIP  before or during the statewide assessment phase, 5 said that 90 days was  adequate. Nonetheless, 21 of 35 state survey respondents reported that the  90-day time frame was insufficient. For example, one respondent reported  that 90 days is too short a time to perform tasks necessary for developing a  program improvement plan, such as analyzing performance data, involving  diverse groups of stakeholders in planning decisions, and obtaining the  approval of state officials. Survey results indicate that increasing numbers  of states are developing their PIPs early in the CFSR process, which may  reflect ACF’s emphasis on PIP development. The following figure shows  that of the 18 states reviewed in 2001, only 2 started developing their PIPs  before or during the statewide assessment phase. Among states reviewed  in 2003, this share increased to 5 of 9.
Evidence suggests that lengthy time frames for PIP approval have not  necessarily delayed PIP implementation, and ACF has made efforts to  reduce the time the agency takes to approve states’ PIPs. For example,  officials in 3 of the 5 states we visited told us they began implementing  new action steps before ACF officially approved their plans because many  of the actions in their PIPs were already under way. In addition, according  to our survey, of the 28 states reporting on this topic, 24 reported that they  had started implementing their PIP before ACF approved it. Further, our  analysis shows that the length of time between the PIP due date, which  statute sets at 90 days after the release of the final CFSR report, and final  ACF PIP approval has ranged considerably—from 45 to 349 business days.  For almost half of the plans, ACF’s approval occurred 91 to 179 business  days after the PIP was due. As shown in figure 4, however, our analysis  indicates that ACF has recently reduced the time lapse by 46 business days  between states’ PIP due dates and ACF’s PIP approval.
The shorter time lapse for PIP approval may be due, in part, to the ACF’s  emphasis on PIP development. According to one official, ACF has directed  states to concentrate on submitting a plan that can be quickly approved.  Another ACF official added that because of ACF’s assistance with PIP  development, states are now submitting higher-quality PIPs that require  fewer revisions.
section_title: State and Federal Uncertainties Cloud PIP Implementation and Monitoring
paragraphs: Program improvement planning has been ongoing, but uncertainties have  made it difficult for states to implement their plans and ACF to monitor  state performance. Such uncertainties include not knowing whether state  resources are adequate to implement the plans and how best to monitor  state reforms. In answering a survey question about PIP implementation  challenges, a number of states identified insufficient funding, staff, and  time—as well as high caseloads—as their greatest obstacles. Figure 5  depicts these results.
In regards to funding, an official from Pennsylvania commented that  because of the state’s budget shortfall, no additional funds were available  for the state to implement its improvement plan, so most counties must  improve outcomes with little or no additional resources. A Massachusetts  official reported that fiscal problems in his state likely would lead the state  to lay off attorneys and caseworkers and to cut funding for family support  programs. While state officials acknowledged that they do not have  specific estimates of PIP implementation expenses because they have not  tracked this information in their state financial systems, many states  indicated that to cope with financial difficulties, they had to be creative  and use resources more efficiently to fund PIP strategies. Of the 26 states  responding to a question in our survey on PIP financing, 12 said that they  were financing the PIP strategies by redistributing current funding, and 7  said that they were using no-cost methods. In an example of the latter,  Oklahoma officials reported pursuing in-kind donations from a greeting  card company so that they could send thank-you notes to foster parents,  believing this could increase foster parent retention and engagement.
States also reported that PIP implementation has been affected by staff  workloads, but these comments were mixed. In Wyoming, for example,  caseworkers told us that their high caseloads would prevent them from  implementing many of the positive action steps included in their  improvement plan. In contrast, Oklahoma caseworkers told us that the  improvement plan priorities in their state—such as finding permanent  homes for children—have helped them become more motivated, more  organized, and more effective with time management. For example, one  caseworker explained that she is quicker now at locating birth fathers who  were previously uninvolved in the child’s life because she uses the Internet  to search for these fathers’ names. She said this new way of exploring  leads and information—a strategy that stemmed from PIP development— has been motivating and rewarding because it has decreased the time  spent tracking down paternal relatives and increased the number of  available placements for the child.
ACF officials expressed uncertainty about how best to monitor states’  progress and apply estimated financial penalties when progress was slow  or absent, and 3 of the 5 states we visited reported frustration with the  limited guidance ACF had provided on the PIPs quarterly reporting  process. For example, 4 regional offices told us that they did not have  enough guidance on or experience with evaluating state quarterly reports.  Some regional offices told us they require states to submit evidence of  each PIP action step’s completion, such as training curricula or revised  policies, but one ACF official acknowledged that this is not yet standard  procedure, although the agency is considering efforts to make the  quarterly report submission procedures more uniform. Moreover, ACF  staff from one region told us that because PIP monitoring varies by region,  they were concerned about enforcing penalties. Finally, shortly before  California’s quarterly report was due, state officials told us they still did  not know how much detail to provide, how to demonstrate whether they  had completed certain activities, or what would happen if they did not  reach the level of improvement specified in the plan.
Based on data from the states that have been reviewed to date, the  estimated financial penalties range from a total of $91,492 for North  Dakota to $18,244,430 for California, but the impact of these potential  penalties remains unclear. While ACF staff from most regional offices told  us that potential financial penalties are not the driving force behind state  reform efforts, some contend that the estimated penalties affect how  aggressively states pursue reform in their PIPs. For example, regional  office staff noted that one state’s separate strategic plan included more  aggressive action steps than those in its PIP because the state did not want  to be liable for penalties if it did not meet its benchmarks for  improvement. State officials also had mixed responses as to how the  financial penalties would affect PIP implementation. An official in  Wyoming said that incurring the penalties was equivalent to shutting down  social service operations in one local office for a month, while other  officials in the same state thought it would cost more to implement PIP  strategies than it would to incur financial penalties if benchmarks were  unmet. Nevertheless, these officials also said that while penalties are a  consideration, they have used the CFSR as an opportunity to provide  better services. One official in another state agreed that it would cost more  to implement the PIP than to face financial penalties, but this official was  emphatic in the state’s commitment to program improvement.
section_title: ACF’s Focus Rests Almost Exclusively on Implementing the CFSR
paragraphs: To implement the CFSRs, ACF has focused its activities almost entirely on  the four phases of the review process. However, staff in several regions  report limitations in providing assistance to states in helping them to meet  key federal goals. Although regional staff conduct site visits to states for  reasons beyond the CFSR process, conducting the CFSR on-site reviews  and providing PIP-related assistance to states account for the majority of  regions’ time and travel budgets, according to ACF officials. Further,  regional office staff said that more frequent visits with state personnel— visits outside of the CFSR process in particular—would allow them to  better understand states’ programs and cultivate relationships with state  officials. In addition, state officials in all five of the states we visited said  that ACF technical assistance needed improvement, acknowledging that in  some cases regional office staff were stretched thin by CFSR demands and  in other cases that assistance from resource center staff lacked focus.  While ACF officials in the central office said that the CFSR has become the  primary method for evaluating states’ performance, they acknowledged  that regional staff might still be adjusting to the new way ACF oversees  child welfare programs. Further, they told us that ACF is currently  reevaluating the entire structure of its training and technical assistance, in  part to address these concerns.
ACF officials told us that the learning opportunities in the Children’s  Bureau are intentionally targeted at the CFSR, but staff in 3 regions told us  that this training should cover a wider range of subjects—including topics  outside of the CFSR process—so that regional officials could better meet  states’ needs. All 18 of the courses that ACF has provided to its staff since  2001 have focused on topics such as writing final CFSR reports and using  data for program improvement. ACF officials in the central office said that  the course selection reflects both the agency’s prioritization of the CFSR  process and staff needs. To ascertain staff training needs, ACF surveyed  regional staff in October 2002, and ACF officials told us they used the  survey results in deciding which courses to offer. Our analysis of this  survey, however, showed that it focused only on training topics directly  related to the CFSR, so it might offer only limited information on whether  regional officials wanted training on other topics. Specifically, the survey  asked staff to check their top 5 training choices from among   11 CFSR-related topics. While survey respondents were also given the  opportunity to write in additional training tropics they desired, only 2 of  the survey’s 27 respondents did so. One indicated a greater need for  training on Indian child welfare issues, and another expressed a desire to  learn more about the entire child welfare system. Although it is not  possible to determine whether more respondents would have prioritized  non-CFSR training areas had the survey been designed to elicit such  information, our interviews with regional staff suggest that some of them  wish to obtain additional non-CFSR training. For example, a staff member  from one region told us she has not been adequately trained in child  welfare and believed that her credibility was damaged when a state  wanted advice that she could not provide on how to help older youth  prepare to exit from foster care.
In addition to offering training, ACF organizes biennial conferences for  state and federal child welfare officials. Nonetheless, staff from 5 regions  told us that they wanted more substantive interaction with their ACF  colleagues, such as networking at conferences, to increase their overall  child welfare expertise. Staff from 6 of the 10 regions told us that their  participation in conferences is limited because of funding constraints.
Further, staff in all 10 regions provide ongoing assistance or ad hoc  counseling to states, but staff from 6 regions told us they would like to  conduct site visits with states more regularly to improve their  relationships with state officials and provide more targeted assistance. For  example, staff in most regions told us that they assist states predominantly  by e-mail and telephone on topics such as interpretation of Title IV-E  eligibility criteria. Additionally, staff in 7 regions said that they sometimes  visit with states to participate in state planning meetings—part of the  annual child and family services planning effort—or to give presentations  at state conferences on topics such as court improvement. However, staff  in 4 regions felt their travel funds were constrained and explained that  they try to stretch their travel dollars by addressing states’ non-CFSR  needs, such as court improvements, during CFSR-related visits.  While an  ACF senior official from central office confirmed that CFSR-related travel  constituted 60 percent of its 2002 child welfare-monitoring budget, this  official added that CFSR spending represents an infusion of funding rather  than a reprioritization of existing dollars and stated that regional  administrators have discretion over how the funds are allocated within  their regions. In addition, the same official stated that he knew of no  instance in which a region requested more money for travel than it  received.
Concerns from state officials in all 5 of the states we visited echoed those  of regional office staff and confirmed the need for improvements to the  overall training and technical assistance structure, while respondents’  comments on our survey showed more mixed perceptions on the quality of  assistance they received. For example, state officials in New York and  Wyoming commented that ACF staff from their respective regional offices  did not have sufficient time to spend with them on CFSR matters because  regional staff were simultaneously occupied conducting reviews in other  states. Further, an Oklahoma state official commented that assistance  from one resource center was not as specific or helpful as desired.  Specifically, when the state asked the resource center to provide a  summary of other states’ policies regarding the intake of abuse and neglect  allegations, the resource center did not provide an analysis of sufficient  depth for the state to explore possible reforms.
According to state survey respondents, however, satisfaction with the  training and technical assistance provided by regional offices varied by  CFSR phase. For example, among states reviewed in 2001, 2002, and 2003,  satisfaction was generally highest in the statewide assessment phase, but  then dropped during on-site review and PIP development before rising  again in the PIP implementation phase. Across all phases of the CFSR  process, however, states reviewed in 2003 had much higher levels of  satisfaction with regional office assistance than those states reviewed in  2001, suggesting improvements to regional office training and technical  assistance as the process evolved. Further, based on survey data and our  follow-up calls with selected states, satisfaction was also mixed in regard  to resource center provided assistance. For example, among states  reporting in our survey on the quality of assistance provided by the  resource center for Organizational Improvement and the resource center  for Information Technology—the two resource centers that provide  specific support to states regarding data issues and PIP development— satisfaction was generally lower in every phase among states reviewed in  2003 than among states reviewed in 2001. The only exception to this was  during the PIP development phase, for which states reviewed in 2003  reported higher levels of satisfaction with the resource center for  Organizational Improvement than states reviewed in 2001, suggesting  positive responses to the on-site training and technical assistance this  resource center has recently been providing to aid states in their PIP  planning efforts.
ACF officials told us the CFSR has become the agency’s primary  mechanism for monitoring states and facilitating program improvement,  but they acknowledged that regional office staff might not have realized  the full utility of the CFSR as a tool to integrate all existing training and  technical assistance efforts. Further, according to ACF officials, meetings  to discuss a new system of training and technical assistance are ongoing,  though recommendations were not available at the time of publication.  Levels of resource center funding, the scope and objectives of the  resource centers’ work, and the contractors who operate the resource  centers are all subject to change before the current cooperative  agreements expire at the close of fiscal year 2004.
section_title: Conclusions
paragraphs: ACF and the states have devoted considerable resources to the CFSR  process, but concerns remain regarding the validity of some data sources  and the limited use of all available information to determine substantial  conformity.  Further, no state to date has passed the threshold for  substantial conformity on all CFSR measures.  The majority of states  surveyed agreed that CFSR results are similar to their own evaluation of  areas needing improvement. However, without using more reliable data— and in some cases, additional data from state self-assessments—to  determine substantial conformity, ACF may be over- or underestimating  the extent to which states are actually meeting the needs of the children  and families in their care. These over- or underestimates can, in turn,  affect the scope and content of the PIPs that states must develop in  response. As states face difficult budget decisions, accurate performance  information could be critical to deciding how best to allocate resources.  We previously reported on the reliability of state-reported child welfare  data and recommended that HHS consider additional ways to enhance the  guidance and assistance offered to states to help them overcome the key  challenges in collecting and reporting child welfare data. In response to  this recommendation, HHS said that ACF has provided extensive guidance  on how states can improve the quality of their data and acknowledged that  additional efforts were under way.
In addition, the PIP development, approval, and monitoring processes  remain unclear to some, potentially reducing states’ credibility with their  stakeholders and straining the federal/state partnership. Similarly, regional  officials are unclear as to how they can accomplish their various training  and technical assistance responsibilities, including the CFSR. Without  clear guidance on how to systematically prepare and monitor PIP-related  documents, and how regional officials can integrate their many oversight  responsibilities, ACF has left state officials unsure of how their progress  over time will be judged and potentially complicated its own monitoring  efforts.
section_title: Recommendations for Executive Action
paragraphs: To ensure that ACF uses the best available data in measuring state  performance, we recommend that the Secretary of HHS expand the use of  additional data states may provide in their statewide assessments and  consider alternative data sources when available, such as longitudinal data  that track children’s placements over time, before making final CFSR  determinations.
In addition, to ensure that ACF regional offices and states fully understand  the PIP development, approval, and monitoring processes, and that  regional offices fully understand ACF’s prioritization of the CFSR as the  primary mechanism for child welfare oversight, we recommend that the  Secretary of HHS take the following two actions:  issue clarifying guidance on the PIP process and evaluate states’ and  regional offices’ adherence to this instruction, and    provide guidance to regional offices explaining how to better integrate the  many training and technical assistance activities for which they are  responsible, such as participation in state planning meetings and the  provision of counsel to states on various topics, with their new CFSR  responsibilities.
section_title: Agency Comments
paragraphs: We received comments on a draft of this report from HHS.  These  comments are reproduced in appendix IV. HHS also provided technical  clarifications, which we incorporated where appropriate.
HHS generally agreed with our findings and noted that the CFSR process  has already focused national attention on child welfare reform, but  because the CFSR is the first review of its kind, HHS is engaged in  continuous monitoring and improvement of the process. However, in its  technical comments, HHS commented that while it acknowledges that the  CFSR is its top priority, it disagreed with our statement that HHS’s focus  rests exclusively on implementing the CFSR, stating that the  Administration for Children and Families (ACF) also conducts other  oversight efforts, such as Title IV-E eligibility reviews and AFCARS  assessments. While we acknowledged ACF’s other oversight activities in  the background section of the report, this report focuses primarily on the  CFSR and we reflected the comments that ACF officials made throughout  the course of our work that the CFSR was the primary tool for monitoring  state performance and that it served as the umbrella for all monitoring  activities undertaken by central and regional ACF staff.
HHS further noted in its technical comments that we were wrong to  suggest that federal staff do not know how to monitor state PIPs or assess  financial penalties. However, we do not report that ACF is unsure of how  to monitor PIPs or how to assess financial penalties—rather, we reported  that ACF regional staff have not received sufficient guidance on how to  best monitor PIPs and that ACF officials have not decided how or when to  apply such penalties, even though two states to date have completed their  initial PIP implementation timeframe and all states reviewed thus far are  engaged in PIP development and implementation.
With regard to our first recommendation, HHS acknowledged that several  steps are under way to address necessary data improvements and said that  states have begun to submit more accurate information in their AFCARS  and NCANDS profiles, with HHS’s assistance. HHS also commented that  we failed to properly emphasize the states’ responsibility to improve  overall data quality. We believe that our report, as well as our previous  report on child welfare data and states’ information systems, addresses  HHS’s activities and the steps many states have taken to enhance their  CFSR data. Given that many states have developed independent data  collection tools—and included findings from these instruments in their  statewide assessments—our recommendation is meant to encourage HHS  to work more closely with all states to supplement their AFCARS and  NCANDS data in order to improve the determinations made about state  performance.
In addition, HHS commented that our report emphasized the limitations of  the 50 case sample size without focusing on the expenses and the  increased state and federal staff time that would likely be associated with  efforts to increase the sample size. We agree that additional expenses and  staff time would likely be needed to increase the sample size and  recommended that ACF use additional data—beyond the information  collected from the 50 case reviews and ACFARS and NCANDS data—to  develop a more accurate picture of state performance. This information  could include the data that many states already collect on their  performance, such as longitudinal information tracking children from their  time of entry into the system.
In response to our second recommendation, HHS said that it has  continued to provide technical assistance and training to states and  regional offices, when appropriate. HHS noted that it is committed to  continually assessing and addressing training and technical assistance  needs. In this context, our recommendation was intended to encourage  HHS to enhance existing training efforts and focus both on state and on  regional officials’ needs in understanding and incorporating the CFSR  process into their overall improvement and oversight efforts.
We also provided a copy of our draft report to child welfare officials in the  five states we visited—California, Florida, New York, Oklahoma, and  Wyoming. We received comments from California, Florida, New York, and  Oklahoma, all of which generally agreed with our findings and provided  various technical comments, which we also incorporated where  appropriate.
We are sending copies of this report to the Secretary of Health and Human  Services, state child welfare directors, and other interested parties. We  will make copies available to others on request. In addition, the report will  be available at no charge on GAO’s Web site at http://www.gao.gov. If you  or your staff have any questions or wish to discuss this material further,  please call me at (202) 512-8403 or Diana Pietrowiak at (202) 512-6239. Key  contributors to this report are listed in appendix IV.
section_title: Appendix I: Objectives, Scope, and Methodology
subsections: section_title: Objectives
paragraphs: The objectives of our study were to report on (1) ACF’s and the states’  experiences preparing for and conducting the statewide assessments and  on-site reviews; (2) ACF’s and the states’ experiences developing, funding,  and implementing items in the PIP; and (3) additional efforts that ACF has  taken beyond the CFSR to ensure that all states meet federal goals of  safety, permanency, and well-being.
section_title: Scope and Methodology
paragraphs: To gather information about ACF’s and the states’ experiences with the  CFSR and PIP process, we utilized multiple methodologies to solicit  information from both ACF and the states, including (1) a Web-based  survey to state child welfare agencies; (2) site visits to five states; (3) a  content analysis of all 31 PIPs available as of January 1, 2004;   (4), interviews with ACF officials in Washington and all regional offices,  directors of all resource centers, and child welfare experts nationwide;  and (5) a review of CFSR regulations and the available guidance offered to  states. We conducted our work between May 2003 and February 2004 in  accordance with generally accepted government auditing standards.
subsections: section_title: Survey
paragraphs: To gather information about states’ experiences with each phase of the  CFSR and PIP process, we distributed a Web-based survey to all 50 states,  the District of Columbia, and Puerto Rico on July 30, 2003. We pretested  the survey instrument with officials in the District of Columbia, Kentucky,  and Maryland; and after extensive follow-up, we received survey  responses from all 50 states and the District of Columbia for a 98 percent  response rate. We did not independently verify the information obtained  through the survey.
The survey asked a combination of questions that allowed for open-ended  and close-ended responses. Because some states had not yet begun their  statewide assessments and others had already submitted quarterly PIP  progress reports at the time that our survey was released, the instrument  was designed with skip patterns directing states to comment only on the  CFSR stages that they had begun or completed to that point. Therefore,  the number of survey respondents for each question varied depending on  the number of states that had experienced that stage of the CFSR and PIP  processes.
To supplement the survey and elaborate on survey responses, we selected  10 states with which to conduct follow-up phone calls based on their  answers to the survey’s open-ended questions.  These calls helped us  obtain more specific examples about states’ experiences preparing for the  CFSR; developing, funding, and implementing a PIP; and working with  ACF to improve their child welfare systems.
Because this was not a sample survey, there are no sampling errors.  However, the practical difficulties of conducting any survey may introduce  errors, commonly referred to as nonsampling errors. For example,  difficulties in how a particular question is interpreted, in the sources of  information that are available to respondents, or in how the data are  entered into a database or were analyzed, can introduce unwanted  variability into the survey results. We took steps in the development of the  questionnaire, the data collection, and the data analysis to minimize these  nonsampling errors. For example, social science survey specialists  designed the questionnaire in collaboration with GAO staff with subject  matter expertise. Then, the draft questionnaire was pretested with a  number of state officials to ensure that the questions were relevant, clearly  stated, and easy to comprehend. When the data were analyzed, a second,  independent analyst checked all computer programs. Since this was a  Web-based survey, respondents entered their answers directly into the  electronic questionnaire. This eliminated the need to have the data keyed  into a database thus removing an additional source of error.
section_title: Site Visits
paragraphs: To gather more detailed information about the states’ experiences with the  CFSR and PIP process, we selected five states to visit—California, Florida,  New York, Oklahoma, and Wyoming—based on the timing and results of  each state’s CFSR, as well as their differences in location, size of child  welfare population, degree of privatization of services, size of tribal  populations, and whether they had state or locally administered systems.  In preparation for the visits and to understand the unique circumstances in  each state, we obtained and reviewed relevant literature from each of the  five states, such as the statewide assessment, the CFSR final report, and  any available PIPs or quarterly reports. Additionally, we reviewed relevant  past or current litigation that may affect the states’ delivery of services as  identified by the National Center for Youth Law’s Litigation Docket (2002).
During our visits to each state, we talked with officials from the state child  welfare agency along with officials and staff from at least one local agency  office that was selected for the CFSR on-site review. Specifically, in each  state we spoke with state and local officials responsible for guiding the  states’ efforts throughout the review process; CFSR on-site reviewers; and  stakeholders, including judges, child advocates, private providers, foster  parents, and child welfare staff. Some detailed information regarding key  CFSR milestones among the five states we visited is included in appendix  III.
section_title: Content Analysis of Available PIPs
paragraphs: To learn about states’ improvement strategies, we conducted a content  analysis of the 31 available PIPs that ACF had approved by January 1,  2004. For each of these PIPs, we classified the state’s action steps as  relating to one or more of the following: policies and procedures, data  collection and analysis, staff supports, foster parent supports or services  and resources for children and families, state legislative supports, and  federal technical assistance. Table 2 in the report summarizes how we  classified the PIP strategies and indicates the number of states including  each strategy in its PIP.
section_title: Interviews
paragraphs: To gather information about ACF’s experience with the CFSR and PIP  process, we interviewed ACF officials in Washington, D.C., who are  involved in the CFSR process and ACF staff in all 10 of the regional  offices; directors of each resource center; and ACF contractors working  on CFSR-related activities. Further, we observed the final debriefing  sessions for three states—South Carolina, Virginia, and Washington— during the weeks of their respective on-site reviews.
In addition to our interviews with ACF officials, we also interviewed 10  prominent child welfare experts and researchers, such as those affiliated  with the Chapin Hall Center for Children, the Child Welfare League of  America, the National Coalition on Child Protection Reform, and the  University of California at Berkeley, to learn additional information about  the states’ experiences with the CFSR process, including information  about states’ concerns with the reliability of CFSR data, states’  involvement of tribes as stakeholders, and the media’s coverage of  egregious child welfare cases.
section_title: Review of ACF Guidance to States
paragraphs: To gather information about CFSR regulations and the available training  and technical assistance offered to states, we reviewed ACF’s regulations,  its policy memorandums, and the CFSR manual it makes available to  states. In addition, we obtained and reviewed a list of all of the resource  centers’ training and technical assistance activities provided to the five  states we visited during our site visits.
section_title: Appendix II: List of Outcomes and Systemic Factors and Their Associated Items
paragraphs: (Items with an asterisk have associated national standards.)
subsections: section_title: Outcome factors
paragraphs: Outcome 1: Children are, first and foremost, protected from abuse and  neglect.
Item 1: Timeliness of initiating investigations on reports of child  maltreatment  Item 2: Repeat maltreatment    Recurrence of maltreatment*   Incidence of child abuse and/or neglect in foster care*  Outcome 2: Children are safely maintained in their own homes  whenever possible and appropriate.
Item 3: Services to family to protect child(ren) in home and prevent  removal  Item 4: Risk of harm to child(ren)
Outcome 3: Children have permanency and stability in their living  conditions.
Item 5: Foster care re-entries*  Item 6: Stability of foster care placement*  Item 7: Permanency goal for child  Item 8: Reunification, guardianship, or permanent placement with relatives    Length of time to achieve permanency goal of reunification*    Length of time to achieve permanency goal of adoption* Item 10: Permanency goal of other planned permanent living arrangement  Outcome 4: The continuity of family relationships and connections is  preserved for children.
Item 11: Proximity of foster care placement  Item 12: Placement with siblings  Item 13: Visiting with parents and siblings in foster care  Item 14: Preserving connections  Item 15: Relative placement  Item 16: Relationship of child in care with parents  Outcome 5: Families have enhanced capacity to provide for their  children’s needs.
Item 17: Needs and services of child, parents, foster parents  Item 18: Child and family involvement in case planning  Item 19: Worker visits with child  Item 20: Work visits with parent(s)
Outcome 6: Children receive appropriate services to meet their  educational needs.
Item 21: Education needs of the child  Outcome 7: Children receive adequate services to meet their physical  and mental health needs.
Item 22: Physical health of child  Item 23: Mental health of child.
section_title: Systemic Factors
paragraphs: Systemic factor 1: Statewide information system  Item 24: State is operating a statewide information system that, at a  minimum, can identify the status, demographic characteristics, location,  and goals for the placement of every child who is (or within the  immediately preceding 12 months has been) in foster care.
Systemic factor 2: Case review system  Item 25: Provides a process that ensures that each child has a written case  plan to be developed jointly with the child’s parent(s) that includes the  required provisions  Item 26: Provides a process for the periodic review of the status of each  child, no less frequently than once every 6 months, either by a court or by  administrative review.
Item 27: Provides a process that ensures that each child in foster care  under the supervision of the state had a permanency hearing in a qualified  court or administrative body no later than 12 months from the date the  child entered foster care and no less frequently than every 12 months  thereafter.
Item 28: Provides a process for termination of parental rights proceedings  in accordance with the provisions of the Adoption and Safe Families Act.
Item 29: Provides a process for foster parents, pre-adoptive parents, and  relative caregivers of children in foster care to be notified of, and have an  opportunity to be heard in, any review or hearing held with respect to the  child.
Systemic factor 3: Quality assurance system  Item 30: The state has developed and implemented standards to ensure  that children in foster care are provided quality services that protect the  safety and health of children.
Item 31: The state is operating an identifiable quality assurance system  that is in place in the jurisdictions where the services included in the Child  and Family Services Plan are provided, evaluates the quality of services,  identifies strengths and needs of the service delivery system, provides  relevant reports, and evaluates program improvement measures  implemented.
Systemic factor 4: Training  Item 32: The state is operating a staff development and training program  that supports the goals and objectives in the Child and Family Services  Plan, addresses services provided under Titles IV-B and IV-E, and provides  initial training for all staff who deliver these services.
Item 33: The state provides for ongoing training for staff that addresses the  skills and knowledge base needed to carry out their duties with regard to  the services included in the Child and Family Services Plan.
Item 34: The state provides training for current or prospective foster  parents, adoptive parents, and staff of state-licensed or approved facilities  that care for children receiving foster care or adoption assistance under  Title IV-E that addresses the skills and knowledge base needed to carry  out their duties with regard to foster and adopted children.
Systemic factor 5: Service array  Item 35: The state has in place an array of services that assess the  strengths and needs of children and families and determine other service  needs, address the needs of families in addition to individual children in  order to create a safe home environment, enable children to remain safely  with their parents when reasonable, and help children in foster and  adoptive placements achieve permanency.
Item 36: The services in item 35 are accessible to families and children in  all political jurisdictions covered in the State’s Child and Family Services  Plan.
Item 37: The services in item 35 can be individualized to meet the unique  needs of children and families served by the agency.
Systemic factor 6: Agency responsiveness to the community  Item 38: In implementing the provisions of the Child and Family Services  Plan, the state engages in ongoing consultation with tribal representatives,  consumers, services providers, foster care providers, the juvenile court,  and other public and private child- and family-serving agencies and  includes the major concerns of these representatives in the goals and  objectives of the Child and Family Services Plan.
Item 39: The agency develops, in consultation with these representatives,  annual reports of progress and services delivered pursuant to the Child  and Family Services Plan.
Item 40: The state’s services under the Child and Family Services Plan are  coordinated with services or benefits of other federal or federally assisted  programs serving the same population.
Systemic factor 7: Foster and adoptive parent licensing, recruitment,  and retention  Item 41: The state has implemented standards for foster family homes and  child care institutions that are reasonably in accord with recommended  national standards.
Item 42: The standards are applied to all licensed or approved foster family  homes or child care institutions receiving Title IV-E or IV-B funds.
Item 43: The state complies with federal requirements for criminal  background clearances as related to licensing or approving foster care and  adoptive placements and has in place a case-planning process that  includes provisions for addressing the safety of foster care and adoptive  placements for children.
Item 44: The state has in place a process for ensuring the diligent  recruitment of potential foster and adoptive families that reflect the ethnic  and racial diversity of children in the state for whom foster and adoptive  homes are needed.
Item 45: The state has in place a process for the effective use of cross- jurisdictional resources to facilitate timely adoptive or permanent  placements for waiting children.
section_title: Appendix III: Dates on Which Site Visit States Reached CFSR Milestones
subsections: section_title: Total number of outcomes and systemic factors not in substantial conformity
section_title: Appendix IV: Comments from the Department of Health and Human Services
section_title: Appendix V: GAO Contacts and Acknowledgments
subsections: section_title: GAO Contacts
section_title: Staff Acknowledgments
paragraphs: In addition to those named above, Elizabeth Caplick and Catherine Roark  made key contributions to this report. Amy Buck, Karen Burke,   Jason Kelly, Stuart Kaufman, Luann Moy, and Jerome Sandau also  provided key technical assistance.
section_title: Related GAO Products
paragraphs: Child Welfare: Improved Federal Oversight Could Assist States in  Overcoming Key Challenges. GAO-04-418T. Washington, D.C.: January 28,  2004.
D.C. Family Court: Progress Has Been Made in Implementing Its  Transition. GAO-04-234. Washington, D.C.: January 6, 2004.
Child Welfare: States Face Challenges in Developing Information  Systems and Reporting Reliable Child Welfare Data. GAO-04-267T.  Washington, D.C.: November 19, 2003.
Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide  States Additional Information to Improve Services. GAO-03-956.  Washington, D.C.: September 12, 2003.
Child Welfare: Most States Are Developing Statewide Information  Systems, but the Reliability of Child Welfare Data Could be Improved.  GAO-03-809. Washington, D.C.: July 31, 2003.
D.C. Child and Family Services: Key Issues Affecting the Management of  Its Foster Care Cases. GAO-03-758T. Washington, D.C.: May 16, 2003.
Child Welfare and Juvenile Justice: Federal Agencies Could Play a  Stronger Role in Helping States Reduce the Number of Children Placed  Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.:  April 21, 2003.
Foster Care: States Focusing on Finding Permanent Homes for Children,  but Long-Standing Barriers Remain. GAO-03-626T. Washington, D.C.:  April 8, 2003.
Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare  Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March  31, 2003.
Foster Care: Recent Legislation Helps States Focus on Finding  Permanent Homes for Children, but Long-Standing Barriers Remain.  GAO-02-585. Washington, D.C.: June 28, 2002.
District of Columbia Child Welfare: Long-Term Challenges to Ensuring  Children’s Well- Being. GAO-01-191. Washington, D.C.: December 29, 2000.
Child Welfare: New Financing and Service Strategies Hold Promise, but  Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000.
Foster Care: States’ Early Experiences Implementing the Adoption and  Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999.
Foster Care: HHS Could Better Facilitate the Interjurisdictional  Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19,  1999.
Foster Care: Effectiveness of Independent Living Services Unknown.  GAO/HEHS-00-13. Washington, D.C.: November 10, 1999.
Foster Care: Kinship Care Quality and Permanency Issues.   GAO/HEHS-99-32. Washington, D.C.: May 6, 1999.
Juvenile Courts: Reforms Aim to Better Serve Maltreated  Children. GAO/HEHS-99-13. Washington, D.C.: January 11, 1999.
Child Welfare: Early Experiences Implementing a Managed Care  Approach. GAO/HEHS-99-8. Washington, D.C.: October 21, 1998.
Foster Care: Agencies Face Challenges Securing Stable Homes for  Children of Substance Abusers. GAO/HEHS-98-182. Washington, D.C.:  September 30, 1998.