Report to Congressional Requesters

United States General Accounting Office 

GAO

April 2004 CHILD AND FAMILY 

SERVICES REVIEWS 

Better Use of Data and 

Improved Guidance 

Could Enhance HHS’s 

Oversight of State 

Performance 

GAO-04-333 
www.gao.gov/cgi-bin/getrpt?GAO-04-333.

To view the full product, including the scope 

and methodology, click on the link above. 

For more information, contact Cornelia Ashby 

at (202) 512-8403 or AshbyC@gao.gov. 

Highlights of GAO-04-333, a report to 

Congressional Requesters 

April 2004

CHILD AND FAMILY SERVICES REVIEWS

Better Use of Data and Improved 

Guidance Could Enhance HHS's 

Oversight of State Performance 

ACF and many state officials perceive the CFSR as a valuable process and a 

substantial undertaking, but some data enhancements could improve its 

reliability. ACF staff in 8 of the 10 regions considered the CFSR a helpful tool

to improve outcomes for children. Further, 26 of 36 states responding to a 

relevant question in our survey commented that they generally or completely 

agreed with the results of the final CFSR report, even though none of the 41 

states with final CFSR reports released through 2003 has achieved 

substantial conformity on all 14 outcomes and systemic factors—see figure 

below. Additionally, both ACF and the states have dedicated substantial 

financial and staff resources to the process. Nevertheless, several state 

officials and child welfare experts we interviewed questioned the accuracy 

of the data used in the review process. While ACF officials contend that 

stakeholder interviews and case reviews complement the data profiles, many 

state officials and experts reported that additional data from the statewide 

assessment could bolster the evaluation of state performance. 

State Performance on the 14 CFSR Outcomes and Systemic Factors 

Program improvement planning is under way, but uncertainties have 

affected the development, funding, and implementation of state PIPs. 

Officials from 3 of the 5 states we visited said ACF’s PIP-related instructions 

were unclear, and at least 9 of the 25 states reporting on PIP implementation 

in our survey said that insufficient funding and staff were among the greatest 

challenges. While ACF has provided some guidance, ACF and state officials 

remain uncertain about PIP monitoring efforts and how ACF will apply 

financial penalties if states fail to achieve their stated PIP objectives. 

Since 2001, ACF’s focus has been almost exclusively on the CFSRs and 

regional staff report limitations in providing assistance to states in helping 

them to meet key federal goals. While staff from half of ACF’s regions told us 

they would like to provide more targeted assistance to states, and state 

officials in all 5 of the states we visited said that ACF’s existing technical 

assistance efforts could be improved, ACF officials acknowledged that 

regional staff might still be adjusting to the new way ACF oversees child 

welfare programs. 

In 2001, the Department of Health 

and Human Services’ (HHS) 

Administration for Children and 

Families (ACF) implemented the 

Child and Family Services Reviews 

(CFSR) to increase states’ 

accountability. The CFSR uses 

states’ data profiles and statewide 

assessments, as well as interviews 

and an on-site case review, to 

measure state performance on 14 

outcomes and systemic factors, 

including child well-being and the 

provision of caseworker training. 

The CFSR also requires progress 

on a program improvement plan 

(PIP); otherwise ACF may apply 

financial penalties. This report 

examines (1) ACF’s and the states’ 

experiences preparing for and 

conducting the statewide 

assessments and on-site reviews; 

(2) ACF’s and the states’ 

experiences developing, funding, 

and implementing items in PIPs; 

and (3) any additional efforts that 

ACF has taken beyond the CFSR to 

help ensure that all states meet 

federal goals related to children’s 

safety, permanency, and well-being. 

GAO recommends that the 

Secretary of HHS ensure that ACF 

uses the best available data to 

measure state performance, clarify 

PIP guidance, and help regional 

offices better integrate their 

oversight activities. In commenting 

on this report, HHS generally 

agreed with GAO’s findings and 

said that it is committed to 

continually monitoring and 

improving the CFSR process. 
Page i GAO-04-333 Child and Family Services Reviews 

Letter 1

Results in Brief 3

Background 5

The CFSR Is a Valuable yet Substantial Undertaking, but Data 

Enhancements Could Improve Its Reliability 8

Program Improvement Planning Under Way, but Uncertainties 

Challenge Plan Development, Implementation, and Monitoring 18

ACF’s Focus Rests Almost Exclusively on Implementing the CFSR 28

Conclusions 31

Recommendations for Executive Action 32

Agency Comments 33

Appendix I Objectives, Scope, and Methodology 37 

Objectives 37

Scope and Methodology 37

Appendix II List of Outcomes and Systemic Factors and Their 

Associated Items 41 

Outcome factors 41

Systemic Factors 43

Appendix III Dates on Which Site Visit States Reached CFSR 

Milestones 47 

Appendix IV Comments from the Department of Health and 

Human Services 48 

Appendix V GAO Contacts and Acknowledgments 52 

GAO Contacts 52

Staff Acknowledgments 52

Related GAO Products 53 

Contents 
Page ii GAO-04-333 Child and Family Services Reviews 

Tables 

Table 1: Items Most Frequently Assigned a Rating of Area Needing 

Improvement among the 41 States Reviewed in 2001, 2002, 

and 2003 11

Table 2: Number of States Including Each of the PIP Strategy 

Categories Used in This Study 20

Table 3: ACF Regulations and Guidance on Program Improvement 

Planning 22

Figures 

Figure 1: Four Phases of the CFSR Process 6

Figure 2: State Performance on the 14 CFSR Outcomes and 

Systemic Factors 10

Figure 3: Variation in the Start of States’ PIP Development 24

Figure 4: Average Time Lapse between States’ PIP Due Dates and 

ACF’s PIP Approval 25

Figure 5: Most Common Challenges Affecting States’ PIP 

Implementation 26

Abbreviations 

ACF Administration for Children and Families 

AFCARS Adoption and Foster Care Analysis and Reporting System 

ASFA Adoption and Safe Families Act 

CFSR Child and Family Services Review 

CIP court improvement program 

FTE full-time equivalent 

HHS Department of Health and Human Services 

NCANDS National Child Abuse and Neglect Data System 

PIP program improvement plan 

SACWIS Statewide Automated Child Welfare Information System 

This is a work of the U.S. government and is not subject to copyright protection in the 

United States. It may be reproduced and distributed in its entirety without further 

permission from GAO. However, because this work may contain copyrighted images or 

other material, permission from the copyright holder may be necessary if you wish to 

reproduce this material separately. 
Page 1 GAO-04-333 Child and Family Services Reviews 

April 20, 2004 

Congressional Requesters 

Recent media reports have called into question a number of states’ 

abilities to protect the more than 800,000 children estimated to spend 

some time in foster care each year. To help states investigate reports of 

child abuse and neglect, provide placements to children outside their 

homes, and deliver services to help keep families together, the federal 

government provides approximately $7 billion dollars to states annually. In 

addition, the federal Department of Health and Human Services (HHS) 

monitors states’ compliance with key federal goals, specified in part by the 

Adoption and Safe Families Act (ASFA) of 1997, to keep children safe and 

ensure their placement in stable and permanent homes. 

In 2001, HHS—through its Administration for Children and Families’ 

(ACF) Children’s Bureau—began implementing a new federal review 

system, known as the Child and Family Services Reviews (CFSR), to hold 

states accountable for improving child welfare outcomes. Unlike prior 

federal reviews—which determined states’ adherence to certain process 

measures—ACF designed the CFSRs as an outcome-oriented approach to 

assess children’s safety; their timely placement in permanent homes; and 

their mental, physical, and educational well-being. ACF also developed a 

number of factors to assess states’ performance in systemic areas, such as 

staff training and foster parent licensing.1

 According to ACF, this CFSR 

process combines statewide assessments, which the states complete using 

a profile of their recent child welfare data; on-site reviews of child welfare 

 1

The CFSR measures state performance on 45 performance items, which correspond to 

7 outcomes and 7 systemic factors. The outcomes relate to children’s safety, permanency, 

and well-being, and the systemic factors address state agency management and 

responsiveness to the community. Six national standards, as reported in the Adoption and 

Foster Care Analysis and Reporting System (AFCARS) and the National Child Abuse and 

Neglect Data System (NCANDS), apply to 5 of the 45 items. Three of these standards are 

based on the 75th percentile of all states’ performance—adoption; stability of foster care 

placements; and length of time to achieve reunification, guardianship, or permanent 

placement with relatives—because a higher incidence is desirable. However, the remaining 

three standards—recurrence of maltreatment, incidence of child abuse/neglect in foster 

care, and foster care re-entries—are based on the 25th percentile of state performance, 

because lower incidence is a desired outcome for these measures. 

United States General Accounting Office 

Washington, DC 20548 
Page 2 GAO-04-333 Child and Family Services Reviews 

case files; and interviews with stakeholders2 to provide both qualitative 

and quantitative information that helps describe and clarify complex child 

welfare issues. Further, states are required to develop and implement 

program improvement plans (PIP) to improve their child welfare practice 

and capacity when they are found to be deficient. Pursuant to CFSR 

regulations, ACF can withhold federal funds if states do not show 

adequate progress implementing their PIPs. 

To better understand the CFSR process, you asked us to examine 

(1) ACF’s and the states’ experiences preparing for and conducting the 

statewide assessments and on-site reviews; (2) ACF’s and the states’ 

experiences developing, funding, and implementing items in their PIPs; 

and (3) additional efforts, if any, that ACF has taken beyond the CFSR to 

help ensure that all states meet federal goals of safety, permanency, and 

well-being for children. 

To conduct our work, we surveyed all 50 states, the District of Columbia, 

and Puerto Rico regarding their interactions with ACF, the resources and 

staffing required to prepare for the review and develop and implement 

their PIPs, and any assistance ACF provided throughout the process. We 

achieved a 98 percent response rate from this survey, as Puerto Rico was 

the only non-respondent. To supplement this information, we also engaged 

in intensive post-survey follow-up phone calls with key states in each 

phase of the CFSR process. Further, we visited 5 states—California, 

Florida, New York, Oklahoma, and Wyoming—to obtain first-hand 

information on states’ experiences. We selected these states for diversity 

in their location, size, program administration, performance on the CFSR, 

and the timing of their review. We also examined all 31 approved PIPs 

available as of January 1, 2004, to categorize common strategies states 

developed to improve their performance. In addition, we conducted 

interviews with ACF’s senior officials and regional staff from all 10 

regions, ACF contractors, directors and staff from all 10 national resource 

centers, and key child welfare experts.3

 We conducted our work between 

 2

The term stakeholder refers to two groups: (1) agency stakeholders, such as judges or 

advocates, whose responsibilities are closely related to the work of the child welfare 

agency and who can comment on the agency’s overall performance on outcomes and 

systemic factors, and (2) case-specific stakeholders, such as parents, caseworkers, 

children, or others who are interviewed to provide first-hand information that supplements 

reviewers’ assessment of paper or electronic case files. 

3

Each of the 10 resource centers operates under a 5-year cooperative agreement with HHS 

to help states implement federal child welfare legislation and provide training and technical 

assistance to states. 
Page 3 GAO-04-333 Child and Family Services Reviews 

May 2003 and February 2004 in accordance with generally accepted 

government auditing standards. A more detailed discussion of our scope 

and methodology appears in appendix I. 

ACF and many state officials perceive the CFSR as a valuable process—

highlighting many areas needing improvement—and a substantial 

undertaking, but some state officials and child welfare experts told us that 

data enhancements could improve its reliability. ACF staff in 8 of the 

10 regions considered the CFSR a helpful tool to improve outcomes for 

children, and officials from 8 regions noted that the CFSRs were more 

intensive and comprehensive than other types of reviews they had 

conducted in the past. Further, 26 of the 36 states responding to a relevant 

question in our survey commented that they generally or completely 

agreed with the results of the final CFSR report, even though none of the 

41 states with final CFSR reports released through 2003 has achieved 

substantial conformity on all 14 outcomes and systemic factors. In 

addition, both ACF and the states have dedicated substantial financial and 

staff resources to the process. For example, ACF committed $6.6 million 

to hire contractors for review assistance, and based on survey responses 

from the 45 states that reported on staff resources, an average of 47 fulltime staff equivalents per state participated in the statewide assessment 

phase. Nevertheless, several state officials and child welfare experts we 

interviewed questioned the accuracy of the data used to compile state 

profiles and establish the national standards. While ACF officials in the 

central office contend that stakeholder interviews and case reviews 

complement the data profiles, several state officials and experts reported 

that additional data from the statewide assessment could bolster the 

evaluation of state performance. 

The program improvement planning process for the CFSR, which all states 

with final reports have been required to undertake, has been characterized 

by many uncertainties, such as those related to federal guidance and 

monitoring and the availability of state resources to implement the plans. 

Despite such uncertainties, states have included a variety of strategies in 

their PIPs to address weaknesses identified in the review. Such strategies 

include involving foster and birth parents in identifying needed services 

and developing tools to assess the risk of harm to children. To assist states 

with PIP development, ACF issued regulations and guidance to states for 

developing the plans, but state responses to this assistance have been 

mixed. For example, state officials attributed a lengthy PIP approval 

process, in part, to unclear regulations and insufficient guidance. Our 

analysis of the time lapse between the date states must submit a PIP and 

Results in Brief 
Page 4 GAO-04-333 Child and Family Services Reviews 

ACF’s PIP approval date shows a range from 45 to 349 business days. 

Further, 6 of the 21 states reporting on the PIP approval process said that 

ACF did not clearly describe the process to them, and officials in 3 of the 5 

states we visited said they were unclear as to ACF’s expectations 

regarding the format and content of the plans. In response to a question on 

PIP implementation challenges, at least 9 of 25 states reported that 

insufficient funding, staff, and time, as well as high caseloads, were the 

most challenging factors. For example, one survey respondent reported 

that unless the state receives additional funding, it would need to lay off 

social workers and attorneys, making it difficult to implement some of its 

improvement strategies. ACF officials noted that they have clarified PIP 

expectations and increased technical assistance over time, but they 

acknowledge that they are still unsure how best to monitor state PIP 

implementation and have not yet determined when and how to apply the 

prescribed financial penalties for states’ failures to improve outcomes. 

Since 2001, ACF’s focus has been almost exclusively on implementing the 

CFSRs, and regional staff report limitations in providing assistance to 

states in helping them to meet key federal goals. For example, ACF 

provides internal training for its staff and technical assistance to states, 

but all 18 courses provided to regional staff have dealt with the various 

phases of the review. Staff in 2 regional offices told us that they have not 

been able to counsel states on such topics as contracting out for child 

welfare services and maximizing the receipt of federal child welfare funds 

because they have lacked the applicable training. In addition, while ACF 

organizes biennial conferences for state and federal child welfare officials, 

staff from 5 regions told us that they wanted more substantive interaction 

with their ACF colleagues to increase their overall child welfare expertise. 

While regional officials conduct site visits to states, CFSR on-site reviews 

and PIP-related assistance account for the majority of regions’ time and 

travel budgets, and ACF staff from 5 regions said that more frequent visits 

with state personnel—outside of the CFSR process—would allow them to 

better understand states’ programs and cultivate relationships with state 

officials. Further, state officials in all 5 of the states we visited said that 

ACF’s technical assistance needed improvement, acknowledging, in some 

cases, that regional office staff were stretched thin by CFSR demands and, 

in other cases, that the assistance from resource center staff lacked focus 

or practicality. Although ACF officials told us the CFSR has become the 

agency’s primary mechanism for monitoring states and facilitating 

program improvement, they acknowledged that regional staff might still be 

adjusting to the new way ACF oversees child welfare programs. 
Page 5 GAO-04-333 Child and Family Services Reviews 

Given the emphasis that ACF has placed on the CFSRs—and the resources 

and time states have invested to prepare for the reviews and address their 

deficiencies through program improvement plans—we are recommending 

that the Secretary of HHS ensure that ACF uses the best available data to 

measure state performance. We are also recommending that the Secretary 

clarify PIP guidance and provide guidance to regional officials on how to 

better integrate their many oversight responsibilities. In commenting on a 

draft of this report, HHS generally agreed with our findings and 

recommendations. HHS acknowledged that the CFSR is a new process 

that continues to evolve, and also noted several steps it has taken to 

address the data quality concerns we raise in this report. We believe that 

our findings fully address these initial actions, as well as the substantial 

resources the agency has already dedicated to the review process. 

However, to improve its oversight of state performance, our 

recommendations are meant to encourage HHS to take additional actions 

to improve its use of data in conducting these reviews and enhance PIPrelated guidance and regional officials’ understanding of how to 

incorporate the CFSR process into their overall improvement and 

oversight efforts. 

ACF’s Children’s Bureau administers and oversees federal funding to 

states for child welfare services under Titles IV-B and IV-E of the Social 

Security Act, and states and counties provide these child welfare services, 

either directly or indirectly through contracts with private agencies.4

Among other activities, ACF staff are responsible for developing 

appropriate policies and procedures for states to follow to obtain and use 

federal child welfare funds and conduct administrative reviews of states’ 

case files to ensure that children served by the state meet statutory 

eligibility requirements. 

In 2001, ACF launched a new outcome-oriented process, known as the 

Child and Family Services Reviews (CFSR), to improve its existing 

monitoring efforts, which had once been criticized for focusing exclusively 

on states’ compliance with regulations rather than on their performance 

over a full range of child welfare services. Passage of the 1997 Adoption 

 4

Title IV-B of the Social Security Act, consisting of two subparts, is the primary source of 

federal funding for services to help families address problems that lead to child abuse and 

neglect and to prevent the unnecessary separation of children from their families. Funding 

under Title IV-E of the Social Security Act is used primarily to pay for the room and board 

of children in foster care. 

Background 
Page 6 GAO-04-333 Child and Family Services Reviews 

and Safe Families Act (ASFA) helped spur the creation of the CFSR by 

emphasizing the outcomes of safety, permanency, and well-being for 

children. Subsequently, ACF consulted with state officials, child welfare 

experts, and other interested parties, and conducted pilot CFSR reviews in 

14 states. In January 2000, ACF released a notice of proposed rule making 

and published final CFSR regulations. In March 2001, ACF conducted the 

first of its state reviews in Delaware. By March of 2004, ACF had 

completed an on-site review in all 50 states, the District of Columbia, and 

Puerto Rico. Although ACF plans to initiate a second round of reviews, an 

official start date for this process has not yet been determined. 

As figure 1 indicates, the CFSR is a four-phase process that involves state 

staff as well as central and regional ACF officials. This process begins with 

a statewide assessment in the first phase. 

Figure 1: Four Phases of the CFSR Process 

The assessment of state performance continues in the second phase, most 

commonly known as the on-site review, when ACF sends a team of 

reviewers to three sites in the state for one week to assess state 

performance. A list of all the outcomes and systemic factors and their 

associated items appears in appendix II. In assessing performance, ACF 

relies, in part, on its own data systems, known as the National Child Abuse 

and Neglect Data System (NCANDS) and the Adoption and Foster Care 

Analysis and Reporting System (AFCARS), which were designed prior to 

Phase 1: Phase 2: (6-7 months

later) 

Phase 3: (within 90 days

after report release)

Phase 4: (2 years after 

PIP approval)

State develops a

self-assessment.

ACF conducts its

weeklong on-site review.

State develops program 

improvement plan (PIP).

State completes PIP 

implementation.

Result Result Result

ACF and state 

use statewide 

assessment 

information to 

select 2 

locations for 

on-site review. 

(3rd location is 

state's largest 

metropolitan 

area).

ACF's 

contractor 

prepares a 

final report 

based in part 

on the findings 

of the on-site 

review and 

releases this 

report to the 

state.

ACF approves 

PIP and state 

submits 

quarterly 

reports to ACF 

for monitoring.

Result

ACF assesses 

the state's 

achievements 

against 

negotiated 

benchmarks 

and 

determines 

whether or not 

financial 

penalties 

apply.

1

2

1 1

2

3

Source: GAO analysis.

State and ACF 

review and correct 

statewide data profile.

State conducts focus 

groups and surveys, 

engages 

stakeholders and 

staff.

ACF draws a sample 

of cases and state 

prepares files from 

50 cases.

State and ACF select 

and train review 

team.

Review team 

conducts file review 

and interviews 

stakeholders.

State and ACF 

negotiate 

benchmarks and 

action steps.
Page 7 GAO-04-333 Child and Family Services Reviews 

CFSR implementation to capture, report, and analyze the child welfare 

information collected by the states.5

 Today, these systems provide the 

national data necessary for ACF to calculate national standards for key 

CFSR performance items with which all states’ data will be compared. 

After the on-site review, ACF prepares a final report for the state—

identifying areas needing improvement, as well as the outcomes and 

systemic factors for which the state was determined not to be in 

substantial conformity—and provides the state with an estimated financial 

penalty.6

 As a result, the state must develop a 2-year PIP with action steps 

to address its noted deficiencies and performance benchmarks to measure 

progress. Once ACF approves the PIP, states are required to submit 

quarterly progress reports, which ACF uses to monitor improvement. 

Pursuant to CFSR regulations, federal child welfare funds can be withheld 

if states do not show adequate progress as a result of PIP implementation, 

but these penalties are suspended during the 2-year implementation term. 

As of January 2004, no financial penalties had been applied, but according 

to data on the 41 states for which final CFSR reports have been released 

through December 2003, potential penalties range from $91,492 for North 

Dakota to $18,244,430 for California. 

ACF staff in HHS’s 10 regional offices provide technical assistance to 

states through all phases of the CFSR process, and they are also 

responsible for reviewing state planning documents required by Title IV-B, 

assisting with state data system reviews, and assessing states’ use of IV-E 

funds. In addition to these efforts, ACF has established cooperative 

agreements with 10 national resource centers to help states implement 

federal legislation intended to ensure the safety, permanency, and well-

 5

States began voluntarily reporting to NCANDS in 1990, and in 1995 started reporting to 

AFCARS on the demographic characteristics of adoptive and foster children and their 

parents, as well as foster children’s type of placement and permanency goals. We recently 

issued a report on states’ child welfare information systems and the reliability of child 

welfare data. See Child Welfare: Most States Are Developing Statewide Information 

Systems, but the Reliability of Child Welfare Data Could Be Improved, GAO-03-809

(Washington, D.C.: July 31, 2003). 

6

States achieve substantial conformity on outcomes when at least 90 percent of applicable 

cases are substantially achieved; stakeholder interviews confirm that state plan and other 

program requirements are in place and functioning as described in the applicable 

regulations or statute; and performance on items with national standards meets the 

applicable threshold. The formula for calculating penalties is based in part on each state’s 

allocation of federal child welfare funds from Titles IV-B and IV-E and the number of 

outcomes and systemic factors for which substantial conformity has not been achieved. 
Page 8 GAO-04-333 Child and Family Services Reviews 

being of children and families. ACF sets the resource centers’ areas of 

focus, and although each center has a different area of expertise, such as 

organizational improvement or information technology, all of them 

conduct needs assessments, sponsor national conference calls with states, 

collaborate with other resource centers and agencies, and provide on-site 

training and technical assistance to states.7

Members of the 108th Congress introduced a proposal to provide federal 

incentive payments directly to states that demonstrate significant 

improvements to their child welfare systems. At the time of publication, 

the House of Representatives was considering H.R. 1534, the Child 

Protective Services Improvement Act, which contains provisions to award 

grants to states with approved PIPs and additional bonuses to states that 

have made considerable progress in achieving their PIP goals for the 

previous year. 

ACF and many state officials perceive the CFSR as a valuable process—

highlighting many areas needing improvement—and a substantial 

undertaking, but some state officials and child welfare experts told us that 

data enhancements could improve its reliability. ACF staff in 8 of the 

10 regions considered the CFSR a helpful tool to improve outcomes for 

children. Further, 26 of the 36 states responding to a relevant question in 

our survey commented that they generally or completely agreed with the 

results of the final CFSR report, even though none of the 41 states with 

final CFSR reports released through 2003 has achieved substantial 

conformity on all 14 outcomes and systemic factors. In addition, both ACF 

and the states have dedicated substantial financial and staff resources to 

the process. However, several state officials and child welfare experts we 

interviewed questioned the accuracy of the data used to compile state 

profiles and establish the national standards. While ACF officials in the 

central office contend that stakeholder interviews and case reviews 

compliment the data profiles, many state officials and experts reported 

that additional data from the statewide assessment could bolster the 

evaluation of state performance. 

 7

Resource centers may provide unlimited assistance to states by phone or e-mail, but onsite training and technical assistance are restricted to 10 days each year. States must first 

obtain regional office approval before on-site training and technical assistance can occur. 

ACF officials in both the regions and in headquarters indicated that they have been flexible 

with extensions beyond the 10-day cap. 

The CFSR Is a 

Valuable yet 

Substantial 

Undertaking, but Data 

Enhancements Could 

Improve Its Reliability 
Page 9 GAO-04-333 Child and Family Services Reviews 

ACF and state officials support the objectives of the review, especially in 

focusing on children’s outcomes and strengthening relationships with 

stakeholders, and told us they perceive the process as valuable. ACF staff 

in 8 of the 10 regions considered the CFSR a helpful tool to improve 

outcomes for children. Also, ACF officials from 8 regional offices noted 

that the CFSRs were more intensive and more comprehensive than the 

other types of reviews they had conducted in the past, creating a valuable 

tool for regional officials to monitor states’ performance. In addition, state 

officials from every state we visited told us that the CFSR process helped 

to improve collaboration with community stakeholders. For example, a 

court official in New York said that the CFSR acted as a catalyst for 

improved relations between the state agency and the courts, and he 

believes that this has contributed to more timely child abuse and neglect 

hearings. Additionally, a state official in Florida said that the CFSR 

stimulated discussions among agency staff about measuring and improving 

outcomes, particularly the data needed to examine outcomes and the 

resources needed to improve them. Furthermore, state staff from 4 of the 

5 states we visited told us the CFSR led to increased public and legislative 

attention to critical issues in child welfare. For example, caseworkers in 

Wyoming told us that without the CFSR they doubted whether their state 

agency’s administration would have focused on needed reforms. They 

added that the agency used the CFSR findings to request legislative 

support for the hiring of additional caseworkers. 

In addition to affirming the value associated with improved stakeholder 

relations, the ACF officials we talked to and many state officials reported 

that the process has been helpful in highlighting the outcomes and 

systemic factors, as well as other key performance items that need 

improvement. According to our survey, 26 of the 36 states that commented 

on the findings of the final CFSR report indicated that they generally or 

completely agreed with the findings, even though performance across the 

states was low in certain key outcomes and performance items. For 

example, not one of the 41 states with final reports released through 2003 

was found to be in substantial conformity with either the outcome 

measure that assesses the permanency and stability of children’s living 

situations or with the outcome measure that assesses whether states had 

enhanced families’ capacity to provide for their children’s needs. 

Moreover, across all 14 outcomes and systemic factors, state performance 

ranged from achieving substantial conformity on as few as 2 outcomes and 

The CFSR Is a Valuable 

Process for ACF and the 

States 
Page 10 GAO-04-333 Child and Family Services Reviews 

systemic factors to as many as 9.8 As figure 2 illustrates, the majority of 

states were determined to be in substantial conformity with half or fewer 

of the 14 outcomes and systemic factors assessed. 

Figure 2: State Performance on the 14 CFSR Outcomes and Systemic Factors 

States’ performance on the outcomes related to safety, permanency, and 

well-being—as well as the systemic factors—is determined by their 

performance on an array of items, such as establishing permanency goals, 

ensuring worker visits with parents and children, and providing accessible 

services to families. The CFSR showed that many states need 

improvement in the same areas, and table 1 illustrates the 10 items most 

 8

California and Puerto Rico were determined to be in substantial conformity on 2 outcomes 

and systemic factors, while North Dakota achieved substantial conformity on 9. 
Page 11 GAO-04-333 Child and Family Services Reviews 

frequently rated as needing improvement across all 41 states reviewed 

through 2003. 

Table 1: Items Most Frequently Assigned a Rating of Area Needing Improvement 

among the 41 States Reviewed in 2001, 2002, and 2003 

Item 

Number of states 

assigned a rating of area 

needing improvement 

Assessing the needs and services of child, parents, and 

foster parents 40

Assessing mental health of child 37

Establishing the most appropriate permanency goal for the 

child 36

Demonstrating efforts to involve child and family in caseplanning activities 36

Ensuring stability of foster care placements 35

Achieving a child’s goal of adoption 35

Providing a process that ensures that each child has a 

written case plan to be developed jointly with the child’s 

parent(s) that includes the required provisions 35

Ensuring workers conduct face-to-face visits with parent(s) 34

Providing services that are accessible to families and 

children in all political jurisdictions covered in the state’s 

Child and Family Services Plan 31

Ensuring workers conduct face-to-face visits with child 31

Source: GAO analysis of the 41 states’ final CFSR reports released through December 31, 2003. 

Given the value that ACF and the states have assigned to the CFSR 

process, both have spent substantial financial resources and staff time to 

prepare for and implement the reviews. In fiscal years 2001-2003, when 

most reviews were scheduled, ACF budgeted an additional $300,000 

annually for CFSR-related travel. In fiscal year 2004, when fewer reviews 

were scheduled, ACF budgeted about $225,000. To further enhance its 

capacity to conduct the reviews, and to obtain additional logistical and 

technical assistance, ACF spent approximately $6.6 million annually to 

hire contractors. Specifically, ACF has let three contracts to assist with 

CFSR-related activities, including training reviewers to conduct the on-site 

reviews, tracking final reports and PIP documents, and, as of 2002, writing 

the CFSR final reports. Additionally, ACF hired 22 new staff to build 

central and regional office capacity and dedicated 4 full-time staff and 

2 state government staff temporarily on assignment with ACF to assist 

with the CFSR process. To build a core group of staff with CFSR expertise, 

ACF and the States Report 

That Reviews Have Been a 

Substantial Undertaking 
Page 12 GAO-04-333 Child and Family Services Reviews 

ACF created the National Review Team, composed of central and regional 

office staff with additional training in and experience with the review 

process. In addition, to provide more technical assistance to the states, 

ACF reordered the priorities of the national resource centers to focus their 

efforts primarily on helping states with the review process. 

Like ACF, states also spent financial resources on the review. While some 

states did not track CFSR expenses—such as staff salaries, training, or 

administrative costs—of the 25 states that reported such information in 

our survey, the median expense to date was $60,550, although states 

reported spending as little as $1,092 and as much as $1,000,000 on the 

CFSR process.9

 For example, California officials we visited told us that 

they gave each of the state’s three review sites $50,000 to cover the salary 

of one coordinator to manage the logistics of the on-site review. Although 

ACF officials told us that states can use Title IV-E funds to pay for some of 

their CFSR expenses, only one state official addressed the use of these 

funds in our survey, commenting that it was not until after the on-site 

review occurred that the state learned these funds could have been used 

to offset states’ expenses.10 Furthermore, 18 of 48 states responding to a 

relevant question in our survey commented that insufficient funding was a 

challenge—to a great or very great extent—in preparing for the statewide 

assessment, and 11 of 40 states responded that insufficient funding was 

similarly challenging in preparing for the on-site review. Officials in other 

states reported that because available financial resources were 

insufficient, they obtained non-financial, in-kind donations to cover 

expenses associated with the on-site review. For example, a local site 

coordinator in Oklahoma informed us that while the state budgeted $7 per 

reviewer each day of the on-site review for food, refreshments, and other 

expenses, she needed to utilize a variety of resources to supplement the 

state’s budget with donations of supplies and food from local companies 

and agency staff. 

 9

These values are state-reported and reflect officials’ estimates of costs associated with all 

CFSR-related activities except those incurred during PIP implementation. In reporting on 

their expenses, states were instructed to include the value of training, travel, infrastructure, 

technology, food, administrative supplies, and any other expenses associated with the 

CFSR process. States were also asked to provide supporting documentation for this 

particular question, but most states were unable to provide documentation. Many states 

reported that they did not track CFSR-related expenses. The 25 states that did provide 

estimates were in different phases of the CFSR. 

10ACF provided preliminary data, based on an inquiry of its regional offices, showing that 

25 states have charged some CFSR-related costs to Title IV-E. 
Page 13 GAO-04-333 Child and Family Services Reviews 

States reported that they also dedicated staff time to prepare for the 

statewide assessment and to conduct the on-site review, which sometimes 

had a negative impact on some staffs’ regular duties. According to our 

survey, 45 states reported dedicating up to 200 full-time staff equivalents 

(FTEs), with an average of 47 FTEs, to the statewide assessment process.11

Similarly, 42 states responded that they dedicated between 3 and 

130 FTEs, with an average of 45 FTEs, to the on-site review process. To 

prepare for their own reviews, officials in all 5 states we visited told us 

that they sent staff for a week to participate as reviewers in other states. 

Additionally, local site coordinators in 4 of the 5 states we visited reported 

that planning for the CFSR on-site review was a full-time job involving 

multiple staff over a period of months. An official in Florida also told us 

that the extensive preparation for the CFSR dominated the work of the 

quality assurance unit, which was about 20 people at that time. In addition 

to preparing for the review, staff in all 5 of the states we visited, who 

served as reviewers in their own states, reported that to meet their 

responsibility as case reviewers they worked more than 12-hour days 

during the week of the on-site review. For some caseworkers, dedicating 

time to the CFSR meant that they were unable or limited in their ability to 

manage their typical workload. For example, Wyoming caseworkers 

whose case files were selected for the on-site review told us that they 

needed to be available to answer reviewers’ questions all day every day 

during the on-site review, which they said prevented them from 

conducting necessary child abuse investigations or home visits. 

Child welfare-related stakeholders—such as judges, lawyers, and foster 

parents—also contributed time to the CFSR, but some states found it took 

additional staff resources to initiate and maintain stakeholder involvement 

over time. According to our survey, 46 states reported that an average of 

about 277 stakeholders were involved with the statewide assessment, and 

42 states responded that an average of about 126 stakeholders participated 

in the on-site review.12 However, state officials told us that it was difficult 

 11The number of FTEs participating in each phase of the CFSR is state-reported. While 

states were not given specific instructions for how to calculate FTEs, they were asked to 

report only on the phases of the CFSR that they had started or completed. Therefore, 

states’ responses varied depending on the phase of the CFSR process they were in and the 

methods they used to calculate FTEs. 

12These values are state-reported and represent states’ estimates regarding the number of 

stakeholders that participated in each phase of the CFSR process. In answering this 

question, states were provided with a list of stakeholders, such as advocates and court 

officials, to guide their estimates, but they were not asked to indicate the duration of 

stakeholder involvement during each phase. 
Page 14 GAO-04-333 Child and Family Services Reviews 

to recruit and maintain stakeholder involvement because long time lapses 

sometimes occurred between CFSR phases. For example, more than a 

year can elapse between completion of the statewide assessment and the 

initiation of PIP development—both key points at which stakeholder 

participation is critical. Nonetheless, all 5 of the states we visited tried to 

counter this obstacle by conducting phone and in-person interviews, 

holding focus groups, or giving presentations in local communities to 

inform and recruit stakeholders to be involved in the CFSR process. As a 

result, some stakeholders were involved throughout the process. For 

example, a tribal representative in Oklahoma assisted with the 

development of the statewide assessment, was interviewed during the onsite review, and provided guidance on the needs of Indian children during 

PIP development. 

State officials in all 5 states, as well as child welfare experts, reported on 

several data improvements that could enhance the reliability of CFSR 

findings. In particular, they highlighted inaccuracies with the AFCARS and 

NCANDS data that are used for establishing the national standards and 

creating the statewide data profiles, which are then used to determine if 

states are in substantial conformity. These concerns echoed the findings of 

a prior GAO study on the reliability of these data sources, which found 

that states are concerned that the national standards used in the CFSR are 

based on unreliable information and should not be used as a basis for 

comparison and potential financial penalty.13 Several of the state officials 

we visited and surveyed also questioned the reliability of data given the 

variation in states’ data-reporting practice, which they believe may 

ultimately affect the validity of the measures and may place some states at 

a disadvantage. Furthermore, many states needed to resubmit their 

statewide data after finding errors in the data profiles ACF would have 

used to measure compliance with the national standards.14 According to 

our national survey, of the 37 states that reported on resubmitting data for 

the statewide data profile, 23 needed to resubmit their statewide data at 

least once, with 1 state needing to resubmit as many as five times to 

accurately reflect revised data. Four states reported in our survey that they 

did not resubmit their data profiles because they did not know they had 

 13See GAO-03-809. 

14ACF provides states with their statewide data about 6 months prior to the on-site review, 

during which time states are allowed to make corrections to the data and resubmit the 

updated data so it can be used when determining state conformity with CFSR measures. 

States and Child Welfare 

Experts Report That 

Several Data 

Improvements Could 

Enhance CFSR Reliability 
Page 15 GAO-04-333 Child and Family Services Reviews 

this option or they did not have enough time to resubmit before the 

review. 

In addition to expressing these data concerns, child welfare experts as 

well as officials in all of the states we visited commented that existing 

practices that benefit children might conflict with actions needed to attain 

the national standards. Specifically, one child welfare expert noted that if 

a state focused on preventing child abuse and neglect while the child was 

still living at home, and only removed the most at-risk children, the state 

may perform poorly on the CFSR reunification measure—assessing the 

timeliness of children’s return to their homes from foster care 

arrangements—because the children they did remove would be the 

hardest to place or most difficult to reunite. Additionally, officials in New 

York said that they recently implemented an initiative to facilitate 

adoptions. Because these efforts focus on the backlog of children who 

have been in foster care for several years, New York officials predict that 

their performance on the national standard for adoption will be lower 

since many of the children in the initiatives have already been in foster 

care for more than 2 years. 

These same state officials and experts also commented that they believe 

the on-site review case sample of 50 cases is too small to provide an 

accurate picture of statewide performance, although ACF officials stated 

that the case sampling is supplemented with additional information. Of the 

40 states that commented in our survey on the adequacy of the case 

sample size for the on-site review, 17 states reported that 50 cases were 

very or generally inadequate to represent their caseload.15 Oklahoma 

officials we visited also commented that they felt the case sample size was 

too small, especially since they annually assess more than 800 of their own 

cases—using a procedure that models the federal CFSR—and obtain 

higher performance results than the state received on its CFSR. 

Furthermore, because not every case in the states’ sample is applicable to 

each item measured in the on-site review, we found that sometimes as few 

as one or two cases were being used to evaluate states’ performance on an 

item. Specifically, an ACF contractor said that several of the CFSR items 

measuring children’s permanency and stability in their living conditions 

 15According to our calculations—which assumed that the attribute of interest occurred in 

about 50 percent of the cases—a sample size of 50 would produce percentage estimates 

with a 95 percent margin of error of approximately plus or minus 14 percentage points. 

This level of variability is a limitation when attempting to interpret estimates based on this 

sample size. 
Page 16 GAO-04-333 Child and Family Services Reviews 

commonly have very few applicable cases. For example, Wyoming had 

only two on-site review cases applicable for the item measuring the length 

of time to achieve a permanency goal of adoption, but for one of these 

cases, reviewers determined that appropriate and timely efforts had not 

been taken to achieve finalized adoptions within 24 months, resulting in 

the item being assigned a rating of area needing improvement.16 While ACF 

officials acknowledged the insufficiency of the sample size,17 they contend 

that the case sampling is augmented by stakeholder interviews for all 

items and applicable statewide data for the five CFSR items with 

corresponding national standards, therefore providing sufficient evidence 

for determining states’ conformity. 

All of the states we visited experienced discrepant findings between the 

aggregate data from the statewide assessment and the information 

obtained from the on-site review, which complicated ACF’s determination 

of states’ performance. Each of the 5 states we visited had at least one, and 

sometimes as many as three, discrepancies between its performance on 

the national standards and the results of the case review. We also found 

that in these 5 states, ACF had assigned an overall rating of area needing 

improvement for 10 of the 11 instances where discrepancies occurred. 

ACF officials acknowledged the challenge of resolving data discrepancies, 

noting that such complications can delay the release of the final report and 

increase or decrease the number of items that states must address in their 

PIPs. While states have the opportunity to resolve discrepancies by 

submitting additional information explaining the discrepancy or by 

requesting an additional case review, only one state to date has decided to 

pursue the additional case review.18 In addition, among the states we 

 16Because one of the two cases applicable to the adoption measure was assigned a rating of 

area needing improvement, 50 percent of the cases for this item were assigned a rating of 

area needing improvement. As a result, the item was given an overall rating of area needing 

improvement since both cases would have needed to be assigned a rating of strength for 

this item to meet the 85 percent threshold necessary to assign an overall rating of strength. 

17An ACF statistician also confirmed that the CFSR sample is too small to generalize to the 

states’ populations and that the three sites, from which cases are selected, also are not 

representative. 

18Virginia requested an additional case review to resolve a discrepancy between the 

statewide data and on-site review findings for the item measuring the state’s performance 

on foster care re-entries. According to an ACF regional official, the state met the national 

standard for this item but the case review findings showed the state did not meet the 

threshold for this measure. At the time of publication, ACF and the state were still 

finalizing plans to conduct the additional case review, and until the review is completed, 

the state cannot receive its final report. 
Page 17 GAO-04-333 Child and Family Services Reviews 

visited, for example, one state acknowledged that it has not opted to 

pursue the supplemental case review because doing so would place 

additional strain on its already limited resources. 

Several state officials and experts told us that additional data from the 

statewide assessments—or other data sources compiled by the states—

could bolster the evaluation of states’ performance, but they found this 

information to be missing or insufficiently used in the final reports. 

According to our survey, of the 34 states that commented on the adequacy 

of the final report’s inclusion of the statewide assessment, 10 states 

reported that too little emphasis was placed on the statewide assessment.19

Specifically, 1 state reported that the final report would have presented a 

more balanced picture of the state’s child welfare system if the statewide 

assessment were used as a basis to compare and clarify the on-site review 

findings. Further, child welfare experts and state officials from California 

and New York—who are using alternative data sources to AFCARS and 

NCANDS, such as longitudinal data that track children’s placements over 

time—told us that the inclusion of this more detailed information would 

provide a more accurate picture of states’ performance nationwide. North 

Carolina officials also reported in our survey that they tried to submit 

additional longitudinal data—which they use internally to conduct 

statewide evaluations of performance—in their statewide assessment, but 

HHS would not accept the alternative data for use in evaluating the state’s 

outcomes. An HHS official told us that alternative data are used only to 

assess state performance in situations where a state does not have 

NCANDS data, since states are not mandated to have these systems. 

Given their concerns with the data used in the review process, state 

officials in 4 of the 5 states believed that the threshold for achieving 

substantial conformity was difficult to achieve. One state official we 

visited acknowledged that she believed child welfare agencies should be 

pursuing high standards to improve performance, but she questioned the 

level at which ACF established the thresholds. While an ACF official told 

us that different thresholds for the national standards had been 

considered, ACF policy makers ultimately concluded that a more rigorous 

threshold would be used. ACF officials recognize that they have set a high 

standard. However, they believe it is attainable and supportive of their 

overall approach to move states to the standard through continuous 

improvement. 

 19The remaining 24 respondents reported that the emphasis was “just about right.” 
Page 18 GAO-04-333 Child and Family Services Reviews 

In preparation for the next round of CFSRs, ACF officials have formed a 

Consultation Work Group of ACF staff, child welfare administrators, data 

experts, and researchers who will propose recommendations on the CFSR 

measures and processes. The group began its meetings prior to our 

publication, but no proposals were available. 

Forty-one states are engaged in program improvement planning, but many 

uncertainties, such as those related to federal guidance and monitoring 

and the availability of state resources, have affected the development, 

implementation, and funding of the PIPs. State PIPs include strategies 

such as revising or developing policies, training caseworkers, and 

engaging stakeholders, and ACF has issued regulations and guidance to 

help states develop and implement their plans. Nevertheless, states 

reported uncertainty about how to develop their PIPs and commented on 

the challenges they faced during implementation. For example, officials 

from 2 of the states we visited told us that ACF had rejected their PIPs 

before final approval, even though these officials said that the plans were 

similar in the level of detail included in previously approved PIPs that 

regional officials had provided. Further, at least 9 states responding to our 

survey indicated that insufficient time, funding, and staff, as well as high 

caseloads, were the greatest challenges to PIP implementation. As states 

progress in PIP implementation, some ACF officials expressed a need for 

more guidance on how to monitor state accomplishments, and both ACF 

and state officials were uncertain about how the estimated financial 

penalties would be applied if states fail to achieve the goals described in 

their plans. 

State plans include a variety of strategies to address weaknesses identified 

in the CFSR review process. However, because most states had not 

completed PIP implementation by the time of our analysis, the extent to 

which states have improved outcomes for children has not been 

determined.20 While state PIPs varied in their detail, design, and scope, 

according to our analysis of 31 available PIPs, these state plans have 

focused to some extent on revising or developing policies; reviewing and 

reporting on agency performance; improving information systems; and 

 20At the time of our analysis, only Delaware and North Carolina had completed the 2-year 

term of their PIPs, and to date, ACF is still analyzing the states’ progress and has not 

determined if there has been overall improvement or if it will apply financial penalties. 

Program 

Improvement 

Planning Under Way, 

but Uncertainties 

Challenge Plan 

Development, 

Implementation, and 

Monitoring 

State Plans Include a 

Variety of Strategies to 

Address Identified 

Weaknesses 
Page 19 GAO-04-333 Child and Family Services Reviews 

engaging stakeholders such as courts, advocates, foster parents, private 

providers, or sister agencies in the public sector.21 Table 2 shows the 

number of states that included each of the six categories and 

subcategories of strategies we developed for the purposes of this study, 

and appendix I details our methodology for this analysis. 

 21Although 41 states were developing or implementing PIPs at the time of publication, we 

reviewed the 31 available PIPs that ACF had approved as of January 1, 2004. 
Page 20 GAO-04-333 Child and Family Services Reviews 

Table 2: Number of States Including Each of the PIP Strategy Categories Used in This Study 

PIP strategy category Description (number of states that included the strategy in their PIP) 

Policies and procedures Review, modify, or develop/implement any policy, procedure, or case practice standard 

(31) 

Enhance foster home/parent licensing standards (7) 

Develop child and family assessment tools, such as protocols for risk/safety 

determinations (28) 

Identify and adopt any promising practices (19) 

Data collection and analysis Review and report on agency performance through self-assessments or internal 

audits/review (31) 

Apply federal CFSR or similar process for internal statewide case reviews (16) 

Improve information and data collection systems (31) 

Staff supports Train and develop caseworkers (through dissemination and training on policy or 

through revisions to overall curriculum) (30) 

Assess and monitor staff responsibilities, skills, or performance (24) 

Recruit additional staff/retain staff (14) 

Lower caseloads (11) 

Increase caseworker pay (1) 

Foster parent supports/services and 

resources for children and families 

Train and develop foster families’/providers’ skills and capacities (27) 

Recruit and retain foster families (22) 

Increase involvement of foster or birth families in case (18) 

Expand service array for children and families (includes developing or enhancing 

transportation systems to transport siblings and parents for visits, creating one-stop 

centers for assistance, modifying visitation services, and providing any additional 

support services) (27) 

Engage stakeholders such as courts, advocates, foster homes, private providers, or 

sister agencies in public sector, e.g., mental health (can include consultation, training, 

or formal partnering to improve services or placements) (31) 

Create or improve monitoring of contracts with private providers to enhance service 

delivery (includes development of performance based or outcome-based contracts or 

other evaluations of provider performance) (25) 

State legislative supports State request for legislative action to support any of the above strategies (20) 

Federal technical assistance State request technical assistance from ACF or any resource center to support any of 

the above strategies (27) 

Source: GAO analysis. 

Our analysis found that every state’s PIP has included policy revisions or 

creation to improve programs and services. For example, to address 

unsatisfactory performance in the prevention of repeat maltreatment, 

California’s PIP includes a policy change, pending legislative approval, 

granting more flexibility to counties in determining the length of time they 

spend with families to ensure child safety and improve family functioning 

before closing cases. Additionally, 30 of the plans included caseworker 
Page 21 GAO-04-333 Child and Family Services Reviews 

training; 20 of the plans included requests for state legislative action; and 

27 of the plans included requests for federal technical assistance from ACF 

or the resource centers. For example, California planned to ask its 

legislature to allow the agency to consolidate standards for foster care and 

adoption home studies, believing this would facilitate the adoption of 

children in the state. In addition, New York worked with its legislature to 

secure additional funding to improve the accessibility of independent 

living and adoption services for children and families. 

Our analysis also showed that many states approached PIP development 

by building on state initiatives in place prior to the on-site review. Of the 

42 surveyed states reporting in our survey on this topic, 30 said that their 

state identified strategies for the PIP by examining ongoing state 

initiatives. For example, local officials in New York City and state officials 

in California told us that state reform efforts—borne in part from legal 

settlements—have become the foundation for the PIP. In New York, the 

Marisol case was one factor in prompting changes to the state’s child 

welfare financing structure. 22 Subsequently, the state legislature 

established a quality enhancement fund. Much of this money today 

supports strategies in New York’s PIP, such as permanency mediation to 

support family involvement in case planning and new tools to better assess 

children’s behavioral and mental health needs. California state officials 

also informed us that state reform efforts initiated by the governor prior to 

the CFSR, such as implementing a new system for receiving and 

investigating reports of abuse and neglect and developing more early 

intervention programs, became integral elements in the PIP. 

 22The plaintiffs in this class action lawsuit filed in 1995 were 11 children, all of whom were 

abused, sometimes severely. The plaintiffs sought to restructure New York City’s child 

welfare system and charged that the city mishandled their cases and deprived them of their 

rights under the state and federal constitutions as well as various state and federal laws. 

Marisol A. v. Giuliani, 929 F. Supp. 662, 669 (S.D.N.Y. 1996). As part of its settlement with 

the parties, the state agreed to monitor the city’s staffing requirements and conduct audits 

of the city’s treatment of children in nine specified areas. Marisol A. v. Giuliani, 185 

F.R.D. 152, 159-160 (S.D.N.Y. 1999). 
Page 22 GAO-04-333 Child and Family Services Reviews 

ACF has provided states with regulations and guidance to facilitate PIP 

development, but some states believe the requirements have been unclear. 

Some of the requirements for program improvement planning are outlined 

in the following table. 

Table 3: ACF Regulations and Guidance on Program Improvement Planning 

Stage Requirements Time frame 

Development of the 

PIP 

States are required to develop a PIP to address all 

of the outcomes and systemic factors determined 

not to be in substantial conformity as a result of a 

CFSR. Although ACF does not require a specific 

format for the PIP, it does offer states a suggested 

format, called the PIP matrix, to use. 

States must submit the PIP to ACF for approval within 

90 calendar days from the date that the state received 

written notification that it is not operating in substantial 

conformity. 

Approval of the PIP States must include certain information in the PIP, 

including measurable goals, action steps required to 

correct each deficiency identified in the final report, 

anticipated dates of completion for each action step, 

amount of progress the state will make toward 

national standards, benchmarks to measure state 

progress implementing the PIP, and technical 

assistance needs. 

No time frame is specified for ACF review and approval 

of the PIP. However, ACF guidance says that the 

regional office should give prompt attention to the PIP 

when it is submitted and take the minimum amount of 

time necessary to review it and respond to the state. If 

ACF notifies the state that the PIP is unacceptable, the 

state has 30 calendar days to resubmit the plan for 

approval. 

Implementation and 

monitoring of the PIP 

ACF will monitor the state’s progress in completing 

the provisions of the approved PIP by evaluating 

quarterly status reports and reviewing the state’s 

Annual Progress and Services Report, which 

incorporates the state’s progress in implementing 

the PIP. 

States must submit status reports to ACF on a quarterly 

basis, and the PIP must be implemented within 2 years.

Source: ACF regulations at 45 C.F.R. 1355.35 and guidance. 

Some states in our survey indicated that the guidance ACF had provided 

did not clearly describe the steps required for PIP approval. In addition, 

some state officials believe that even ACF’s more recent efforts to improve 

PIP guidance have also been insufficient. Of the 21 states reporting on the 

PIP approval process in our survey, 6 states—4 reviewed in 2001 and 

2 reviewed in 2002—said that ACF did not clearly describe its approval 

process, and another 8 states reported that ACF described the approval 

process as clearly as it did unclearly. Further, several states commented in 

our survey that several steps in the approval process were unclear to 

them, such as how much detail and specificity the agency expects the plan 

to include, what type of feedback states could expect to receive, when 

states could expect to receive such feedback, and whether a specific 

format was required. Officials in the states we visited echoed survey 

Insufficient Guidance 

Hampered State Planning 

Efforts, but ACF Has 

Taken Steps to Clarify 

Expectations and Improve 

Technical Assistance 
Page 23 GAO-04-333 Child and Family Services Reviews 

respondents’ concerns with officials from 3 of the 5 states informing us 

that ACF had given states different instructions regarding acceptable PIP 

format and content. For example, California and Florida officials told us 

that their program improvement plans had been rejected prior to final 

approval, even though they were based on examples of approved plans 

that regional officials had provided. In addition, California officials told us 

that they did not originally know how much detail the regional office 

expected in the PIP and believed that the level of detail the regional office 

staff ultimately required was too high. Although some steps may be 

duplicative, officials in California said that the version of their plan that 

the region accepted included 2,932 action steps—a number these officials 

believe is too high given their state’s limited resources and the 2-year time 

frame to implement the PIP. 

ACF officials have undertaken several steps to clarify their expectations 

for states and to improve technical assistance, but state responses to this 

assistance have been mixed. For example, in 2002, 2 years after ACF 

released the CFSR regulations and a procedures manual, ACF offered 

states additional guidance and provided a matrix format to help state 

officials prepare their plans. ACF officials told us the agency is also 

helping states through a team approach to providing on-site technical 

assistance. Under this approach, when ACF determines a state is slow in 

developing its PIP, the agency sends a team of staff from ACF and 

resource centers to the state to provide intensive on-site technical 

assistance. An official from West Virginia who had received this team 

assistance reported that ACF’s support was very beneficial. Further, ACF 

has attempted to encourage state officials to start developing program 

improvement plans before the final report is released. To do so, the agency 

has provided training to state officials and stakeholders almost 

immediately after the completion of the on-site review. ACF has sent staff 

from the resource center for Organizational Improvement to provide such 

training. An official from Utah, however, reported that the resource center 

training on PIP development had been general, and she wished the 

resource center staff had better tailored its assistance and provided more 

examples of strategies other states are pursuing to improve. 

Analysis of state survey responses indicates that starting to develop 

improvement plans early can make the 90-day time frame to prepare a PIP 

seem adequate. Of 9 states reporting that they started developing their PIP 

before or during the statewide assessment phase, 5 said that 90 days was 

adequate. Nonetheless, 21 of 35 state survey respondents reported that the 

90-day time frame was insufficient. For example, one respondent reported 

that 90 days is too short a time to perform tasks necessary for developing a 
Page 24 GAO-04-333 Child and Family Services Reviews 

program improvement plan, such as analyzing performance data, involving 

diverse groups of stakeholders in planning decisions, and obtaining the 

approval of state officials. Survey results indicate that increasing numbers 

of states are developing their PIPs early in the CFSR process, which may 

reflect ACF’s emphasis on PIP development. The following figure shows 

that of the 18 states reviewed in 2001, only 2 started developing their PIPs 

before or during the statewide assessment phase. Among states reviewed 

in 2003, this share increased to 5 of 9. 

Figure 3: Variation in the Start of States’ PIP Development 

Evidence suggests that lengthy time frames for PIP approval have not 

necessarily delayed PIP implementation, and ACF has made efforts to 

reduce the time the agency takes to approve states’ PIPs. For example, 

officials in 3 of the 5 states we visited told us they began implementing 

new action steps before ACF officially approved their plans because many 

of the actions in their PIPs were already under way. In addition, according 

to our survey, of the 28 states reporting on this topic, 24 reported that they 

had started implementing their PIP before ACF approved it. Further, our 

analysis shows that the length of time between the PIP due date, which 
Page 25 GAO-04-333 Child and Family Services Reviews 

statute sets at 90 days after the release of the final CFSR report, and final 

ACF PIP approval has ranged considerably—from 45 to 349 business days. 

For almost half of the plans, ACF’s approval occurred 91 to 179 business 

days after the PIP was due. As shown in figure 4, however, our analysis 

indicates that ACF has recently reduced the time lapse by 46 business days 

between states’ PIP due dates and ACF’s PIP approval. 

Figure 4: Average Time Lapse between States’ PIP Due Dates and ACF’s PIP 

Approval 

Notes: Data for states reviewed in 2003 are not included because only 2 states’ PIPs had been 

approved by March 1, 2004. 

We used the date that states are required to submit a PIP to calculate the time lapse. Therefore, the 

time lapse is calculated from day 90 through the date ACF ultimately approved the plans. States may 

have submitted their plans prior to or after the 90-day time frame designated in regulations, but ACF’s 

contractor—charged with tracking CFSR-related milestones—does not track the actual PIP 

submission dates. 

The shorter time lapse for PIP approval may be due, in part, to the ACF’s 

emphasis on PIP development. According to one official, ACF has directed 

states to concentrate on submitting a plan that can be quickly approved. 

Another ACF official added that because of ACF’s assistance with PIP 

development, states are now submitting higher-quality PIPs that require 

fewer revisions. 

Program improvement planning has been ongoing, but uncertainties have 

made it difficult for states to implement their plans and ACF to monitor 

state performance. Such uncertainties include not knowing whether state 

resources are adequate to implement the plans and how best to monitor 

state reforms. In answering a survey question about PIP implementation 

challenges, a number of states identified insufficient funding, staff, and 

time—as well as high caseloads—as their greatest obstacles. Figure 5 

depicts these results. 

State and Federal 

Uncertainties Cloud PIP 

Implementation and 

Monitoring 
Page 26 GAO-04-333 Child and Family Services Reviews 

Figure 5: Most Common Challenges Affecting States’ PIP Implementation 

Note: This is based on responses from 25 states. The results reported in the figure are a sum of the 

states reporting that the issue was a challenge to PIP implementation to a very great extent, great 

extent, moderate extent, or some/little extent. States not included answered no extent, no basis to 

judge, or not applicable. 

In regards to funding, an official from Pennsylvania commented that 

because of the state’s budget shortfall, no additional funds were available 

for the state to implement its improvement plan, so most counties must 

improve outcomes with little or no additional resources. A Massachusetts 

official reported that fiscal problems in his state likely would lead the state 

to lay off attorneys and caseworkers and to cut funding for family support 

programs. While state officials acknowledged that they do not have 

specific estimates of PIP implementation expenses because they have not 

tracked this information in their state financial systems, many states 

indicated that to cope with financial difficulties, they had to be creative 

and use resources more efficiently to fund PIP strategies. Of the 26 states 

responding to a question in our survey on PIP financing, 12 said that they 

were financing the PIP strategies by redistributing current funding, and 7 

said that they were using no-cost methods. In an example of the latter, 
Page 27 GAO-04-333 Child and Family Services Reviews 

Oklahoma officials reported pursuing in-kind donations from a greeting 

card company so that they could send thank-you notes to foster parents, 

believing this could increase foster parent retention and engagement. 

States also reported that PIP implementation has been affected by staff 

workloads, but these comments were mixed. In Wyoming, for example, 

caseworkers told us that their high caseloads would prevent them from 

implementing many of the positive action steps included in their 

improvement plan. In contrast, Oklahoma caseworkers told us that the 

improvement plan priorities in their state—such as finding permanent 

homes for children—have helped them become more motivated, more 

organized, and more effective with time management. For example, one 

caseworker explained that she is quicker now at locating birth fathers who 

were previously uninvolved in the child’s life because she uses the Internet 

to search for these fathers’ names. She said this new way of exploring 

leads and information—a strategy that stemmed from PIP development—

has been motivating and rewarding because it has decreased the time 

spent tracking down paternal relatives and increased the number of 

available placements for the child. 

ACF officials expressed uncertainty about how best to monitor states’ 

progress and apply estimated financial penalties when progress was slow 

or absent, and 3 of the 5 states we visited reported frustration with the 

limited guidance ACF had provided on the PIPs quarterly reporting 

process. For example, 4 regional offices told us that they did not have 

enough guidance on or experience with evaluating state quarterly reports. 

Some regional offices told us they require states to submit evidence of 

each PIP action step’s completion, such as training curricula or revised 

policies, but one ACF official acknowledged that this is not yet standard 

procedure, although the agency is considering efforts to make the 

quarterly report submission procedures more uniform. Moreover, ACF 

staff from one region told us that because PIP monitoring varies by region, 

they were concerned about enforcing penalties. Finally, shortly before 

California’s quarterly report was due, state officials told us they still did 

not know how much detail to provide, how to demonstrate whether they 

had completed certain activities, or what would happen if they did not 

reach the level of improvement specified in the plan. 

Based on data from the states that have been reviewed to date, the 

estimated financial penalties range from a total of $91,492 for North 

Dakota to $18,244,430 for California, but the impact of these potential 

penalties remains unclear. While ACF staff from most regional offices told 

us that potential financial penalties are not the driving force behind state 
Page 28 GAO-04-333 Child and Family Services Reviews 

reform efforts, some contend that the estimated penalties affect how 

aggressively states pursue reform in their PIPs. For example, regional 

office staff noted that one state’s separate strategic plan included more 

aggressive action steps than those in its PIP because the state did not want 

to be liable for penalties if it did not meet its benchmarks for 

improvement. State officials also had mixed responses as to how the 

financial penalties would affect PIP implementation. An official in 

Wyoming said that incurring the penalties was equivalent to shutting down 

social service operations in one local office for a month, while other 

officials in the same state thought it would cost more to implement PIP 

strategies than it would to incur financial penalties if benchmarks were 

unmet. Nevertheless, these officials also said that while penalties are a 

consideration, they have used the CFSR as an opportunity to provide 

better services. One official in another state agreed that it would cost more 

to implement the PIP than to face financial penalties, but this official was 

emphatic in the state’s commitment to program improvement. 

To implement the CFSRs, ACF has focused its activities almost entirely on 

the four phases of the review process. However, staff in several regions 

report limitations in providing assistance to states in helping them to meet 

key federal goals. Although regional staff conduct site visits to states for 

reasons beyond the CFSR process, conducting the CFSR on-site reviews 

and providing PIP-related assistance to states account for the majority of 

regions’ time and travel budgets, according to ACF officials. Further, 

regional office staff said that more frequent visits with state personnel—

visits outside of the CFSR process in particular—would allow them to 

better understand states’ programs and cultivate relationships with state 

officials. In addition, state officials in all five of the states we visited said 

that ACF technical assistance needed improvement, acknowledging that in 

some cases regional office staff were stretched thin by CFSR demands and 

in other cases that assistance from resource center staff lacked focus. 

While ACF officials in the central office said that the CFSR has become the 

primary method for evaluating states’ performance, they acknowledged 

that regional staff might still be adjusting to the new way ACF oversees 

child welfare programs. Further, they told us that ACF is currently 

reevaluating the entire structure of its training and technical assistance, in 

part to address these concerns. 

ACF officials told us that the learning opportunities in the Children’s 

Bureau are intentionally targeted at the CFSR, but staff in 3 regions told us 

that this training should cover a wider range of subjects—including topics 

outside of the CFSR process—so that regional officials could better meet 

ACF’s Focus Rests 

Almost Exclusively on 

Implementing the 

CFSR 
Page 29 GAO-04-333 Child and Family Services Reviews 

states’ needs. All 18 of the courses that ACF has provided to its staff since 

2001 have focused on topics such as writing final CFSR reports and using 

data for program improvement. ACF officials in the central office said that 

the course selection reflects both the agency’s prioritization of the CFSR 

process and staff needs. To ascertain staff training needs, ACF surveyed 

regional staff in October 2002, and ACF officials told us they used the 

survey results in deciding which courses to offer. Our analysis of this 

survey, however, showed that it focused only on training topics directly 

related to the CFSR, so it might offer only limited information on whether 

regional officials wanted training on other topics. Specifically, the survey 

asked staff to check their top 5 training choices from among 

11 CFSR-related topics. While survey respondents were also given the 

opportunity to write in additional training tropics they desired, only 2 of 

the survey’s 27 respondents did so.23 One indicated a greater need for 

training on Indian child welfare issues, and another expressed a desire to 

learn more about the entire child welfare system. Although it is not 

possible to determine whether more respondents would have prioritized 

non-CFSR training areas had the survey been designed to elicit such 

information, our interviews with regional staff suggest that some of them 

wish to obtain additional non-CFSR training. For example, a staff member 

from one region told us she has not been adequately trained in child 

welfare and believed that her credibility was damaged when a state 

wanted advice that she could not provide on how to help older youth 

prepare to exit from foster care. 

In addition to offering training, ACF organizes biennial conferences for 

state and federal child welfare officials. Nonetheless, staff from 5 regions 

told us that they wanted more substantive interaction with their ACF 

colleagues, such as networking at conferences, to increase their overall 

child welfare expertise. Staff from 6 of the 10 regions told us that their 

participation in conferences is limited because of funding constraints. 

Further, staff in all 10 regions provide ongoing assistance or ad hoc 

counseling to states, but staff from 6 regions told us they would like to 

conduct site visits with states more regularly to improve their 

relationships with state officials and provide more targeted assistance. For 

example, staff in most regions told us that they assist states predominantly 

by e-mail and telephone on topics such as interpretation of Title IV-E 

eligibility criteria. Additionally, staff in 7 regions said that they sometimes 

 23ACF could not provide a response rate. 
Page 30 GAO-04-333 Child and Family Services Reviews 

visit with states to participate in state planning meetings—part of the 

annual child and family services planning effort—or to give presentations 

at state conferences on topics such as court improvement. However, staff 

in 4 regions felt their travel funds were constrained and explained that 

they try to stretch their travel dollars by addressing states’ non-CFSR 

needs, such as court improvements, during CFSR-related visits. While an 

ACF senior official from central office confirmed that CFSR-related travel 

constituted 60 percent of its 2002 child welfare-monitoring budget, this 

official added that CFSR spending represents an infusion of funding rather 

than a reprioritization of existing dollars and stated that regional 

administrators have discretion over how the funds are allocated within 

their regions. In addition, the same official stated that he knew of no 

instance in which a region requested more money for travel than it 

received. 

Concerns from state officials in all 5 of the states we visited echoed those 

of regional office staff and confirmed the need for improvements to the 

overall training and technical assistance structure, while respondents’ 

comments on our survey showed more mixed perceptions on the quality of 

assistance they received. For example, state officials in New York and 

Wyoming commented that ACF staff from their respective regional offices 

did not have sufficient time to spend with them on CFSR matters because 

regional staff were simultaneously occupied conducting reviews in other 

states. Further, an Oklahoma state official commented that assistance 

from one resource center was not as specific or helpful as desired. 

Specifically, when the state asked the resource center to provide a 

summary of other states’ policies regarding the intake of abuse and neglect 

allegations, the resource center did not provide an analysis of sufficient 

depth for the state to explore possible reforms. 

According to state survey respondents, however, satisfaction with the 

training and technical assistance provided by regional offices varied by 

CFSR phase. For example, among states reviewed in 2001, 2002, and 2003, 

satisfaction was generally highest in the statewide assessment phase, but 

then dropped during on-site review and PIP development before rising 

again in the PIP implementation phase. Across all phases of the CFSR 

process, however, states reviewed in 2003 had much higher levels of 

satisfaction with regional office assistance than those states reviewed in 

2001, suggesting improvements to regional office training and technical 

assistance as the process evolved. Further, based on survey data and our 

follow-up calls with selected states, satisfaction was also mixed in regard 

to resource center provided assistance. For example, among states 

reporting in our survey on the quality of assistance provided by the 
Page 31 GAO-04-333 Child and Family Services Reviews 

resource center for Organizational Improvement and the resource center 

for Information Technology—the two resource centers that provide 

specific support to states regarding data issues and PIP development—

satisfaction was generally lower in every phase among states reviewed in 

2003 than among states reviewed in 2001. The only exception to this was 

during the PIP development phase, for which states reviewed in 2003 

reported higher levels of satisfaction with the resource center for 

Organizational Improvement than states reviewed in 2001, suggesting 

positive responses to the on-site training and technical assistance this 

resource center has recently been providing to aid states in their PIP 

planning efforts.24

ACF officials told us the CFSR has become the agency’s primary 

mechanism for monitoring states and facilitating program improvement, 

but they acknowledged that regional office staff might not have realized 

the full utility of the CFSR as a tool to integrate all existing training and 

technical assistance efforts. Further, according to ACF officials, meetings 

to discuss a new system of training and technical assistance are ongoing, 

though recommendations were not available at the time of publication. 

Levels of resource center funding, the scope and objectives of the 

resource centers’ work, and the contractors who operate the resource 

centers are all subject to change before the current cooperative 

agreements expire at the close of fiscal year 2004.25 

ACF and the states have devoted considerable resources to the CFSR 

process, but concerns remain regarding the validity of some data sources 

and the limited use of all available information to determine substantial 

conformity. Further, no state to date has passed the threshold for 

substantial conformity on all CFSR measures. The majority of states 

surveyed agreed that CFSR results are similar to their own evaluation of 

 24According to interviews with ACF regional staff, only one region is tracking states’ 

satisfaction with the resource centers systematically. While some regions conduct ad hoc 

conversations with state officials after a resource center has provided assistance, ACF does 

not have a formal mechanism for collecting or analyzing states’ views on the quality of 

assistance provided. 

25Funding levels for each resource center have varied over their 5-year cooperative 

agreements, but our analysis of 1999-2003 data shows that since the implementation of the 

CFSRs, the funding for the resource center for Information Technology and the resource 

center for Organizational Improvement—the two resource centers that provide specific 

support to states regarding data issues and PIP development—has consistently and 

substantially increased. 

Conclusions 
Page 32 GAO-04-333 Child and Family Services Reviews 

areas needing improvement. However, without using more reliable data—

and in some cases, additional data from state self-assessments—to 

determine substantial conformity, ACF may be over- or underestimating 

the extent to which states are actually meeting the needs of the children 

and families in their care. These over- or underestimates can, in turn, 

affect the scope and content of the PIPs that states must develop in 

response. As states face difficult budget decisions, accurate performance 

information could be critical to deciding how best to allocate resources. 

We previously reported on the reliability of state-reported child welfare 

data and recommended that HHS consider additional ways to enhance the 

guidance and assistance offered to states to help them overcome the key 

challenges in collecting and reporting child welfare data. In response to 

this recommendation, HHS said that ACF has provided extensive guidance 

on how states can improve the quality of their data and acknowledged that 

additional efforts were under way. 

In addition, the PIP development, approval, and monitoring processes 

remain unclear to some, potentially reducing states’ credibility with their 

stakeholders and straining the federal/state partnership. Similarly, regional 

officials are unclear as to how they can accomplish their various training 

and technical assistance responsibilities, including the CFSR. Without 

clear guidance on how to systematically prepare and monitor PIP-related 

documents, and how regional officials can integrate their many oversight 

responsibilities, ACF has left state officials unsure of how their progress 

over time will be judged and potentially complicated its own monitoring 

efforts. 

To ensure that ACF uses the best available data in measuring state 

performance, we recommend that the Secretary of HHS expand the use of 

additional data states may provide in their statewide assessments and 

consider alternative data sources when available, such as longitudinal data 

that track children’s placements over time, before making final CFSR 

determinations. 

In addition, to ensure that ACF regional offices and states fully understand 

the PIP development, approval, and monitoring processes, and that 

regional offices fully understand ACF’s prioritization of the CFSR as the 

primary mechanism for child welfare oversight, we recommend that the 

Secretary of HHS take the following two actions: 

• issue clarifying guidance on the PIP process and evaluate states’ and 

regional offices’ adherence to this instruction, and 

Recommendations for 

Executive Action 
Page 33 GAO-04-333 Child and Family Services Reviews 

• provide guidance to regional offices explaining how to better integrate the 

many training and technical assistance activities for which they are 

responsible, such as participation in state planning meetings and the 

provision of counsel to states on various topics, with their new CFSR 

responsibilities. 

We received comments on a draft of this report from HHS. These 

comments are reproduced in appendix IV. HHS also provided technical 

clarifications, which we incorporated where appropriate. 

HHS generally agreed with our findings and noted that the CFSR process 

has already focused national attention on child welfare reform, but 

because the CFSR is the first review of its kind, HHS is engaged in 

continuous monitoring and improvement of the process. However, in its 

technical comments, HHS commented that while it acknowledges that the 

CFSR is its top priority, it disagreed with our statement that HHS’s focus 

rests exclusively on implementing the CFSR, stating that the 

Administration for Children and Families (ACF) also conducts other 

oversight efforts, such as Title IV-E eligibility reviews and AFCARS 

assessments. While we acknowledged ACF’s other oversight activities in 

the background section of the report, this report focuses primarily on the 

CFSR and we reflected the comments that ACF officials made throughout 

the course of our work that the CFSR was the primary tool for monitoring 

state performance and that it served as the umbrella for all monitoring 

activities undertaken by central and regional ACF staff. 

HHS further noted in its technical comments that we were wrong to 

suggest that federal staff do not know how to monitor state PIPs or assess 

financial penalties. However, we do not report that ACF is unsure of how 

to monitor PIPs or how to assess financial penalties—rather, we reported 

that ACF regional staff have not received sufficient guidance on how to 

best monitor PIPs and that ACF officials have not decided how or when to 

apply such penalties, even though two states to date have completed their 

initial PIP implementation timeframe and all states reviewed thus far are 

engaged in PIP development and implementation.

With regard to our first recommendation, HHS acknowledged that several 

steps are under way to address necessary data improvements and said that 

states have begun to submit more accurate information in their AFCARS 

and NCANDS profiles, with HHS’s assistance. HHS also commented that 

we failed to properly emphasize the states’ responsibility to improve 

overall data quality. We believe that our report, as well as our previous 

Agency Comments 
Page 34 GAO-04-333 Child and Family Services Reviews 

report on child welfare data and states’ information systems, addresses 

HHS’s activities and the steps many states have taken to enhance their 

CFSR data. Given that many states have developed independent data 

collection tools—and included findings from these instruments in their 

statewide assessments—our recommendation is meant to encourage HHS 

to work more closely with all states to supplement their AFCARS and 

NCANDS data in order to improve the determinations made about state 

performance. 

In addition, HHS commented that our report emphasized the limitations of 

the 50 case sample size without focusing on the expenses and the 

increased state and federal staff time that would likely be associated with 

efforts to increase the sample size. We agree that additional expenses and 

staff time would likely be needed to increase the sample size and 

recommended that ACF use additional data—beyond the information 

collected from the 50 case reviews and ACFARS and NCANDS data—to 

develop a more accurate picture of state performance. This information 

could include the data that many states already collect on their 

performance, such as longitudinal information tracking children from their 

time of entry into the system. 

In response to our second recommendation, HHS said that it has 

continued to provide technical assistance and training to states and 

regional offices, when appropriate. HHS noted that it is committed to 

continually assessing and addressing training and technical assistance 

needs. In this context, our recommendation was intended to encourage 

HHS to enhance existing training efforts and focus both on state and on 

regional officials’ needs in understanding and incorporating the CFSR 

process into their overall improvement and oversight efforts. 

We also provided a copy of our draft report to child welfare officials in the 

five states we visited—California, Florida, New York, Oklahoma, and 

Wyoming. We received comments from California, Florida, New York, and 

Oklahoma, all of which generally agreed with our findings and provided 

various technical comments, which we also incorporated where 

appropriate. 

We are sending copies of this report to the Secretary of Health and Human 

Services, state child welfare directors, and other interested parties. We 

will make copies available to others on request. In addition, the report will 

be available at no charge on GAO’s Web site at http://www.gao.gov. If you 

or your staff have any questions or wish to discuss this material further, 
Page 35 GAO-04-333 Child and Family Services Reviews 

please call me at (202) 512-8403 or Diana Pietrowiak at (202) 512-6239. Key 

contributors to this report are listed in appendix IV. 

Cornelia M. Ashby, Director, 

Education, Workforce, and 

Income Security Issues 
Page 36 GAO-04-333 Child and Family Services Reviews 

List of Congressional Requesters 

The Honorable Tom DeLay 

Majority Leader 

House of Representatives 

The Honorable Charles E. Grassley 

Chairman, Committee on Finance 

United States Senate 

The Honorable George Miller 

Ranking Democratic Member 

Committee on Education and the Workforce 

House of Representatives 

The Honorable Charles B. Rangel 

Ranking Democratic Member 

Committee on Ways and Means 

House of Representatives 

The Honorable Fortney Pete Stark 

Ranking Democratic Member 

Subcommittee on Health 

Committee on Ways and Means 

House of Representatives 

The Honorable Benjamin L. Cardin 

Ranking Democratic Member 

Subcommittee on Human Resources 

Committee on Ways and Means 

House of Representatives 
Appendix I: Objectives, Scope, and 

Methodology 

Page 37 GAO-04-333 Child and Family Services Reviews 

The objectives of our study were to report on (1) ACF’s and the states’ 

experiences preparing for and conducting the statewide assessments and 

on-site reviews; (2) ACF’s and the states’ experiences developing, funding, 

and implementing items in the PIP; and (3) additional efforts that ACF has 

taken beyond the CFSR to ensure that all states meet federal goals of 

safety, permanency, and well-being. 

To gather information about ACF’s and the states’ experiences with the 

CFSR and PIP process, we utilized multiple methodologies to solicit 

information from both ACF and the states, including (1) a Web-based 

survey to state child welfare agencies; (2) site visits to five states; (3) a 

content analysis of all 31 PIPs available as of January 1, 2004; 

(4), interviews with ACF officials in Washington and all regional offices, 

directors of all resource centers, and child welfare experts nationwide; 

and (5) a review of CFSR regulations and the available guidance offered to 

states. We conducted our work between May 2003 and February 2004 in 

accordance with generally accepted government auditing standards. 

To gather information about states’ experiences with each phase of the 

CFSR and PIP process, we distributed a Web-based survey to all 50 states, 

the District of Columbia, and Puerto Rico on July 30, 2003. We pretested 

the survey instrument with officials in the District of Columbia, Kentucky, 

and Maryland; and after extensive follow-up, we received survey 

responses from all 50 states and the District of Columbia for a 98 percent 

response rate.1

 We did not independently verify the information obtained 

through the survey. 

The survey asked a combination of questions that allowed for open-ended 

and close-ended responses. Because some states had not yet begun their 

statewide assessments and others had already submitted quarterly PIP 

progress reports at the time that our survey was released, the instrument 

was designed with skip patterns directing states to comment only on the 

CFSR stages that they had begun or completed to that point. Therefore, 

the number of survey respondents for each question varied depending on 

the number of states that had experienced that stage of the CFSR and PIP 

processes. 

 1

Puerto Rico did not respond to our survey. 

Appendix I: Objectives, Scope, and 

Methodology 

Objectives 

Scope and 

Methodology 

Survey 
Appendix I: Objectives, Scope, and 

Methodology 

Page 38 GAO-04-333 Child and Family Services Reviews 

To supplement the survey and elaborate on survey responses, we selected 

10 states with which to conduct follow-up phone calls based on their 

answers to the survey’s open-ended questions.2

 These calls helped us 

obtain more specific examples about states’ experiences preparing for the 

CFSR; developing, funding, and implementing a PIP; and working with 

ACF to improve their child welfare systems. 

Because this was not a sample survey, there are no sampling errors. 

However, the practical difficulties of conducting any survey may introduce 

errors, commonly referred to as nonsampling errors. For example, 

difficulties in how a particular question is interpreted, in the sources of 

information that are available to respondents, or in how the data are 

entered into a database or were analyzed, can introduce unwanted 

variability into the survey results. We took steps in the development of the 

questionnaire, the data collection, and the data analysis to minimize these 

nonsampling errors. For example, social science survey specialists 

designed the questionnaire in collaboration with GAO staff with subject 

matter expertise. Then, the draft questionnaire was pretested with a 

number of state officials to ensure that the questions were relevant, clearly 

stated, and easy to comprehend. When the data were analyzed, a second, 

independent analyst checked all computer programs. Since this was a 

Web-based survey, respondents entered their answers directly into the 

electronic questionnaire. This eliminated the need to have the data keyed 

into a database thus removing an additional source of error. 

To gather more detailed information about the states’ experiences with the 

CFSR and PIP process, we selected five states to visit—California, Florida, 

New York, Oklahoma, and Wyoming—based on the timing and results of 

each state’s CFSR, as well as their differences in location, size of child 

welfare population, degree of privatization of services, size of tribal 

populations, and whether they had state or locally administered systems. 

In preparation for the visits and to understand the unique circumstances in 

each state, we obtained and reviewed relevant literature from each of the 

five states, such as the statewide assessment, the CFSR final report, and 

any available PIPs or quarterly reports. Additionally, we reviewed relevant 

 2

The 10 states participating in our phone follow-up survey were Arkansas, Iowa, Kansas, 

Mississippi, North Dakota, New Jersey, Pennsylvania, Rhode Island, Utah, and West 

Virginia. 

Site Visits 
Appendix I: Objectives, Scope, and 

Methodology 

Page 39 GAO-04-333 Child and Family Services Reviews 

past or current litigation that may affect the states’ delivery of services as 

identified by the National Center for Youth Law’s Litigation Docket (2002). 

During our visits to each state, we talked with officials from the state child 

welfare agency along with officials and staff from at least one local agency 

office that was selected for the CFSR on-site review. Specifically, in each 

state we spoke with state and local officials responsible for guiding the 

states’ efforts throughout the review process; CFSR on-site reviewers; and 

stakeholders, including judges, child advocates, private providers, foster 

parents, and child welfare staff. Some detailed information regarding key 

CFSR milestones among the five states we visited is included in appendix 

III. 

To learn about states’ improvement strategies, we conducted a content 

analysis of the 31 available PIPs that ACF had approved by January 1, 

2004. For each of these PIPs, we classified the state’s action steps as 

relating to one or more of the following: policies and procedures, data 

collection and analysis, staff supports, foster parent supports or services 

and resources for children and families, state legislative supports, and 

federal technical assistance. Table 2 in the report summarizes how we 

classified the PIP strategies and indicates the number of states including 

each strategy in its PIP. 

To gather information about ACF’s experience with the CFSR and PIP 

process, we interviewed ACF officials in Washington, D.C., who are 

involved in the CFSR process and ACF staff in all 10 of the regional 

offices; directors of each resource center; and ACF contractors working 

on CFSR-related activities. Further, we observed the final debriefing 

sessions for three states—South Carolina, Virginia, and Washington—

during the weeks of their respective on-site reviews. 

In addition to our interviews with ACF officials, we also interviewed 10 

prominent child welfare experts and researchers, such as those affiliated 

with the Chapin Hall Center for Children, the Child Welfare League of 

America, the National Coalition on Child Protection Reform, and the 

University of California at Berkeley, to learn additional information about 

the states’ experiences with the CFSR process, including information 

about states’ concerns with the reliability of CFSR data, states’ 

involvement of tribes as stakeholders, and the media’s coverage of 

egregious child welfare cases. 

Content Analysis of 

Available PIPs 

Interviews 
Appendix I: Objectives, Scope, and 

Methodology 

Page 40 GAO-04-333 Child and Family Services Reviews 

To gather information about CFSR regulations and the available training 

and technical assistance offered to states, we reviewed ACF’s regulations, 

its policy memorandums, and the CFSR manual it makes available to 

states. In addition, we obtained and reviewed a list of all of the resource 

centers’ training and technical assistance activities provided to the five 

states we visited during our site visits. 

Review of ACF Guidance 

to States 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 41 GAO-04-333 Child and Family Services Reviews 

(Items with an asterisk have associated national standards.) 

Child Safety 

Outcome 1: Children are, first and foremost, protected from abuse and 

neglect. 

Item 1: Timeliness of initiating investigations on reports of child 

maltreatment 

Item 2: Repeat maltreatment 

• Recurrence of maltreatment* 

• Incidence of child abuse and/or neglect in foster care* 

Outcome 2: Children are safely maintained in their own homes 

whenever possible and appropriate. 

Item 3: Services to family to protect child(ren) in home and prevent 

removal 

Item 4: Risk of harm to child(ren) 

Permanency for Children 

Outcome 3: Children have permanency and stability in their living 

conditions. 

Item 5: Foster care re-entries* 

Item 6: Stability of foster care placement* 

Item 7: Permanency goal for child 

Item 8: Reunification, guardianship, or permanent placement with relatives 

• Length of time to achieve permanency goal of reunification* 

Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Outcome factors 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 42 GAO-04-333 Child and Family Services Reviews 

Item 9: Adoption 

• Length of time to achieve permanency goal of adoption* 

Item 10: Permanency goal of other planned permanent living arrangement 

Outcome 4: The continuity of family relationships and connections is 

preserved for children. 

Item 11: Proximity of foster care placement 

Item 12: Placement with siblings 

Item 13: Visiting with parents and siblings in foster care 

Item 14: Preserving connections 

Item 15: Relative placement 

Item 16: Relationship of child in care with parents 

Child and Family Well-being 

Outcome 5: Families have enhanced capacity to provide for their 

children’s needs. 

Item 17: Needs and services of child, parents, foster parents 

Item 18: Child and family involvement in case planning 

Item 19: Worker visits with child 

Item 20: Work visits with parent(s) 

Outcome 6: Children receive appropriate services to meet their 

educational needs. 

Item 21: Education needs of the child 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 43 GAO-04-333 Child and Family Services Reviews 

Outcome 7: Children receive adequate services to meet their physical 

and mental health needs. 

Item 22: Physical health of child 

Item 23: Mental health of child. 

Systemic factor 1: Statewide information system 

Item 24: State is operating a statewide information system that, at a 

minimum, can identify the status, demographic characteristics, location, 

and goals for the placement of every child who is (or within the 

immediately preceding 12 months has been) in foster care. 

Systemic factor 2: Case review system 

Item 25: Provides a process that ensures that each child has a written case 

plan to be developed jointly with the child’s parent(s) that includes the 

required provisions 

Item 26: Provides a process for the periodic review of the status of each 

child, no less frequently than once every 6 months, either by a court or by 

administrative review. 

Item 27: Provides a process that ensures that each child in foster care 

under the supervision of the state had a permanency hearing in a qualified 

court or administrative body no later than 12 months from the date the 

child entered foster care and no less frequently than every 12 months 

thereafter. 

Item 28: Provides a process for termination of parental rights proceedings 

in accordance with the provisions of the Adoption and Safe Families Act. 

Item 29: Provides a process for foster parents, pre-adoptive parents, and 

relative caregivers of children in foster care to be notified of, and have an 

opportunity to be heard in, any review or hearing held with respect to the 

child. 

Systemic Factors 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 44 GAO-04-333 Child and Family Services Reviews 

Systemic factor 3: Quality assurance system 

Item 30: The state has developed and implemented standards to ensure 

that children in foster care are provided quality services that protect the 

safety and health of children. 

Item 31: The state is operating an identifiable quality assurance system 

that is in place in the jurisdictions where the services included in the Child 

and Family Services Plan are provided, evaluates the quality of services, 

identifies strengths and needs of the service delivery system, provides 

relevant reports, and evaluates program improvement measures 

implemented. 

Systemic factor 4: Training 

Item 32: The state is operating a staff development and training program 

that supports the goals and objectives in the Child and Family Services 

Plan, addresses services provided under Titles IV-B and IV-E, and provides 

initial training for all staff who deliver these services. 

Item 33: The state provides for ongoing training for staff that addresses the 

skills and knowledge base needed to carry out their duties with regard to 

the services included in the Child and Family Services Plan. 

Item 34: The state provides training for current or prospective foster 

parents, adoptive parents, and staff of state-licensed or approved facilities 

that care for children receiving foster care or adoption assistance under 

Title IV-E that addresses the skills and knowledge base needed to carry 

out their duties with regard to foster and adopted children. 

Systemic factor 5: Service array 

Item 35: The state has in place an array of services that assess the 

strengths and needs of children and families and determine other service 

needs, address the needs of families in addition to individual children in 

order to create a safe home environment, enable children to remain safely 

with their parents when reasonable, and help children in foster and 

adoptive placements achieve permanency. 

Item 36: The services in item 35 are accessible to families and children in 

all political jurisdictions covered in the State’s Child and Family Services 

Plan. 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 45 GAO-04-333 Child and Family Services Reviews 

Item 37: The services in item 35 can be individualized to meet the unique 

needs of children and families served by the agency. 

Systemic factor 6: Agency responsiveness to the community 

Item 38: In implementing the provisions of the Child and Family Services 

Plan, the state engages in ongoing consultation with tribal representatives, 

consumers, services providers, foster care providers, the juvenile court, 

and other public and private child- and family-serving agencies and 

includes the major concerns of these representatives in the goals and 

objectives of the Child and Family Services Plan. 

Item 39: The agency develops, in consultation with these representatives, 

annual reports of progress and services delivered pursuant to the Child 

and Family Services Plan. 

Item 40: The state’s services under the Child and Family Services Plan are 

coordinated with services or benefits of other federal or federally assisted 

programs serving the same population. 

Systemic factor 7: Foster and adoptive parent licensing, recruitment, 

and retention 

Item 41: The state has implemented standards for foster family homes and 

child care institutions that are reasonably in accord with recommended 

national standards. 

Item 42: The standards are applied to all licensed or approved foster family 

homes or child care institutions receiving Title IV-E or IV-B funds. 

Item 43: The state complies with federal requirements for criminal 

background clearances as related to licensing or approving foster care and 

adoptive placements and has in place a case-planning process that 

includes provisions for addressing the safety of foster care and adoptive 

placements for children. 

Item 44: The state has in place a process for ensuring the diligent 

recruitment of potential foster and adoptive families that reflect the ethnic 

and racial diversity of children in the state for whom foster and adoptive 

homes are needed. 
Appendix II: List of Outcomes and Systemic 

Factors and Their Associated Items 

Page 46 GAO-04-333 Child and Family Services Reviews 

Item 45: The state has in place a process for the effective use of crossjurisdictional resources to facilitate timely adoptive or permanent 

placements for waiting children. 
Appendix III: Dates on Which Site Visit States 

Reached CFSR Milestones 

Page 47 GAO-04-333 Child and Family Services Reviews 

 California Florida New York Oklahoma Wyoming

Date the state submitted its statewide assessment 7/02 5/01 4/01 1/02 6/02

Date ACF began the on-site review 9/23/02 8/6/01 6/18/01 3/18/02 7/8/02

Date ACF released the final CFSR report 1/10/03 4/23/02 1/9/02 7/1/02 4/1/03

Time lapse in release of final report 75 days 182 days 143 days 71 days 187 days

Date the state’s PIP was due 4/10/03 7/22/02 4/9/02 9/29/02 6/30/03

Date ACF approved the PIP 6/24/03 4/30/03 4/14/03 1/22/03 1/1/04

Time lapse in PIP approval 54 days 203 days 265 days 83 days 134 days

Date the state submitted its first quarterly report 10/29/03 7/30/03 8/11/03 5/15/03 4/30/04

Total number of outcomes and systemic factors not in 

substantial conformitya

12 8 8 9 10

Estimated financial penalties $18,244,430 $3,639,552 $2,284,407 $727,300 $137,369

Source: GAO analysis of ACF data. 

a

The total number of outcomes and systemic factors evaluated in the CFSR is 14. 

Appendix III: Dates on Which Site Visit States 

Reached CFSR Milestones 
Appendix IV: Comments from the Department of Health and Human Services 

Page 48 GAO-04-333 Child and Family Services Reviews 

Appendix IV: Comments from the 

Department of Health and Human Services 
Appendix IV: Comments from the Department of Health and Human Services 

Page 49 GAO-04-333 Child and Family Services Reviews 
Appendix IV: Comments from the Department of Health and Human Services 

Page 50 GAO-04-333 Child and Family Services Reviews 
Appendix IV: Comments from the Department of Health and Human Services 

Page 51 GAO-04-333 Child and Family Services Reviews 
Appendix V: GAO Contacts and Acknowledgments 

Page 52 GAO-04-333 Child and Family Services Reviews 

Diana Pietrowiak, (202) 512-6239 

Joy Gambino, (202) 512-2838 

In addition to those named above, Elizabeth Caplick and Catherine Roark 

made key contributions to this report. Amy Buck, Karen Burke, 

Jason Kelly, Stuart Kaufman, Luann Moy, and Jerome Sandau also 

provided key technical assistance. 

Appendix V: GAO Contacts and 

Acknowledgments 

GAO Contacts 

Staff 

Acknowledgments 
Related GAO Products 

Page 53 GAO-04-333 Child and Family Services Reviews 

Child Welfare: Improved Federal Oversight Could Assist States in 

Overcoming Key Challenges. GAO-04-418T. Washington, D.C.: January 28, 

2004. 

D.C. Family Court: Progress Has Been Made in Implementing Its 

Transition. GAO-04-234. Washington, D.C.: January 6, 2004. 

Child Welfare: States Face Challenges in Developing Information 

Systems and Reporting Reliable Child Welfare Data. GAO-04-267T. 

Washington, D.C.: November 19, 2003. 

Child Welfare: Enhanced Federal Oversight of Title IV-B Could Provide 

States Additional Information to Improve Services. GAO-03-956. 

Washington, D.C.: September 12, 2003. 

Child Welfare: Most States Are Developing Statewide Information 

Systems, but the Reliability of Child Welfare Data Could be Improved. 

GAO-03-809. Washington, D.C.: July 31, 2003. 

D.C. Child and Family Services: Key Issues Affecting the Management of 

Its Foster Care Cases. GAO-03-758T. Washington, D.C.: May 16, 2003. 

Child Welfare and Juvenile Justice: Federal Agencies Could Play a 

Stronger Role in Helping States Reduce the Number of Children Placed 

Solely to Obtain Mental Health Services. GAO-03-397. Washington, D.C.: 

April 21, 2003. 

Foster Care: States Focusing on Finding Permanent Homes for Children, 

but Long-Standing Barriers Remain. GAO-03-626T. Washington, D.C.: 

April 8, 2003. 

Child Welfare: HHS Could Play a Greater Role in Helping Child Welfare 

Agencies Recruit and Retain Staff. GAO-03-357. Washington, D.C.: March 

31, 2003. 

Foster Care: Recent Legislation Helps States Focus on Finding 

Permanent Homes for Children, but Long-Standing Barriers Remain. 

GAO-02-585. Washington, D.C.: June 28, 2002. 

District of Columbia Child Welfare: Long-Term Challenges to Ensuring 

Children’s Well- Being. GAO-01-191. Washington, D.C.: December 29, 2000. 

Related GAO Products 
Related GAO Products 

Page 54 GAO-04-333 Child and Family Services Reviews 

Child Welfare: New Financing and Service Strategies Hold Promise, but 

Effects Unknown. GAO/T-HEHS-00-158. Washington, D.C.: July 20, 2000. 

Foster Care: States’ Early Experiences Implementing the Adoption and 

Safe Families Act. GAO/HEHS-00-1. Washington, D.C.: December 22, 1999. 

Foster Care: HHS Could Better Facilitate the Interjurisdictional 

Adoption Process. GAO/HEHS-00-12. Washington, D.C.: November 19, 

1999. 

Foster Care: Effectiveness of Independent Living Services Unknown.

GAO/HEHS-00-13. Washington, D.C.: November 10, 1999. 

Foster Care: Kinship Care Quality and Permanency Issues. 

GAO/HEHS-99-32. Washington, D.C.: May 6, 1999. 

Juvenile Courts: Reforms Aim to Better Serve Maltreated 

Children. GAO/HEHS-99-13. Washington, D.C.: January 11, 1999. 

Child Welfare: Early Experiences Implementing a Managed Care 

Approach. GAO/HEHS-99-8. Washington, D.C.: October 21, 1998. 

Foster Care: Agencies Face Challenges Securing Stable Homes for 

Children of Substance Abusers. GAO/HEHS-98-182. Washington, D.C.: 

September 30, 1998. 

(130259) 
The General Accounting Office, the audit, evaluation and investigative arm of 

Congress, exists to support Congress in meeting its constitutional responsibilities 

and to help improve the performance and accountability of the federal 

government for the American people. GAO examines the use of public funds; 

evaluates federal programs and policies; and provides analyses, 

recommendations, and other assistance to help Congress make informed 

oversight, policy, and funding decisions. GAO’s commitment to good government 

is reflected in its core values of accountability, integrity, and reliability. 

The fastest and easiest way to obtain copies of GAO documents at no cost is 

through the Internet. GAO’s Web site (www.gao.gov) contains abstracts and fulltext files of current reports and testimony and an expanding archive of older 

products. The Web site features a search engine to help you locate documents 

using key words and phrases. You can print these documents in their entirety, 

including charts and other graphics. 

Each day, GAO issues a list of newly released reports, testimony, and 

correspondence. GAO posts this list, known as “Today’s Reports,” on its Web site 

daily. The list contains links to the full-text document files. To have GAO e-mail 

this list to you every afternoon, go to www.gao.gov and select “Subscribe to e-mail 

alerts” under the “Order GAO Products” heading. 

The first copy of each printed report is free. Additional copies are $2 each. A 

check or money order should be made out to the Superintendent of Documents. 

GAO also accepts VISA and Mastercard. Orders for 100 or more copies mailed to a 

single address are discounted 25 percent. Orders should be sent to: 

U.S. General Accounting Office 

441 G Street NW, Room LM 

Washington, D.C. 20548 

To order by Phone: Voice: (202) 512-6000 

TDD: (202) 512-2537 

Fax: (202) 512-6061 

Contact: 

Web site: www.gao.gov/fraudnet/fraudnet.htm 

E-mail: fraudnet@gao.gov

Automated answering system: (800) 424-5454 or (202) 512-7470 

Jeff Nelligan, Managing Director, NelliganJ@gao.gov (202) 512-4800 

U.S. General Accounting Office, 441 G Street NW, Room 7149 

Washington, D.C. 20548 

GAO’s Mission 

Obtaining Copies of 

GAO Reports and 

Testimony 

Order by Mail or Phone 

To Report Fraud, 

Waste, and Abuse in 

Federal Programs 

Public Affairs