Foster Care Program: Improved Processes Needed to Estimate Improper Payments and Evaluate Related Corrective Actions
Released: 2012-03-28T00:00:00


Highlights: section_title: Why GAO Did This Study
paragraphs: Each year, hundreds of thousands of the nation’s most vulnerable children are removed from their homes and placed in foster care. While states are primarily responsible for providing safe and stable out-of-home care for these children, Title IV-E of the Social Security Act provides federal financial support. The Administration for Children and Families (ACF) in the Department of Health and Human Services (HHS) is responsible for administering and overseeing federal funding for Foster Care. Past work by the HHS Office of Inspector General (OIG), GAO, and others have identified numerous financial deficiencies associated with the Title IV-E Foster Care program. GAO was asked to determine the extent to which (1) ACF’s estimation methodology generated a reasonably accurate and complete estimate of improper payments across the Foster Care program and (2) ACF’s corrective actions reduced Foster Care program improper payments. To complete this work, GAO reviewed HHS’s fiscal year 2010 improper payments estimation procedures, conducted site visits, and met with cognizant ACF officials.

section_title: What GAO Found
paragraphs: Although ACF has established a process to calculate a national improper payment estimate for the Foster Care program, the estimate is not based on a statistically valid methodology and consequently does not reflect a reasonably accurate estimate of the extent of Foster Care improper payments. In addition, the estimate deals with only about one-third of the federal expenditures for Foster Care, and is therefore incomplete. ACF’s methodology for estimating Foster Care improper payments was approved by the Office of Management and Budget (OMB) in 2004 with the understanding that continuing efforts would be taken to improve the accuracy of ACF’s estimates of improper payments in the ensuing years. ACF, however, continued to generally follow its initial methodology which, when compared to federal statistical guidance and internal control standards, GAO found to be deficient in all three phases of ACF’s estimation methodology—planning, selection, and evaluation. These deficiencies impaired the accuracy and completeness of the Foster Care program improper payments estimate of $73 million reported for fiscal year 2010.

ACF has reported significantly reduced estimated improper payments for its Foster Care maintenance payments, from a baseline of 10.33 percent for fiscal year 2004 to a 4.9 percent error rate for fiscal year 2010. However, the validity of ACF’s reporting of reduced error rates is questionable. GAO found that ACF’s ability to reliably assess the extent to which its corrective actions reduced improper payments was impaired by weaknesses in its requirements for state-level corrective actions. For example, ACF used the number of cases found in error rather than the dollar amount of improper payments identified to determine whether or not a state was required to implement corrective actions. As such, some states with higher improper payment dollar error rates were not required to implement actions to reduce these rates. GAO also found deficiencies in ACF’s Audit Resolution Tracking and Monitoring System that limited its ability to efficiently track and compare trends across states’ Single Audit findings.

section_title: What GAO Recommends
paragraphs: GAO is making seven recommendations to help improve ACF’s methodology for estimating improper payments for the Foster Care program and its corrective action process. HHS agreed that its improper payment estimation efforts can and should be improved, and generally concurred with four of the recommendations and agreed to continue to study the remaining three recommendations. GAO reaffirms the need for all seven recommendations.

section_title: Letter
paragraphs: Each year, hundreds of thousands of our nation’s most vulnerable  children are removed from their homes and placed in foster care, often  due to abuse or neglect. While states are primarily responsible for  providing safe and stable out-of-home care for these children until they  are returned safely home, placed with adoptive families, or placed in other  arrangements, Title IV-E of the Social Security Act provides states some  federal financial support in this area. The Administration for Children and  Families (ACF) under the Department of Health and Human Services  (HHS) is responsible for administering this program and overseeing Title  IV-E funds. HHS’s reported fiscal year 2010 outlays to states for their  Foster Care programs under Title IV-E totaled more than $4.5 billion  serving over 408,000 children as of September 30, 2010.

Past work by the HHS Office of Inspector General (OIG), GAO, and  others have identified numerous deficiencies in state claims associated  with the Title IV-E Foster Care program. In particular, the HHS OIG found  hundreds of millions of dollars in unallowable claims associated with Title  IV-E funding. A 2006 GAO report also found variations in costs states  claimed under the Title IV-E program and recommended a number of  actions HHS should take to better safeguard federal resources. In  addition, annual state-level audits have identified weaknesses in states’  use of federal funds, such as spending on unallowed activities or costs  and inadequate state monitoring of federal funding.

Pub. L. No. 107-300, 116 Stat. 2350 (Nov. 26, 2002). IPIA was amended by the Improper  Payments Elimination and Recovery Act of 2010, Pub. L. No. 111-204, 124 Stat. 2224  (July 22, 2010). The changes made by this law to IPIA estimation and reporting were first  implemented for fiscal year 2011 reporting.

Foster Care program since 2005. In its fiscal year 2010 agency financial  report, HHS reported estimated improper payments for Foster Care of  about $73 million, which contributed to the total governmentwide  improper payment estimate of over $125 billion for that year.

Given continuing concerns about financial vulnerabilities in the Foster  Care program, you asked us to conduct a review of the program’s  improper payment estimate and reduction strategies. Specifically, our  objectives were to (1) determine the extent to which ACF’s estimation  methodology generated a reasonably accurate and complete estimate of  Foster Care improper payments and (2) determine the extent to which  ACF’s corrective actions reduced improper payments.

To address these objectives, we reviewed applicable improper payment  legislation, Title IV-E of the Social Security Act, as amended, and HHS  regulations on Title IV-E eligibility reviews, related Office of Management  and Budget (OMB) guidance and ACF’s internal guidance, results from  Title IV-E eligibility reviews, past GAO and OIG reports, and internal  control standards. We also reviewed improper payment information  reported in the Improper Payments Section of HHS’s fiscal year 2010  agency financial report.

To further address the first objective, we performed independent  assessments of ACF’s methodology, including statistical sampling  analysis using OMB’s statistical guidance, GAO guidance, and federal  internal control standards as criteria to determine the accuracy and  completeness of ACF’s reported fiscal year 2010 improper payment  estimate for the Foster Care program.conducted interviews with ACF officials and its contractor to clarify our  understanding of the methodology. We also obtained information about  ACF’s pilot to develop a methodology for estimating improper payments  related to administrative costs for the Title IV-E Foster Care program. The  scope of our review did not include an assessment of individual states’  processes or payment systems.

As part of this assessment, we  To further address the second objective, we reviewed agency policies  and procedures and states’ Program Improvement Plans (PIP) used to  address the root causes of improper payments identified from the Title IV- E eligibility reviews, which ACF conducts to ensure that states are  claiming federal reimbursement only for eligible children; evaluated the  compliance thresholds ACF uses to require states to implement corrective  actions; and conducted interviews with officials from ACF’s central office.  We also obtained information about other ACF monitoring activities over  states, such as its tracking and monitoring of states’ efforts to address the  findings identified in the Single Audits.

In addition, we conducted site visits at 3 of ACF’s 10 regional offices  (Philadelphia, PA; Chicago, IL; and San Francisco, CA) where we  interviewed ACF staff on how Title IV-E eligibility reviews are conducted  and how they work with states to implement corrective actions and follow  up on Single Audit findings. We selected these three regional offices for  site visits because they provided oversight of states that collectively  claimed over half of the total federal share of Foster Care maintenance  payments made in fiscal year 2009, the most recent data available at the  time of our review for site visit selection. We also selected these regional  offices to achieve variation in the numbers of error cases and amount of  disallowed claims found during Title IV-E eligibility reviews.

We conducted this performance audit from February 2011 through March  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives. See appendix I for  additional details on our objectives, scope, and methodology.
subsections: section_title: Background
subsections: section_title: Foster Care Program
paragraphs: Title IV-E of the Social Security Act provides for states to obtain federal  reimbursement for the costs of their Foster Care programs. While states  may provide foster care services to a range of children outlined by state  laws and regulations, they may only claim Title IV-E Foster Care funds for  children meeting eligibility criteria outlined in the Social Security Act (see  table 1).

Title IV-E authorizes states to receive federal reimbursement for  “maintenance payments” to support expenses for a foster care child, such  as food, clothing, shelter, and school supplies. The federal government  matches the amounts states pay for maintenance costs under the  Medicaid rate. The Medicaid rate varies by state and by year and, for  fiscal year 2010, ranged from 50 to 83 percent. In addition to  maintenance costs, Title IV-E authorizes states to receive reimbursement  for other costs incurred to manage the program. Those other costs and   fall under the following three main  the allowable reimbursement ratescategories:

Child placement services and other administrative activities  (administrative costs), which generally cover expenses states incur in  identifying eligible children, referring them to services, and planning  for permanent placement. These can also include administrative costs  used to serve foster care “candidate” children, who are at-risk for  foster care but still reside in the home. These costs are matched at 50  percent.

State and local training costs (training costs), which are matched at 75  percent.

Statewide Automated Child Welfare Information System (SACWIS)  development, installation, and operation costs (SACWIS costs).  SACWIS helps states manage their child welfare cases and report  related information to the federal government. These SACWIS costs  are matched at 50 percent.

Since 2002, HHS has also approved states to receive federal  reimbursement for demonstration project costs involving the waiver of  certain provisions of Title IV-E. The waivers grant states flexibility in the  use of Title IV-E foster care funds for “demonstration projects” of  alternative services that promote safety, permanency, and well-being for  children in the foster care system, so long as the projects do not cost the  federal government more than the states would have received under Title  IV-E. As of June 2010, nine states have active Title IV-E waiver  agreements.

section_title: Program Trends and Costs
paragraphs: Data from HHS shows that the average number of children served by  Title IV-E Foster Care funds has declined, from over 197,000 in fiscal  year 2008 to 181,000 in fiscal year 2010.

HHS and child welfare experts have cited a number of reasons for this  decline. For example, they noted that a child is required to qualify for the  Aid to Families with Dependent Children program (a means-tested  program based on a federally defined poverty line) as it was in effect on  July 16, 1996, in order to be eligible for Title IV-E. Because income limits  for the program have remained static while inflation has raised nominal  incomes for some families, fewer children are eligible. For example, to be  considered as below the federal poverty line, a family comprised of 4  persons, including 2 children, had to have an annual income below  $15,911 in 1996, as compared to $22,113 in 2010. However, the $15,911  threshold continues to be used each year to determine eligibility for the  Title IV-E Foster Care program. In addition, states sometimes use other  federal programs for children who could otherwise have been claimed  under Title IV-E, because the other programs, such as Title XX’s Social  Services Block Grants, Medicaid, and Temporary Assistance to Needy  Families, provide federal reimbursement for a broader range of  services.

Of the $4.5 billion in total Title IV-E Foster Care funds paid to states in  fiscal year 2010, ACF reported that maintenance costs made up 34  percent of the total while administrative costs accounted for the largest  share of the costs at 44 percent. Figure 1 shows federal outlays, as  reported by HHS, by type of expenditure for fiscal year 2010. Title IV-E  expenditures by type and by state for fiscal year 2010 are presented in  appendix II.

ACF is responsible for the administration and oversight of Title IV-E  funding to states. ACF staff are located in ACF’s headquarters (Central  Office) and its 10 regional offices. Collectively, these ACF offices  oversee states’ financial internal control processes for the Title IV-E  program and monitor their performance and compliance with federal child  welfare laws. One key oversight activity related to state Foster Care  programs is ACF’s Title IV-E eligibility reviews, as required under the   ACF has conducted these  Social Security Act and HHS regulations.reviews since 2000. They are intended to help evaluate whether state  claims for federal reimbursement for Foster Care maintenance costs are  valid and accurate. Title IV-E eligibility reviews are to be conducted by  teams composed of both federal and state staff, and are to include (1)  desk reviews to ensure that the correct amount of maintenance costs was  claimed on behalf of foster care children during the review period and (2)  site visits to states to ensure that maintenance costs were claimed only  for children who were eligible for the Title IV-E program.

As required by the Social Security Act and HHS regulations, there are two  stages of Title IV-E eligibility reviews, a primary and secondary review.  During a primary review, HHS regulations specify that the review team is  to examine a sample of 80 cases per state, selected from the Adoption  and Foster Care Analysis and Reporting System (AFCARS). Each case  represents a child for whom a Title IV-E Foster Care maintenance  payment was made. If a primary review finds fewer than 5 cases with  errors in either the amounts paid on behalf of a child or in a child’s  eligibility for Title IV-E funds (5 percent of the cases reviewed or fewer),  ACF determines that the state is in substantial compliance with the  regulations. At that point, the state is scheduled to have another primary  review in 3 years.cases are in error (exceeding 5 percent of the number of cases  reviewed), ACF determines that the state is not in substantial compliance  with the regulations. In those instances, ACF requires states to develop a  Program Improvement Plan (PIP) designed to correct the areas of  noncompliance identified, such as payments to unlicensed providers or  incomplete criminal record checks. Any improper payments the review  teams identify during these reviews are classified as disallowed costs   On the other hand, if a primary review finds 5 or more  that, in general, are to be returned to ACF or withheld from future  reimbursement claims.

States required to develop a PIP generally have 1 year to implement the  corrective actions specified in the PIP, after which a secondary review is  to be conducted. During the secondary review, the review team is to  examine a sample of 150 cases, as outlined in HHS regulations. If 10  percent of cases or fewer are found to be in error and if the total dollar  amount found to be in error is less than 10 percent of the total dollar  amount reviewed, then ACF determines that the state is in substantial  compliance. Further, if the state exceeded only one of these secondary  review error thresholds, then ACF would also determine that the state is  in substantial compliance. Only in instances where the state exceeds both  the case percentage and dollar percentage error thresholds of 10 percent  would ACF determine that the state is not in substantial compliance. HHS  regulations require such a state to repay a disallowance percentage  applied to its Title IV-E claims during the review period. After conducting a  secondary review, ACF would then schedule another primary review in 3  years (see figure 2).

Another key ACF oversight activity related to state Foster Care programs  is the monitoring of findings from state-level audits conducted under the  Single Audit Act and OMB Circular No. A-133, known as Single Audits.  The Single Audit Act requires an annual audit of states, local  governments, and non-profit organizations that expend $500,000 or more  of federal funds in a given year. ACF regional offices are to work with  states to resolve Single Audit findings related to the Foster Care program  to help ensure that states are using funds in accordance with program  requirements and addressing financial management weaknesses. ACF  started using the Audit Resolution Tracking and Monitoring System  (ARTMS) in 2010 to provide online processing and real-time tracking of  ACF’s audit follow-up process. The National External Audit Review  Center is a specialized function of the HHS OIG that serves as a  clearinghouse to determine which state Single Audit report findings HHS  is responsible for resolving. When ACF receives Single Audit finding data  from the OIG’s National External Audit Review Center, HHS headquarters  staff upload the data into ARTMS and assign the audit finding data to the  appropriate ACF regional office staff for resolution. Consistent with OMB  Circular No. A-50, ACF considers an audit finding resolved when the  auditor and the state agree on action to be taken. ARTMS provides  users with notifications of tasks to be performed, such as when an audit is  assigned to a financial management specialist for follow-up, and allows  them to submit and view all audit resolution information online.

section_title: Improper Payments Information Act of 2002 and Implementing Guidance
paragraphs: The Improper Payments Information Act (IPIA) was enacted in November  2002 to enhance the accuracy and integrity of federal payments.requires agencies to:  review all programs and activities and identify those that are  susceptible to significant improper payments; obtain a statistically valid estimate of the annual amount of improper  payments, including the gross total of over- and underpayments, in  those susceptible programs and activities;  report to the Congress estimates of the annual amount of improper  payments in their susceptible programs and activities and;  for estimates exceeding $10 million, implement a plan to reduce  improper payments.

OMB’s implementing guidance for IPIA, in effect for fiscal year 2010  reporting, required that for any programs and activities identified as  susceptible to significant improper payments, agencies must develop a  statistically valid methodology, or other methodology approved in  advance by OMB, to estimate the annual amount of improper payments,  including a gross total of both underpayments and overpayments. The  Foster Care program was deemed a risk susceptible program and  therefore required to address the IPIA reporting requirements.guidance also requires that, as part of their plan to reduce improper  payments for all programs and activities with improper payments  exceeding $10 million, agencies identify the reasons their programs and  activities are at risk of improper payments (also known as root causes),  set reduction targets for future improper payment levels and a timeline  within which the targets will be reached, and ensure that agency  managers and accountable officers are held accountable for reducing  improper payments.

ACF annually reports to HHS—for inclusion in its agency financial report  used to report to the Congress—an improper payment estimate for Foster  Care program maintenance payments based on results of Title IV-E  eligibility reviews required under the Social Security Act. For programs  administered at the state level such as Foster Care, OMB guidance  provides that statistically valid estimates of improper payments may be  provided at the state level either for all states or for all sampled states  annually. These state-level improper payment estimates should then be  used to generate a national dollar estimate and improper payment rate.  With prior OMB approval, ACF has taken its existing Title IV-E eligibility  review process, already in place under the Social Security Act, and  leveraged it for IPIA estimation. OMB granted this approval in December  2004 with the expectation that continuing efforts would be taken to  improve the accuracy of ACF’s estimates of improper payments in the  ensuing years. ACF provides a national estimated error rate based on a  rolling average of error rates identified in states examined on a 3-year  cycle. As a result, ACF’s IPIA reporting for each year is based on new  data for about one-third of the states and previous years’ data for the  remaining two-thirds of states. While each state sample represents a  distinct 6-month period under review, the national “composite sample”  reflects a composite period under review that encompasses a 3-year  period (the Title IV-E eligibility review 3-year improper payment cyclical  process). To calculate a national estimate of improper payments, ACF  uses error rates that span a 3-year period of Title IV-E eligibility reviews in  the 50 states, the District of Columbia, and Puerto Rico. ACF applies the  percentage dollar error rate from the sample to the total payments for the  period under review for each state. Improper payment error rates by state  for fiscal year 2010, as calculated by ACF, can be found in appendix III.

On July 22, 2010, the Improper Payments Elimination and Recovery Act  of 2010 (IPERA) was enacted. IPERA amended IPIA, and established  additional requirements related to federal agency management  accountability, compliance and noncompliance determinations based on  an Inspector General’s assessment of an agency’s adherence to IPERA  requirements and reporting that determination, and an opinion on internal  controls over improper payments. Specifically, one new IPERA provision  calls for federal agencies’ Inspectors General to annually determine  whether their respective agencies are in compliance with key IPERA  requirements such as meeting annual reduction targets for each program  assessed to be at risk of and measured for improper payments, and to  report on their determinations to the agency head, the Congress, and the  Comptroller General.

section_title: ACF’s Methodology for Estimating Foster Care Program Improper Payments Is Not Statistically Valid or Complete
paragraphs: ACF’s methodology, which resulted in a reported $73 million (or 4.9  percent) estimate of improper payments in the Foster Care program for   had deficiencies in all three phases of its estimation  fiscal year 2010,methodology—planning, selection, and evaluation—when compared to  OMB’s statistical guidance, GAO guidance, and federal internal control  standards, as summarized in table 2. Specifically, ACF’s estimation  methodology (1) did not consider nearly two-thirds of reported federal  Foster Care program payments for fiscal year 2010, (2) was not based on  a probability sample of payments, (3) lacked specific procedures for  identifying underpayments and duplicate payment errors, and (4) used a  flawed process for aggregating state-level data into an overall national  error rate. As a result, ACF’s methodology is not statistically valid or  complete, and these deficiencies impair the accuracy and completeness  of its reported Foster Care program improper payment estimate.

ACF’s annual IPIA reporting for the Foster Care program is incomplete,  as it is limited to identifying improper payments for only one type of  program payment activity—maintenance payments. For fiscal year 2010,  as shown in figure 1 of this report, maintenance payments represented 34  percent of the total federal share of expenditures for the Foster Care  program. Administrative and other payments were not considered in  ACF’s IPIA estimation process and thus, not included in the Foster Care  program improper payment estimate of about $73 million for fiscal year  2010. Administrative costs accounted for 44 percent of the total federal  share of expenditures for the Foster Care program, while other costs  accounted for the remaining 22 percent. These other costs include  operational and development costs associated with SACWIS; training  costs; and state demonstration projects to provide alternative services  and support for children in the Foster Care system. Figure 3 shows the  portion of Foster Care program outlays considered in ACF’s methodology  for estimating associated improper payments for IPIA reporting.

Because ACF’s methodology does not include an estimate for improper  payments related to its administrative payment activity, the related  payment errors that meet the definition of improper payments were not  accounted for or included in the reported estimate for the Foster Care  program. OMB’s December 2004 approval of ACF’s proposed  methodology included an expectation that ACF would develop a plan and  timetable to test administrative expenses by April 2005.

Consistent with this expectation, in order to begin exploring the issues of  accounting for and including administrative costs, ACF established a  working group in 2006. Then, in 2007, ACF initiated an Administrative  Cost Review (ACR) pilot to examine how certain state agencies  accumulate costs that are included in their expenditure claims for federal  financial participation and to identify improper administrative payments  within those pilot states.

Seven states volunteered for pilots held from  fiscal years 2007 through 2011, and two more states are scheduled for  fiscal year 2012. Pilot reports for two states have provided estimates for  a gross improper payment total of $11.3 million for the period October 1,  2008, through March 31, 2009. These amounts were not included in  ACF’s estimated amounts for improper payments. According to ACF, it  will use the results of the ACR pilots to determine the feasibility of  developing a methodology to estimate an administrative error rate as part  of the calculation of the national Foster Care improper payment error rate.  However, as of December 15, 2011, ACF had not yet made a decision  with respect to when these reviews would be implemented and,  ultimately, whether to establish a methodology to estimate improper  payments related to administrative costs.

Although ACF did not consider Foster Care administrative expenditures in  its fiscal year 2010 IPIA estimation process, its Title IV-E eligibility  reviews identified disallowed administrative costs (or improper payment  amounts) which were added to the amount of any claims disallowances.  For fiscal year 2010, disallowed administrative costs that ACF  documented from the Title IV-E eligibility reviews totaled $2.4 million;  however, this amount was not included in its Foster Care improper  payment estimate. According to ACF, administrative payments are not  currently included as part of the reported improper payment estimate  because this disallowed amount is based on a calculation and not directly  determined from a case file review. ACF calculates the administrative  cost disallowance by allocating an average administrative cost for any  ineligible time period identified during a Title IV-E eligibility review.
subsections: section_title: ACF’s Methodology for Selecting Cases to Review Is Also Deficient
paragraphs: The methodology ACF used to estimate improper maintenance payments  was not based on a probability sample of payments, which is needed for  a direct estimate of the payment error rate and total amount of dollars that  were improperly paid.approaches for estimating improper payments in the Foster Care   In 2004, ACF proposed three options to OMB as  program. OMB approved ACF’s plan to derive the estimate using error  dollars per case from state review samples for its base error rate  calculation, with the expectation that continuing attention to the statistical  processes used would be needed to obtain the best estimate of  erroneous payment rates. OMB’s approval reflected the idea that the  methods initially used would incorporate annual improvements to the  accuracy of improper payment estimates. However, other than a change  in 2008 to derive the estimate using sample state dollar error rates, ACF  generally continues to use the same methodology outlined in 2004.

We found that ACF selected a sample from a universe of all cases  receiving Title IV-E Foster Care payments during the period under review.  This population of Foster Care cases is drawn from AFCARS. However,  AFCARS does not contain any Title IV-E financial data that links a Title  IV-E payment amount to each case file. Lacking such payment data, ACF  relies on states to provide payment histories for all cases selected for  review. According to ACF, it was already using AFCARS to select  samples for the Title IV-E eligibility reviews prior to IPIA implementation.  Consistent with OMB’s approval of its methodology, ACF opted to use  AFCARS data instead of creating or identifying a new data source for  meeting IPIA requirements. ACF officials stated that utilizing this existing  source of data reduced the burden on states by not requiring them to  draw their own samples and employed the AFCARS database in a  practical manner.

Notwithstanding ACF’s objective of leveraging existing data to estimate  foster care improper payments for addressing IPIA requirements, if ACF  is to make an estimate that accurately represents the target population of  Title IV-E payments, a more direct statistical method would be needed to  select a probability sample of Title IV-E payments. OMB’s Standards and  Guidelines for Statistical Surveys documents the professional principles  and practices that federal agencies are required to adhere to and the  level of quality and effort expected in all statistical activities. According  to these standards and guidelines, probabilistic methods for survey  sampling are one of a variety of methods for sampling that give a known,  non-zero, probability of selection to each member of the target population.  The advantage of probabilistic sampling methods is that sampling error  can be calculated. For the purpose of making a valid estimate with a  measurable sampling error that represents a population, the sample must  be selected using probabilistic methods. The sample results can then be  used to make an inference about the target population, in this instance,  foster care cases that received a maintenance payment.

While it is possible for ACF to estimate a payment error rate and the total  amount of dollars improperly paid for maintenance payments using a  combination of AFCARS and supplemental payment data from the states,  this would require a more complex estimation methodology than ACF  currently uses. Based on our review of the sampling documentation  provided, ACF did not consider key factors such as variation in volume of  payments and dollars of payments across cases and states. In addition,  the population of data that ACF used to select its sample from is not  reliable because ACF’s sampling methodology did not provide for up-front  data quality control procedures to (1) ensure that the population of cases  was complete prior to its sample selection and (2) identify inaccuracies in  the data field used for sample selection.

During our review, we found that the population of Foster Care cases  from AFCARS contained inaccurate information on whether a case had  actually received a Title IV-E Foster Care maintenance payment during  the period under review, reflecting continuing concerns regarding the  accuracy and completeness of AFCARS data. Specifically, ACF had to  replace a high percentage of cases sampled from the database of Foster  Care cases for the fiscal year 2010 reporting period due to inaccurate  information in AFCARS. To ensure that a sufficient number of relevant  sample items are available for review, ACF routinely selects an “over- sample” of cases—cases selected in addition to the required 80 or 150  cases initially selected for the primary and secondary reviews.

Of the original 4,570 sample cases ACF selected for testing in its primary  and secondary reviews for fiscal year 2010, 298 cases (almost 7 percent)  had to be replaced with substitutes taken from the “over-sampled” cases  because the selected cases had not received Title IV-E Foster Care  maintenance payments during the period under review. Of the 298 over- sampled cases used to replace the cases initially selected, 63 cases  (more than 21 percent) then had to be replaced again because those  cases had also not received Title IV-E Foster Care maintenance  payments during the period under review. Although we were able to  determine how many sampled (or over-sampled) cases had to be  replaced because available records showed no Title IV-E payment was  received during the reporting period, neither we nor ACF were able to  determine the extent to which the opposite occurred—the extent to which  cases that had received a payment (and therefore should have been  included in the sample population) had not been coded as receiving Title  IV-E payments. As part of its sampling methodology, ACF has not  established procedures to identify any such occurrences. Therefore, ACF  could not determine whether its sampling universe was complete, i.e.,  whether all of the cases receiving a Foster Care payment were included  in the universe of cases from which it selected sample cases for review.

According to GAO’s Assessing the Reliability of Computer-Processed  Data, reliable data are defined as data that are reasonably complete and  accurate, meet intended purposes, and are not subject to inappropriate  alteration. “Completeness” refers to the extent to which relevant records  are present and the fields in each record are populated appropriately.  “Accuracy” refers to the extent to which recorded data reflect the actual  underlying information. GAO’s Internal Control Management and  Evaluation Tool provides that reconciliations should be performed to  verify data completeness. Also, data validation and editing should be  performed to identify erroneous data. Erroneous data should be captured,  reported, investigated, and promptly corrected.

ACF officials told us they are aware that AFCARS does not contain Title  IV-E payment data and acknowledged that they do not perform  procedures to identify incorrect or missing information in the population  prior to sample selection. However, ACF officials said they continue to  use the data to meet IPIA reporting requirements because it is the only  database that contains case-level information on all children in foster care  for whom the state child welfare agencies have responsibility for  placement, care, or supervision. However, without developing a  statistically valid sampling methodology that incorporates up-front data  quality controls to ensure complete and accurate information on the  population, including payment data, ACF cannot provide assurance that  its reported improper payment estimate accurately and completely  represents the extent of improper maintenance payments in the Foster  Care program.

section_title: ACF’s Methodology Does Not Include Procedures to Identify the Full Extent of Errors Related to Underpayments and Duplicate or Excessive Payments
paragraphs: In its fiscal year 2010 agency financial report, ACF reported that  underpayments and duplicate or excessive payments represented 25  percent of the errors that caused improper payments. While ACF’s  methodology for performing Title IV-E eligibility reviews included written  guidance and a data collection instrument that focused on eligibility  errors, it did not include procedures on how to search for and identify  payment errors related to underpayments and duplicate or excessive  payments during case reviews. Rather, ACF’s procedures only provided  that any observed underpayments and duplicate or excessive payments  are to be disclosed as findings in the state’s final eligibility review report.  Without detailed procedures to guide review teams in the identification of  underpayments and duplicate or excessive payments, ACF’s  methodology cannot effectively assure its review team identifies the full  extent to which any such underpayments or duplicate or excessive  payments exist in the Foster Care program.

As defined in IPIA, improper payments include both overpayments and  underpayments, and an agency’s estimate should reflect both types of  errors. IPIA also includes examples of improper payments, one of which  is duplicate payments. According to GAO’s Standards for Internal Control  in the Federal Government, operational information is needed to  determine whether the agency is achieving its compliance requirements  under various laws and regulations. Information is required on a day-to-day basis, to make operating decisions, monitor performance, and  allocate resources. Pertinent information should be identified, captured,  and distributed in a form and time frame that permits people to perform  their duties efficiently.

ACF compiles the results of all state eligibility reviews to determine the  most common types of payment errors. ACF officials told us that all  review team members receive the same training and the results of the  state reviews are analyzed to ensure consistency and reliability. For fiscal  year 2010, underpayments were the largest percentage of payment errors  (19 percent—as a percentage of all Title IV-E maintenance payment  errors identified in sampled cases). Duplicate or excessive payments  comprised 6 percent of the payment errors. However, the extent of  underpayments and duplicate or excessive payment errors identified  varied widely by state, and in some instances were not identified at all.  The lack of detailed procedures for identifying any such payment errors  may have contributed to the variation or whether the teams found any  errors. For example, our analysis of the Title IV-E eligibility reviews that  comprised the fiscal year 2010 foster care improper payment estimate  identified underpayments in 21 of 51 state reviews. Of the 21 states  where reviewers had identified underpayments, such payments ranged  from 1.3 percent to 12.0 percent of cases reviewed. Similarly, duplicate or  excessive payments were identified in only 16 of 51 states. Of the 16  states that had this type of error, these payments ranged from 1.3 percent  to 5.0 percent.

During our site visits, ACF regional officials told us that states have  differing claiming practices for certain expenses. Specifically, officials in  one regional office said that if a child became eligible during a specific  month, a state could have claimed through the first day of the month for  that child, but chose not to so as to not risk a potential error on a future  Title IV-E eligibility review. According to these regional officials, the  regional offices operate under the presumption that “if a state made the  decision not to claim certain expenses, then the failure to claim is not  considered an underpayment.” These types of decisions would be  discussed during the reviews but ACF guidance does not call for  decisions and their rationale to be formally documented. According to  ACF’s Title IV-E eligibility review guide, potential underpayments are to  be identified during a review of the case record and payment history.  ACF’s Title IV-E eligibility review guide provides that payment histories  should be submitted, but it does not specify what criteria reviewers are to  look for in order to determine instances of underpayments or duplicate or  excessive payments. In August 2011, ACF issued a new attachment to  the Title IV-E eligibility review guide to provide a tool for calculating and  reporting underpayments identified during eligibility reviews. This new  attachment provides a template for recording underpayments for the  period under review, but it does not provide guidance on how to identify  underpayments. Instances of duplicate or excessive payments are to be  reported on other existing attachments in the Title IV-E eligibility review  guide. However, none of these attachments offer additional guidance for  how to identify national underpayments or duplicate or excessive  payments.

section_title: ACF’s Aggregation Methodology Does Not Generate a National Foster Care Program Improper Payment Estimate
paragraphs: ACF’s fiscal year 2010 Foster Care program improper payment estimate  did not appropriately aggregate state improper payment data to derive a  national improper payment estimate (dollars and error rate). ACF  calculated the national estimate of improper payments each year using  data collected in the most recent eligibility review for each of 50 states,  the District of Columbia, and Puerto Rico. According to the information  ACF presented to OMB in December 2004, ACF’s methodology would  calculate the standard error of each state estimate, and of the national  estimate, to examine the extent to which the precision requirements as  specified in OMB’s implementing guidance for IPIA are met. However, the  methodology ACF actually used to aggregate this state-level improper  payment data does not take into account each state’s margin of error,  which is needed to calculate an overall program improper payment  estimate with a 90 percent confidence level generally required by OMB  guidance. Figure 4 depicts, at a high level, ACF’s calculation to derive the  national improper payment estimate for the Foster Care program.

section_title: Validity of Reported Foster Care Program Improper Payment Reductions is Questionable
paragraphs: ACF has reported significantly reduced estimated improper maintenance  payments, from a baseline error rate of 10.33 percent for 2004 to a 4.9  percent error rate for 2010, but the validity of ACF’s reporting of reduced  improper payment error rates is questionable. Examples of corrective  actions ACF has identified include reviews, the requirement for state  improvement plans, on-site training and technical assistance to states,  and outreach to judicial organizations to educate them as to their role in  addressing Foster Care eligibility issues. However, the significant  weaknesses discussed previously concerning ACF’s estimation  methodology impaired the accuracy and completeness of ACF’s reported  improper payment estimate for the Foster Care program. Further, we  found that ACF’s ability to reliably assess the extent to which its  corrective actions reduced Foster Care program improper payments was  impaired by deficiencies in (1) its method for requiring when states  implement corrective actions and (2) information technology limitations  related to monitoring states’ Foster Care program-related Single Audit  findings.
subsections: section_title: Deficiencies in ACF’s Requirements for State Foster Care Program Improper Payment Corrective Actions
paragraphs: We identified three deficiencies in ACF’s process for implementing plans  to reduce improper payments.

ACF did not use reported improper payment error rates—which are  based on the dollar amount of improper payments identified in a  sample of state Foster Care cases—to determine whether or not a  state is required to implement corrective actions.

ACF’s measure for assessing corrective action effectiveness is  through performance on its secondary review of Title IV-E cases,  which has a more lenient passing standard than the primary review.

Not all types of payment errors are required to be addressed in the  PIP.

OMB’s implementing guidance for IPIA requires that agencies put in place  a corrective action plan to reduce improper payments. In addition, ACF’s  internal guidance requires states to implement corrective actions through  a PIP if, during the Title IV-E primary eligibility review, a state is found to  have 5 or more cases in error (exceeding 5 percent of the number of  cases reviewed). While ACF identifies state PIPs as a corrective action  strategy, it does not use the dollar-based estimated improper payments to  determine when a state is required to develop a PIP. Instead, ACF uses  the number of sample cases found in error to determine which states  should develop a PIP. Therefore, some states with improper payment  dollar error rates exceeding 5 percent were not required to implement  corrective actions to reduce these rates. For fiscal year 2010 reporting,  ACF used the results of 44 primary eligibility reviews and 7 secondary  reviews. Of the 44 state primary reviews, 13 had dollar-based estimated  improper payments greater than 5 percent; however, because ACF uses  case error rates as the determining factor for states’ compliance with their  primary reviews, not all states were required to complete a PIP. Of the 13  states, ACF determined 7 were noncompliant in their primary eligibility  reviews because the case error rate exceeded ACF’s threshold of 5  percent (more than 4 of the 80 cases were found in error) and thus, were  required to complete a PIP. The remaining 6 states were found  substantially compliant in their primary reviews as their case error rate  was below the established 5 percent threshold (4 or fewer cases were  found in error). The dollar-based improper payment rates for those 6  states ranged from 5.1 to 19.8 percent—based on the percentage of  improper payment dollars found in the sample. Because improper  payment rates are not used in applying the PIP corrective action strategy,  ACF’s method cannot effectively measure states’ progress over time in  reducing improper payments. It also cannot effectively help determine  whether further action is needed to minimize future improper payments.  This limits the extent to which states are held accountable for the  reduction of improper payments in the Foster Care program.

Upon a state’s implementation of its PIP, ACF conducts a secondary  review to determine whether errors found during the primary review have  been addressed. The secondary review is ACF’s principal tool to measure  a state’s success in implementing actions to reduce Foster Care program  improper payments. These reviews carry the potential financial penalty of  an extrapolated disallowance of the state’s federal share of Title IV-E  expenditures if the state is found to be noncompliant. However, because  ACF’s error threshold to be found noncompliant with a secondary review  is twice as high as that of the primary review (10 percent versus 5  percent), it limits ACF’s ability to provide an effective incentive for states  to focus continuing attention on causes of improper payments. Based on  our analysis of ACF’s Title IV-E eligibility reviews, 27 states have had at  least one secondary review between 2002 and 2010. Of the 27 states that  received a secondary review, 26 states passed this review (meaning that  the error rates were below the ten percent threshold) and only 1 state  failed. Of the 26 states that passed, 13 states (50 percent) would have  failed if the primary review error threshold of 5 percent was in effect. Of  those 13 states, we found at least 3 states that passed the secondary  review with a case error rate over 10 percent because the reported  improper payment dollar based error rate was below 10 percent.

The one state that failed its secondary review in 2003 received an  extrapolated disallowance in accordance with HHS regulations. Since the  eligibility reviews began in 2000, this is the only state found to be  noncompliant with its secondary review. While the extrapolated  disallowance is a financial penalty intended to encourage states to  address causes of improper payments, after incurring an extrapolated  disallowance, this state was again found to be noncompliant based on  ACF’s subsequent review in 2006. As such, the state was again required  to develop and implement a PIP to address the causes of errors identified  in this review. After implementing this PIP, the state was subject to  another secondary review and was found to be compliant with a case  error rate of 6.67 percent and dollar error rate of 2.84 percent. However,  this state would have failed if the primary review error threshold of 5  percent was in effect.

According to ACF officials, it established the 10 percent threshold for  compliance with secondary reviews in 2000 based on states’ error rates  at that time which were between 15 percent and 17 percent (in terms of  both cases and dollars). ACF officials told us the 10 percent appeared to  be a target that states could meet to demonstrate reductions in improper  payments over time. Also, the baseline estimated improper payment error  rate reported for the Foster Care program in 2004 was 10.33 percent.  Since establishing the 10 percent threshold in 2000, ACF has not  conducted a review to validate the continuing propriety of the  performance metric. GAO’s Internal Control Management and Evaluation  Tool provides that an agency should periodically review and validate the  propriety and integrity of both organizational and individual performance  measures and indicators. According to this tool, performance  measurement factors are to be evaluated to ensure they are linked to  mission, goals, and objectives, and that they are balanced and set  appropriate incentives for achieving goals while complying with law,  regulations, and ethical standards. For fiscal year 2010, ACF reported  that error rates for most of the states (33 of 51) were less than 5  percent.

In addition, ACF’s process for overseeing states’ implementation of  improper payment reduction actions has other weaknesses. ACF’s  guidance only requires that the PIP—required if a state has more than  four cases found in error in its primary review—address areas that the  eligibility review identified as needing improvement. Consequently, states’  corrective action plans may not address all types of previously identified  payment errors. There is nothing in the guidance to prevent states from  addressing other areas in the PIP. However, based on our review of the  guidance for developing PIPs and discussions with Central Office and  regional office staff, underpayments and other non-eligibility errors and  eligibility errors outside of the period under review might not be  addressed in the PIP if these types of errors were not a factor in a state’s  compliance. Not including all types of errors in states’ corrective action  plans reduces their effectiveness for addressing the causes of payment  errors as required under IPIA.

section_title: Limitations of ACF’s Audit Tracking System Impair Monitoring Corrective Actions
paragraphs: In 2010, ACF began using a departmentwide system, ARTMS, to track  and monitor the resolution of audit findings for programs it administers,  including audit findings concerning Foster Care program payment  errors.resolution of reported Single Audit findings. Single Audit findings for  states’ Foster Care programs have included, among other issues,  deficiencies in state oversight over subrecipients of federal funds, lack of  training of state personnel on program eligibility requirements, and  potential for unauthorized access to information systems to create and  approve cases. Single Audit reports generally include a summary of prior   ACF utilizes ARTMS as its primary tool for monitoring states’  audit findings describing any recurring issues and any related corrective  actions undertaken by the state agency. Single Audits also generally  provide information about any deficiencies in state agencies’ systems and  processes that can be useful for ACF in monitoring federal expenditures  and identifying and reducing improper payments in the Foster Care  program.

According to ACF officials, ARTMS is designed to track and monitor the  resolution of Single Audit findings by audit report number, but the system  does not enable users to search for specific audit findings by type of  finding, grantee, state, region, or across years. As a result, regional  offices could not use ARTMS to examine trends in the types of findings in  their states in order to ensure that any systemic issues are addressed.  Limitations with ARTMS decrease ACF’s ability to leverage existing  agency data to identify reoccurring issues and other vulnerabilities such  as inadequate state monitoring of federal funding that might not be  identified during the 3-year eligibility review process and could lead to  improper payments. This lack of information could impair ACF’s and  regional offices’ ability to effectively monitor states’ efforts to reduce  improper payments and the effectiveness of corrective action strategies  implemented. According to GAO’s Standards for Internal Control in the  Federal Government, information should be recorded and communicated  to management and others within the entity who need it and in a form and  within a time frame that enables them to carry out their internal control  and other responsibilities.

ACF regional office officials acknowledged limitations with ARTMS related  to functionality in tracking findings. Specifically, a regional office official  told us that ARTMS was not designed to be able to generate reports of all  audit findings for an entire ACF region. Lacking such capability, a user  interested in aggregating the findings would have to obtain the audit  findings from each state and manually combine them outside of ARTMS.  This regional office used a separate internal spreadsheet to track  information related to the audit findings for all states in its purview.  Otherwise, staff would need to view each state’s Single Audit findings  individually within ARTMS, which could be time-consuming. Another  regional office we visited used a similar spreadsheet, which also included  information on the time it takes to close findings. A third regional office we  visited also utilized an off-line spreadsheet as a means to track the  clearance process for closing audit findings.

section_title: Conclusions
paragraphs: A statistically valid approach for estimating improper payments would  help ensure that Foster Care program improper payment estimates are  reasonably accurate and complete, reflecting all types of program  payments including administrative costs, based on complete and accurate  payment data, and aggregated using state-level margins of error.  Developing and implementing a sound methodology is a critical program  management tool for understanding and addressing financial  vulnerabilities in the Foster Care program through approaches such as  identifying underpayments and duplicate or excessive payment errors  consistently across states. While ACF has reported an improper payment  estimate and related reductions for the Foster Care program, the  statistical validity of both is questionable. Further, ACF’s method for  evaluating the effectiveness of states’ implementation of their corrective  action plans has several significant weaknesses, including reliance on  ineffective and dated metrics that do not consider states’ improper  payment dollar error rates in conjunction with targets that have not been  reassessed since 2000.

Similarly, deficiencies in its system for monitoring Single Audit findings  limit ACF’s ability to efficiently track and compare trends across states.  This includes ACF’s ability to measure states’ progress in reducing their  improper payment errors, as well as its ability to reliably and completely  identify and correct vulnerabilities at the state level that could lead to  improper payments. Although OMB’s approval reflected a stated plan for  ACF to implement a process to annually improve the accuracy of its  improper payment estimate, this has not resulted in substantial changes  to the process ACF outlined in 2004. Given the financial accountability  challenges reported for state-administered programs, the ongoing  imbalance between revenues and outlays across the federal government,  and increasing demands for accountability over taxpayer funds, improving  ACF’s ability to identify, reduce, and recover improper payments is  critical. It will be important for ACF to work closely with OMB in examining  and updating its statistical procedures to help assure the validity of ACF’s  estimates.

section_title: Recommendations for Executive Action
paragraphs: In order to more accurately and completely estimate improper payments  for the Foster Care program and ensure that its methodology is  statistically valid, we recommend that the Secretary of Health and Human  Services direct the Assistant Secretary for the Administration for Children  and Families to take the following four actions: augment procedures for estimating and reporting Foster Care  program improper payments, to include administrative costs; develop and implement procedures to provide a statistically valid  methodology for estimating and reporting Foster Care program  improper payments based on complete and accurate payment data; augment guidance to teams gathering state-level Foster Care  program improper payment estimate data to include specific  procedures to follow in identifying and reporting any underpayments  and duplicate or excessive payment errors; and  revise existing procedures for calculating a national improper payment  estimate for the Foster Care program to include a statistically valid  method for aggregating state-level margins of error to derive an  overall, inflation adjusted, program estimate.

To help ensure corrective action strategies effectively reduce Foster Care  program improper payments, we recommend that the Secretary of Health  and Human Services direct the Assistant Secretary for the Administration  for Children and Families to take the following three actions: develop and implement procedures requiring states to implement and  report on corrective actions whenever a state’s estimated improper  payment dollar error rate exceeds a specified target level for the  program; establish and implement procedures requiring periodic assessments  of state-level improper payment target levels, including targets  associated with Title IV-E secondary reviews, for the Foster Care  program for which states are to implement and report on corrective  actions; and enhance ARTMS reporting capabilities to provide data on the status of  actions taken to address Single Audit findings concerning states’  Foster Care program payments, such as providing reporting  capabilities to allow ARTMS users to search for specific audit findings  by type of finding, grantee, state, region, or across years.

section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to the Secretary of Health and Human  Services for comment. In its written comments, reprinted in appendix IV,  HHS agreed that its improper payment estimation efforts can and should  be improved. HHS provided a summary of refinements that it had made to  its improper payment estimation methodology over the years and also  provided information on additional steps it planned to take. HHS stated  that our analysis would be a helpful resource as it continued to improve  its process. With regard to our seven recommendations to help improve  ACF’s methodology to estimate improper payments and its corrective  action process, HHS generally concurred with four of the  recommendations and agreed to continue to study the remaining three  recommendations. HHS also provided technical comments that we  incorporated, as appropriate.

HHS generally concurred with three recommendations we made related  to improving the improper payment estimation methodology for the Foster  Care program. Specifically, HHS generally agreed to (1) estimate and  report improper payments related to administrative costs, (2) provide  specific procedures to identify and report any underpayments and  duplicate or excessive payment errors, and (3) revise procedures for  calculating the aggregate state-level margins of error to derive an overall,  inflation adjusted, program estimate. HHS described several actions  currently under way to address these recommendations. Regarding the  first recommendation, HHS noted that it was continuing to pilot test the  Administrative Cost Reviews, described in this report, in fiscal year 2012;  however, HHS’s response did not indicate when it expects these reviews  will be fully implemented. In its response to the second recommendation,  HHS stated that additional guidance for identifying and reporting  underpayments and duplicate or excessive payment errors will be  included in the updated Eligibility Review Guide and review instrument  during the fiscal year 2012 review cycle. For the third recommendation,  HHS stated that it can and will adjust its calculation to incorporate  individual state margins of error in aggregating the state-level estimates  into the national program estimate. However, HHS also stated that it will  seek to determine whether making this revision would add sufficient value  given that the estimate spans 3 years and that inflation is relatively low.  We maintain that both aggregating state-level margins of error and  factoring for inflation are needed to implement a statistically valid method  for estimating improper payments.

For the other recommendation we made related to the improper payment  estimation methodology for the Foster Care program, HHS stated it would  continue to study our recommendation to develop and implement a  statistically valid Foster Care improper payment methodology based on  complete and accurate payment data. HHS agreed that it should use the  best data available. In its comments on our draft report, HHS  acknowledged that it would be optimal to conduct a separate data  collection to obtain a universe of Title IV-E payments, but stated that it  needs to balance the goal of appropriate measurement with the cost and  burden placed on states. HHS described the quality controls in place over  the AFCARS data to help ensure the information is complete and  accurate prior to selecting case samples for its Title IV-E eligibility  reviews, which form the basis for its Foster Care improper payment    estimate. Examples of such controls include automated system edit  checks within AFCARS, AFCARS Assessment Reviews, and other  outreach efforts to improve state AFCARS reporting. Our report describes  some of the steps ACF has taken to address AFCARS data quality, but  we also point out limitations in these efforts. For example, the AFCARS  Assessment Reviews are not conducted annually for all states and do not  address verifying the accuracy or completeness of the specific data  element that ACF uses to develop its population of foster care cases for  estimating improper payments. HHS also stated that its use of  oversample cases demonstrated that its sampling and oversampling  process is working properly to exclude cases that do not meet the  selection criteria. While this process would identify some cases that did  not meet ACF’s selection criteria, our point in this report is that ACF’s  extensive reliance on the use of oversampling in its methodology is  indicative that the population of cases could contain additional  inaccuracies that may not be identified through its existing process.  Further, as we stated in our report, neither we nor ACF were able to  determine the completeness of the universe of cases used to estimate  Foster Care improper payments, that is, whether all cases that had  actually received a Title IV-E payment were properly coded as such.  Thus, the issues we identified with ACF’s sampling methodology,  provides limited assurance that the reported improper payment estimate  accurately and completely represents the extent of improper maintenance  payments in the Foster Care program.

With respect to our three recommendations to help ensure corrective  strategies effectively reduce Foster Care improper payments, HHS  concurred with one recommendation related to establishing and  implementing procedures for periodic assessments of state-level  improper payment target levels. HHS also agreed to consider another  recommendation to develop and implement corrective action procedures  whenever a state’s estimated improper payment dollar error rate exceeds  a specified target level for the program in conjunction with the  recommendation it concurred with to implement periodic assessments.  HHS highlighted several actions it plans to take to enhance its efforts to  reduce improper payments, such as taking steps to reexamine and  explore the feasibility for lowering the error rate threshold and considering  ways to enhance existing Eligibility Review Guide instructions to address  any eligibility review findings that involve improper payments not  specifically requiring development of a corrective action plan. HHS stated  that it plans to further study our recommendation to enhance ARTMS  reporting capabilities to provide data on the status of actions taken to  address Single Audit findings concerning states’ Foster Care program  payments. HHS stated that the agency would study the value of potential  enhancements to ARTMS in light of the significant relevant data already  available (such as audit information by federal program, grantee name,  audit resolution status, and audit periods). Although these data elements  are currently available in ARTMS, search results are presented by  individual audit reports and include limited information. For example, the  search results provide only the number of findings associated with a  specific audit report but do not provide details of the individual audit  findings that would allow program managers to analyze, for example,  trends in the types of findings in states in order to ensure that any  systemic issues are addressed. We reaffirm our recommendation to  ensure that ACF is able to fully utilize ARTMS as a tool to analyze Single  Audit findings.

As agreed with your offices, unless you publicly announce the contents of  this report earlier, we plan no further distribution until 30 days from the  report date. At that time, we will send copies to the appropriate  congressional committees; the Secretary of Health and Human Services;  and other interested parties. In addition, the report will be available at no  charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact  me at (202) 512-8486 or raglands@gao.gov. Contact points for our  Offices of Public Affairs and Congressional Relations may be found on  the last page of this report. GAO staff who made key contributions to this  report are listed in appendix V.

section_title: Appendix I: Objectives, Scope, and Methodology
paragraphs: The objectives of this report were to (1) determine the extent to which the  Administration for Children and Families’ (ACF) estimation methodology  generated a reasonably accurate and complete estimate of improper  payments across the Foster Care program and (2) determine the extent  to which ACF’s corrective actions reduced improper payments. To  address these objectives, we reviewed the Improper Payments  Information Act of 2002 (IPIA) requirements and related Office of  Management and Budget (OMB) guidance effective for fiscal year 2010,  Department of Health and Human Services (HHS) regulations on Title IV- E eligibility reviews, and ACF’s internal guidance including policies and  procedures on conducting Title IV-E Foster Care eligibility reviews,  computing improper payments, implementing corrective action plans for  reducing improper payments, and monitoring and resolving audit findings  in the Foster Care program. We also reviewed results from the Title IV-E  eligibility reviews for periods 2000 through 2010, and prior GAO and  HHS Office of Inspector General (OIG) reports. In addition, we reviewed  improper payment information reported in HHS’s fiscal year 2010 agency  financial report, Improper Payments Section. We reviewed these  documents to understand ACF’s efforts to address IPIA requirements and  to identify previously reported issues with ACF’s improper payment  reporting.

To further determine the extent to which ACF’s methodology generated a  reasonably accurate and complete estimate of improper payments across  the Foster Care program, we:

Performed an independent analysis of ACF’s sampling methodology,  including a review of the sampling plan and other underlying  documentation, as well as evaluated whether ACF’s sampling  methodology complied with OMB statistical guidance, GAO guidance,  and federal internal control standards as criteria to determine the  accuracy and completeness of ACF’s reported fiscal year 2010  improper payment estimate for the Foster Care program. The scope  of our review did not include an assessment of individual states’  processes or payment systems that are the underlying data that ACF  uses to support the national estimate of Foster Care improper  payments.

Interviewed ACF officials such as the Acting Associate Commissioner  for the Children’s Bureau, its contractor, and staff at selected regional  offices such as program managers and financial specialists to gain an  understanding of (1) the methodology that ACF uses to estimate  improper payments in the Foster Care program in accordance with  IPIA requirements and (2) the administrative cost review pilot at five  states to develop a methodology for estimating related administrative  improper payments. We reviewed available reports for the five pilot  reviews to identify what information ACF obtained from these reviews.

To further determine the extent to which ACF’s corrective action process  reduced improper payments, we:

Reviewed ACF policies and procedures to gain an understanding of  reported corrective action strategies, including the Title IV-E eligibility  review process and development of states’ Program Improvement  Plans (PIP) used to address the root causes of improper payments,  which are identified from the Title IV-E eligibility reviews. We reviewed  applicable states’ PIPs for the period 2001 through 2010. We also  inquired of ACF officials from the Program Implementation Division  within the Children’s Bureau about other monitoring activities in place  for states that did not have a PIP in place for IPIA reporting in fiscal  year 2010.

Assessed compliance thresholds ACF uses to require states to  implement corrective actions against actual performance data to  assess the propriety of established performance measures. As part of  this analysis, we reviewed our internal control standards as guidance  to assess ACF’s evaluation of states’ efforts to implement corrective  actions.

Conducted a walkthrough of ACF’s Audit Resolution Tracking and  Monitoring System (ARTMS) to obtain an understanding of ACF’s  monitoring activities to track and resolve state’s Single Audit findings  for the Foster Care program. In addition, we interviewed officials in  ACF’s Office of Information Services and the Division of Financial  Integrity located in the Central Office and selected representative  regional offices such as Regional Program Managers, and program  and fiscal specialists to determine how ARTMS is used to identify and  correct vulnerabilities that could lead to improper payments.

Examined states’ reported Single Audit findings for fiscal years 2008  through 2010 from ARTMS and a listing of HHS OIG reports on the  Foster Care program to identify vulnerabilities or weaknesses in  states’ operations that may not have been identified through ACF’s  Title IV-E eligibility reviews.

Reviewed agency policies and procedures such as ACF’s Title IV-E  Foster Care Eligibility Review Guide issued in March 2006 which  includes the Title IV-E Foster Care Eligibility On-Site Review  Instrument and Instructions; ACF’s user guide for ARTMS, version  1.2; and ACF’s FY 2010 Corrective Action Plan to Reduce the  Estimate Rate of Improper Payments in the Foster Care Program,  dated November 12, 2010.

In addition, we conducted site visits to three of ACF’s ten regional  offices (Philadelphia, PA; Chicago, IL; and San Francisco, CA). These  three regional offices provided oversight of states who collectively  claimed over half of the total federal share of Foster Care payments  made in fiscal year 2009, the most recent data available at the time of  our review for site visit selection. We also selected these regional  offices to achieve variation in the numbers of error cases and amount  of disallowed claims found during Title IV-E eligibility reviews, which  ACF conducts to help ensure that states are claiming federal  reimbursement only for eligible children. One region represented the  highest number of error cases found in the Title IV-E eligibility reviews  and the highest maintenance payment disallowance. Another region  had the largest amount of foster care maintenance payments in fiscal  year 2009 and the states within this region had high improper  payment rates. Another region had a low number of error cases and  improper payment issues relative to the high amount of maintenance  payments it made to states in its purview. During these site visits, we  interviewed agency personnel such as program managers, regional  grants officers, and financial specialists to gain an understanding of  how Title IV-E eligibility reviews are conducted and how the regional  offices work with states on corrective actions and follow up on Single  Audit findings. We also inquired about other ACF monitoring activities  over states to address financial management weaknesses.

We conducted this performance audit from February 2011 through March  2012 in accordance with generally accepted government auditing  standards. Those standards require that we plan and perform the audit to  obtain sufficient, appropriate evidence to provide a reasonable basis for  our findings and conclusions, based on our audit objectives. We believe  that the evidence obtained provides a reasonable basis for our findings  and conclusions based on our audit objectives.

section_title: Appendix II: Title IV-E Expenditures by Type and by State for Fiscal Year 2010
subsections: section_title: Average monthly number of children 2,097
paragraphs: SACWIS  (operations  and  development)   $ 2,234,570    (662,979) (2,529,684)

section_title: Average monthly number of children 12,724
paragraphs: SACWIS  (operations  and  development)   5,193,320    (490,734)

section_title: Totals

section_title: Appendix III: Estimated Improper Payment Error Rates by State for Fiscal Year 2010 Reporting
subsections: section_title: State Ohio
paragraphs: According to ACF, Florida IV-E Reviews suspended pending completion of a statewide Foster Care  demonstration project and were therefore, not included in the national error rate.

section_title: Appendix IV: Comments from the Department of Health and Human Services

section_title: Appendix V: GAO Contact and Staff Acknowledgments
subsections: section_title: GAO Contact

section_title: Staff Acknowledgments
paragraphs: In addition to the contact named above, Carla Lewis, Assistant Director;  Betty Ward-Zukerman, Assistant Director; Sharon Byrd, Assistant  Director; Sophie Brown; Gabrielle Fagan; Vincent Gomes; and Nhi  Nguyen made key contributions to this report. Also contributing to this  report were Kay Brown; Wilfred Holloway; Francine DelVecchio; Jason  Kirwan; and Karen O’Conor.


Question-summary annotations: questions: question: What is the problem with the ACF national improper pay estimate for the Foster Care Program?
answer: Although ACF has established a process to calculate a national improper payment estimate for the Foster Care program, the estimate is not based on a statistically valid methodology and consequently does not reflect a reasonably accurate estimate of the extent of Foster Care improper payments.
child_questions: question: Why is the estimate incomplete?
answer: In addition, the estimate deals with only about one-third of the federal expenditures for Foster Care, and is therefore incomplete.

question: Under what condition was the estimate approved?
answer: ACF’s methodology for estimating Foster Care improper payments was approved by the Office of Management and Budget (OMB) in 2004 with the understanding that continuing efforts would be taken to improve the accuracy of ACF’s estimates of improper payments in the ensuing years.

question: What deficiencies did GAO find?
answer: ACF, however, continued to generally follow its initial methodology which, when compared to federal statistical guidance and internal control standards, GAO found to be deficient in all three phases of ACF’s estimation methodology—planning, selection, and evaluation.
child_questions: question: How were these deficiencies problematic?
answer: These deficiencies impaired the accuracy and completeness of the Foster Care program improper payments estimate of $73 million reported for fiscal year 2010.

questions: question: What has ACF reported from 2004 to 2010?
answer: ACF has reported significantly reduced estimated improper payments for its Foster Care maintenance payments, from a baseline of 10.33 percent for fiscal year 2004 to a 4.9 percent error rate for fiscal year 2010.
child_questions: question: Why is ACF's report questionable?
answer: GAO found that ACF’s ability to reliably assess the extent to which its corrective actions reduced improper payments was impaired by weaknesses in its requirements for state-level corrective actions.
child_questions: question: How is ACF's report questionable?
answer: For example, ACF used the number of cases found in error rather than the dollar amount of improper payments identified to determine whether or not a state was required to implement corrective actions.

question: What deficiencies did ACF find?
answer: GAO also found deficiencies in ACF’s Audit Resolution Tracking and Monitoring System that limited its ability to efficiently track and compare trends across states’ Single Audit findings.

questions: question: What role do states play in foster care?
answer: While states are primarily responsible for providing safe and stable out-of-home care for these children, Title IV-E of the Social Security Act provides federal financial support.

question: What role does ACF in HHS have?
answer: The Administration for Children and Families (ACF) in the Department of Health and Human Services (HHS) is responsible for administering and overseeing federal funding for Foster Care.

question: What was GAO asked to determine?
answer: GAO was asked to determine the extent to which (1) ACF’s estimation methodology generated a reasonably accurate and complete estimate of improper payments across the Foster Care program and (2) ACF’s corrective actions reduced Foster Care program improper payments.
child_questions: question: How did GAO get information for its task?
answer: To complete this work, GAO reviewed HHS’s fiscal year 2010 improper payments estimation procedures, conducted site visits, and met with cognizant ACF officials.