Foster Care Program: Improved Processes Needed to Estimate Improper Payments and Evaluate Related Corrective Actions
Released: 2012-03-28T00:00:00

Letter

Background

Foster Care Program

Title IV-E of the Social Security Act provides for states to obtain federal reimbursement for the costs of their Foster Care programs. While states may provide foster care services to a range of children outlined by state laws and regulations, they may only claim Title IV-E Foster Care funds for children meeting eligibility criteria outlined in the Social Security Act (see table 1).

Title IV-E authorizes states to receive federal reimbursement for “maintenance payments” to support expenses for a foster care child, such as food, clothing, shelter, and school supplies. The federal government matches the amounts states pay for maintenance costs under the Medicaid rate. The Medicaid rate varies by state and by year and, for fiscal year 2010, ranged from 50 to 83 percent. In addition to maintenance costs, Title IV-E authorizes states to receive reimbursement for other costs incurred to manage the program. Those other costs and fall under the following three main the allowable reimbursement ratescategories:

Child placement services and other administrative activities (administrative costs), which generally cover expenses states incur in identifying eligible children, referring them to services, and planning for permanent placement. These can also include administrative costs used to serve foster care “candidate” children, who are at-risk for foster care but still reside in the home. These costs are matched at 50 percent.

State and local training costs (training costs), which are matched at 75 percent.

Statewide Automated Child Welfare Information System (SACWIS) development, installation, and operation costs (SACWIS costs). SACWIS helps states manage their child welfare cases and report related information to the federal government. These SACWIS costs are matched at 50 percent.

Since 2002, HHS has also approved states to receive federal reimbursement for demonstration project costs involving the waiver of certain provisions of Title IV-E. The waivers grant states flexibility in the use of Title IV-E foster care funds for “demonstration projects” of alternative services that promote safety, permanency, and well-being for children in the foster care system, so long as the projects do not cost the federal government more than the states would have received under Title IV-E. As of June 2010, nine states have active Title IV-E waiver agreements.

Program Trends and Costs

Data from HHS shows that the average number of children served by Title IV-E Foster Care funds has declined, from over 197,000 in fiscal year 2008 to 181,000 in fiscal year 2010.

HHS and child welfare experts have cited a number of reasons for this decline. For example, they noted that a child is required to qualify for the Aid to Families with Dependent Children program (a means-tested program based on a federally defined poverty line) as it was in effect on July 16, 1996, in order to be eligible for Title IV-E. Because income limits for the program have remained static while inflation has raised nominal incomes for some families, fewer children are eligible. For example, to be considered as below the federal poverty line, a family comprised of 4 persons, including 2 children, had to have an annual income below $15,911 in 1996, as compared to $22,113 in 2010. However, the $15,911 threshold continues to be used each year to determine eligibility for the Title IV-E Foster Care program. In addition, states sometimes use other federal programs for children who could otherwise have been claimed under Title IV-E, because the other programs, such as Title XX’s Social Services Block Grants, Medicaid, and Temporary Assistance to Needy Families, provide federal reimbursement for a broader range of services.

Of the $4.5 billion in total Title IV-E Foster Care funds paid to states in fiscal year 2010, ACF reported that maintenance costs made up 34 percent of the total while administrative costs accounted for the largest share of the costs at 44 percent. Figure 1 shows federal outlays, as reported by HHS, by type of expenditure for fiscal year 2010. Title IV-E expenditures by type and by state for fiscal year 2010 are presented in appendix II.

ACF is responsible for the administration and oversight of Title IV-E funding to states. ACF staff are located in ACF’s headquarters (Central Office) and its 10 regional offices. Collectively, these ACF offices oversee states’ financial internal control processes for the Title IV-E program and monitor their performance and compliance with federal child welfare laws. One key oversight activity related to state Foster Care programs is ACF’s Title IV-E eligibility reviews, as required under the ACF has conducted these Social Security Act and HHS regulations.reviews since 2000. They are intended to help evaluate whether state claims for federal reimbursement for Foster Care maintenance costs are valid and accurate. Title IV-E eligibility reviews are to be conducted by teams composed of both federal and state staff, and are to include (1) desk reviews to ensure that the correct amount of maintenance costs was claimed on behalf of foster care children during the review period and (2) site visits to states to ensure that maintenance costs were claimed only for children who were eligible for the Title IV-E program.

As required by the Social Security Act and HHS regulations, there are two stages of Title IV-E eligibility reviews, a primary and secondary review. During a primary review, HHS regulations specify that the review team is to examine a sample of 80 cases per state, selected from the Adoption and Foster Care Analysis and Reporting System (AFCARS). Each case represents a child for whom a Title IV-E Foster Care maintenance payment was made. If a primary review finds fewer than 5 cases with errors in either the amounts paid on behalf of a child or in a child’s eligibility for Title IV-E funds (5 percent of the cases reviewed or fewer), ACF determines that the state is in substantial compliance with the regulations. At that point, the state is scheduled to have another primary review in 3 years.cases are in error (exceeding 5 percent of the number of cases reviewed), ACF determines that the state is not in substantial compliance with the regulations. In those instances, ACF requires states to develop a Program Improvement Plan (PIP) designed to correct the areas of noncompliance identified, such as payments to unlicensed providers or incomplete criminal record checks. Any improper payments the review teams identify during these reviews are classified as disallowed costs On the other hand, if a primary review finds 5 or more that, in general, are to be returned to ACF or withheld from future reimbursement claims.

States required to develop a PIP generally have 1 year to implement the corrective actions specified in the PIP, after which a secondary review is to be conducted. During the secondary review, the review team is to examine a sample of 150 cases, as outlined in HHS regulations. If 10 percent of cases or fewer are found to be in error and if the total dollar amount found to be in error is less than 10 percent of the total dollar amount reviewed, then ACF determines that the state is in substantial compliance. Further, if the state exceeded only one of these secondary review error thresholds, then ACF would also determine that the state is in substantial compliance. Only in instances where the state exceeds both the case percentage and dollar percentage error thresholds of 10 percent would ACF determine that the state is not in substantial compliance. HHS regulations require such a state to repay a disallowance percentage applied to its Title IV-E claims during the review period. After conducting a secondary review, ACF would then schedule another primary review in 3 years (see figure 2).

Another key ACF oversight activity related to state Foster Care programs is the monitoring of findings from state-level audits conducted under the Single Audit Act and OMB Circular No. A-133, known as Single Audits. The Single Audit Act requires an annual audit of states, local governments, and non-profit organizations that expend $500,000 or more of federal funds in a given year. ACF regional offices are to work with states to resolve Single Audit findings related to the Foster Care program to help ensure that states are using funds in accordance with program requirements and addressing financial management weaknesses. ACF started using the Audit Resolution Tracking and Monitoring System (ARTMS) in 2010 to provide online processing and real-time tracking of ACF’s audit follow-up process. The National External Audit Review Center is a specialized function of the HHS OIG that serves as a clearinghouse to determine which state Single Audit report findings HHS is responsible for resolving. When ACF receives Single Audit finding data from the OIG’s National External Audit Review Center, HHS headquarters staff upload the data into ARTMS and assign the audit finding data to the appropriate ACF regional office staff for resolution. Consistent with OMB Circular No. A-50, ACF considers an audit finding resolved when the auditor and the state agree on action to be taken. ARTMS provides users with notifications of tasks to be performed, such as when an audit is assigned to a financial management specialist for follow-up, and allows them to submit and view all audit resolution information online.

Improper Payments Information Act of 2002 and Implementing Guidance

The Improper Payments Information Act (IPIA) was enacted in November 2002 to enhance the accuracy and integrity of federal payments.requires agencies to: review all programs and activities and identify those that are susceptible to significant improper payments; obtain a statistically valid estimate of the annual amount of improper payments, including the gross total of over- and underpayments, in those susceptible programs and activities; report to the Congress estimates of the annual amount of improper payments in their susceptible programs and activities and; for estimates exceeding $10 million, implement a plan to reduce improper payments.

OMB’s implementing guidance for IPIA, in effect for fiscal year 2010 reporting, required that for any programs and activities identified as susceptible to significant improper payments, agencies must develop a statistically valid methodology, or other methodology approved in advance by OMB, to estimate the annual amount of improper payments, including a gross total of both underpayments and overpayments. The Foster Care program was deemed a risk susceptible program and therefore required to address the IPIA reporting requirements.guidance also requires that, as part of their plan to reduce improper payments for all programs and activities with improper payments exceeding $10 million, agencies identify the reasons their programs and activities are at risk of improper payments (also known as root causes), set reduction targets for future improper payment levels and a timeline within which the targets will be reached, and ensure that agency managers and accountable officers are held accountable for reducing improper payments.

ACF annually reports to HHS—for inclusion in its agency financial report used to report to the Congress—an improper payment estimate for Foster Care program maintenance payments based on results of Title IV-E eligibility reviews required under the Social Security Act. For programs administered at the state level such as Foster Care, OMB guidance provides that statistically valid estimates of improper payments may be provided at the state level either for all states or for all sampled states annually. These state-level improper payment estimates should then be used to generate a national dollar estimate and improper payment rate. With prior OMB approval, ACF has taken its existing Title IV-E eligibility review process, already in place under the Social Security Act, and leveraged it for IPIA estimation. OMB granted this approval in December 2004 with the expectation that continuing efforts would be taken to improve the accuracy of ACF’s estimates of improper payments in the ensuing years. ACF provides a national estimated error rate based on a rolling average of error rates identified in states examined on a 3-year cycle. As a result, ACF’s IPIA reporting for each year is based on new data for about one-third of the states and previous years’ data for the remaining two-thirds of states. While each state sample represents a distinct 6-month period under review, the national “composite sample” reflects a composite period under review that encompasses a 3-year period (the Title IV-E eligibility review 3-year improper payment cyclical process). To calculate a national estimate of improper payments, ACF uses error rates that span a 3-year period of Title IV-E eligibility reviews in the 50 states, the District of Columbia, and Puerto Rico. ACF applies the percentage dollar error rate from the sample to the total payments for the period under review for each state. Improper payment error rates by state for fiscal year 2010, as calculated by ACF, can be found in appendix III.

On July 22, 2010, the Improper Payments Elimination and Recovery Act of 2010 (IPERA) was enacted. IPERA amended IPIA, and established additional requirements related to federal agency management accountability, compliance and noncompliance determinations based on an Inspector General’s assessment of an agency’s adherence to IPERA requirements and reporting that determination, and an opinion on internal controls over improper payments. Specifically, one new IPERA provision calls for federal agencies’ Inspectors General to annually determine whether their respective agencies are in compliance with key IPERA requirements such as meeting annual reduction targets for each program assessed to be at risk of and measured for improper payments, and to report on their determinations to the agency head, the Congress, and the Comptroller General.

ACF’s Methodology for Estimating Foster Care Program Improper Payments Is Not Statistically Valid or Complete

ACF’s methodology, which resulted in a reported $73 million (or 4.9 percent) estimate of improper payments in the Foster Care program for had deficiencies in all three phases of its estimation fiscal year 2010,methodology—planning, selection, and evaluation—when compared to OMB’s statistical guidance, GAO guidance, and federal internal control standards, as summarized in table 2. Specifically, ACF’s estimation methodology (1) did not consider nearly two-thirds of reported federal Foster Care program payments for fiscal year 2010, (2) was not based on a probability sample of payments, (3) lacked specific procedures for identifying underpayments and duplicate payment errors, and (4) used a flawed process for aggregating state-level data into an overall national error rate. As a result, ACF’s methodology is not statistically valid or complete, and these deficiencies impair the accuracy and completeness of its reported Foster Care program improper payment estimate.

ACF’s annual IPIA reporting for the Foster Care program is incomplete, as it is limited to identifying improper payments for only one type of program payment activity—maintenance payments. For fiscal year 2010, as shown in figure 1 of this report, maintenance payments represented 34 percent of the total federal share of expenditures for the Foster Care program. Administrative and other payments were not considered in ACF’s IPIA estimation process and thus, not included in the Foster Care program improper payment estimate of about $73 million for fiscal year 2010. Administrative costs accounted for 44 percent of the total federal share of expenditures for the Foster Care program, while other costs accounted for the remaining 22 percent. These other costs include operational and development costs associated with SACWIS; training costs; and state demonstration projects to provide alternative services and support for children in the Foster Care system. Figure 3 shows the portion of Foster Care program outlays considered in ACF’s methodology for estimating associated improper payments for IPIA reporting.

Because ACF’s methodology does not include an estimate for improper payments related to its administrative payment activity, the related payment errors that meet the definition of improper payments were not accounted for or included in the reported estimate for the Foster Care program. OMB’s December 2004 approval of ACF’s proposed methodology included an expectation that ACF would develop a plan and timetable to test administrative expenses by April 2005.

Consistent with this expectation, in order to begin exploring the issues of accounting for and including administrative costs, ACF established a working group in 2006. Then, in 2007, ACF initiated an Administrative Cost Review (ACR) pilot to examine how certain state agencies accumulate costs that are included in their expenditure claims for federal financial participation and to identify improper administrative payments within those pilot states.

Seven states volunteered for pilots held from fiscal years 2007 through 2011, and two more states are scheduled for fiscal year 2012. Pilot reports for two states have provided estimates for a gross improper payment total of $11.3 million for the period October 1, 2008, through March 31, 2009. These amounts were not included in ACF’s estimated amounts for improper payments. According to ACF, it will use the results of the ACR pilots to determine the feasibility of developing a methodology to estimate an administrative error rate as part of the calculation of the national Foster Care improper payment error rate. However, as of December 15, 2011, ACF had not yet made a decision with respect to when these reviews would be implemented and, ultimately, whether to establish a methodology to estimate improper payments related to administrative costs.

Although ACF did not consider Foster Care administrative expenditures in its fiscal year 2010 IPIA estimation process, its Title IV-E eligibility reviews identified disallowed administrative costs (or improper payment amounts) which were added to the amount of any claims disallowances. For fiscal year 2010, disallowed administrative costs that ACF documented from the Title IV-E eligibility reviews totaled $2.4 million; however, this amount was not included in its Foster Care improper payment estimate. According to ACF, administrative payments are not currently included as part of the reported improper payment estimate because this disallowed amount is based on a calculation and not directly determined from a case file review. ACF calculates the administrative cost disallowance by allocating an average administrative cost for any ineligible time period identified during a Title IV-E eligibility review.

ACF’s Methodology for Selecting Cases to Review Is Also Deficient

The methodology ACF used to estimate improper maintenance payments was not based on a probability sample of payments, which is needed for a direct estimate of the payment error rate and total amount of dollars that were improperly paid.approaches for estimating improper payments in the Foster Care In 2004, ACF proposed three options to OMB as program. OMB approved ACF’s plan to derive the estimate using error dollars per case from state review samples for its base error rate calculation, with the expectation that continuing attention to the statistical processes used would be needed to obtain the best estimate of erroneous payment rates. OMB’s approval reflected the idea that the methods initially used would incorporate annual improvements to the accuracy of improper payment estimates. However, other than a change in 2008 to derive the estimate using sample state dollar error rates, ACF generally continues to use the same methodology outlined in 2004.

We found that ACF selected a sample from a universe of all cases receiving Title IV-E Foster Care payments during the period under review. This population of Foster Care cases is drawn from AFCARS. However, AFCARS does not contain any Title IV-E financial data that links a Title IV-E payment amount to each case file. Lacking such payment data, ACF relies on states to provide payment histories for all cases selected for review. According to ACF, it was already using AFCARS to select samples for the Title IV-E eligibility reviews prior to IPIA implementation. Consistent with OMB’s approval of its methodology, ACF opted to use AFCARS data instead of creating or identifying a new data source for meeting IPIA requirements. ACF officials stated that utilizing this existing source of data reduced the burden on states by not requiring them to draw their own samples and employed the AFCARS database in a practical manner.

Notwithstanding ACF’s objective of leveraging existing data to estimate foster care improper payments for addressing IPIA requirements, if ACF is to make an estimate that accurately represents the target population of Title IV-E payments, a more direct statistical method would be needed to select a probability sample of Title IV-E payments. OMB’s Standards and Guidelines for Statistical Surveys documents the professional principles and practices that federal agencies are required to adhere to and the level of quality and effort expected in all statistical activities. According to these standards and guidelines, probabilistic methods for survey sampling are one of a variety of methods for sampling that give a known, non-zero, probability of selection to each member of the target population. The advantage of probabilistic sampling methods is that sampling error can be calculated. For the purpose of making a valid estimate with a measurable sampling error that represents a population, the sample must be selected using probabilistic methods. The sample results can then be used to make an inference about the target population, in this instance, foster care cases that received a maintenance payment.

While it is possible for ACF to estimate a payment error rate and the total amount of dollars improperly paid for maintenance payments using a combination of AFCARS and supplemental payment data from the states, this would require a more complex estimation methodology than ACF currently uses. Based on our review of the sampling documentation provided, ACF did not consider key factors such as variation in volume of payments and dollars of payments across cases and states. In addition, the population of data that ACF used to select its sample from is not reliable because ACF’s sampling methodology did not provide for up-front data quality control procedures to (1) ensure that the population of cases was complete prior to its sample selection and (2) identify inaccuracies in the data field used for sample selection.

During our review, we found that the population of Foster Care cases from AFCARS contained inaccurate information on whether a case had actually received a Title IV-E Foster Care maintenance payment during the period under review, reflecting continuing concerns regarding the accuracy and completeness of AFCARS data. Specifically, ACF had to replace a high percentage of cases sampled from the database of Foster Care cases for the fiscal year 2010 reporting period due to inaccurate information in AFCARS. To ensure that a sufficient number of relevant sample items are available for review, ACF routinely selects an “over- sample” of cases—cases selected in addition to the required 80 or 150 cases initially selected for the primary and secondary reviews.

Of the original 4,570 sample cases ACF selected for testing in its primary and secondary reviews for fiscal year 2010, 298 cases (almost 7 percent) had to be replaced with substitutes taken from the “over-sampled” cases because the selected cases had not received Title IV-E Foster Care maintenance payments during the period under review. Of the 298 over- sampled cases used to replace the cases initially selected, 63 cases (more than 21 percent) then had to be replaced again because those cases had also not received Title IV-E Foster Care maintenance payments during the period under review. Although we were able to determine how many sampled (or over-sampled) cases had to be replaced because available records showed no Title IV-E payment was received during the reporting period, neither we nor ACF were able to determine the extent to which the opposite occurred—the extent to which cases that had received a payment (and therefore should have been included in the sample population) had not been coded as receiving Title IV-E payments. As part of its sampling methodology, ACF has not established procedures to identify any such occurrences. Therefore, ACF could not determine whether its sampling universe was complete, i.e., whether all of the cases receiving a Foster Care payment were included in the universe of cases from which it selected sample cases for review.

According to GAO’s Assessing the Reliability of Computer-Processed Data, reliable data are defined as data that are reasonably complete and accurate, meet intended purposes, and are not subject to inappropriate alteration. “Completeness” refers to the extent to which relevant records are present and the fields in each record are populated appropriately. “Accuracy” refers to the extent to which recorded data reflect the actual underlying information. GAO’s Internal Control Management and Evaluation Tool provides that reconciliations should be performed to verify data completeness. Also, data validation and editing should be performed to identify erroneous data. Erroneous data should be captured, reported, investigated, and promptly corrected.

ACF officials told us they are aware that AFCARS does not contain Title IV-E payment data and acknowledged that they do not perform procedures to identify incorrect or missing information in the population prior to sample selection. However, ACF officials said they continue to use the data to meet IPIA reporting requirements because it is the only database that contains case-level information on all children in foster care for whom the state child welfare agencies have responsibility for placement, care, or supervision. However, without developing a statistically valid sampling methodology that incorporates up-front data quality controls to ensure complete and accurate information on the population, including payment data, ACF cannot provide assurance that its reported improper payment estimate accurately and completely represents the extent of improper maintenance payments in the Foster Care program.

ACF’s Methodology Does Not Include Procedures to Identify the Full Extent of Errors Related to Underpayments and Duplicate or Excessive Payments

In its fiscal year 2010 agency financial report, ACF reported that underpayments and duplicate or excessive payments represented 25 percent of the errors that caused improper payments. While ACF’s methodology for performing Title IV-E eligibility reviews included written guidance and a data collection instrument that focused on eligibility errors, it did not include procedures on how to search for and identify payment errors related to underpayments and duplicate or excessive payments during case reviews. Rather, ACF’s procedures only provided that any observed underpayments and duplicate or excessive payments are to be disclosed as findings in the state’s final eligibility review report. Without detailed procedures to guide review teams in the identification of underpayments and duplicate or excessive payments, ACF’s methodology cannot effectively assure its review team identifies the full extent to which any such underpayments or duplicate or excessive payments exist in the Foster Care program.

As defined in IPIA, improper payments include both overpayments and underpayments, and an agency’s estimate should reflect both types of errors. IPIA also includes examples of improper payments, one of which is duplicate payments. According to GAO’s Standards for Internal Control in the Federal Government, operational information is needed to determine whether the agency is achieving its compliance requirements under various laws and regulations. Information is required on a day-to-day basis, to make operating decisions, monitor performance, and allocate resources. Pertinent information should be identified, captured, and distributed in a form and time frame that permits people to perform their duties efficiently.

ACF compiles the results of all state eligibility reviews to determine the most common types of payment errors. ACF officials told us that all review team members receive the same training and the results of the state reviews are analyzed to ensure consistency and reliability. For fiscal year 2010, underpayments were the largest percentage of payment errors (19 percent—as a percentage of all Title IV-E maintenance payment errors identified in sampled cases). Duplicate or excessive payments comprised 6 percent of the payment errors. However, the extent of underpayments and duplicate or excessive payment errors identified varied widely by state, and in some instances were not identified at all. The lack of detailed procedures for identifying any such payment errors may have contributed to the variation or whether the teams found any errors. For example, our analysis of the Title IV-E eligibility reviews that comprised the fiscal year 2010 foster care improper payment estimate identified underpayments in 21 of 51 state reviews. Of the 21 states where reviewers had identified underpayments, such payments ranged from 1.3 percent to 12.0 percent of cases reviewed. Similarly, duplicate or excessive payments were identified in only 16 of 51 states. Of the 16 states that had this type of error, these payments ranged from 1.3 percent to 5.0 percent.

During our site visits, ACF regional officials told us that states have differing claiming practices for certain expenses. Specifically, officials in one regional office said that if a child became eligible during a specific month, a state could have claimed through the first day of the month for that child, but chose not to so as to not risk a potential error on a future Title IV-E eligibility review. According to these regional officials, the regional offices operate under the presumption that “if a state made the decision not to claim certain expenses, then the failure to claim is not considered an underpayment.” These types of decisions would be discussed during the reviews but ACF guidance does not call for decisions and their rationale to be formally documented. According to ACF’s Title IV-E eligibility review guide, potential underpayments are to be identified during a review of the case record and payment history. ACF’s Title IV-E eligibility review guide provides that payment histories should be submitted, but it does not specify what criteria reviewers are to look for in order to determine instances of underpayments or duplicate or excessive payments. In August 2011, ACF issued a new attachment to the Title IV-E eligibility review guide to provide a tool for calculating and reporting underpayments identified during eligibility reviews. This new attachment provides a template for recording underpayments for the period under review, but it does not provide guidance on how to identify underpayments. Instances of duplicate or excessive payments are to be reported on other existing attachments in the Title IV-E eligibility review guide. However, none of these attachments offer additional guidance for how to identify national underpayments or duplicate or excessive payments.

ACF’s Aggregation Methodology Does Not Generate a National Foster Care Program Improper Payment Estimate

ACF’s fiscal year 2010 Foster Care program improper payment estimate did not appropriately aggregate state improper payment data to derive a national improper payment estimate (dollars and error rate). ACF calculated the national estimate of improper payments each year using data collected in the most recent eligibility review for each of 50 states, the District of Columbia, and Puerto Rico. According to the information ACF presented to OMB in December 2004, ACF’s methodology would calculate the standard error of each state estimate, and of the national estimate, to examine the extent to which the precision requirements as specified in OMB’s implementing guidance for IPIA are met. However, the methodology ACF actually used to aggregate this state-level improper payment data does not take into account each state’s margin of error, which is needed to calculate an overall program improper payment estimate with a 90 percent confidence level generally required by OMB guidance. Figure 4 depicts, at a high level, ACF’s calculation to derive the national improper payment estimate for the Foster Care program.

Validity of Reported Foster Care Program Improper Payment Reductions is Questionable

ACF has reported significantly reduced estimated improper maintenance payments, from a baseline error rate of 10.33 percent for 2004 to a 4.9 percent error rate for 2010, but the validity of ACF’s reporting of reduced improper payment error rates is questionable. Examples of corrective actions ACF has identified include reviews, the requirement for state improvement plans, on-site training and technical assistance to states, and outreach to judicial organizations to educate them as to their role in addressing Foster Care eligibility issues. However, the significant weaknesses discussed previously concerning ACF’s estimation methodology impaired the accuracy and completeness of ACF’s reported improper payment estimate for the Foster Care program. Further, we found that ACF’s ability to reliably assess the extent to which its corrective actions reduced Foster Care program improper payments was impaired by deficiencies in (1) its method for requiring when states implement corrective actions and (2) information technology limitations related to monitoring states’ Foster Care program-related Single Audit findings.

Deficiencies in ACF’s Requirements for State Foster Care Program Improper Payment Corrective Actions

We identified three deficiencies in ACF’s process for implementing plans to reduce improper payments.

ACF did not use reported improper payment error rates—which are based on the dollar amount of improper payments identified in a sample of state Foster Care cases—to determine whether or not a state is required to implement corrective actions.

ACF’s measure for assessing corrective action effectiveness is through performance on its secondary review of Title IV-E cases, which has a more lenient passing standard than the primary review.

Not all types of payment errors are required to be addressed in the PIP.

OMB’s implementing guidance for IPIA requires that agencies put in place a corrective action plan to reduce improper payments. In addition, ACF’s internal guidance requires states to implement corrective actions through a PIP if, during the Title IV-E primary eligibility review, a state is found to have 5 or more cases in error (exceeding 5 percent of the number of cases reviewed). While ACF identifies state PIPs as a corrective action strategy, it does not use the dollar-based estimated improper payments to determine when a state is required to develop a PIP. Instead, ACF uses the number of sample cases found in error to determine which states should develop a PIP. Therefore, some states with improper payment dollar error rates exceeding 5 percent were not required to implement corrective actions to reduce these rates. For fiscal year 2010 reporting, ACF used the results of 44 primary eligibility reviews and 7 secondary reviews. Of the 44 state primary reviews, 13 had dollar-based estimated improper payments greater than 5 percent; however, because ACF uses case error rates as the determining factor for states’ compliance with their primary reviews, not all states were required to complete a PIP. Of the 13 states, ACF determined 7 were noncompliant in their primary eligibility reviews because the case error rate exceeded ACF’s threshold of 5 percent (more than 4 of the 80 cases were found in error) and thus, were required to complete a PIP. The remaining 6 states were found substantially compliant in their primary reviews as their case error rate was below the established 5 percent threshold (4 or fewer cases were found in error). The dollar-based improper payment rates for those 6 states ranged from 5.1 to 19.8 percent—based on the percentage of improper payment dollars found in the sample. Because improper payment rates are not used in applying the PIP corrective action strategy, ACF’s method cannot effectively measure states’ progress over time in reducing improper payments. It also cannot effectively help determine whether further action is needed to minimize future improper payments. This limits the extent to which states are held accountable for the reduction of improper payments in the Foster Care program.

Upon a state’s implementation of its PIP, ACF conducts a secondary review to determine whether errors found during the primary review have been addressed. The secondary review is ACF’s principal tool to measure a state’s success in implementing actions to reduce Foster Care program improper payments. These reviews carry the potential financial penalty of an extrapolated disallowance of the state’s federal share of Title IV-E expenditures if the state is found to be noncompliant. However, because ACF’s error threshold to be found noncompliant with a secondary review is twice as high as that of the primary review (10 percent versus 5 percent), it limits ACF’s ability to provide an effective incentive for states to focus continuing attention on causes of improper payments. Based on our analysis of ACF’s Title IV-E eligibility reviews, 27 states have had at least one secondary review between 2002 and 2010. Of the 27 states that received a secondary review, 26 states passed this review (meaning that the error rates were below the ten percent threshold) and only 1 state failed. Of the 26 states that passed, 13 states (50 percent) would have failed if the primary review error threshold of 5 percent was in effect. Of those 13 states, we found at least 3 states that passed the secondary review with a case error rate over 10 percent because the reported improper payment dollar based error rate was below 10 percent.

The one state that failed its secondary review in 2003 received an extrapolated disallowance in accordance with HHS regulations. Since the eligibility reviews began in 2000, this is the only state found to be noncompliant with its secondary review. While the extrapolated disallowance is a financial penalty intended to encourage states to address causes of improper payments, after incurring an extrapolated disallowance, this state was again found to be noncompliant based on ACF’s subsequent review in 2006. As such, the state was again required to develop and implement a PIP to address the causes of errors identified in this review. After implementing this PIP, the state was subject to another secondary review and was found to be compliant with a case error rate of 6.67 percent and dollar error rate of 2.84 percent. However, this state would have failed if the primary review error threshold of 5 percent was in effect.

According to ACF officials, it established the 10 percent threshold for compliance with secondary reviews in 2000 based on states’ error rates at that time which were between 15 percent and 17 percent (in terms of both cases and dollars). ACF officials told us the 10 percent appeared to be a target that states could meet to demonstrate reductions in improper payments over time. Also, the baseline estimated improper payment error rate reported for the Foster Care program in 2004 was 10.33 percent. Since establishing the 10 percent threshold in 2000, ACF has not conducted a review to validate the continuing propriety of the performance metric. GAO’s Internal Control Management and Evaluation Tool provides that an agency should periodically review and validate the propriety and integrity of both organizational and individual performance measures and indicators. According to this tool, performance measurement factors are to be evaluated to ensure they are linked to mission, goals, and objectives, and that they are balanced and set appropriate incentives for achieving goals while complying with law, regulations, and ethical standards. For fiscal year 2010, ACF reported that error rates for most of the states (33 of 51) were less than 5 percent.

In addition, ACF’s process for overseeing states’ implementation of improper payment reduction actions has other weaknesses. ACF’s guidance only requires that the PIP—required if a state has more than four cases found in error in its primary review—address areas that the eligibility review identified as needing improvement. Consequently, states’ corrective action plans may not address all types of previously identified payment errors. There is nothing in the guidance to prevent states from addressing other areas in the PIP. However, based on our review of the guidance for developing PIPs and discussions with Central Office and regional office staff, underpayments and other non-eligibility errors and eligibility errors outside of the period under review might not be addressed in the PIP if these types of errors were not a factor in a state’s compliance. Not including all types of errors in states’ corrective action plans reduces their effectiveness for addressing the causes of payment errors as required under IPIA.

Limitations of ACF’s Audit Tracking System Impair Monitoring Corrective Actions

In 2010, ACF began using a departmentwide system, ARTMS, to track and monitor the resolution of audit findings for programs it administers, including audit findings concerning Foster Care program payment errors.resolution of reported Single Audit findings. Single Audit findings for states’ Foster Care programs have included, among other issues, deficiencies in state oversight over subrecipients of federal funds, lack of training of state personnel on program eligibility requirements, and potential for unauthorized access to information systems to create and approve cases. Single Audit reports generally include a summary of prior ACF utilizes ARTMS as its primary tool for monitoring states’ audit findings describing any recurring issues and any related corrective actions undertaken by the state agency. Single Audits also generally provide information about any deficiencies in state agencies’ systems and processes that can be useful for ACF in monitoring federal expenditures and identifying and reducing improper payments in the Foster Care program.

According to ACF officials, ARTMS is designed to track and monitor the resolution of Single Audit findings by audit report number, but the system does not enable users to search for specific audit findings by type of finding, grantee, state, region, or across years. As a result, regional offices could not use ARTMS to examine trends in the types of findings in their states in order to ensure that any systemic issues are addressed. Limitations with ARTMS decrease ACF’s ability to leverage existing agency data to identify reoccurring issues and other vulnerabilities such as inadequate state monitoring of federal funding that might not be identified during the 3-year eligibility review process and could lead to improper payments. This lack of information could impair ACF’s and regional offices’ ability to effectively monitor states’ efforts to reduce improper payments and the effectiveness of corrective action strategies implemented. According to GAO’s Standards for Internal Control in the Federal Government, information should be recorded and communicated to management and others within the entity who need it and in a form and within a time frame that enables them to carry out their internal control and other responsibilities.

ACF regional office officials acknowledged limitations with ARTMS related to functionality in tracking findings. Specifically, a regional office official told us that ARTMS was not designed to be able to generate reports of all audit findings for an entire ACF region. Lacking such capability, a user interested in aggregating the findings would have to obtain the audit findings from each state and manually combine them outside of ARTMS. This regional office used a separate internal spreadsheet to track information related to the audit findings for all states in its purview. Otherwise, staff would need to view each state’s Single Audit findings individually within ARTMS, which could be time-consuming. Another regional office we visited used a similar spreadsheet, which also included information on the time it takes to close findings. A third regional office we visited also utilized an off-line spreadsheet as a means to track the clearance process for closing audit findings.

Conclusions

A statistically valid approach for estimating improper payments would help ensure that Foster Care program improper payment estimates are reasonably accurate and complete, reflecting all types of program payments including administrative costs, based on complete and accurate payment data, and aggregated using state-level margins of error. Developing and implementing a sound methodology is a critical program management tool for understanding and addressing financial vulnerabilities in the Foster Care program through approaches such as identifying underpayments and duplicate or excessive payment errors consistently across states. While ACF has reported an improper payment estimate and related reductions for the Foster Care program, the statistical validity of both is questionable. Further, ACF’s method for evaluating the effectiveness of states’ implementation of their corrective action plans has several significant weaknesses, including reliance on ineffective and dated metrics that do not consider states’ improper payment dollar error rates in conjunction with targets that have not been reassessed since 2000.

Similarly, deficiencies in its system for monitoring Single Audit findings limit ACF’s ability to efficiently track and compare trends across states. This includes ACF’s ability to measure states’ progress in reducing their improper payment errors, as well as its ability to reliably and completely identify and correct vulnerabilities at the state level that could lead to improper payments. Although OMB’s approval reflected a stated plan for ACF to implement a process to annually improve the accuracy of its improper payment estimate, this has not resulted in substantial changes to the process ACF outlined in 2004. Given the financial accountability challenges reported for state-administered programs, the ongoing imbalance between revenues and outlays across the federal government, and increasing demands for accountability over taxpayer funds, improving ACF’s ability to identify, reduce, and recover improper payments is critical. It will be important for ACF to work closely with OMB in examining and updating its statistical procedures to help assure the validity of ACF’s estimates.

Recommendations for Executive Action

In order to more accurately and completely estimate improper payments for the Foster Care program and ensure that its methodology is statistically valid, we recommend that the Secretary of Health and Human Services direct the Assistant Secretary for the Administration for Children and Families to take the following four actions: augment procedures for estimating and reporting Foster Care program improper payments, to include administrative costs; develop and implement procedures to provide a statistically valid methodology for estimating and reporting Foster Care program improper payments based on complete and accurate payment data; augment guidance to teams gathering state-level Foster Care program improper payment estimate data to include specific procedures to follow in identifying and reporting any underpayments and duplicate or excessive payment errors; and revise existing procedures for calculating a national improper payment estimate for the Foster Care program to include a statistically valid method for aggregating state-level margins of error to derive an overall, inflation adjusted, program estimate.

To help ensure corrective action strategies effectively reduce Foster Care program improper payments, we recommend that the Secretary of Health and Human Services direct the Assistant Secretary for the Administration for Children and Families to take the following three actions: develop and implement procedures requiring states to implement and report on corrective actions whenever a state’s estimated improper payment dollar error rate exceeds a specified target level for the program; establish and implement procedures requiring periodic assessments of state-level improper payment target levels, including targets associated with Title IV-E secondary reviews, for the Foster Care program for which states are to implement and report on corrective actions; and enhance ARTMS reporting capabilities to provide data on the status of actions taken to address Single Audit findings concerning states’ Foster Care program payments, such as providing reporting capabilities to allow ARTMS users to search for specific audit findings by type of finding, grantee, state, region, or across years.

Agency Comments and Our Evaluation

We provided a draft of this report to the Secretary of Health and Human Services for comment. In its written comments, reprinted in appendix IV, HHS agreed that its improper payment estimation efforts can and should be improved. HHS provided a summary of refinements that it had made to its improper payment estimation methodology over the years and also provided information on additional steps it planned to take. HHS stated that our analysis would be a helpful resource as it continued to improve its process. With regard to our seven recommendations to help improve ACF’s methodology to estimate improper payments and its corrective action process, HHS generally concurred with four of the recommendations and agreed to continue to study the remaining three recommendations. HHS also provided technical comments that we incorporated, as appropriate.

HHS generally concurred with three recommendations we made related to improving the improper payment estimation methodology for the Foster Care program. Specifically, HHS generally agreed to (1) estimate and report improper payments related to administrative costs, (2) provide specific procedures to identify and report any underpayments and duplicate or excessive payment errors, and (3) revise procedures for calculating the aggregate state-level margins of error to derive an overall, inflation adjusted, program estimate. HHS described several actions currently under way to address these recommendations. Regarding the first recommendation, HHS noted that it was continuing to pilot test the Administrative Cost Reviews, described in this report, in fiscal year 2012; however, HHS’s response did not indicate when it expects these reviews will be fully implemented. In its response to the second recommendation, HHS stated that additional guidance for identifying and reporting underpayments and duplicate or excessive payment errors will be included in the updated Eligibility Review Guide and review instrument during the fiscal year 2012 review cycle. For the third recommendation, HHS stated that it can and will adjust its calculation to incorporate individual state margins of error in aggregating the state-level estimates into the national program estimate. However, HHS also stated that it will seek to determine whether making this revision would add sufficient value given that the estimate spans 3 years and that inflation is relatively low. We maintain that both aggregating state-level margins of error and factoring for inflation are needed to implement a statistically valid method for estimating improper payments.

For the other recommendation we made related to the improper payment estimation methodology for the Foster Care program, HHS stated it would continue to study our recommendation to develop and implement a statistically valid Foster Care improper payment methodology based on complete and accurate payment data. HHS agreed that it should use the best data available. In its comments on our draft report, HHS acknowledged that it would be optimal to conduct a separate data collection to obtain a universe of Title IV-E payments, but stated that it needs to balance the goal of appropriate measurement with the cost and burden placed on states. HHS described the quality controls in place over the AFCARS data to help ensure the information is complete and accurate prior to selecting case samples for its Title IV-E eligibility reviews, which form the basis for its Foster Care improper payment estimate. Examples of such controls include automated system edit checks within AFCARS, AFCARS Assessment Reviews, and other outreach efforts to improve state AFCARS reporting. Our report describes some of the steps ACF has taken to address AFCARS data quality, but we also point out limitations in these efforts. For example, the AFCARS Assessment Reviews are not conducted annually for all states and do not address verifying the accuracy or completeness of the specific data element that ACF uses to develop its population of foster care cases for estimating improper payments. HHS also stated that its use of oversample cases demonstrated that its sampling and oversampling process is working properly to exclude cases that do not meet the selection criteria. While this process would identify some cases that did not meet ACF’s selection criteria, our point in this report is that ACF’s extensive reliance on the use of oversampling in its methodology is indicative that the population of cases could contain additional inaccuracies that may not be identified through its existing process. Further, as we stated in our report, neither we nor ACF were able to determine the completeness of the universe of cases used to estimate Foster Care improper payments, that is, whether all cases that had actually received a Title IV-E payment were properly coded as such. Thus, the issues we identified with ACF’s sampling methodology, provides limited assurance that the reported improper payment estimate accurately and completely represents the extent of improper maintenance payments in the Foster Care program.

With respect to our three recommendations to help ensure corrective strategies effectively reduce Foster Care improper payments, HHS concurred with one recommendation related to establishing and implementing procedures for periodic assessments of state-level improper payment target levels. HHS also agreed to consider another recommendation to develop and implement corrective action procedures whenever a state’s estimated improper payment dollar error rate exceeds a specified target level for the program in conjunction with the recommendation it concurred with to implement periodic assessments. HHS highlighted several actions it plans to take to enhance its efforts to reduce improper payments, such as taking steps to reexamine and explore the feasibility for lowering the error rate threshold and considering ways to enhance existing Eligibility Review Guide instructions to address any eligibility review findings that involve improper payments not specifically requiring development of a corrective action plan. HHS stated that it plans to further study our recommendation to enhance ARTMS reporting capabilities to provide data on the status of actions taken to address Single Audit findings concerning states’ Foster Care program payments. HHS stated that the agency would study the value of potential enhancements to ARTMS in light of the significant relevant data already available (such as audit information by federal program, grantee name, audit resolution status, and audit periods). Although these data elements are currently available in ARTMS, search results are presented by individual audit reports and include limited information. For example, the search results provide only the number of findings associated with a specific audit report but do not provide details of the individual audit findings that would allow program managers to analyze, for example, trends in the types of findings in states in order to ensure that any systemic issues are addressed. We reaffirm our recommendation to ensure that ACF is able to fully utilize ARTMS as a tool to analyze Single Audit findings.

As agreed with your offices, unless you publicly announce the contents of this report earlier, we plan no further distribution until 30 days from the report date. At that time, we will send copies to the appropriate congressional committees; the Secretary of Health and Human Services; and other interested parties. In addition, the report will be available at no charge on the GAO website at http://www.gao.gov.

If you or your staffs have any questions about this report, please contact me at (202) 512-8486 or raglands@gao.gov. Contact points for our Offices of Public Affairs and Congressional Relations may be found on the last page of this report. GAO staff who made key contributions to this report are listed in appendix V.

Appendix I: Objectives, Scope, and Methodology

The objectives of this report were to (1) determine the extent to which the Administration for Children and Families’ (ACF) estimation methodology generated a reasonably accurate and complete estimate of improper payments across the Foster Care program and (2) determine the extent to which ACF’s corrective actions reduced improper payments. To address these objectives, we reviewed the Improper Payments Information Act of 2002 (IPIA) requirements and related Office of Management and Budget (OMB) guidance effective for fiscal year 2010, Department of Health and Human Services (HHS) regulations on Title IV- E eligibility reviews, and ACF’s internal guidance including policies and procedures on conducting Title IV-E Foster Care eligibility reviews, computing improper payments, implementing corrective action plans for reducing improper payments, and monitoring and resolving audit findings in the Foster Care program. We also reviewed results from the Title IV-E eligibility reviews for periods 2000 through 2010, and prior GAO and HHS Office of Inspector General (OIG) reports. In addition, we reviewed improper payment information reported in HHS’s fiscal year 2010 agency financial report, Improper Payments Section. We reviewed these documents to understand ACF’s efforts to address IPIA requirements and to identify previously reported issues with ACF’s improper payment reporting.

To further determine the extent to which ACF’s methodology generated a reasonably accurate and complete estimate of improper payments across the Foster Care program, we:

Performed an independent analysis of ACF’s sampling methodology, including a review of the sampling plan and other underlying documentation, as well as evaluated whether ACF’s sampling methodology complied with OMB statistical guidance, GAO guidance, and federal internal control standards as criteria to determine the accuracy and completeness of ACF’s reported fiscal year 2010 improper payment estimate for the Foster Care program. The scope of our review did not include an assessment of individual states’ processes or payment systems that are the underlying data that ACF uses to support the national estimate of Foster Care improper payments.

Interviewed ACF officials such as the Acting Associate Commissioner for the Children’s Bureau, its contractor, and staff at selected regional offices such as program managers and financial specialists to gain an understanding of (1) the methodology that ACF uses to estimate improper payments in the Foster Care program in accordance with IPIA requirements and (2) the administrative cost review pilot at five states to develop a methodology for estimating related administrative improper payments. We reviewed available reports for the five pilot reviews to identify what information ACF obtained from these reviews.

To further determine the extent to which ACF’s corrective action process reduced improper payments, we:

Reviewed ACF policies and procedures to gain an understanding of reported corrective action strategies, including the Title IV-E eligibility review process and development of states’ Program Improvement Plans (PIP) used to address the root causes of improper payments, which are identified from the Title IV-E eligibility reviews. We reviewed applicable states’ PIPs for the period 2001 through 2010. We also inquired of ACF officials from the Program Implementation Division within the Children’s Bureau about other monitoring activities in place for states that did not have a PIP in place for IPIA reporting in fiscal year 2010.

Assessed compliance thresholds ACF uses to require states to implement corrective actions against actual performance data to assess the propriety of established performance measures. As part of this analysis, we reviewed our internal control standards as guidance to assess ACF’s evaluation of states’ efforts to implement corrective actions.

Conducted a walkthrough of ACF’s Audit Resolution Tracking and Monitoring System (ARTMS) to obtain an understanding of ACF’s monitoring activities to track and resolve state’s Single Audit findings for the Foster Care program. In addition, we interviewed officials in ACF’s Office of Information Services and the Division of Financial Integrity located in the Central Office and selected representative regional offices such as Regional Program Managers, and program and fiscal specialists to determine how ARTMS is used to identify and correct vulnerabilities that could lead to improper payments.

Examined states’ reported Single Audit findings for fiscal years 2008 through 2010 from ARTMS and a listing of HHS OIG reports on the Foster Care program to identify vulnerabilities or weaknesses in states’ operations that may not have been identified through ACF’s Title IV-E eligibility reviews.

Reviewed agency policies and procedures such as ACF’s Title IV-E Foster Care Eligibility Review Guide issued in March 2006 which includes the Title IV-E Foster Care Eligibility On-Site Review Instrument and Instructions; ACF’s user guide for ARTMS, version 1.2; and ACF’s FY 2010 Corrective Action Plan to Reduce the Estimate Rate of Improper Payments in the Foster Care Program, dated November 12, 2010.

In addition, we conducted site visits to three of ACF’s ten regional offices (Philadelphia, PA; Chicago, IL; and San Francisco, CA). These three regional offices provided oversight of states who collectively claimed over half of the total federal share of Foster Care payments made in fiscal year 2009, the most recent data available at the time of our review for site visit selection. We also selected these regional offices to achieve variation in the numbers of error cases and amount of disallowed claims found during Title IV-E eligibility reviews, which ACF conducts to help ensure that states are claiming federal reimbursement only for eligible children. One region represented the highest number of error cases found in the Title IV-E eligibility reviews and the highest maintenance payment disallowance. Another region had the largest amount of foster care maintenance payments in fiscal year 2009 and the states within this region had high improper payment rates. Another region had a low number of error cases and improper payment issues relative to the high amount of maintenance payments it made to states in its purview. During these site visits, we interviewed agency personnel such as program managers, regional grants officers, and financial specialists to gain an understanding of how Title IV-E eligibility reviews are conducted and how the regional offices work with states on corrective actions and follow up on Single Audit findings. We also inquired about other ACF monitoring activities over states to address financial management weaknesses.

We conducted this performance audit from February 2011 through March 2012 in accordance with generally accepted government auditing standards. Those standards require that we plan and perform the audit to obtain sufficient, appropriate evidence to provide a reasonable basis for our findings and conclusions, based on our audit objectives. We believe that the evidence obtained provides a reasonable basis for our findings and conclusions based on our audit objectives.

Appendix II: Title IV-E Expenditures by Type and by State for Fiscal Year 2010

Average monthly number of children 2,097

SACWIS (operations and development) $ 2,234,570 (662,979) (2,529,684)

Average monthly number of children 12,724

SACWIS (operations and development) 5,193,320 (490,734)

Totals

Appendix III: Estimated Improper Payment Error Rates by State for Fiscal Year 2010 Reporting

State Ohio

According to ACF, Florida IV-E Reviews suspended pending completion of a statewide Foster Care demonstration project and were therefore, not included in the national error rate.

Appendix IV: Comments from the Department of Health and Human Services

Appendix V: GAO Contact and Staff Acknowledgments

GAO Contact

Staff Acknowledgments

In addition to the contact named above, Carla Lewis, Assistant Director; Betty Ward-Zukerman, Assistant Director; Sharon Byrd, Assistant Director; Sophie Brown; Gabrielle Fagan; Vincent Gomes; and Nhi Nguyen made key contributions to this report. Also contributing to this report were Kay Brown; Wilfred Holloway; Francine DelVecchio; Jason Kirwan; and Karen O’Conor.