Head Start: A More Comprehensive Risk Management Strategy and Data Improvements Could Further Strengthen Program Oversight
Published: 2008-02-12T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In February 2005, GAO issued a report that raised concerns about the effectiveness of the Department of Health and Human Services (HHS) Administration for Children and Families' (ACF) oversight of about 1,600 local organizations that receive nearly $7 billion in Head Start grants. GAO was asked to report on (1) ACF's progress in conducting a risk assessment of the Head Start program and ensuring the accuracy and reliability of data from its annual Program Information Report (PIR) survey of grantees, (2) efforts to improve on-site monitoring of grantees, and (3) how data are used to improve oversight and help grantees meet program standards. For this report, GAO surveyed a nationally representative sample of Head Start program directors and interviewed ACF officials. GAO also reviewed ACF studies on the validity of PIR data and conducted tests of data from the 2006 PIR database.
section_title: What GAO Found
paragraphs: ACF has not undertaken a comprehensive assessment of risks that may limit Head Start's ability to meet federal program objectives, despite GAO's 2005 recommendation, and little progress has been made to ensure that the data from its annual PIR survey of grantees, which could facilitate such an assessment, are reliable. To conduct a comprehensive risk assessment, ACF needs to identify external and internal risks, estimate their significance, and decide how to best manage them. While ACF says it is working to establish two systems to address programwide risk, our analysis suggests that these systems fall short of that goal. The first system, by which ACF assesses grantees before providing new funds each year, only assesses risk posed to the program by poorly performing grantees and does not allow for a broader assessment of other sources of risk, such as improper payments to contractors. The second system, a new, integrated management information system, has been in development for over 4 years and it is not clear how it will facilitate a more comprehensive risk assessment for the Head Start program. Both initiatives depend, in part, on data from the annual PIR survey of grantees, which have been found to be unreliable. ACF has taken steps to improve oversight of Head Start grantees by implementing a more rigorous process for certifying reviewers who conduct on-site monitoring visits, implementing new processes to improve the consistency of reviews, and working to establish a system for evaluating reviews on an ongoing basis. Now, ACF verifies reviewers' qualifications and requires them to pass online tests in writing and computer literacy. Reviewers must also complete ongoing training and are evaluated by their peers at the end of each review. ACF has also taken a number of steps to improve the consistency and objectivity of reviews, including developing a Web-based data collection tool to facilitate information gathering, assigning review team leaders from outside the grantee's home region to increase independence, and centralizing the review and preparation of monitoring reports. ACF is also working to establish an ongoing system for evaluating its on-site review process. ACF uses data to track grantee performance and target assistance to underperforming grantees, but weaknesses may have hindered these efforts to improve grantee performance. For example, ACF does not have clear criteria for determining which grantees need additional oversight as part of its refunding analysis. Such decisions are made on an ad-hoc basis, which may result in grantees with similar problems receiving different levels of oversight. Moreover, prior to the December 2007 reauthorization of Head Start, ACF was limited in its ability to increase competition for grants to replace underperforming grantees. Under the new law, ACF will have more flexibility to open competition for Head Start grants to other prospective grantees when current grantees fail to deliver high-quality, comprehensive Head Start programs.

section_title: Letter
paragraphs: In February 2005, we issued a report that raised concerns about the  effectiveness of oversight of the Head Start program. We made a number  of recommendations to improve the Department of Health and Human  Services (HHS) Administration for Children and Families’ (ACF) oversight  of local organizations that receive Head Start program grants and the  grantees’ financial management. Head Start is one of the largest federal  early childhood programs. In fiscal year 2007, Head Start provided nearly  $7 billion to 1,604 local organizations that provide a wide range of services  to over 900,000 primarily low-income children, from birth to age 5, and  their families. These services are aimed at improving the social  competence, learning skills, and health and nutrition status of low-income  children so that they can begin school ready to learn. ACF spent $25.6  million in fiscal year 2006 to conduct its oversight activities.
Since our 2005 report, ACF has implemented a number of changes  designed to improve its monitoring and other oversight of grantees.  However, policymakers have raised new concerns about the scope of  these changes and whether they are likely to strengthen the performance  of Head Start grantees, suggesting a need to reassess ACF’s monitoring  and oversight activities. To respond to your request for more information  about ACF’s monitoring and oversight of the Head Start program, we  examined:  1.  ACF’s progress in conducting a risk assessment of the Head Start  program and ensuring the accuracy and reliability of data from its  annual Program Information Report (PIR) survey of grantees,  2.  ACF’s efforts to improve its on-site monitoring processes, and  3.  ACF’s use of data to improve oversight and help grantees meet  performance standards.
To address these objectives, we visited and interviewed Office of Head  Start and ACF officials and their staff in Washington, D.C. We interviewed  staff from all of ACF’s 10 regional offices and reviewed relevant  documentation from each of these offices. We administered a Web-based  survey to a nationally-representative sample of Head Start and Early Head  Start program directors. We selected our sample of directors from the  population of directors whose programs were the subject of ACF’s on-site  reviews from October 2005 through March 2007 using a newly revised on- site review process. In total, we surveyed 329 program directors, asking  them about their experiences during the most recent on-site reviews, and  their views about the changes to the process. Of the 329 program  directors, 261 responded—for a response rate of 79 percent. Throughout  this report, when we refer to our survey results, we use the terms  “program directors” and “grantees” interchangeably. Finally, we tested the  reliability of two of ACF’s administrative databases—the first contains  data on the results of each grantee’s on-site review, and the second  contains data from ACF’s annual PIR survey of grantees. To assess the  reliability of on-site review data, we relied on our 2005 assessment of the  reliability of these data, performed additional electronic tests of data  elements used for an analysis of the extent of repeat on-site review  findings, and reviewed new information about the database. Based on our  previous and updated assessments, we find the on-site review data used  for our analysis of on-site review findings to be sufficient for the purpose  of this report. However, we identified concerns about the overall reliability  of data from ACF’s annual PIR survey of grantees, which we discuss more  fully in this report. To further address objective 1, we reviewed a recent  ACF study on the validity of PIR survey data. To further address objective  2, we met with the contractor responsible for coordinating the on-site  review process and tracking on-site review results, and representatives  from the National Head Start Association. We also examined patterns of  on-site review results in areas related to program governance, record- keeping and reporting, and fiscal management for all programs reviewed  in both fiscal years 2003 and 2006 to assess the extent of repeat findings,  and obtained the results of a study conducted by ACF to evaluate its on- site review process. For additional details about our scope and  methodology, see appendix I. Our work was conducted from February  2007 through December 2007 in accordance with generally accepted  government auditing standards.
subsections: section_title: Background
paragraphs: The Head Start program was established in 1965 to promote the school  readiness of low-income children by enhancing their cognitive, social, and  emotional development by providing a range of individualized services to  pre-school aged children and their families. The program is overseen by  ACF, which awards grants directly to a network of about 1,600 public and  private nonprofit and for-profit agencies to help pay for health,  educational, nutritional, social, and other services to primarily low-income  children from birth to age 5, and their families.
ACF monitors the success of local agencies that receive Head Start grants  in meeting Head Start program goals and complying with program  requirements by conducting on-site monitoring reviews of grantee  programs every 3 years; administering an extensive, annual PIR survey of  grantees; and reviewing required financial reports and annual audit  reports. Reviewers assess Head Start grantee compliance with all program  requirements, including those specified in the Improving Head Start Act,  the Head Start Program Performance Standards, and other relevant  federal, state, and local regulations. These requirements consist of  administrative, financial management, and other standards, such as using  age-appropriate materials to help children learn to recognize letters and  numbers, and providing safe play areas.
By law, each Head Start grantee must receive a full review at least once  every 3 years. New grantees must receive a full review after completion of  their first year of providing Head Start services and at least once every 3  years thereafter. ACF’s policy is to conduct these reviews on-site. Except  for new grantees, Head Start grantees are reviewed on a rotating basis, and  approximately one-third of all grantees are monitored each year. Reviews  are conducted by a team of reviewers led by a federal Head Start program  specialist from one of ACF’s 10 regional offices.
In our February 2005 report, we identified a number of weaknesses in  ACF’s oversight of Head Start grantees. Specifically, we found that ACF  did not have a strategy for bringing information from its various  monitoring processes together in order to comprehensively assess Head  Start program risks, and identified problems with each of its strategies for  monitoring grantees. We found that ACF did not have procedures to  ensure that on-site reviewers performed their responsibilities in  accordance with established guidelines or to ensure that managers and  staff in ACF regional offices were held accountable for the quality of the  on-site reviews. We also found that ACF did not have procedures for  independently verifying data submitted by grantees in its annual PIR  survey, which, in addition to providing information about grantee  performance, is used to provide information to Congress and the public  about important program characteristics, such as program design and  staffing, and numbers and characteristics of children enrolled and  attending Head Start programs nationwide. Finally, we found that ACF  made limited use of financial reports and audits to ensure that all grantees  effectively resolved financial management problems and had made little  use of its authority to terminate grantees that did not meet program,  financial management, and other requirements, and fund new grantees to  replace them. We made a number of recommendations to address the  problems that we identified, including:  1.  producing a comprehensive risk assessment of the Head Start  2.  strengthening on-site reviewer training and certification procedures;  3.  developing a more consistent approach to conducting on-site reviews  4. implementing a quality assurance process that ensures on-site reviews  are conducted within established guidelines and ACF managers are  held accountable for the quality of on-site reviews;  5.  ensuring the accuracy of PIR survey data by independently verifying  key data submitted by grantees, or ensuring that grantees have systems  in place to collect and report accurate, verifiable data;  6.  making greater use of available information on the status and use of  7.  taking steps to obtain competition for grants that are being refunded if  it is determined that current grantees have failed to meet program,  financial management, or other requirements.
In 2006, ACF reorganized its regional offices in order to streamline  program operations. (See fig. 1.) Currently, ACF regional program staff  report directly to its central program offices, rather than to regional office  administrators. The regional administrators no longer have direct  authority to manage individual program activities. As a result, Head Start  program specialists are directly accountable to central office management.  Also, financial management specialists, who monitor financial  management of all ACF grants, including Head Start grants, are now  directly accountable to the central Office of Grants Management, which is  located within the ACF Office of Administration. The Office of  Administration provides support to ACF’s program offices on a range of  administrative issues, such as managing personnel, information resources,  procurement, and grants. The Office of Grants Management carries out the  Office of Administration’s grants administration duties and provides  leadership and technical guidance to ACF program and regional offices on  grant operations and grants management issues.
The Head Start program was revised and reauthorized in the December  2007 Improving Head Start Act. Prior to this, the program was last  reauthorized in 1998, for fiscal years 1999 through 2003. In the years  between the 1998 and 2007 reauthorizations, the program remained funded  through the annual appropriations process.
section_title: ACF Has Not Undertaken a Comprehensive Risk Assessment, and Data Reliability and Other Challenges Remain
paragraphs: ACF has not undertaken a comprehensive assessment of risks to the  federal Head Start program, despite our 2005 recommendation, and little  progress has been made in ensuring that the data from its annual PIR  survey of grantees, which could facilitate such an assessment, are reliable.  Although ACF has two systems in development to address risk  assessment, neither system provides for a comprehensive, programwide  risk assessment for the Head Start program. Further, both systems depend  to some extent on unreliable data from the annual PIR survey of grantees.  Although ACF has known about the problems with PIR survey data, it has  done little to address them.
subsections: section_title: ACF Has Not Undertaken a Comprehensive Risk Assessment
paragraphs: ACF has not undertaken a comprehensive assessment of risks to the  federal Head Start program. Risk assessment is one of five internal control  standards that together provide the foundation for effective program  management and help government program managers achieve desired  results through effective stewardship of public resources. To carry out a  comprehensive risk assessment, program managers need to identify  program risks from both external and internal sources, estimate the  significance of these risks, and decide what steps should be taken to best  manage them. Although such an assessment would not assure that  program risks are completely eliminated, it would provide reasonable  assurance that such risks are being minimized. For the Head Start  program, this might include anticipating and developing strategies to  minimize the impact of changes in resources available to oversee and  assist local grantees, or to develop initiatives to address social and  demographic changes that may result in changing service needs for  families with young children.
In 2005, we reported a similar finding, noting that despite efforts to collect  information and assess risks, ACF did not have a strategy for bringing this  information together in order to comprehensively assess program risks,  and recommended that ACF produce a comprehensive risk assessment of  the Head Start program. To address our 2005 finding, ACF has attempted  to bring together information about local programs and to assess risk on a  grantee-by-grantee basis; however, it has yet to develop a systematic  approach to assessing risks that may arise from other sources and, if  undetected, could hinder ACF’s ability to achieve Head Start program  objectives, or for developing strategies to prioritize and address risks  proactively.
section_title: New Systems Could Facilitate Comprehensive Risk Assessment but Limitations Exist
paragraphs: ACF is in the process of developing two systems that may help it assess  programwide risks; however, significant limitations or uncertainty exist  with respect to each that could constrain ACF’s ability to use them to  conduct a meaningful risk assessment. The first system, the refunding  analysis system, is a process whereby ACF evaluates the performance of  individual grantees each year before it refunds, or renews, their grants.  The second system, the Head Start Enterprise System (HSES), is still  under development. As envisioned by ACF, the HSES may one day  integrate all available Head Start program data into a single, interactive  database that may one day facilitate analysis across many program areas.  The first system is limited in how it could be used for risk assessment, and  the completion of the second system is uncertain. The refunding analysis  system is limited because it assesses risk from only one source—grantee  performance—and does not assess other types of risk, such as inadequate  procedures for ensuring that staff follow policies for monitoring grantee  activities or for minimizing payments that are not in accordance with  program requirements. Although the planned HSES has the potential for  assessing a wider range of potential program risks, it has been in  development for at least 4 years, and it is unclear when or how it will  actually be used.
The refunding analysis system is an evolving system for evaluating grantee  financial management and performance annually and determining which  grantees require additional assistance. Each month, regional program staff  who are responsible for overseeing Head Start grantees bring together and  assess all available information about grantees that are scheduled to re- apply for their grants. The information is reviewed by grants management  staff and Head Start central office staff. Although the refunding analysis  system is intended to provide analysis of grantee performance prior to  refunding, it serves primarily as a means of identifying grantees that need  assistance and not as a means of discontinuing grants for underperforming  grantees.
Although the refunding analysis system allows grantees that are  considered high risk to be brought to the attention of Head Start program  managers, it does not allow for a broader assessment of other sources of  risk, such as those we previously identified. For example, in 2005, we  identified improper payments to contractors as a source of potential risk  for the Head Start program. Under the Improper Payments Information  Act of 2002, agencies are required to annually identify programs and  activities that may be susceptible to significant improper payments;  provide Congress with the annual estimated amount of improper  payments; and, for programs and activities with estimated improper  payments that exceed $10 million, report on actions taken to reduce  improper payments. In addition, the Improving Head Start Act requires the  Secretary of HHS to submit a report to the appropriate congressional  committees certifying that HHS has completed a risk assessment to  determine which ACF programs are at significant risk of making improper  payments, and describing the actions HHS will take to reduce these  improper payments. Since fiscal year 2004, ACF has taken limited action  to minimize improper payments by collecting data on payments to  grantees that do not meet the requirement that at least 90 percent of the  children who are enrolled in Head Start programs must be from low- income families. However, ACF officials stated that, due to resource  constraints, they do not have plans to track other types of improper  payments, such as overpayments to grantees that serve fewer children  than are reported to be enrolled in their local Head Start programs and  excessive compensation paid to Head Start program staff. Overpayments  to grantees with programs that have enrollment below their funded levels  are not uncommon. In April 2007, the HHS Inspector General reported that  in the 2006 program year, fewer than half (40 percent) of Head Start  grantees were fully enrolled and that enrollment levels by grantee ranged  from full enrollment to as low as 68 percent of funded enrollment.  Overall, this translated into 5 percent of Head Start slots that were funded  but not filled. The HHS Inspector General also found that only 11 percent  of grantees had reported enrollment levels to ACF that matched their  actual enrollment levels, and questioned the ability of 26 percent of  grantees to maintain accurate attendance records and to determine  enrollment accurately. The HHS Inspector General has also conducted a  series of audits of Head Start programs that identified unreasonable levels  of compensation to Head Start program executives. Although ACF  requires grantees to provide information about Head Start program staff  salaries and compensation as part of their annual refunding applications,  ACF has not estimated the extent to which excessive compensation may  be a problem, or verified the extent to which information provided by  grantees is accurate.
section_title: ACF Has Made Little Progress toward Ensuring Accuracy of Key Data
paragraphs: In developing its new systems, ACF plans to use data from its annual PIR  survey of grantees, which several studies over the past 12 years have  determined to be unreliable. Specifically, the refunding analysis system  uses PIR data on grantees’ enrollment of children with disabilities and  provision of medical and dental treatments as factors when determining  the risk that an individual grantee will fail to meet program standards. In  addition, ACF plans to use the PIR database in the HSES. Our February  2005 report found discrepancies in the 2003 PIR database. During our  current review, we conducted similar tests to check for data consistency  in the 2006 PIR database and found that it continues to provide some  inconsistent data. Our findings are consistent with a more recent study  funded by ACF that was undertaken in response to our 2005  recommendation to address the accuracy of PIR data. Specifically, ACF’s  2007 study found that the PIR data reported by individual Head Start  grantees are frequently inaccurate and may be unreliable for grantee  monitoring or risk assessment purposes.
The 2007 study found that data submitted by grantees for the PIR survey  may be unreliable due to the length and complexity of the survey. The  survey includes over 130 questions and provides data on program  operations, enrollment, staff and their qualifications, services for children  and families, and other information used for policymaking and  accountability. All Head Start grantees are required to submit PIR data  every year. ACF’s 1995 study on PIR data validity also suggested that the  length of the survey reduced the accuracy of PIR data submitted by  grantees, and its 2007 study further suggested that instructions provided to  grantees for completing the PIR may be unclear and could lead to grantees  submitting incorrect data. Based on our survey of program directors, we  estimate that over half of all Head Start grantees spend more than 24 hours  to complete the PIR. In our survey, we solicited comments from program  directors about potential obstacles to completing the PIR and they cited  various obstacles, such as unclear instructions and questions that may  change from one year to the next.
In addition to using the PIR data to assess progress of individual grantees,  ACF aggregates the PIR data to provide national, regional, and state-level  statistics on Head Start. ACF uses the aggregate data to report to Congress  and the public on the performance of the Head Start program. Head Start  grantees report using the PIR survey to help manage their programs. A  majority of grantees report using the PIR survey to help ensure  compliance with federal laws and regulations, compare the performance  of their program to national or regional benchmarks, and observe trends in  their own performance over time. Moreover, the Improving Head Start Act  requires ACF to use the PIR as one determinant of whether grantees meet  program and financial management requirements and standards, as part of  a new system for renewing Head Start grants. Reliance on systems that  contain inaccurate data can mislead policymakers and program managers  and result in inappropriate decisions.
In its 2007 study, ACF asserted that the national statistics produced by the  PIR present a reasonable estimate of the services provided by the Head  Start program even though the data collected from individual grantees are  unreliable. However, if there are actual errors in grantee-reported data, the  nationally-reported PIR statistics might present a false picture of the  services provided by the Head Start program. For example, the study  estimated that grantees over-reported the number of children that received  medical exams by 3.6 percentage points and noted that problematic  record-keeping on the part of grantees or physicians might account for  some of the discrepancy between the PIR statistics and the study’s  estimates.  Given that the Head Start program provides services to more  than 900,000 children, over-reporting in the number of children that  received medical exams by 3.6 percentage points could mean that as many  as 32,000 fewer children may be receiving medical exams than the  nationally-reported PIR statistics indicate.
The 2007 study found that ACF lacked procedures to independently verify  the accuracy of the data. Although ACF has built internal consistency  checks into the PIR database, these checks will not detect inaccurate data  as long as the grantee reports data consistently throughout its PIR report.  In our 2005 report, we also found that ACF lacked a data verification  process and recommended that ACF either (1) independently verify key  data submitted through the survey or (2) ensure that grantees have  systems in place to collect and report accurate, verifiable data.
The 2007 study offered several recommendations for enhancing the  reliability of PIR data. Most of the study’s recommendations would  address the accuracy of data reported by individual grantees, to allow ACF  to better assess grantee performance. For example, the study recommends  that ACF perform regular validation of the PIR data submitted by grantees,  possibly during the triennial on-site monitoring reviews. Alternatively, one  recommendation focuses on ACF’s use of the PIR as a tool for generating  national statistics, and suggests accomplishing this through a more limited  survey to a random sample of grantees, thereby reducing the overall  burden on grantees. ACF officials told us that they have not yet developed  any plans to implement the specific recommendations from its 2007 study.
section_title: ACF Improved On-site Monitoring by Strengthening Review Processes and Is Working to Establish a System to Evaluate Reviews
paragraphs: ACF has implemented several changes aimed at improving the quality and  consistency of its on-site reviews of Head Start grantees, in response to  our 2005 recommendations. These changes directly address our previous  findings regarding the lack of procedures for ensuring that review teams  were following on-site review protocols or for ensuring that managers and  staff in ACF regional offices are held accountable for the quality of the  reviews. Specifically, ACF has implemented a more rigorous process for  certifying reviewers and new processes to improve the consistency of  reviews, and is working to establish a system for evaluating reviews on an  ongoing basis.
subsections: section_title: ACF Has Implemented More Rigorous Certification Procedures for Reviewers
paragraphs: ACF has implemented more rigorous procedures for ensuring that a  sufficient number of qualified reviewers are available to help conduct  required on-site reviews of Head Start programs. Danya International, Inc.  (Danya) manages the on-site review process under a contract with ACF  and monitors whether reviewers meet all of the necessary qualifications,  such as having a Bachelor’s degree and at least 3 years of work experience  in a field related to early childhood development or public program  management, and whether they comply with ongoing training and  performance requirements. Danya’s polices require that reviewers who do  not satisfactorily meet the necessary qualifications or who fail to comply  with ongoing requirements for reviewers cannot participate in an on-site  review. According to Danya, 174 reviewers were placed on hold or  removed from the reviewer pool in fiscal years 2006 and 2007 because they  did not meet the necessary qualifications.
Procedures for recruitment of on-site reviewers are more systematic than  in the past. Previously, ACF relied on informal networking among  individuals affiliated with the Head Start program to recruit new  reviewers. Now, Danya procedures provide for an ongoing assessment of  the composition of the current reviewer pool and the numbers of  reviewers needed to carry out reviews in a given year, and a targeted  recruitment strategy to address any shortfalls in the numbers or types of  reviewers needed. For example, to address a shortfall of reviewers who  speak Spanish or have experience in Native American issues, Danya  representatives may attend conferences to recruit new reviewers with  needed special skills and experience, such as conferences sponsored by  the National Hispanic Head Start Association or the National Indian  Education Association. Each month, a three-person panel of qualified  reviewers screens all new applications to determine which applicants  appear to have the required skills and experience. Applicants who meet  the initial screening requirements are asked to provide more detailed  employment and education information, which is then verified by Danya.
In addition to meeting the basic requirements for qualifying to become an  on-site reviewer, Danya procedures require that new and current  reviewers alike must meet minimum training and performance  requirements before they are assigned to a review team. New reviewers  are required to complete a basic training course and must successfully  complete a Head Start monitoring review as a trainee under the  supervision of an experienced coach. All reviewers are required to  complete any new training that may be specified during a given fiscal year,  stay informed about changes to the review process, and successfully  complete online tests in writing and computer literacy. Also, members of  the reviewer pool who are employees of Head Start programs, known as  peer reviewers, who work for programs that are in serious noncompliance  with program requirements are not eligible to participate in on-site  reviews. ACF also requires the review team leader and report coordinator  to complete evaluations for all members of the review team; in addition,  each team member must assess the performance of several colleagues on  the team. If a reviewer’s performance is rated as unsatisfactory by two or  more raters on a single review, Danya will follow up to verify the raters’  assessments, if necessary, and consult with ACF to decide whether the  reviewer should be placed on probation and allowed to participate in an  additional review for further assessment, or whether the reviewer should  be dropped from the reviewer pool.
section_title: New Processes Increase Consistency of Reviews
paragraphs: In fiscal year 2006, ACF implemented new procedures to improve the  quality and consistency of on-site reviews. Concerns about the lack of  independence of on-site review team leaders prompted changes in how  team leaders are assigned. As a result, each review team is now led by ACF  program staff from a region other than the grantee’s home region.  Concerns regarding inconsistencies in the findings cited by different teams  of reviewers for the same grantees led to changes in how findings are  developed and reported. The review teams collect only data and facts  during their review: they do not draw conclusions from the information  they gather or prepare the final report. Instead, reviewers record their  observations in a centralized, Web-based system, noting any potential  areas of noncompliance. The reviewers’ notes are then submitted for  centralized review by ACF. The draft review report is prepared centrally  and includes findings of noncompliance with program requirements or  deficiency—a more serious form of noncompliance that can lead to  termination of the grant. The draft report is then forwarded to the  grantee’s home region and to the review team leader’s region for review  and comment. Any comments received are incorporated into the report as  necessary to clarify the reviewers’ observations, and the final report is  signed by the Director of the Office of Head Start and then issued to the  grantee.
ACF has also revised its on-site review protocols to encourage a more  efficient and uniform approach to conducting on-site reviews. In fiscal  year 2007, ACF implemented a more uniform set of on-site review  protocols, under which every grantee is asked the same questions relating  to 10 distinct program areas, such as health services, fiscal management  and education and early childhood development services. The protocols  encourage a more targeted assessment of whether or not grantees are in  compliance with program regulations, and no longer provide for reporting  about program strengths, such as the provision of services that extend  beyond what is required by regulation. Prior to the on-site review team’s  visit, grantees receive a 30-day notification and are asked to prepare a  uniform set of documents for reviewers to examine before meeting with  the grantee. During the review, reviewers are required to enter the  information they gather into a central, Web-based system, which facilitates  sharing evidence among reviewers and tracking whether reviewers have  completed all necessary tasks.
ACF has also implemented uniform corrective action periods and  mandatory follow-up visits when grantees are found to be either  noncompliant or deficient. At the time of our previous review, ACF relied  on grantees to self-certify that they had corrected any problems identified  during audits or on-site reviews and only made follow-up visits to grantees  that had been found deficient. Now, when grantees are found to be  noncompliant with Head Start program regulations, ACF allows them 90  days to resolve the underlying problems and bring their programs into  compliance. If found deficient, grantees are given 6 months to fully resolve  deficiencies before ACF will take action to terminate their grants, though  ACF will allow additional time if a grantee can justify its request for an  extension. At the end of the relevant corrective action period, ACF  conducts follow-up reviews in order to verify that grantees have resolved  the problems identified during the initial on-site review.
ACF officials said that most grantees have adjusted well to the new on-site  review process, although there were some initial negative reactions from  the grantees when the new procedures were first introduced. ACF officials  told us that they encourage grantees to provide feedback on the review  process and on the conduct of on-site reviewers. They have also  established formal procedures for review team leaders to report violations  of the on-site code of conduct by reviewers and to replace reviewers while  the team is on-site, if necessary. Based on our survey results, we estimate  that 9 percent of grantees encountered problems with reviewers during  their most recent review that required outside intervention. For example,  one respondent reported several problems, including trouble scheduling  the review, a reviewer with a conflict of interest, and overly aggressive  reviewers.
Our survey results suggest that grantees have mixed views about the  revised on-site monitoring procedures. Generally, directors of programs  reviewed under the revised procedures had positive views of the reviewers  but had less positive views of specific changes in the procedures and the  extent to which their most recent on-site review had led to improvements  in their Head Start programs. Specifically, most directors of programs that  were reviewed under the revised procedures were very or somewhat  satisfied with how their most recent on-site reviews were conducted. Most  directors also found that the review teams had adhered to the new  protocols and that the review teams demonstrated an understanding of  program requirements. They were almost evenly split over whether having  program specialists from outside their home region lead their review had a  positive or negative effect on the review process, but about three- quarters thought that the focus on reporting only noncompliance had a  negative or very negative effect on the on-site review process. When asked  the extent to which their most recent reviews led to program  improvements in each of the 10 program component areas, directors  generally reported that the review led to few or no improvements in most  program areas, with the exception of health services and program design  and management, which directors generally thought had improved to a  greater extent than other program areas as a result of their most recent  reviews.
Although we cannot directly attribute any improvements in Head Start  program management to these changes, our analysis of on-site review data  does suggest that there may have been some improvements in Head Start  program management since our last review. In 2005, we reported that  about 76 percent of all grantees that had on-site reviews in 2000 had been  found to be out of compliance with one or more standards in the areas of  fiscal management, program governance, or record-keeping and reporting.  We also reported that, in subsequent reviews, 53 percent of those same  grantees had been cited again for problems in these same program areas.  Our more recent analysis of on-site review data shows that, for grantees  reviewed in 2003, about 71 percent were found to be out of compliance  with standards in the same three areas of program management. In 2006,  29 percent of grantees reviewed were cited again for problems in these  three areas.
In response to our 2005 recommendation, ACF has begun to implement  procedures for assessing the quality of on-site reviews. In 2006, ACF  conducted a one-time study of the consistency of on-site reviews. This  study revealed some differences in the findings identified by different  teams of reviewers that visited the same grantees. However, ACF  determined none of the non-duplicated preliminary findings identified by  re-review teams were serious enough to meet the regulatory definition of a  program deficiency. The Improving Head Start Act further requires that  on-site reviews are to be conducted in a manner that includes periodic  assessments of the reliability of the process. ACF is currently working to  establish an ongoing system for evaluating its on-site review process that  would address this new requirement.
section_title: ACF Uses Data to Identify Grantees in Need of Assistance, but Weaknesses Have Hindered Efforts to Improve Grantee Performance
paragraphs: To improve Head Start grantee performance, ACF uses data from multiple  sources to identify underperforming Head Start grantees. For example, it  uses data to direct resources to grantees in need of training and technical  assistance. It also uses data to identify high-risk grantees that may need  additional oversight. However, ACF’s use of data to improve grantee  performance has faced limitations. For example, ACF does not have clear  criteria for determining which grantees need additional oversight. In  addition, ACF had limits on its ability to obtain competition for grants  prior to the recent reauthorization of Head Start.
subsections: section_title: ACF Uses Data to Track Grantee Activities and Performance and to Target Resources to Underperforming Grantees
paragraphs: ACF uses data from multiple sources to track Head Start grantee  performance and to identify grantees with program weaknesses. In  addition to findings from on-site monitoring reviews, ACF also assesses  grantees’ performance through analysis of their audit reports, financial  reports, and other sources of information. One of the ways that ACF uses  data to assess grantee performance is by calculating risk levels for each  grantee, through its refunding analysis system. ACF determines grantee  risk levels by using various indicators, such as findings from on-site  monitoring reviews, turnover of key program staff, PIR survey data, and  negative media coverage.
After ACF identifies underperforming grantees, it targets resources from  its training and technical assistance (T/TA) network to help these grantees  address their program weaknesses. ACF gives grantees with deficiencies  priority for receiving services from the T/TA network. Priority for  receiving services is then given, in order, to other grantees in non- compliance or at risk for deficiencies, grantees new to providing Head  Start services, and grantees with new directors or key staff. Grantees with  deficiencies are sometimes offered on-site assistance to address their  program weaknesses. The T/TA network also assists grantees through  workshops, cluster training for groups of grantee program staff and  management, presentations at local and national conferences, and other  activities. For example, T/TA network staff members affiliated with ACF’s  regional office in Atlanta have provided clustered training to grantees  covering topics like literacy, domestic violence, and guidance related to  ACF’s on-site monitoring review process. In addition, all grantees are  required to submit an annual T/TA plan to ACF, identifying their T/TA  needs for the coming year. Grantees can use multiple data sources to  identify their T/TA needs, including on-site monitoring review reports and  community assessments.
section_title: Lack of Criteria for Initiating Additional On- site Reviews Undermines ACF’s Ability to Address Severely Underperforming Grantees
paragraphs: ACF uses the refunding analysis system as an opportunity to identify  program weaknesses before it reviews grantees’ annual grant applications.  If the process identifies an underperforming grantee—those designated as  “high risk”—ACF may decide to initiate a special on-site review to  determine whether the grantee is deficient and whether the grantee should  ultimately be terminated. If the grantee is deemed deficient, ACF can  require the grantee to correct the deficiencies within specified time  frames, or to begin a quality improvement process, after which ACF will  assess whether the grantee has corrected all deficiencies. In all cases,  grantees that do not correct identified deficiencies are subject to  termination proceedings. However, ACF’s criteria for deciding which  grantees are subject to the special on-site review are unclear. These  decisions are typically made on an ad-hoc basis, which may result in  grantees with similar problems receiving inconsistent levels of oversight.  We have reported that consistency is an essential component to ensuring  performance accountability in federal grants.
Prior to enactment of the Improving Head Start Act, statutory provisions  limited ACF’s ability to terminate underperforming grantees at the time an  on-site review or the annual refunding analysis showed inadequate  performance. ACF was required to grant priority to existing grantees  when making funding decisions, unless ACF determined that the grantee  failed to meet program, financial management, or other requirements  established by ACF. However, before ACF could terminate a failing  grantee and open the grant to competition from other prospective  grantees, ACF was required to provide the grantee with official notice and  an opportunity for a hearing on its termination—or convince the grantee  to relinquish its grant. Moreover, if a grantee appealed ACF’s termination  decision, ACF was required to pay the grantee’s legal fees until the hearing  process was completed.
Recent work related to grants management has pointed to competition for  grants as a way to facilitate grant accountability. For example, the  Domestic Working Group, chaired by the Comptroller General of the  United States, cited grant competition as a key area of opportunity for  improving grant accountability. It noted that grant competition promotes  fairness and openness in the selection of grantees, and that agencies can  better ensure that grantees have the capability to efficiently and effectively  meet grant goals by incorporating evaluation criteria focused on factors  indicative of success into the competition process.
The recent reauthorization of Head Start amends some of the requirements  and procedures for refunding Head Start grantees, including the  introduction of time limits for Head Start grants. This will give ACF the  ability to open competition for Head Start grants to additional grantees,  thereby enhancing ACF’s ability to remove severely underperforming  grantees from the program, on a regular basis.
section_title: Conclusions
paragraphs: In light of federal budget limitations and increasing expectations for  program accountability, ACF’s ability to demonstrate effective  stewardship over billions of dollars in Head Start grants has never been  more critical. Since our 2005 review, ACF has made significant  improvements in its procedures for monitoring local Head Start programs.  In particular, the agency has taken steps to bring together information  about individual grantees from various sources in order to identify those  that are struggling to meet Head Start performance standards, and to  target assistance to help these grantees strengthen their programs. This  risk-based approach is promising and provides a foundation for a more  strategic, comprehensive approach to managing the Head Start program.  Nevertheless, ACF’s current initiatives do not yet constitute a  comprehensive plan for managing program risks. Without a more  comprehensive approach to identifying risks, threats to ACF’s ability to  achieve Head Start program objectives will likely go undetected until a  problem arises. For example, undetected improper payments could result  in a severe reduction in the funds available to pay for services for children.
Although ACF has made progress since we last reported in 2005 toward  strengthening its oversight of the approximately 1,600 local agencies that  receive Head Start program grants, its systems for doing so depend in part  on data that are unreliable. If ACF does not act to address the weaknesses  in its data, it cannot depend on its new systems to provide it with reliable  information on grantee performance. The lack of reliable information  about local program activities further compromises ACF’s ability to  manage risks by limiting its ability to understand whether problems are  isolated or national in scope, as well as whether they arise from individual  grantee failures or from weaknesses in the broader structure of the  program itself. A lack of sound information also calls into question the  credibility of ACF’s reporting to Congress and the American public on the  services provided by the Head Start program.
Even if ACF conducts a comprehensive risk assessment of the Head Start  program and works to improve the accuracy of its data, it will still face  challenges addressing risks posed by the most severely underperforming  grantees. The annual refunding process could be used to link funding  opportunities to performance. For example, if ACF develops clear and  objective criteria for deciding which grantees are subject to a special on- site review, it could ensure that all grantees with similar problems receive  similar levels of oversight. Without such criteria, special on-site reviews  will continue to be conducted on an ad-hoc basis and, as a result, ACF may  continue to fund poorly performing grantees that do not receive special  on-site reviews without providing them the additional oversight they need.
section_title: Recommendations for Executive Action
paragraphs: To improve its management and oversight of the Head Start program, we  are making the following four recommendations to HHS’s Assistant  Secretary for Children and Families:    More fully implement our 2005 recommendation by developing a strategic  and comprehensive approach to assessing Head Start programwide risks.  A comprehensive, programwide risk assessment should take all program  risks into account. Specifically, a comprehensive risk assessment should  include an assessment of risks arising from external sources, such as  social and demographic changes that may affect the availability or demand  for Head Start program services, as well as from internal sources, such as  underperforming grantees, differences in how regional offices implement  program policies and procedures, or the availability of sound data to help  manage the program. A comprehensive risk assessment should also  include strategies for minimizing risks that could significantly limit the  ability of ACF and grantees to help grantees deliver high-quality programs.
Look for cost-effective ways to expand ACF’s efforts to comply with the  Improper Payments Information Act of 2002 and to address our 2005  recommendation by collecting data on and estimating the extent of  improper payments made for unallowable activities and other  unauthorized purposes. These should take into account various aspects of  the program and should not be limited to improper payments to grantees  that enroll too many children from families that do not meet the program’s  income eligibility requirement.
Take additional steps to ensure the accuracy of PIR data by determining  which elements of the PIR are essential for program management and  focus resources on a streamlined version of the PIR that would be  required annually of all grantees, with the responses verified periodically.  If additional information is needed to produce national estimates of a  wider range of Head Start program services, ACF should include the  relevant, additional data items in an expanded version of the PIR, which  could be administered to a random, representative sample of grantees  each year.
Develop clear criteria for determining which grantees require more  thorough reviews—such as special, on-site reviews—as a result of its  refunding analysis system.
section_title: Agency Comments and Our Evaluation
paragraphs: We provided a draft of this report to ACF for comment. The full text of  these comments appears in appendix II. ACF agreed with two of our  recommendations, and emphasized progress already made toward  developing a comprehensive risk assessment process and toward reducing  improper payments. Specifically, ACF noted that it plans to implement a  programwide risk management process in early 2008. ACF also said that it  has developed a new integrated data management system. While we agree  that ACF’s planned programwide risk management process and integrated  data management system are important initiatives that may facilitate  programwide risk assessment, both systems have yet to be fully  implemented and it remains to be seen how these systems will actually be  used to proactively manage the Head Start program nationally. ACF also  emphasized the progress it has made to reduce the frequency and amount  of improper payments arising from participants’ ineligibility, and that its  focus on eligibility is consistent with how ACF has implemented the  Improper Payments Information Act of 2002 for other programs, and was  approved by HHS and OMB management. We agree that ACF’s progress  toward reducing the improper payment error rate due to Head Start  program ineligibility from 3.6 percent in 2003 to 1.3 percent in 2006 is  commendable, and acknowledge ACF’s commitment to reducing improper  payments for the Head Start program. However, as noted in ACF’s  comments, the agency’s improved monitoring efforts and planned risk  management initiatives may help ACF to identify further areas for study  related to improper payments, and we encourage ACF to pursue these  areas, as practical. ACF agreed with our recommendation to review and  streamline its annual PIR survey of grantees, noting that it will continue to  use the results of its own PIR validation study to inform the design of  future surveys, take steps to periodically verify grantee responses, and  leverage technological improvements to capture program data more  frequently and consistently. Finally, ACF said it will soon have the ability  to define realistic criteria for determining which grantees require more  thorough or special reviews, and noted that improvements in monitoring,  and  changes to the process for designating grantees resulting from  reauthorization, should help enable it to do so. We’re encouraged that ACF  said it should have both its risk management process and its integrated  data management systems operational this year. Both systems will play an  important role in its efforts to better target its oversight efforts.
section_title: Appendix I: Scope and Methodology
paragraphs: To answer our research objectives, we interviewed Administration for  Children and Families (ACF) and Office of Head Start officials and their  staff in Washington, D.C., and in the ACF regional offices in Philadelphia  and Atlanta. We conducted interviews with staff from all of ACF’s 10  regional offices, and reviewed relevant documentation from each of these  offices. We visited two regional offices—Philadelphia and Atlanta—to help  us develop interview and data collection protocols and conducted  telephone interviews with the remaining eight regional offices.
To obtain grantees’ opinions on ACF’s oversight of the Head Start  program, we administered a Web-based survey to a nationally- representative sample of Head Start and Early Head Start program  directors. Our target population consisted of directors whose programs  were the subject of on-site reviews from October 2005 through March  2007, and were reviewed under ACF’s newly revised on-site review  process. Based on data supplied by ACF and the contractor responsible  for coordinating the on-site reviews, we identified a total population of 598  Head Start grantees. From this population, we selected a stratified random  probability sample of 329 grantees. We stratified the population based on  whether or not the grantee has any delegate agencies and the ACF region  in which the grantee operates. We also included in our sample all grantees  that participated in ACF’s 2006 one-time study of the consistency of its on- site reviews. With this probability sample, each grantee had a nonzero  probability of being selected, and that probability could be computed for  any grantee. Each grantee selected in the sample was subsequently  weighted in the analysis to account statistically for all the grantees in the  population.
Because we followed a probability procedure based on random selections,  our sample is only one of a large number of samples that we might have  drawn. Since each sample could have provided different estimates, we  express our confidence in the precision of our particular sample’s results  as a 95 percent confidence interval. This is the interval that would contain  the actual population value for 95 percent of the samples we could have  drawn. As a result, we are 95 percent confident that each of the confidence  intervals in this report will include the true values in the study population.
All percentage estimates from this survey have margins of error of plus or  minus 10 percent or less, unless otherwise noted.
We developed our survey questions based on feedback that we received  during three focus group sessions that we conducted with Head Start  program directors whose programs were reviewed under the revised on- site review process. After we drafted the survey questionnaire, we  conducted a series of six pretests by telephone to check that (1) the  questions were clear and unambiguous, (2) terminology was used  correctly, and (3) the information could feasibly be obtained. We made  changes to the content and format of the questionnaire as necessary  during the pretesting process.
The survey was fielded between July 12, 2007, and August 20, 2007.  Of the  329 program directors in our sample, 261 responded—for a response rate  of 79 percent.
To assess the changes ACF made to its on-site monitoring review process,  we met with the contractor responsible for coordinating the reviews, as  well as with representatives from the National Head Start Association, and  reviewed the results of a study conducted by ACF to evaluate its on-site  review process.
We also analyzed the extent to which ACF’s on-site monitoring reviews  identified repeat noncompliance by Head Start grantees, using 2003 and  2006 data from the on-site review database that is currently maintained by  Danya International, Inc. (Danya). We used four data elements from this  database for our analysis: Grantee ID, Fiscal Year, Review Type, and Core  Area. We further limited our analysis to three core areas: program  governance, record-keeping and reporting, and fiscal management. To  assess the reliability of these data, we relied on both our 2005 assessment  of the reliability of 2003 data, and performed additional tests. Specifically,  we conducted electronic testing of both the 2003 and 2006 data and found  no missing or out-of-range entries for any of the four elements that we  used, and obtained additional information about the 2006 data from  Danya. Based on our previous and updated assessments, we find the 2003  and 2006 data sufficient for the purpose of this engagement.
We analyzed the on-site review data for 2003 and 2006 to obtain, for each  year, the numbers of (1) all grantees reviewed, (2) grantees receiving  triennial or first-year reviews only, and (3) grantees cited for deficiency or  noncompliance in one, two, or all three of the core areas of interest  (program governance, record-keeping and reporting, and fiscal  management). To obtain the number of grantees with repeat  noncompliance, we computed the number of grantees cited for  noncompliance or deficiency in both 2003 and 2006, in each of the same  core areas. The computer code used to derive these numbers was  subjected to independent review within GAO and was found to be  accurate.
We reviewed ACF studies from 1995 and 2007 on the validity and accuracy  of PIR data, the results of which are discussed in this report. We also  conducted consistency tests on PIR data from the 2006 PIR database,  similar to the tests that we conducted on the 2003 PIR database for our  2005 report on Head Start oversight. Overall we conducted 29 tests, across  all three sections of the PIR database: Enrollment and Program  Operations, Program Staff and Qualifications, and Child and Family  Services. In 9 of our 29 tests, PIR data contained inconsistent data that did  not sum to the expected totals. In each of the nine tests that failed, less  than 1 percent of the 2,696 data items failed. Our findings indicate that the  PIR database contains some inconsistent data.
Our work was conducted from February 2007 through December 2007 in  accordance with generally accepted government auditing standards.
section_title: Appendix II: Comments from the Department of Health and Human Services
section_title: Appendix III: GAO Contacts and Staff Acknowledgments
subsections: section_title: GAO Contact
section_title: Acknowledgments
paragraphs: The following individuals made important contributions to this report: Bill  J. Keller, Regina Santucci, Christopher W. Backley, Susannah Compton,  James Ashley, Cindy Gilbert, Alison Martin, George Quinn, Jerry Sandau,  Elizabeth Wood, Kimberly Brooks, Jacqueline Nowicki, Doreen Feldman,  Alexander Galuten, and James Rebbe.
section_title: Appendix IV: Related GAO Products
paragraphs: Improper Payments: Incomplete Reporting under the Improper  Payments Information Act Masks the Extent of the Problem.   GAO-07-254T. Washington, D.C.: December 5, 2006.
Grants Management: Enhancing Performance Accountability   Provisions Could Lead to Better Results. GAO-06-1046. Washington, D.C.:  September 29, 2006.
Head Start: Progress and Challenges in Implementing Transportation  Regulations. GAO-06-767R. Washington, D.C.:  July 27, 2006.
Head Start: Comprehensive Approach to Identifying and Addressing  Risks Could Help Prevent Grantee Financial Management Weaknesses.  GAO-05-176. Washington, D.C.: February 28, 2005.
Head Start: Better Data and Processes Needed to Monitor  Underenrollment. GAO-04-17. Washington, D.C.: December 4, 2003.
Head Start: Increased Percentage of Teachers Nationwide Have Required  Degrees, but Better Information on Classroom Teachers’ Qualifications  Needed. GAO-04-5. Washington: D.C.: October 1, 2003.
Managing for Results: Efforts to Strengthen the Link between Resources  and Results at the Administration for Children and Families. GAO-03-9.  Washington, D.C.: December 10, 2002.
Strategies to Manage Improper Payments: Learning from Public and  Private Sector Organizations. GAO-02-69G. Washington, D.C.: October  2001.
Standards for Internal Control in the Federal Government.   GAO/AIMD-00-21.3.1. Washington, D.C.: November 1999.