Department of Health and Human Services 
OFFICE OF 
 
INSPECTOR GENERAL 
 
 
 
 
  H S G
EAD  TART  RANT 
 
  RECOMPETITION:  
 
E I
A  RLY  MPLEMENTATION 
 
R S
  ESULTS  UGGEST 
 
O
PPORTUNITIES FOR 
I  
MPROVEMENT
 
Suzanne Murrin 
Deputy Inspector General for 
Evaluation and Inspections 
 
August 2016 
OEI-12-14-00650   
EXECUTIVE SUMMARY    
HEAD START GRANT RECOMPETITION:  EARLY IMPLEMENTATION RESULTS 
SUGGEST OPPORTUNITIES FOR IMPROVEMENT 
OEI-12-14-00650 
WHY WE DID THIS STUDY    
The Head Start program is the largest Federal investment in early childhood education. The 
Improving Head Start for School Readiness Act of 2007 required the Administration for 
Children and Families (ACF) to begin awarding 5-year grants for Head Start and to require 
grantees that ACF determines are not providing a high-quality and comprehensive Head Start 
program to recompetei.e., to participate in open competition for funding renewal.  In 
response, ACF began in 2012 to implement the Designation Renewal System (DRS).  The DRS 
uses seven trigger conditions to assess a subset of grantees (known as a cohort) each year and 
determine which grantees will be required to recompete.  These changes are intended to improve 
the quality of grantees receiving Head Start funds.  However, stakeholders have raised concerns 
about the efficacy and fairness of this process.   
HOW WE DID THIS STUDY 
To review the second cohort of grantees to undergo DRS assessment and recompetition, we 
combined data on (1) grantees characteristics; (2) their performance histories; (3) the DRS 
determinations as to which grantees had their grants automatically renewed and which were 
required to recompete, and (4) the outcomes of those recompetitions.  We summarized the DRS 
determinations and recompetition results, and we compared grantees DRS determinations to 
other, non-DRS performance data that ACF collects.  Finally, we reviewed DRS determinations 
and recompetition outcomes for a subgroup of grantees that had lower performance on 
10 selected measures than did their peers. 
WHAT WE FOUND 
We found that one-third of grantees were required under the DRS to recompete for funding 
renewal.  Grantees DRS determinations were not linked to the number of Head Start enrollees 
they served, the types of areas (i.e., rural or urban) where their centers were located, the 
proportion of their enrollees who were from non-English-speaking families, or the proportion of 
their enrollees who were from very poor households.  Of grantees required to recompete, 
approximately three-quarters had their grants renewed for an additional 5-year term.  More than 
half of these grantees were the sole applicants for their respective grants.  We also found that 
DRS determinations were largely inconsistent with other ACF performance data.  Additionally, 
few grantees with lower performance on a hybrid of 10 DRS and non-DRS performance 
measures left the Head Start program through the DRS and recompetition processes.  Overall, 
92 percent of Head Start grantees had their grants renewed.    
WHAT WE RECOMMEND  
We recommend that ACF proactively monitor grantees performance results to verify that 
grantees designated under the DRS for automatic, noncompetitive renewal perform better than 
their peers.  Additionally, ACF should take steps to increase the number of applicants for 
recompeted grants.  ACF concurred with both recommendations.   
TABLE OF CONTENTS 
 
Objectives ....................................................................................................1 
Background ..................................................................................................1 
Methodology ................................................................................................6 
Findings........................................................................................................9 
One-third of Head Start grantees were required to recompete for 
funding .............................................................................................9 
Of grantees required to recompete, approximately three-quarters 
had their grants renewed ................................................................10 
DRS determinations were largely inconsistent with other 
performance data ............................................................................ 11 
Few grantees with lower performance on selected measures than 
their peers left the Head Start program through the DRS and 
recompetition processes .................................................................12 
Conclusion and Recommendations ............................................................14 
Agency Comments and OIG Response......................................................16 
Appendixes ................................................................................................17 
A:  Head Start Eligibility ...............................................................17 
B:  Designation Renewal System Trigger Conditions ...................18 
C:  Detailed Methodology ..............................................................21 
D:  Comparison of Automatically Renewed vs. Recompeting 
Grantees:  Statistical Testing ..........................................................26 
E:  Agency Comments ...................................................................28 
Acknowledgments......................................................................................30 
 
 
 
   
OBJECTIVES 
1.  To assess Designation Renewal System (DRS) determinations 
regarding which Head Start grantees are required to recompete for 
funding. 
2.  To describe Head Start grant renewal decisions the Administration for 
Children and Families (ACF) made under recompetition. 
3.  To determine the extent to which grantees with lower performance on 
selected measures left the Head Start program through the DRS and 
recompetition processes. 
RATIONALE 
With a budget of over $9 billion and serving more than 1 million children 
each year, the Head Start program is the largest Federal investment in 
early childhood education.  The Improving Head Start for School 
Readiness Act of 20071 required ACF to begin awarding 5-year grants, 
rather than the indefinite-term grants used in the past, and to require 
grantees who do not provide high quality and comprehensive services to 
participate in open competition for renewal.   
In response, in late 2011, ACF began assessing grantees through the 
Designation Renewal System (DRS) to determine which grantees would 
be required to recompete, or participate in open competition for renewal.  
These changes are intended to improve overall program quality.  However, 
some stakeholders have raised concerns about whether the DRS places a 
disproportionate burden on certain types of grantees (for example, those 
serving linguistically diverse populations), as well as whether grantees that 
ACF designated under the DRS for automatic renewal are actually of 
higher quality than those required to undergo recompetition.2  This study is 
the first national review of the DRS and recompetition processes. 
BACKGROUND  
Head Start:  Overview 
Head Start is a nationwide grant program designed to promote school 
readiness in children from low-income families.  ACF awards funds to 
approximately 1,700 Head Start grantees, who provide early childhood 
education; medical, dental, and mental health care; and nutrition services.  
 
1 P. L. No. 110-134 (Dec. 12, 2007). 
2 For example, as expressed in public comments to ACF regarding the DRS 
implementing regulations.  See 76 Fed. Reg. 70011-70029 (Nov. 9, 2011). 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           1 
   
Grantees may be public agencies, private nonprofit and for-profit 
organizations, tribal governments, or school systems.  The Presidents 
budget request for FY 2017 included $9.6 billion for the program.3   
The Office of Head Start within ACF administers four types of Head Start 
programs.  Traditional Head Start programs, which are most common, 
serve preschoolers (primarily ages 3 and 4) and account for over 
80 percent of children enrolled in Head Start.  Early Head Start programs 
serve infants, toddlers, and pregnant women.  Migrant and Seasonal Head 
Start programs serve the migrant and seasonal worker community.  
Finally, American Indian-Alaska Native (AI/AN) Head Start programs 
serve AI/AN communities.  
Grantees must implement and comply with a variety of eligibility and 
enrollment requirements.  In general, children are eligible for Head Start if 
they are of the appropriate age and if the family is homeless, is eligible for 
or receiving public assistance, is caring for foster children, and/or meets 
income guidelines.4  A child who meets these requirements and whose 
familys income comes primarily from agricultural work is eligible for 
Migrant or Seasonal Head Start.5  For a detailed explanation of Head Start 
eligibility, see Appendix A.  
Performance Monitoring and Oversight 
Federal law requires ACF to perform periodic monitoring and oversight of 
Head Start grantees.6  ACF conducts the following monitoring reviews to 
determine whether grantees comply with requirements and standards:7 
  Triennial reviews are conducted at least once during each 3-year 
period to assess grantee compliance with all program areas.  In   
FY 2015, ACF began transitioning from triennial reviews to more 
frequent assessments, increasing the focus on quality while 
continuing to examine compliance.8 
  First-year reviews are reviews of each newly designated Head 
Start grantee immediately after the grantee completes its first 
program year. 
  Followup reviews are conducted for grantees with areas of 
noncompliance or with one or more deficiencies (described below) 
 
3 Department of Health and Human Services, HHS FY 2017 Budget in Brief,             
February 2016.   
4 Head Start Act  645(a)(1)(A) (42 U.S.C.  9840(a)(1)(A)).  
5 45 CFR  1305.4(g). 
6 Head Start Act  641A(c) (42 U.S.C.  9836A(c)). 
7 Head Start Act  641A(c)(1).  
8 ACF, Report to Congress on Head Start MonitoringFiscal Year 2014, p. 31-32. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           2 
   
in order to determine whether these grantees have corrected 
previously identified problems.  
  ACF may initiate other reviews as appropriate, e.g., if issues with 
a grantees performance are brought to ACFs attention.  These 
reviews are conducted on an as-needed basis and focus on 
assessing a specific concern. 
ACF reviews data collected during monitoring reviews and determines 
whether grantees comply with all requirements.  Monitoring reports may 
include the following types of findings: 
  Deficiencies9 indicate that a grantee exhibits systemic or 
substantial noncompliance with significant State or Federal 
requirements.  Examples include requirements regarding threats to 
childrens health or safety or the misuse of Head Start funds.  ACF 
may terminate any grantee that fails to correct a deficiency finding 
within the designated timeframe.10   
  Noncompliances11 indicate that a grantee is out of compliance with 
a requirement, but not to a level that constitutes a deficiency.  
Noncompliances require a written timeline of correction and may 
also result in technical assistance or guidance from ACF.  If a 
grantee does not correct a noncompliance within the specified 
timeframe, ACF reclassifies the noncompliance as a deficiency.12 
  Strengths indicate new or innovative practices that help the grantee 
overcome challenges, improve service quality, and/or surpass 
performance indicators. 
Additionally, ACF calculates a variety of performance indicators based on 
information that grantees self-report through the annual Program 
Information Report (PIR).  These performance indicators describe various 
aspects of services provided during the preceding program year (e.g., the 
proportion of children with disabilities who received targeted services).  
ACF makes grantees scores on these performance indicators publicly 
available at both the individual and national levels.13 
 
9 Head Start Act  637 (42 U.S.C.  9832).  
10 Head Start Act  641A(e) (42 U.S.C.  9836A(e)). 
11 45 CFR  1304.61. 
12 Ibid. 
 13 For example, see ACF, Office of Head Start  Head Start Services Snapshot National 
(2014-2015).  Accessed at http://eclkc.ohs.acf.hhs.gov/hslc/data/psr/2015/services-
snapshot-hs-2014-2015.pdf on April 25, 2016. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           3 
   
Grant Renewal 
Historically, Head Start grants were indefinite in term, and grantees 
remained in the program unless their grants were terminated for cause.  
The Improving Head Start for School Readiness Act of 200714 amended 
the Head Start Act to establish 5-year terms for Head Start grants.  The 
law further required that grantees determined not to be delivering a     
high-quality and comprehensive Head Start program must participate in 
open competition for renewal, or recompete.15   
To determine which grantees would recompete, the Head Start Act, as 
amended, required the Secretary of Health and Human Services to develop 
a system to identify Head Start grantees that deliver a high-quality and 
comprehensive Head Start Program that meets the educational, health, 
nutritional, and social needs of the children and families it serves, and 
meets program and financial management requirements and 
standards....16  Grantees that do not meet this requirement are subject to 
open competition for grant renewal.17  Specifically, ACF posts the grant as 
a Funding Opportunity Announcement, and if the incumbent grantee seeks 
renewal, it must apply alongside other interested entities so that ACF can 
select the most qualified provider through open competition.  To promote 
competition, ACF has provided guidance to eligible organizations through 
an online grant application toolkit18 and conducted community meetings to 
raise awareness of upcoming competitions. 
DRS.  Through a final rule effective December 9, 2011, ACF established 
the DRS to identify grantees that deliver a high-quality and 
comprehensive Head Start program and whose grants can therefore be 
automatically renewed without competition.19  Specifically, regulations for 
the DRS describe seven trigger conditions indicative of quality 
concerns.  If one or more of these seven conditions applies to a Head Start 
grantee, it must recompete if it seeks grant renewal.  The DRS trigger 
conditions can be summarized as follows: 
1.  One or more deficiency findings over the prior review period.  
2.  Low score on one or more domains of the Classroom Assessment 
Scoring System (CLASS), an observational tool used to assess the 
quality of teacher-student interactions in preschool classrooms.  
 
14 P. L. No. 110-134. 
15 Head Start Act  641(c)(7)(A) (42 U.S.C.  9836(c)(7)(A)). 
16 Head Start Act  641(c)(1) (42 U.S.C.  9836(c)(1)). 
17 Head Start Act  641(c)(7)(A) (42 U.S.C.  9836(c)(7)(A)). 
18 The toolkit is available online at https://eclkc.ohs.acf.hhs.gov/hslc/grants/grant-toolkit.     
    Accessed on May 15, 2016  
19 45 CFR  1307.1. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           4 
   
Specifically, the grantee either scores in the bottom 10 percent of 
grantees in any of the three CLASS domains (emotional support, 
classroom organization, and instructional support) or fails to meet the 
minimum quality standard in any of the three CLASS domains.  
3.  A determination that the grantee is at risk of failing to continue 
functioning as a going concern (i.e., a determination that the grantee 
is at risk of financial failure). 
4.  Revocation by a State or local licensing agency of a grantees license 
to operate a Head Start or Early Head Start center or program.  
5.  Failure to establish program goals for school readiness or take steps to 
achieve those goals. 
6.  Suspension from the Head Start program by ACF. 
7.  Debarment by any Federal or State department or agency or 
disqualification from the Child and Adult Care Food Program. 
The first trigger condition (deficiency findings) is drawn solely from 
ACFs onsite monitoring reviews.  However, when ACF is determining 
whether a grantee will have its grant automatically renewed or will be 
required to recompete, it does not consider other findings from onsite 
monitoring (such as noncompliances and strengths).  Similarly, ACF does 
not consider grantees performance on PIR-based indicators. 
The DRS trigger conditions are described in detail at 45 CFR  1307.3.  
(See Appendix B.)   
DRS Implementation.  The DRS became effective December 9, 2011, with 
the goal of implementation over a 3-year period.20  Transitions from grants 
for indefinite periods to grants for 5-year periods were staggered over the 
implementation period so that in each year, only a subset of grantees 
(known as a cohort) would be assessed through the DRS.  Under the DRS, 
each grantee either has its grant noncompetitively renewed or is required 
to recompete.   
When ACF assessed the first cohort under the DRS, it did so using five of 
the seven DRS trigger conditions; it did not consider the conditions related 
to CLASS or school readiness goals.21  For the second and third cohorts, 
ACF took all seven DRS trigger conditions into account.  ACF has stated 
that by the end of 2016, it will have transitioned all grantees to 5-year 
grants. 
 
20 ACF, Report to Congress on the Final Head Start Program Designation Renewal 
System, p. 36. 
21 45 CFR  1307.7(b). 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           5 
   
METHODOLOGY 
Scope 
We reviewed the second cohort (hereafter, Cohort 2) of Head Start and 
Migrant and Seasonal Head Start grantees22 to go through DRS assessment 
and recompetition, which took place from 2012 through 2014.  We 
excluded Early Head Start and AI/AN Head Start grantees from this 
review, because ACF uses somewhat different DRS criteria and processes 
for these programs.  
Data Sources 
We combined data from several ACF sources and systems.  We drew 
demographic, descriptive, and service data from the PIR, which grantees 
submit to ACF annually.  Additionally, we used ACFs formulas23 to 
calculate four PIR-based performance indicators from these data.  We 
obtained information on deficiencies, noncompliances, and strengths24 
from ACFs performance-monitoring system.  Finally, ACF provided the 
Office of Inspector General (OIG) with grantees final DRS 
determinations (i.e., grant automatically renewed vs. grantee required to 
recompete), the DRS trigger conditions that were present, CLASS scores, 
and recompetition results.  We reviewed all available data for the 361 
grantees in Cohort 2.   
Analysis 
To assess determinations made under the DRS in its second year of 
implementation, we calculated the proportion of Cohort 2 grantees that 
were required under the DRS to recompete for their grants, identified the 
most common DRS trigger conditions, and compared characteristics of 
grantees that were required to recompete with characteristics of those that 
were not.   
We also compared grantees DRS determinations to their past scores on six 
selected performance measures that ACF collects but does not use in its 
DRS assessments.  Of these, two measures (noncompliances and 
strengths) were drawn from the results of monitoring reviews.  The 
remaining four measures (preventive and primary care; disability services; 
and two measures of teacher qualifications) were drawn from ACFs     
 
22 Head Start and Migrant and Seasonal Head Start, combined, account for 85 percent of 
all Head Start Program enrollees. 
23 ACFs formulas for all PIR performance indicators are available at 
https://eclkc.ohs.acf.hhs.gov/hslc/data/pir.    
24 ACF monitoring review reports describe grantee strengths in narrative format. We 
categorized grantees strengths as reported in monitoring reports according to the seven 
programmatic areas in the Head Start Monitoring Protocol (e.g., child development and 
education, fiscal integrity, etc.). 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           6 
   
PIR-based performance indicators.  We consulted with ACF in selecting 
these measures to confirm that they were relevant and accurate bases for 
assessment and comparison of grantees.  When comparing groups of 
grantees, we used permutation testing to determine whether observed 
differences were most likely due to meaningful association or random 
variation.     
To describe grant renewal decisions made during the second year of 
recompetition, we determined the proportion of recompeted grants that 
were renewed and the number of applicants for each grant.   
To determine the extent to which grantees with lower performance on 
selected measures left the Head Start program through the DRS and 
recompetition processes, we examined the DRS and recompetition 
outcomes for a subset of grantees that underperformed on 10 selected 
performance measures relative to their peers.  These measures included the 
six selected non-DRS measures described above, as well as four key 
measuresdeficiency findings and three separate CLASS scoresthat 
ACF uses in its DRS determinations regarding who must recompete.25  
(See Graphic 1 on the next page.)  A grantee met our criteria as lower 
performing relative to its peers if it a) scored in the bottom 10 percent of 
Cohort 2 grantees on four or more measures, or b) scored in the bottom 
5 percent of Cohort 2 grantees on two or more measures. 
See Appendix C for a detailed description of our sources and analysis.  
   
 
25 Specifically, we included the DRS trigger conditions that drove the vast majority of 
DRS ratings:  deficiencies and low scores on any of the three CLASS domains.  The 
remaining DRS conditions affected zero, one, or two grantees each (see Table 1 on 
page 7) and so were not included in the algorithm.  
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           7 
   
Graphic 1:  Head Start Performance Data Used in OIG Analysis 
 
Head Start Performance Data:                DRS Determinations:  
  Onsite Reviews  Seven Trigger Conditions 
  Deficiencies    Deficiencies 
 
  Noncompliances    CLASS scores 
  Emotional support 
    Strengths 
  Instructional support 
  Classroom organization 
 
  Imminent financial failure 
Head Start Performance Data:              
  PIR Indicators    License revocation 
  Failure to set program goals 
  Preventive and primary care 
 
services    Suspension 
    Disability services    Debarment 
  Teacher qualifications   
 
   
 
   
OIG Analysis: OIG Analysis:
   
  DRS Determinations Compared to Six Grantees With Lower Performance on 
   
Selected Non-DRS Measures 10 Selected Measures
    Noncompliances    Noncompliances 
  Strengths    Strengths 
 
  Preventive and primary care    Preventive and primary care 
services  services 
 
  Disability services    Disability services 
 
  Teacher qualifications    Teacher qualificationsA.A. or 
associates degree (A.A.) or  higher 
 
higher 
  Teacher qualificationsB.A. or 
    Teacher qualifications  higher 
bachelors degree (B.A.) or higher 
  Deficiencies 
 
  CLASS:  Emotional support 
  CLASS:  Instructional support 
 
  CLASS:  Classroom organization 
   
                     Limitations 
As an early implementation review, we examined the second cohort of 
grantees to undergo DRS assessment and recompetition.  Later cohorts 
may perform differently.  Additionally, some data used in this review 
(specifically, PIR data) is reported by grantees; we did not independently 
verify its accuracy. 
Standards 
This study was conducted in accordance with the Quality Standards for 
Inspection and Evaluation issued by the Council of the Inspectors General 
on Integrity and Efficiency. 
 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           8 
   
FINDINGS 
One-third of Head Start grantees were required to 
recompete for funding 
Of the 361 grantees in Cohort 2, 115 (32 percent) had at least one DRS 
trigger condition and were therefore required to recompete for their grants.  
The most common DRS triggers were deficiency findings and/or low 
CLASS scores.  We found no correlation between whether a grantee was 
required to recompete and its enrollment size, its location type (rural or 
urban), the extent to which it served a non-English-speaking population, or 
the extent to which it served a high-poverty population.   
Grantees were most often required to recompete because of 
deficiency findings and/or low CLASS scores 
Although there are 7 possible DRS trigger conditions, nearly all of the 
115 grantees that were required to recompete had either deficiency 
findings or low CLASS scores.  Of these grantees, 112 (97 percent) had a 
single DRS trigger, while 3 grantees (3 percent) had more than one DRS 
trigger (see Table 1). 
Table 1:  Cohort 2 DRS Trigger Conditions 
Number of Grantees  Percentage of 
DRS Trigger Conditions 
(n=115)  Grantees (n=115) 
Deficiency findings  69  60% 
Low CLASS scores  46*  40% 
Determination that grantee is at risk of financial 
failure (i.e., at risk of failing to continue as a  2  2% 
going concern) 
Revocation of license to operate by a State or 
1  1% 
local licensing agency 
Failure to establish and use program goals for 
0  0% 
school readiness 
Suspension from the Head Start program  0  0% 
Debarment by any Federal or State department 
0  0% 
or agency 
  Source:  OIG analysis of ACF data. 
  Numbers do not total 100 percent due to grantees with multiple trigger conditions. 
  *Includes 7 grantees that had scores below the minimum quality threshold and 39 grantees that had scores that   
   were in the bottom 10 percent of grantees but were at or above the minimum quality threshold. 
 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           9 
   
DRS determinations were unrelated to grantees enrollment 
size, location type (rural or urban), or the extent to which they 
served a non-English-speaking or high-poverty population 
Some stakeholders have expressed concerns that certain types of 
grantees for example, those serving linguistically diverse populations
 
might fare disproportionately worse under the new system.  However, we 
found that DRS determinations regarding which grantees would be 
required to recompete were not correlated with grantees enrollment size, 
the proportion of grantees centers located in rural areas, the proportion of 
families served who did not speak English at home, or the proportion of 
families served who had experienced homelessness during the year 
(a measure of extreme poverty).  This suggests that the DRS did not 
disadvantage these categories of grantees.  See Appendix D for details and 
statistical testing results. 
Of grantees required to recompete, approximately 
three-quarters were renewed 
The 115 grantees with 1 or more DRS trigger conditions were required to 
recompete for grant renewal.  Of these grantees, 85 (74 percent) were 
ultimately renewed for an additional 5-year grant term, while 27 grantees 
(23 percent) were not renewed.  Grantees that were not renewed included 
16 that applied but were not selected; 5 that did not apply or that declined 
an award; 4 that relinquished their grants; and 2 that were terminated by 
ACF.  An additional three grantees were not renewed for a 5-year term but 
were asked to temporarily continue services while ACF reposted the grant 
announcement (see Table 2). 
Table 2:  Outcomes of Cohort 2 Grant Recompetitions 
Number of Grantees  Percentage of 
Recompetition Outcomes 
(n=115)  Grantees (n=115) 
Grantee renewed  85*  74% 
Grantee not renewed  27  23% 
Grantee temporarily continued services 
3  3% 
while grant announcement reposted 
  Source:  OIG analysis of ACF data. 
  *Includes 3 grantees that were partially renewed, i.e., their respective grants were renewed for part, but not all, of  
   the original service areas. 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           10 
   
Of grantees that were renewed through recompetition,               
64 percent were the sole applicants for their respective grants 
When grants were recompeted, there were typically few applicants.  
Although the number of applicants for recompeted grants ranged from 0 to 
13, the average posting drew 2 applicants.  Of the 85 grantees that were 
renewed after their grants were recompeted, 54 (64 percent) were the sole 
applicants for their respective grants, meaning that they faced no 
competition.   
In general, if ACF chooses not to renew a grant for which the incumbent 
grantee was the sole applicant, it has limited options for ensuring the 
continuity of Head Start services.  In Cohort 2, there was only one 
recompetition in which the incumbent grantee was the sole applicant and 
was not selected for renewal.  In that instance, ACF appointed an interim 
operator to provide Head Start services until a qualified long-term 
grantee for the service area could be identified. 
DRS determinations were largely inconsistent with 
other performance data 
DRS determinations regarding which grantees must recompete are based 
on seven trigger conditions.  However, ACF collects substantial additional 
performance data for monitoring and management purposes.  We found 
that DRS determinations were generally inconsistent with the six other 
performance measures we selected for review (see Graphic 1 on page 6).   
Overall, we found that grantees designated for automatic, noncompetitive 
renewal had performed significantly better than other grantees on only one 
selected measure:  the number of prior noncompliances.  This difference 
was substantialautomatically renewed grantees had received an average 
of 2.46 noncompliance findings in prior ACF monitoring reviews, 
compared to 4.44 noncompliance findings for grantees that ACF required 
to recompete for renewal.  However, there was no significant difference 
between the two sets of grantees on the other five selected performance 
measures we reviewed.  These measures were:  
  The number of programmatic areas (e.g., child development and 
education, fiscal integrity, etc.) in which the grantee exhibited 
strengths 
  The proportion of children that received preventive and primary 
care on schedule 
  The proportion of children with disabilities that received 
appropriate services for those disabilities 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           11 
   
  The proportion of preschool classes in which at least one teacher 
had an A.A. or higher in early childhood education or a related 
field 
  The proportion of preschool teachers that had a B.A. or higher in 
early childhood education or a related field 
See Appendix C for additional detail about the selection and use of these 
performance measures.  See Appendix D for statistical testing results.   
Few grantees with lower performance on selected 
measures than their peers left the Head Start program 
through the DRS and recompetition processes 
We examined the performance of all Cohort 2 grantees by using a hybrid 
of 10 DRS and non-DRS measures to identify grantees that 
underperformed relative to their peers (see Graphic 1 on page 6).26  These 
10 measures, which include both ACF monitoring results and self-reported 
grantee service data, provide a useful summary of a grantees 
performance.  We note that relatively lower performance on these 
measures is not evidence that a grantee should not be in the Head Start 
programfull grantee performance assessments take into account a broad 
array of nuanced information, and grant renewal decisions depend in part 
on the number and quality of applicants for the grant.  That said, it is 
reasonable to expect that as a group, grantees with lower performance 
would tend to fare poorly under DRS assessment and recompetition.  
However, we found that relatively few grantees with lower performance 
on selected measures left the Head Start program through the DRS and 
recompetition processes.  Of the 301 grantees in Cohort 2 that had 
complete performance data27, 43 grantees (14 percent) met our criteria as 
lower performing on 10 selected measures relative to their peers.28  
 
26 The 10 measures included the following: a) 3 key measures assessed during ACF 
onsite monitoring reviews (deficiencies, noncompliances, and strengths); b) 3 CLASS 
scores assigned by ACF-contracted reviewers based on classroom observation 
(instructional support, emotional support, and classroom organization); and c) 4 selected 
performance indicators that ACF calculates from grantees self-reported PIR data 
(preventive and primary care, services for children with disabilities, teachers that had a 
B.A. or higher, and classrooms in which a teacher had an A.A. or higher).  Of these 
measures, deficiencies and CLASS scores are used in determining grantees DRS ratings, 
while the remaining six measures are collected by ACF for management and information 
purposes but are not considered in the DRS assessment. 
27 Complete performance data was not available for 60 of the 361 grantees.  This was 
primarily because we had CLASS scores only for the grantees that received a triennial 
review in FY 2012. 
28 Specifically, they scored in the bottom 10 percent of Cohort 2 grantees on four or more 
measures or scored in the bottom 5 percent of Cohort 2 grantees on two or more 
measures. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           12 
   
Of these, 11 grantees (26 percent) were not renewed and therefore left the 
Head Start program.  ACF renewed the remaining 32 lower performing 
grantees for an additional 5-year term.  
Of the 32 lower performing grantees that were renewed, 10 had been 
designated under the DRS for automatic renewal without competition.  
The remaining 22 renewed grantees were renewed through the 
recompetition process.  Of those 22 grantees, 12 were the sole applicants 
for their respective grants (see Table 3). 
Table 3:  DRS and Recompetition Outcomes for Cohort 2 Grantees With 
Lower Performance on Selected Measures 
Number of Lower  Percentage Lower 
Outcome  Performing Grantees  Performing of Grantees 
(n=43)  (n=43) 
Designated under DRS for  10  23% 
automatic, noncompetitive renewal 
Required to recompete and won  22*  51% 
renewal 
Required to recompete and did not 
11  26% 
win renewal 
Total  43  100% 
  Source:  OIG analysis of ACF data. 
  *Includes 12 grantees that were the sole applicants for their respective grants 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           13 
   
CONCLUSION AND RECOMMENDATIONS 
When ACF began implementing recompetition, stakeholders raised 
concerns about whether the DRS could accurately determine which 
grantees were of lower quality and should therefore recompete.  We found 
that DRS determinations regarding which grantees were required to 
recompete were often inconsistent with other ACF performance data.  
Further, few grantees with lower performance on selected measures than 
their peers left the Head Start program through the DRS and recompetition 
processes.  This was in part because recompeted grants typically had few 
applicants; in many recompetitions, the incumbent grantee was the sole 
applicant.  Taken together, these facts suggest opportunities for 
improvement.   
However, our results should allay some concerns expressed by 
stakeholdersspecifically, that grantees serving certain populations might 
be at a disadvantage under the new system.  We found that grantees fared 
similarly under the DRS regardless of enrollment size, type of location 
(rural vs. urban), the extent to which they served a non-English-speaking 
population, or the extent to which they served a high-poverty population.   
For the cohort we reviewed, recompetition resulted in little grant turnover:  
of the 361 grantees, 246 were designated under the DRS for automatic, 
noncompetitive renewal, and an additional 85 recompeted and won 
renewal.  Overall, 92 percent of Head Start grantees in Cohort 2 retained 
their grants.  
The DRS and recompetition processes are still relatively early in 
implementation; the final cohort of grantees will transition to 5-year grants 
later this year.  As ACF moves forward, we recommend the following:  
ACF should proactively monitor grantee performance results 
to verify that grantees designated for automatic, 
noncompetitive renewal perform better than their peers 
The purpose of the DRS is to help ACF predict which grantees will 
provide the highest quality services over the next 5 years and can thus 
have their grants automatically renewed.  As an early implementation 
review, OIG compared grantees DRS determinations to the only 
performance data currently availablegrantees past performance results 
on other ACF-collected measures.  However, a more important 
comparison will involve how these grantees perform in future years.   
As ACF shifts to its new system of more frequent grantee monitoring, with 
a greater focus on quality in addition to compliance, it should take the 
opportunity to continually assess the extent to which its DRS 
determinations accurately predict future performance.  If the results of 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           14 
   
ACFs ongoing monitoring reviews demonstrate that grantees designated 
for noncompetitive renewal perform no better than their peers, the DRS 
trigger conditions should be reassessed.  ACF must ensure that it is 
accurately identifying the grants that would most benefit from 
recompetition to improve program quality.  
ACF should take additional steps to increase the number of 
applicants for recompeted grants  
More than half of grantees who recompeted and won renewal were the 
sole applicants for their respective grants, requiring ACF to either reselect 
the incumbent grantee or appoint a temporary grantee to avoid a disruption 
of Head Start services.  ACF has made efforts to promote competition, 
such as providing an online application toolkit and conducting community 
meetings.  However, despite these actions, many Cohort 2 recompetitions 
involved only the incumbent grantee.  ACF should take additional steps to 
ensure robust, meaningful competition for Head Start grants.   
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           15 
   
AGENCY COMMENTS AND OFFICER OF INSPECTOR GENERAL 
RESPONSE 
ACF concurred with both of our recommendations.  Regarding our first 
recommendation, ACF stated that it plans to assess the DRS conditions 
after the implementation of the DRS for all grantee cohorts.  ACF noted 
that this assessment will include how the DRS conditions and other ACF 
measures relate to quality, as recommended by OIG.  Regarding our 
second recommendation, ACF agreed that more competition is desirable 
but described several challenges to achieving robust competition for Head 
Start grants.  ACF stated that it will continue to provide an online toolkit 
to facilitate the application process, that it plans to issue a final rule 
streamlining Head Start requirements in ways that will improve 
transparency and accessibility to applicants, and that it will look for 
additional steps to encourage competition after it completes the initial 
implementation of the DRS. 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           16 
   
APPENDIX A 
Head Start Eligibility 
Grantees must implement and comply with a variety of eligibility and 
enrollment requirements.  In general, children are eligible for Head Start if 
they are of the appropriate age and if the family is homeless, is eligible for 
or receiving public assistance, is caring for foster children, or has an 
income below the poverty line.29, 30  A child who meets these requirements 
and whose familys income comes primarily from agricultural work is 
eligible for Migrant or Seasonal Head Start.31  
The Improving Head Start for School Readiness Act of 200732 amended 
the Head Start Act to allow grantees to enroll children from families with 
incomes between 100 and 130 percent of the poverty line, provided that 
families in this income range do not exceed 35 percent of the grantees 
total enrollment.  Grantees must establish and implement outreach and 
enrollment policies before enrolling children from these families.33  In 
addition, grantees retained the flexibility to make up to 10 percent of their 
enrollment opportunities available to children from families exceeding 
these income guidelines when there are other significant needs facing the 
family.34 
Eligibility rules differ slightly for AI/AN grantees.  For these grantees, up 
to 49 percent of enrollment may consist of children from families above 
the poverty line.35   
Head Start grantees that meet certain conditions, such as being located in 
areas with populations of 1,000 or less, may establish their own criteria for 
eligibility within established parameters.36   
   
 
29 Head Start Act  645(a)(1)(B) (42 U.S.C.  9840(a)(1)(B)).  
30 In 2014, the poverty line was $23,850 for a family of four.  79 Fed. Reg. 3593-3594 
(Jan. 22, 2014). 
31 45 CFR  1305.4(g). 
32 P. L. No. 110-134. 
33 Head Start Act  645(a)(1)(B)(iii) (42 U.S.C.  9840(a)(1)(B)(iii)). 
34 Head Start Act  645(a)(1)(B)(iii)(I) (42 U.S.C.  9840(a)(1)(B)(iii)(I)). 
35 45 CFR  1305.4(e)(1)(iv) mandates that 51 percent of enrolled children be 
categorically eligible and/or income-eligible. 
36 Head Start Act  645(a)(2) (42 U.S.C.  9840(a)(2)).   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           17 
   
APPENDIX B 
Designation Renewal System Trigger Conditions 
Regulations at 45 CFR  1307.3 describe the conditions that serve as 
ACFs basis for determining whether a grantee will be required to 
recompete for renewal: 
 
 1307.3 Basis for determining whether a Head Start agency will 
be subject to an open competition. 
A Head Start or Early Head Start agency shall be required to 
compete for its next five years of funding whenever the 
responsible HHS official determines that one or more of the 
following seven conditions existed during the relevant time 
period covered by the responsible HHS officials review under 
 1307.7 of this part: 
(a) An agency has been determined by the responsible HHS 
official to have one or more deficiencies on a single review 
conducted under section 641A(c)(1)(A), (C), or (D) of the Act 
in the relevant time period covered by the responsible HHS 
officials review under section 1307.7. 
(b) An agency has been determined by the responsible HHS 
official based on a review conducted under section 
641A(c)(1)(A), (C), or (D) of the Act during the relevant time 
period covered by the responsible HHS officials review under 
 1307.7 not to have: 
(1) After December 9, 2011, established program goals for 
improving the school readiness of children participating in its 
program in accordance with the requirements of section 
641A(g)(2) of the Act and demonstrated that such goals: 
(i) Appropriately reflect the ages of children, birth to five, 
participating in the program; 
(ii) Align with the Head Start Child Development and Early 
Learning Framework, State early learning guidelines, and the 
requirements and expectations of the schools, to the extent that 
they apply to the ages of children, birth to five, participating in 
the program and at a minimum address the domains of 
language and literacy development, cognition and general 
knowledge, approaches toward learning, physical well-being 
and motor development, and social and emotional 
development; 
(iii) Were established in consultation with the parents of 
children participating in the program. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           18 
   
(2) After December 9, 2011, taken steps to achieve the school 
readiness goals described under paragraph (b)(1) of this section 
demonstrated by: 
(i) Aggregating and analyzing aggregate child-level assessment 
data at least three times per year (except for programs 
operating less than 90 days, which will be required to do so at 
least twice within their operating program period) and using 
that data in combination with other program data to determine 
grantees progress toward meeting its goals, to inform parents 
and the community of results, and to direct continuous 
improvement related to curriculum, instruction, professional 
development, program design and other program decisions; and 
(ii) Analyzing individual ongoing, child-level assessment data 
for all children birth to age five participating in the program 
and using that data in combination with input from parents and 
families to determine each childs status and progress with 
regard to, at a minimum, language and literacy development, 
cognition and general knowledge, approaches toward learning, 
physical well-being and motor development, and social and 
emotional development and to individualize the experiences, 
instructional strategies, and services to best support each child. 
(c) An agency has been determined during the relevant time 
period covered by the responsible HHS officials review under 
 1307.7: 
(1) After December 9, 2011, to have an average score across all 
classrooms observed below the following minimum thresholds 
on any of the three CLASS: 
Pre-K domains from the most recent CLASS: Pre-K 
observation: 
(i) For the Emotional Support domain the minimum threshold 
is 4; 
(ii) For the Classroom Organization domain, the minimum 
threshold is 3; 
(iii) For the Instructional Support domain, the minimum 
threshold is 2; 
(2) After December 9, 2011, to have an average score across all 
classrooms observed that is in the lowest 10 percent on any of 
the three CLASS: Pre-K domains from the most recent 
CLASS: Pre-K observation among those currently being 
reviewed unless the average score across all classrooms 
observed for that CLASS: Pre-K domain is equal to or above 
the standard of excellence that demonstrates that the classroom 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           19 
   
interactions are above an exceptional level of quality. For all 
three domains, the standard of excellence is a 6. 
(d) An agency has had a revocation of its license to operate a 
Head Start or Early Head Start center or program by a State or 
local licensing agency during the relevant time period covered 
by the responsible HHS officials review under  1307.7 of this 
part, and the revocation has not been overturned or withdrawn 
before a competition for funding for the next five-year period is 
announced. A pending challenge to the license revocation or 
restoration of the license after correction of the violation shall 
not affect application of this requirement after the competition 
for funding for the next five-year period has been announced. 
(e) An agency has been suspended from the Head Start or Early 
Head Start program by ACF during the relevant time period 
covered by the responsible HHS officials review under             
 1307.7 of this part and the suspension has not been 
overturned or withdrawn. If there is a pending appeal and the 
agency did not have an opportunity to show cause as to why 
the suspension should not have been imposed or why the 
suspension should have been lifted if it had already been 
imposed under 45 CFR part 1303, the agency will not be 
required to compete based on this condition. If an agency has 
received an opportunity to show cause, the condition will be 
implemented regardless of appeal status. 
(f) An agency has been debarred from receiving Federal or 
State funds from any Federal or State department or agency or 
has been disqualified from the Child and Adult Care Food 
Program (CACFP) any time during the relevant time period 
covered by the responsible HHS officials review under                  
 1307.7 of this part but has not yet been terminated or denied 
refunding by ACF. (A debarred agency will only be eligible to 
compete for Head Start funding if it receives a waiver 
described in 2 CFR 180.135.) 
(g) An agency has been determined within the twelve months 
preceding the responsible HHS officials review under                       
 1307.7 of this part to be at risk of failing to continue 
functioning as a going concern. The final determination is 
made by the responsible HHS official based on a review of the 
findings and opinions of an audit conducted in accordance with 
section 647 of the Act; an audit, review or investigation by a 
State agency; a review by the National External Audit Review 
(NEAR) Center; or an audit, investigation or inspection by the 
Department of Health and Human Services Office of Inspector 
General. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           20 
   
APPENDIX C 
Detailed Methodology 
This review used multiple data sources to review the second cohort of 
Head Start grantees to undergo DRS assessment and participate in 
recompetition.  We reviewed all 361 grantees in the cohort.   
Data Sources 
We used data from the following ACF sources and systems: 
Program Information Report (PIR).  All grantees are required to submit 
PIR data to ACF annually.  These are summary data that describe a wide 
range of characteristics of grantees and the populations they serve, such as 
location, number of children served, number of children in each eligibility 
category, etc.  ACF also calculates and makes public a variety of 
performance indicators based on grantees PIR data.37  We used ACFs 
formulas when calculating PIR performance indicators used in this review.  
We used PIR data that grantees reported for the 20112012 program year, 
because this period most closely aligned with the Cohort 2 monitoring 
process.   
Performance monitoring system.  For all grantees in Cohort 2, we 
reviewed data from ACFs performance monitoring system to identify 
noncompliances, deficiencies, and strengths identified during triennial 
reviews and other monitoring reviews from October 2009 through 
June 2014.  These data included counts of noncompliances and 
deficiencies and narrative descriptions of grantee strengths.  We 
determined the number of separate categories of strengths for each grantee 
by conducting a qualitative review of strength narratives and categorizing 
them according to the seven programmatic areas described in the Head 
Start Monitoring Protocol.38  Because the majority of issues are identified 
during triennial reviews, when grantees had received more than one 
triennial review during the period, we included only the most recent 
triennial review.  
DRS.  For each grantee, we reviewed DRS data provided by ACF to 
determine the final DRS determination and the DRS trigger conditions that 
were present.  
 
37 For example, see ACF, Office of Head Start  Head Start Services Snapshot National 
(20142015).  Accessed at http://eclkc.ohs.acf.hhs.gov/hslc/data/psr/2015/services-
snapshot-hs-2014-2015.pdf on April 25, 2016. 
38 The seven programmatic areas in the Protocol are program governance; management 
systems; fiscal integrity; eligibility, recruitment, selection, enrollment, and attendance; 
child health and safety; family and community engagement; and child development and 
education. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           21 
   
CLASS scores.  For each grantee, ACF provided OIG with the CLASS 
scores used in the DRS assessment.  
Grant recompetition results.  For each grant that was recompeted, ACF 
provided OIG with (1) a list of applicants for each recompeted grant and 
(2) the result of the recompetition, i.e., whether the incumbent grantee was 
fully renewed, partially renewed, or not renewed.  
Analysis 
Assessing DRS determinations.  To assess determinations made under the 
DRS in the second year of implementation, we calculated the proportion 
of Head Start grantees in Cohort 2 that were required to compete for 
renewal.  We also identified the most common DRS triggers.   
Further, to determine whether certain types of grantees fared better or 
worse under the DRS, we compared descriptive characteristics of grantees 
that were, and were not, required to recompete.  Specifically, we identified 
and compared, for each group:  
  Average funded enrollment.  A grantees funded enrollment is the 
total number of enrollees the program was funded to serve for the 
enrollment year.  This information is reported as part of the PIR. 
  Average proportion of grantees centers located in rural areas.  To 
determine this, we obtained Head Start center addresses from ACF 
and coded them as rural or urban based on Rural-Urban 
Commuting Area (RUCA) codes.39  We then calculated the 
proportion of each grantees centers that were in rural locations. 
  Average proportion of families who speak a language other than 
English at home.  This information is reported as part of the PIR. 
  Average proportion of families served who experienced 
homelessness during the program year.  We used this item as 
a proxy for extreme poverty.  This information is reported as part 
of the PIR.   
We consulted with ACF in the selection of these descriptive factors.   
 
39 RUCA codes were developed through a collaborative project between HHSs Health 
Resources and Service Administration, the U.S. Department of Agricultures Economic 
Research Service, and the WWAMI [Washington, Wyoming, Alaska, Montana, and 
Idaho] Rural Health Research Center.  We used the standard dichotomous definition of 
urban and rural.  Specifically, we coded centers in locations with RUCA codes 1.0, 
1.1, 2.0, 2.1, 3.0, 4.1, 5.1, 7.1, 8.1, and 10.1 as urban.  We coded centers in locations with 
RUCA codes 4.0, 4.2, 5.0, 5.2, 6.0, 6.1, 7.0, 7.2, 7.3, 7.4, 8.0, 8.2, 8.3, 8.4, 9.0, 9.1, 9.2, 
10.0, 10.2, 10.3, 10.4, 10.5, and 10.6 as rural. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           22 
   
For each factor, we compared the two groups to determine whether any 
differences exist between grantees that were automatically renewed under 
the DRS and those that were required to recompete.  To determine whether 
observed differences between the two groups were consistent with random 
distribution or whether they reflect meaningful associations, we conducted 
permutation testing (using 500,000 permutations).  Specifically, we used 
analytic software to re-randomize the population 500,000 times and 
compared the distribution of key variables to that observed in the 
population.  Permutation test p-values less than or equal to 0.0500 indicate 
a significant difference, i.e., a difference that is most likely not attributable 
to random variation. 
Additionally, to determine the extent to which DRS determinations 
(regarding which grantees would be required to recompete) were 
consistent with other performance data, we compared the DRS 
determinations for Cohort 2 grantees to the grantees past performance on 
six measures that are collected by ACF but are not included in the DRS:   
  number of noncompliances (identified during ACF monitoring 
reviews); 
  number of categories of strengths (identified during ACF 
monitoring reviews and categorized by OIG using the seven 
programmatic areas described in the Head Start Monitoring 
Protocol); 
  proportion of children who are up to date on a schedule of 
preventive and primary care per the States schedule (an 
ACF-defined performance indicator calculated from PIR data); 
  proportion of preschool children with an individualized education 
plan for one of the primary disabilities reported in the PIR who 
received special education or related services for those disabilities 
(an ACF-defined performance indicator calculated from PIR data); 
  proportion of preschool classrooms in which at least one teacher 
met the degree/credential requirements of Section 648A(3)(B) of 
the Head Start Act, i.e., had an A.A. or higher in early childhood 
education or equivalent (an ACF-defined performance indicator 
calculated from PIR data); and  
  proportion of preschool teachers who met the degree/credential 
requirements of Section 648A(2)(A) of the Head Start Act, i.e., 
a B.A. or higher in early childhood education or equivalent (an 
ACF-defined performance indicator calculated from PIR data). 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           23 
   
We consulted with ACF in the selection of these performance measures to 
confirm that they were relevant and accurate bases for assessment and 
comparison of grantees.   
For each measure, we compared the two groups average performance to 
determine whether grantees designated for automatic, noncompetitive 
renewal exhibited superior past performance.  To determine whether the 
observed differences between the groups were consistent with random 
distribution or reflected meaningful associations, we conducted 
permutation testing (using 500,000 permutations). 
Describing grant renewal decisions under recompetition.  To describe 
grant renewal decisions made in the second year of recompetition, we 
reviewed applicants for each recompeted grant in Cohort 2 and the 
outcomes of those recompetitions.  We determined the proportion of 
recompeted grants that were renewed vs. the proportion awarded to a 
different grantee.  As part of this analysis, we also calculated the average 
number and range of applicants, as well as how often the incumbent 
grantee was the sole applicant.   
Determining whether grantees with lower performance on selected 
measures left the Head Start program through the DRS and recompetition 
processes.  To identify lower performing grantees, we first selected 
10 performance measures that encompass a range of grantee 
responsibilities and include both DRS and non-DRS measures.  These 
included: 
  Three measures drawn from ACFs onsite monitoring reviews:  the 
number of deficiencies, number of noncompliances, and number of 
categories of strengths; 
  Three CLASS scores that ACF-contracted reviewers determined on 
the basis of classroom observation:  instructional support, 
emotional support, and classroom organization; and 
  Four selected measures that we calculated (using ACFs 
performance indicator formulas) from grantee-reported PIR data:  
provision of preventive and primary care; services for children 
with disabilities; percentage of classrooms in which at least one 
teacher had an A.A. in early childhood education or equivalent; 
and percentage of preschool teachers overall with a B.A. in early 
childhood education or equivalent.  
Of the above 10 measures, ACF includes 4 measuresthe number of 
deficiencies and the 3 CLASS scoresin its DRS assessments.  ACF 
collects the data for the remaining six measures for management purposes 
but does not use them in DRS assessments. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           24 
   
For each measure, we calculated the scores for the bottom fifth percentile 
and bottom tenth percentile among Cohort 2 grantees.40  A grantee met our 
criteria as lower performing relative to its peers if it (a) scored in the 
bottom 10 percent of Cohort 2 grantees on 4 or more measures, or 
(b) scored in the bottom 5 percent of Cohort 2 grantees on 2 or more 
measures.  We then summarized the DRS and recompetition results for 
this subset of grantees. 
   
 
40 For example, for noncompliances, a higher number is a worse score.  As a result, the 
bottom fifth and tenth percentiles therefore include the grantees with the most 
noncompliances.  In contrast, for a measure of the percentage of teachers with specified 
credentials, a higher number is a better score, and therefore the bottom fifth and tenth 
percentiles include the grantees with the lowest percentage of credentialed teachers. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           25 
   
APPENDIX D 
Comparison of Automatically Renewed vs. Recompeting 
Grantees:  Statistical Testing  
Descriptive Factors 
We conducted permutation testing to determine whether observed 
differences between automatically renewed grantees and recompeting 
grantees reflected random variation or meaningful association.  For each 
the four descriptive factors we testedenrollment size, proportion of 
centers in a rural location, proportion of families who do not speak 
English at home, and proportion of families experiencing homelessness 
permutation testing yielded p-values greater than 0.0500, indicating no 
significant association between these variables and grantees DRS 
determinations.  In other words, our results indicate that the DRS did not 
unduly disadvantage these categories of grantees.  See Table 1. 
Table 1:  Comparison of Descriptive Factors:  Statistical Testing 
Grantees  Grantees 
Designated for  Required to  Permutation 
Descriptive Factors 
Automatic  Recompete  Test P-Value* 
Renewal (n=246)  (n=115) 
Average size (number of 
children the grantee is  556  607  >0.9999 
funded to serve) 
Average percentage of 
grantees centers in rural  34%  29%  0.9917 
locations 
Average percentage of 
families served who do not  21%  22%  >0.9999 
speak English at home 
Average percentage of 
families served who 
5.6%  3.8%  0.0538 
experienced homelessness 
(extreme poverty) 41 
  Source:  OIG analysis of ACF data. 
  * A p-value greater than 0.0500 indicates that the observed difference is most likely attributable to  
    random variation. 
 
 
41 This result suggests a possible marginal association (p-value of 0.0538) between the 
DRS determination and the percentage of a grantees families that experienced 
homelessness.  However, the direction of the association favored grantees serving a 
higher proportion of homeless families, suggesting that grantees serving more 
impoverished populations are not disadvantaged under the DRS. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           26 
   
Performance Data 
We also used permutation testing to determine whether grantees DRS 
determinations were consistent with other ACF performance data.  For one 
of the six performance measures we reviewedpast noncompliance 
findingsthe difference between grantees that were automatically 
renewed and grantees that were required to recompete was highly 
significant.  However, for the remaining five measures, testing resulted in 
p-values greater than 0.0500, indicating no significant difference between 
the two groups.  See Table 2. 
Table 2:  Comparison of Performance Measures:  Statistical Testing 
Grantees  Grantees 
Designated for  Required to  Permutation 
Performance Measures 
Automatic  Recompete  Test P-Value* 
Renewal (n=246)  (n=115) 
Average number of noncompliances  2.46  4.44  <0.0001 
Average number of strength 
1.59  1.40  0.6418 
categories 
Average percentage of children 
current on preventive and primary  93%  93%  >0.9999 
health care services 
Average percentage of preschool 
children receiving appropriate  98%  94%  0.0685 
disability services 
Average percentage of classes in 
which at least one teacher has an 
88%  88%  >0.9999 
associates degree or higher in early 
childhood education or a related field 
Average percentage of preschool 
teachers with a bachelors degree or 
63%  60%  0.9993 
higher in early childhood education or 
a related field 
  Source:  OIG analysis of ACF data. 
  * A p-value greater than 0.0500 indicates that the observed difference is most likely attributable to  
    random variation. 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           27 
   
APPENDIX E 
 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           28 
   
 
   
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           29 
   
ACKNOWLEDGMENTS 
This report was prepared under the direction of Dave Tawes, Regional 
Inspector General for Evaluation and Inspections in the Baltimore regional 
office, and Louise Schoggen, Assistant Regional Inspector General.  
Louise Schoggen served as the team leader for this study.  Central office 
staff who provided support include Kevin Farber, Joanne Legomsky, 
Christine Moritz, Melicia Seay, and Sherri Weinstein.  
 
 
Head Start Grant Recompetition:  Early Results Suggest Opportunities for Improvement (OEI-12-14-00650)           30 
   
Office of Inspector General
http://oig.hhs.gov  
 
The mission of the Office of Inspector General (OIG), as mandated by Public Law 95452, as  
amended, is  to protect the integrity of the Department of  Health and Human Services  
(HHS) programs, as  well  as the health  and welfare of individuals served by those programs.  
This statutory mission is carried  out through a nationwide network of audits, investigations,  
and inspections conducted by the following operating components: 
Office of Audit Services 
The Office  of  Audit Services  (OAS) provides auditing services  for HHS, either by  conducting  
audits  with its own audit resources or by  overseeing  audit work done by others.  Audits  
examine the performance of HHS programs and/or its grantees and contractors in carrying  
out their respective responsibilities and are intended  to provide independent assessments of 
HHS programs and operations.  These assessments help reduce waste, abuse, and  
mismanagement and promote economy and efficiency  throughout  HHS. 
Office of Evaluation and Inspections 
The Office  of  Evaluation and Inspections (OEI)  conducts national evaluations to  provide 
HHS, Congress, and the public with timely, useful, and reliable information on significant 
issues.  These evaluations focus on preventing fraud,  waste, or abuse  and promoting  
economy, efficiency, and effectiveness of departmental programs.  To promote impact, OEI 
reports also present practical recommendations for improving program operations.  
Office of Investigations 
The Office of Investigations (OI) conducts criminal, civil, and administrative investigations  
of fraud and misconduct  related to HHS programs, operations, and individuals.  With  
investigators working in all 50 States and the District of Columbia, OI  utilizes its resources 
by actively  coordinating with the Department  of Justice  and other Federal, State, and local 
law enforcement authorities.  The investigative efforts of OI often lead to  criminal 
convictions, administrative sanctions,  and/or  civil monetary  penalties.  
Office of Counsel to the Inspector General 
The Office of Counsel to the  Inspector  General (OCIG) provides general legal services to 
OIG, rendering  advice and opinions on HHS programs and operations and  providing all  
legal support for OIGs  internal operations.  OCIG represents  OIG in all civil and 
administrative fraud and  abuse cases involving HHS programs, including False Claims Act, 
program exclusion, and civil monetary penalty cases.  In  connection with these cases, OCIG 
also negotiates and monitors corporate integrity agreements.  OCIG renders advisory 
opinions, issues compliance program  guidance, publishes fraud alerts, and provides other  
guidance  to  the health care industry concerning the anti-kickback statute and other  OIG  
enforcement authorities.