Head Start: Further Development Could Allow Results of New Test to Be Used for Decision Making
Published: 2005-05-17T00:00:00

Fast Facts:

Highlights:
section_title: Why GAO Did This Study
paragraphs: In September 2003, the Head Start Bureau, in the Department of Health and Human Services (HHS) Administration for Children and Families (ACF), implemented the National Reporting System (NRS), the first nationwide skills test of over 400,000 4- and 5-year-old children. The NRS is intended to provide information on how well Head Start grantees are helping children progress. Given the importance of the NRS, this report examines: what information the NRS is designed to provide; how the Head Start Bureau has responded to concerns raised by grantees and experts during the first year of implementation; and whether the NRS provides the Head Start Bureau with quality information.
section_title: What GAO Found
paragraphs: The Head Start Bureau developed the NRS to gauge the extent to which Head Start grantees help children progress in specific skill areas, including understanding spoken English, recognizing letters, vocabulary, and early math. Due to time constraints and technical matters, the Head Start Bureau adapted portions of other assessments for use in the NRS. Head Start Bureau officials have responded to some concerns raised during the first year of NRS implementation, but other issues remain. For example, the Head Start Bureau has modified training materials and is exploring the feasibility of sampling. However, it is not monitoring whether grantees are inappropriately changing instruction to emphasize areas covered in the NRS. Head Start Bureau officials have said NRS results will eventually be used for program improvement, targeting training and technical assistance, and program accountability; however, the Head Start Bureau has not stated how NRS results will be used to realize these purposes. Currently, results from the first year of the NRS are of limited value for accountability purposes because the Head Start Bureau has not shown that the NRS meets professional standards for such uses, namely that (1) the NRS provides reliable information on children's progress during the Head Start program year, especially for Spanish-speaking children, and (2) its results are valid measures of the learning that takes place. The NRS also may not provide sufficient information to target technical assistance to the Head Start centers and classrooms that need it most.

section_title: Letter
paragraphs: In fall 2003, the federal Head Start program initiated a nationwide skills  test of over 400,000 4- and 5-year-old children. This test, called the Head  Start National Reporting System (NRS), is intended to meet a long- standing need for systematic information on how well specific Head Start  grantees are helping children learn. Head Start is designed to promote  school readiness and healthy development among poor preschool children  and provides services to nearly 1 million children, generally between the  ages of 3 and 5, through nearly 1700 grantees. These grantees or their  delegates provide services at about 19,000 Head Start centers nationally,  with each grantee having from 1 to over 100 centers. For nearly a decade  the Head Start Bureau (HSB) and the U.S. Department of Health and  Human Services (HHS) have been engaged in promoting accountability  and moving toward a results-oriented evaluation of Head Start. The NRS  builds on this work. The NRS was developed in response to President  Bush’s April 2002 announcement of the “Good Start, Grow Smart” early  childhood initiative that directed HHS to develop a national accountability  system to ensure that every Head Start grantee will assess the progress  made by children in early literacy, language, and numeracy skills.
Head Start teachers, or others trained as NRS assessors, administer the  NRS to children individually in the fall and spring of the Head Start year.  The NRS begins with a game of “Simon Says,” lasts about 15 minutes, and  includes four sub-tests designed to screen for understanding of spoken  English and to assess skills in recognizing letters, vocabulary, and early  math. During the test, an assessor sits across from a child at a table and  asks scripted questions of the child, and the child responds by verbally  identifying or pointing to pictures, numbers, or letters that are contained  in a 3-ring binder. The assessor marks the child’s responses on a  computer-readable scoring sheet. While all of the children are given at  least the portion of the English-language assessment that screens for  understanding of spoken English, children whose primary language is  Spanish are also assessed using a Spanish version of the NRS. Children  who speak both English and Spanish are given both versions of the NRS  and scores from both tests are reported separately.
Although other evaluations of children’s skills and Head Start performance  exist, the NRS differs from them in its scale, type, and purpose. The NRS is  a standardized test intended for all prekindergarten Head Start children. It  represents the first time that HSB will use children’s performance on a  standardized test to measure how well specific Head Start grantees are  helping children progress. Many in the Head Start community and beyond  agree that it is a laudable goal to look at Head Start at the national and  grantee levels to determine whether Head Start achieves its stated  objectives. However, there have been significant concerns about whether  the NRS, as currently composed, is the right way to accomplish this goal.
Given the importance HSB places on measuring Head Start performance  and the concerns about the NRS, we examined (1) what information the  NRS is designed to provide, (2) how HSB has responded to  implementation issues raised by the Head Start grantees and experts  during the first year of NRS implementation, and what issues remain to be  addressed, and (3) whether the NRS provides HSB with the quality of  information it needs to meet its purposes.
To answer these questions, we collected and analyzed information from  multiple sources. To determine what information the NRS is designed to  provide, we interviewed representatives from HSB, its contractors, and  early childhood professional organizations and we reviewed documents  chronicling the steps HSB took in developing the NRS. To examine how  HSB responded to implementation issues raised by Head Start grantees  and experts during the first year of NRS implementation and what issues  remain to be addressed, we interviewed representatives from HSB and  randomly sampled Head Start grantees and delegates from the population  of all Head Start grantees and delegates during the 2003-2004 school year.  We received responses from 80 percent of the grantees and delegates we  surveyed. We also visited 12 Head Start grantees in 5 states (Colorado,  Maryland, Massachusetts, Rhode Island, and Virginia), to interview staff  who conducted the assessments and to observe them administering the  NRS to children. The states and grantees chosen for site visits were  judgmentally selected to include a range of enrollment sizes, types of  program, rural and urban locations, and linguistic populations. Finally, to  examine whether the NRS provides HSB with the quality of information it  needs to meet its goals, we reviewed the professionally accepted  standards for test development, interviewed all of the members of the  Technical Work Group—a team of experts convened to assist HSB and its  contractors in the design and implementation of the NRS—and consulted  with individuals recommended by the National Academy of Sciences as  experts in the areas of test design and the educational testing of Spanish- speaking and bilingual children. These independent experts reviewed  documents provided by HSB and its contractors pertaining to the  adequacy and appropriateness of the assessment. See appendix I for  additional information on our scope and methodology. We conducted our  work between May 2004 and February 2005 in accordance with generally  accepted government auditing standards.
subsections: section_title: Background
paragraphs: Established in 1965, Head Start is a federally funded early childhood  development program that served over 900,000 children at a cost of  $6.8 billion in 2004. Head Start offers low-income children a broad range of  services, including educational, medical, dental, mental health, nutritional,  and social services. Children enrolled in Head Start are generally between  the ages of 3 and 5 and come from varying ethnic and racial backgrounds.  Head Start is administered by HSB within ACF. HSB awards Head Start  grants directly to local grantees. Grantees may develop or adopt their own  curricula and practices within federal guidelines. Grantees may contract  with other organizations—called delegate agencies—to run all or part of  their local Head Start programs. Each grantee or delegate agency may  have one or more centers, each containing one or more classrooms. In this  report, the term “grantee” is used to refer to both grantees and delegate  agencies. Figure 1 provides information on the numbers of Head Start  grantees, delegate agencies, centers and classrooms.
Since the inception of Head Start, questions have been raised about the  effectiveness of the program. In 1998, we reported that Head Start lacked  objective information on performance of individual grantees and Congress  enacted legislation requiring HSB to establish specific educational  standards applicable to all Head Start programs and allowed development  of local assessments to measure whether the standards are met. HSB  implemented this legislation by developing the Child Outcomes  Framework to guide Head Start grantees in their ongoing assessment of  the progress of children. The Framework covers a broad range of child  skill and development areas and incorporates each of the legislatively  mandated goals, such as that children “use and understand an increasingly  complex and varied vocabulary” and “identify at least 10 letters of the  alphabet.”
Since 2000, HSB has required every Head Start grantee to include each of  the areas in the Framework in the child assessments that each grantee  adopts and implements. The eight broad areas included in the Framework  are language development, literacy, mathematics, science, creative arts,  social and emotional development, approaches to learning, and physical  health and development. Grantees are permitted to determine how to  assess children’s progress in these areas. These assessments are to align  with the grantee’s curriculum; as a result the specific assessments vary  across the grantees. The assessments occur 3 times each year and  generally involve observing the children during normal classroom  activities. The results of the assessments are used for the purposes of  individual program improvement and instructional support and are not  aggregated across grantees or systematically shared with federal officials.  The NRS, prompted by the April 2002 announcement of President Bush’s  Good Start, Grow Smart initiative, builds on the 1998 legislation by  requiring all Head Start programs to implement the same assessment,  twice a year, to all 4- and 5-year-old Head Start participants who will  attend kindergarten the following year.
When President Bush announced this initiative in April 2002, it called for  full implementation in fall 2003; as a result the NRS was developed and  preparations for implementation occurred within an 18-month period. See  figure 2. Shortly after the President announced this initiative, HSB hired a  contractor to assist it in developing and implementing the NRS. The  contractor, working closely with HSB, was responsible for the design and  field testing of the NRS, including developing training materials to support  national implementation of the reporting system by grantees. HSB also  worked with the Technical Work Group and others throughout  implementation of the NRS. The Technical Work Group includes 16  experts in such areas as child development, educational testing, and  bilingual education. They advised HSB on the selection of assessments, the  appropriateness of the assessments in addressing the mandated indicators,  the technical merit of the assessments, and the overall design of the NRS.  While the Technical Work Group members offered advice, the group  members were not always in agreement with each other and HSB was not  obligated to act on any of the advice it received. A list of the Technical  Work Group members and their professional affiliations is included in  appendix I.
Through focus groups, teleconferences, and various correspondences,  HSB officials communicated to Head Start grantees the purpose of the  NRS and their plans for administering the assessment. Focus groups and  discussions were held with various interested parties, including Head Start  managers and directors and experts from universities and the public  sector, on issues ranging from strengths and limitations of various  assessment tools to strategies for assessing non-English speaking children.  HSB also received input through a 60-day public comment period, from  mid-April to June 2003.
Another contractor developed a Computer-Based Reporting System  (CBRS) for the NRS. Local Head Start staff use the CBRS to enter  descriptive information about their grantees, centers, classrooms,  teachers, and children, as shown in table 1, as well as to keep track of  which children have been assessed. HSB analyzes the descriptive  information from the CBRS in conjunction with the child assessment data  to develop reports on the progress of specific subgroups of children. For  example, HSB can report separately on the average scores of children  enrolled in part-day programs and those enrolled in full-day programs.
HSB, with assistance from the contractors, worked to ensure local staff  received adequate training on administering the assessment and using the  CBRS, and provided guidance on how to obtain consent from parents.  Training and certification of all assessors was required so that all  assessors would administer the NRS in the same way. Two-and-a-half day  training sessions were held at eight sites throughout the U.S. and Puerto  Rico during July and August 2003. Roughly 2,800 individuals completed the  training, of which 484 were certified in both English and Spanish. In turn,  these certified trainers held training sessions locally to train and certify  additional staff who would be able to administer assessments.
The development of educational tests is a science in itself, to which  university departments, professional organizations, and private companies  are devoted. Among the most important concepts in test development are  validity and reliability. Validity refers to whether the test results mean  what they are expected to mean and whether evidence supports the  intended interpretations of test scores for a particular purpose. Reliability  refers to whether or not a test yields consistent results. Validity and  reliability are not properties of tests; rather, they are characteristics of the  results obtained using the tests. For example, even if a test designed for  4th graders were shown to produce meaningful measures of their  understanding of geometry, this wouldn’t necessarily mean that it would  do so when administered to 2nd or 6th graders or with a change in  directions allowing use of a compass and ruler. Test developers typically  implement “pilot” tests that represent the actual testing population and  conditions and they use data from the pilot to evaluate the reliability and  validity of a test. This process generally takes more than 1 year, especially  if the test is designed to measure changes in performance.
In the remainder of the report, we will discuss how the focus of the NRS  was determined and the assessment was developed, HSB’s response to  problems in initial implementation as well as some implementation issues  that remain unaddressed, and the extent to which the assessment meets  the professional and technical standards to support specific purposes  identified by HSB.
section_title: NRS Assesses Selected Skills Using Adaptations of Other Assessments
paragraphs: The NRS assesses vocabulary, letter recognition, simple math skills, and  screens for understanding of spoken English. As initially conceived by  HSB, the NRS was to gauge the progress of Head Start children in 13  congressionally mandated indicators of learning. However, time  constraints and technical matters precluded HSB from assessing children  on all of the indicators and led HSB to consider, and eventually adopt,  portions of other assessments for use in the NRS.
The 18 months from announcing the Good Start, Grow Smart initiative, of  which the NRS is a part, to implementing the assessment was not enough  time for HSB to develop a completely new assessment. Therefore, HSB,  with the advice of its contractor and the Technical Work Group, chose to  borrow material from existing assessments. Concerns raised by Technical  Work Group members and the contractor about the length and complexity  of the assessment and the technical adequacy of individual components  eventually led to limiting the areas assessed in the NRS, from 13 skills to 6.
The six legislatively mandated skills that HSB targeted included whether  children in Head Start:    use increasingly complex and varied spoken vocabulary;    understand increasingly complex and varied vocabulary;   identify at least 10 letters of the alphabet;    know numbers and simple math operations, such as addition and  subtraction;  for non-English speaking children, demonstrate progress in listening to  and understanding English; and  for non-English speaking children, show progress in speaking English.
In April and May of 2003 an assessment that included 5 components  covering the 6 skills was field tested with 36 Head Start programs to  examine the basic adequacy of the NRS, as well as the method for training  assessors, and the use of the CBRS. The field test also included a Spanish  version of the NRS. Based on the field test, one component–-phonological  awareness, or one’s ability to hear, identify, and manipulate sounds–-was  eliminated. While this component examined an area that experts have  linked to prevention of reading difficulties, the test used to assess it was  problematic. HSB moved forward with the other components of the NRS.  The four components of the NRS each measure one or more of the six  legislatively-mandated indicators.
The four components that comprise the NRS are from the following tests:    Oral Language Development Scale (OLDS) of the Pre-Language  Assessment Scale 2000 (Pre-LAS 2000),    Third Edition of the Peabody Picture Vocabulary Test (PPVT-III),    Head Start Quality Research Centers (QRC) letter-naming exercise, and    Early Childhood Longitudinal Study of a kindergarten cohort (ECLS-K)  math assessment.
Some or all of each test was previously used for other studies, and the  PPVT and letter naming were previously used in studies of Head Start  children. Three of the four tests were modified from their original version,  as shown in table 2. Figures 3 and 4 are examples from the letter naming  and early math skills components of the NRS. Figure 5 is an example of  the type of item used in the vocabulary (PPVT) component of the NRS.
Here are some letters of the alphabet.
GESTURE WITH A CIRCULAR MOTION AT LETTERS AND SAY:  Point to all the letters that you know and tell me the name of  each one. Go slowly and show me which letter you’re naming.
INDICATE ONLY CORRECTLY NAMED LETTERS ON ANSWER SHEET.
WHEN CHILD STOPS NAMING LETTERS, SAY:  Look carefully at all of them. Do you know any more?
KEEP ASKING UNTIL CHILD DOESN’T KNOW ANY MORE.