Head Start: Research Insufficient to Assess Program Impact
Released: 1998-03-26T00:00:00

Letter

Background

Since 1965, Head Start’s primary goal has been to improve the social competence of children in low-income families, that is, their everyday ability to deal with both their current environment and later responsibilities in school and life. This considers the relationships between cognitive and intellectual development, physical and mental health, nutritional needs, and other factors. Head Start delivers, or provides access to, a wide range of services—educational, medical, dental, nutrition, mental health, and social services. HHS administers the Head Start program through its Head Start Bureau within the Administration for Children and Families (ACF). coordinate with public health agencies to obtain health services, while others contract with local physicians. Although all grantees operate under one set of performance standards, they have a great deal of discretion in implementing those standards, resulting in programs that vary.

In addition to providing services to children and families, Head Start sees one of its roles as a national laboratory for child development. Consequently, Head Start uses much of its discretionary research funding for demonstrations and studies of program innovations. The amount of funds allocated to research, demonstration, and evaluation has represented about 2 percent of the Head Start budget over the years. About $12 million (about 0.3 percent of the Head Start budget) was so allocated for fiscal year 1997.

The main focus of the program’s research, according to Head Start Bureau officials, has been to improve the program by exploring ways to maximize and sustain Head Start benefits. In addition, Head Start funds studies designed to answer questions on the effectiveness of new or innovative service delivery strategies. Such studies typically involve special program efforts and demonstration projects conducted on a trial basis at a few Head Start sites that focus on practices or services not typically found in regular Head Start programs.

The passage of the Results Act in 1993 has heightened the importance of the type and direction of this research. The Results Act is designed to hold federal agencies accountable for achieving program results. The act specifically requires that agencies clearly define their missions, establish long-term strategic goals as well as annual goals linked to them, measure their performance according to their performance goals, and report on their progress. Agencies are also expected to perform discrete program evaluations and to use information from these evaluations to improve their programs. impact as differences in outcomes caused by Head Start participation. Essentially, impact evaluations are the only way to answer the question, “Is this program making a difference?” Impact evaluation is a form of program evaluation that assesses the net impact of a program by comparing its outcomes with an estimate of what would have happened without the program. This form of program evaluation is used when external factors are known to influence the program’s outcomes; it isolates program contributions from other factors that may affect the achievement of program objectives. The most reliable way to determine program impact is to compare a group of Head Start participants with an equivalent group of nonparticipants. The preferred method for establishing that the groups are equivalent at the outset is to randomly assign participants to either a Head Start group or a comparison group, although other methods are valuable for estimating a program’s net impact.

Evaluation of Past Research

In 1997, we reported the results of our work on identifying what existing studies suggest about Head Start’s impact. To conclude that impact has been demonstrated, one would expect to see either (1) a sufficient number of reasonably well-designed individual studies whose findings could appropriately be combined to provide information on national impact or (2) at least one large-scale evaluation using a nationally representative sample. After locating and screening 600 studies and consulting with many early childhood researchers and officials at the Head Start Bureau, we identified only 22 studies that met the criteria for inclusion in our analysis. other early childhood programs for low-income families has been growing. Thus, the Congress needs to know with some certainty whether the federal investment in Head Start is making a difference.

In commenting on our earlier report, HHS said that the existing research on Head Start’s impact was substantial and that the Department’s strategy to expand this research was appropriate for determining both the program’s impact and its quality. HHS also indicated plans to evaluate the feasibility of conducting impact studies such as we recommended.

HHS supported its claim that the existing research was substantial by noting the findings from a 1985 research synthesis of studies conducted in the 1960s and 1970s and two more recent studies. We disagreed, however, that findings drawn from studies more than 20 years old adequately support claims about the current program’s impact. As noted, the current Head Start program operates in an environment that has changed in the last 20 years, when other, non-Head Start comprehensive early childhood services were not as available. Similarly, the findings from the two more recent studies did not support conclusions about program impact that can be generalized to the national program. Even though these two studies were larger than others we had found, both had significant methodological limitations.

Current Initiatives

HHS’ current initiatives reflect its opinion that a randomized control group is not necessary to measure Head Start’s impact. The current initiatives HHS describes as assessing impact include (1) the development of new performance measures, (2) a longitudinal study called FACES, and (3) a collaborative effort with NCES. assess the net impact of the Head Start program. It will allow Head Start to define and assess program outcomes, such as improved language skills, that it could then use to compare Head Start participants’ outcome results with those of a control group to determine impact.

Another HHS initiative, FACES, is a study of a representative sample of Head Start children and their families intended to show whether Head Start is reaching its goal of improving children’s social competence. According to HHS, for the spring 1997 pilot, data were collected from a sample of 2,400 families with children enrolled in 160 randomly selected centers in 40 Head Start programs nationwide. The full study will collect data from 3,200 families at program entry, exit, and at the end of kindergarten. HHS will conduct a more comprehensive validation substudy of 120 families. Researchers will use well-established and widely used scales, assessments, and observational protocols and specially tailored questionnaires to collect data on children’s vocabulary, emergent literacy and mathematical skill, perceptual-motor development, and social and communicative competence before and after Head Start participation.

Head Start officials describe FACES as a way to draw conclusions about Head Start’s impact in part because it will use nationally normed instruments. In addition, some of the FACES data elements will be the same as those in a Department of Education national household education survey. This will allow for comparing certain FACES results with a nationally representative sample of low-income children. It is not clear from our work so far how HHS will use the nationally normed data. According to HHS officials, the study will not compare Head Start children and their families with a randomly assigned control group of other children and families or with any other group. collect data from parents and children, including descriptions of children’s preschool experience and standardized tests in areas such as achievement and psychomotor development. This database will be available as a public-use tape for Head Start as well as other researchers. Head Start could use this database to compare groups of children in non-Head Start preschool programs with those in Head Start programs to assess program impact.

Improvement on Initiatives

Head Start’s initiatives are headed in the right direction because of their increased focus on outcomes and research that could be expanded to compare outcomes for children in Head Start with those for similar children and families not served by the program. It is not clear how or whether Head Start will make these comparisons, however, using nationally normed tests or comparison group data from NCES. In addition, either of those research designs provides a much weaker basis for drawing conclusions about impact than a study with randomized assignment. For example, if Head Start uses NCES data for comparisons, the results could provide some indication of program impact. Some question will always remain, however, about the degree to which preexisting differences in the groups may have affected study results. True experimental designs, also called randomized trials, eliminate such questions. Randomized trials are comparison group studies that randomly assign study participants to either a treatment or control group. In the case of Head Start, these studies would require recruiting more eligible children than the program can serve. From these recruits, some would be randomly assigned to Head Start; the rest, the unserved children, would constitute the control group. HHS officials cited ethical considerations of assigning children to an unserved control group as one of the difficulties in conducting randomized trials. suggested that Head Start conduct randomized trials to study regular Head Start programs because this type of study provides the most conclusive information on program impact. In fact, the evaluation of the Early Head Start program, now under way, has randomly assigned potential participants to Early Head Start or a control group that has not received Early Head Start services. Control groups of randomly assigned participants are important to determining impact because they prevent mistakenly attributing outcomes to program effects when these outcomes are really caused by other factors. For instance, a recent evaluation of the Comprehensive Child Development Program, a demonstration project involving comprehensive early childhood services like those of Head Start, found positive changes in the families participating. The study had a control group, however, and researchers discovered that the control group families also had similar positive changes. They concluded therefore that the positive changes could not be attributed to the program.

Conclusions

Although impact research can be costly and time consuming, the federal government has made a considerable financial investment in the Head Start program; therefore, Head Start warrants a close examination to determine what the public is getting for its investment. Head Start has devoted substantial resources to research and evaluation activities, including some long-term studies and studies involving comparison groups. Although these have been worthwhile efforts, they have not sufficiently focused on evaluating results. HHS is taking steps that may help lay the groundwork for efforts to evaluate the net impact of Head Start program services. Identifying performance measures is an important first step in building a research and impact evaluation base for Head Start. In addition, this effort could yield a set of common measures upon which a body of research, including impact research, could be built. Similarly, the information gained in FACES should be extremely useful, especially to the extent that it is nationally representative.

HHS efforts, however, do not include plans for a research study or set of studies that will definitively compare the outcomes achieved by Head Start children and their families with those achieved by similar non-Head Start children and families. Although definitive results could take years to obtain, questions about Head Start’s impact will remain unanswered unless these plans are expanded.

Messrs. Chairmen, this concludes my statement. I would be happy to answer any questions you or members of the Subcommittees may have.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.