Early Childhood Programs: The Use of Impact Evaluations to Assess Program Effects
Released: 2001-04-16T00:00:00

Letter

Background

The use of impact evaluations to assess the effect of a program or other intervention is an accepted practice in areas as diverse as medicine, economics, and social services. Some frequently cited examples of impact evaluations involve experiments designed to test the efficacy of drugs by randomly assigning participants to a group that will either receive or not receive the experimental drug. Impact evaluations are also used in other areas to assess program effectiveness. One such study is currently under way for the Job Corps. In addition, several state-level impact evaluations are being used to study welfare-to-work programs. The Congress has mandated studies to ensure that agencies conduct the kinds of evaluations it deems necessary. In both authorizing and reauthorizing legislation, the Congress has required evaluations of a program or demonstration program.

Impact evaluations have been used to evaluate and develop programs in the early childhood education and care arena. For example, HHS sponsored an impact evaluation of a demonstration project called the Comprehensive Child Development program to test the effectiveness of the program for low-income families and their young children. HHS incorporated many of the lessons learned from this study in the Early Head Start program.

The 11 federal programs that serve 25,000 or more young children in some capacity are administered by either Education or HHS (table 1). The programs differ in a number of ways, including their goals and the portion of funds they devote to young children. For example, Head Start’s goal is school readiness and most of its budget is spent on children younger than age 5. All Head Start grantees provide services that are designed to enhance school readiness. In contrast, SSBG, which devotes only 11 percent of its funds for early childhood education and care, is a flexible source of funds that allows states wide discretion to fund a variety of social services, one of which can be child care. Its goal is primarily to help families achieve self-sufficiency. The other programs we reviewed fund early childhood education and care in various ways but often have primary goals that are only indirectly related to children or school readiness.

The passage of the Government Performance and Results Act (GPRA) of 1993 influenced how agencies assess the effectiveness of their programs. GPRA shifted the focus of accountability for federal programs from inputs, such as staffing and activity levels, to outcomes. GPRA requires that each federal agency develop a multiyear strategic plan identifying the agency’s mission and long-term goals. To develop the plans, each agency must connect its long-term strategic goals with daily activities. While GPRA requires agencies to measure the outcomes of their programs, it does not require agencies to conduct formal program evaluations such as impact evaluations. However, GPRA requires agencies to summarize the findings of program evaluations in their annual performance reports.

Impact Evaluations Are the Most Definitive Method for Determining a Program’s Effect on Its Participants

Many researchers consider impact evaluations to be the best method for determining a program’s effect on its participants because they isolate a program’s contribution from the effects of other factors that could have influenced participant outcomes. Impact evaluations can be designed in several ways, but most fall into two categories: experimental and quasiexperimental designs.

Impact evaluations are used to isolate the influences of a program being studied from other influences in an individual’s life. In the case of a child, many influences affect his or her development (fig. 1). As a child grows, he or she acquires new knowledge and new skills. Nutrition, health, family, and community as well as education and care play roles in his or her learning. In light of all these influences, it becomes difficult to distinguish between the effects of the program and the other factors that influence a child’s learning.

To isolate the program’s influences on a child, an impact evaluation studies two groups of children: those receiving program services and a similar group not receiving program services. Researchers compare the relevant outcomes, such as reading ability, of these two groups of children to determine the program’s effect.

Most impact evaluations are one of two types—experimental or quasiexperimental—although they can be designed in several ways. The two designs differ primarily in the way that the comparison groups are developed. In an experimental design, the comparison group is referred to as the “control” group. It is composed of individuals randomly selected from possible program participants. In a quasiexperimental impact evaluation, the comparison group is composed of individuals who share characteristics with program participants, but who may or may not have ever sought program services.

Experimental Design

The more rigorous of the two types of impact evaluations is the experimental design (fig. 2). It randomly assigns individuals to either a group that will receive program services or a group that will not receive program services. As applicants apply for program services, each has an equal chance of being placed in a group that will receive program services or a group that will not receive program services. Although individual applicants are different, random assignment distributes those differences equally between the two groups, resulting in two groups that are presumed to have no systematic differences in their characteristics. Although individuals in the control group do not receive services from the program under study, they may receive similar services through other programs.

At various points during an experimental impact study, researchers measure the relevant outcomes of the members of both groups. The group that is not receiving services demonstrates what would have happened without the program. Thus, when the progress of the two groups is compared, any differences between them can be attributed to the influences of the program. This way, the use of a control group serves to rule out alternative explanations for differences in the outcomes of program participants and nonparticipants. However, because control group members may receive services similar to those provided through the program under study, the differences in outcomes between the two groups may be less than they would have been had the control group not received those similar services.

The Denial of Services to Certain Eligible Study Participants Heightens the Complexity and Difficulty of Experimental Impact Evaluations

The denial of the program services under study to eligible individuals in control groups—a necessary aspect of experimental impact evaluations— raises a number of issues. For example, experimental designs cannot be used when a program must serve all eligible individuals. The denial of services can also raise ethical issues, the result of which can be reluctance on the part of program staff and officials to participate in impact studies. These issues, along with other challenges characteristic of complex evaluations, can make experimental impact evaluations especially difficult, time-consuming, and expensive to perform.

Ethical issues arise in the selection of an experimental impact evaluation control group. As we stated earlier, a control group consists of participants who are denied services. Several factors may complicate the creation of the control group, and in fact denying services for the purpose of creating a control group may not be possible. For example, a program may be required to serve all eligible individuals, as is the case with Special Education programs. Even when forming a control group does not raise these kinds of issues, program staff and officials may be ethically opposed to denying services to individuals, especially for evaluations requiring several years to complete. To overcome such opposition, an additional investment of time may be necessary to gain the trust of program staff. The organization conducting the evaluation may need to provide training to staff so that they understand the need for random assignment and can collect data if necessary.

In addition to the possible opposition of staff members, people seeking program services can be reluctant to agree to forego program participation. Researchers must gain the acceptance and informed consent of both participants and control group members (or, in the case of evaluations involving minors, their guardians). In some instances, control group members or their families may require monetary or other incentives to participate. Over the course of the evaluation, incentives may need to be sustained or increased.

The ethical issues associated with experimental designs, along with other challenges characteristic of complex evaluations, can make them difficult to complete and add to the time and expense they require. Experimental impact evaluations are often complex, multiyear studies awarded through competitive bids to experienced research firms. Considerable attention must be given to both study planning and execution. For example, researchers often prepare detailed plans describing the study, its methodologies, and issues that can affect the study. In many cases, aspects of the study must be pilot tested before the larger study can be undertaken. Then, once evaluations are under way, individuals may need to be tracked over several years. For example, an evaluation that begins well before a child starts school—for example, at age 3 years—may require 3 years or more of data collection to track the child through the beginning of his or her school years. In addition, special efforts must be made to limit the number of study participants who leave the study, especially from within the control group.

Quasiexperimental Design

The other, less rigorous type of impact evaluation is quasiexperimental design. When randomly assigning individuals to a control group is not a feasible option, quasiexperimental impact evaluations can be used to compare the performance of program participants on various measures to individuals not in the program. In a quasiexperimental design, methods other than random assignment are used to create a comparison group. A comparison group can be developed in a variety of ways. One way is to use a set of individuals who have similar characteristics to the group receiving the program services under study. For example, the group might live in the same neighborhood as the group receiving services and have family and income characteristics similar to those receiving serivces.

The extent to which quasiexperimental comparison groups are actually similar to program participants is not entirely known. For this reason, quasiexperimental evaluations cannot rule out all of the factors that influence participants’ outcomes. Other factors could explain differences between the two groups, making it difficult to conclude, with certainty, the effect of the program being examined. When well planned and executed, however, such designs can provide some indication of program impact.

Impact Evalutions Are Being Used to Assess the Effectiveness of Early Education Programs

The two programs most focused on early childhood education—Head Start and Even Start—are currently being studied using impact evaluations with an experimental design. HHS is conducting impact studies of Head Start and Early Head Start, and Education is using an impact evaluation to study Even Start. Both of these programs are intended to improve children’s school readiness and educational and developmental outcomes, for example by encouraging enhanced literacy. Impact evaluations are not currently being used to evaluate the early education and care services provided by the other nine programs we reviewed whose goals are not as directly focused on early childhood education as Even Start and Head Start. For many of these programs, providing early childhood education and care is one of many allowable services, not the primary program focus.

The Head Start Study

The Head Start evaluation is a $28.3-million, nationally representative, longitudinal impact evaluation. The contract for the evaluation was awarded in fall 2000 to Westat Inc., a research firm, in collaboration with the Urban Institute, the American Institute for Research, and Decision Information Resources. The evaluation is still in its early stages and data collection has not yet commenced. Before data collection begins in fall 2002, a pilot study will be conducted to test various procedures and measures and to investigate the variations that exist across different communities with respect to Head Start programs and the availability of other care options for low-income children. The pilot study will be conducted in spring 2001.

In fall 2002, researchers will begin to collect data on about 5,000 to 6,000 3- and 4-year-olds from 75 programs and communities across the country; data collection will continue through the spring of the children’s first grade of elementary school. It is estimated that this evaluation will take more than 6 years to complete. The Congress mandated this evaluation and specified that it be an impact evaluation by stating that children should be randomly assigned to either a group that will receive Head Start services or one that will not receive Head Start services.

The study has two primary goals. The first is to determine the national impact of Head Start on children’s school readiness by comparing children in Head Start to children not in Head Start. Data in areas related to school readiness, such as cognition, physical well-being, social and emotional development, and language usage, will be collected from both groups of children. In addition, information on parenting practices, family structure, demographics, and socioeconomics will be collected. The second goal is to determine under which conditions and for which children Head Start works best.

In crafting the study proposal, Westat noted the difficulties inherent in denying services to eligible children in order to create the control group and suggested ways to soften the ethical objections to random assignment. For example, the proposal allows Head Start staff to provide information to control group families on alternative services in the community, and exempts a small number of children with significant disabilities from the random assignment. The plan also acknowledges the need to make parents comfortable with the process and, in some instances, provides financial incentives for participation. At the same time, Westat noted the need to maintain the integrity of the random assignment over a number of years. To this end, the plan calls for providing information to Head Start staff about the need for random assignment. In addition, Head Start grantees will need to agree to exclude children assigned to the control group from enrolling at other Head Start centers within their jurisdictional control during the study. The final report is due in December 2006.

The issue of Head Start’s effectiveness was highlighted in 1997 when we reported that the body of research on the Head Start program was insufficient to draw conclusions about its national impact. The report recommended that a study or studies be undertaken to determine the impact of the program nationally. The following year, we reiterated these points in another report and added that the federal government’s significant financial investment in Head Start, including plans to increase the number of children served, warranted definitive evaluation studies, even though they may be costly.

Based on these recommendations and the testimony of research methodologists and early childhood experts, the Congress mandated that HHS fund and conduct an evaluation to determine, on a national level, the impact of Head Start on the children it serves. The Congress also called for an expert panel to develop recommendations regarding the study design to determine if, overall, Head Start programs’ impacts are consistent with the primary goal of school readiness. The panel included 30 experts in the areas of program evaluation and research, education, early childhood care and care policy, and economics. The panel outlined a framework for the impact study and deliberated extensively on the feasibility, ethics, and credibility of random assignment. It concluded that random assignment represented the best approach for answering the question of Head Start’s impact.

The Early Head Start Study

The Early Head Start program is part of Head Start and serves pregnant women, infants, and toddlers. The Early Head Start Study, which was also congressionally mandated, began in 1995 shortly after the program was established. It will cost about $21 million and take 6 years to complete. This longitudinal study of about 3,000 families and their children will assess children 14, 24, and 36 months after birth. The Early Head Start study includes an impact component as well as an implementation study that we discuss later in this report. Mathematica Policy Research, Inc. (a research organization), and researchers at Columbia University are conducting the study in collaboration with 15 local research teams.

The impact portion of the study will examine Early Head Start programs at 17 local sites that include a mix of urban and rural sites as well as sites serving individuals with different racial and ethnic backgrounds. These 17 sites were selected to reflect the characteristics of all programs funded in the early years of the program.

As the Early Head Start grantees recruited families, the families were randomly assigned to either a group that will receive Early Head Start program services or a control group that will not receive program services. (Control group families could receive other services in the community.) The final report, which will follow children up to their third birthday, is due in 2002. HHS has initiated another study which will follow these children through the spring before their entry into kindergarten to evaluate the program’s impact on school readiness, among other things. This follow-up evaluation was not congressionally mandated.

The Even Start Study

The estimated $3.6 million Even Start study will also examine several local projects using an experimental design. The study is supported through program funds that the Congress made available for evaluation. The mandate for the Even Start study was included in the program’s original legislation and, although that legislation has been amended several times, the requirement for an independent evaluation to determine the performance and effectiveness of Even Start has remained unchanged.

Funded in 1997, the study provides for an experimental design to test the effectiveness of 18 projects serving 400 Even Start families and 200 control group families. The contract, awarded to Abt Associates, Inc. (a research organization), will, among other things, measure children’s school readiness. The study will take 6 years to complete and the final report is expected in summer 2003.

The current study is actually the second Even Start impact study conducted using an experimental design. The first evaluation included an impact evaluation that examined Even Start programs operated by five grantees. The small number of sites examined by the study and the lack of information on control group experiences did not permit conclusions about program effectiveness.

Impact evaluations are not currently being used to evaluate the early education and care services provided by the other nine programs we reviewed. Their goals are not as focused on early childhood education as those of Even Start and Head Start. For many of these programs, providing early childhood education and care is one of many allowable services, rather than the primary program focus (fig. 3). Researchers have used a variety of study designs, other than impact evaluations, to understand more about these types of programs with respect to early childhood education and care services.

Different Types of Studies Inform HHS and Education About Administered Programs

HHS and Education promote and sponsor many types of studies, other than impact evaluations, that can provide a wide variety of data valuable to program managers and policymakers. Different study designs are used depending on the questions to be answered, the nature of the program being studied, and the type of information needed. Often, to answer varied, complex, and interrelated questions, policymakers may need to use several different designs to assess a single program. Policymakers’ desire to know a program’s effect does not preclude their need to know about its implementation and other details that can be revealed through different types of studies.

Implementation Evaluation

An implementation evaluation, also called a process evaluation, can assess the extent to which a program is being administered as the Congress anticipated. An implementation evaluation typically assesses whether program activities conform to statutory and regulatory requirements, program design, and professional standards or customer expectations. Generally, an implementation evaluation necessitates collecting information for comparison to legislative objectives. For example, if legislation requires a program to serve a particular population with certain characteristics, researchers conducting an implementation evaluation would examine the demographics of participants to determine if the program is indeed serving this population.

Implementation evaluations provide information on many aspects of a program. For example, the first Even Start study was largely an implementation study, and its findings were used to improve program design and shape future research. The study was broad in scope, but examined the characteristics of Even Start participants, projects, and services to assess how closely they resembled what had been envisioned for the program. The study served as a catalyst for changes in the program’s legislation, including a shift in focus on those most in need. In addition, as a result of the study’s findings, teen parents are now eligible for program services.

The Early Head Start study under way also includes an implementation component. The implementation study gathered information on participants and services to determine if the program was implemented as intended. Information from the implementation study will be integrated with the results of the impact analysis.

Outcome Evaluation

An outcome evaluation assesses how closely a program’s achievements are aligned with program goals. This type of evaluation answers the question, “Are participants achieving intended outcomes?”

Outcome evaluations can determine how well children are doing at a particular time. However, without the use of an experimental design, researchers cannot attribute this performance to any one influence. Unlike an impact evaluation, outcome evaluations are unable to isolate the influences of the program under study.

HHS and Education use outcome evaluations to find out how well children in agency-sponsored programs perform basic skills. For example, Education is sponsoring two national studies with outcome components for its special education programs. One study is an early intervention study. This study will follow a nationally representative sample of children from birth to age 3, monitoring them during and after the program, to describe their progress in such areas as eating and dressing. The other study follows 3- to 5-year-olds through early elementary school and will also address questions about children’s outcomes. HHS funded the Family and Child Experiences Survey (FACES) in 1996 to collect outcome data from a nationally representative sample of Head Start programs, children, and families. FACES collects a range of data that includes cognitive, social, emotional, and physical development of Head Start children; the well-being and accomplishments of Head Start families; and the quality of Head Start classrooms.

Research Studies

Research studies are used to collect information to improve an agency’s understanding about a particular area, such as early childhood education, or the issues surrounding a particular area.

Often, research studies are used to test ideas that have arisen in an area but are not yet proven. For example, HHS believes that increasing the knowledge of how child care systems work is a critical step toward improving the availability and quality of child care services. To explore whether this is the case, HHS is using research studies to understand the complex nature of child care markets at the state, local, and national levels. Thus, some of the questions framing HHS’ child care research agenda include: What does child care look like today? How do child care variations influence child development and well-being? How do child care variations affect family economic self-sufficiency and well-being?

Agency Comments

We provided a draft of this report to the Departments of Education and Health and Human Services for comment, and they did not provide comments. However, Education’s Office of Elementary and Secondary Education and Office of Special Education and Rehabilitative Services and HHS’ Administration for Children and Families provided technical comments, which we incorporated where appropriate.

We are sending copies of this report to the Honorable Richard Durbin, Ranking Member of the Subcommittee on Oversight of Government Management, Restructuring and the District of Columbia, Committee on Governmental Affairs; the Honorable Roderick R. Paige, Secretary of Education; the Honorable Tommy G. Thompson, Secretary of Health and Human Services; relevant congressional committees, and other interested parties. We will also make copies available to others on request.

Please contact me on (202) 512-7215 if you or your staff have any questions about this report. Other GAO contacts and staff acknowledgments are listed in appendix I.

Appendix I: GAO Contacts and Staff Acknowledgments

GAO Contacts

Staff Acknowledgments

In addition to those named above, John Smale and Corinna Nicolaou made key contributions to this report.