Healthy Start: Preliminary Results From National Evaluation Are Not Conclusive
Released: 1998-07-15T00:00:00

Letter

Background

In 1991, HRSA announced that it would fund 10 Healthy Start sites and issued guidance on how communities could obtain a grant. By July 1991, HRSA had received 40 applications, and in September of that year, it began funding 15 communities for a 5-year demonstration project. In 1996, funding for these communities was extended for a sixth year. In 1994, HRSA began funding seven new communities—called special projects—and funding for these was also extended in 1996 for an additional year. Forty-one additional communities have been awarded grants since 1997, and these now share funding with the 15 original sites and 5 of the special projects judged by HRSA to have been successful.

To be eligible for the original grants, a community had to have an average annual infant mortality rate of at least 1.5 times the national average between 1984 and 1988—that is, 15.7 deaths per 1,000 live births—and at least 50 but no more than 200 infant deaths per year. Applicants had to be local or state health departments, other publicly supported provider organizations, tribal organizations, private nonprofit organizations, or consortia of these organizations. HRSA required only a few specific activities of all sites to provide grantees flexibility to make their projects relevant to local circumstances.

Healthy Start’s principal goal to reduce infant mortality has usually been stated as a 50-percent reduction in infant mortality, attributable to the program, over 5 years. Healthy Start also aims to achieve improvements in other outcomes—such as reductions in low birthweight, improved maternal health, and increased community awareness of threats to infant health—that are expected to help reduce infant mortality. In addition, Healthy Start was designed to demonstrate how a program based on innovation, community commitment and involvement, increased access to care, service integration, and personal responsibility could work in a variety of locations with high infant mortality.

From fiscal year 1991 (a planning year that preceded the 5-year demonstration), through fiscal year 1998, program funding for Healthy Start has totaled more than $600 million. Healthy Start’s fiscal year 1992 funding was less than half of what was initially proposed, and the number of grantees was greater. Instead of $171 million being spread over 10 sites, funding for the first year of the demonstration was $64 million spread over 15 sites.

In 1997, HRSA concluded the demonstration phase of Healthy Start and began the “replication phase” in 40 (now 41) new sites. In addition to providing Healthy Start services in their own communities, the established Healthy Start communities—the original 15 sites and 5 of the special projects—are mentoring several of the new sites. While the new sites receive, on average, somewhat less funding than the established sites, funding is shared among all sites.

The National Evaluation Consists of Both Impact and Process Evaluations

In September 1993, HRSA contracted with MPR to conduct the national evaluation of the Healthy Start program. This is currently funded with about $4.8 million, paid from the 1-percent set-aside for evaluation of health programs. The original contract called for MPR to evaluate the first 4 years of the 5-year demonstration program and contained an option for HRSA to request evaluation of the fifth year. In 1995, HRSA exercised that option, and the contract now requires that the evaluation cover all 5 years of the originally planned demonstration. Although the demonstration was extended for a year, HRSA currently has no plans to request that MPR evaluate the sixth and final year of the demonstration phase.

The national evaluation, focused only on the original 15 sites, is designed to determine whether Healthy Start changed the rate of infant mortality and related outcomes, what factors contributed to any effects the program may have had, and how successful approaches to lessening infant mortality can be replicated in other communities. Although each Healthy Start community is unique and the details of the delivery of any one service may differ across communities, many of the services are common to all sites: outreach and case management; support services, such as transportation and nutrition education; enhancements to clinical services; and public information campaigns.

The national evaluation has two major components: an impact evaluation and a process evaluation. The impact evaluation is used to determine whether the infant mortality rates in Healthy Start communities have declined and whether related outcomes have improved. The process evaluation describes how the program actually operates. In its final evaluation report, MPR intends to synthesize these two components, linking outcomes with processes to determine why Healthy Start has or has not succeeded in communities and which strategies are likely to be successful elsewhere.

MPR’s Preliminary Evaluation Results Were Not Conclusive

In the fall of 1997, MPR reported some preliminary evaluation results, including a draft interim report on its impact evaluation, which led to press accounts suggesting a variety of interpretations about the success of the Healthy Start program. We believe these preliminary evaluation results were not conclusive. Although the impact evaluation suggested that Healthy Start has not reduced infant mortality, such conclusions about the program would be premature because the impact evaluation does not include data from all the program sites or data from all the years of the program. Moreover, the process evaluation indicates that program implementation in many communities was slow and, therefore, that the impact data analysis may not be representative of a mature Healthy Start program.

Preliminary Analysis of Program Impact May Not Be Meaningful Because of the Limited Data Available

The national evaluation’s analysis of Healthy Start’s effect on infant mortality and related outcomes is preliminary—MPR characterized its October 1997 report as a draft. Because of problems obtaining data from some of the states’ departments of health, only 9 of the 15 program sites to be evaluated were represented in the analysis. In addition, the analysis is related to only the first 3 of the 6 years of program operation. Moreover, for illustrative purposes, MPR has limited its principal impact analysis to data from only the last of those 3 years, 1994. However, if, as HRSA believes, fiscal year 1995 was the first fully operational year, even 1994 data may not reflect the communities’ mature programs.

To determine program impact, MPR is conducting two types of analysis: availability and participation. The availability analysis compares a Healthy Start community and two similar communities without Healthy Start to determine if the presence of the program in a community has an effect on infant mortality and related outcomes. The participation analysis compares, within a Healthy Start community, mothers who were clients of the program and mothers who were not. Both analyses can be used to study infant mortality; however, the availability analysis directly addresses the issue of reducing infant mortality in entire communities, while the participation analysis is restricted to outcomes for program participants.

The national evaluation’s availability analysis found that for 1994, the overall infant mortality rate in Healthy Start communities was about the same as that in comparison communities. Applied to the individual sites, the analysis found that of the nine Healthy Start communities analyzed, only one experienced a significant reduction in infant mortality relative to its comparison sites. MPR similarly found that the neonatal and postneonatal mortality rates—two components of infant mortality—were not significantly reduced in the Healthy Start communities relative to the comparison sites. In its analysis of birth outcomes considered to be risk factors for infant mortality at eight of the Healthy Start communities, MPR found that in 1994, the low birthweight rate was reduced in only one community, the preterm birth rate was reduced in two other communities, and the rate at which women received adequate or better prenatal care was improved in five communities. None of the analyses of data pooled from all sites yielded significant differences between sites with and without Healthy Start.

The national evaluation’s participation analysis of Healthy Start’s effect on infant mortality has not been completed because of problems of data availability. The participation analysis of related outcomes, like the availability analysis, yielded little evidence of program effect in the eight communities analyzed. Participation in Healthy Start was not associated with reductions in low or very low birthweight rates or preterm birth rates. In postpartum interviews with participants and nonparticipants, conducted in 1996, after all sites became fully operational, MPR found that participants were more likely to rate their prenatal care experience more highly and to be using birth control. However, no significant differences between participants and nonparticipants were reported for the receipt of services or health behaviors during pregnancy.

Process Evaluation Indicates That Program Implementation Was More Gradual Than Anticipated

The national evaluation’s process evaluation is intended to provide a detailed picture of what happened over time when Healthy Start was implemented at the various sites and assess its success in meeting its process objectives, such as hiring and retaining staff and putting the planned program in place. The evaluation, which, according to HRSA’s project officer for the evaluation, is to result in a series of reports, indicates thus far that the Healthy Start program was implemented largely as originally envisioned but more gradually than expected.

MPR’s implementation report, a major portion of the process evaluation, provides an overview of program implementation in 14 of the 15 original sites and draws conclusions about these projects. The report includes detailed information on the development of the projects, the barriers to successful implementation, and the gaps between what was planned and what resulted. In addition, the report presents perceptions of variations across projects with respect to a variety of criteria, such as staff stability and consumer participation in the process. It also contains timelines indicating for each site when specific program components became operative. These timelines demonstrate, for example, that only 4 of the 14 sites had all their planned services operational by October 1994. The implementation report concludes with lessons learned, which are organized into four categories: community context, organization and administration, community involvement, and service delivery.

MPR has also completed a report on the infant mortality review process at the various Healthy Start sites. It indicates that, in general, the review programs are operational but with varying degrees of success in identifying the factors leading to infant mortality in their communities.

Two detailed reports on specific interventions are available only in draft form. One describes, in greater detail than the implementation report, program participants, including comparisons of participants and nonparticipants with respect to the use of health and social services, satisfaction with services, and health-related behaviors, such as birth control and breast feeding. The other describes how outreach and case management were delivered, with consideration given to both similarities and differences across sites.

The Degree to Which Specific Program Characteristics Contribute to Reductions in Infant Mortality Is Not Yet Known

Because the impact and process evaluations are not finished, their synthesis has not yet begun. MPR expects to be able to draw conclusions about the program characteristics that are most effective in improving maternal and child health outcomes and the circumstances under which they are most likely to succeed when it integrates the impact and process evaluations. The synthesis of the impact and process components of the evaluation to be presented in the final report will be based on impact data from years one through four of the demonstration. This synthesis may have to be revised when more impact data are available.

MPR’s Final Report Will Be Based on Fewer Years of Impact Data and Program Operation Than Originally Planned

The final report as currently planned will not be the final evaluation of Healthy Start. The final report will contain an analysis of outcomes through 1995 and a synthesis of this with the findings of the process evaluation reports. Thus, it will assess the program’s impact on infant mortality through the fourth year of the demonstration, not the fifth year as planned. Further, these data will reflect the impact of only 1 or 2 years during which the program was fully operational. An addendum to the report, planned to follow a year later, will contain an updated analysis of outcomes through 1996. The addendum will assess impact on infant mortality through the fifth year of the demonstration and thus will reflect the impact of only 2 or 3 years during which the program was fully operational. However, by evaluating the sixth year of the demonstration, it would be possible to obtain an analysis of 3 or 4 years of impact data from the mature program.

Evaluating the sixth year of the demonstration would likely enhance the value of the investment in MPR’s evaluation for several reasons. First, including data from the sixth year of the demonstration would allow evaluation of the years in which all 15 sites have been fully operational. Second, additional data would represent the effects of a more mature and potentially more effective program, which would likely provide more definitive answers about Healthy Start’s success. Third, having more years of data would increase the likelihood of detecting small but real effects of the program. Further, it is possible that data from the more mature years of the program will reflect program impact on the wider Healthy Start community, not just direct participants in program services and activities. In addition to hoped-for effects on the pregnant clients of Healthy Start, there may be effects of program services and education on those same women at other times, such as before or early in their next pregnancy; indirect effects on their social network, such as their male partners, friends, and sisters; and indirect effects on the community in general.

HRSA’s project officer for the national evaluation notes that the cost associated with analyzing results for an additional year would be about $100,000; this would be inexpensive relative to the total national evaluation cost of about $5 million. Since states collect vital records on births and deaths routinely, funds would be needed only to obtain, analyze, and report on the data and to revise the synthesis of these data with the process evaluation.

Conclusions

Since the national evaluation of the Healthy Start program has yet to be completed, preliminary results should not be used to conclude that the program has or has not achieved its goals. HRSA and MPR plan for the “final” report of the national evaluation to include an extensive description of the program, indicate whether it has reduced infant mortality rates at Healthy Start sites, and provide an analysis of how program characteristics have influenced outcomes. However, the final report will analyze infant mortality data from only 4 years of the demonstration. Primarily because implementation of the projects was slower than anticipated, data from the first 4 years of the demonstration may be insufficient for judging the success of Healthy Start in lowering infant mortality. Thus, even the final report will be inconclusive. Analysis of the fifth year of the demonstration, as planned, will help strengthen the evaluation, but this analysis will not reflect as many years of mature program operation as possible. Thus, at a relatively modest cost, MPR’s evaluation would be further strengthened by including data from the sixth and final year of the demonstration.

Recommendation

To increase the value of the investment in the national evaluation of Healthy Start, we recommend that HRSA contract with MPR to expand the evaluation to include impact data from the sixth year.

Agency Comments

In commenting on a draft of this report, HRSA agreed with our findings and indicated that it intends to add funds to the MPR contract to include impact data from the sixth year of the demonstration. HRSA and MPR provided a number of technical comments that we incorporated as appropriate.

As arranged with your office, unless you announce its contents earlier, we plan no further distribution of this report until 30 days after the date of this letter. We will then send copies to the Secretary of Health and Human Services, to the Administrator of HRSA, and to others who are interested. We will also make copies available to others on request.

Please contact me at (202) 512-7119 if you or your staff have any questions. You may also contact Michele Orza, Assistant Director, at 512-9228, or Donald Keller, Senior Evaluator, at 512-2932.

MPR’s Methodology and Its Limitations

The Availability Analysis

The availability analysis of infant mortality and related birth outcomes is part of Mathematica Policy Research’s (MPR) attempt to determine if Healthy Start has, as intended, reduced infant mortality at program sites, looking at the vital statistics for entire program and comparison areas where, respectively, the program is or is not available. It does this without concern about the participation in the program of specific persons. It attempts to separate any change that may occur in outcomes at program sites that is attributable to Healthy Start from change in outcomes at those sites that would have occurred without the program—for example, changes stemming from the national trends not related to health and social interventions, such as the persisting decline in infant mortality experienced almost everywhere in the United States. It does this for each outcome of interest, obtained from the state health department’s vital records of linked births and deaths for the sites of interest, by (1) comparing each program site with two comparison sites without Healthy Start, selected (matched) for similarity to the program site with respect to race and ethnicity, infant mortality rate, and trend in infant mortality over the pre-Healthy Start period and (2) statistically adjusting the data for differences between program and comparison site mothers on variables, also obtained from vital records, believed to affect the outcome. To the extent that, as a result of site selection and statistical adjustment, the program and comparison sites do not differ in expected infant mortality rate, then the comparison of the program and comparison site adjusted outcomes should be a valid indication of the effectiveness of the program.

MPR’s approach involves accepted statistical methods with known limitations. One limitation stems from the possibility that program and comparison site mothers will systematically differ in ways, such as poverty level, that affect outcomes but are not taken into account in the selection of comparison sites and are not available for use in statistically adjusting the data. Such a difference could bias the estimation of the difference between program and comparison sites in outcomes. Nevertheless, MPR appears to have taken reasonable precautions to minimize the likelihood of bias. MPR did this, for example, by using two comparison sites, not just one, for each program site and by avoiding the selection of comparison areas known to have interventions similar to Healthy Start. Further, MPR shared information on its site selections with each of the 15 sites and sought their comments and agreement on the choices. reductions when such differences, in fact, exist. Roughly speaking, power depends on the number of observations—in this case, live births—in the analysis, and it is therefore often a problem when that number is not controlled by the design of the study. In the case of Healthy Start, this implies that the ability to detect a real difference between program and comparison communities depends upon whether the comparison involves, for example, communities relatively small in population, large communities, or all communities pooled. With respect to infant mortality, MPR reports that, using all data from 1984 to 1994, the minimal detectable difference in infant mortality is computed to be 31 percent, 7 percent, and 6 percent for small, large, and all communities, respectively. This means that if there are real differences between Healthy Start and comparison communities, but these differences are smaller than we have the power to detect (that is, smaller than the percentages listed above), then we will mistakenly conclude that the program has no effect on infant mortality. Since power depends on the number of observations, increasing the number of years of data included in the analysis will increase the ability to detect any difference that may exist.

A third potential limitation concerns the number of statistical tests performed in the complete impact analysis. If 15 sites and seven different birth-related outcome variables are considered, then at least 105 statistical tests will be done. With as large a number of tests as this being done, it is likely that a portion of them will yield statistically significant results by chance alone. This means that even if Healthy Start has no effect on infant mortality, we will mistakenly conclude that it does have one in a certain percentage of the statistical tests conducted. There are statistical methods of dealing with this problem. If they are not employed in the final analysis, then differences that are statistically significant by chance alone will occur more often than is considered acceptable by statistical convention.

The Participation Analysis

MPR’s participation analysis of birth outcomes is part of MPR’s attempt to determine the effect of participation in Healthy Start within program sites. It compares 1995 birth outcomes between participants and nonparticipants in each project area. Participants in the program’s prenatal activities are identified from program files, the Minimum Data Set required of all Healthy Start sites, and their birth certificates are flagged. Their birth outcomes are then compared with those of nonparticipants or participants with limited prenatal program involvement.

This kind of analysis is limited by possible preexisting differences between participants and nonparticipants and by the very definition of participant. Since participation in Healthy Start is voluntary, it is possible that participants systematically differ from nonparticipants. Program providers may, for example, tend to attract persons who are especially knowledgeable about services or already well connected to the health care system. Under these circumstances, program participants would be expected to have better outcomes even without Healthy Start. Alternatively, participants might be especially needy and at high risk for poor outcomes, in which case they would be expected to have relatively poor outcomes. MPR deals with this by statistically adjusting the data on the basis of information that may reflect these preexisting differences, background information from birth certificates, and any other available sources. Although it is difficult to be certain that outcomes have been adjusted for all possible systematic differences between the groups being compared, MPR has stated that participants tend to be at high risk for poor birth outcomes, thereby making any potential finding of better outcomes for them than for nonparticipants more convincing of the program’s value.

The question of who is a participant must be answered in order to conduct the participation analysis. It turns out not to be easily answered because (1) the Minimum Data Sets of many sites have been slow in developing into accurate record systems, (2) it is not always clear whether a participant’s involvement has been intense enough to classify that person as a participant, and (3) when supplementary information has been sought from new mothers about their involvement in the program it is not always clear what criteria they use for judging whether or not to claim to be participants. Moreover, these problems vary somewhat from site to site, making it difficult to be sure that all participation analyses are comparable. meaningful to the extent that the preexisting differences between participants and nonparticipants can be taken into account.

The Process Evaluation

MPR’s process evaluation is an effort to use both qualitative and quantitative information to assess the degree to which Healthy Start has implemented its program as conceived of, how it serves its target population, and how these processes developed over time. This description of Healthy Start can be considered the documentation of the program’s second goal—to demonstrate what happens when this kind of effort is mounted. Its methods are varied, including making site visits with and telephone calls to project staff, examining the client records of the Minimum Data Set, the postpartum survey of participants and nonparticipants, running focus groups with service providers and with members of the communities, as well as using documents and vital records. Many aspects of the process evaluation are not complete and will not therefore be described further, but one major document—the implementation report—is complete.

The implementation report is based mainly on site visits, expenditure reports from each project, and the client records of the Minimum Data Set. Further, two independent teams of site visitors rated certain dimensions of administrative success using a modified Delphi consensus reaching process. Although they may not be avoidable, the limitations of this report are those common to most process evaluations that are heavily qualitative. The methods employed provide a wealth of information suitable to inform those who would develop similar programs about what to expect if different options of organization, administration, and mode of service delivery are attempted. However, the essential subjectivity of interview methods makes it difficult to know how closely other evaluators would agree with the conclusions drawn.

The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 37050 Washington, DC 20013 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (202) 512-6061, or TDD (202) 512-2537.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.