CDC's National Immunization Survey: Methodological Problems Limit Survey's Utility
Released: 1996-09-19T00:00:00

Letter

Background

CDC’s Immunization Program

The CDC’s National Immunization Program provides grants to states and 28 urban IAP areas for the purpose of controlling vaccine-preventable diseases. The Congress made available at least $142 million for these grants in fiscal 1995. The portion of these funds received by a particular grantee is based largely upon the amount received the previous year. In addition to these funds, consistent with statements of the Senate Appropriations Committee, CDC has awarded annual incentive grants to states since fiscal 1994 to improve the immunization levels of 2-year-olds.For awards in fiscal 1994, CDC allocated incentive grants based on state-supplied estimates of the percentage of fully immunized 2-year-olds. To establish a common basis for awarding subsequent grants and to monitor progress toward early childhood immunization objectives, the CDC designed, and starting in fiscal 1994 began to conduct, the National Immunization Survey. In fiscal 1996, $33 million was allocated for such incentive grants.

States’ Previous Methods for Measuring Immunization Among Preschoolers

With the advent of the NIS, states had no further obligation to produce statewide coverage estimates and were able to use the grant funds formerly devoted to such measurement for other activities. However, most states’ former methods for estimating immunization coverage were much less expensive than the NIS, which CDC has heretofore financed at no cost to the states. Lately, CDC has made inquiries of state health officers regarding their willingness to devote certain percentages of grant funds to support the NIS (see p. 9).

To meet CDC’s former requirement for measuring preschoolers’ immunization coverage, all states used either school retrospective surveys or other population-based methods to estimate immunization coverage. Most states estimated immunization coverage among preschoolers through reviewing the immunization records of children entering first grade or kindergarten to determine whether their immunizations were up-to-date when the children were younger, typically when they were 2 years old.

This method has both disadvantages and advantages. It produces estimates that are about 3 years old by the time the data are gathered, and immunizations may be selectively collected on school records relative to the minimum state requirements for school entry, which vary to some extent across states and may not include the newer vaccines. Because the retrospective method uses data that are already collected for the purpose of verifying immunization at school entry, it is fairly inexpensive and enables some states to develop estimates of immunization coverage at substate levels for the use of counties or state health districts. Records for those immunizations required for school entry should provide more accurate dates of immunization than can be obtained in interviews with parents, who frequently do not have ready access to immunization information.

Those states that did not use the retrospective method used others, such as birth certificate surveys and registry-based methods, that required more original data collection than the retrospective survey, but produced more current coverage estimates while providing the states with other benefits or additional information about their specific activities. In 1995, CDC dropped its requirement that grantees produce an independent assessment of preschool immunization coverage with the view that the estimates from its new NIS would supplant the data that had formerly been gathered by grantees.

Sources of Survey Error

In general, in assessing the quality of survey findings, analysts should consider a variety of types of error that may affect a survey result. These include errors that arise because (1) surveys only involve a sample of the population of interest, (2) some of the sampled individuals may not respond to the survey, and (3) some of the population of interest may not be covered by the group from which the sample was chosen. In addition, there are problems associated with interviewers, the respondent, or the questionnaire, such as unclear questions or respondents’ difficulty in recalling the answers. What is commonly quoted in the reporting of poll results as the “margin of error,” typically plus or minus 3 percent for a random sample of 1,500, represents only the error attributable to the first factor named above. Assessing the quality of survey results also requires considering the extent to which the other sources of error may have affected the accuracy of survey findings.

Scope and Methodology

To respond to your request, we met with officials of the National Immunization Program and the National Center for Health Statistics and with staff of the CDC contractor conducting the National Immunization Survey. We reviewed documents describing the structure, performance, and results of the survey. We also reviewed literature on telephone survey methodology and parental recall of children’s immunization status. The methodology report for the 1995 survey was not available as of June 18, 1996, when we conducted our exit conference with CDC, and thus, our review of survey methodology was limited to the procedures employed in the 1994 survey and reports of NIS findings issued through June 1996. We understand from NCHS officials that, since issuance of the 1994 methodology report, procedures for using provider data to adjust survey results have been documented and sensitivity analyses have been conducted to measure the impact of changes in various assumptions inherent in the adjustment of survey results.

To provide information on survey costs, we requested that agency officials provide data on total payments under the survey contract and estimates of the costs of related agency activities. We also reviewed the survey contract and trends in the costs billed under the contract. We did not independently verify the payments for the survey or the CDC cost estimates, though we did review the invoices from the survey contractor and assess the agency’s cost estimates for their consistency with the activities the agencies conducted. With this exception, our work was conducted in accordance with generally accepted government auditing standards between March 25 and June 18, 1996.

Finally, we surveyed state immunization program managers regarding how they had used the results of the NIS and their costs for previous survey approaches. In addition, CDC provided a list of six former CDC contractors, officials, and current grantees that they recommended we contact. We contacted some of these individuals and asked them to provide comments consistent with their familiarity with the survey’s cost and methodology.

Principal Findings

Survey Costs

The cost of the survey includes three major components—expenditures under the contract issued to conduct the survey and the costs of survey-related activities conducted by NCHS and NIP. Both NCHS and NIP were involved in managing the data collection contract and providing statistical analysis of survey data. In addition to these roles, NIP gathered and reviewed data from the survey respondents’ immunization providers. When problems with survey software created a need for a larger interviewing staff, some work was done by the Bureau of the Census, but costs for this work are included in estimates provided by NCHS. Table 1 shows the costs of the NIS contract, survey assistance provided by NCHS and the Bureau of the Census, and NIP’s survey-related activities. Only two quarters of data were collected in fiscal 1994; 1995 was the first fiscal year in which the survey operated in all four quarters. Extraordinary expenses were incurred in fiscal 1995 when the agency discovered it needed to reinterview 1994 survey participants in order to identify their immunization providers.

The contract to conduct the NIS provides the recipient with a fixed fee and all reasonable costs for conducting the survey. Expenditures under the survey contract have risen at twice the rate anticipated at its signing, reaching nearly the full face amount of the contract halfway through the 54-month performance period. Contractor and agency representatives attribute the higher rate of expenditures to difficulties arising from the need to replace survey management software; the higher-than-expected number of calls required to identify households in the sampling frame; and the addition of a study to check parents’ responses against provider records, which increased the complexity of estimating survey results because of the need to adjust them with provider-derived information. The number of calls required to identify eligible households will continue to be an important determinant of survey costs.

According to estimates from CDC and invoices from the contractor, costs for the NIS have been roughly $25 million through March 30, 1996, including the $13 million for fiscal 1995, the first complete year of data collection. Insofar as a number of extraordinary expenses were incurred in fiscal 1995, CDC officials anticipate that final survey costs will decrease in fiscal 1996 and future years. However, for fiscal 1997, the agency has requested $16 million for the survey and its administration, as it requested and received in fiscal years 1995 and 1996 based on expenditures in the early implementation stage of the survey. CDC officials indicated that the balance of funds received in 1995 for the NIS (about $3 million) was spent on other assessment activities, such as the NHIS and its provider record check study, the Clinic Assessment Software Application, and the provision of technical assistance to the states. However, we have not independently verified this information.

In its report accompanying the fiscal 1996 appropriations, the Senate Appropriations Committee noted its concern that the national findings of the NIS duplicate the findings from the NHIS and that the annual cost of the survey cannot be justified by its utility. The Committee noted particularly that the survey does not provide significant information on high-risk communities for targeting purposes, and in some respects, it duplicates surveys conducted by each state. In the justification for its fiscal 1997 budget request, CDC acknowledged these concerns and noted that it was holding ongoing discussions with, among others, the Association of State and Territorial Health Officials (ASTHO) and the Council of State and Territorial Epidemiologists in which “various options related to the NIS” were being considered.

For example, CDC explored with ASTHO the level of willingness among state health officers to finance the survey through state grant funds distributed by CDC rather than directly through CDC appropriations. However, ASTHO surveys of its members found that many of the larger states and urban areas were not prepared to devote 6-10 percent of their immunization infrastructure grants to support of the survey. This is consistent with the findings of our survey of state immunization program managers, which indicated that while the NIS findings were widely used to communicate with the news media and respond to legislative inquiries, they were not used by most states for targeting their activities or designing interventions.

Methodological Challenges

Survey Design Incorporates Inefficiencies

NIS surveyors identify households with children between 19 and 35 months old by dialing random telephone numbers and asking a short set of screening questions to assess the presence of children in the correct age range. Surveyors ask for the number of doses of various vaccines the child has received and a variety of demographic information. Even with sampling refinements implemented by the contractor, only a small proportion of randomly generated telephone numbers results in contacting a residence that includes children between 19 and 35 months old. CDC reported that roughly 1.2 million telephone numbers were called to complete 25,247 interviews during the first three quarters of data collection (47 numbers per respondent, with an average of 4-5 calls per number required to reach a respondent). Thus, roughly 200 calls are initiated per completed interview. In view of the size of this undertaking, there was some thought at the time the survey was planned of using it to gather additional health data, but these plans never came to fruition and the final survey addressed only immunization issues. “only when the first-phase element survey costs are smaller than those for the second phase by a large factor . . . the first-phase sample identifies the members of the rare population inexpensively, and the survey items are then collected from them in the second phase.”

For the NIS, the reverse is true. It appears that CDC is spending a large sum of money on the first phase of the survey, which provides low-quality immunization data but identifies the sample for the second phase, which provides high-quality immunization data from provider records, albeit for a smaller number of children. Although the provider-supplied data improves the accuracy of survey results, earlier recognition of the problems with relying solely on household data might have led to consideration of more efficient data collection methods.

As of June 1996, summary coverage estimates had been published for the first five quarters of NIS data collection (April 1994-June 1995). CDC shares the survey results with state programs shortly before their publication in Morbidity and Mortality Weekly Report. Thus, the survey findings are available to states and the general public about a year after data are collected.

Households Without a Telephone Are Excluded From the Survey

“The high nontelephone noncoverage rates in many of the IAP areas and the large differences between telephone and nontelephone children’s vaccination rates indicate that the potential for noncoverage bias is considerable in several IAP areas. Any candidate estimation technique for the NIS must recognize this potentially large bias, and attempt to adjust for differences between the telephone and nontelephone groups.”

Appendix III shows the estimated percentage of households with a 2-year-old child that lack a telephone in each of the IAP areas, and table 2 provides national data from the 1992 and 1993 National Health Interview Surveys detailing the difference in reported immunization rates between children in households with and without telephones.

Although only about 5 percent of all U.S. households lack a telephone, the absence of one is more than twice as common in households with children under 2 years old (11.7 percent). However, these national data mask the wide variation among IAP areas in the percentages of households with children under 2 lacking telephones, which ranges from 2 to 25 percent across the 50 states and 28 urban IAP areas. Exclusion of households without a telephone requires that the survey results be adjusted to account for the positive bias that may result. However, there is no consistent source of information on the immunization rates among children in households without telephones in each area where the NIS is conducted. Consequently, the adjustment for noncoverage of children without telephones is based on a complex procedure involving the application of a statistical model of the probability that a fully vaccinated child in a related national survey resides in a household with a telephone. It is not possible to know whether these adjustments are accurate in each of the states and urban areas covered by the NIS.

Response Rates Vary Widely

The response rate is the estimated proportion of the target group (in this case, households with telephones and age-eligible children) that actually provided data. This rate is important in evaluating survey findings because, to the extent that nonrespondents might have answered differently from those who completed the survey, a large nonresponse rate indicates that survey findings will incorporate bias and require adjustment. For example, CDC analyses of NIS respondents indicated that, as a group, they differed in some respects from census and vital statistics estimates for the population; they slightly overrepresented mothers with more than 12 years of education and in some areas were more likely to report household incomes exceeding $50,000 and less likely to report income below $10,000. Thus, answers from those types of respondents who tended to be underrepresented were weighed more heavily in adjusting survey results to arrive at final coverage estimates. Such adjustments will remove bias to the extent that immunization coverage is similar between respondents and demographically similar nonrespondents. However, there is no clear way to test this assumption in the various areas surveyed.

For the calendar year 1994 survey, contractors estimated that the overall response rate was 69.5 percent. Appendix III identifies the overall response rates reported for each surveyed area. Although households determined to be eligible through their completion of the screening questions had high rates of cooperation with the full interview, they represented a smaller portion of the potential households than would have been expected based on census data, indicating that some 17.3 percent of eligible households with telephones were never reached, refused cooperation during the screening phase, or inaccurately responded to the questions about age-eligible children. Although a response rate in this range is not atypical of telephone surveys, nonresponse rates tend to run higher for telephone interviewing than for personal visitation. Also, while overall response rates varied tremendously across states and urban areas, nonresponse to particular questions ranged as high as 26 percent. When combined, these factors sometimes reduce to below 50 percent the effective response rates for key questions (for example, how many times has your child received a polio vaccine?), raising concerns about the accuracy of resulting estimates.

Household Respondents Have Difficulty Providing Accurate Data

The potential to use household surveys for the collection of childhood immunization data is limited by the accuracy with which household respondents can supply information on children’s immunization status.Data available to CDC before the initiation of the NIS, including a report commissioned by the agency in 1975 to review the United States Immunization Survey, questioned the assumption that parents could accurately recall immunization history. Even as the NIS was initiated in 1994, NCHS had a study in progress to assess the accuracy of responses to the immunization supplement of the NHIS. It is well documented that survey respondents have trouble accurately recalling the occurrences of, distinctions among, and number of, events that are not particularly salient, or that are similar in nature, or that are repeated more than a few times over a long time period. As a result, when surveyed, they sometimes forget when the events occurred and are confused as to how many of which types of events occurred. As a rule, if events are socially desirable, respondents tend to overreport them.

The NIS asks about the receipt of 14 different immunizations, given in repeated sets, varying in number, over a 1- to 3-year period. Respondents may not understand the differences among the various types of shots and probably consider getting shots socially desirable. As noted, these elements are among the factors associated with inaccurate reporting. To the extent that a parent is able to answer from an up-to-date vaccination record, few of these errors would occur, but significant portions of NIS respondents did not have a shot card and consequently reported from memory. Others apparently used shot cards that were not up-to-date.

In December 1994, after the first two quarters of NIS data collection, CDC acknowledged the need to check parents’ responses against provider records. At that time, NCHS had determined from its surveys assessing the accuracy of parental responses to immunization questions in the NHIS that household respondent reports of vaccinations contain a number of errors that result in underestimation of the “true” vaccination coverage levels. NCHS concluded that, although respondent information was necessary for estimation and demographic analysis, household respondent records of immunizations are often not sufficiently up-to-date to provide accurate information, errors in reports from recall exist, and the household information must be adjusted using provider data. Using the findings from the NHIS substudy, NCHS and NIP attempted to adjust the NIS estimates. However, these adjustments resulted in estimates that did not differentiate the IAP areas. Therefore, CDC determined that a provider substudy similar to the one being conducted in connection with the NHIS was needed to produce accurate vaccination coverage level estimates from the NIS. We reviewed the level of agreement between household reports and physician records from the NIS substudy and confirmed that it was generally only “poor” or “fair” based on the application of recognized statistical criteria.

Earlier recognition of this problem might have led to more serious consideration of other survey methods.

Coverage Estimates Failed to Meet Desired Levels of Precision

The survey plan called for precision of plus or minus 5 percent for a coverage estimate of 50 percent, meaning that the margin of error would have been narrower for more extreme coverage estimates. Owing to various factors, the actual estimates produced by the survey in its first year had margins of error that were often larger. As these margins of error increase, the survey’s capacity to detect changes in immunization coverage decreases: it becomes more difficult to distinguish a change of a particular size from simple error in the estimates.

CDC officials have indicated that the survey is useful in that it permits them to rank states and helps to motivate the lower ranking states to take positive action to improve immunization coverage. However, partly because survey estimates did not meet planned levels of precision, there appear to be remarkably few differences across states. For example, for the most recently published four quarters of NIS data (quarter 3 of 1994 through quarter 2 of 1995), in 31 states, the estimated percentage of children up-to-date in their immunizations could not be statistically distinguished from the national percentage of children up-to-date. (See figure 1.)

Moreover, the survey is unlikely to show change from quarter to quarter. The Final Sampling Plan for the survey notes, “it will only be possible to detect very large changes between adjacent annualized estimates.” For example, a move from 50- to 70-percent coverage would have been the smallest detectable change had the planned level of precision been achieved. As a result, there are no statistically significant changes in full coverage across the first three sets of survey results published by CDC for any of the 78 states or urban areas surveyed.

The smallest change that the survey is likely to detect between successive years for a particular IAP area (for example, quarters 1-4, 1995, versus quarters 1-4, 1996) may in some areas approach the size of the largest change observed between successive years in recent years’ data from the NHIS for antigens that had been recommended before every child in the survey cohort was born. Thus, even if changes of a typical size were occurring, the survey results might create the false impression of a lack of progress. At a minimum, the survey’s broad margins of error indicate that reporting such statistics each quarter is neither necessary nor advisable. Moreover, the imprecision of the survey estimates combined with their narrow range raises questions about whether the survey provides an improved basis for distribution of incentive funds across states.

NCHS officials acknowledged that they had considered reporting the results only semi-annually. However, even this may be too frequent. For those vaccines that have been recommended for a number of years—measles, polio, 3 doses of diphtheria, tetanus, and pertussis—coverage is 80 percent or higher, limiting the size of any increases that might occur.

Identification of Pockets of Need

CDC officials have indicated that they view identification of pockets of children in need of more timely immunization as a state responsibility rather than a federal one. Although a departmental statement accompanying the fiscal 1997 budget request had indicated the NIS would be useful in identifying pockets of need, HHS officials told us that the statement was in error. CDC has indicated that the National Immunization Survey was not designed to identify such “pockets of need,” and consequently, it does not do so. Our survey of state immunization program managers confirmed that they generally drew upon other data for this purpose.

Instead, the primary objectives CDC has for the NIS have been monitoring state progress in achieving childhood immunization objectives, permitting comparison of current coverage rates across states, and awarding incentive funds available to CDC grantees based on their immunization of certain percentages of preschool children. In this connection, we note that the accomplishment of national immunization goals is simultaneously tracked through supplements to the NHIS and that the cost of mounting the NIS (roughly $13 million in fiscal 1995) has been large relative to the total amount of incentive funds it is used to distribute ($33 million in fiscal 1996). We have noted above the survey’s limitations for monitoring changes in immunization coverage.

Although the NIS can produce national statistics for some nongeographically defined subgroups, the sample size of the NIS is not large enough to provide subgroup statistics for each state or urban area. On a national basis, the NHIS provides these same subgroup statistics with the exception of immunization coverage estimates for persons of Hispanic and Asian origin. CDC has suggested that the NIS can be used to evaluate immunization activities; however, the NIS does not currently collect information that could link immunization coverage to specific programs. For example, CDC has encouraged immunization among in participants in the Special Supplemental Food Program for Women Infants, and Children (WIC). However, state estimates of immunization coverage by WIC participation derived from the NIS would have unacceptably large sampling error unless the survey sample size were increased at substantial expense.

Conclusion and Matters for Consideration

We have not had the opportunity to assess the NIS in light of the list of additional purposes for the survey provided to us by HHS after our exit conference on this study. Further, our survey of state immunization directors turned up anecdotal evidence that a few states view the NIS favorably even though they are unable to use it to target pockets of underimmunized children. However, while the NIS has provided estimates of current state-specific immunization levels for awarding incentive grants and monitoring progress toward early childhood immunization objectives, it has significant limitations when used for these purposes.

First, of the appropriation that it has requested for fiscal 1997, CDC has requested $16 million for the survey and its administration. However, the actual costs of the NIS are now expected to be between $12 and $13 million, and even these amounts would render it an inefficient method of allocating incentive grants expected to total $33 million. Second, the NIS does not provide useful quarterly measurements of statewide immunization levels, and even annual estimates may not be suitable for monitoring the level of annual change that is likely to occur in immunization coverage. Third, the NIS does not assist in the systematic targeting of underimmunized children, a particular concern if HHS is to achieve levels of disease reduction and elimination established as goals for the end of this decade. To follow up on this report, we intend to continue to study the various means of identifying pockets of children in need of immunization.

State officials did make use of the NIS findings in communicating with their legislators and the press; however, these objectives could be met by previous methods at markedly lower cost. Moreover, the survey provides only a statewide or citywide indicator of immunization coverage. Insofar as this indicator is not linked to any specific component of the unique set of immunization initiatives pursued by a particular CDC grantee, it is not surprising that it is not useful in helping states to diagnose problems in their ongoing activities, target their efforts, or design interventions.

CDC has also stressed the motivational benefit of ranking states. Apart from the concerns we have raised about the survey’s capacity to rank states, it is difficult to quantify the benefits of this ranking.

In view of these limitations, the Congress may wish to reconsider the NIS’s benefits relative to its cost. At a minimum, the Congress may want to ensure that the CDC appropriation reflects a more accurate estimate of the survey’s cost.

Agency Comments

We provided a draft of this report to CDC officials for their comments, which are reprinted in appendix IV. CDC does not dispute the cost we reported for the NIS or that CDC’s fiscal 1997 budget request for the survey exceeds by at least $3 million the survey costs the agency anticipates in fiscal 1997. CDC disagrees with some of our findings regarding the survey’s methodology and our suggestion that the Congress may wish to consider NIS’ benefits relative to its costs. However, the agency bases some of its objections on statements that incorporate inaccurate representations of our findings regarding the validity of survey estimates and factual and technical errors, which we have identified in appendix IV.

CDC indicates that following our presentation of our findings to the agency in late June, we failed to assess all the benefits of the survey that they had identified. However, the additional benefits asserted by CDC after our work was completed break no new ground. Each of these putative benefits stems from the use of the survey findings to compare state performance, monitor changes in immunization coverage across time, or evaluate intervention efforts. However, with few exceptions, our findings cast doubt on the appropriateness or practicality of such uses of survey results in view of the survey’s broad margins of error for particular states and urban areas, the generally high level of coverage for individual vaccines, and the difficulty of attributing changes across time or place to any particular causal factor.

CDC asserts that the survey provides an early warning of precipitous changes in immunization coverage; however, we are concerned that the survey may lend a false sense of security by obscuring the existence of substantial pockets of underimmunized children. For example, a recent household survey of central and southeast Seattle found an immunization coverage rate of 57 percent, in contrast to the 79 percent reported by the NIS for the King County area incorporating Seattle. Further, NIS data are not generally analyzed and released until a year after data collection. We agree with the CDC that the survey is technically capable of detecting changes in use of newly introduced vaccines, but CDC already monitors these changes on a national basis through its NHIS. Other means, such as sales and distribution reports, may be available for monitoring the initial uptake of newer vaccines at less expense.

Some data from the late 1980s indicated that immunization coverage levels in the preschool population were quite low and highly variable across areas. While the NIS might have been more useful under those circumstances, it appears the situation has changed. Coverage for particular diseases is now quite high, and coverage for long-recommended vaccines has not been highly variable across states. While the survey does provide more timely immunization coverage data than the retrospective surveys that were formerly used for such data collection, it does so at much higher cost. Thus, in the interest of using immunization resources most efficiently, we have suggested that the cost of collecting and analyzing these data be weighed against their continued utility.

As we agreed with your office, we are sending copies of this report to other interested congressional committees, the Secretary of HHS, the Director of CDC, and other federal and state officials. We will also make copies available to others upon request.

If you have any questions or would like additional information, please contact me, at (202) 512-3092, or Sushil K. Sharma, Assistant Director, at (202) 512-3460. Other major contributors to this report are listed in appendix V.

Recommended Childhood Immunization Schedule

This schedule was approved by the Advisory Committee on Immunization Practices, the American Academy of Pediatrics, and the American Academy of Family Physicians.

Vaccines are listed under the routinely recommended ages. Bars indicate the range of acceptable ages for vaccination. Shaded bars indicate catch-up vaccination: at 11-12 years of age, hepatitis B vaccine should be administered to children not previously vaccinated, and varicella zoster virus vaccine should be administered to children not previously vaccinated who lack a reliable history of chicken pox.

The Urban IAP Areas for Which the NIS Produces Immunization Coverage Estimates

The urban IAP project names are listed below in roman type; the names in bold identify the largest city in the IAP county project.

Atlanta, Georgia (Fulton/DeKalb Counties) Baltimore, Maryland Birmingham, Alabama (Jefferson County) Boston, Massachusetts Chicago, Illinois Cleveland, Ohio (Cuyahoga County) Columbus, Ohio (Franklin County) Dallas, Texas (Dallas County) Detroit, Michigan El Paso, Texas (El Paso County) Houston, Texas Indianapolis, Indiana (Marion County) Jacksonville, Florida (Duval County) Los Angeles, California Memphis, Tennessee (Shelby County) Miami, Florida (Dade County) Milwaukee, Wisconsin (Milwaukee County) Nashville, Tennessee (Davidson County) New Orleans, Louisiana New York City, New York Newark, New Jersey Philadelphia, Pennsylvania (Philadelphia County) Phoenix, Arizona (Maricopa County) San Antonio, Texas (Bexar County) San Diego, California (San Diego County) San Jose, California (Santa Clara County) Seattle, Washington (King County) Washington, DC (District of Columbia)

Households With 2-Year-Old Children That Lack a Telephone and Overall Response Rates for Households With Telephonesa

Jefferson County (Birmingham)

Maricopa County (Phoenix)

Duval County (Jacksonville)

Dade County (Miami)

Fulton/DeKalb County (Atlanta)

Marion County (Indianapolis) (continued)

New York City (5 counties) (continued)

Shelby County (Memphis)

Davidson County (Nashville)

Bexar County (San Antonio)

King County (Seattle) (Table notes on next page)

By area surveyed, for quarters 2 through 4, 1994.

Comments From the Department of Health and Human Services

The following are GAO’s comments on the Department of Health and Human Services’ letter dated July 22, 1996.

GAO Comments

1. CDC has mischaracterized our findings. Although we have identified several issues that raise questions about accuracy, neither we nor CDC can validate the accuracy of survey results. The accuracy of the NIS results depends on the accuracy of the assumptions inherent in CDC’s adjustment of the survey results, some of which are untestable. The results of the NHIS are used to adjust the results of the NIS. Thus, while the similarity of the two surveys is reassuring, the NHIS cannot provide an independent assessment of the NIS’ accuracy. In any event, the agreement of the national estimates does not ensure that the local estimates are accurate. 2. The various benefits asserted by CDC derive from the application of the NIS to monitoring immunization rates and to comparing them across states. We acknowledged both of these objectives in the second paragraph of our report.

Many potential benefits or purposes could be asserted for the survey, but its use in any of these capacities is limited by the low precision, narrow range, and unverified accuracy of the survey estimates. 3. It is true that surveys, to varying degrees, customarily require the types of adjustments applied to the NIS to correct for biases introduced by nonresponse and limitations in survey coverage. However, the adjustment of NIS results for exclusion of households without telephone service required a somewhat greater leap of faith than customary adjustments for telephone noncoverage.

The success of such adjustments usually depends on the extent to which the variable being measured can be accurately predicted by demographic characteristics that are available or can be inferred for both nontelephone and telephone households. As we have noted in the report, based on data from the NHIS, which is an in-person survey, there are large differences in immunization coverage between children in households with and without telephones. These differences are not completely explained by demographic differences between telephone and nontelephone households.

Furthermore, although telephone ownership varies substantially across the surveyed areas, there are no consistent sources of state and local data on differences in immunization coverage between telephone and nontelephone households. Consequently, the extent to which this adjustment improved the accuracy of state and local survey results is unclear. 4. CDC has acknowledged that the NIS does not identify pockets of children in need of more timely immunization, and most state immunization program managers have told us that the NIS does not help them in targeting their efforts or designing interventions, although it does relieve them of CDC’s previous requirement that they collect statewide coverage data on their own. We are studying alternative means for identifying pockets of need.

Although there is currently no other means of comparing statewide immunization coverage data, the NHIS, as we have noted, tracks coverage changes at the national level. In addition, other methods were used in the past to collect statewide coverage information, albeit through a variety of methods across states. 5. It is true that the sample size of the NIS should afford the calculation of rates for such subgroups on a national basis. The NHIS is not currently large enough to provide childhood immunization coverage information on these two groups. 6. CDC states that the NIS is an “important public health management tool” and notes that Missouri, Arizona, and Idaho have taken steps intended to improve immunization coverage in the wake of NIS results. However, we have some concern that the NIS provides no guidance on the type of action that is appropriate or where it is appropriate. It is not necessarily clear that placing special emphasis on the states with the lowest survey estimates for coverage with a combination of four vaccines is the most appropriate way to prevent a disease outbreak. States with high estimates may nonetheless include significant pockets of underimmunized children. 7. CDC provides no evidence that the NIS is cost-effective. As we note in our conclusion, it is markedly more expensive than the retrospective surveys previously used to generate statewide coverage data. Presuming that the capacity to measure differences between states is an important objective, the NIS’ capacity to meet this objective is limited by the broad margins of error in survey estimates and variations in survey participation and coverage. It is similarly limited with respect to monitoring changes in immunization coverage across time. As with previous state surveys, there is no guarantee that the NIS provides unbiased estimates of immunization coverage. 8. The NIS can detect small changes on a quarterly basis only at the national level. Survey results are not released until roughly a year after data collection, and it is doubtful that a 1-percent change in national coverage should or would be construed as an early warning in the context of very high vaccine-specific rates. In any case, national coverage statistics are also available from the NHIS. Availability of the NIS results did not prevent the recent outbreak of measles in Utah.

Sudden drops in immunization levels for a particular disease in other countries have been associated with problems, such as sudden concerns about vaccine safety, that were evident apart from immunization measurement. There was concern and widespread publicity in the mid-1970s in both the United Kingdom and Japan about reports of encephalitis following the receipt of pertussis vaccine. The reduced utilization of this vaccine was precipitous and observable from sources other than national survey data. 9. While states with lower immunization estimates may be motivated by the NIS findings to improve coverage, the findings do not indicate where the problem lies within these states or what corrective actions are needed. We remain concerned that they may provide a false sense of security to other areas that actually face significant problems (for example, specific pockets of low immunization within states with generally high coverage rates). In addition, the motivational effects of such quarterly ranking may diminish over time. Finally, CDC’s argument presumes that states will be more motivated to act by data collected through the NIS than they would have been by data collected locally or through other means. We disagree. 10. The CDC has indicated that the NIS was not intended to identify pockets of need and consequently does not do so. The NIS may actually deflect attention from some serious problem areas because they are incorporated in larger areas for survey purposes. For example, the Seattle-King County Department of Public Health and the University of Washington conducted a separate household survey of Central and Southeast Seattle using the same age group and reference dates as the NIS, but finding that 57 percent of children in this part of the city were fully immunized, in contrast to the NIS rate of 79 percent up-to-date for all of King County in the same time period.11. Because of the wide margins of error of survey estimates, the NIS is probably not sufficiently sensitive to permit evaluation of interventions or policy changes in particular areas or subgroups. Although national changes in immunization coverage may be monitored with greater precision, changes in national or local immunization coverage might be attributable to factors other than policy changes (for example, trends in the demographic characteristics of children to be immunized). Moreover, policy changes typically occur in groups and are implemented gradually, which would make it quite difficult to attribute any observed movements in immunization coverage to a single change or a combination of changes.

In this context, it seems inadvisable to draw conclusions about particular state activities based solely on the results of the NIS. Similarly, with cross-state comparisons, multiple interventions are linked to each area and subgroup, as well as variations in demographic and other factors, making it difficult to disentangle the reasons for any differences observed across states and cities in the NIS findings. 12. We agree that the NIS is technically capable of detecting the rapid and dramatic changes in coverage that typically accompany the recommendation of new vaccines. However, on a national level, the NHIS also reports on the uptake of newly recommended vaccines. Sales and distribution reports may provide a less expensive means of monitoring the uptake of such vaccines in particular areas. 13. Even small states had produced statewide coverage estimates using previous methods. However, it is difficult for small states to justify the use of $165,000 in infrastructure funding for a random digit dialing immunization survey such as the NIS. Under a proposal CDC has floated with states, surveys in small states would be subsidized by “contributions” of a percentage of federal grant funds from larger states. However, in view of immunization needs, 20 state health officers surveyed by ASTHO could not justify devoting 6.5-10 percent of their infrastructure funds to survey support. Twenty-four states told ASTHO they were willing to contribute 10 percent of their 1995 infrastructure grant toward the survey in the event that federal funding was discontinued, but their prospective contributions would have totaled $4.6 million—much less than the survey’s reported annual cost. 14. As we have noted, the precision of current estimates raises questions about whether the survey does, in fact, provide an improved basis for the distribution of incentive funds. Moreover, the amount expended on the survey is substantial in comparison to the amount of such funds available for distribution. 15. Most state immunization program managers indicated that the NIS results were not useful in targeting their activities. Although a low result may provide some states with a general incentive to do better, it provides no guidance as to how to accomplish any improvement. 16. The collection of such data will enhance the information derived from the tremendous number of phone contacts with ineligible households made in conducting the NIS. However, the collection of immunization data may continue to drive the number of calls required (and hence the cost of the survey) because households containing two-year-olds would likely continue to be the rarest population sampled. In any case, the utility of the survey for collecting other data does not bear upon its usefulness for collecting information on immunization. 17. CDC agrees with the cost we reported for the NIS. We did not verify CDC’s claims regarding its use of the funds that were not applied to the survey. While CDC anticipates that future costs will be lower, it has not requested modification of its fiscal 1997 budget request to reflect these lower costs. 18. The poor quality of immunization data gathered from household respondents had been documented before the NIS was planned. Thus, although the provider surveys may have reduced the inaccuracies contained in these household data, the survey might have been more efficiently designed had the limitations of household data been acknowledged in survey planning. Earlier recognition of this problem would have supported more serious exploration of other survey methods. 19. It should be noted that CDC’s comments compare the survey estimates to a standard different from the target established in the contract and survey plan. Survey plans are ordinarily drawn by determining the sample size necessary to achieve an acceptably precise result if the value of the measured variable is near 50 percent, the point at which the largest sample will be required to achieve a given level of precision (for example, plus or minus 5 percent with 95-percent confidence). This is exactly the sampling target specified in CDC’s contract with the survey organization. Insofar as the immunization levels measured by the survey are well above 50 percent, had the targets established in the contract been met, the estimates would show precision better than plus or minus 5 percent.

Further, CDC’s statement that, “Seventy-one of the 78 areas met or exceeded the requirement that the margin of error be within five percent of the value of the estimate itself,” does not conform to the first four quarters of survey results published by CDC (see MMWR, Feb. 23, 1996, pp. 148-49). These indicate that, for 4:3:1 coverage, only 23 of the 78 estimates met or exceeded the criterion that the margin of error be within 5 percent of the value of the estimate itself. For 4:3:1:3 coverage, the number meeting or exceeding this criterion was only 16 of the 78.

Whether the survey estimates met this or any other criterion is less important than the fact that their precision, if not improved, is generally only sufficient to detect, reliably, changes of a size larger than has typically been observed on an annual basis.

While the addition of provider data has helped correct some substantial errors incorporated in household responses, it has not reduced the margins of error for survey estimates. 20. We do not find that the survey documents high levels of variability in results across IAPs. Although CDC correctly states that Alabama’s result was statistically different from the result for 21 other IAP areas (11 states and 10 cities), it cannot be statistically distinguished from the results in 56 others. CDC is correct that, in most cases, differences of at least 10 points can be statistically distinguished, as we show in figure 1 for 4:3:1 coverage, but there is only a 24-point range in the state estimates for full coverage, so the majority of the state estimates—31—are not far enough apart for their difference from the national estimate to be confidently attributed to anything more than sampling error. The range of estimates for coverage with particular vaccines is generally narrower. 21. The NIS can detect reasonably small changes in national coverage between consecutive four-quarter annualized estimates, though the first two successive annualized estimates for 4:3:1 coverage were not different. However, even at the national level, for most of the antigens and series, the smallest reliably detectable change (at conventional levels of significance) is slightly larger than 1 percent.

At conventional levels of significance, it is impossible to judge differences as small as 5 percent to be statistically significant when most estimates have 95 percent margins of error of 5 percent or greater. Our report quotes a statement in a document issued by the survey contractor noting that the survey can detect only very large changes (for example, a 20-percent increase from 50 percent) between successive quarterly annualized estimates in the various areas surveyed. The margins originally planned would have been no larger than plus or minus 5 percent. However, survey documentation NCHS provided to us notes that “Confidence intervals for the vaccination coverage estimates are somewhat wider than originally planned because provider information is not available for all children in the sample.” In addition, for data collected in quarters 2 through 4 of 1994, the number of completed child-level interviews was less than 90 percent of the sample size called for in the design specifications for roughly a third of the IAPs. This too, would have the effect of increasing the margins of error for survey estimates. 22. While the NIS applies the same methodology across states, the range of state results is not as broad as expected and the performance of many states cannot be differentiated. In any case, in making such comparisons with the NIS, it is important to take into account the wide variations in survey coverage and response rates across states and urban areas. 23. We noted that the retrospective survey approach has both advantages and disadvantages, including the timeliness of data. Retrospective surveys do not produce results as quickly as the NIS; however, even the NIS issues results about a year after data collection, and thus it appears equally ill-suited to provide an early warning. 24. As we have noted in appendix III, the NIS in some areas excludes a similar proportion of children living in households without telephone service. 25. This is generally true, although the costs of a household survey can be comparable in some urban areas, as suggested by recent experience in Norfolk and Seattle. 26. There may be some economies of scale in centralizing the surveys under a single contract, but these must be weighed against the costs of limiting potential bidders to firms equipped to handle a task of this large scale. Conducting separate surveys would have the advantage of permitting the questions to be tailored to provide additional data about state and local initiatives. 27. It is true that the full cost of a random digit dialing survey such as the NIS would be more difficult for smaller states to bear. ASTHO officials reported that many smaller states were unwilling to continue participation in the survey if it meant funding the full cost of their own random digit dialing survey through their infrastructure funding. However, it should be noted that all states have recent experience conducting other types of statewide immunization surveys. 28. Minimal staff hours are generally involved in retrospective surveys. While this is not true of household surveys, states may also contract for such services if they continue to be required. 29. As noted in our report, the Congress may wish to weigh the cost of the NIS against its benefits in order to ensure the most efficient use of immunization resources.

Major Contributors to This Report

Program Evaluation and Methodology Division

Office of General Counsel

George Bogart, Senior Attorney The first copy of each GAO report and testimony is free. Additional copies are $2 each. Orders should be sent to the following address, accompanied by a check or money order made out to the Superintendent of Documents, when necessary. VISA and MasterCard credit cards are accepted, also. Orders for 100 or more copies to be mailed to a single address are discounted 25 percent.

U.S. General Accounting Office P.O. Box 6015 Gaithersburg, MD 20884-6015 Room 1100 700 4th St. NW (corner of 4th and G Sts. NW) U.S. General Accounting Office Washington, DC Orders may also be placed by calling (202) 512-6000 or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and testimony. To receive facsimile copies of the daily list or any list from the past 30 days, please call (202) 512-6000 using a touchtone phone. A recorded menu will provide information on how to obtain these lists.