United States General Accounting Office

GAO Report to the Honorable

Dale Bumpers, U.S. Senate

September 1996 CDC’S NATIONAL

IMMUNIZATION SURVEY

Methodological Problems

Limit Survey’s Utility

G O A

years

1921 - 1996

GAO/PEMD-96-16

GAO United States

General Accounting Office

Washington, D.C. 20548

Program Evaluation and

Methodology Division

B-272491 

September 19, 1996

The Honorable Dale Bumpers

United States Senate

Dear Senator Bumpers:

Recommendations for childhood immunization indicate that children

should receive most of their immunizations before they are 19 months old.

(See appendix I.) To monitor the extent to which this goal is met at the

national level, the Centers for Disease Control and Prevention (CDC) has,

since 1991, administered a supplement to the National Health Interview

Survey (NHIS). States have also monitored this goal with their own

methods.

In 1994, CDC decided to centralize the collection of immunization coverage

data for preschoolers in each state and certain urban areas. The purposes

CDC cited for this effort included monitoring state progress in achieving

childhood immunization objectives, permitting comparison of current

coverage rates across states, and awarding incentive funds available to CDC

grantees based on their immunization of certain percentages of preschool

children.

Thus, to augment the national estimates provided by the NHIS, in 1994, CDC

initiated the National Immunization Survey (NIS) to produce current and

comparable immunization coverage estimates for children aged 19 to 35

months in each state and 28 urban areas receiving CDC funds to implement

immunization action plans (IAPs).1

 (See appendix II.) Unlike the NHIS,

which is a face-to-face household survey, the NIS is conducted by

telephone. Unlike the other state immunization surveys, the NIS is

conducted under a federal contract jointly managed by CDC’s National

Immunization Program (NIP) and National Center for Health Statistics

(NCHS) and financed directly by the CDC rather than through the grant funds

received by each state and urban area. CDC has noted that the NIS is the

largest survey ever conducted to assess vaccination coverage levels of

children in the United States.

1

The decision to develop coverage estimates for children up to 35 months is grounded in

methodological realities rather than disease prevention objectives. The NHIS sample reaches only a

limited number of households with children; thus, to allow a sufficient sample size, the immunization

supplement measures coverage in children up to 35 months. Other age ranges might have been chosen

in designing the NIS; however, this age range was carried over from the design for the NHIS.

Page 1 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Although national, antigen-specific immunization rates are generally high,

there are still areas and communities at continued risk of disease

outbreaks (sometimes called “pockets of need”) because they have

concentrations of children who have not received timely immunizations.2

This report responds to your request that we assess (1) the cost of the NIS,

(2) the methods used by CDC to conduct the survey, and (3) the utility of

the survey in identifying “pockets” of children in need of more timely

immunization.

Results in Brief The cost of the NIS for the most recently completed fiscal year—1995—was

about $13 million, using estimates provided by the Department of Health

and Human Services (HHS) on the costs of the data collection contract and

other survey-related expenditures. These expenditures are large relative to

the incentive awards the survey results are used to distribute, for which

approximately $33 million was available in fiscal 1996. For fiscal 1997, CDC

has requested $16 million for the survey and its administration, as it

requested and received in fiscal years 1995 and 1996. However, even with

the extraordinary expenses incurred in 1995, survey costs did not reach

$16 million.3 CDC officials indicated that the balance of funds received for

the NIS (about $3 million) was spent on other assessment activities.

However, we did not independently verify this statement.

With respect to survey methodology, a number of difficulties are inherent

in applying telephone survey methods to the estimation of preschool

immunization coverage. Although the random-number telephone survey

permits rapid centralized data collection, it also has some widely

recognized weaknesses, including the exclusion of households that lack a

telephone, lower response rates than typify other survey methods, and

inefficiencies in identifying particular types of households. Recognizing

the biases inherent in the survey technique and problems with

respondents’ recall of children’s immunization history, CDC has made

various adjustments to the results of the NIS. The accuracy of the coverage

estimates, consequently, depends to some extent on the validity of a highly

complex set of assumptions undergirding these adjustments.

The precision of survey estimates and their narrow range raise additional

concerns about the utility of this approach. The precision of survey

2

See U.S. General Accounting Office, Vaccines for Children: Reexamination of Program Goals and

Implementation Needed to Ensure Vaccination (GAO/PEMD-95-22; June 15, 1995).

3

In fiscal 1995, CDC had to reinterview fiscal 1994 survey respondents in order to obtain their consent

to contact their children’s immunization providers.

Page 2 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

estimates (or their “margin of error”) governs the survey’s capacity to

detect differences in immunization coverage levels over time and across

immunization projects. For a variety of reasons, coverage estimates did

not achieve planned levels of precision, and the survey can detect only

very large changes from quarter to quarter. Moreover, although large

differences were anticipated across states, the range of actual results is

comparatively narrow. It is doubtful that the precision of survey estimates

is adequate for detecting the modest changes that occur from year to year

in most coverage levels. Yet CDC uses the survey to measure, quarterly,

states’ progress toward early childhood immunization objectives and to

reward, annually, those that have reached the highest levels of coverage.

Finally, CDC officials told us that the NIS was not designed to identify

pockets of children in need of more timely immunization, and

consequently, it does not do so. HHS officials indicated that their previous

statement to the contrary was in error.4

 Furthermore, the NIS was not

designed to suggest specific programs or interventions to boost

immunization rates, and our survey of state immunization program

managers confirmed that it did not.

The survey was never intended to identify pockets of children in need of

more timely immunization, and state estimates are higher and closer

together than originally anticipated. Moreover, in view of the survey’s cost

and its methodological limitations for the purposes of comparing state

performance, monitoring progress toward immunization objectives, and

distributing incentive funds, we have serious concerns about its utility and

efficiency. At your request, we are continuing to study the various means

of identifying pockets of need.

Background

CDC’s Immunization

Program

The CDC’s National Immunization Program provides grants to states and 28

urban IAP areas for the purpose of controlling vaccine-preventable

diseases. The Congress made available at least $142 million for these

grants in fiscal 1995. The portion of these funds received by a particular

grantee is based largely upon the amount received the previous year. In

addition to these funds, consistent with statements of the Senate

Appropriations Committee, CDC has awarded annual incentive grants to

4

See Office of the Assistant Secretary for Management and Budget, “Department of Health and Human

Services: The Fiscal Year 1997 Budget,” March 19, 1996, p. 28.

Page 3 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

states since fiscal 1994 to improve the immunization levels of 2-year-olds.5

For awards in fiscal 1994, CDC allocated incentive grants based on

state-supplied estimates of the percentage of fully immunized 2-year-olds.

To establish a common basis for awarding subsequent grants and to

monitor progress toward early childhood immunization objectives, the CDC

designed, and starting in fiscal 1994 began to conduct, the National

Immunization Survey. In fiscal 1996, $33 million was allocated for such

incentive grants.

States’ Previous Methods

for Measuring

Immunization Among

Preschoolers

With the advent of the NIS, states had no further obligation to produce

statewide coverage estimates and were able to use the grant funds

formerly devoted to such measurement for other activities. However, most

states’ former methods for estimating immunization coverage were much

less expensive than the NIS, which CDC has heretofore financed at no cost

to the states. Lately, CDC has made inquiries of state health officers

regarding their willingness to devote certain percentages of grant funds to

support the NIS (see p. 9).

To meet CDC’s former requirement for measuring preschoolers’

immunization coverage, all states used either school retrospective surveys

or other population-based methods to estimate immunization coverage.

Most states estimated immunization coverage among preschoolers

through reviewing the immunization records of children entering first

grade or kindergarten to determine whether their immunizations were

up-to-date when the children were younger, typically when they were 2

years old.

This method has both disadvantages and advantages. It produces

estimates that are about 3 years old by the time the data are gathered, and

immunizations may be selectively collected on school records relative to

the minimum state requirements for school entry, which vary to some

extent across states and may not include the newer vaccines.6

 Because the

retrospective method uses data that are already collected for the purpose

of verifying immunization at school entry, it is fairly inexpensive and

enables some states to develop estimates of immunization coverage at

5

S. Rept. 143, 103d Cong., 1st Sess. 63 (1993), accompanying the fiscal 1994 HHS appropriation; S. Rept.

318, 103d Cong., 2d Sess. 56 (1994), accompanying the fiscal 1995 HHS appropriation; and S. Rept. 145,

104th Cong., 1st Sess. 53 (1995), accompanying the fiscal 1996 HHS appropriation.

6

See T.V. Murphy, P. Pastor, S.B. Turner, et al., “Estimating immunization coverage from school-based

childhood immunization records,” Pediatric Infectious Disease Journal, 14:7 (1995), 561-67.

Page 4 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

substate levels for the use of counties or state health districts.7 Records

for those immunizations required for school entry should provide more

accurate dates of immunization than can be obtained in interviews with

parents, who frequently do not have ready access to immunization

information.

Those states that did not use the retrospective method used others, such

as birth certificate surveys and registry-based methods, that required more

original data collection than the retrospective survey, but produced more

current coverage estimates while providing the states with other benefits

or additional information about their specific activities. In 1995, CDC

dropped its requirement that grantees produce an independent assessment

of preschool immunization coverage with the view that the estimates from

its new NIS would supplant the data that had formerly been gathered by

grantees.8

Sources of Survey Error In general, in assessing the quality of survey findings, analysts should

consider a variety of types of error that may affect a survey result.9

 These

include errors that arise because (1) surveys only involve a sample of the

population of interest, (2) some of the sampled individuals may not

respond to the survey, and (3) some of the population of interest may not

be covered by the group from which the sample was chosen. In addition,

there are problems associated with interviewers, the respondent, or the

questionnaire, such as unclear questions or respondents’ difficulty in

recalling the answers. What is commonly quoted in the reporting of poll

results as the “margin of error,” typically plus or minus 3 percent for a

random sample of 1,500, represents only the error attributable to the first

factor named above. Assessing the quality of survey results also requires

7

See L.E. Rodewald, K.J. Roghmann, P.G. Szilagyi, et al., “The school-based immunization survey: An

inexpensive tool for measuring vaccine coverage,” American Journal of Public Health, 83:12 (1993),

1749-51.

8

Some states have continued to conduct the retrospective survey to preserve the capacity to compare

current coverage levels against older data derived from retrospective methods. The retrospective

surveys generally measure immunization coverage on the 2nd birthday (24 months), while the NIS

measures immunization coverage between 19 and 35 months of age. Since the measured

immunizations are due by 18 months, the estimates provided by the NIS are based on an age range

allowing a median of 9 (and as many as 17) additional months to obtain immunizations, while the

retrospective surveys allow only 6 additional months to reach up-to-date status. Consequently, the

estimates of up-to-date rates derived from the NIS will tend to be higher owing to methodological

differences between the surveys rather than changes in the immunization status of preschoolers. It

should also be noted that the NIS measures immunization coverage in areas (states and cities) defined

by political boundaries. Very different coverage estimates may emerge from surveys of particular areas

or populations incorporated in these states and cities.

9

See P.E. Converse and M.W. Traugott, “Assessing the accuracy of polls and surveys,” Science, 234

(1986), 1094-98.

Page 5 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

considering the extent to which the other sources of error may have

affected the accuracy of survey findings.

Scope and

Methodology

To respond to your request, we met with officials of the National

Immunization Program and the National Center for Health Statistics and

with staff of the CDC contractor conducting the National Immunization

Survey. We reviewed documents describing the structure, performance,

and results of the survey. We also reviewed literature on telephone survey

methodology and parental recall of children’s immunization status. The

methodology report for the 1995 survey was not available as of June 18,

1996, when we conducted our exit conference with CDC, and thus, our

review of survey methodology was limited to the procedures employed in

the 1994 survey and reports of NIS findings issued through June 1996. We

understand from NCHS officials that, since issuance of the 1994

methodology report, procedures for using provider data to adjust survey

results have been documented and sensitivity analyses have been

conducted to measure the impact of changes in various assumptions

inherent in the adjustment of survey results.

To provide information on survey costs, we requested that agency officials

provide data on total payments under the survey contract and estimates of

the costs of related agency activities. We also reviewed the survey contract

and trends in the costs billed under the contract. We did not independently

verify the payments for the survey or the CDC cost estimates, though we

did review the invoices from the survey contractor and assess the agency’s

cost estimates for their consistency with the activities the agencies

conducted. With this exception, our work was conducted in accordance

with generally accepted government auditing standards between March 25

and June 18, 1996.

Finally, we surveyed state immunization program managers regarding how

they had used the results of the NIS and their costs for previous survey

approaches. In addition, CDC provided a list of six former CDC contractors,

officials, and current grantees that they recommended we contact. We

contacted some of these individuals and asked them to provide comments

consistent with their familiarity with the survey’s cost and methodology.

Page 6 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Principal Findings

Survey Costs The cost of the survey includes three major components—expenditures

under the contract issued to conduct the survey and the costs of

survey-related activities conducted by NCHS and NIP. Both NCHS and NIP

were involved in managing the data collection contract and providing

statistical analysis of survey data. In addition to these roles, NIP gathered

and reviewed data from the survey respondents’ immunization providers.

When problems with survey software created a need for a larger

interviewing staff, some work was done by the Bureau of the Census, but

costs for this work are included in estimates provided by NCHS. Table 1

shows the costs of the NIS contract, survey assistance provided by NCHS

and the Bureau of the Census, and NIP’s survey-related activities. Only two

quarters of data were collected in fiscal 1994; 1995 was the first fiscal year

in which the survey operated in all four quarters. Extraordinary expenses

were incurred in fiscal 1995 when the agency discovered it needed to

reinterview 1994 survey participants in order to identify their

immunization providers.

Table 1: Estimates of Total NIS-related

Expendituresa Cost

Fiscal year

Survey

contractb

NCHS and

Census

Bureauc NIPd Total

1994 (2 quarters)e $4.0 $1.8 $0.1 $5.9

1995 (4 quarters) 11.3 1.2 0.4 12.9

1996 (2 quarters)f 4.9 0.7 0.2 5.8

Total (8 quarters) $20.2 $3.7 $0.7 $24.6

aIn millions.

bBased on an NCHS summary provided to us in April 1996. These amounts cover contractor

billings through March 30, 1996.

cEstimates taken from an NCHS summary memo on administrative costs provided to us in

April 1996. For purposes of consistency, the estimate provided for fiscal 1996 has been halved in

order to approximate the costs for the first two quarters.

dRevised estimates provided to us by NIP on June 17, 1996.

eJanuary 1, 1994 to September 30, 1994; in fiscal 1994, only two quarters of survey data were

collected (between April and September).

f

First two quarters of fiscal 1996.

Source: Data from NCHS and NIP.

Page 7 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

The contract to conduct the NIS provides the recipient with a fixed fee and

all reasonable costs for conducting the survey. Expenditures under the

survey contract have risen at twice the rate anticipated at its signing,

reaching nearly the full face amount of the contract halfway through the

54-month performance period. Contractor and agency representatives

attribute the higher rate of expenditures to difficulties arising from the

need to replace survey management software; the higher-than-expected

number of calls required to identify households in the sampling frame; and

the addition of a study to check parents’ responses against provider

records, which increased the complexity of estimating survey results

because of the need to adjust them with provider-derived information. The

number of calls required to identify eligible households will continue to be

an important determinant of survey costs.

According to estimates from CDC and invoices from the contractor, costs

for the NIS have been roughly $25 million through March 30, 1996, including

the $13 million for fiscal 1995, the first complete year of data collection.

Insofar as a number of extraordinary expenses were incurred in fiscal

1995, CDC officials anticipate that final survey costs will decrease in fiscal

1996 and future years. However, for fiscal 1997, the agency has requested

$16 million for the survey and its administration, as it requested and

received in fiscal years 1995 and 1996 based on expenditures in the early

implementation stage of the survey. CDC officials indicated that the balance

of funds received in 1995 for the NIS (about $3 million) was spent on other

assessment activities, such as the NHIS and its provider record check study,

the Clinic Assessment Software Application, and the provision of technical

assistance to the states. However, we have not independently verified this

information.

In its report accompanying the fiscal 1996 appropriations, the Senate

Appropriations Committee noted its concern that the national findings of

the NIS duplicate the findings from the NHIS and that the annual cost of the

survey cannot be justified by its utility.10 The Committee noted particularly

that the survey does not provide significant information on high-risk

communities for targeting purposes, and in some respects, it duplicates

surveys conducted by each state. In the justification for its fiscal 1997

budget request, CDC acknowledged these concerns and noted that it was

holding ongoing discussions with, among others, the Association of State

and Territorial Health Officials (ASTHO) and the Council of State and

Territorial Epidemiologists in which “various options related to the NIS”

were being considered.

10S. Rept. 145, 104th Cong., 1st Sess. 55 (1995).

Page 8 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

For example, CDC explored with ASTHO the level of willingness among state

health officers to finance the survey through state grant funds distributed

by CDC rather than directly through CDC appropriations. However, ASTHO

surveys of its members found that many of the larger states and urban

areas were not prepared to devote 6-10 percent of their immunization

infrastructure grants to support of the survey.11 This is consistent with the

findings of our survey of state immunization program managers, which

indicated that while the NIS findings were widely used to communicate

with the news media and respond to legislative inquiries, they were not

used by most states for targeting their activities or designing

interventions.12

Methodological Challenges

Survey Design Incorporates

Inefficiencies

NIS surveyors identify households with children between 19 and 35 months

old by dialing random telephone numbers and asking a short set of

screening questions to assess the presence of children in the correct age

range. Surveyors ask for the number of doses of various vaccines the child

has received and a variety of demographic information. Even with

sampling refinements implemented by the contractor, only a small

proportion of randomly generated telephone numbers results in contacting

a residence that includes children between 19 and 35 months old. CDC

reported that roughly 1.2 million telephone numbers were called to

complete 25,247 interviews during the first three quarters of data

collection (47 numbers per respondent, with an average of 4-5 calls per

number required to reach a respondent). Thus, roughly 200 calls are

initiated per completed interview. In view of the size of this undertaking,

there was some thought at the time the survey was planned of using it to

gather additional health data, but these plans never came to fruition and

the final survey addressed only immunization issues.

With the view that the data provided by parents and other household

contacts would be sufficient to produce accurate immunization coverage

estimates, the NIS was initially designed as a telephone survey of

households using a modified random digit-dialing technique. When it later

became clear that the data derived from household interviews would not,

on their own, be sufficient to produce accurate estimates of immunization

11The total financing available from those who indicated a willingness to support the NIS through the

donation of 10 percent of infrastructure funds was $4.6 million.

12Thirty-four states told us they had not used the NIS results to target program activities, and 36 told us

they had not used them to design programs or interventions.

Page 9 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

coverage, a second phase of the survey was added—the Provider Record

Check Study. In this second phase, household contacts are asked for the

names of medical providers, who are then approached for independent

information on the dates of immunization for children identified through

the household survey. Thus, to some extent, the NIS was reconceptualized

as a survey of providers using a two-phase design. However, two-phase

designs are usually most efficient

“only when the first-phase element survey costs are smaller than those for the second

phase by a large factor . . . [as when] the first-phase sample identifies the members of the

rare population inexpensively, and the survey items are then collected from them in the

second phase.”13

For the NIS, the reverse is true. It appears that CDC is spending a large sum

of money on the first phase of the survey, which provides low-quality

immunization data but identifies the sample for the second phase, which

provides high-quality immunization data from provider records, albeit for a

smaller number of children. Although the provider-supplied data improves

the accuracy of survey results, earlier recognition of the problems with

relying solely on household data might have led to consideration of more

efficient data collection methods.

As of June 1996, summary coverage estimates had been published for the

first five quarters of NIS data collection (April 1994-June 1995).14 CDC shares

the survey results with state programs shortly before their publication in

Morbidity and Mortality Weekly Report. Thus, the survey findings are

available to states and the general public about a year after data are

collected.

Households Without a

Telephone Are Excluded From

the Survey

The telephone survey technique that CDC adopted for the NIS has certain

inherent biases, notably exclusion of any household without a working

telephone. The Department of Commerce’s National Telecommunications

and Information Administration has recently noted that more than

6 million households still lack a telephone and that low-income and

minority communities are substantially below the national average for

13Graham Kalton, Introduction to Survey Sampling (Beverly Hills, Calif: Sage, 1983), p. 48.

14See CDC, “State and national vaccination coverage levels among children aged 19-35 months—United

States, April-December 1994,” MMWR, 44 (August 25, 1995), 613, 619-23; “National, state, and urban

area vaccination coverage levels among children aged 19-35 months—United States, April 1994-March

1995,” MMWR 45 (February 23, 1996), 145-50; and “National, State, and Urban Area Vaccination

Coverage Levels Among Children Aged 19-35 Months—United States, July 1994-June 1995,” MMWR,

45:24 (June 21, 1996), 508-13.

Page 10 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

telephone penetration.15 The methodology report prepared for NCHS by the

survey contractor notes that

“The high nontelephone noncoverage rates in many of the IAP areas and the large

differences between telephone and nontelephone children’s vaccination rates indicate that

the potential for noncoverage bias is considerable in several IAP areas. Any candidate

estimation technique for the NIS must recognize this potentially large bias, and attempt to

adjust for differences between the telephone and nontelephone groups.”16

Appendix III shows the estimated percentage of households with a

2-year-old child that lack a telephone in each of the IAP areas, and table 2

provides national data from the 1992 and 1993 National Health Interview

Surveys detailing the difference in reported immunization rates between

children in households with and without telephones.

Table 2: NHIS Estimates of Children

With Up-To-Date Immunizations in

Households With and Without

Telephonesa

1992 1993 Respondents

have a shot

card?

With a

telephone

Without a

telephone Total

With a

telephone

Without a

telephone Total

Yes 64.8% 47.8% 62.9% 65.2% 49.1% 63.6%

No 53.3 38.3 52.3 71.3 58.1 70.2

Total 58.2 42.3 56.6 68.6 53.9 67.3

aUp-to-date immunization is defined as the receipt of at least 4 doses of diphtheria and tetanus

toxoids and pertussis vaccine/diphtheria and tetanus toxoids, 3 doses of oral polio virus vaccine,

and 1 dose of measles-mumps-rubella vaccine.

Source: Abt Associates, Inc., The National Immunization Survey (NIS) 1994 Annual Methodology

Report, chapter VIII, table 1. (Submitted to NCHS under contract no. 200-94-7009.)

Although only about 5 percent of all U.S. households lack a telephone, the

absence of one is more than twice as common in households with children

under 2 years old (11.7 percent). However, these national data mask the

wide variation among IAP areas in the percentages of households with

children under 2 lacking telephones, which ranges from 2 to 25 percent

across the 50 states and 28 urban IAP areas. Exclusion of households

without a telephone requires that the survey results be adjusted to account

for the positive bias that may result. However, there is no consistent

source of information on the immunization rates among children in

15See National Telecommunications and Information Administration. Falling Through the Net: A

Survey of the “Have Nots” in Rural and Urban America (Washington, D.C.: U.S. Department of

Commerce, July 1995).

16See Abt Associates, Inc., The National Immunization Survey (NIS) 1994 Annual Methodology Report,

prepared for the National Center for Health Statistics and the National Immunization Program of the

Centers for Disease Control and Prevention under contract no. 200-94-7009, p. 159.

Page 11 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

households without telephones in each area where the NIS is conducted.

Consequently, the adjustment for noncoverage of children without

telephones is based on a complex procedure involving the application of a

statistical model of the probability that a fully vaccinated child in a related

national survey resides in a household with a telephone. It is not possible

to know whether these adjustments are accurate in each of the states and

urban areas covered by the NIS.

Response Rates Vary Widely The response rate is the estimated proportion of the target group (in this

case, households with telephones and age-eligible children) that actually

provided data.17 This rate is important in evaluating survey findings

because, to the extent that nonrespondents might have answered

differently from those who completed the survey, a large nonresponse rate

indicates that survey findings will incorporate bias and require adjustment.

For example, CDC analyses of NIS respondents indicated that, as a group,

they differed in some respects from census and vital statistics estimates

for the population; they slightly overrepresented mothers with more than

12 years of education and in some areas were more likely to report

household incomes exceeding $50,000 and less likely to report income

below $10,000. Thus, answers from those types of respondents who tended

to be underrepresented were weighed more heavily in adjusting survey

results to arrive at final coverage estimates. Such adjustments will remove

bias to the extent that immunization coverage is similar between

respondents and demographically similar nonrespondents. However, there

is no clear way to test this assumption in the various areas surveyed.

For the calendar year 1994 survey, contractors estimated that the overall

response rate was 69.5 percent. Appendix III identifies the overall

response rates reported for each surveyed area. Although households

determined to be eligible through their completion of the screening

questions had high rates of cooperation with the full interview, they

represented a smaller portion of the potential households than would have

been expected based on census data, indicating that some 17.3 percent of

eligible households with telephones were never reached, refused

17Assessing the extent of error introduced by nonresponse is somewhat complicated in a random

digit-dialing telephone survey of the type employed for the NIS. This is true because surveyors are not

simply calling a list of houses with children in the appropriate age range, but a list of random

telephone numbers. If every telephone number drawn into the sample could be categorized according

to its business or residential status and, if residential, according to whether the household included a

child in the appropriate age range, calculation of response rates would be a simple matter. However,

nonresponse may occur well before this is accomplished, as when a sampled number results in

reaching an individual who hangs up before any information can be gathered. Some of these instances

may represent eligible households who refused participation before it could be determined that they

contained children in the correct age range. Others may represent numbers that did not belong to

households, but to businesses or numbers that were not assigned.

Page 12 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

cooperation during the screening phase, or inaccurately responded to the

questions about age-eligible children. Although a response rate in this

range is not atypical of telephone surveys, nonresponse rates tend to run

higher for telephone interviewing than for personal visitation.18 Also, while

overall response rates varied tremendously across states and urban areas,

nonresponse to particular questions ranged as high as 26 percent. When

combined, these factors sometimes reduce to below 50 percent the

effective response rates for key questions (for example, how many times

has your child received a polio vaccine?), raising concerns about the

accuracy of resulting estimates.

Household Respondents Have

Difficulty Providing Accurate

Data

The potential to use household surveys for the collection of childhood

immunization data is limited by the accuracy with which household

respondents can supply information on children’s immunization status.19

Data available to CDC before the initiation of the NIS, including a report

commissioned by the agency in 1975 to review the United States

Immunization Survey, questioned the assumption that parents could

accurately recall immunization history.20 Even as the NIS was initiated in

1994, NCHS had a study in progress to assess the accuracy of responses to

the immunization supplement of the NHIS.

21 It is well documented that

18See, for example, NCHS, An Experimental Comparison of Telephone and Personal Health Interview

Surveys, Data Evaluation and Methods Research, Series 2, No. 106 (Washington, D.C.: U.S.

Government Printing Office, August 1987).

19See, for example, K.P. Goldstein, F.J. Kviz, and R.S. Daum, “Accuracy of immunization histories

provided by adults accompanying preschool children to a pediatric emergency department,” Journal of

the American Medical Association, 270:18 (1993), 2190-94. This study indicated that only 8 percent of

adults questioned in an inner-city emergency room could accurately recall how many of each vaccine

their preschool child had received and that many did not know whether the child had received a

measles vaccination despite a recent epidemic in the city. See also M.D. Joffe and A. Luberti, “Effect of

emergency department immunization on compliance with primary care,” Pediatric Emergency Care,

10:6 (1994), 317-19.

Reports from other countries include: M.A. Soljak, “How many children are fully immunised?” New

Zealand Medical Journal (January 25, 1984), 37-39; P. Hawe, A. Wilson, P. Fahey, et al., “The validity of

parental report of vaccination as a measure of a child’s immunisation status,” The Medical Journal of

Australia, 155 (1991), 681-86; P.A. McKinney, F.E. Alexander, C. Nicholson, et al., “Mothers’ reports of

childhood vaccinations and infections and their concordance with general practitioner records,”

Journal of Public Health Medicine, 13:1 (1991), 13-22; and J.J. Valadez and L.H. Weld, “Maternal recall

error of child vaccination status in a developing nation,” Journal of the American Public Health

Association, 82:1 (1992), 120-22.

20See J. Bean, L. Burmesister, P. Isaacson, et al., “Estimation of Community Immunization Levels,”

report based on CDC Grant No. 21-74-545 (PIC ID number 0436), July 1, 1975. The specific

recommendations of this study of the methods used in the United States Immunization Survey

included, “To institute as soon as practicable additional studies of validity of immunization history by

both record search and comparison with serum antibody levels.”

21Elizabeth R. Zell, James T. Massey, and Trena M. Ezzati-Rice, “An Overview of the National Health

Interview Survey and the State and Local Area Immunization Coverage and Health Survey,” presented

at the National Immunization Conference, Los Angeles, May 1995.

Page 13 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

survey respondents have trouble accurately recalling the occurrences of,

distinctions among, and number of, events that are not particularly salient,

or that are similar in nature, or that are repeated more than a few times

over a long time period. As a result, when surveyed, they sometimes forget

when the events occurred and are confused as to how many of which

types of events occurred. As a rule, if events are socially desirable,

respondents tend to overreport them.

The NIS asks about the receipt of 14 different immunizations, given in

repeated sets, varying in number, over a 1- to 3-year period. Respondents

may not understand the differences among the various types of shots and

probably consider getting shots socially desirable. As noted, these

elements are among the factors associated with inaccurate reporting. To

the extent that a parent is able to answer from an up-to-date vaccination

record, few of these errors would occur, but significant portions of NIS

respondents did not have a shot card and consequently reported from

memory.22 Others apparently used shot cards that were not up-to-date.

In December 1994, after the first two quarters of NIS data collection, CDC

acknowledged the need to check parents’ responses against provider

records. At that time, NCHS had determined from its surveys assessing the

accuracy of parental responses to immunization questions in the NHIS that

household respondent reports of vaccinations contain a number of errors

that result in underestimation of the “true” vaccination coverage levels.

NCHS concluded that, although respondent information was necessary for

estimation and demographic analysis, household respondent records of

immunizations are often not sufficiently up-to-date to provide accurate

information, errors in reports from recall exist, and the household

information must be adjusted using provider data. Using the findings from

the NHIS substudy, NCHS and NIP attempted to adjust the NIS estimates.

However, these adjustments resulted in estimates that did not differentiate

the IAP areas. Therefore, CDC determined that a provider substudy similar

to the one being conducted in connection with the NHIS was needed to

produce accurate vaccination coverage level estimates from the NIS. We

reviewed the level of agreement between household reports and physician

records from the NIS substudy and confirmed that it was generally only

“poor” or “fair” based on the application of recognized statistical criteria.23

22As combination vaccines are introduced and children receive immunizations on an increased variety

of schedules, the task of obtaining accurate data on receipt of particular antigens from records or

recall may increase in complexity.

23See D.G. Altman, Practical Statistics for Medical Research (New York: Chapman and Hall, 1991), p.

404.

Page 14 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Earlier recognition of this problem might have led to more serious

consideration of other survey methods.

Coverage Estimates Failed to

Meet Desired Levels of

Precision

The survey plan called for precision of plus or minus 5 percent for a

coverage estimate of 50 percent, meaning that the margin of error would

have been narrower for more extreme coverage estimates. Owing to

various factors, the actual estimates produced by the survey in its first

year had margins of error that were often larger. As these margins of error

increase, the survey’s capacity to detect changes in immunization coverage

decreases: it becomes more difficult to distinguish a change of a particular

size from simple error in the estimates.

CDC officials have indicated that the survey is useful in that it permits them

to rank states and helps to motivate the lower ranking states to take

positive action to improve immunization coverage. However, partly

because survey estimates did not meet planned levels of precision, there

appear to be remarkably few differences across states. For example, for

the most recently published four quarters of NIS data (quarter 3 of 1994

through quarter 2 of 1995), in 31 states, the estimated percentage of

children up-to-date in their immunizations could not be statistically

distinguished from the national percentage of children up-to-date. (See

figure 1.)

Page 15 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Figure 1: Estimated Percentages of Children in Each State Whose Immunizations Are Up-To-Date and 95-Percent

Confidence Boundaries for Each Estimatea

States

100

0

10

20

30

40

90

Upper limit


Lower limit


Estimate


National estimate (plus or minus 1 percent)

50

60

70

80

Percent

aFrom July 1994 through June 1995 for children 19-35 months old. Up-to-date immunization is

defined here as having received at least 4 doses of diphtheria and tetanus toxoids and pertussis

vaccine/diphtheria and tetanus toxoids, 3 doses of oral polio virus vaccine, and 1 dose of a

measles-mumps-rubella vaccine.

Source: Based on NIS data published in “National, State, and Urban Area Vaccination Coverage

Levels Among Children Aged 19-35 Months—United States, July 1994-June 1995,” MMWR, 45:24

(June 21, 1996), 508-13.

Page 16 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Moreover, the survey is unlikely to show change from quarter to quarter.

The Final Sampling Plan for the survey notes, “it will only be possible to

detect very large changes between adjacent annualized estimates.” For

example, a move from 50- to 70-percent coverage would have been the

smallest detectable change had the planned level of precision been

achieved.24 As a result, there are no statistically significant changes in full

coverage across the first three sets of survey results published by CDC for

any of the 78 states or urban areas surveyed.

The smallest change that the survey is likely to detect between successive

years for a particular IAP area (for example, quarters 1-4, 1995, versus

quarters 1-4, 1996) may in some areas approach the size of the largest

change observed between successive years in recent years’ data from the

NHIS for antigens that had been recommended before every child in the

survey cohort was born.25 Thus, even if changes of a typical size were

occurring, the survey results might create the false impression of a lack of

progress. At a minimum, the survey’s broad margins of error indicate that

reporting such statistics each quarter is neither necessary nor advisable.

Moreover, the imprecision of the survey estimates combined with their

narrow range raises questions about whether the survey provides an

improved basis for distribution of incentive funds across states.

NCHS officials acknowledged that they had considered reporting the results

only semi-annually. However, even this may be too frequent. For those

vaccines that have been recommended for a number of years—measles,

polio, 3 doses of diphtheria, tetanus, and pertussis—coverage is 80 percent

or higher, limiting the size of any increases that might occur.

Identification of Pockets of

Need

CDC officials have indicated that they view identification of pockets of

children in need of more timely immunization as a state responsibility

rather than a federal one. Although a departmental statement

accompanying the fiscal 1997 budget request had indicated the NIS would

be useful in identifying pockets of need, HHS officials told us that the

statement was in error. CDC has indicated that the National Immunization

Survey was not designed to identify such “pockets of need,” and

consequently, it does not do so. Our survey of state immunization program

24See Abt Associates, Inc., State and Local Area Immunization Coverage and Health Survey: Final

Sampling Plan (March 7, 1994), p. 19. (Submitted to NCHS under contract no. 200-94-7009.)

25For more newly recommended vaccines, such as Hib and Hep-B, the NHIS has estimated that

coverage only increased 15 and 14 percent, respectively, from the third quarter of 1993 to the second

quarter of 1994. Sales or distribution reports might also be used to monitor uptake of newer vaccines.

Page 17 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

managers confirmed that they generally drew upon other data for this

purpose.

Instead, the primary objectives CDC has for the NIS have been monitoring

state progress in achieving childhood immunization objectives, permitting

comparison of current coverage rates across states, and awarding

incentive funds available to CDC grantees based on their immunization of

certain percentages of preschool children. In this connection, we note that

the accomplishment of national immunization goals is simultaneously

tracked through supplements to the NHIS and that the cost of mounting the

NIS (roughly $13 million in fiscal 1995) has been large relative to the total

amount of incentive funds it is used to distribute ($33 million in fiscal

1996). We have noted above the survey’s limitations for monitoring

changes in immunization coverage.

Although the NIS can produce national statistics for some

nongeographically defined subgroups, the sample size of the NIS is not

large enough to provide subgroup statistics for each state or urban area.

On a national basis, the NHIS provides these same subgroup statistics with

the exception of immunization coverage estimates for persons of Hispanic

and Asian origin. CDC has suggested that the NIS can be used to evaluate

immunization activities; however, the NIS does not currently collect

information that could link immunization coverage to specific programs.

For example, CDC has encouraged immunization among in participants in

the Special Supplemental Food Program for Women Infants, and Children

(WIC). However, state estimates of immunization coverage by WIC

participation derived from the NIS would have unacceptably large sampling

error unless the survey sample size were increased at substantial expense.

Conclusion and

Matters for

Consideration

We have not had the opportunity to assess the NIS in light of the list of

additional purposes for the survey provided to us by HHS after our exit

conference on this study. Further, our survey of state immunization

directors turned up anecdotal evidence that a few states view the NIS

favorably even though they are unable to use it to target pockets of

underimmunized children. However, while the NIS has provided estimates

of current state-specific immunization levels for awarding incentive grants

and monitoring progress toward early childhood immunization objectives,

it has significant limitations when used for these purposes.

First, of the appropriation that it has requested for fiscal 1997, CDC has

requested $16 million for the survey and its administration. However, the

Page 18 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

actual costs of the NIS are now expected to be between $12 and

$13 million, and even these amounts would render it an inefficient method

of allocating incentive grants expected to total $33 million. Second, the NIS

does not provide useful quarterly measurements of statewide

immunization levels, and even annual estimates may not be suitable for

monitoring the level of annual change that is likely to occur in

immunization coverage. Third, the NIS does not assist in the systematic

targeting of underimmunized children, a particular concern if HHS is to

achieve levels of disease reduction and elimination established as goals for

the end of this decade. To follow up on this report, we intend to continue

to study the various means of identifying pockets of children in need of

immunization.

State officials did make use of the NIS findings in communicating with their

legislators and the press; however, these objectives could be met by

previous methods at markedly lower cost. Moreover, the survey provides

only a statewide or citywide indicator of immunization coverage. Insofar

as this indicator is not linked to any specific component of the unique set

of immunization initiatives pursued by a particular CDC grantee, it is not

surprising that it is not useful in helping states to diagnose problems in

their ongoing activities, target their efforts, or design interventions.

CDC has also stressed the motivational benefit of ranking states. Apart from

the concerns we have raised about the survey’s capacity to rank states, it

is difficult to quantify the benefits of this ranking.

In view of these limitations, the Congress may wish to reconsider the NIS’s

benefits relative to its cost. At a minimum, the Congress may want to

ensure that the CDC appropriation reflects a more accurate estimate of the

survey’s cost.

Agency Comments We provided a draft of this report to CDC officials for their comments,

which are reprinted in appendix IV. CDC does not dispute the cost we

reported for the NIS or that CDC’s fiscal 1997 budget request for the survey

exceeds by at least $3 million the survey costs the agency anticipates in

fiscal 1997. CDC disagrees with some of our findings regarding the survey’s

methodology and our suggestion that the Congress may wish to consider

NIS’ benefits relative to its costs. However, the agency bases some of its

objections on statements that incorporate inaccurate representations of

our findings regarding the validity of survey estimates and factual and

technical errors, which we have identified in appendix IV.

Page 19 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

CDC indicates that following our presentation of our findings to the agency

in late June, we failed to assess all the benefits of the survey that they had

identified. However, the additional benefits asserted by CDC after our work

was completed break no new ground. Each of these putative benefits

stems from the use of the survey findings to compare state performance,

monitor changes in immunization coverage across time, or evaluate

intervention efforts. However, with few exceptions, our findings cast

doubt on the appropriateness or practicality of such uses of survey results

in view of the survey’s broad margins of error for particular states and

urban areas, the generally high level of coverage for individual vaccines,

and the difficulty of attributing changes across time or place to any

particular causal factor.

CDC asserts that the survey provides an early warning of precipitous

changes in immunization coverage; however, we are concerned that the

survey may lend a false sense of security by obscuring the existence of

substantial pockets of underimmunized children. For example, a recent

household survey of central and southeast Seattle found an immunization

coverage rate of 57 percent, in contrast to the 79 percent reported by the

NIS for the King County area incorporating Seattle. Further, NIS data are not

generally analyzed and released until a year after data collection. We agree

with the CDC that the survey is technically capable of detecting changes in

use of newly introduced vaccines, but CDC already monitors these changes

on a national basis through its NHIS. Other means, such as sales and

distribution reports, may be available for monitoring the initial uptake of

newer vaccines at less expense.

Some data from the late 1980s indicated that immunization coverage levels

in the preschool population were quite low and highly variable across

areas. While the NIS might have been more useful under those

circumstances, it appears the situation has changed. Coverage for

particular diseases is now quite high, and coverage for long-recommended

vaccines has not been highly variable across states. While the survey does

provide more timely immunization coverage data than the retrospective

surveys that were formerly used for such data collection, it does so at

much higher cost. Thus, in the interest of using immunization resources

most efficiently, we have suggested that the cost of collecting and

analyzing these data be weighed against their continued utility.

As we agreed with your office, we are sending copies of this report to

other interested congressional committees, the Secretary of HHS, the

Page 20 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
B-272491 

Director of CDC, and other federal and state officials. We will also make

copies available to others upon request.

If you have any questions or would like additional information, please

contact me, at (202) 512-3092, or Sushil K. Sharma, Assistant Director, at

(202) 512-3460. Other major contributors to this report are listed in

appendix V.

Sincerely yours,

Kwai-Cheung Chan

Director of Program Evaluation

 in Physical Systems Areas

Page 21 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Contents

Letter 1

Appendix I 

Recommended

Childhood

Immunization

Schedule

24

Appendix II 

The Urban IAP Areas

for Which the NIS

Produces

Immunization

Coverage Estimates

26

Appendix III 

Households With

2-Year-Old Children

That Lack a

Telephone and Overall

Response Rates for

Households With

Telephones

27

Appendix IV 

Comments From the

Department of Health

and Human Services

31

Page 22 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Contents

Appendix V 

Major Contributors to

This Report

46

Tables Table 1: Estimates of Total NIS-related Expenditures 7

Table 2: NHIS Estimates of Children With Up-to-Date

Immunizations in Households With and Without Telephones

11

Figures Figure 1: Estimated Percentages of Children in Each State Whose

Immunizations Are Up-to-Date and 95-Percent Confidence

Boundaries for Each Estimate

16

Figure 1.1: Recommended Childhood Immunization Schedule for

the United States, January-June 1996

24

Abbreviations

ASTHO Association of State and Territorial Health Officials

CDC Centers for Disease Control and Prevention

HHS Department of Health and Human Services

IAP Immunization action plan

NCHS National Center for Health Statistics

NHIS National Health Interview Survey

NIP National Immunization Program

NIS National Immunization Survey

WIC Special Supplemental Food Program for Women, Infants,

and Children

Page 23 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix I 

Recommended Childhood Immunization

Schedule

This schedule was approved by the Advisory Committee on Immunization

Practices, the American Academy of Pediatrics, and the American

Academy of Family Physicians.

Vaccines are listed under the routinely recommended ages. Bars indicate

the range of acceptable ages for vaccination. Shaded bars indicate

catch-up vaccination: at 11-12 years of age, hepatitis B vaccine should be

administered to children not previously vaccinated, and varicella zoster

virus vaccine should be administered to children not previously vaccinated

who lack a reliable history of chicken pox.

Figure 1.1: Recommended Childhood Immunization Schedule for the United States, January-June 1996

Vaccine

Birth 1

Months Old Years Old

Hepatitis Ba,b

Diptheria, tetanus, pertussisc

H. influenzae type bd

Polioe

Measles, mumps, rubellaf

Varicella zoster virus 


vaccineg

2 4 6 12 15 18 4-6 11-12 14-16

MMR

Var Varg

or MMRf MMRf

OPV OPV OPV e

Hibd Hib Hib Hibd

DTP DTP DTP DTPc

(DTaP at 15+m) Td DTP or


DTaP

Hep Bb Hep B-2 Hep B-3

Hep B-1

OPV

Page 24 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix I 

Recommended Childhood Immunization

Schedule

aInfants born to HBsAg-negative mothers should receive 2.5 μg of Merck vaccine (Recombivax

HB) or 10 μg of SmithKline Beecham (SB) vaccine (Engerix-B). The 2nd dose should be

administered 1 month after the 1st dose.

Infants born to HBsAg-positive mothers should receive 0.5 mL hepatitis B immune globulin

(HBIG) within 12 hours of birth, and either 5 μg of Merck vaccine (Recombivax HB) or 10 μg of SB

vaccine (Engerix-B) at a separate site. The 2nd dose is recommended at 1-2 months of age and

the 3rd dose at 6 months of age.

Infants born to mothers whose HBsAg status is unknown should receive either 5 μg of Merck

vaccine (Recombivax HB) or 10 μg of SB vaccine (Engerix-B) within 12 hours of birth. The 2nd

dose of vaccine is recommended at 1 month of age and the 3rd dose at 6 months of age.

bAdolescents who have not previously received 3 doses of hepatitis B vaccine should initiate or

complete the series at the 11- or 12-year-old visit. The 2nd dose should be administered at least 1

month after the 1st dose, and the 3rd dose should be administered at least 4 months after the 1st

dose and at least 2 months after the 2nd dose.

cDTP4 may be administered at 12 months of age, if at least 6 months have elapsed since DTP3.

DTaP (diphtheria and tetanus toxoids and acellular pertussis vaccine) is licensed for the 4th

and/or 5th vaccine dose(s) for children aged 15 months and may be preferred for these doses in

this age group. Td (tetanus and diphtheria toxoids, absorbed, for adult use) is recommended at

11-12 years of age if at least 5 years have elapsed since the last dose of DTP, DTaP, or DT.

dThree H. influenzae type b (Hib) conjugate vaccines are licensed for infant use. If PRP-OMP

(PedvaxHIB [Merck]) is administered at 2 and 4 months of age, a dose of 6 months is not

required. After completing the primary series, any Hib conjugate vaccine may be used as a

booster.

eOral poliovirus vaccine (OPV) is recommended for routine infant vaccination. Inactivated

poliovirus vaccine (IPV) is recommended for persons with a congenital or acquired immune

deficiency disease or an altered immune status as a result of disease or immunosuppressive

therapy, as well as their household contacts, and is an acceptable alternative for other persons.

The primary 3-dose series for IPV should be given with a minimum interval of 4 weeks between

the 1st and 2nd doses and 6 months between the 2nd and 3rd doses.

f

The 2nd dose of MMR is routinely recommended at 4-6 years of age or at 11-12 years of age, but

may be administered at any visit, provided at least 1 month has elapsed since receipt of the 1st

dose.

gVaricella zoster virus vaccine (Var) can be administered to susceptible children any time after 12

months of age. Unvaccinated children who lack a reliable history of chicken pox should be

vaccinated at the 11- or 12-year-old visit.

Source: Adapted from American Academy of Pediatrics, Committee on Infectious Diseases,

“Recommended Childhood Immunization Schedule,” Pediatrics, 97(1), 143 and 145-a.

Page 25 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix II 

The Urban IAP Areas for Which the NIS

Produces Immunization Coverage Estimates

The urban IAP project names are listed below in roman type; the names in

bold identify the largest city in the IAP county project.

Atlanta, Georgia (Fulton/DeKalb Counties)

Baltimore, Maryland

Birmingham, Alabama (Jefferson County)

Boston, Massachusetts

Chicago, Illinois

Cleveland, Ohio (Cuyahoga County)

Columbus, Ohio (Franklin County)

Dallas, Texas (Dallas County)

Detroit, Michigan

El Paso, Texas (El Paso County)

Houston, Texas

Indianapolis, Indiana (Marion County)

Jacksonville, Florida (Duval County)

Los Angeles, California

Memphis, Tennessee (Shelby County)

Miami, Florida (Dade County)

Milwaukee, Wisconsin (Milwaukee County)

Nashville, Tennessee (Davidson County)

New Orleans, Louisiana

New York City, New York

Newark, New Jersey

Philadelphia, Pennsylvania (Philadelphia County)

Phoenix, Arizona (Maricopa County)

San Antonio, Texas (Bexar County)

San Diego, California (San Diego County)

San Jose, California (Santa Clara County)

Seattle, Washington (King County)

Washington, DC (District of Columbia)

Page 26 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix III 

Households With 2-Year-Old Children That

Lack a Telephone and Overall Response

Rates for Households With Telephonesa

Area

Households with

2-year-old children that

lack a telephone

Overall response rate

among households with

telephoneb

U.S. total 11.72% 69.5%

Alabama 20.30

Jefferson County

(Birmingham)

9.03 64.5

Rest of state 66.3

Alaska 9.73 100.0

Arizona 25.41

Maricopa County (Phoenix) 11.79 69.8

Rest of state 67.5

Arkansas 22.11 62.2

California 6.09

Los Angeles 6.94 78.2

Santa Clara 2.21 85.4

San Diego 3.86 73.7

Rest of state 79.5

Colorado 8.21 68.5

Connecticut 5.92 60.2

Delaware 7.49 67.3

District of Columbia 7.36 52.1

Florida 11.91

Duval County (Jacksonville) 10.68 71.8

Dade County (Miami) 9.05 58.8

Rest of state 51.6

Georgia 16.96

Fulton/DeKalb County

(Atlanta)

8.31 64.2

Rest of state 75.0

Hawaii 3.24 81.8

Idaho 11.81 85.1

Illinois 7.16

Chicago 17.14 69.2

Rest of state 71.0

Indiana 13.53

Marion County

(Indianapolis)

12.63 76.0

Rest of state 71.6

Iowa 7.25 65.8

(continued)

Page 27 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix III 

Households With 2-Year-Old Children That

Lack a Telephone and Overall Response

Rates for Households With Telephonesa

Area

Households with

2-year-old children that

lack a telephone

Overall response rate

among households with

telephoneb

Kansas 8.91% 73.4%

Kentucky 19.94 57.7

Louisiana 14.97

Orleans Parish 15.31 60.6

Rest of state 70.0

Maine 7.09 62.4

Maryland 4.00

Baltimore City 16.34 62.9

Rest of state 77.8

Massachusetts 3.42

Boston 8.89 53.5

Rest of state 66.5

Michigan 7.38

City of Detroit 12.75 80.2

Rest of state 72.4

Minnesota 3.98 71.4

Mississippi 23.25 71.2

Missouri 11.15 71.7

Montana 12.13 71.9

Nebraska 7.75 76.3

Nevada 8.31 67.6

New Hampshire 6.33 67.3

New Jersey 6.50

City of Newark 23.32 68.3

Rest of state 56.0

New Mexico 23.65 73.3

New York 6.63

New York City (5 counties) 13.93 61.2

Rest of state 68.8

North Carolina 14.44 67.5

North Dakota 5.95 77.8

Ohio 10.99

Cuyahoga County 7.59 64.8

Franklin County 4.67 65.1

Rest of state 69.1

Oklahoma 18.39 60.1

Oregon 9.34 65.1

(continued)

Page 28 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix III 

Households With 2-Year-Old Children That

Lack a Telephone and Overall Response

Rates for Households With Telephonesa

Area

Households with

2-year-old children that

lack a telephone

Overall response rate

among households with

telephoneb

Pennsylvania 5.02%

Philadelphia 10.25 62.0%

Rest of state 68.3

Rhode Island 7.24 61.0

South Carolina 15.86 63.9

South Dakota 11.37 78.3

Tennessee 17.33

Shelby County (Memphis) 9.43 80.0

Davidson County

(Nashville)

9.67 60.8

Rest of state 66.6

Texas 18.46

Dallas County 13.10 67.1

City of Houston 17.73 71.9

El Paso County 17.22 100.0

Bexar County (San Antonio) 14.49 71.8

Rest of state 83.3

Utah 4.93 100.0

Vermont 7.64 69.0

Virginia 11.07 68.2

Washington 7.64

King County (Seattle) 2.09 69.2

Rest of state 76.7

West Virginia 23.61 56.9

Wisconsin 4.45

Milwaukee County 9.47 69.0

Rest of state 73.0

Wyoming 9.32 85.4

(Table notes on next page)

Page 29 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix III 

Households With 2-Year-Old Children That

Lack a Telephone and Overall Response

Rates for Households With Telephonesa

aBy area surveyed, for quarters 2 through 4, 1994.

bThe overall response rate is defined as ratio between the number of households completing the

interview for at least one child and 0.0508 times the total number of households identified in the

sample. (Census data indicate that 5.8 percent of telephone households contain 2-year-old

children.) The total number of households identified in the sample is defined as the total of

(1) households identified as ineligible; (2) households determined to be eligible; (3) households

for which eligibility was not determined; (4) 90 percent of the numbers that reached an answering

machine or service, but did not result in determination of eligibility; and (5) 90 percent of the

numbers that were categorized as likely households but for which household status was not

clearly determined. The overall response rate calculated in this fashion is slightly higher than the

response rate derived when all possible households are used in the calculation.

Source: Abt Associates, Inc., State and Local Area Immunization Coverage Health Survey: Final

Sampling Plan (March 7, 1994), Exhibit I pp. 4-6, submitted to the National Center for Health

Statistics under contract number 200-94-7009; and The National Immunization Survey (NIS) 1994

Annual Methodology Report, Table 12 Appendix, Chapter V, submitted to the National Center for

Health Statistics under contract no. 200-94-7009.

Page 30 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

Note: GAO comments

supplementing those in the

report text appear at the

end of this appendix.

See comments 1 and 2.

See comment 3.

See comment 4.

See comment 1.

See comment 1.

See comment 5.

Page 31 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 6.

See comment 2.

See comment 7.

See comment 8.

See comment 9.

See comment 10.

Page 32 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 11.

See comment 12.

See comment 11.

See comment 13.

See comment 14.

See comment 15.

See comment 8.

See comment 11.

See comment 16.

Page 33 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 17.

See comment 3.

See comment 1.

Page 34 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 18.

See comment 19.

See comment 20.

See comment 21.

Page 35 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 21.

See comment 22.

See comment 23.

Page 36 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

See comment 24.

See comment 25.

See comment 26.

See comment 27.

See comment 28.

See comment 29.

Page 37 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

The following are GAO’s comments on the Department of Health and

Human Services’ letter dated July 22, 1996.

GAO Comments 1. CDC has mischaracterized our findings. Although we have identified

several issues that raise questions about accuracy, neither we nor CDC can

validate the accuracy of survey results. The accuracy of the NIS results

depends on the accuracy of the assumptions inherent in CDC’s adjustment

of the survey results, some of which are untestable. The results of the NHIS

are used to adjust the results of the NIS. Thus, while the similarity of the

two surveys is reassuring, the NHIS cannot provide an independent

assessment of the NIS’ accuracy. In any event, the agreement of the

national estimates does not ensure that the local estimates are accurate.

2. The various benefits asserted by CDC derive from the application of the

NIS to monitoring immunization rates and to comparing them across states.

We acknowledged both of these objectives in the second paragraph of our

report.

Many potential benefits or purposes could be asserted for the survey, but

its use in any of these capacities is limited by the low precision, narrow

range, and unverified accuracy of the survey estimates.

3. It is true that surveys, to varying degrees, customarily require the types

of adjustments applied to the NIS to correct for biases introduced by

nonresponse and limitations in survey coverage. However, the adjustment

of NIS results for exclusion of households without telephone service

required a somewhat greater leap of faith than customary adjustments for

telephone noncoverage.

The success of such adjustments usually depends on the extent to which

the variable being measured can be accurately predicted by demographic

characteristics that are available or can be inferred for both nontelephone

and telephone households. As we have noted in the report, based on data

from the NHIS, which is an in-person survey, there are large differences in

immunization coverage between children in households with and without

telephones. These differences are not completely explained by

demographic differences between telephone and nontelephone

households.

Furthermore, although telephone ownership varies substantially across

the surveyed areas, there are no consistent sources of state and local data

Page 38 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

on differences in immunization coverage between telephone and

nontelephone households. Consequently, the extent to which this

adjustment improved the accuracy of state and local survey results is

unclear.

4. CDC has acknowledged that the NIS does not identify pockets of children

in need of more timely immunization, and most state immunization

program managers have told us that the NIS does not help them in targeting

their efforts or designing interventions, although it does relieve them of

CDC’s previous requirement that they collect statewide coverage data on

their own. We are studying alternative means for identifying pockets of

need.

Although there is currently no other means of comparing statewide

immunization coverage data, the NHIS, as we have noted, tracks coverage

changes at the national level. In addition, other methods were used in the

past to collect statewide coverage information, albeit through a variety of

methods across states.

5. It is true that the sample size of the NIS should afford the calculation of

rates for such subgroups on a national basis. The NHIS is not currently large

enough to provide childhood immunization coverage information on these

two groups.

6. CDC states that the NIS is an “important public health management tool”

and notes that Missouri, Arizona, and Idaho have taken steps intended to

improve immunization coverage in the wake of NIS results. However, we

have some concern that the NIS provides no guidance on the type of action

that is appropriate or where it is appropriate. It is not necessarily clear

that placing special emphasis on the states with the lowest survey

estimates for coverage with a combination of four vaccines is the most

appropriate way to prevent a disease outbreak. States with high estimates

may nonetheless include significant pockets of underimmunized children.

7. CDC provides no evidence that the NIS is cost-effective. As we note in our

conclusion, it is markedly more expensive than the retrospective surveys

previously used to generate statewide coverage data. Presuming that the

capacity to measure differences between states is an important objective,

the NIS’ capacity to meet this objective is limited by the broad margins of

error in survey estimates and variations in survey participation and

coverage. It is similarly limited with respect to monitoring changes in

immunization coverage across time. As with previous state surveys, there

Page 39 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

is no guarantee that the NIS provides unbiased estimates of immunization

coverage.

8. The NIS can detect small changes on a quarterly basis only at the

national level. Survey results are not released until roughly a year after

data collection, and it is doubtful that a 1-percent change in national

coverage should or would be construed as an early warning in the context

of very high vaccine-specific rates. In any case, national coverage statistics

are also available from the NHIS. Availability of the NIS results did not

prevent the recent outbreak of measles in Utah.

Sudden drops in immunization levels for a particular disease in other

countries have been associated with problems, such as sudden concerns

about vaccine safety, that were evident apart from immunization

measurement. There was concern and widespread publicity in the

mid-1970s in both the United Kingdom and Japan about reports of

encephalitis following the receipt of pertussis vaccine. The reduced

utilization of this vaccine was precipitous and observable from sources

other than national survey data.

9. While states with lower immunization estimates may be motivated by

the NIS findings to improve coverage, the findings do not indicate where

the problem lies within these states or what corrective actions are needed.

We remain concerned that they may provide a false sense of security to

other areas that actually face significant problems (for example, specific

pockets of low immunization within states with generally high coverage

rates). In addition, the motivational effects of such quarterly ranking may

diminish over time. Finally, CDC’s argument presumes that states will be

more motivated to act by data collected through the NIS than they would

have been by data collected locally or through other means. We disagree.

10. The CDC has indicated that the NIS was not intended to identify pockets

of need and consequently does not do so. The NIS may actually deflect

attention from some serious problem areas because they are incorporated

in larger areas for survey purposes. For example, the Seattle-King County

Department of Public Health and the University of Washington conducted

a separate household survey of Central and Southeast Seattle using the

same age group and reference dates as the NIS, but finding that 57 percent

of children in this part of the city were fully immunized, in contrast to the

Page 40 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

NIS rate of 79 percent up-to-date for all of King County in the same time

period.1

11. Because of the wide margins of error of survey estimates, the NIS is

probably not sufficiently sensitive to permit evaluation of interventions or

policy changes in particular areas or subgroups. Although national

changes in immunization coverage may be monitored with greater

precision, changes in national or local immunization coverage might be

attributable to factors other than policy changes (for example, trends in

the demographic characteristics of children to be immunized). Moreover,

policy changes typically occur in groups and are implemented gradually,

which would make it quite difficult to attribute any observed movements

in immunization coverage to a single change or a combination of changes.

In this context, it seems inadvisable to draw conclusions about particular

state activities based solely on the results of the NIS. Similarly, with

cross-state comparisons, multiple interventions are linked to each area

and subgroup, as well as variations in demographic and other factors,

making it difficult to disentangle the reasons for any differences observed

across states and cities in the NIS findings.

12. We agree that the NIS is technically capable of detecting the rapid and

dramatic changes in coverage that typically accompany the

recommendation of new vaccines. However, on a national level, the NHIS

also reports on the uptake of newly recommended vaccines. Sales and

distribution reports may provide a less expensive means of monitoring the

uptake of such vaccines in particular areas.

13. Even small states had produced statewide coverage estimates using

previous methods. However, it is difficult for small states to justify the use

of $165,000 in infrastructure funding for a random digit dialing

immunization survey such as the NIS. Under a proposal CDC has floated

with states, surveys in small states would be subsidized by “contributions”

of a percentage of federal grant funds from larger states. However, in view

of immunization needs, 20 state health officers surveyed by ASTHO could

not justify devoting 6.5-10 percent of their infrastructure funds to survey

support. Twenty-four states told ASTHO they were willing to contribute 10

percent of their 1995 infrastructure grant toward the survey in the event

that federal funding was discontinued, but their prospective contributions

1

See Seattle-King County Department of Public Health and The University of Washington School of

Public Health and Community Medicine (February 1996). “Household Immunization Survey of Central

and Southeast Seattle April-August 1995: Final Report.”

Page 41 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

would have totaled $4.6 million—much less than the survey’s reported

annual cost.

14. As we have noted, the precision of current estimates raises questions

about whether the survey does, in fact, provide an improved basis for the

distribution of incentive funds. Moreover, the amount expended on the

survey is substantial in comparison to the amount of such funds available

for distribution.

15. Most state immunization program managers indicated that the NIS

results were not useful in targeting their activities. Although a low result

may provide some states with a general incentive to do better, it provides

no guidance as to how to accomplish any improvement.

16. The collection of such data will enhance the information derived from

the tremendous number of phone contacts with ineligible households

made in conducting the NIS. However, the collection of immunization data

may continue to drive the number of calls required (and hence the cost of

the survey) because households containing two-year-olds would likely

continue to be the rarest population sampled. In any case, the utility of the

survey for collecting other data does not bear upon its usefulness for

collecting information on immunization.

17. CDC agrees with the cost we reported for the NIS. We did not verify CDC’s

claims regarding its use of the funds that were not applied to the survey.

While CDC anticipates that future costs will be lower, it has not requested

modification of its fiscal 1997 budget request to reflect these lower costs.

18. The poor quality of immunization data gathered from household

respondents had been documented before the NIS was planned. Thus,

although the provider surveys may have reduced the inaccuracies

contained in these household data, the survey might have been more

efficiently designed had the limitations of household data been

acknowledged in survey planning. Earlier recognition of this problem

would have supported more serious exploration of other survey methods.

19. It should be noted that CDC’s comments compare the survey estimates

to a standard different from the target established in the contract and

survey plan. Survey plans are ordinarily drawn by determining the sample

size necessary to achieve an acceptably precise result if the value of the

measured variable is near 50 percent, the point at which the largest sample

will be required to achieve a given level of precision (for example, plus or

Page 42 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

minus 5 percent with 95-percent confidence). This is exactly the sampling

target specified in CDC’s contract with the survey organization. Insofar as

the immunization levels measured by the survey are well above 50 percent,

had the targets established in the contract been met, the estimates would

show precision better than plus or minus 5 percent.

Further, CDC’s statement that, “Seventy-one of the 78 areas met or

exceeded the requirement that the margin of error be within five percent

of the value of the estimate itself,” does not conform to the first four

quarters of survey results published by CDC (see MMWR, Feb. 23, 1996, pp.

148-49). These indicate that, for 4:3:1 coverage, only 23 of the 78 estimates

met or exceeded the criterion that the margin of error be within 5 percent

of the value of the estimate itself. For 4:3:1:3 coverage, the number

meeting or exceeding this criterion was only 16 of the 78.

Whether the survey estimates met this or any other criterion is less

important than the fact that their precision, if not improved, is generally

only sufficient to detect, reliably, changes of a size larger than has typically

been observed on an annual basis.

While the addition of provider data has helped correct some substantial

errors incorporated in household responses, it has not reduced the

margins of error for survey estimates.

20. We do not find that the survey documents high levels of variability in

results across IAPs. Although CDC correctly states that Alabama’s result was

statistically different from the result for 21 other IAP areas (11 states and

10 cities), it cannot be statistically distinguished from the results in 56

others. CDC is correct that, in most cases, differences of at least 10 points

can be statistically distinguished, as we show in figure 1 for 4:3:1 coverage,

but there is only a 24-point range in the state estimates for full coverage,

so the majority of the state estimates—31—are not far enough apart for

their difference from the national estimate to be confidently attributed to

anything more than sampling error. The range of estimates for coverage

with particular vaccines is generally narrower.

21. The NIS can detect reasonably small changes in national coverage

between consecutive four-quarter annualized estimates, though the first

two successive annualized estimates for 4:3:1 coverage were not different.

However, even at the national level, for most of the antigens and series,

the smallest reliably detectable change (at conventional levels of

significance) is slightly larger than 1 percent.

Page 43 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

At conventional levels of significance, it is impossible to judge differences

as small as 5 percent to be statistically significant when most estimates

have 95 percent margins of error of 5 percent or greater. Our report quotes

a statement in a document issued by the survey contractor noting that the

survey can detect only very large changes (for example, a 20-percent

increase from 50 percent) between successive quarterly annualized

estimates in the various areas surveyed. The margins originally planned

would have been no larger than plus or minus 5 percent. However, survey

documentation NCHS provided to us notes that “Confidence intervals for

the vaccination coverage estimates are somewhat wider than originally

planned because provider information is not available for all children in

the sample.”2

 In addition, for data collected in quarters 2 through 4 of 1994,

the number of completed child-level interviews was less than 90 percent of

the sample size called for in the design specifications for roughly a third of

the IAPs. This too, would have the effect of increasing the margins of error

for survey estimates.

22. While the NIS applies the same methodology across states, the range of

state results is not as broad as expected and the performance of many

states cannot be differentiated. In any case, in making such comparisons

with the NIS, it is important to take into account the wide variations in

survey coverage and response rates across states and urban areas.

23. We noted that the retrospective survey approach has both advantages

and disadvantages, including the timeliness of data. Retrospective surveys

do not produce results as quickly as the NIS; however, even the NIS issues

results about a year after data collection, and thus it appears equally

ill-suited to provide an early warning.

24. As we have noted in appendix III, the NIS in some areas excludes a

similar proportion of children living in households without telephone

service.

25. This is generally true, although the costs of a household survey can be

comparable in some urban areas, as suggested by recent experience in

Norfolk and Seattle.

26. There may be some economies of scale in centralizing the surveys

under a single contract, but these must be weighed against the costs of

limiting potential bidders to firms equipped to handle a task of this large

scale. Conducting separate surveys would have the advantage of

2

“National Immunization Survey: EZGuide to Estimation” (undated), p. 7.

Page 44 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix IV 

Comments From the Department of Health

and Human Services

permitting the questions to be tailored to provide additional data about

state and local initiatives.

27. It is true that the full cost of a random digit dialing survey such as the

NIS would be more difficult for smaller states to bear. ASTHO officials

reported that many smaller states were unwilling to continue participation

in the survey if it meant funding the full cost of their own random digit

dialing survey through their infrastructure funding. However, it should be

noted that all states have recent experience conducting other types of

statewide immunization surveys.

28. Minimal staff hours are generally involved in retrospective surveys.

While this is not true of household surveys, states may also contract for

such services if they continue to be required.

29. As noted in our report, the Congress may wish to weigh the cost of the

NIS against its benefits in order to ensure the most efficient use of

immunization resources.

Page 45 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Appendix V 

Major Contributors to This Report

Program Evaluation

and Methodology

Division

Betty Ward-Zukerman, Project Manager

Richard C. Weston, Senior Social Science Analyst

Brian Keenan, Assistant Director for Survey Methodology

Venkareddy Chennareddy, Referencer

Office of General

Counsel

George Bogart, Senior Attorney

(973441) Page 46 GAO/PEMD-96-16 Methodological Problems Limit Utility of NIS
Ordering Information

The first copy of each GAO report and testimony is free.

Additional copies are $2 each. Orders should be sent to the

following address, accompanied by a check or money order

made out to the Superintendent of Documents, when

necessary. VISA and MasterCard credit cards are accepted, also.

Orders for 100 or more copies to be mailed to a single address

are discounted 25 percent.

Orders by mail:

U.S. General Accounting Office

P.O. Box 6015

Gaithersburg, MD 20884-6015

or visit:

Room 1100

700 4th St. NW (corner of 4th and G Sts. NW)

U.S. General Accounting Office

Washington, DC

Orders may also be placed by calling (202) 512-6000 

or by using fax number (301) 258-4066, or TDD (301) 413-0006.

Each day, GAO issues a list of newly available reports and

testimony. To receive facsimile copies of the daily list or any

list from the past 30 days, please call (202) 512-6000 using a

touchtone phone. A recorded menu will provide information on

how to obtain these lists.

For information on how to access GAO reports on the INTERNET,

send an e-mail message with "info" in the body to:

info@www.gao.gov

or visit GAO’s World Wide Web Home Page at:

http://www.gao.gov

PRINTED ON RECYCLED PAPER
United States

General Accounting Office

Washington, D.C. 20548-0001

Official Business

Penalty for Private Use $300

Address Correction Requested

Bulk Rate

Postage & Fees Paid

GAO

Permit No. G100