Vicki Pineau1, Kirk Wolter1, Ben Skalland1, and Wei Zeng1

, 

Carla Black2, Christina Dorell2 , Meena Khare2, and David Yankey2 

1NORC at the University of Chicago

2Centers for Disease Control and Prevention

Modeling Total Survey Error in the 2011 

National Immunization Survey (NIS): 

Pre-School Children and Teens

2013 ASA Joint Statistical Meetings

The findings and conclusions in this paper are those of the 

author(s) and do not necessarily represent the views of 

NORC at the University of Chicago or the Centers for 

Disease Control and Prevention.
• Household CATI survey followed by a mailed provider 

record check study(PRC)

• Landline RDD sampling frame before 2011 for official estimates

• Since 2011 Cell RDD samples have been added

• Household interview screens

 NIS-Child: 19-35 months of age

 NIS-Teen: 13-17 years of age

• Assess and monitor vaccination coverage rates of 

children and teens in the United States. 

• Publish official vaccination estimates at the national, 

state, and selected local levels

• Sponsored by the Centers for Disease Control and 

Prevention

NIS Program Overview

2 
• TSE is the sum of errors from each survey stage and 

includes:

• Sampling Error 

• Non Sampling Error 

• Noncoverage

• Nonresponse 

• TSE Analysis can help answer questions:

• What is the size of bias due to noncoverage and nonresponse 

and what is its impact on estimated vaccination coverage rates?

• What is the impact of different weighting methodologies on the 

total survey errors?

3 

Total Survey Error (TSE)
Comparison of TSE Analysis in the NIS (2010 vs. 2011)

2010

•

•

Assessed bias for multiple NIS-Child and NISTeen vaccination coverage estimates

Considered noncoverage bias potentially 

introduced by missing the cell phone only and 

zero-bank population along with no phone 

population

2011

• Repeated TSE to include improvement in 

population coverage by including a RDD cell 

phone frame

Assessed bias under two different dual frame 

weighting approaches (shrinkage weighting vs. 

unbiased weighting)

Assessed nonresponse bias from decreased 

response rates

•

•
Universe of Eligible Children

Stage 1: Coverage 

Stage 2: Response

 Adequate Provider Data --

Respondents

(p2, μ2)

No Adequate Provider Data --

Nonrespondents

(1-p2, μ2A)

Telephone Number in RDD 

Dual Sampling Frame

(p1, μ1)

No Telephone (p1A, μ1A)

Zero-Bank (Landline Only)

 (p1B, μ1B)

Universe of Eligible Children 

5 
Total Survey Error Analysis Steps:

A. Develop a model describing the survey stages at which 

component error may enter: 

1. Coverage

2. Response (Both the Household interview and the mailed PRC)

Bias due to Nonresponse

q1 = μˆ1 − μ1 = pˆ 2 q2 + (1− pˆ 2 )q2A − v2 (q2 − q2A )

Total Bias

Methodology for TSE Analysis (1)

6 

)()(

1( ˆˆ ) ˆˆ

ˆ

111111

1111111

000

A BA B

BA ABAA

qqvqqv

qpqpqpp

q

−+−

+++−−=

−= μμ

P denotes the corresponding probability of living in such households.

V denotes the sum of sampling and nonsampling error in the estimated probability. 

 μ denotes the conditional mean of vaccination coverage rates among 

children/teens living in a certain household;
B. Obtain best estimates of each component error from 

sources with higher coverage and/or response rates

1) Vital Statistics data

2) American Community Survey (ACS)

3) National Health Interview Survey (NHIS) and NHISProvider Record Check Study (NHIS-PRC)

C. Generate a Monte Carlo simulated dataset using the 

best sources of component error to estimate total bias 

in vaccination coverage rate estimates and associated 

sampling error.

Methodology for TSE Analysis (2)

7 
2011 Simulation Inputs -- NIS-Child (1)

8 

3%

SE=0.4%

0.4%

SE=0.3%

78%

SE=1.4%

18%

SE=1.6%

0%

10%

20%

30%

40%

50%

60%

70%

80%

No Phone LL Only-Zero Bank Nonrespondents Respondents

Distribution of Eligible Children (P’s)
2011 Simulation Inputs -- NIS-Child (2)

9 

67.3%

82.0%

75.4% 76.7% 77.0% 76.8%

30%

40%

50%

60%

70%

80%

90%

100%

No Phone LL Only-Zero

Bank

Design WT Shrinkage WT Unbiased WT

Noncoverage Nonresponse Respondents

Best Estimated Vaccination Coverage Rates for 4:3:1:3:3:1 

Series (μ’s)
• Design Weights

• Adjust for the selection probabilities

• Adjust for the multiplicity of telephone lines

• Adjust for the in household selection for the landline sample

• Adjust for the overlap of the landline and cell sample. 

• Unbiased Weights and Shrinkage Weights

• Full set of weighting adjustment including adjustments for 

selection probabilities, nonresponse, combining the landline and 

cell samples, and raking.

• Unbiased weights: used the true cell phone only cases to 

represent the cell phone only population

• Shrinkage weights: used the true cell phone only cases along 

with some borrowed landline cases to represent the cell phone 

only population

10

Three Different Weighted Estimates
2011 Simulation Inputs -- NIS-Child (3)

11

4+ DTaP 1+ MMR 4:3:1:3:3:1

mean SE mean SE mean SE

No phone Non-Covered

LL-only zero-bank

67.3% 9.3% 82.0% 7.2% 67.3% 9.3%

86.1% 4.3% 93.0% 3.0% 82.0% 4.8%

NonRespondent Non-APD 83.4% 1.6% 91.6% 1.2% 75.4% 1.8%

Respondent

NIS weighted (Design 

wt) 84.0% 0.7% 91.2% 0.5% 76.7% 0.8%

NIS final weighted 

(Shrinkage wt) 84.6% 0.5% 91.6% 0.4% 77.0% 0.6%

NIS final weighted 

(Unbiased wt) 84.4% 0.6% 91.5% 0.5% 76.8% 0.7%
2011 Simulation Inputs -- NIS-Teen

12

1+ Tdap 1+ MenACWY 1+ HPV among females

mean SE mean SE mean SE

Non-Covered No phone 52.7% 13.6% 54.0% 12.1% 67.0% 17.3%

LL-only zero-bank 81.9% 3.6% 70.8% 4.3% 46.8% 6.3%

Non-Respondent Non-APD 74.9% 1.1% 67.9% 1.2% 46.4% 1.8%

Respondent NIS weighted (Design wt) 80.2% 0.5% 72.2% 0.6% 51.4% 1.0%

NIS final weighted (Shrinkage wt) 78.2% 0.5% 70.5% 0.5% 53.0% 0.8%

NIS final weighted (Unbiased wt) 78.0% 0.6% 69.9% 0.6% 52.8% 1.0%

1.1% 

SE=0.14%

0.4% 

SE=0.25%

86.7% 

SE=0.5% 

11.8% 

SE=0.2% 

Distribution of Eligible Teen (P’s)

No Phone

LL-Only Zero Bank

Nonrespondents

Respondents

52.72%

81.9%

74.9% 80.2% 78.2% 78.0%

30%

40%

50%

60%

70%

80%

90%

100%

No Phone Zero Bank Design WT Shrinkage WT Unbiased WT

Noncoverage Nonresponse Respondents

Best Estimated Vaccination Coverage 

Rates for 1+ Tdap (μ’s)
Result 1: Estimated Total Bias Using Three Alternative 

Weights – 2011 NIS-Child

1.3 1.4 1.5

-0.1

0.3 0.3 0.9 1.3 1.5

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

Estimated Total Bias %

4:3:1:3:3:1 Series 1+MMR 4+DTaP

13
14

Result 2: Estimated Incremental Bias due to Noncoverage

and Nonresponse – 2011 NIS-Child

1.1 

-0.4 

0.4 

0.2 

0.3 

0.5 

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

4:3:1:3:3:1 1+MMR 4+DTaP

Incremental Bias % 

Nonresponse Bias Noncoverage bias
Result 3: Estimated Total Bias using 3 Alternative Weights 

– 2011 NIS-Teen

Statistically significant at the 95% confidence level

4.9

2.7 2.9

3.9

1.5 2.2

4.3

5.7 5.9

-4.0

-2.0

0.0

2.0

4.0

6.0

8.0

10.0

Estimated Total Bias %

1+Tdap 1+MenACWY 1+HPV

15
Result 4: Estimated Incremental Bias due to Noncoverage

and Nonresponse – 2011 NIS-Teen

16

4.7 

3.8 

4.5 

0.2 

0.1 

-0.2 

-1.0

0.0

1.0

2.0

3.0

4.0

5.0

6.0

1+Tdap 1+MenACWY 1+ HPV

Incremental Bias % 

Nonresponse Bias Noncoverage bias
Discussion (1)

17

• The estimated bias associated with population noncoverage is 

found to be small for both NIS-Child and NIS-Teen with the dual 

frame telephone sample design.

• NIS-Child: 

• The total bias in the 2011 vaccination coverage rates examined are quite 

small (<=1.5%) and not significantly different from zero, which is comparable 

to 2010 results.

• NIS-Teen: 

• Statistically significant biases were detected in the 2011 NIS-Teen 

vaccination coverage rates ranging from 2.9% to 5.9% (under shrinkage 

weights)

• The estimated bias from nonresponse dominate the total survey error. 
Discussion (2)

18

• Use of shrinkage weights in 2011 was effective at reducing 

variance in vaccination coverage rate estimates without 

increasing bias.

• Limitation wise, as with other TSE studies, results are based on 

several assumptions, multiple sources of data, and models used 

in simulation. 

• Total Survey Error analysis will be repeated using 2012 NIS 

Child and Teen data which has a larger cell-phone sample.
Thank You!

Wei Zeng

ZENG-WEI@NORC.ORG