National Immunization Survey-Teen

Error Profile for the 2022 NIS-Teen

Centers for Disease Control and Prevention

National Center for Immunization 

and Respiratory Diseases

Presented by:

NORC at the University of Chicago

August 2023
Error Profile for the 2022 NIS-Teen i

Acknowledgments

The development and production of the Error Profile for the 2022 NIS-Teen is a team effort 

that has included contributions from many individuals (listed in alphabetical order) in two 

organizations: 

National Center for Immunization and Respiratory Diseases, CDC – Laurie D. Elam-Evans, 

Sean Hu, Cassandra Pingali, and James A. Singleton. 

NORC at the University of Chicago – Zachary H. Seeskin, Benjamin Skalland, Kirk Wolter, and 

Yuhei Koshino.
Error Profile for the 2022 NIS-Teen i

Table of Contents

1. Introduction........................................................................................................1

2. Part I: Comparisons of NIS-Teen Data to External Sources..............................2

2.1 Demographic Distributions: Comparison of NIS-Teen Distributions to Population Distributions...2

2.2 Comparison of NIS-Teen and IISAR Vaccination Coverage Estimates........................................5

What is IISAR?..............................................................................................................................5

Visual Comparison of Vaccination Coverage Estimates ...............................................................6

Adolescent Participation Rate (APR) ............................................................................................8

Negative Relationship between Difference in Vaccination Coverage and APR............................9

2.3 Comparison of Adolescent by Type of Health Insurance Coverage ...........................................10

2.4 Comparison of Vaccination Coverage Estimates: NIS-Teen v. State Immunization Surveys.....12

Comparison to Other IIS-Based Vaccination Coverage Estimates .............................................16

3. Part II: Assessment of Total Survey Error for NIS-Teen Vaccination 

Coverage Estimates ............................................................................................17

3.1 Sampling-Frame Coverage Error................................................................................................19

3.2 Nonresponse Error......................................................................................................................23

Components of Nonresponse in NIS-Teen .................................................................................23

Weight Adjustment for Nonresponse Error .................................................................................23

Assessment of Nonresponse Error .............................................................................................24

NIS-Teen Item Nonresponse Rates ............................................................................................26

3.3 Measurement Error.....................................................................................................................27

3.4 Sampling Error............................................................................................................................29

3.5 Total Survey Error Distribution....................................................................................................30

3.6 Assessment of the Change in Bias Using the Bridging Cohort Method......................................33

4. Summary .........................................................................................................34

References ..........................................................................................................38
Error Profile for the 2022 NIS-Teen ii

List of Figures 

Figure 2.1: Scatterplot of NIS-Teen (in %) v. IISAR (in %) Vaccination Coverage Estimates for 

1+ Tdap: 56 Estimation Areas, 2021....................................................................................7

Figure 2.2: Scatterplot of NIS-Teen (in %) v. IISAR (in %) Vaccination Coverage Estimates for 

1+ Tdap: Six 2013-2017 Sentinel Sites, 2021......................................................................7

Figure 2.3: Scatterplot of IISAR (in %) Vaccination Coverage Estimate for 1+ Tdap v. APR (in %): 

56 Estimation Areas, 2021 ...................................................................................................8

Figure 2.4: Scatterplot of the Difference (NIS-Teen minus IISAR) (in %) v. APR (in %) with 

Regression Line: 56 Estimation Areas, 2021 .......................................................................9

Figure 2.5a: Comparison of Vaccination Coverage Estimates (in %): GIS v. NIS-Teen in GA, 2017 

and 2018 ............................................................................................................................14

Figure 2.5b: Comparison of 1+HPV Coverage Estimates (in %) by Sex: GIS v. NIS-Teen in GA, 

2017 and 2018 ...................................................................................................................14

Figure 2.6a: Comparison of Vaccination Coverage Estimates (in %): KIS v. NIS-Teen in KS, 

Various Years.....................................................................................................................15

Figure 2.6b: Comparison of 1+HPV Coverage Estimates (in %) by Sex: KIS v. NIS-Teen in KS, 

2016 ...................................................................................................................................16
Error Profile for the 2022 NIS-Teen iii

List of Tables 

Table 2.1: One-Way Demographic Distributions Among Adolescents with Adequate Provider 

Data vs. Population Control: NIS-Teen, United States, 2022..................................................4

Table 2.2: Comparison of Alternative Estimates of Health Insurance Coverage Among Adolescents: 

NIS-Teen v. ACS v. CPS ASEC v. NHIS for 2021 and NIS-Teen v. CPS ASEC for 2022....10

Table 2.3: Sample Sizes of Adolescents in Georgia and Kansas ..........................................................13

Table 3.1: Percentage of Age-Eligible Adolescents in the Population by Telephone Status by Year: 

NIS-Teen, United States, 2012-2021 ....................................................................................20

Table 3.2: Vaccination Coverage Estimates and Standard Errors of Select Vaccines and Vaccine

Series for the Cell-Phone and LLO Domains: NIS-Teen, United States, 2017 .....................21

Table 3.3: Vaccination Coverage Estimates and Standard Errors of Select Vaccines and Vaccine 

Series for Adolescents 13-17 Years in the Cell-Phone and Phoneless Domains: 

NHIS-PRC, United States, 2012 ...........................................................................................21

Table 3.4: Preliminary Assessment of Sampling-Frame Coverage Error and Mean True 

Vaccination Coverage Estimate (in %): NIS-Teen, United States, 2022...............................22

Table 3.5: Estimated Parameters of Nonresponse Error in Vaccination Coverage Estimates (in %)

Derived from the 2022 NIS-Teen and the 2021 NHIS...........................................................26

Table 3.6: Item Nonresponse Rates, NIS-Teen Household Interview, United States, 2022 ..................27

Table 3.7: Estimated Under-Reporting Error by Vaccine: NIS-Teen, United States, 2017 and 2019 ....29

Table 3.8: Vaccination Coverage Estimates and Standard Errors Using Design Weight and Final 

Weight: NIS-Teen, United States, 2022 ................................................................................30

Table 3.9: Total Survey Error Model Inputs by Stages: NIS-Teen, United States, 2022........................31

Table 3.10: Mean and 95% Credible Interval for the Estimated TSE Distribution and Component 

Error Distributions: NIS-Teen, United States, 2022 ..............................................................32

Table 3.11: Difference between the Estimates* for the Bridging Birth Cohort: NIS-Teen, 

United States, 2021 vs. 2022 ................................................................................................34
Error Profile for the 2022 NIS-Teen 1

1. Introduction 

Total survey error (TSE) is the difference between a survey estimate and the true value of the 

corresponding population parameter. TSE is the net effect of sampling error and all forms of 

nonsampling error, including sample-frame coverage error, error due to survey nonresponse, and 

errors of measurement (such as reporting, record checking, coding, and other processing errors). TSE 

excludes conceptual errors committed in deciding what should be measured in the survey and 

judgmental errors made in interpreting the survey findings or in making public policy based on the 

survey data.

The main aim of this report is to provide a well-rounded but brief discussion of what is known about 

TSE for 2022 NIS-Teen estimated vaccination coverage at the national level. The statistics and 

methodology of the NIS have been described by Smith, Hoaglin, Battaglia et al. (2005) and Wolter, 

Smith, Khare et al. (2017).

The report is written in two parts. The first part, which appears in Section 2, compares NIS-Teen 

statistics to corresponding benchmarks derived from censuses or large reference surveys, such as the 

National Health Interview Survey and the American Community Survey. A large difference between an 

NIS-Teen statistic and its corresponding benchmark is likely a signal of error in the NIS-Teen or of 

definitional differences between NIS-Teen and benchmark concepts. A small difference may be a signal 

of good accuracy in the NIS-Teen or simply an indicator that the NIS-Teen statistic and its benchmark 

are consistent with one another. This part of the report examines demographic statistics, vaccination 

coverage estimates, and health insurance statistics.

The second part of the report, set forth in Section 3, focuses attention on the NIS-Teen estimated 

vaccination coverage. The material presents what is known from special evaluation studies about the 

component errors and the total error in the vaccination coverage estimates. The section culminates with 

discussion of the distribution of TSE in the 2022 NIS-Teen and of the change in TSE between the 2021

and 2022 NIS-Teen. 

The report closes in Section 4 with a summary of findings and limitations.

Throughout the report, we analyze total survey error for the following vaccines: 1+ Tdap; 

1+ MenACWY; and up-to-date (UTD) status for HPV among the total age-eligible population, among 

females, and among males. Assessments are conducted at the national level (50 states plus District of 

Columbia). 
Error Profile for the 2022 NIS-Teen 2

2. Part I: Comparisons of NIS-Teen Data 

to External Sources

We begin by comparing NIS-Teen demographic distributions (adolescent age, adolescent sex, 

adolescent race/ethnicity, mother’s education, and mother’s age) to benchmark distributions derived 

from the Census Bureau’s Population Estimates Program (PEP) and American Community Survey 

(ACS). Second, we compare NIS-Teen vaccination coverage estimates to estimates set forth in the 

Immunization Information Systems Annual Report (IISAR). Third, we compare health insurance 

distributions derived from the NIS-Teen Health Insurance Module (HIM) to corresponding benchmark 

distributions obtained from (i) the National Health Interview Survey (NHIS), (ii) the ACS, and (ii) the 

Current Population Survey Annual Social and Economic Supplement (CPS ASEC). Finally, we compare 

NIS-Teen vaccination coverage estimates to those available from state immunization surveys from 

Georgia and Kansas, and we discuss literature comparing NIS-Teen vaccination coverage estimates to 

independent estimates based on immunization information systems for Florida and Wisconsin. 

2.1 Demographic Distributions: Comparison of NIS-Teen 

Distributions to Population Distributions

A direct method of estimating survey error is to compare the survey estimates to benchmark estimates 

from other, higher quality sources. While high-quality benchmark estimates of national vaccination 

coverage are not available, we can compare the survey estimates of demographic distributions to those 

derived from the PEP and ACS data.

To create benchmark demographic distributions for adolescents aged 13 to 17 in 2022, we began by 

obtaining PEP data for 2021 by the combination of state, single year of age (12 to 16 in 2021), sex, and 

race/ethnicity (These data are made available from the Census Bureau at approximately a one-year 

lag, and the 2021 estimates were the most recent available at this writing). An adjustment was then 

made within age groups to remove the institutionalized population because the NIS-Teen target 

population excludes institutionalized adolescents. The distribution of education level for mothers of 

adolescents 13-17 years was estimated using the combined 2019, 2020, and 2021 one-year ACS 

Public-Use Microdata Sample (PUMS); these mother’s education distribution estimates were produced 

within the combination of state and adolescent age, sex, and race/ethnicity and then applied to the 

overall PEP estimates for each combination to estimate the total number of adolescents by mother’s 

education for each combination. The distribution of mother’s age was estimated from one-year 2021 

ACS data and applied to the overall PEP estimates to estimate the total number of adolescents by 

mother’s age. Finally, adjustments were made for mortality and foreign immigration; these steps 

account for changes in the adolescent population totals in the one-year period between the reference 
Error Profile for the 2022 NIS-Teen 3

day (July 1, 2021) for the 2021 PEP counts and the midpoint of the reference year (July 1, 2022) for the 

2022 NIS-Teen.

We produced 2022 NIS-Teen national-level demographic distribution estimates first using design 

weights and then using the final weights. The design weights reflect the sample design but do not 

include any adjustments for sampling-frame noncoverage or interview nonresponse and are not 

calibrated to population control totals. Final weights are the design weights, with adjustments for 

noncoverage, nonresponse, and calibration to population control totals. (See the footnotes to Table 2.1 

for the demographics used in this calibration.)

Table 2.1 compares 2022 NIS-Teen national-level survey estimates of demographic distributions for 

adolescents with adequate provider data to benchmark distributions for adolescent age, sex, 

race/ethnicity, mother’s education, and mother’s age. The survey distributions of adolescent age, 

adolescent sex, and mother’s age are very close to the population distributions, even when using the 

design weights. The design-weighted distribution of adolescent race/ethnicity differs from the population 

distribution, with larger differences for non-Hispanic White only adolescents (57.3 percent in survey, 

50.2 percent in population) and Hispanic adolescents (19.1 percent in survey, 25.7 percent in 

population). These differences are substantially smaller when final weights are used (25.6 percent 

survey estimate for Hispanic adolescents, 25.7 percent in population; 48.5 percent survey estimate for 

non-Hispanic White only adolescents, 50.2 percent in population). Note that the final-weighted 

distribution does not exactly match the population distribution because final weights are calibrated to 

only three race/ethnicity categories – Hispanic, non-Hispanic Black, and other (including non-Hispanic 

White) – and in some geographic areas categories for calibration may be collapsed due to small 

sample sizes. 

Differences between survey estimates and population values are observed for mother’s education. The 

survey over-represents adolescents whose mothers have a four-year college degree when the design 

weights are used (51.9 percent in survey, 37.4 percent in population) and under-represents 

adolescents whose mothers have no high school degree, only a high school degree, or some college. 

When the final weights are used, the survey still over-represents adolescents whose mothers have a 

four-year college degree (44.2 percent in survey, 37.4 percent in population) and under-represents 

adolescents whose mothers have some college (23.3 percent in survey, 30.3 percent in population). 

(Final survey weights are calibrated to population totals for less than high school, high school, and 

more than high school, but are not calibrated separately for some college vs. four-year degree.)
Error Profile for the 2022 NIS-Teen 4

Table 2.1: One-Way Demographic Distributions Among Adolescents with Adequate Provider 

Data vs. Population Control: NIS-Teen, United States, 2022 

Demographic

Population 

Percentage

Design Weighted Final Weighted*

Survey 

Estimate

Survey 

Estimate – 

Population 

Percentage 

Survey 

Distribution

Survey 

Estimate – 

Population 

Percentage

Adolescent Age

13 years 19.7 19.7 ± 1.1 0.1 ± 1.1 20.7 ± 1.3 1.0 ± 1.3

14 years 20.4 19.4 ± 1.1 -1.0 ± 1.1 19.4 ± 1.1 -1.0 ± 1.1

15 years 20.3 20.4 ± 1.1 0.1 ± 1.1 20.2 ± 1.2 -0.1 ± 1.2

16 years 19.9 21.5 ± 1.2 1.6 ± 1.2 20.6 ± 1.2 0.6 ± 1.2

17 years 19.7 18.9 ± 1.1 -0.7 ± 1.1 19.2 ± 1.1 -0.5 ± 1.1

Adolescent Sex

Male 51.2 52.6 ± 1.4 1.4 ± 1.4 51.2 ± 1.5 0.0 ± 1.5

Female 48.8 47.4 ± 1.4 -1.4 ± 1.4 48.8 ± 1.5 0.0 ± 1.5

Adolescent Race/Ethnicity

Hispanic 25.7 19.1 ± 1.1 -6.6 ± 1.1 25.6 ± 1.4 -0.1 ± 1.4

Non-Hispanic White only 50.2 57.3 ± 1.4 7.1 ± 1.4 48.5 ± 1.4 -1.7 ± 1.4

Non-Hispanic Black only 13.7 11.3 ± 1.0 -2.5 ± 1.0 13.6 ± 1.0 -0.1 ± 1.0

Other 10.4 12.4 ± 0.9 2.0 ± 0.9 12.3 ± 0.9 1.9 ± 0.9

Mother Education

Less than high school 11.9 6.6 ± 0.7 -5.4 ± 0.7 11.5 ± 1.2 -0.4 ± 1.2

High school 20.4 15.7 ± 1.0 -4.7 ± 1.0 21.0 ± 1.3 0.6 ± 1.3

Some college 30.3 25.9 ± 1.2 -4.4 ± 1.2 23.3 ± 1.2 -7.1 ± 1.2

4-year college graduate 37.4 51.9 ± 1.4 14.5 ± 1.4 44.2 ± 1.4 6.9 ± 1.4

Mother Age

< 35 years 7.6 5.3 ± 0.5 -2.3 ± 0.5 7.0 ± 0.8 -0.6 ± 0.8

35-44 years 46.8 43.5 ± 1.4 -3.4 ± 1.4 43.4 ± 1.4 -3.5 ± 1.4

>= 45 years 45.6 51.2 ± 1.4 5.7 ± 1.4 49.6 ± 1.5 4.1 ± 1.5

Note: Excludes U.S. territory samples.

Note: The notation ± signifies 95% confidence intervals.

* Final provider-phase weights are calibrated within each geographic estimation area to marginal totals for 

adolescent age (13-14, 15-17), adolescent sex (male, female), adolescent race/ethnicity (Hispanic, nonHispanic Black only, other), mother’s education (less than high school, high school, more than high 

school), household telephone status (cell-phone-only, other), and quintile of the estimated propensity to 

have adequate provider data for the adolescent, given the household interview was completed for the 

adolescent.

Comparisons of demographic distributions were made between survey estimates and population values 

for all two-way combinations of adolescent age, adolescent sex, adolescent race/ethnicity, and 

mother’s education, first using design weights and then using final weights. While final weights are 

controlled to marginal population totals for these characteristics individually, the weights are not 
Error Profile for the 2022 NIS-Teen 5

controlled to totals for cross-classifications of these characteristics. Differences between survey 

estimates and population values for cross-classifications involving adolescent age, sex, and 

race/ethnicity are all less than 5.0 percentage points when final weights are used.

2.2 Comparison of NIS-Teen and IISAR Vaccination Coverage 

Estimates 

This section compares NIS-Teen vaccination coverage estimates to Immunization Information System 

Annual Report (IISAR) data in 2021.1 The comparison is given for estimates for coverage of the 1+ 

Tdap vaccine using the data available from IISAR, recognizing that the findings may not apply to other 

vaccine series. Agreement between the vaccination coverage estimates signals consistency between 

the NIS-Teen and IISAR, and it may signal that both sources provide an accurate measurement of the 

true vaccination coverage in the age-eligible adolescent population (13- to 17-year-old adolescents). 

Lack of agreement between the vaccination coverage estimates signals inconsistency and that at least 

one or both sources are less accurate.

Our work in this subsection is divided into four parts. First, we describe some definitions we will use in 

this analysis. Second, we compare the vaccination coverage estimates in NIS-Teen and IISAR visually 

using scatterplots. Third, we introduce the concept of the IIS (Immunization Information System) 2

Adolescent Participation Rate (APR). Finally, we demonstrate through regression analysis that the 

difference between the 1+Tdap vaccination coverage estimates in NIS-Teen and IISAR is related to the 

APR. 

What is IISAR?

The IISAR is an annual assessment of IIS activity among the 64 immunization program awardees, 

which include the 50 states, 6 cities (Chicago, District of Columbia, Houston, New York City, 

Philadelphia and San Antonio), and 8 U.S. territories (American Samoa, Guam, Marshall Islands, 

Micronesia, Northern Mariana Islands, Palau, Puerto Rico and Virgin Islands). To evaluate each 

awardee’s performance, the immunization program manager in the awardee area is asked to complete 

a self-administered and web-based questionnaire asking for demographic and immunization 

information, public and private provider site participation levels, and information about fulfillment of IIS 

functional standards. Because the questionnaire is self-administered and web-based, some awardees 

may report partial data or no data at all. 

1 https://www.cdc.gov/vaccines/programs/iis/annual-report-iisar/overview.html

2 State IIS are computer databases that aspire to contain information about all of the doses of all vaccines administered to all 

adolescent residents within the state. It is known that different state IIS vary in their completeness of both 

adolescents and the doses they received. https://www.cdc.gov/vaccines/programs/iis/about.html
Error Profile for the 2022 NIS-Teen 6

NCIRD provides competitive supplemental funds to grantees that have achieved consistently high 

standards. During the period 2013-2017, six grantees were recognized as sentinel sites, including 

Michigan, Minnesota, North Dakota, New York City, Oregon, and Wisconsin.3 Because of the higher 

standards they achieved regarding participation rates and other indicators of IIS quality, vaccination 

coverage estimates reported in IISAR by sentinel sites are thought to be relatively more accurate than 

vaccination estimates reported by non-sentinel sites. In this analysis, the sentinel sites are mentioned 

as 2013-2017 sentinel sites. 

In what follows, we compare 2021 NIS-Teen vaccination estimates to 2021 IISAR vaccination 

estimates. Because 2022 IISAR vaccination coverage estimates are not available as of this writing, the 

2021 comparison will serve as the most current information available about the relative accuracy of the 

2022 NIS-Teen. 

Visual Comparison of Vaccination Coverage Estimates

Figures 2.1 and 2.2 display plots of the NIS-Teen vaccination coverage estimate versus the IISAR 

vaccination coverage estimate for 1+ Tdap for the year 2021. Figure 2.1 includes only 56 of the 

estimation areas used in the NIS-Teen; it does not include points corresponding to 8 U.S. territories. 

Figure 2.2 includes only the six 2013-2017 sentinel sites. Some of the areas are missing from some of 

the plots due to missing values in the IISAR. IISAR vaccination coverage estimates use an IIS count of 

vaccinated adolescents as the numerator but a U.S. Census count of adolescents living in the area as 

the denominator; this can result in some IISAR vaccination rates being greater than 100 percent, for 

example if adolescents in the IIS have moved away from the area. 

In all plots, the straight line through the origin reflects the yy = xx line. Points above the line represent 

areas in which the NIS-Teen vaccination coverage estimate is greater than the IISAR estimate, and 

points below the line represent areas in which the IISAR estimate is greater. The line itself represents 

complete agreement between the estimates. In addition, the color and symbol of each point signifies 

the magnitude of the difference between the NIS-Teen and IISAR rates. 

The plots reveal that in some areas the NIS-Teen vaccination coverage estimates are greater than the 

IISAR estimates and vice versa. There is reasonably good agreement between the two estimates in five 

of the six 2013-2017 sentinel sites; in New York City, the IISAR estimate is much greater than 100 

percent, possibly because the IIS contains many adolescents not currently living in New York City. 

Generally, IISAR vaccination coverage estimates tend to be lower in non-sentinel areas. For IIS that 

achieve high standards, NIS-Teen and IISAR vaccination estimates are reasonably similar.

3 https://www.cdc.gov/vaccines/programs/iis/activities/sentinel-sites.html
Error Profile for the 2022 NIS-Teen 7

Figure 2.1: Scatterplot of NIS-Teen (in %) v. IISAR (in %) Vaccination Coverage Estimates for 1+ 

Tdap: 56 Estimation Areas, 2021 

Figure 2.2: Scatterplot of NIS-Teen (in %) v. IISAR (in %) Vaccination Coverage Estimates for 1+ 

Tdap: Six 2013-2017 Sentinel Sites, 2021 
Error Profile for the 2022 NIS-Teen 8

Adolescent Participation Rate (APR)

To test the hypothesis that increasing quality of a state’s IIS data implies increasing agreement 

between the NIS-Teen and IISAR vaccination estimates, we introduce the APR4 and evaluate it as a 

possible measure of the quality of the IIS. In Figure 2.3, we plot the IISAR vaccination coverage 

estimate for 1+ Tdap versus the APR for the year 2021, including the 56 core estimation areas. Points 

for the six 2013-2017 sentinel sites are labeled, and these sites are generally located in the upper right 

corner of the plot, corresponding to higher values of APR and the IISAR vaccination estimate. Note that 

the IISAR APR use an IIS count of vaccinated adolescents as the numerator but a U.S. Census count 

of adolescents living in the area as the denominator; this can result in some IISAR vaccination rates 

being greater than 100 percent.

Figure 2.3: Scatterplot of IISAR (in %) Vaccination Coverage Estimate for 1+ Tdap v. APR (in %): 56 

Estimation Areas, 2021 

We fit a linear regression model to the points in Figure 2.3, and the corresponding fit is represented by 

the solid line depicted in the figure. The association of APR with the IISAR vaccination coverage 

estimate is positive and strongly statistically significant. The Pearson correlation is 0.656 with a 95% 

confidence interval of [0.475, 0.784]. We conclude that APR is positively associated with the 

4 Proportion of adolescents in the area who have two or more doses of any vaccine recorded in the IIS relative to a U.S. 

Census Bureau count of adolescents living in the area. This can result in some APR values being greater than 100 percent. 
Error Profile for the 2022 NIS-Teen 9

completeness of the vaccination histories of the adolescents. APR appears to be a reasonable, but not 

necessarily comprehensive, measure of the quality of data in the IIS databases. 

Negative Relationship between Difference in Vaccination Coverage and APR

We conducted further evaluation of the hypothesis that increasing quality of state IIS data is associated 

with increasing agreement between the NIS-Teen and IISAR vaccination coverage estimates. 

Specifically, taking APR to be the measure of IIS quality, we calculated the difference between the 1+ 

Tdap coverage estimates in NIS-Teen and IISAR and fit a simple linear regression model relating the 

difference to the APR. Figure 2.4 presents the scatter plot of the difference versus the APR for the set 

of 56 core estimation areas in 2021, and the straight line depicted in the figure is the regression line. 

The APR has a strong and statistically significant relationship with the difference. The coefficient on the 

APR is negative (-0.99 percentage points with a 95% confidence interval of [-1.32, -0.66]), which 

implies that the difference declines with increasing APR, or in other words, the difference between NISTeen and IISAR vaccination coverage estimates declines as IIS quality increases. As the APR, as an 

indicator of IIS data quality, increases, IISAR vaccination estimates tend to converge towards NIS-Teen 

vaccination estimates, thus potentially supporting the accuracy of the NIS-Teen vaccination coverage 

estimates.

Figure 2.4: Scatterplot of the Difference (NIS-Teen minus IISAR) (in %) v. APR (in %) with Regression 

Line: 56 Estimation Areas, 2021 
Error Profile for the 2022 NIS-Teen 10

We conducted an additional analysis pooling the data over the period 2016 to 2021 to achieve greater 

precision and regressed the difference between NIS-Teen and IISAR vaccination coverage estimates 

on APR, dummy variables for year, and state dummy variables. Similar to the 2021 only results in 

Figure 2.4, the pooled results showed that APR has a statistically significant, negative relationship with 

the difference, with a regression slope coefficient of -0.90 percentage points with a 95% confidence 

interval of [-0.98, -0.82]. 

Summarizing, we have presented evidence in this section that APR is a reasonable, though not 

comprehensive, measure of the quality of IIS adolescent data. As state IIS data quality indicators 

improve, the difference between the NIS-Teen and IISAR vaccination coverage estimates declines. 

Because NIS-Teen uses a consistent methodology in all areas, it may be reasonable to extrapolate this 

finding and suggest that IISAR and NIS-Teen vaccination coverage estimates would tend to agree in all 

areas, if the IIS was of consistently high quality in all areas.

2.3 Comparison of Adolescent by Type of Health Insurance 

Coverage 

In this subsection, we compare NIS-Teen health insurance estimates to those from the ACS, the CPS 

ASEC, and the NHIS. We discuss the percentages of adolescents with any private insurance coverage, 

any public insurance coverage, and no insurance coverage with a comparison in Table 2.2. Before 

reviewing the results of the table, we provide an overview of each data source. 

Table 2.2: Comparison of Alternative Estimates of Health Insurance Coverage Among Adolescents: 

NIS-Teen, ACS, CPS ASEC, and NHIS for 2021 and NIS-Teen and CPS ASEC for 2022 

Type of 

Health 

Insurance 

Coverage

2021 2022 

ACS CPS ASEC NHIS NIS-Teena CPS ASEC NIS-Teena

Any privateb 61.3% 63.5% 56.0% 65.3% 63.1% 62.8%

Any publicc 38.8% 34.8% 41.4% 41.0% 36.5% 44.1%

Uninsuredd 5.8% 6.7% 4.8% 3.0% 5.5% 3.4%

a NIS-Teen estimates were produced among adolescents with adequate provider data using the final NIS-Teen 

survey weights, which are adjusted for noncoverage and nonresponse and calibrated to demographic 

population control totals.

b Private: Includes coverage provided through an employer or union or purchased directly from an insurance 

company that helps pay for both doctor visits and hospital stays.

c Public: Includes Medicaid, CHIP, Indian Health Service, TRICARE, CHAMPUS and CHAMP-VA.

d Uninsured: Adolescents are defined as uninsured if they do not have private insurance that helps pay for 

both doctor visits and hospital stays and do not have any other form of health insurance.
Error Profile for the 2022 NIS-Teen 11

Conducted by the U.S. Census Bureau, the ACS is an ongoing survey that provides essential 

information about the population of the United States on an annual basis, including statistics related to 

social, housing, economic, and demographic characteristics of the population. Estimates in Table 2.2 

contain information on health insurance status by the adolescent population aged 13 to 17 years based 

on the 2021 ACS5 at the national level, the most recently available ACS data as of this writing. ACS 

interviews are conducted throughout the calendar year, and the ACS instrument assesses health 

insurance status as of the date of the interview. 

CPS ASEC is conducted in March of every year. While CPS is a monthly household survey conducted 

by the U.S. Census Bureau and the Bureau of Labor Statistics and designed mainly for measuring 

employment and unemployment, CPS ASEC provides additional detailed statistics related to household 

income, poverty, health insurance status, and other topics. The CPS ASEC asks current health 

insurance coverage status as of the time of the interview. Based on data from the March 2021 and 

2022 CPS ASEC6, national-level estimates of the health insurance distribution in 2021 and 2022 among 

adolescents aged 13 to 17 years are shown in Table 2.2.

The NHIS is a cross-sectional household interview survey, conducted by the National Center for Health 

Statistics, that covers the civilian noninstitutionalized population in the United States. The objective of 

the NHIS is to monitor the health status of the U.S. population. In addition to collecting variables related 

to health status, the survey collects many demographic and socioeconomic characteristics of 

household members. NHIS7 national-level estimates of the health insurance distribution in 2021 for 

adolescents aged 13 to 17 years are shown in Table 2.2, as 2021 is the most recently available NHIS 

data. In the NHIS, health insurance status is assessed as of the time of the interview.

In reviewing Table 2.2, we find NIS-Teen estimates, which are also based on health insurance status 

as of the date of the interview, to be larger than those from the ACS, CPS ASEC, and NHIS for the 

privately-insured population for 2021 and similar to the CPS ASEC estimate for 2022. We find the 

estimates for public health insurance in NIS-Teen to be larger than the corresponding estimates from 

ACS and CPS ASEC in 2021, about the same as the NHIS for 2021, and larger than the CPS ASEC 

estimate for 2022. Finally, we find NIS-Teen estimates of the size of the uninsured population are less

than corresponding estimates from ACS, CPS ASEC, and NHIS in 2021 and also less than the CPS 

ASEC estimate for 2022. The differences in estimates between the NIS-Teen and the other three 

sources could be due to differential error in the NIS-Teen relative to the other sources (due to 

differential sample-frame coverage error, nonresponse error, or measurement error), or to definitional 

differences (questionnaire differences) in how health insurance status is measured.

5 US Census Bureau. 2021 ACS PUMS Data. Retrieved from https://www.census.gov/programssurveys/acs/microdata/access/2021.html

6 US Census Bureau. 2021 and 2022 CPS ASEC dataset. Retrieved from https://www.census.gov/data/datasets/timeseries/demo/cps/cps-asec.html

7 NHIS - Data, Questionnaires and Related Documentation. Retrieved from https://www.cdc.gov/nchs/nhis/dataquestionnaires-documentation.htm
Error Profile for the 2022 NIS-Teen 12

2.4 Comparison of Vaccination Coverage Estimates: NIS-Teen 

v. State Immunization Surveys

In this subsection, we compare vaccination coverage estimates from NIS-Teen and state vaccination 

surveys. Agreement between the estimates signals consistency between the two sources and may 

indicate the accuracy of the sources. Disagreement between the estimates signals inconsistency and 

that at least one of the sources may be inaccurate.

The objective here is to compare NIS-Teen to state-sponsored surveys that are independent of the 

corresponding IIS. We are aware of only two states that have conducted an immunization survey of its 

resident adolescents: Georgia (GA) and Kansas (KS). Thus, we now compare NIS-Teen vaccination 

coverage rates for Georgia to those from the Georgia Adolescent Immunization Study (henceforth 

abbreviated GIS), which is sponsored and conducted by Georgia Department of Public Health 

(Machado, 2017 and 2018), and NIS-Teen vaccination coverage rates for Kansas to those from an 

analysis of Kansas BRFSS (Behavioral Risk Factor Surveillance System) survey data (henceforth 

abbreviated KIS), conducted by Kansas Department of Health and Environment (Gillespie, 2018). This 

report focuses on the most recently available data from these surveys, 2011 to 2016 for GIS and 2017 

to 2018 for KIS.

Each annual GIS is based on a two-stage sample of seventh grade students, with vaccination history 

information collected from the Georgia Registry of Immunization Transactions and Services. In the first 

stage, up to thirty middle schools (public and private) are selected from each of eighteen state health 

districts, while in the second stage, a random sample of students is selected from within each selected 

school. Each KIS is based on a dual-frame random digit dial (RDD) survey of noninstitutionalized 

adults, 18 years and older. Households are screened for the presence of children aged 17 or younger 

in the household, and one eligible child is randomly selected for households with multiple children. 

From 2011 to 2016, parents and guardians were asked about adolescent vaccinations and vaccination 

histories. Published estimates include children younger than 13, whereas NIS-Teen is based on 

adolescents aged 13 to 17 identified in an RDD telephone survey of Georgia and Kansas households, 

with provider reporting of vaccination histories. Table 2.3 gives the sample sizes for the surveys. KIS 

and especially GIS are much larger in sample size than NIS-Teen.
Error Profile for the 2022 NIS-Teen 13

Table 2.3: Sample Sizes of Adolescents in Georgia and Kansas

Year State Vaccination 

Surveys

NIS-Teen in State, Adolescents 

with Adequate Provider Data

Georgia

2017 6,191 299

2018 7,057 345

Kansas

2011 971 359

2012 568 297

2013 1,280 205

2014 641 234

2015 1,073 228

2016 670 207

Sources:

https://dph.georgia.gov/sites/dph.georgia.gov/files/Adolescent%20Immu

nization%20Study%202017%20Final%20Report_Georgia.pdf

https://dph.georgia.gov/sites/dph.georgia.gov/files/Immunizations/2018%

20GAIS%20final_web.pdf

https://www.kansaspharmacyfoundation.org/pdf/vaccines/vaccinationrates-Adolescent-Coverage-2016.pdf

Figures 2.5a and 2.5b present a comparison of GIS and NIS-Teen vaccination coverage estimates for 

the two most recently available years: 2017 and 2018. Differences for 1+ Tdap are small to moderate, 

with the difference for 2017 statistically significant (pp = 0.05). The differences for 1+ MenACWY are 

small, 1.1 percentage points and 0.7 percentage points in 2017 and 2018, respectively. The largest 

differences are for 1+ HPV and UTD for HPV. For males and females combined, NIS-Teen estimates 

for 1+ HPV are 16.4 and 17.5 percentage points greater than GIS estimates, in 2017 and 2018, 

respectively. Similarly, for males and females analyzed separately, NIS-Teen estimates for 1+ HPV are 

much larger than GIS estimates. All differences for 1+ HPV are highly statistically significant.

Differences may be due to the inclusion of 12-year-olds for the GIS who are less likely to have had their 

first dose of HPV vaccine.
Error Profile for the 2022 NIS-Teen 14

Figure 2.5a: Comparison of Vaccination Coverage Estimates (in %): GIS v. NIS-Teen in GA, 

2017 and 2018

Figure 2.5b: Comparison of 1+HPV Coverage Estimates (in %) by Sex: GIS v. NIS-Teen in GA, 

2017 and 2018

96.6 96.4

47.9

23.3

95.7 95.5

50.6

23.4

93.3 95.3

64.3

45.7

94.2 94.8

68.1

49.6

0

10

20

30

40

50

60

70

80

90

100

1+Tdap 1+MenACWY 1+HPV Complete HPV series

GIS 2017 (7th Grade Adolescents) GIS 2018 (7th Grade Adolescents)

NIS-Teen 2017 (Adolescents aged 13-17) NIS-Teen 2018 (Adolescents aged 13-17)

0

10

20

30

40

50

60

70

80

90

100

2017 2018 2017 2018

1+HPV females 1+HPV males

GIS (7th Grade Adolescents) NIS-Teen (Adolescents aged 13-17)
Error Profile for the 2022 NIS-Teen 15

The comparison of NIS-Teen and KIS involves multiple years. The KIS assessed Tdap vaccination 

coverage in 2011 and 2014, 1+ MenACWY coverage in 2012 and 2015, and HPV coverage in 2013 

and 2016. In Figures 2.6a and 2.6b, we compare KIS and NIS-Teen vaccination coverage estimates for 

these specific years. The smallest differences are 1.1 and 1.2 percentage points for 1+ Tdap in 2011 

and 2014, respectively. The difference for 1+ MenACWY in 2012, -1.3 percentage points, is also small, 

but the difference in 2015, 11.5 percentage points, is larger and statistically significant (p=0.00). NISTeen estimates of HPV vaccination coverage are much larger than the corresponding KIS estimates. In 

2016, the difference in estimates for 1+ HPV is -22.3 percentage points for males and females 

combined, -28.7 percentage points for females, and -15.9 percentage points for males, all of which are 

highly statistically significant. These differences may be due to (1) KIS estimates being based on 

parental reports while NIS-Teen estimates are based on provider vaccination histories, (2) differential 

nonresponse error for the KIS and NIS-Teen, and (3) the KIS including children younger than 13

whereas NIS-Teen estimates are for 13- to 17-year-olds.

Figure 2.6a: Comparison of Vaccination Coverage Estimates (in %): KIS v. NIS-Teen in KS, 

Various Years 

80.2 81.0

54.6 52.2

19.6

10.1

29.5

14.9

79.1 79.8

55.9

63.7

32.3

20.0

51.8

35.6

0

10

20

30

40

50

60

70

80

90

100

1+Tdap

2011

1+Tdap

2014

1+MenACWY

2012

1+MenACWY

2015

1+HPV

2013

Complete

HPV series

2013

1+HPV

2016

Complete

HPV series

2016

KIS NIS-Teen 2017 (Adolescents aged 13-17)
Error Profile for the 2022 NIS-Teen 16

Figure 2.6b: Comparison of 1+HPV Coverage Estimates (in %) by Sex: KIS v. NIS-Teen in KS, 2016

Comparison to Other IIS-Based Vaccination Coverage Estimates

Additional comparisons of estimates of vaccination coverage from state IIS to estimates from NIS-Teen

have been studied. First, Staras et al. (2021) examined estimates of vaccination coverage for 13- to 17-

year-olds with 1+ doses of HPV vaccine as well as for UTD for HPV vaccination as determined from 

Florida’s IIS, known as the State Health Online Tracking System (SHOTS). They compared these

estimates to those from NIS-Teen from 2009 to 2018. The authors find that the estimates from Florida 

SHOTS and NIS-Teen are similar for males, while estimates from Florida SHOTS are higher than NISTeen for females for 1+ doses of HPV vaccination. The findings are consistent with the range of levels 

of agreement found between IISAR and NIS-Teen estimates of vaccination coverage across different 

estimation areas described in Section 2.2. 

Second, we compared state-level estimates for UTD for HPV from the Wisconsin Immunization 

Registration (WIR) as reported by Weatherer et al. (2021) to corresponding estimates from NIS-Teen.

The estimates from NIS-Teen were higher than those derived from WIR. The 2019 Wisconsin NIS-Teen 

UTD for HPV estimate among 13- to 17-year-old adolescents was 56.0% with a 95% margin of error of 

9.0 percentage points, compared with 45.6% reported from WIR. The 2016 NIS-Teen estimate was 

45.5% with a 6.5 percentage point margin of error, compared with 34.0% from WIR. It is possible that 

these differences reflect under-reporting of HPV vaccination coverage for WIR for these years. 

33.7

25.8

62.4

41.7

0

10

20

30

40

50

60

70

80

90

100

1+HPV females

2016

1+HPV males

2016

KIS NIS-Teen 2016 (Adolescents aged 13-17)
Error Profile for the 2022 NIS-Teen 17

In summary, research on estimates from Florida and Wisconsin IIS vaccination coverage estimates is 

consistent with other evidence on rates of agreement between IIS and NIS-Teen estimates, and some 

differences may be due to under-reporting of vaccination coverage in IIS data. Additionally, state survey 

estimates of vaccination coverage from Georgia (via the GIS) and Kansas (via the KIS) exhibit 

reasonable agreement with NIS-Teen vaccination coverage estimates for 1+ Tdap and 1+ MenACWY. 

However, NIS-Teen HPV vaccination coverage estimates are much higher than the corresponding 

estimates from GIS and KIS, likely because GIS assesses younger adolescents (seventh grade 

students tend to be 12 years old, before the main uptake of HPV vaccine, and below the NIS-Teen 

eligible age range that better represents the main uptake). Additionally, the KIS assessment is based 

on parental recall, known to be less accurate than provider reporting, and includes younger children. 

KIS and GIS may have differential nonresponse error relative to NIS-Teen. After making allowance for 

the known differences in survey methods, it is not unreasonable to conclude that both GIS and NISTeen for Georgia and KIS and NIS-Teen for Kansas tend towards consistent measurements.

3. Part II: Assessment of Total Survey Error for 

NIS-Teen Vaccination Coverage Estimates 

In this part of the report, we assess the total survey error in NIS-Teen vaccination coverage estimates 

using the framework developed and implemented in Molinari et al. (2011) and Wolter et al. (2017). We 

decompose TSE into components of sampling and nonsampling error, and then assemble the best 

information available about the magnitude of each component error from specialized evaluation studies.

We view each component error as a random variable subject to a conditional distribution, given the 

outcome of the NIS-Teen. The mean of the conditional distribution is estimated from numerical 

evidence obtained in the corresponding evaluation study. The variance of the distribution, reflecting 

both variability in the evaluation survey samples and other uncertainties in our knowledge about the 

component error, is estimated from internal evidence within the evaluation study and possibly additional 

professional judgment. After assembling the best available information about each of the component 

errors, we combine the information to produce a total survey error distribution, using a Monte Carlo 

method. 

Before proceeding to consider the component errors, we introduce some notation that will be helpful in 

this section. Let μμ0 denote the true but unknown vaccination coverage in the age-eligible population of 

adolescents, and let μμ̂denote the NIS-Teen estimate of vaccination coverage. The TSE in the 

vaccination coverage estimate is then given by

(1)
Error Profile for the 2022 NIS-Teen 18

We use a three-stage model for TSE, where Stage 1 represents error due to the sampling-frame’s 

under-coverage of the population of age-eligible adolescents, Stage 2 represents error due to 

nonresponse among sampled units, and Stage 3 represents measurement error among the responding 

units. The model for the first stage (sampling-frame coverage) is

(2)

where μμ1is the true vaccination coverage for the age-eligible adolescents covered by the sampling 

frame, μμ1AA is the true vaccination coverage for the age-eligible adolescents not covered by the 

sampling frame, and pp1AA is the proportion of the age-eligible population not covered by the sampling 

frame. The model at the second stage (response) is

(3)

where μμ2 is the true vaccination coverage for adolescents who respond to NIS-Teen, μμ2AA is the true 

vaccination coverage for adolescents who do not respond, and pp2AA is the proportion of adolescents who

do not respond. Finally, the model at the third stage (measurement) is

(4)

where μμ3 is the true vaccination coverage of adolescents for whom accurate response is given to the 

survey, μμ3AA is the true vaccination coverage of adolescents for whom inaccurate response is given to 

the survey, and pp3AA is the proportion of adolescents for whom inaccurate response is given. 

Combining all three stages together, the true vaccination coverage can be written as

(5)

We can also write the TSE as 

(6)

where is the error due to noncoverage, is the error due to nonresponse, and 

is the error due to inaccurate reporting by survey respondents.
Error Profile for the 2022 NIS-Teen 19

The seven parameters on the right side of (5) are . Estimates of the 

values of these seven parameters, based on the analyses to be presented in Sections 3.2 through 3.4 

below, are denoted by . Let denote the estimated variancecovariance matrix of φφ. We assume the seven parameters are independently distributed and that

, where each is our estimate of the variance of the 

corresponding parameter. 

We assume our knowledge about the true parameters, φφ, can be acceptably represented by a 

probability distribution, with parameters and . We assess TSE by making random draws of φφ and ΣΣ

from their distributions. For each draw, we use equation (5) to produce a draw from the distribution of 

the true vaccination coverage in the overall age-eligible population (say, μμ0

∗) and we compute 

qq∗ = μμ̂ − μμ0

∗ as a draw from the distribution of TSE. We obtain the distribution of TSE using 10,000 such 

draws.

Having established our model and notation, we now consider sampling-frame coverage error in the 

NIS-Teen, which arises because the sampling frame omits direct representation of the landline-only 

(LLO) and phoneless populations. Second, we consider nonresponse error in the NIS-Teen, which 

comes about due to nonresponse in the random digit dial (RDD) telephone survey of households, to 

failure of the parental respondent to give consent to contact the adolescent’s immunization providers,

and to missing vaccination histories in the provider record check given consent. Third, we consider 

response or measurement error in the provider reporting of vaccination histories. This component of 

error has also been referred to as under-ascertainment of vaccination histories. Fourth, we consider 

error in the NIS-Teen due to sampling, i.e., error because the survey observes only about 1 out of 

1,300 adolescents in the age-eligible population. Fifth, we combine the foregoing component error 

distributions, resulting in the TSE distribution of the vaccination coverage estimate for the 2022 

NIS-Teen. Finally, we close this section by examining the change in the TSE, from the 2021 NIS-Teen

to the 2022 NIS-Teen, using the bridging cohort method, first described in Yankey, Hill, Elam-Evans

et al. (2015).

3.1 Sampling-Frame Coverage Error

Sampling-frame coverage errors arise in a survey when the sampling frame does not include the entire 

target population. In 2018, the NIS-Teen began using a single-frame cell-phone RDD design, which 

omits direct representation of adolescents in LLO and phoneless households. To account for the 

excluded population groups, the NIS-Teen weighting methodology makes adjustment to the weights by 

raking the weights to select demographic characteristics of the population of adolescents 13-17 years. 

The assumption embedded in this procedure is that, after controlling for these characteristics, the 

vaccination coverage in the population not represented on the sampling frame equals the coverage in 

the population represented on the frame. However, it is possible that estimated vaccination coverage of 
Error Profile for the 2022 NIS-Teen 20

adolescents in the omitted domains differ from the rates of adolescents in the included domains, which 

may introduce bias into the estimator of the vaccination coverage. 

In this section, we attempt to measure the bias in the estimated vaccination coverage introduced by 

sampling-frame coverage error. Table 3.1 displays the proportion of adolescents 13-17 years in the 

population by telephone status by year for 2012-2021 based on estimates from the NHIS, as 2021 is 

the most recently available year of NHIS data. The proportion of adolescents with cell-phone-only 

(CPO) status is increasing throughout this period and the proportion with dual-user status is 

decreasing. The estimated proportion in cell-phone households (i.e., CPO and dual-user combined), 

was relatively steady until 2018, before increasing from 95.5% in 2018, 98.1% in 2019, 98.6% in 2020, 

and 99.2% in 2021. As a result, the estimated proportion of uncovered households decreased from 

4.5% in 2018 to 0.8% in 2021. 

Table 3.1: Percentage of Age-Eligible Adolescents in the 

Population by Telephone Status by Year: NIS-Teen, United States, 

2012-2021 

Year Cell-PhoneOnlyDual-User LandlineOnlyPhoneless

2012 36.2 58.6 3.5 1.7

2013 40.5 55.3 2.4 1.8

2014 44.8 50.5 3.0 1.7

2015 49.3 47.0 2.1 1.7

2016 55.2 41.2 1.9 1.7

2017 55.7 40.4 1.7 2.2

2018 56.9 38.5 2.2 2.3

2019 63.2 34.8 0.8 1.1

2020 65.0 33.7 0.7 0.7

2021 72.3 27.0 0.1 0.6

Source: Produced using the methods of Blumberg, Ganesh, Luke, and 

Gonzales (2013) applied to data from the 2012-2021 National Health 

Interview Survey sponsored by CDC’s National Center for Health Statistics 

(https://www.cdc.gov/nchs/nhis/index.htm).

The 2022 NIS-Teen did not directly measure LLO or phoneless adolescents, and to assess vaccination 

coverage estimates in these domains and determine whether they differ from estimates in the 

combined cell-phone domain, we must turn to other sources. Specifically, the 2012-2017 NIS-Teen 

samples directly represented LLO adolescents and thus permit comparison of their vaccination 

coverage estimates to the corresponding estimates of adolescents in cell-phone households. Table 3.2 

displays the vaccination coverage estimates for 2017, the closest such year to 2022. We observe that 

vaccination coverage estimates are generally higher in the cell-phone domain than in the LLO domain, 

yet none of the differences are statistically significant.
Error Profile for the 2022 NIS-Teen 21

Table 3.2: Vaccination Coverage Estimates and Standard Errors of Select Vaccines and 

Vaccine Series for the Cell-Phone and LLO Domains: NIS-Teen, United States, 2017

Cell-Phone Domain Landline-Only Domain Difference

Variance/Series Estimate Standard 

Error

Estimate Standard 

Error

Estimate Standard 

Error

1+ Tdap 88.7 0.46 86.9 3.79 1.9 3.8

1+ MenACWY 85.3 0.48 73.4 6.69 11.8 6.7

UTD for HPV 48.7 0.66 40.3 7.29 8.5 7.3

UTD for HPV 

among females 53.1 0.97 55.0 9.70 -1.9 9.8

UTD for HPV 

among males 44.5 0.88 31.6 10.13 12.8 10.2

∗ pp ≤ 0.05.

Since NIS-Teen has never included direct sampling of phoneless adolescents, we study vaccination 

estimates in the phoneless domain using estimates from the 2012 National Health Interview SurveyProvider Record Check (NHIS-PRC).8 Table 3.3 shows the vaccination coverage estimates for select 

vaccines and vaccine series for adolescents from the cell-phone domain versus those who are in the 

phoneless domain. Caution must be taken due to the very small sample size in the phoneless domain, 

but estimated differences in vaccination coverage estimates between the cell-phone domain and the 

phoneless domain are small relative to the standard errors of the differences.

Table 3.3: Vaccination Coverage Estimates and Standard Errors of Select Vaccines and 

Vaccine Series for Adolescents 13-17 Years in the Cell-Phone and Phoneless Domains: 

NHIS-PRC, United States, 2012

Cell-Phone Domain Phoneless Domain Difference

Vaccine Estimate Standard 

Error Estimate Standard Error Estimate Standard Error

1+ Tdap 80.5 0.84 80.9 10.92 -0.4 11.0

1+ MenACWY 71.8 0.98 77.5 10.10 -5.7 10.1

1+ HPV among 

femalesa 51.7 1.49 41.3 13.06 10.4 13.1

∗ pp ≤ 0.05. a Estimates for HPV vaccination with respect to the current recommendations are not available from the 

NHIS-PRC, and are not available for males or for the overall adolescent population; the estimates for 

HPV in table can be presented only for 1+ HPV vaccination among females.

8 2012 was the last, and therefore most recent, year for which the NHIS-PRC was conducted, and thus for which a direct 

measurement was obtained of the vaccination status of phoneless adolescents.
Error Profile for the 2022 NIS-Teen 22

The foregoing tables can be translated into an assessment of sampling-frame coverage error in 2022 

NIS-Teen estimated vaccination coverage. We can also write the true vaccination coverage estimate as 

(7)

where qq1 = pp1AA(μμ1 − μμ1AA) equates to sampling-frame coverage error. To fully assess the distribution of 

total survey error in the NIS-Teen, we will require estimates of the parameters, pp̂1AA, μμ̂1AA, and μμ̂1, and 

their standard errors, which we will present in Section 3.5. Here we simply observe that pp̂1AA will be 

obtained from the landline-only and phoneless columns on the right side of Table 3.1 for 2021 (the most 

recent year available), and μμ̂1AA will be obtained from the results of the 2022 NIS-Teen and the 

Difference columns on the right side of Tables 3.2 and 3.3. The estimate of vaccination coverage in the 

sampling-frame covered population, μμ̂1, will be obtained from the results of the 2022 NIS-Teen and 

from work we will do in Sections 3.2 and 3.3 on nonresponse error and measurement error.

As a preliminary assessment of the effect of sampling-frame coverage error, we can estimate the true 

vaccination coverage estimate ignoring the effects of nonresponse and measurement error. In this 

circumstance, the NIS-Child vaccination estimate is essentially μμ1. Then, Table 3.4 presents estimates 

of qq1 and of the true vaccination estimate, μμ0. For all five vaccination coverage estimates, we estimate 

that sampling-frame coverage error is 0.1 percentage points or less and the estimated error is less than 

the standard error of μμ0. 

Table 3.4: Preliminary Assessment of Sampling-Frame Coverage Error and Mean 

True Vaccination Coverage Estimate (in %): NIS-Teen, United States, 2022 

Vaccine

(2022 

NIS-Teen 

Vaccination 

Coverage 

Estimate)

a

(Mean of ,

the True 2022 

Vaccination 

Coverage) 

Standard 

Error of 

1+ Tdap 89.9 0.0 89.9 0.5

1+ MenACWY 88.6 0.0 88.6 0.5

UTD for HPV 62.6 0.1 62.5 0.7

UTD for HPV among females 64.6 0.1 64.6 1.0

UTD for HPV among males 60.6 0.1 60.5 1.0

a The estimated sampling-frame coverage error, , is obtained by combining information in 

Table 3.2 about the landline-only population in 2017 with information in Table 3.3 about the 

phoneless population in 2012.
Error Profile for the 2022 NIS-Teen 23

3.2 Nonresponse Error 

There are two types of nonresponse error impacting NIS-Teen, unit nonresponse error due to not 

obtaining responses (or completed interviews) for all adolescents sampled and item nonresponse error 

due to missing questionnaire items among survey respondents. This section focuses on assessing 

survey error due to unit nonresponse. We conclude with a review of 2022 NIS-Teen item nonresponse 

rates. NIS-Teen vaccination coverage estimates are based on provider-reported vaccination histories; 

incomplete (or missing vaccination) information on these histories is a form of measurement error or 

under-reporting error, which is assessed in Section 3.3. 

Components of Nonresponse in NIS-Teen

Unit nonresponse error in NIS-Teen estimates of vaccination coverage is the error arising because

completed interviews are not obtained for all adolescents sampled. Unit nonresponse arises at four 

steps in the survey process, as follows: (1) failure to resolve the selected telephone number as an 

occupied household or some other known entity, (2) failure to screen the household for the presence of 

an age-eligible adolescent, (3) failure to complete the telephone interview of an eligible household, and 

(4) failure to obtain consent to contact the adolescent’s vaccination providers or failure to obtain 

sufficient information from providers to determine the adolescent’s vaccination status, given consent. 

We do not observe the vaccination statuses of adolescents for whom either the household interview is 

missing or the provider record check is missing. This section assesses the extent of nonresponse error 

in the 2022 NIS-Teen estimates of vaccination coverage for three different vaccines, including 1+ Tdap, 

1+ MenACWY, and UTD for HPV, with HPV analyzed for the overall population of adolescents and 

separately for males and females.

Weight Adjustment for Nonresponse Error

NIS-Teen addresses error due to nonresponse by using weight adjustments that correct for known 

differences between adolescents in responding and nonresponding households based on observable 

characteristics. Specifically, weighting cells are defined based on sample frame information known for 

both respondents and nonrespondents, and weights are adjusted by a factor inversely proportional to 

the response rate within each cell. Calibration of the weights to demographic population totals also 

serves to adjust for differences between the responding sample and the population. The NIS-Teen

weighting methodology is described in detail in Wolter, Smith, Khare et al. (2017).

The weighting adjustment method assumes that nonresponse is a missing at random process (Rubin 

1976), or that the conditional distribution of vaccination coverage on the characteristics used to form the 

weighting cells and calibration dimensions is the same whether or not the data are missing. This 

assumption, while widely used for weighting adjustments, is generally untestable since we do not 

observe vaccinations statuses for the nonrespondents. Thus, further methods are needed to assess the 

extent of nonresponse error after conducting weighting adjustments. 
Error Profile for the 2022 NIS-Teen 24

Assessment of Nonresponse Error

To inform our TSE models, an estimate of the proportion of adolescents with adequate provider data 

among adolescents in households corresponding to the sampled telephone numbers is needed. The 

2022 NIS-Teen realization rate9 of adolescents with adequate provider data was 2.8 percent with a 

standard error of 0.03 percentage points. Dividing the realization rate by the sampling-frame coverage 

rate estimated in Section 3.1 yields an estimate of the proportion of adolescents with adequate provider 

data among those covered by the sampling frame of also 2.8 percent, or 97.2 percent without adequate 

provider data, with a standard error of 0.2 percentage points. These two numbers (97.2% with a 

standard error of 0.2 percentage points) serve as model inputs pp̂2AA and for the TSE analysis.

We now assess the extent of nonresponse error both before conducting nonresponse weighting 

adjustments as well as the residual error after accounting for such adjustments. It is common in TSE 

analyses to compare estimates derived from the survey under study, NIS-Teen in this instance, with 

those from leading reference surveys (Biemer, 2010). A reasonable benchmark for the 2022 NIS-Teen 

is data available from the 2021 NHIS, because it provides representation of the same population of 

adolescents as the NIS-Teen, is known to be a premier health survey of the general population in the 

United States, is conducted using face-to-face interviewing methods, and has a relatively high response 

rate, with the final response rate for the Sample Child component in 2021 being 49.9%.10 

Comparing NIS-Teen estimates to those based on the 2021 NHIS enables estimation of nonresponse 

error. If nonresponse error is minimal in the NHIS, then the comparison to the NIS-Teen can be taken 

as a measure of nonresponse error in the NIS-Teen. To ensure comparability to the population covered 

by the NIS-Teen, we examine NHIS adolescents who are in the corresponding age range and have a 

working cell-phone in the family.

While the NHIS does not produce direct estimates of vaccination coverage, we can compare indirect 

vaccination coverage estimates derived from the NHIS to the direct vaccination coverage estimates

derived from the NIS-Teen. We take advantage of the range of variables that are common to both the 

2022 NIS-Teen and the 2021 NHIS and produce estimates of nonresponse error for the following 

vaccination series: 1+ Tdap, 1+ MenACWY, and UTD for HPV. Specifically, we estimate logistic 

regression models for vaccination statuses in the NIS-Teen with variables common to both surveys as 

independent variables. We then use the fitted models to produce multiple imputations of vaccination 

statuses for the NHIS case set, using the 2021 NHIS Public Use File (CDC, 2022). Then, we estimate 

vaccination coverage using the NHIS data after pooling the survey-weighted estimates across the 

multiply-imputed datasets. We treat the estimates based on imputations of the NHIS as the true 

9 The realization rate (Skalland, 2011) is calculated as the ratio of the unadjusted survey estimate of the size of the target 

population to an external estimate of the true size of the target population, and can be interpreted as the product of the 

coverage rate of the sampling frame and the response rate.

10 See p. 26 of https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHIS/2021/srvydesc-508.pdf 
Error Profile for the 2022 NIS-Teen 25

vaccination coverage among the population covered by the NIS-Teen and estimate nonresponse error 

in the NIS-Teen estimates by taking the difference between the NIS-Teen and the NHIS estimates.

We note that a common method for nonresponse analysis is to apply modeling, including logistic 

regression, to develop predictions or imputations of key variables among nonrespondents to develop 

full-response key estimates and compare to estimates based on respondents alone (U.S. Census 

Bureau, 2019). The method we employ in this study extends this concept to applying predictions or 

imputations to a reference survey. The final report of the NCES/NISS Task Force on Nonresponse Bias 

Analysis (National Institute of Statistical Sciences, 2009) recommends conducting multiple imputation 

when employing such methods to account for the uncertainty in estimates of nonresponse error due to 

missing data.11 

Table 3.5 compares the estimates of vaccination coverage based on the NHIS-imputed data and NISTeen provider-reported data. Two NIS-Teen estimates are presented, one based on applying design 

weights and another based on applying the final weights that reflect adjustments for noncoverage and 

nonresponse. Presenting both sets of estimates shows how the NIS-Teen estimates, before and after 

weighting adjustments, compare to the estimates based on NHIS imputations. The table further shows 

the percentage point difference between the NHIS-based estimate and the NIS-Teen estimate based 

on design weights. It includes a tt-statistic for testing the difference between the two vaccination 

coverage estimates, accounting for the uncertainty in both estimates.

The NIS-Teen estimates based on design weights are 0.4 percentage points higher than the estimates 

based on NHIS imputations for 1+ Tdap, 0.3 percentage points higher for 1+ MenACWY, and 0.9 

percentage points lower for UTD for HPV. None of these differences are statistically significant (pvalues of 0.73, 0.81, and 0.60 respectively), accounting for uncertainty in both the NHIS and NIS-Teen 

estimates. When examining HPV estimates by sex, the NIS-Teen estimates based on design weights 

are 1.0 percentage points lower for females (p-value 0.67) and 0.7 percentage points higher for males 

(p-value 0.78). For our TSE model, we estimate μμ̂2AA and by first taking the difference in estimates 

between (b) the respondent set and (a) the full-response set and then using our estimate of pp̂2AA above 

to derive μμ̂2AA as an estimate of the vaccination coverage among nonrespondents.

The NIS-Teen estimates based on final weights are 0.5 percentage points lower than the estimates 

based on NHIS imputations for 1+ Tdap, 0.4 percentage points lower for 1+ MenACWY, and 1.5

percentage points lower for UTD for HPV. Examining HPV estimates by sex, NIS-Teen estimates are 

2.1 percentage points lower for females and 1.1 percentage points lower for males. Overall, when 

viewing the estimates based on NHIS imputations (a) as the full-response estimate of vaccination 

coverage, these results suggest that nonresponse error in these NIS-Teen estimates is small, although 

the standard errors reflect uncertainty in our knowledge about the extent of nonresponse error.

11 Details of our methods for estimating nonresponse error are available from the authors.
Error Profile for the 2022 NIS-Teen 26

Table 3.5: Estimated Parameters of Nonresponse Error in Vaccination Coverage Estimates (in %) 

Derived from the 2022 NIS-Teen and the 2021 NHIS

Statistics 1+ 

Tdap

1+ 

MenACWY

UTD for 

HPV

UTD for 

HPV 

among 

Females

UTD for 

HPV 

among 

Males

(a) Estimate Based on NHIS Imputations 90.4 89.0 64.1 66.7 61.7

Standard Error 1.0 1.0 1.5 2.3 2.1

(b) NIS-Teen Estimate (Design Weighted) 90.8 89.3 63.2 65.7 61.1

Standard Error 0.4 0.5 0.7 1.0 1.0

(c) NIS-Teen Estimate (Final Weighted) 89.9 88.6 62.6 64.6 60.6

Standard Error 0.5 0.5 0.7 1.0 1.0

(d) Difference, (b) – (a) 0.4 0.3 -0.9 -1.0 -0.7

Standard Error of Difference 1.0 1.1 1.7 2.5 2.3

t-statistic 0.35 0.25 -0.53 -0.42 -0.28

p-value (2-sided) 0.73 0.81 0.60 0.67 0.78

(d) Difference, (c) – (a) -0.5 -0.4 -1.5 -2.1 -1.1

Standard Error of Difference 1.1 1.1 1.7 2.5 2.3

t-statistic -0.43 -0.38 -0.92 -0.83 -0.48

p-value (2-sided) 0.67 0.70 0.36 0.41 0.63

One caveat is that the findings depend on the fit of the models used for imputation and the assumption 

that the conditional distributions of the NHIS case and NIS-Teen case vaccination statuses on the 

model variables are the same. The goodness-of-fit for the imputation models is not particularly strong 

(pseudo-RR2 ≤ 0.10), resulting in large standard errors for estimates based on NHIS imputations. This 

fact further illuminates the extent of uncertainty in our estimates of nonresponse error. 

NIS-Teen Item Nonresponse Rates 

Thus far, this section has focused on assessing error in vaccination coverage rates due to unit 

nonresponse to NIS-Teen. Here, we present item nonresponse rates for the household interview 

portion of the survey in Table 3.6, focusing on socio-demographic variables used in raking procedures

for survey weighting. Most item nonresponse rates in this table are low and less than five percent. We 

also include exact family income which a higher item nonresponse rate of 26.5%, although the majority 

of exact-income nonrespondents completed the follow-up cascade of income questions, which 

establish tight income bounds. 
Error Profile for the 2022 NIS-Teen 27

Table 3.6: Item Nonresponse Rates, NIS-Teen Household Interview, United States, 2022

Variable

2022 Item 

Nonresponse 

Rate (Percent)*

Sex of adolescent 0.6

Hispanicity of adolescent 0.4

Race of adolescent with multiple race category 4.7

Education of mother 1.5

Household phone status – landline-only, cell-phone-only, or landline- and cellphone 1.0

Family income 26.5

 Exact income not reported but income cascade completed 18.1

 Exact income not reported and income cascade not completed 8.4

* Unweighted percent of "don't know" or "refused" responses among respondents asked the question. 

For race of child, percent also includes "other" responses that could not be back-coded into one or more 

of the race categories presented in the questionnaire. Rates presented in this table exclude U.S. 

territories.

3.3 Measurement Error

In this subsection, we assess one source of measurement error in the NIS-Teen: provider underreporting of the adolescent’s vaccination status. Throughout, we assume if a provider reported a 

vaccination for a given adolescent, it was actually given. We consider an adolescent to have underreported vaccination status if the adolescent is truly up-to-date for the vaccine but the adolescent is 

classified as not up-to-date based on the vaccination history reported by the adolescent’s provider(s). 

That is, adolescents with under-reporting are up-to-date but are reported as not up-to-date; adolescents 

without under-reporting are either both truly up-to-date and reported as up-to-date or are truly not up-todate and are reported as not up-to-date. All adolescents with under-reporting are, by definition, truly upto-date.

To assess under-reporting in provider-reported vaccination histories, we rely on projects sponsored by 

CDC in which the NIS-Teen sample of adolescents in selected geographic estimation areas was 

matched to the state or local IIS. For each of these projects, the NIS-Teen interview requested parental 

consent to contact both the adolescent’s vaccination providers and the local IIS. Adolescents for whom 

consent was obtained were matched to their respective IIS databases. Then, for the set of matched 

adolescents, we compared each adolescent’s vaccination status based on the provider report(s) to the 

adolescent’s vaccination status when both the provider(s) and the IIS reports are included in a 

combined vaccination history. 

We take the combined history to offer the best available information about the adolescent’s true 

vaccination status, and we view the NIS-Teen provider-reported history to be possibly subject to an 

under-reporting mechanism. This mechanism, often called under-ascertainment, can arise if some but 

not all of the adolescent’s providers were nominated by the household respondent, if the nominated 
Error Profile for the 2022 NIS-Teen 28

provider’s contact information was reported incorrectly by the household respondent, if not all of the 

nominated providers responded to the mailed Immunization History Questionnaire, or if respondent 

providers did not have complete vaccination records (such as when an adolescent moved out-of-area 

and vaccination records were not forwarded).

From the match studies, we estimated the proportion of adolescents with under-reported vaccination 

status. For each given vaccine, we determined the subset of matched adolescents for whom measured 

vaccination status (i.e., up-to-date or not up-to-date) from the combined (provider and IIS) vaccination 

history was equivalent to the vaccination status from the NIS-Teen provider-reported vaccination history 

alone. Then we made the reasonable assumption that equivalency of the measured vaccination 

statuses is a sign of accurate reporting in the NIS-Teen Provider Record Check. In other words, if the 

IIS did not add information about vaccination status beyond that already embodied within the NIS-Teen 

provider-reported data, then we took the NIS-Teen data to be accurate (not under-reported). If the 

adolescent was up-to-date based on the combined (provider and IIS) vaccination history but not up-todate based on the NIS-Teen vaccination history alone, then we classified the adolescent as having an 

under-reported vaccination status in the NIS-Teen. Among the adolescents with adequate NIS-Teen 

provider data that were in the IIS and had two or more doses in the IIS, we estimated the NIS-Teen 

under-reporting rate for the vaccine as the design-weighted proportion classified as having underreported vaccination status for the vaccine. 

Recent sources of information for assessing under-reporting in the NIS-Teen are the match projects 

completed in 2017 and 2019. In 2017, match projects were conducted in twenty jurisdictions: Arkansas, 

Georgia, Idaho, Iowa, Louisiana, Maine, Michigan, Mississippi, Nevada, New Mexico, New York City, 

North Carolina, North Dakota, Oklahoma, Rhode Island, South Dakota, Vermont, Washington, 

Wisconsin, and Wyoming. In 2019, match projects were conducted in eight jurisdictions: Arkansas, 

Kansas, Louisiana, Missouri, Nevada, New York City, Vermont, and Washington. Because only a 

subset of jurisdictions participated in match projects, we estimated the standard error of the estimated 

under-reporting rate by treating each selected IIS as a cluster sampled from the population of IIS in the 

United States.

Table 3.7 presents the estimated under-reporting error for the vaccines and vaccine series under study.
Error Profile for the 2022 NIS-Teen 29

Table 3.7: Estimated Under-Reporting Error by Vaccine: NIS-Teen, United States, 2017 

and 2019

Under-reporting Error

Vaccine

Estimate 

(percentage 

points) 

Standard 

Error 

(percentage 

points)

≥1 Tdap at or after age 10 years 4.7 0.33 

≥1 Meningococcal ACWY 4.1 0.33 

UTD for HPV* 2.5 0.25 

UTD for HPV among females 2.3 0.33 

UTD for HPV among males 2.5 0.38 

Note: National-level under-reporting in NIS-Teen provider-reported vaccination status was 

estimated using data from the 2017 and 2019 IIS-NIS Match Projects. Among adolescents with 

adequate provider data found in the IIS database with two or more IIS doses, those classified as 

up-to-date based on the combined IIS-NIS vaccination history but not up-to-date based on the 

NIS-Teen vaccination history alone were considered to have under-reported NIS-Teen vaccination 

status.

* ≥3 doses, or ≥2 doses if 1st dose before age 15 and at least 5 months – 4 days between 1st 

and 2nd doses

3.4 Sampling Error

Sampling error arises from the fact that we observe only a random sample of the adolescent 

population, not the entire population. Table 3.8 presents estimated vaccination coverage and 

corresponding standard errors for 2022 NIS-Teen at the national level. We calculated the standard 

errors using the Taylor series method, first for the design-weighted vaccination coverage estimate and 

then for the final-weighted vaccination coverage estimate. The design weights reflect the sample design 

but do not include adjustments for noncoverage, nonresponse, nor calibration to population control 

totals. Final weights are the design weights, with adjustments for noncoverage, nonresponse, and 

calibration to population control totals.

The national-level standard errors for these five vaccination coverage estimates are small, ranging from 

approximately 0.4 to 1.0 percentage points.
Error Profile for the 2022 NIS-Teen 30

Table 3.8: Vaccination Coverage Estimates and Standard Errors Using Design Weight 

and Final Weight: NIS-Teen, United States, 2022

Vaccine

Design Weight Final Weight

Estimate

(%)

Standard 

Error 

(percentage 

points)

Estimate

(%)

Standard 

Error 

(percentage 

points)

1+ Tdap at or after age 10 years 90.8 0.4 89.9 0.5

1+ MenACWY 89.3 0.5 88.6 0.5

UTD for HPV* 63.2 0.7 62.6 0.7

 UTD for HPV among females 65.7 1.0 64.6 1.0

 UTD for HPV among males 61.1 1.0 60.6 1.0

Note: Excludes U.S. territory samples.

* ≥3 doses, or ≥2 doses if 1st dose before age 15 and at least 5 months minus 4 days between 

1st and 2nd doses

3.5 Total Survey Error Distribution

This section consolidates the component assessments of sampling-frame coverage error (Section 3.1), 

nonresponse error (Section 3.2), measurement error (Section 3.3), and sampling error (Section 3.4) to 

develop our estimates of total survey error for estimated vaccination coverage corresponding to 1+ 

Tdap, 1+ MenACWY, and UTD for HPV both overall and separately for males and females. The section 

culminates with the presentation of total survey error distributions, constructed using the methodology 

described in Molinari, Wolter, Skalland et al. (2011) and Wolter, Pineau, Skalland et al. (2017). For 

each estimate, we review the distribution of total survey error across 10,000 Monte Carlo simulations, 

treating the mean of the distribution as the point estimate of total error and the interval between the 

2.5th percentile and the 97.5th percentile as the 95% credible interval of total error. 

At the beginning of Part II of this report, we presented our TSE model and its seven parameters. Table 

3.9 contains the values of these seven parameters and their standard errors we used in the model for 

TSE in the 2022 NIS-Teen. These values arise from the analyses described in Sections 3.2 through 3.4 

above.12 We assume the logit transformations of the inputs are normally distributed and independent 

(i.e., no covariance between inputs).

12 μμ̂1AA and μμ̂2AA, which were estimated based on the NHIS-PRC and models built from NIS-Teen vaccination data, respectively, 

have been adjusted upwards to account for provider under-reporting error in those surveys, assuming the same level of underreporting error as was estimated in Section 3.4.
Error Profile for the 2022 NIS-Teen 31

Table 3.9: Total Survey Error Model Inputs by Stages: NIS-Teen, United States, 2022

Parameter 1+ Tdap 1+ MenACWY UTD for 

HPV

UTD for HPV 

among 

Females

UTD for HPV 

among Males

Stage 1: Sampling-Frame Coverage Error

0.8% 0.8% 0.8% 0.8% 0.8%

0.2% 0.2% 0.2% 0.2% 0.2%

94.7% 95.7% 54.9% 58.5% 52.4%

9.3% 8.6% 11.1% 11.2% 9.7%

Stage 2: Nonresponse Error

97.2% 97.2% 97.2% 97.2% 97.2%

0.2% 0.2% 0.2% 0.2% 0.2%

95.1% 93.1% 66.6% 69.0% 64.3%

1.0% 1.1% 1.6% 2.4% 2.2%

Stage 3: Measurement Error

4.7% 4.1% 2.5% 2.3% 2.6%

0.3% 0.3% 0.3% 0.3% 0.4%

100.0% 100.0% 100.0% 100.0% 100.0%

0.0% 0.0% 0.0% 0.0% 0.0%

95.3% 93.1% 64.8% 67.2% 62.7%

0.6% 0.6% 0.7% 1.0% 1.0%

Table 3.10 presents the means and 95% credible intervals of total survey error distributions and 

component error distributions based on 10,000 Monte Carlo draws from the input (or component) error 

distributions and application of the TSE model set forth in equations (1) - (6). The means of the 

estimated TSE distributions are -5.0 percentage points for 1+ Tdap, -4.4 percentage points for 1+ 

MenACWY, -3.9 percentage points for HPV, -4.2 percentage points for HPV among females, and -3.5

percentage points for HPV among males. The 95% credible intervals for the two UTD for HPV total 

error estimates by sex include 0. These results suggest that the 2022 NIS-Teen may have somewhat 

underestimated the true vaccination coverage for 1+ Tdap, 1+ MenACWY, and UTD for HPV. The 

largest estimated component of error in absolute value for all three series is measurement error, i.e., 

provider under-reporting error. 
Error Profile for the 2022 NIS-Teen 32

Table 3.10: Mean and 95% Credible Interval for the Estimated TSE Distribution and 

Component Error Distributions: NIS-Teen, United States, 2022

Vaccine or 

Series Component 

Mean TSE

(percentage 

points)

95% Credible Interval

(percentage points)

1+ Tdap

TSE (final weighted) -5.0 (-6.8, -2.6)* 

TSE (design weighted) -4.2 (-5.9, -1.8)* 

Noncoverage error 0.1 (-0.1, 0.6)

Nonresponse error 0.4 (-1.6, 3.0)

Measurement error -4.7 (-5.6, -3.6)* 

Sampling error 0.0 (-1.1, 1.4)

1+ 

MenACWY

TSE (final weighted) -4.4 (-6.3, -2.1)* 

TSE (design weighted) -3.7 (-5.6, -1.4)* 

Noncoverage error 0.1 (-0.1, 0.6)

Nonresponse error 0.3 (-1.8, 2.8)

Measurement error -4.1 (-5.1, -3.0)* 

Sampling error 0.0 (-1.2, 1.3)

UTD for HPV

TSE (final weighted) -3.9 (-6.7, -0.8)* 

TSE (design weighted) -3.2 (-6.1, -0.2)* 

Noncoverage error 0.2 (-0.1, 0.4)

Nonresponse error -0.9 (-4.1, 2.4)

Measurement error -2.5 (-3.8, -1.0)* 

Sampling error 0.0 (-1.4, 1.5)

UTD for HPV 

among 

females

TSE (final weighted) -4.2 (-8.5, 0.4)

TSE (design weighted) -3.2 (-7.4, 1.4)

Noncoverage error 0.1 (-0.1, 0.4)

Nonresponse error -1.0 (-5.7, 3.9)

Measurement error -2.3 (-4.2, -0.3)* 

Sampling error 0.0 (-2.0, 2.0)

UTD for HPV 

among 

males

TSE (final weighted) -3.5 (-7.5, 0.7)

TSE (design weighted) -3.0 (-7.0, 1.2)

Noncoverage error 0.2 (-0.1, 0.4)

Nonresponse error -0.6 (-5.2, 4.1)

Measurement error -2.6 (-4.6, -0.6)* 

Sampling error 0.0 (-2.0, 2.1)

* 95% credible interval does not include 0.
Error Profile for the 2022 NIS-Teen 33

3.6 Assessment of the Change in Bias Using the Bridging 

Cohort Method

In the previous section, we assessed TSE in the 2022 NIS-Teen estimated vaccination coverage, while 

in the current section, we assess the change in TSE between vaccination coverage estimates produced 

from the 2021 and 2022 NIS-Teen samples. Change is now measured using the bridging cohort 

method first described in Yankey, Hill, Elam-Evans et al. (2015). 

Each survey quarter includes adolescents born within 21 quarterly birth cohorts. Every pair of adjacent 

survey quarters spans 22 quarterly birth cohorts, of which 20 are in common and 2 are not in common. 

In turn, every survey year includes adolescents born within 24 quarterly birth cohorts. Every pair of 

adjacent survey years spans 28 quarterly birth cohorts, of which 20 are in common and 8 are not in 

common. We shall call the common quarters the bridging cohort, and for 2021 and 2022, the bridging 

cohort extends from adolescents born in Q1 2004 through adolescents born in Q4 2008. 

Consider a vaccination series with coverage estimated from the bridging cohort as of a given 

adolescent age, such as 13 years. Two estimates are possible, one using the sample of adolescents in 

the bridging cohort within the 2021 NIS-Teen sample and the second using the corresponding sample 

of adolescents within the 2022 NIS-Teen sample. Ideally, the two estimators should exhibit the same 

mean value. A large difference between the two estimates may signal a change in the expectation of 

the estimator from one survey year to the next, which could result from a change in the distribution of 

sampling-frame coverage error, nonresponse error, or measurement error. Differences may also result 

simply from the effects of random sampling error.

Table 3.11 presents the two estimated vaccination estimates for adolescents as of 13 years of age for 

the 2021-2022 bridging cohort. The columns on the right side of the table reveal the differences 

between the 2022 and 2021 estimates for the bridging cohort, the estimated standard errors of the 

differences, and the p-values associated with statistical tests of the hypothesis that the expectations of 

the two estimators are the same. Summarizing, we do not observe any statistically significant 

differences between the 2022 and 2021 vaccination coverage estimates for the 2021-2022 bridging 

cohort. That is, there is no statistical evidence of a change in mean TSE between 2021 and 2022. 
Error Profile for the 2022 NIS-Teen 34

Table 3.11: Difference between the Estimates* for the Bridging Birth Cohort: NIS-Teen, United States, 

2021 vs. 2022

2021 2022 Difference

Description Estimate

Standard 

Error Estimate

Standard 

Error Estimate

Standard 

Error

pp-value for 

Test of No 

Difference

1+ Tdap by 13 years 88.8 0.62 88.0 0.55 -0.8 0.83 0.361

1+ MenACWY by 13 

years

83.3 0.76 83.0 0.63 -0.4 0.99 0.716

UTD for HPV& by 13 

years 36.9 0.94 36.8 0.75 -0.1 1.20 0.961

UTD for HPV by 13 

years among 

females

38.8 1.38 38.8 1.10 -0.4 1.76 0.805

UTD for HPV by 13 

years among males 35.5 1.27 35.0 1.03 -0.5 1.64 0.770

* Estimates were computed among adolescents with adequate provider data, excluding U.S. territories. The 

bridging birth cohort used for this analysis includes adolescents born between January 2004 and December 

2008. Final provider-phase weights for 2021 were ratio-adjusted within each monthly birth cohort such that their 

sum within monthly birth cohort equals the sum of the final provider-phase weights for 2022 within the 

corresponding monthly birth cohort. 

& ≥3 doses, or ≥2 doses if 1st dose before age 15 and at least 5 months minus 4 days between 1st and 2nd 

doses.

4. Summary 

We profiled the sources of error in 2022 NIS-Teen statistics at the national level (excluding U.S.

territories) for the total age-eligible population of adolescents. We compared NIS-Teen statistics to 

corresponding values from benchmark surveys and other external sources (Part I) and assessed 

component and total error in vaccination coverage estimates through a series of specialized evaluation 

studies (Part II). Wherever possible, we used 2022 sources and studies to assess error in the 2022

NIS-Teen. Where 2022 sources were not available, we reported information from prior year sources as 

the best information available for understanding error in the 2022 NIS-Teen. 

In Part I, we compared NIS-Teen demographic distributions (age, sex, race/ethnicity, mother’s 

education, mother’s age) to benchmark values derived from the U.S. Census Bureau’s PEP and ACS 

data. When using design weights that have not been calibrated to external population totals, 

demographic distributions as estimated by the survey are generally close to the benchmark 

distributions. Before calibration, the NIS-Teen somewhat over-represented non-Hispanic White-only 

adolescents, under-represented Hispanic adolescents, and over-represented adolescents whose 
Error Profile for the 2022 NIS-Teen 35

mothers are college graduates. When using final weights that have been calibrated to external 

population totals, the differences between survey estimates and population values narrowed, but the 

2022 NIS-Teen still over-represented adolescents whose mothers are college graduates and underrepresented adolescents whose mothers have some college but not a four-year degree.

We compared NIS-Teen vaccination coverage estimates to IISAR vaccination coverage estimates and 

found that NIS-Teen vaccination coverage estimates are higher in many jurisdictions. For five of the six 

2013-2017 sentinel sites, we found good agreement between NIS-Teen and IISAR estimates. (For the 

sixth sentinel site, the IISAR estimate was over 100 percent and therefore a clear over-estimate.)

Further, we determined that the adolescent participation rate is a reasonable indicator of the quality of 

the corresponding IIS database. We learned that the difference between NIS-Teen and IISAR 

vaccination coverage estimates declines as the adolescent participation rate increases (i.e., as the 

quality of the IIS increases). The findings are consistent with the view that IIS vaccination coverage 

estimates converge towards NIS-Teen vaccination estimates as the quality of the IIS increases.

We compared NIS-Teen health insurance distributions to similar distributions produced by the ACS, 

CPS ASEC, and NHIS. The surveys use somewhat different definitions of insurance status. 

Nevertheless, we found the four distributions to be broadly similar, but with some modest differences. 

The NIS-Teen estimate of percent of adolescents with any public insurance was higher than the 

corresponding estimates from ACS, CPS ASEC, and NHIS, and the NIS-Teen estimate of uninsured 

adolescents was lower than the estimates from the benchmark surveys.

Finally, we compared NIS-Teen vaccination coverage estimates to comparable statistics produced by 

immunization surveys sponsored by states. Results of the Georgia Adolescent Immunization Study 

(GIS) and an analysis of Kansas BRFSS data (KIS) were the only such statistics available at this time. 

GIS and NIS-Teen, and KIS and NIS-Teen displayed reasonably similar vaccination coverage 

estimates for 1+ Tdap and 1+ MenACWY. For males and females both separately and combined, NISTeen vaccination coverage estimates for UTD for HPV were substantially and statistically significantly 

higher than the corresponding estimates from GIS and KIS. In the case of GIS, this finding is likely due 

to differences in the age groups included in the study populations, and in the case of KIS, to incomplete 

reporting by the parental respondents and the likely inclusion of 12-year-olds for estimates. 

In Part II of the report, we evaluated NIS-Teen vaccination coverage estimates with respect to sampleframe coverage error, nonresponse error, measurement error, sampling error, and total survey error. 

We also assessed the change in total survey error from 2021 to 2022. 

The NIS-Teen cell-phone RDD sampling frame fails to include the LLO and phoneless populations, and 

we assessed vaccination coverage estimates in the former using data collected in the 2017 NIS-Teen 

and in the latter using data collected in the 2012 NHIS Provider Record Check. The vaccination 

coverage estimates in the LLO population tended to be less than the vaccination coverage estimates in 

the population included in the sampling-frame, and the results were somewhat mixed with regard to the 
Error Profile for the 2022 NIS-Teen 36

phoneless population. Because the sampling-frame uncovered population is so small relative to the 

covered population, however, we found mean sampling-frame coverage error to be 0.1 percentage 

points or less for each of the vaccine series examined.

We compared the 2021 NHIS and 2022 NIS-Teen to assess nonresponse error in the 2022 NIS-Teen. 

The NHIS does not offer direct estimates of vaccination coverage. Instead, we used a model-based 

technique to impute NHIS vaccination status and then compared the resulting NHIS vaccination 

coverage estimates (treated as vaccination coverage estimates void of nonresponse error) to NIS-Teen 

vaccination coverage estimates, with the difference treated as nonresponse error in the NIS-Teen. 

Incorporating all sources of missing data, including (1) nonresolution of telephone numbers, (2) 

nonresponse to the screener, (3) failure to complete the interview, (4) non-consent to contact providers, 

and (5) nonresponse from providers, we estimated that over 90% of the age-eligible sample of 

households failed to respond to the NIS-Teen. Despite this large percentage, we found mean 

nonresponse error in vaccination coverage estimates to be modest and not statistically significant for all 

of the vaccine series examined. 

We used twenty IIS-NIS match studies conducted in 2017 and eight additional match studies conducted 

in 2019 to assess measurement error, or under-ascertainment, in the NIS-Teen vaccination coverage 

estimates. In this work, the standard of truth for a given child is taken to be the synthesis of the NISTeen and IIS vaccination histories. We found measurement error was by far the largest component of 

error in NIS-Teen vaccination coverage estimates. We found measurement error depressed observed 

vaccination coverage estimates by about two to five percentage points, and these findings were 

statistically significant. Under-ascertainment of adolescent vaccination history may arise due to the 

failure of the household respondent to nominate all of the adolescent’s vaccination providers, failure of 

the nominated vaccination providers to respond, or failure of the responding providers to report all of 

the vaccinations that the adolescent has received.

We combined all of the component errors and assessed the distribution of total survey error in the NISTeen vaccination coverage rates, using a Monte Carlo technique. For the 1+ Tdap vaccination 

coverage rate, we found the mean of the TSE distribution to be -5.0 percentage points with a 95% 

credible interval of (-6.8, -2.6) percentage points. That is, the NIS-Teen vaccination coverage rate was 

on average about 5.0 percentage points too low. For the 1+ MenACWY vaccination coverage rate, we 

found the mean of the TSE distribution to be -4.4 percentage points with a 95% credible interval of 

(-6.3, -2.1) percentage points, and for the UTD for HPV vaccination coverage rate, we found the mean 

of the TSE distribution to be -3.9 percentage points with a credible interval of (-6.7, -0.8) percentage

points. Again, under-ascertainment of the provider-reported vaccination history dominated total survey 

error. Estimates of nonresponse error have wider 95% credible intervals reflecting that those estimates 

have larger uncertainty than other error components. 

Finally, using the bridging cohort method, we assessed a change in bias between 2021 and 2022 NISTeen by comparing vaccination coverage estimates for adolescents born between the first quarter of 
Error Profile for the 2022 NIS-Teen 37

2004 and the fourth quarter of 2008 available from both survey years. In conducting comparisons for 

five key vaccination coverage estimates, we found no statistically significant differences and no 

evidence of a change in mean TSE between 2021 and 2022.

Our results for the 2022 NIS-Teen are subject to various limitations. The comparisons to benchmark 

distributions in Part I are flawed because the benchmark source usually uses somewhat different 

concepts or definitions than the NIS-Teen. Our comparison of NIS-Teen and IISAR vaccination 

coverage estimates is limited to 1+ Tdap, and the findings may not apply to other vaccine series. In 

Part II, the results are based on input distributions for the component errors as estimated using our best 

available information from external sources and studies, but these inputs may not be accurate. While 

large-sample theory motivates our choice of the normal family of distributions, we have not validated 

this choice. Two key external sources of the information on components errors are the NHIS and state 

or local IIS. The NHIS is based on a smaller sample size than the NIS-Teen, its NHIS Provider Record 

Check (used in the study of sampling-frame coverage error) is likely subject to many of the same 

measurement issues as the NIS-Teen Provider Record Check, and it is subject to its own nonresponse 

and sampling-frame coverage errors. To study nonresponse error in the NIS-Teen, we utilized imputed 

vaccination statuses in the NHIS rather than provider-reported statuses, because the NHIS Provider 

Record Check was terminated in 2013. IIS may underestimate vaccination coverage to some extent

(e.g., to miss some resident children and some vaccine doses within included children), and 

completeness may vary substantially from one state or local area to the next. Our results are based on 

work with IIS in only 22 areas across 28 studies. Our results are also based on an assumption of 

independence of the component errors and this assumption might not be accurate. We conducted the 

TSE analysis for selected national-level vaccination coverage estimates, and the results do not 

necessarily extend to other vaccines, states or estimation areas, or socio-demographic domains. 
Error Profile for the 2022 NIS-Teen 38

References

Biemer, P. P. (2010). Total survey error: Design, implementation, and evaluation. Public Opinion 

Quarterly, 74(5), 817-848.

Blumberg, S.J., Ganesh, N., Luke J.V., and Gonzales G. (2013). Wireless substitution: State-level 

estimates from the National Health Interview Survey, 2012. National health statistics reports; no 70. 

Hyattsville, MD: National Center for Health Statistics. Available from: 

https://www.cdc.gov/nchs/data/nhsr/nhsr070.pdf

CDC (2022). National Health Interview Survey: 2021 Survey Description. Division of Health Interview 

Statistics, National Center for Health Statistics, U.S. Department of Health and Human Services. 

https://ftp.cdc.gov/pub/Health_Statistics/NCHS/Dataset_Documentation/NHIS/2021/srvydesc-508.pdf

Cohen, R.A., Terlizzi, E.P., Martinez, M.E. (2018). Health Insurance Coverage: Early Release of

Estimates from the National Health Interview Survey, 2018. National Center for Health Statistics, 

Hyattsville, MD. 

Gillespie, K. (2018). Adolescent Vaccination Coverage in Kansas, an Analysis of BRFSS Surveys

2011-2016. Kansas Department of Health and Environment, Topeka, KS. 

https://www.kansaspharmacyfoundation.org/pdf/vaccines/vaccination-rates-Adolescent-Coverage2016.pdf

Machado, F.R. (2017). Georgia Adolescent Immunization Study 2017. Georgia Department of Public 

Health, Atlanta, Georgia. 

https://dph.georgia.gov/sites/dph.georgia.gov/files/Adolescent%20Immunization%20Study%202017%2

0Final%20Report_Georgia.pdf

Machado, F.R. (2018). Georgia Adolescent Immunization Study 2018. Georgia Department of Public 

Health, Atlanta, Georgia. 

https://dph.georgia.gov/sites/dph.georgia.gov/files/Immunizations/2018%20GAIS%20final_web.pdf

Molinari, N.M., Wolter, K., Skalland, B., Montgomery, R., Khare, M., Smith, P.J., Barron, M.L., 

Copeland, K., Santos, K., and Singleton, J.A. (2011). Quantifying bias in a health survey: Modeling total 

survey error in the National Immunization Survey. Statistics in Medicine, 30, 505-514.

National Institute of Statistical Sciences. (2009). NCES/NISS task force on nonresponse bias analysis: 

Final report. NISS, Research Triangle Park, NC.

Rubin, D. B. (1976). Inference and missing data. Biometrika, 63(3), 581-592.

Skalland, B. (2011). An alternative to the response rate for measuring a survey’s realization of the 

target population. Public Opinion Quarterly, 75(1), 89-98.
Error Profile for the 2022 NIS-Teen 39

Smith, P.J., Hoaglin, D.C., Battaglia, M.P., Khare, M., and Barker, L.E. (2005). Statistical methodology 

of the National Immunization Survey, 1994-2002. Vital and Health Statistics 2(138). National Center for 

Health Statistics, Hyattsville, MD.

Staras, S. A., Huo, T., Rothbard, S. M., Hall, J. M., Cho, H. D., Guo, Y., Richardson, E., Salloum, R. G., 

Thompson, L. A., Silver, N. L., and Shenkman, E. A. (2021). Human Papillomavirus Vaccination and 

Human Papillomavirus–Associated Cancer Rates Within Florida Counties. American Journal of 

Preventive Medicine, 61(6), 812-820.

U.S. Census Bureau (2019). 2017 National Survey of Children’s Health: Nonresponse Bias Analysis. 

Suitland, MD.

Weatherer, A. C., Pritzl, S. L., Kerch, S., Li, Z., and LoConte, N. K. (2021). Current Trends in HPV 

Vaccine Uptake: Wisconsin and United States, 2016-2019. WMJ: Official publication of the State 

Medical Society of Wisconsin, 120(1), 62-65.

Wolter, K.M., Pineau, V.J., Skalland, B., Zeng, W., Singleton, J.A., Khare, M., Zhao, Z., Yankey, D., and 

Smith, P.J. (2017). Total survey error assessment for sociodemographic subgroups in the 2012 U.S. 

National Immunization Survey. In Total Survey Error in Practice, edited by Biemer, P.P., de Leeuw, E., 

Eckman, S., Edwards, B., Kreuter, F., Lyberg, L.E., Tucker, N.C., and West, B.T. John Wiley & Sons, 

Hoboken, New Jersey.

Wolter, K. M., Smith, P.J., Khare, M, Welch, B., Copeland, K. R., Pineau, V. J., and Davis, N. (2017). 

Statistical methodology of the National Immunization Survey, 2005-2014. Vital and Health Statistics 

1(61). National Center for Health Statistics, Hyattsville, MD.

Yankey, D., Hill, H.A., Elam-Evans, L.D., Khare, M., Singleton, J.A., Pineau, V., and Wolter, K. (2015). 

Estimating change in telephone survey bias in an era of declining response rates and transition to 

wireless telephones – evidence from the National Immunization Survey (NIS), 1995-2013. Presented at 

the annual conference of the American Association for Public Opinion Research, Hollywood, FL.