# Presentation: 794585

## National Immunization Survey: Data Quality and Public-Use Data Files

***National Immunization Survey: ******Data Quality and Public-Use Data Files***

**Meena Khare, **
**National Center for Health Statistics **
**Michael P. Battaglia, **
**Abt Associates Inc.**
**July 16, 2002**

## Outline

***National Immunization Survey***
***Quality of the data***
***Confidentiality issues ***
***Public Use Data Files (PUF)***
*** ******Summary***

**Notes:**
The PMO Management Advisory Team was formulated at AFCEE to assist HQ USAFE/CEC in setting up a PMO that will manage the $1B housing improvement projects in Germany over the next 10 years.  The team collected data that will assist in establishing the guidelines and governing body for the PMO.  This data will be used to create a program management plan for the PMO.

## National Immunization Survey (NIS) 

***Large ongoing RDD survey, conducted by CDC since 1994***
***Measures****** vaccination coverage among children aged 19-35   months at National, State, and Urban area levels (78 IAP areas)***
***Monitors****** Healthy People 2000 and 2010 goals of immunization coverage***
_***>***_***90% Coverage: 4DTP, 3Polio, 1MMR, 3HepB, 3Hib, and 4:3:1:3 series***
***Monitors introduction of new vaccines***
***<4% of households in the U.S. contain child of age 19-35 months***

**Notes:**
- The keys to success must be in-grained in the minds of each member of the PMO.  With these goals in mind, the PMO will be able to successfully attain the goals.  
- Communication and execution are major keys to success.  
- State-of-the-Art automated computer-based systems are necessary in order to have up-to-the-minute reports.  
- The PMO team needs to be adequately trained.  
- All processes should be set-up in a logical manner (processes should flow).  
- The PMO should also take lessons learned from other PMOs and implement those ideas.

## NIS Operations

## NIS Data Collection

***Household CATI Screener and Interview ***
**Parent/Guardian *****(most knowledgeable person)***
**Socio-demographic information: ***mother and child** *
**Shot card use or memory recall **
**Vaccination dates (shot card only)**
**Provider’s contact information with consent**
***Provider Record Check Study (mailed IHQ)***
**Provider’s office (e.g., staff, nurse, manager)**
**Mail, Fax, Telephone **
**Completed IHQ or copy of medical records**
**Matched on DOB, Gender, Name**
**Vaccination history**
**Provider’s information**

## Slide 6

## Goals of Quality Control in the NIS

***Evaluate and improve quality of the information collected***
***Reduce/eliminate discrepancies***
***Improve quality of the estimates***
***Reduce potential bias in official estimates of vaccination coverage rates***

## Quality Control Procedures used in the NIS

***Extensive monitoring and QC review of the data collected from CATI and mailed IHQs***
***Automated edit software for processing combined household and provider data ***
***Manual ‘Matching Sheet’ review of discrepant cases ***
***Resolution by using guidelines developed by expert immunization program managers ***

## Review of Data Errors in the IHQ

***Data Entry Validation***
**Double data entry**
**Approximately 400 forms reviewed/quarter**
**Error rates ranged from 0.23% - 0.67% (~30,000 fields with data, NIS Q4/1999-Q4/2001)**
*** ******~350-1000 manual Matching Sheet reviews/quarter for discrepant data***
***Most errors in dates are found in the original provider-reported records***
***<1% children had data errors ***

## Observed Discrepancies in Dates

***DOB***
**Between household (HH) and provider(s)**
**Between multiple providers for a child**
***Shot dates***
**Between HH and provider(s)**
**Within a single provider**
**Between multiple providers for a child**

##  DOB and Shot dates 
122 children classified as age-ineligible
69 IHQs filled out for the ‘wrong’ child
21 children had changes in best DOB
2163 children had shot dates before DOB or vaccination interval < 1 month, or other shot date edits

Hepatitis B ‘birth’ dose
361 children had missing provider ‘birth dose' of Hepatitis B and providers had checked the ‘given at birth’ box on IHQ
34 children had imputed dates from the household shot card
327 children had imputed dates from the distribution of provider-reported dates for the birth dose

**122 children classified as age-ineligible**
**69 IHQs filled out for the ‘wrong’ child**
**21 children had changes in best DOB**
**2163 children had shot dates before DOB or vaccination interval < 1 month, or other shot date edits**
_***Hepatitis B ‘birth’ dose***_
**361 children had missing provider ‘birth dose' of Hepatitis B and providers had checked the ‘given at birth’ box on IHQ**
***34 children had imputed dates from the household shot card***
***327 children had imputed dates from the distribution of provider-reported dates for the birth dose***

***Matching Sheet Review, 2000 NIS***

## Results of Quality Control Evaluation

***No single data source is 100% complete and accurate***
***Some discrepancies in DOB and vaccination dates/number of doses could not be verified and remained after edits ***
***Number of discrepancies increased if more than one immunization provider per child (1.35 providers per child in 1999 NIS ) was identified ***
***Errors in ***
**Original Immunization records **
**Locating records for the correct child **
**Transcribing shot dates to the IHQ**
**Illegible dates, transposed dates, missing day/month, incorrect dates (especially year)**
**Shot date interval< 1 month, or  shot dates<DOB**
**Missing ‘first’ Hepatitis B dose given at birth***** ***

## ‘Best’ Vaccination Value Research

*** ******Currently, coverage estimates from the ***_***provider***_*** data are used as ***_***official ***_***estimates ***
***The goal of ‘best’ value research is to obtain the most complete and accurate immunization data possible from ***_***two***_*** sources***
**Supplement incomplete provider data with household data for children who are UTD from ‘shot card’ **
**Substitute completely missing provider data with household data for children who are 4:3:1:3 UTD from ‘shot card’ **

## NIS 2000 Sample

***35,960 households with eligible children 19-35 months***
***33,477 (93.1%)of eligible households with completed interviews***
***34,087 age-eligible children with completed interviews***
***22,958 (67.4%) children with completed interviews and ‘adequate’ provider data***

## Sources of Immunization Histories, 2000 NIS

| Source | Household data from Shot Card  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| ‘Adequate’ Provider data  | Yes |  | NO |  | Total |  |
|  | n | % | n | % | n | % |
| Yes | 11,963 | 73.8 | 10,995 | 61.5 | 22,958 | 67.4 |
| No | 4,249 | 26.2 | 6,880 | 38.5 | 11,129 | 32.6 |
| Total | 16,212 | 47.6* | 17,875 | 52.4* | 34,087 | 100.0 |

**** row percents***

**Notes:**
 "children with adequate provider data" refers to children for whom sufficient vaccination
history information is obtained from their providers to determine whether
they are up-to-date with respect to the recommended vaccination schedule.

## Assessment of Provider-Reported 4:3:1:3* Coverage Estimates among Children with Household Data,2000 NIS

| Source of HH reports
and measures of quality | Shot Card | ‘Memory’ Recall** |
| --- | --- | --- |
| Sample Size  | 11,963 | 5,526 |
| Sensitivity | 67.1 | 44.8 |
| Specificity | 61.8 | 65.3 |
| Misclassification (prov)
(HH UTD)/ Prov NUTD) | 7.4 | 8.7 |
| Overall agreement | 66.1 | 49.9 |
| Overall disagreement | 33.9 | 50.1 |
| Net Difference (HH Cov - Prov cov) | -19.1 | -32.7 |

****4DTP/3Polio/1MCV/3Hib***
***** Other 5,469 non-shot card children have missing 4:3:1:3 UTD status***

## Comparison of Vaccine-specific Coverage Rates by Source of Immunization Histories, 2000 NIS   

***Best Value: Combined Provider and household ‘shot card’ data***
***Revised Best Value: After adjusting Best Value for missing provider data among non-shot card children ***

## NIS Data Files and Estimates

***In-house analytic files***
***Public-use data files (PUF )***
***National, State, and Urban area level estimates of vaccination coverage***
**Internet: **_**[www.cdc.gov/nip/coverage](http://www.cdc.gov/nip/coverage)**_** **
**MMWR **
**Journal Articles**
**Articles in Proceedings**
**Presentations at National and State Conferences**
**Publications are listed on NIS Web site**

## Confidentiality Issues

***Legal obligation to protect confidentiality of respondents and reduce risk of disclosure***
***Sample size (4 quarters)***
- ~ 440 children per IAP with interview data
- ~ 290 children per IAP with provider data
***Only telephone households (90% HHs with telephone)***
***Rare population (children 19-35 months)***
***Geographic identifiers***
**State, Urban Areas**
**Census Region**
**Smaller geographic areas (e.g., Zip code, county)**
***Socio-demographic details***
***Analytic cells and cell sizes (within IAP area)***
**Age, gender, race/ethnicity**
**WIC participation, Poverty status**
**Minimum population size**_**>**_**5 in unique cells**
***Date of birth, shot dates, interview dates***
***Clearance by the NCHS Disclosure Review Board (DRB)***

## The NIS PUF

***Public-use data files (1995-2000): Child-level records with 78 IAP area (state and urban) identifiers ******(2001 NIS PUF by Fall 2002)***
***Approximately 35,000 age-eligible children with household interview data***
***Approximately 23,000 children with household interview ***_***and***_*** ‘adequate’ provider data; on average 295 children per IAP area ***
***PUFs released on the Internet and CD-ROMs***
**Internet : **_**[www.](http://www.cdc.gov/nis)**__**[cdc](http://www.cdc.gov/nis)**__**[.](http://www.cdc.gov/nis)**__**[gov](http://www.cdc.gov/nis)**__**[/](http://www.cdc.gov/nis)**__**[nis](http://www.cdc.gov/nis)**_** (1995-2000)**
**CD_ROMs No 1-5**

**Notes:**
Information on about 1000 variables including child’s age, sex, race/ethnicity, and mother’s age, mother’s race/ethnicity, mother’s education, family income, and poverty status along with geographic identifiers are released in the PUF  

## Contents of The NIS PUF

***Household Interview data items***
**Unique sequence ID, age at interview (months), gender, race/ethnicity, ...**
**Mother’s age group, education, marital status, ...**
**WIC participation, income, poverty status,...**
**Vaccine specific immunization history (#doses, up-to-date status) **
**IAP area and State identifiers, Census Region **
***Provider data items***
**Vaccine-specific Immunization History: **
**    ****age at vaccination(days/months), number of doses, up-to-date status (UTD)**
**Providers’ information (e.g., facility type, VFC participation)**

## Contents of NIS PUF (cont.)

***Composite vaccine-specific and vaccine series flags with up-to-date status***
***Sampling weights and key analytic variables***
**Sampling weights: **
**    ****Children with household interview (HY_WGT)**
**    ****Children with Provider data(W0)**
**Stratum + PSU identification variables**

## Summary

***Complicated Childhood Immunization Schedule ***
***No single source is 100% complete or accurate***
***<1% children were observed with discrepancies in household and/or provider reported data***
***Data reported from written ‘shot card’ are of high quality; Shot card use varies by State/IAP area***
***High vaccine-specific agreement in the UTD status between HH ‘shot card’ and provider report(s) when***_*** HH reported UTD***_

## Substantial underreporting in vaccination coverage rates from the household reports

Strongly supports the use of provider-reported immunization histories in computing vaccination coverage estimates

Official estimates from provider reports may slightly underestimate coverage rates due to missing provider data; however, provider data produce comparable estimates across IAP areas

***Strongly supports the use of provider-reported immunization histories in computing vaccination coverage estimates***
***Official estimates from provider reports may slightly underestimate coverage rates due to missing provider data; however, provider data produce comparable estimates across IAP areas***

***Summary***

## Visit the Web Sites

***For information on the NIS PUF and coverage estimates ***
_**[http://www.cdc.gov/nis](http://www.cdc.gov/nis)**_
_**[http://www.cdc.gov/nip/coverage](http://www.cdc.gov/nip/coverage)**_