Innovative Uses of Paradata 

in the SIPP

Matthew Marlay

Holly Fee

Jason Fields

U.S. Census Bureau

Presented at 

FCSM

December 2015

This work is released to inform interested parties of ongoing research and to encourage discussion of 

work in progress. Any views or opinions expressed in the paper are the authors’ own and do not 

necessarily reflect the views or opinions of the U.S. Census Bureau.
Outline

 SIPP Overview

 Types and Uses of Paradata

 Certification Test

 Debriefing Instrument

 Audit Trails (Keystroke files)

 CARI

 Contact History Instrument (CHI)

 Unified Tracking System (UTS) 

2 
The Survey of Income and 

Program Participation (SIPP)

Longitudinal survey collecting data and 

measuring change for topics such as: 

 Economic Well-being

 Family Dynamics

 Education

 Assets

 Health Insurance

 Childcare

 Food Security

3 
2014 SIPP Survey Design

 Sample is multi-stage, stratified sample of 

the noninstitutionalized, civilian U.S. 

population

 Sample size 53,000 households

 4-year panel

 Conducted in waves, each 1 year long

 3-4-month interview period

4 
Interviewers

 Used 1,345 field representatives (FRs)

 ~300 new hires

 Remainder experienced interviewers

 Sample size was ~53,000 households

 Approximately 40 cases per FR

 Interviews all done in-person

 Yielded just under 30,000 completed cases

5 
Interviewer Training

 Two-day generic Census training

 New hires only

 Covers cross-survey skills

 Four-day classroom training

 All SIPP FRs

 Specific to SIPP

 Pre- and post-classroom self-study 

modules 

 Ends with certification test

6 
Certification Test

72 questions, divided into 8 sections:

1. Field Procedures (11)

2. Event History Calendar (12)

3. Programs (6)

4. Movers (15)

5. Content (10)

6. Noninterviews (6)

7. Medicare vs. Medicaid (7)

8. Blaise/Instrument Navigation (5)

7 
Certification Test

0

20

40

60

80

100

120

0%6%

11%17%22%28%33%39%44%50%56%61%67%72%78%83%89%94%

100%

Number of Takers

Score (in Percent)

8 
Certification Test

0

10

20

30

40

50

60

70

80

90

100

Percent

Certification test section

Mean Certification Test Scores by Regional Office

Total

Atlanta

Chicago

Denver

Los Angeles

New York

Philadelphia

9 
Certification Test

0

10

20

30

40

50

60

70

80

90

100

Percent

Certification test section

Mean Certification Test Scores by Position 

Total

FR

Other

10 
Debriefing Instrument

 Administered to FRs when they complete 

their SIPP interviews

 Asks them about the type and frequency 

of encountered problems in each section of 

SIPP interview:

 Question wording

 Response-related

 Technical

 Context/flow/redundancy

11 
Debriefing Instrument

 Most FRs report that they “never or 

almost never” experienced problems in 

the field

 Wording and response-related problems are 

reported more frequently than either technical 

or context, flow and redundancy problems 

 Most FRs (87%) provide feedback for at 

least one open-ended question

12 
0

10

20

30

40

50

60

70

80

90

100

Percent

Section

 Never or almost never

 Rarely

 Sometimes

 Often

 Always or almost always

Debriefing Instrument

13 
Debriefing Instrument

 Examples of resulting instrument updates:

 Update Health Insurance FAQ to better 

explain different kinds of health insurance

 Allow minimum values of $0 in most Assets 

fields

 New FR notes explaining how to handle 

timeshares, leased vehicles, and reinvested 

dividends

14 
Debriefing Instrument

 Debriefing one of the best ways we have 

of getting FRs’ direct feedback

 Item notes are also useful, but debriefing 

covers whole interviewing period

 These results help drive continuous 

improvement of our instrument, training, 

and processes

15 
Audit Trails

 Audit trail files are a record of all of the keystrokes 

entered by a field representative (FR) during an interview

 Audit trail files can be used to create paradata on 

such things as:

 Section timers, 

 Don’t know/refused counts, 

 Help screen calls,

 Checks encountered,

 Item-level notes left, and

 FR navigation throughout the instrument

16 
Audit Trails

Statistics (Completed cases)

Total Range

Variables Mean SD Median Min Max

Don’t Know (CTRL+D) 13.33 15.61 9.00 0 214

Refuse (CTRL+R) 4.46 15.60 0.00 0 385

Help Call Screens (F1) 0.37 0.92 0.00 0 24

Field Case Notes (F7) 0.76 2.90 0.00 0 120

Survey Time (in minutes) 102.41 51.89 92.68 6.9 682.73

18 
CARI 

 Computer-Assisted Recorded Interviewing

 FRs must obtain consent from each respondent to 

record the interview 

 Records interactions between Field Representatives 

(FRs) and respondents 

 The goal of CARI is to ensure the accuracy and quality 

of data collected

 Monitor and improve the FR’s performance

 Identify difficult or problematic questions 

20 
CARI 

We consider data inauthentic when 

interviewers do not collect them directly 

from respondent

 FRs may feel they already know the answer 

(and do not confirm it)

 FRs may skip a question

 Occasionally, FRs may fabricate a response 

entirely

21 
CARI 

Errors in question administration arise 

when questions are presented differently

 FRs do not read the questions as worded

 Speed/volume of interviewer’s voice does not 

match respondents’

 FRs do not probe or lead the respondent

22 
CARI 

 Behavioral problems arise when the 

interviewer’s conduct is inappropriate for 

a Census Bureau employee

 Off-topic personal discussions

 Overly task-oriented

 Unfriendly or hostile

 SIPP is dependent on respondents having 

a good relationship with FRs

23 
Contact History Instrument (CHI)

 Keeps a history of every contact attempt 

for every case

 Collects information about the kind of 

response received (if contact is made)

 Reluctant respondent, hostile respondent, etc.

 Also collects FR’s observation about 

housing unit/environmental conditions

 Condition of the sample unit

 Presence of bars on windows

24 
UTS

 Unified Tracking System

 A Census Bureau application that 

provides real-time survey information

 Allows for detailed monitoring of FRs, 

costs, and cases

 Can also track overall survey progress

 For example, have we contacted as many 

cases as we should, given where we are in the 

interviewing window?

25 
Conclusion 

 SIPP (and the Census Bureau more 

generally) has access to more paradata 

than we have ever had in the past

 Effective use of this paradata for FR 

monitoring and performance helps us 

improve interview administration

 We can also evaluate data quality and 

survey design, allowing continuous 

improvement and streamlining

26 
THANK YOU!

Matthew.C.Marlay@census.gov

Census.SIPP@census.gov

http://www.census.gov/sipp

27