Evolution of a Phoenix: 

Re-engineering the 

Survey of Income and Program Participation – Event History Calendar 

(SIPP-EHC) 

Jason Fields 

US Census Bureau 

BLS CE Survey Redesign Seminar 

Washington, DC 

July 17th 2012, BLS 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 1 
SIPP Re-engineering 

Implement Improvements to SIPP 

 - reduce costs 

 - reduce R burden 

 - improve processing system 

 - modernize instrument 

 - expand/enhance use of admin records 

Key Design Change: 

 - annual interview, 12-month reference pd., 

 Event History Calendar methods 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 2 
EHC Interviewing 

Human memory 

 - structured/organized 

 - links and associations 

EHC exploits memory structure 

 - links between the occurrence and timing of events 

EHC encourages active assistance to Rs 

 - flexible approach to help elicit an autobiographical “story” 

Goal – facilitate the recall and reporting of accurate monthly level 

information over the full prior calendar year. 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 3 
SIPP Re-engineering Field Test Plans 

- Proof of concept test - Sample, design, results 

 - 2008 paper and pencil reinterview test 

- EHC CAPI test - Hurdles and highlights 

 - 2010 Integrated Blaise and C# instrument prototype 

- CAPI revised test 

 - 2011 Test improvements to the wave 1 instrument, training, and 

 expand sample to all regional offices. 

 - Interwave locating experiment 

 - 2012 Test wave 2 concepts and instrument, examine movers and 

attrition issues, dependent interviewing methods and refine training. 

 - 2013 wave 3 interview allows returns to household and additional 

mover and dependent interviewing evaluation 

- 2014 SIPP-EHC is the production SIPP instrument 

 - A new Phoenix "A Bird of a Different Color" 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 4 
Re-contact 

Activities 

2013SIPP-EHC 

Wave 3 Inst. 

SIPP 2008 Panel – Waves 1 – 12 (Rotation 1 field months) 

2008

Sep 

2009

Jan - May - Sep - 

2010

Jan - May - Sep - 

2011

Jan - May - Sep - 

2012

Jan - May - Sep - 

2013

Jan - May - Sep - 

2014

Jan 

Paper Test Eval. Analysis 

2010 SIPP-EHC 

Instrument Dev. 

Processing and 

Evaluation 2010 SIPP-EHC

Dress Rehearsal 

Ref. Period – CY2009 

Field work 

Extension w13-w16 

2011 SIPP-EHC 

Inst. Dev. 

2011SIPP-EHC 

Dress Rehearsal 

Ref. Period – CY2010 

Processing and 

Evaluation 

Field work 

Wave 1 

2012 SIPP-EHC Wave 2 Inst. 

2012 SIPP-EHC 

Ref. Pd – CY2011 

Processing and 

Evaluation 

Field work 

Wave 2 

2013 SIPP-EHC 

Ref. Period – CY2012 

Processing and 

Evaluation 

Field work 

Wave 1 

2014 SIPP Panel 

Inst. Refinement 

Production 

2014 SIPP Panel Wave 1 

Ref. Period – CY2013 

Field work 

Wave 1 

6 Regions – 8k hhlds – 10 States 

12 Regions-4k hhlds-20 States–Test of Wave 1 & 2 [Feedback and movers] 

Materials 

Prep 

12 Regions-Full Production Panel 2010 based sample 

2012 SIPP-EHC CARI 

SIPP-EHC Development and Implementation for 2014 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 5 
Developing the CAPI SIPP-EHC : Challenges Faced 

Developing new technical capacity. 

 - experimentation 

 - limitations 

 - evolution 

Crisis planning. 

 - limited lead time / preparation 

 - changing goals and required flexibility 

New procedures. 

 - training and acceptance 

 - development and refinement of procedures 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 6 
SIPP-EHC CAPI Comparisons 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 7 
Probability of Employment and Program Participation and 

Monthly Time Trend 2010 SIPP-EHC and 2008 Panel data 

matched in MSIPP 

8 

An Initial Evaluation of the 2010 Field Test of Re-Engineered SIPP. SIPP-EHC Data Evaluation Workgroup, March 2011 

(Rebecca Chenevert, Ashley Edwards, Renee Ellis, Jason Fields, Graton Gathright, David Hedengren, Carolyn Hronis, 

Jeongsoo Kim, Lindsay Monte, Daniel Perez-Lopez, Michelle Sandhoff, Marina Vornovytskyy, and Rachael Walsh)

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 8 
9 

Probability of Employment and Program Participation and 

Monthly Time Trend 2010 SIPP-EHC and 2008 Panel data 

matched in MSIPP 

An Initial Evaluation of the 2010 Field Test of Re-Engineered SIPP. SIPP-EHC Data Evaluation Workgroup, March 2011 

(Rebecca Chenevert, Ashley Edwards, Renee Ellis, Jason Fields, Graton Gathright, David Hedengren, Carolyn Hronis, 

Jeongsoo Kim, Lindsay Monte, Daniel Perez-Lopez, Michelle Sandhoff, Marina Vornovytskyy, and Rachael Walsh)

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 9 
Percent Enrolled by Age - Adults Age 15 and Over: 

2010 SIPP Geo-Matched versus SIPP-EHC

*= Estimates are significantly different at the p < .10 level 

Source: U.S. Census Bureau, Survey of Income and Program Participation, 2008 Panel Waves 5-8, 

2011 SIPP-EHC.

0

1 0

2 0

3 0

4 0

5 0

6 0

7 0

8 0

9 0

100

1 5 16 and

1 7

18 and

1 9

20 and

2 1

22 to

2 4

25 to

2 9

30 to

3 4

35 to

4 4

45 to

5 4

55 and

over

SIPP Geo-Matched SIPP-EHC

* 

Measuring School Enrollment in the 2011 SIPP-EHC Field Test. Stephanie Ewert and Sarah Crissey, January 2012 - FCSM

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 10
0.0

2.0

4.0

6.0

8.0

10.0

12.0

14.0

16.0

18.0

20.0

Jan Feb Mar Apr May June July Aug Sep Oct Nov Dec

Geo Matched SIPP

SIPP-EHC 2011

SIPP-EHC 2010

Source: U.S. Census Bureau, Survey of Income and Program Participation, Panel 2008 Waves 5-8, 2010 SIPPEHC, 2011 SIPP-EHC.

Measuring School Enrollment in the 2011 SIPP-EHC Field Test. Stephanie Ewert and Sarah Crissey, January 2012 - FCSM

Percent Enrolled by Month - Adults Age 15 and Over: 

SIPP 2008, SIPP-EHC 2010, and SIPP-EHC 2011 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 11
Mover Rates – Preliminary Evaluation of 2008 SIPP 

and 2011 SIPP-EHC 

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Percent Moved

Movers by Month for the 2010 

Calendar Year

2008 SIPP

2008 SIPP Matched 

2011 Geography

2011 SIPP-EHC

Evaluating Residence History Information in the SIPP-EHC. Matthew Marlay and Peter Mateyka, January 2012 - FCSM

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 12
• Reduced capture of income sources although higher income amounts 

reported in the SIPP-EHC 

• Higher monthly poverty rates, greater number of poverty spells for 

unrelated individuals, and longer duration poverty spells captured in 

SIPP-EHC 

- Variation by calendar years as well as family types 

• Respondents in CY2009 

- SIPP-EHC less likely to exit an initial poverty spell than 

respondents in production SIPP 

- no significant difference across instruments in CY2010 

• SIPP-EHC does a better job of capturing poverty exits within the 

interview period 

Initial Poverty Measurement Evaluation Findings 

Ashley Edwards, 2012 – Presented at the 2012 PAA 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 13
Administrative Data 

Comparisons 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 14
False Negatives – 

(Does not report receipt when Administrative Data Does) 

2010 SIPP-EHC and 2008 Panel data 

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

SSI OASDI MEDICARE SNAP

SIPP

SIPPEHC

15

Accuracy of Reporting about Program Participation in SIPP-EHC Field Test Data. 

Graton Gathright, Martha Stinson, and Lori Reeder, FCSM 2011 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 15
False Positives – 

(Reports receipt when Administrative Data does Not) 

2010 SIPP-EHC and 2008 Panel data 

0.0%

5.0%

10.0%

15.0%

20.0%

25.0%

30.0%

35.0%

40.0%

45.0%

50.0%

SSI OASDI MEDICARE SNAP

SIPP

SIPP-EHC

16

Accuracy of Reporting about Program Participation in SIPP-EHC Field Test Data. 

Graton Gathright, Martha Stinson, and Lori Reeder, FCSM 2011 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 16
Paradata Evaluations 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 17
Paradata elements currently in use with SIPP-EHC 

• audit trail data from the Blaise/C# instrument 

• certification test for interviewer training 

• interviewer characteristics 

• census experience 

• prior SIPP experience 

• supervisory status 

• demographics 

• contact history instrument 

• mileage, case load, supervisor observation 

• neighborhood observation 

Paradata - Use for Evaluation and 

Development of the SIPP-EHC 18
Interview Burden and Length 

2010 SIPP-EHC and 2011 SIPP-EHC 

2010

SIPP-EHC 

2011

SIPP-EHC 

Questions Asked

Full Sample 

Mean 280.5 281.5

Min 156 138

Max 492 445

Adult, Non-Type Z 

Mean 300.7 305.3

Min 173 152

Max 492 445

Interview Length

Personal Interview Length (Min.) 42.36 32.06

Household Interview Length (Min.) 100.84 76.75

19

Source: U.S. Census Bureau, Survey of Income and Program Participation-Event History Calendar (SIPP-EHC), 2010 and 2011. 

Reducing Respondent Burden: Evaluating the Progress of the SIPP-EHC

Rachael Walsh and Jason Fields, FCSM 2012

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 19
Interviewer Learning Curve 

30

35

40

45

50

55

60

1 5 10 15 20 25 30 35

Predicted Interview Length (minutes)

Interviewer Experience (# of personal interview)

2010

2011

Interviewer Effect on Conversational Interviews: Results from the 2010 & 2011 SIPP-EHC

Rachael Walsh, PAA 2012

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 20
2011 Interviewer Progress Curve for Interview Length 

by Certification Exam Score. 

Certifying Interviewers: The role of testing interviewers to improve data quality 

Rachael Walsh, 2012

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 21
2011 Interviewer Progression Curve for Person 

Non-Response by Certification Exam Score. 

Source: U.S. Census Bureau, Survey of Income and Program Participation-Event History Calendar (SIPP-EHC), 2011. 

 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 22
A Challenge - Dependent Data & Seams 

• Dependent Interviewing – a critical component. 

• Why use dependent interviewing (DI)? 

– Bias 

• Seam 

• Recall 

– Burden 

• Interview length 

• Cognitive challenge 

– Data quality 

• Longitudinal consistency 

• Corrections / improvements 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 23
SIPP-EHC Dependent Interviewing (DI) 

• Differentiates Spells expected to continue from closed 

spells that the interviewer should use as memory cues. 

• Utilizes DI to provide bounding to the Interviewer / 

Respondent timeline resolution. 

• CQ sections utilize DI to remind and continue with 

resolving current year‟s monthly receipt. 

• Focus on minimizing the impact to processing and 

assisting data handling tasks. 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 24
SIPP–EHC 

Contents 

25 Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 
2011 SIPP-EHC Completed Calendar 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 26
2012 SIPP-EHC Calendar w/ Dependent Data 

Complete spells 

representing information 

that was reported last 

time, but not active at the 

time of the interview. 

Should be left alone. 

Dependent 

Text 

The 

information 

displayed here 

changes 

depending on 

the dependent 

information. 

Provisional spells 

representing information 

that was reported last 

time, and current at the 

time of the interview. 

Needs to be extended. 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 27
DI Constraints and Considerations 

• 2004 SIPP DI success but processing challenge 

– Over 3100 dependent items 

• Formatting necessary bring data from instrument output 

to instrument input. 

• 2012 SIPP-EHC 

– Provisional = 197 (limited data plus processing created 

longitudinal Job and Residence ID‟s) 

– Complete = 339 (passes full data back in addition to 

processing created longitudinal Job and Residence ID‟s) 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 28
DI Constraints and Considerations 

• Including DI has many sources for error in implementation 

– Reformatting data from the instrument to the processing system 

– Translating months 13 through the interview into months 1 to n 

during the subsequent wave reference period. (Months, changes in 

hours/earnings, and weeks for transitions) 

– Preparing ascii input file for the Blaise instrument 

– Translating input file into proper fields and Blaise function 

– Testing and verifying proper feedback functions 

– Field verification 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 29
Evaluation Plans 

• Issues related to successful use of dependent data 

• Flexibility for Interviewer/Respondent interaction 

• False transitions and mis-timed transitions – recall or 

seams. 

• Mover individuals 

• Changing respondents 

• Respondent Identification Policy 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 30
Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 31

Handling the Unexpected 

- Instrument development testing and retesting 

- Delay in fielding 2012 SIPP-EHC

- Disruption to training development 

- Non-response and RIP 

- Some comparisons are made difficult 

Final Redesign 

- 2013 SIPP-EHC „last dry run,‟ implemented CCB 

Redesign Evaluation 

- Continued use of survey comparisons, administrative 

evaluations, and paradata 
Current Status 

• Interviewing for 2012 SIPP-EHC Wave 3 in all 12 regional office 

areas (20 States) 

– Interviewing conducted in May and June 2012. 

• Reviewing and making changes to content and design (Dependent 

Interviewing for 2013 – Wave 3). 

• Reviewing and revising training materials and methods 

• Planning for 2014 Production Implementation of SIPP-EHC 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 32
Lining the nest. 

- crosswalks 

- workshops 

- data utilities 

- SIPP small grants 

- users groups 

- concerns to evaluate 

- overhaul public perception 

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 33
February 2007 

April/May 2007 

Contact: Jason.M.Fields@Census.Gov 

SIPP-EHC 2012 SIPP-EHC 2010 

URL: http://www.census.gov/sipp

Evolution of a Phoenix: Re-engineering 

the SIPP-EHC 34