# Presentation: 707113

## The SIPP Event History Calendar Field Test:  Analysis Plans and Preliminary Report 

- Jeff Moore
- Statistical Research Division, U.S. Census Bureau
- Jason Fields
- Housing and Household Economic Statistics Division, U.S. Census Bureau
- ASA/SRM SIPP Working Group Meeting
- September 16, 2008

## Overview  

- Background:
- 	- SIPP “re-engineering”
- 	- event history calendar (EHC) methods
- Goals and Design of the EHC Field Test
- Evaluation Plans
- Preliminary Results *[not yet available]*

## SIPP Re-Engineering

- Implement Improvements to SIPP
- 		- reduce costs
- 		- reduce burden
- 		- improve processing system
- 		- modernize instrument
- 		- expand/enhance use of admin records
- Key Design Change:  Annual Interviewing

## EHC Interviewing (1)

- Human Memory
- 		- structured/organized
- 		- links and associations
- EHC Exploits Memory Structure
- 		- links between to-be-recalled events
- 		- coherence, consistency, sequence
- EHC Encourages Active Assistance to Rs

## EHC Interviewing (2)

- Evaluation:  EHC vs. Q-List Comparisons
- 		- various methods
- 		- in general:  positive data quality results
- BUT, Important Research Gaps
- 		- data quality for need-based programs?
- 		- extended reference period?

## Field Test Goals & Design

- _Basic Goal:_
- Can an EHC interview collect data of comparable
- (or better) quality than standard SIPP?
- 		- month-level data
- 		- one 12-month ref pd interview vs. three
- 		  4-month ref pd interviews
- 		- especially for need-based programs
- _Basic Design:_
- EHC re-interview of SIPP sample HHs

## Design Details (1)

- _Main Sample:_
- SIPP Wave 10-11-12 Interview Cases
- 	- reported on CY-2007 via SIPP [Fig. 1]
- _Supplemental Sample:_
- SIPP Wave 8 Sample Cut Cases
- 	- dropped from SIPP in 2006; “unprimed”
- EHC Re-Interview in 2008, about CY-2007

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (2)

- Two Sites
- 	- Illinois (all)
- 	- Texas (4 metro areas)
- N = 1,945 Addresses
- 	- cooperating HHs in SIPP
- Sample Distribution:

| ILLINOIS | (n = 914) | TEXAS | (n = 1,031) |
| --- | --- | --- | --- |
| W10-11-12 | Sample Cut | W-10-11-12 | Sample Cut |
| 487 | 427 | 609 | 422 |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (3)

- Administrative Records
- (for some characteristics, and with R approval)
- 	- Medicare
- 	- Social Security retirement, disability
- 	- SSI
- 	- TANF
- 	- Food Stamps
- 	- [Medicaid?]

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (4)

- EHC Questionnaire [handout]
- 	- paper-and-pencil
- 	- 12-month, CY-2007 reference period
- 	- selected SIPP topics (“domains”)
- 	- start with landmark events
- 	- within domains, anchor on “now”
- 	- month-level (at least) detail
- Sample of Addresses, Not People
- 	- post-interview clerical match to SIPP

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (5)

- $40 Incentive, Non-Contingent
- Same Response Rules as SIPP
- 	- EHC interview for all adults (15+)
- 	- self-response preferred (proxy permitted)
- Field Staff: Census Bureau FRs
- 	- most with some interview experience
- 	- ~1/3 with SIPP experience
- 	- 3-day training on EHC methods

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (6a)

- Field Period:  Mid-April thru Late June 2008
- Outcomes:
- 	- 1,627 HH interviews
- 	- 3,318 individual EHC interviews
- 	- 2,747 EHC Rs matched to SIPP

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Design Details (6b)

|  | ILLINOIS |  | TEXAS |  | TOTAL |  |
| --- | --- | --- | --- | --- | --- | --- |
|  | W-10-11-12 | Sample Cut | W-10-11-12 | Sample Cut | W-10-11-12 | Sample Cut |
| Address Sample
(1,945) | 487 | 427 | 609 | 422 | 1,096 | 849 |
| Completed HH Interviews
(1,627) | 417
(91%) | 347
(89%) | 518
(91%) | 345
(92%) | 935
(91%) | 692
(91%) |
| Completed Individual EHC Interviews
(3,318) | 866
(99%) | 707
(98%) | 1,056
(99%) | 689
(99%) | 1,922
(99%) | 1,396
(99%) |
| Interviewed Adults Matched to SIPP
(2,747) | 767
(89%) | 588
(83%) | 890
(84%) | 502
(73%) | 1,657
(86%) | 1,090
(78%) |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Evaluation Plans (1)

- Compare SIPP and EHC Survey Reports
- 	- same people
- 	- same time period
- 	- same characteristics
- Data Quality Comparison using Admin Records
- (later)
- Evaluation of “Priming” Bias

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Evaluation Plans (2)

- Other Evaluations
- 	- R debriefing form
- 	- FR “case report” debriefing form
- 	- FR debriefing focus groups
- 	- interview observations
- Focus on EHC Interview Process

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (1a)

|  |  | SIPP | Report |
| --- | --- | --- | --- |
|  |  | No | Yes |
| EHC | No | a | b |
| Report | Yes | c | d |

- 2x2 Consistency Table for “Participation”
- (Employed?  Enrolled?  Insured?  etc.)
- 	- for each characteristic
- 	- for each month of CY-2007
- 	- unweighted / unedited data

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (1b)

|  |  | SIPP | Report |
| --- | --- | --- | --- |
|  |  | No | Yes |
| EHC | No | a | b |
| Report | Yes | c | d |

- b=c	 equivalent data quality
- 		   (high if (b+c)/N~0; low if (b+c)/N is large)
- b>c	 EHC “underreporting” (rel. to SIPP)
- b<c	 SIPP “underreporting” (rel. to EHC)

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (1c)

|  |  | SIPP | Report |
| --- | --- | --- | --- |
|  |  | No | Yes |
| EHC | No | a | b |
| Report | Yes | c | d |

- Patterns of Consistency/Inconsistency
- 	- b>c for most months?  b<c?  mixed?
- 	- early months vs. late months?

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (2a)

- Total Reported Months of “Participation”
- 	- by Qtr / combined Qtrs / whole year

|  |  | SIPP Participation Months – Q(n) |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | 0 | 1 | 2 | 3 |
| EHC
Participation
Months – Q(n) | 0 |  |  |  |  |
|  | 1 |  |  |  |  |
|  | 2 |  |  |  |  |
|  | 3 |  |  |  |  |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (2b)

- Patterns of Off-Diag Clustering Across Time
- 	- above for most Qtrs?  below?  mixed?
- 	- early Qtrs vs. late Qtrs?

|  |  | SIPP Participation Months – Q(n) |  |  |  |
| --- | --- | --- | --- | --- | --- |
|  |  | 0 | 1 | 2 | 3 |
| EHC
Participation
Months – Q(n) | 0 |  |  |  |  |
|  | 1 |  |  |  |  |
|  | 2 |  |  |  |  |
|  | 3 |  |  |  |  |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (2c)

- Patterns of Off-Diag Clustering Across Time
- 	- above for most Qtrs?  below?  mixed?
- 	- early Qtrs vs. late Qtrs?
- # Reporting *At Least 1* Month of Participation

|  |  | SIPP Participation Months – Q(n) |  |
| --- | --- | --- | --- |
|  |  | 0 | 1+ |
| EHC
Participation
Months – Q(n) | 0 |  |  |
|  |  |  |  |
|  | 1+ |  |  |
|  |  |  |  |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (3)

- Other “Participation” Comparisons:
- 	- *ANY* need-based program participation?
- 	  (by month / Qtr / combined Qtrs / year)
- 			 or
- 	- *ANY* health insurance coverage
- 			 [etc.]
- 	- alignment/sequencing across domains
- 	  (e.g., moves & jobs, employment &
- 	   health insurance, etc.)

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (4a)

- Month-to-Month Transitions (yesno; noyes)
- SIPP’s Staggered Interview Design:
- 	- each month-pair is a “seam” for 1⁄4 sample
- 	- each month-pair is off-seam for 3⁄4 sample
- Compare Reporting of Transitions

|  | Jan-
Feb | Feb-
Mar | Mar-
Apr | Apr-
May | May-
Jun | Jun-
Jul | Jul-
Aug | Aug-
Sep |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SIPP – Seam Cases |  |  |  |  |  |  |  |  |
| SIPP – Off-Seam Cases |  |  |  |  |  |  |  |  |
| EHC |  |  |  |  |  |  |  |  |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (4b)

- Seam Bias:
- 	- too much Δ across interview “seams”
- 	- too little Δ within a single interview
- EHC Δ rates below SIPP’s (seam), and above SIPP’s (off-seam)  Improved Quality

|  | Jan-
Feb | Feb-
Mar | Mar-
Apr | Apr-
May | May-
Jun | Jun-
Jul | Jul-
Aug | Aug-
Sep |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| SIPP – Seam Cases |   (++)   |  |  |  |  |  |  |  |
| SIPP – Off-Seam Cases |   (- -)   |  |  |  |  |  |  |  |
| EHC |   (0)   |  |  |  |  |  |  |  |

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (5a)

- Income Amount Report Comparisons
- 	- unemployment benefits
- 	- disability income ($)
- 	- workers’ comp
- 	- Social Security ($)
- 	- Medicare Part B deduction ($)
- 	- TANF ($)
- 	- Food Stamps ($)
- 	- SSI ($)
- ($)=admin records

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (5b)

- $$ Comparison is Less Straightforward
- Continuous $$ Variable 
- 	- arbitrary definition(s) of “agreement”
- 	- disagreements are directional
- Limited to “Yes/Yes” Cases

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Compare SIPP/EHC Reports (5c)

- $$ Reporting Comparisons
- 	- mean amount (EHC; SIPP; difference)
- 	- levels of correspondence
- 			(e.g., ±5%; ±5-10%; ±10-25%;
- 			 ±25-50%; >±50%)
- 	- direction of differences
- 			($EHC > $SIPP; $EHC=$SIPP (±1%);
- 			 $EHC < $SIPP)
- 	- timing of amount changes

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Assessment of “Priming” (6a)

- W-10-11-12 Rs Provide CY-2007 Data Twice
- 	- first SIPP, then EHC
- Are Their EHC Reports Biased?
- 	- e.g., more accurate EHC response
- 	- could bias field test interpretation
- Control Group:  W-8 Sample Cut
- 	- last SIPP response in Jun-Sep 2006
- 	- “unprimed” re: CY-2007 (_not_ SIPP content)

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Assessment of “Priming” (6b)

- Compare Distributions for Key Characteristics
- 	- e.g., monthly “participation” reports
- 	- weighted (sub-sampling; attrition)
- Similarity of Profiles  Extent/Nature of Priming Bias
- Admin Records for Some Characteristics
- 	- meaning of distribution differences
- 	- may also reveal hidden quality diffs

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Guidance, Questions, Advice...

- Questions?
- Thoughts/Comments...?
- 	- on the evaluation approach?
- 	- about additional analyses?
- 	- about how to weigh evidence from the
- 	  field test in deciding whether or not to
- 	  adopt a 12-month EHC?

**Notes:**
switch gears to SIPP	
US Census Bureau longitudinal survey program
continuous since 1984; new panel every (1,2,3,4) years
focus:	income / wealth / poverty
participation in gov’t-sponsored income transfer programs 
dynamics (flows into & out of those programs) / impacts
i’er-administered, via computer-assisted q’aire (since 1996)
4-month interview cycle; monthly data
LOTS of seam bias problems
currently completing the 12th  (final) wave of a panel started in 2004; new panel in Feb 08;  then:  re-engineered (2011?  2012?)

## Thank you very much!

- jeffrey.c.moore@census.gov
- 301-763-4975