RTI International is a registered trademark and a trade name of Research Triangle Institute. www.rti.org

Studying Changes in an Ongoing In-Person 

Survey Using a Web Survey

Stephanie Eckman

Research supported by ASA / NSF / BLS Fellowship
Has anyone in this household had 

any expenses for:

Oil change, lubrication or oil filter?

Yes

In what month?

What was the total cost?

Did this expense include labor?

May

$49.99

No

Has anyone in this household had 

any expenses for:

Tire purchases or mounting? Yes?
National Academy Review

▪ This sequence is repeated dozens of times during each interview and 

may affect respondent behavior. 

▪ It seems likely that respondents learn quickly...that the interview will 

last longer if they answer “yes” to these screening questions.

▪ Fifty percent of field representatives said that it happened frequently 

or very frequently.

Measuring What We Spend: Towards a New Consumer Expenditure Survey 

(2013, 84-85)
Do Rs engage in motivated misreporting 

in Wave 1 of the CE?
Interleafed Format

▪ Filter – Follow up – Follow up ▪ Filter – Follow up – Follow up ▪ Filter – Follow up – Follow up
▪ Filter ▪ Filter ▪ Filter ▪ Filter – Follow up – Follow up – Follow up – Follow up – Follow up – Follow up

Grouped Format
Previous research on MM 

▪ CE-like questions:

– Kreuter et al (2011) The Effects of Asking Filter Questions in Interleafed versus Grouped Format. SMR

– Eckman et al (2014) Assessing the Mechanisms of Misreporting to Filter Questions in Surveys. POQ

– Daikeler et al (2019) Motivated Misreporting in Smartphone Surveys. SSCR

– Bach et al (2019) Misreporting Among Reluctant Respondents. JSSAM

▪ CE: underreporting in later waves

– Bosley et al (1999), Shields & To (2005), Yan & Copeland (2010), Bach & Eckman (2019)

▪ CE: MM in Wave 1

– McBride (2013)

Barring experiment within CE, 

how can we estimate underreporting?

Full list: http://www.stepheckman.com/project/motivated_misreporting
How to Measure MM in CE

▪ CE asks filters only in interleafed format

▪ Web survey in both formats

X Interleafed Grouped

Consumer 

Expenditure ● ●

Nonprob

Web Survey

● ●

● ●
Similar Imputation Studies

▪ Rendall et al (2013)

▪ Powers et al (2005)

Christensen et al (2006)

Kolenikov & Kennedy (2014)

Park et al (2016)

▪ Kim et al (2020)
Similar Imputation Studies

▪ Rendall et al (2013)

▪ Powers et al (2005)

Christensen et al (2006)

Kolenikov & Kennedy (2014)

Park et al (2016)

▪ Kim et al (2020)

Survey X1 X2 Y

Survey 1 ● ●

Survey 2 ● ● ●

Model Y|X1, X2

Variables are jointly observed

Cross-Survey imputation
Similar Imputation Studies

▪ Rendall et al (2013)

▪ Powers et al (2005)

Christensen et al (2006)

Kolenikov & Kennedy (2014)

Park et al (2016)

▪ Kim et al (2020)

Mode X Yweb Yphone

Web ● ●

Phone ● ●

Impute responses in better mode
Similar Imputation Studies

▪ Rendall et al (2013)

▪ Powers et al (2005)

Christensen et al (2006)

Kolenikov & Kennedy (2014)

Park et al (2016)

▪ Kim et al (2020)

Survey X Y

Probability ●

Nonprobability ● ●

Impute variable of interest on 

probability survey
How to Measure MM in CE

▪ Eckman (2020) design:

X Interleafed Grouped

Consumer 

Expenditure ● ●

Nonprob

Web Survey

● ●

● ●
Web Survey Questionnaire

▪ Section 1: Demographics

▪ Section 2: Household characteristics

▪ Section 6: Income

Section # Filters # Follow Ups

Utilities 5 4

Clothing 6 5

Insurance 5 5

Order of sections 

randomized

Order within sections 

forwards/backwards
Imputation Approach

▪ Multiple imputation of Yes/No to filter question

– 6 models

– 5 predictions from each model

▪ Predictors:

– Format (grouped or interleafed)

– Survey (CE or web)

– Item 

– Section order

– Respondent & HH characteristics

Pseudo-R2: 0.20-0.22

AUC: 0.80
Compare Background Variables

23 variables

20 with significantly different distributions 

CE vs Web

Good representation in each category
Compare Purchases
Results

Interleafed

% Yes

Grouped

% Yes

Difference

% points

Consumer 

Expenditure 33.5

Web Survey 36.8 42.9 6.1 %points

(0.80)

5.1 %points

(1.5) 38.5

Utilities: 0.2 %points

Clothing: 9.9 %points*

Insurance: 5.1 %points*
Take Aways

▪ Evidence that motivated misreporting occurs in Wave 1

▪ Effect of misreporting on CPI unknown

▪ Imputation approach needs more theoretical support

▪ Approach could be used to assess hypothetical changes to ongoing 

surveys when experimentation not possible
Thank You

Stephanie Eckman

Fellow, Survey Research

RTI International

seckman@rti.org

http://stepheckman.com

@stephnie
Comparison of Imputation Models
Case base for each model is 50,848 filters

Each a variable interacted with each b variable