1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Examining Relationships between 

Doorstep Concerns and Edit Check 

Frequency in the CE Interview Survey

Taylor J. Wilson

Economist

Bureau of Labor Statistics

American Association for Public Opinion Research (AAPOR)

18 May 2019
2 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
4 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Statement of Purpose

 New machine readable error audit trails made accessible to 

internal researchers.

 Enables researchers to explore new questions that previously 

could not be supported with data. 

 This research explores whether interviewer doorstep concerns 

are related to the frequency of edit checks over the course of 

the interview. This is just one of many potential questions that 

can be investigated!
5 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Introduction

 Consumer Expenditure Interview Survey

Administered in person for at least the first 

interview

A household is selected to be in the survey 

four times

Entirely voluntary with approximately a 60% 

response rate

Designed to capture large, recurring expenses

– Diary Survey designed to capture smaller expenses
6 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data: Error Audit Trails

 A new dataset that tracks errors 

triggered in the survey instrument by 

the interviewer.

 Errors are classified by type

Hard Edit Checks (You can’t do that!)

Soft Edit Checks (Are you sure this is right?)

 Interviewer response to edit checks

Suppress, Go to, and Escape
8 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data: Hard and Soft Edit Checks

 Hard Edit Check Example:

 Soft Edit Check Example:

• WARNING! You are about to DELETE the ENTIRE current household 

and create a NEW REPLACEMENT case.

• THE VALUE ENTERED IS UNUSUALLY HIGH OR LOW PLEASE VERIFY

• You must enter all ten digits of the contact person's phone number. No 

spaces.
9 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Data: Contact History Attempt

 Contact history attempt data are created by the interviewer. 

 Track important information related to the interview

Outcome of the contact attempt? (e.g., completed interview, 

demolished household, partial interview, etc.)

Were there doorstep concerns? (e.g., hostile, too busy, etc.)

 For this analysis we are interested in completed interviews 

with doorstep concerns for the most recent releasable quarters 

of data (2018 Q1 and 2017 Q4).
10 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
11 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Methods: Hypotheses

 In the ideal case, any given door step concern would not impact 

the number of edit checks. 

 In reality, interviewers are human so some doorstep concerns may 

influence the frequency of edit checks. A rushed respondent may 

try and speed through the interview resulting in more typos.

 Reduction in edit checks is also a plausible outcome. A respondent 

who was skeptical but convinced by the interviewer may be easier 

to work with than an apathetic respondent. 
12 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Methods: Define Response Variables

 Total number of edit 

checks

Hard

Soft

 Total number of 

resolutions

Suppress

Go to 

Escape Frequency of Suppressions
13 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Methods: Poisson Regression

 Count data lend themselves 

to a Poisson regression

Assumption that the response 

is Poisson distributed

– Mean and variance are equal

 Potentially overdispersed

Greater variability than is 

expected by the model

Poisson Distributions by Parameter Values
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Methods: Poisson Regression

 Overdispersion is present in the response variables due to the 

excess number of zeros. 

 Therefore, the analysis will be modified by using a zero-inflated 

Poisson regression. 

Deals with the excess zeros by estimating them separately from the 

Poisson process. 

Zero-inflated Negative Binomial Model used as a robustness check.

 Household demographics used as controls (e.g., race, region, 

income, etc.)
15 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Methods: Poisson Regression

 Produce incidence rate ratios (IRR) to determine the 

multiplicative likelihood compared to the baseline. 

1.5 IRR, for example, indicates that the result is 1.5 times as likely 

compared to the control group. 

 Detectable differences vs. Meaningful differences

Large samples often lead to over rejection of hypotheses

Consider both magnitude and significance of result

– Detectable IRR of 0.98 doesn’t matter much in a practical sense
16 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Results: Descriptive Statistics

 75% of interviews reported “No Concerns” 

There is a small, but statistically detectable reduction in edit checks for 

households reporting no concerns. 

 The most common concern was “too busy” at 11% of the 

sample.

 The least common concern was “Hostile or threatens FR” at 

0.1% of the sample. (A positive thing)

 The average number of edit checks is about 5.2 per interview 

with a standard deviation of 6.4.
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Results: Most Common Doorstep Concerns
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Results: IRR for Errors/Edit Check

 Very few doorstep concerns resulted in any changes from the 

control group.

Generally, homogenous interviewer behavior is preferred.

 Only ‘intends to quit the survey’ resulted in a statistically 

significant increase in frequency of edit checks controlling for 

demographics. (IRR = 1.4) : Potentially satisficing on the part of 

the interviewer?

 ‘Not interested’ and ‘Too many interviews’ both had 

meaningful* IRR < 1 
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Reduced frequency of 

errors

Increased frequency of 

errors

No deviation from 

control

Not interested

Too many interviews

Survey is voluntary

Privacy Concerns

Intends to quit the survey Too busy

Interview takes too much time

Scheduling difficulties

Anti-government concerns

Do not understand survey

Hang-up/Slams door

Family issues

Gave information last time

Too many personal questions

Other
22 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Results: Incidence Rate Ratios for Suppressions

 Since ‘Escape’ is an incredibly sparse solution for dealing with 

errors (only 9% of all error resolutions), Suppressions were 

modeled.

 Again, almost all doorstep concerns did not result in any 

changes from the control. 

 ‘Survey is voluntary’ and ‘Too many interviews’ both correlate 

with a reduced frequency of suppressions at meaningful levels.
23 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
24 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Reduced frequency of 

Suppressions

Increased frequency of 

Suppressions

No deviation from 

control or sample size 

too small

Survey is voluntary

Too many interviews

Not interested

Too busy

Privacy concerns

Interview takes too much time

Scheduling difficulties

Survey is voluntary

Anti-government concerns

Do not understand survey

Hang-up/Slams door

Family issues

Gave information last time

Too many personal questions

Intends to quit the survey

Other
25 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Roadmap

 Introduction

 Data

 Methods

 Results

 Conclusion
26 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Conclusions

 The majority of door step concerns are not associated with the 

number of edit checks or suppressions. 

 Of the top 5 most common doorstep concerns, only “privacy 

concerns” was significantly associated with edit checks.

 Observing a reduction in errors or suppressions could be a result 

of interviewer effects (i.e., interviewers who get completed 

interviews from challenging households may be 

characteristically different)
27 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Conclusions: Limitations

 Interviewer characteristic data are not available to researchers 

so they cannot be included in the model.

Some interviewers could be better than others at getting respondents 

to complete the interview. 

Could be a function of experience, general safety of the area, etc. 

 Data analyzed are the most recently available. 
28 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Questions?
Contact Information

29 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Taylor J. Wilson

Economist

Bureau of Labor Statistics

www.bls.gov/cex

202-691-6550

wilson.taylor@bls.gov