Introduction

The Proposed Methods

Simulation Results

Next Step

Reducing Nonresponse Bias through

Responsive Design and External Benchmarks

Julia Lee

University of Michigan

July 17, 2012

Thesis committee: S. Heeringa, R. Little, T. Raghunathan, R. Valliant

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Goals of the Project

1 To improve respondent representativeness

2 To assess the nature of nonresponse

3 To adjust for nonresponse

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Outline

Introduction

The proposed method

Simulation results

Next steps

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Current Practice

Alternatives

Current Practice

Reduce nonresponse bias at the analysis stage:

Weighting class methods

Propensity score methods

Calibration

(Imputation)

Challenges:

Need nonrespondent information

Assume ignorable nonresponse pattern

Extreme and highly variable weights occur

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Current Practice

Alternatives

Alternatives

Reduce nonresponse bias at the design and data collection stages:

Actively control for nonresponse bias at design stage by

adaptively improving respondent representativeness.

Effectively use frame data, contextual data, paradata, and

benchmark information to obviate the need for nonrespondent

information.

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

Responsive Design Procedure

Objectives:

Obviate the need for nonrespondent information

Obtain more representative respondent pool

Terminology:

Benchmark survey: capture desired target population, such as

American Community Survey

Current Survey: survey that you are conducting

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

Responsive Design Procedure

Setting: Surveys with multi-phase data collection

The procedures:

1 Complete first phase of data collection.

2 Combine with benchmark information.

3 Augument with frame data, contextual data, and paradata.

4 Model the origin of each data point (1=benchmark, 0=

current survey) in terms of covariates.

5 Compute ratio of propensity score density (Rps ) between

benchmark and current survey.

6 Sample next phase subjects using Rps .

7 Iterate steps 2 through 6 until acceptable representativeness

or budget reached.

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

The problem

How do we know propensity scores of next phase subjects before

they respond?

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

Data structure

Y1 Y2 Y3 Y4 X1 X2 X3 Z1 Z2

Bench 1 √ √ √ √ √ √ √ √ √

Bench 1 √ √ √ √ √ √ √ √ √

Bench 1 √ √ √ √ √ √ √ √ √ ... 1 √ √ √ √ √ √ √ √ √

S1 0 √ √ √ √ √ √ √ √ √

S1 0 √ √ √ √ √ √ √ √ √ ... 0 √ √ √ √ √ √ √ √ √

S2 0 √ √ √ √ √

S2 0 √ √ √ √ √

S2 0 √ √ √ √ √

S2 0 √ √ √ √ √ ... 0 √ √ √ √ √

Notation: 

Ys are survey variables

Xs are common covariates across benchmark survey and the sample survey.

Zs are auxiliary data or contextual data from frame, registry, or interview observations, etc.

Missing 

data

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

The key step 1: Imputation

Estimate propensity score of next samples using imputed covariates

Y1 Y2 Y3 Y4 X1 X2 X3 Z1 Z2

Bench 1 √ √ √ √ √ √ √ √ √

Bench 1 √ √ √ √ √ √ √ √ √

Bench 1 √ √ √ √ √ √ √ √ √ ... 1 √ √ √ √ √ √ √ √ √

S1 0 √ √ √ √ √ √ √ √ √

S1 0 √ √ √ √ √ √ √ √ √ ... 0 √ √ √ √ √ √ √ √ √

S2 0 ▲ ▲ ▲ ▲ √ √ √ √ √

S2 0 ▲ ▲ ▲ ▲ √ √ √ √ √

S2 0 ▲ ▲ ▲ ▲ √ √ √ √ √

S2 0 ▲ ▲ ▲ ▲ √ √ √ √ √ ... 0 ▲ ▲ ▲ ▲ √ √ √ √ √

Notation: 

Ys are survey variables

Xs are common covariates across benchmark survey and the sample survey.

Zs are auxiliary data or contextual data from frame, registry, or interview observations, etc.

Imputation

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Responsive Design Procedure

Data Structure

The key steps

The key step 2: Rps

Define an acceptance/rejection process on the original sampling

frame, to reduce or eliminate bias relative to the benchmark survey.

Must satisfy:

πP(Z|accept) + (1 − π)P(Z) = PB (Z)

where π is the fraction of the combined data that are newly drawn.

What we want is P(accept|Z). Choose P(Z) to be propensity

score density and use Bayes Theorem to obtain

P(accept|Z) ∝

PB (Z)

P(Z)

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

NHIS vs BRFSS: Covariates in the propensity score model

The usual suspects:

Geographic region

Demographic: gender, age, race, marital status,

Socio-economic status: education, income categories, work

status

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

NHIS vs BRFSS: Observed Data

−4 −2 0 2 4

0.0 0.1 0.2 0.3 0.4 0.5 0.6

Density

Phase 1

Bench

phase 2

phase 3

phase 4

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

NHIS vs BRFSS: Observed Data

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

BRFSS NHIS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 1

UNWT Propensity Scores

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

BRFSS NHIS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 2

UNWT Propensity Scores

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

BRFSS NHIS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 3

UNWT Propensity Scores

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

BRFSS NHIS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 4

UNWT Propensity Scores

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

NHIS vs BRFSS: Responsive Design

−4 −2 0 2 4

0.0 0.5 1.0 1.5

Density

Phase 1

Bench

Sample 2

−4 −2 0 2 4

0.0 0.5 1.0 1.5

Density

Phase 2

Bench

Sample 3

−4 −2 0 2 4

0.0 0.5 1.0 1.5

Density

Phase 3

Bench

Sample 4

−4 −2 0 2 4

0.0 0.5 1.0 1.5

Density

Phase 4

Bench

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

ACS vs CE: Observed Data

−4 −2 0 2 4 6 8

0.0 0.2 0.4 0.6 0.8 1.0

Density

Phase 1

Bench

phase 2

phase 3

phase 4

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

ACS vs CE: Observed Data

● ●●●●●

●●●●●●

●●●●

CE ACS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 1

UNWT Propensity Scores

●● ●●●●●

●●●●●

●●●●●

●●

●●●●●

●

●●●●

●

●

●

●

●●●●●

●●

●

●●●●●●

●●●●●

CE ACS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 2

UNWT Propensity Scores

●

●●●● ●●●

●●●

●●●

●●

●●●●●●●●●

●●

●

●●●●●

●●●●●●●

●●●●

●

●

●●●●

●

●●

●●●●●●●

●

●

●●●

●●●●●

●

●

●●●

●

●

●●●●●

●●●●●

●

●●●●●●●●●

●

●●●

●●

CE ACS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 3

UNWT Propensity Scores

●●

●

●

●●●

●

●

●

●

●

●●

●

●●● ●●●●

●

●●●●●●●

●

●

●●

●

●

●●

●●●

●●●●

●●●●●

●●●●●

●

●

●●

●

●●●

●

●●●●

●●●●

●●●

●

●

●●

●

●

●●●

●

●●●●●

●

●●●

●●

●

●

●●●

●●●

●●●●

●●

●●

●●

●

●

●●

●●●●●

●●

●●●●●

●●●

●●●

●

●●

●

●●

●●

●

●●

●

●●●●●

●

●●

●

●●●●●

●

●●●

●●●

●●●

●

●

●●

●●

●●●

●●●●●

●●●●●

●●●

CE ACS

0.0 0.2 0.4 0.6 0.8 1.0

Calendar Quarter 4

UNWT Propensity Scores

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Example Data

ACS vs CE: Observed Data

● ●●●●●

●

●●●●●

●●

●●●

●●●●●●●

●●●

●

●●

●●●

●

●●●

●

●●

●●●●●

●

●●●●

●●●

●

●●

●

●●

●

●

●●●●●●

●

●

●●●●

●

●●●●●●

●●●

●

●●●

●●●●

●

●

●●●●●●

●●

●

●●

●●

●●●●

●●●●

●

●●

●

●●●●●●●

●●●●●●

●●

●●

●

●

●●●●●

CE ACS

0.70 0.75 0.80 0.85 0.90 0.95 1.00

Calendar Quarter 1

UNWT Propensity Scores

●

● ●

●●

●●

●●

●

●●

●●●●

●●●●●

●

●●

●●

●●●

●●●●●

●●●

●

●●

●

●●

●●●

●

●●

●

●●

●

●

●

●

●

●●●●

●

●

●●

●

●

●

●

●

●

●●●

●●

●

●●●

●●●●●

●●●

●

●

●●●

●

●●

●

●●●●●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●●●●

●

●

●

●

●●●

●

●●●

●●●

●

●

●●

●●

●●

●●

●

●

●●

●

●●

●●●

●

●

●●

●

●

●

●

●

●●

●

●

●●●●●

●

●

●

●●●●●

●

●

●●●

●

●

●

●●

●

●

●●

●●●

●●●

●

●

●

●

●●

●●●●●

●

●

●

●

●●●●●

●●

●●

●

●

●●

●●

●●●●

●

●

●

●

●●

●

●

●●

●

●

●

●●●

●●●

●●

●●●●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●●●

●●●●

●

●

●

●

●●

●●●

●●

●

●●●

●

●

●

●

●●

●

●●

●●

●

●

●●●●

●

●●●●

●

●●●

●

●●●

●●●

●

●

●

●

●●●●

CE ACS

0.70 0.75 0.80 0.85 0.90 0.95 1.00

Calendar Quarter 2

UNWT Propensity Scores

●

●

●

●●●●

●

●

●

●

●

●

●●

●●●

●

●

●

●

●●

●●

●

●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●●

●

●

●●

●●

●

●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

●●●

●●●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●

●●

●

●●

●

●●●

●

●●

●

●●

●

●●

●

●

●

●

●

●●

●

●●●

●●

●

●●

●●

●●

●●

●

●

●

●●

●●●

●

●●●

●

●

●●

●●●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●●

●●

●●●

●●

●

●

●

●

●

●●●

●●

●

●●

●

●

●

●

●●

●

●

●●

●

●

●

●●●

●

●

●●

●

●●

●

●

●●●

●●●

●●

●

●

●

●●

●●

●

●●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●●

●

●

●

●

●

●●

●

●

●

●

●●

●

●

●●

●●

●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●●

●●

●●●

●

●

●●

●

●

●●

●●●

●

●

●

●

●●●

●●

●●

●

●

●●

●

●

●●

●

●

●

●

●●●

●

●●

●

●

●

●●

●●

●

●

●●●

●●

●

●

●

●

●

●●

●●

●

●

●

●●

●●●

●●

●

●

●●●●

●

●

●

●

●●

●

●

●

●●●●

●●

●●

●●

●●

●

●

●

●

●

●

●●

●

●

●

●●

●

●

●

CE ACS

0.70 0.75 0.80 0.85 0.90 0.95 1.00

Calendar Quarter 3

UNWT Propensity Scores

●

●●●●

●

●

●●

●

●

●

●

●

●●

●

●

●●●

●

●

●

●●●●

●

●

●

●

●●

●

●

●

●●

●●●

●

●

●

●

●

●

●●●

●●

●●

●

●

●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●

●●

●●●

●

●

●

●

●

●●●

●

●

●●

●

●●

●

●

●

●

●

●●

●

●

●●●●●●

●●

●

●

●●

●

●●

●

●

●●

●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●●●●

●

●

●

●●●●

●●●●

●

●

●

●

●●

●

●●●●

●

●

●

●●●

●

●

●●

●●●

●●

●●

●

●

●

●

●

●

●

●

●●

●

●

●●●

●●●

●

●●

●

●●

●

●●

●●

●

●

●

●

●

●

●

●

●

●

●

●

●●●

●●●

●●

●

●

●

●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●

●●

●●

●

●●

●●●

●●

●

●

●●

●●●

●

●

●●

●

●

●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●

●●●

●

●

●●

●

●

●

●

●

●

●

●●

●●

●

●

●

●

●●●

●●

●

●●

●

●

●

●

●●

●

●

●

●●●●●

●

●

●

●

●●

●

●

●

●●

●●●●●

●

●

●

●

●

●

●

●

●

●

●

●●

●

●

●●

●●

●

●●

●

●●●●

●

●

●

●

●●

●

●

●

●

●●

●

●

●

●

●●

●

●●

●

●●

●

●●●●

●

●

●

●

●

●

●●●

●

●

●

●

●●●

●

●

●●

●●

●●●

●

●

●

●

●

●●●

●

●●

●

●●●

●

●

●

●

●●

●●●

●●

●●

CE ACS

0.70 0.75 0.80 0.85 0.90 0.95 1.00

Calendar Quarter 4

UNWT Propensity Scores

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Model fit

Model fit and similarity measures

Model fit diagnostics

Distance measure on densities

Hellinger distance to quantify the similarity between two

probability distributions

H

2 =

1

2

Z √

dP −

p

dQ2

where P and Q represent the propensity score density from

benchmark and current survey, respectively.

Balance measure on covariates

Absolute distance

Julia Lee CE 2012 Survey Methods Symposium
Introduction

The Proposed Methods

Simulation Results

Next Step

Model fit

Thank you!

Comments are appreciated!

Contact: julialee@umich.edu

Julia Lee CE 2012 Survey Methods Symposium