1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Modeling Topics in Survey 

Interviewer Notes

Wendy Martinez

Terrance Savitsky

US Bureau of Labor Statistics

SDSS 2018

The views expressed are those of the authors and do not necessarily reflect policies 

of the U.S. Bureau of Labor Statistics.
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Origin

 Graduate courses in

Computational statistics

Exploratory data analysis

 Other Ed Wegman students

Jeffrey Solka – finite mixture models

Angel Martinez – text analysis

 Builds on prior work with Lucilla Tan
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Major Points of Analysis

 Use two data sources

 Sample unit behavior

 Text describing reason for refusal

 Use two types of text encodings

 Term-document matrix

 Bigram proximity matrices

 Cluster text using

 Model-based clustering

 Bayes clustering

 Find important concerns in clusters using classification trees

 Cluster IDs are ‘class labels’

 Coded behaviors are ‘features’
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Background – CE

 Data source: 

The Consumer Expenditure Interview Survey (CE) – provides information 

on the buying habits of America’s consumers, including data on 

expenditures, income, and demographics.

 For more details about the Consumer Expenditure program: 

http://www.bls.gov/cex

GOAL: Associate a sample unit’s sentiment (doorstep concerns 

from Contact History Instrument) regarding the survey with the 

reasons for non-response (Survey Instrument – SI)
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Study Sample

 Wave 1 sample units from CE collection April 2012 

through March 2014

 18,031 distinct sample units

 25% were non-respondents

30% of these refused for Other reasons

 Reasons not captured by codes in SI

 Only know reason through text analysis
6 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Data Sources – 2 Instruments

Attempt to 

contact sample 

unit member

Contact made

Interview 

conducted

Non-response/

Refusal Unable to 

contact

Data Source 2: 

Contact History Instrument 

(CHI)

doorstep concerns 

Data Source 1

Survey Instrument (SI)

“Other” reasons 

Attempts to contact sample unit Final outcome
7 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Survey Nonresponse Inputs

Data Source 1 (SI)

Page in Survey Instrument
8 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Unstructured 

text field

Page in Survey Instrument

“Other Refusal” Reason – A Document

Data Source 1 (SI)
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Examples of text documents from Data Source 1

"DOESNT DO SURVEYS". SOMEWHAT HOSTILE

"Its voluntary; I just dont want to do it."

"Special family situation"

"VOLUNTARY NO THANKS"

"just not interested"

"makes it a policy not to do such things"

100% Day

roomates and dont want bothered

old lady said dsnt wnt to participate

999999999999999999999999999999999999999999999

?? Im closing this case for another FR

ALREADY DNE OTHER SURVEYS TOO INVASIVE

ANTI GOV

ATTORNEY TOLD THEM THEY DIDNT HAVE TO DO IT

AVOIDANCE

AVOIDANCE, SILENT REFUSAL

Absolutely will not answer questions
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Highest Frequency Words

Refusal Corpus (SI) 

Most frequent words in the text narrative

Highest Frequency

(1 – 10)

Highest Frequency

(11 – 20)

privacy doesn

refusal door

avoidance government

silent health

issues voluntary

survey concerns

participate personal

refused gov

not govt

anti family
11 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Door “Doorstep concern” indicators from 

Data Source 2 (CHI)

 Interviewers report observations of sample unit reactions to 

the survey request.

 Associate concern codes with refusal reasons

 CHI revised after 2013 data collections – fewer items.
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Process the Text

 Used MATLAB and R

 Text narrative from a non-responding sample unit 

is a “document.”

 Preprocessed text

Removed special characters and stop words

Converted to lower case

 Size of corpus

1,283 documents (n)

760 unique words (p)
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Exploratory Process

1. Encode documents using raw frequencies

1. TDM – Term-document Matrix

2. BPM – Bigram Proximity Matrix

2. Reduce dimensionality–ISOMAP – nonlinear 

approach

Chose d = 4

Used cosine distance

3. Conduct cluster analysis

1. Model-based Clustering

2. Bayes Clustering

4. Associate clusters of interviewer notes (refusal 

reasons) with doorstep concerns
14 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Encode the Text – TDM

 The most common approach is the bag of 

words or term-document matrix (TDM).

 The rows correspond to words.

 The columns correspond to documents.

 The (i,j) -th entry in the matrix is the number 

of times the i -th word appears in the j -th

document.

 These are the raw frequencies.
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Encode the Text – BPM

 TDM – each document coded as a vector

 Bigram Proximity Matrix (BPM) – each document 

coded as a matrix

 The rows and columns in BPMk(k-th document) 

correspond to words.

 The (i,j) -th entry in the matrix BPMkis the number 

of times the i -th word appears before the j -th

word.

 BPMkis reshaped as a row in the data matrix.
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

The Data

 Interviewer note in survey instrument is a 

document.

 Recall the size of corpus:

1,283 documents (n)

760 unique words (p)

 Size of data matrix using TDM is 1,283 x 760

 Size of data matrix using BPM encoding is 1,283 x 

579,121

 The BPM uses the period for all end of sentence punctuation.

 The period is counted as a word. 
17 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

ISOMAP Dimensions for TDM Encoding

-1.0 -0.5 0.0 0.5 1.0 1.5 -1.0 -0.5 0.0 0.5 1.0 1.5

Isomap 1

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0 1.5

Isomap 2 -1.0 -0.5 0.0 0.5 1.0

Isomap 3 -1.0 -0.5 0.0 0.5 1.0

-1.0 -0.5 0.0 0.5 1.0 -1.0 -0.5 0.0 0.5 1.0

Isomap 4

 ISOMAP–estimates of geodesic 

distance used with classical 

multi-dimensional scaling

 Each point is a document in an 

ISOMAP embedding.

 Clustered these data.
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

ISOMAP Dimensions for BPM Encoding

- 1 0 1 2 - 1 0 1 2

Isomap 1

-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 -1.5 -1.0 -0.5 0.0 0.5 1.0 -1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 - 1 0 1 2

Isomap 2 -1.5 -0.5 0.5 1.0 1.5

Isomap 3 -1.5 -0.5 0.0 0.5 1.0

-1.5 -1.0 -0.5 0.0 0.5 1.0 1.5 -1.5 -0.5 0.5 1.5

Isomap 4

 See a little more structure 

with this embedding.

 Clustered these data.

 Around 15 clusters found.
19 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Cluster Analysis

 Model-Based Clustering:

 Estimate a probability density function for cluster structure

 Model is finite sum (mixture) of multivariate Gaussians

 Each term is a cluster – very flexible structure

 Provides estimate of number of groups

 Bayes Clustering:

 Limit of a Dirichlet process (DP) model as the noise variance contracts 

on zero

 Converts posterior distribution to penalized optimization

 Use Carlinski-Harabaz statistic to select penalty parameter (in turn 

determines number of clusters)

 Connects DP to k-means
20 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Connect Clusters with Concerns

 Cluster ID for each narrative of non-response

 Construct classification trees

Use cluster IDs (SI) as class labels

Use doorstep concerns (CHI) as features

Variable chosen to ‘best’ split into subsets

Indication of ‘importance’

Feature/Predictor

Contact History:

Doorstep concern codes

Class/Response

Survey Instrument: 

Cluster ID for text narrative
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Model-Based Clustering using TDM Data Matrix

Classification Tree: using Cluster IDs 

from ISOMAP – Model based clustering 

as response variables and doorstep 

concern codes as predictors.

Voluntary

Not interested

Privacy

Anti-government
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Model-Based Clustering using BPM Data Matrix

Privacy

Not interested

Anti-government

Voluntary

Content
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Bayes Clustering using TDM Data Matrix

Anti-government

Privacy

Not interested Family
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Bayes Clustering using BPM Data Matrix

Privacy

Not interested

Family

Content

Anti-government
25 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Recap

 Used two types of encodings

 Term-document matrix

 Bigram proximity matrices – captures some word order

 Explored two types of cluster approaches – both estimate 

number of clusters

 Model-based clustering

 Flexible clusters

 Not appropriate for high-dimensional data

 Bayes clustering

 Similar to k-means clustering – looks for spherical clusters

 Can be used with high-dimensional data

 Associated clusters with sample unit behavior
26 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Discussion

 Compare cluster approaches – MBC and Bayes

Similar estimates on the number of clusters ~ 15 

Same major concerns – Privacy, anti-government, not 

interested, voluntary

Bayes – different concern – Family

 Compare encodings – TDM and BPM

Some similar concerns

BPM uncovered different concern – Survey content
27 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Application

 Important reasons for nonresponse are not captured by the 

existing codes – enhance the survey instrument. 

 Missing these reasons could adversely affect non-response 

bias analyses. 

 Understand refusal reasons and sentiment to better tailor 

information about the usefulness of government statistics and 

measures taken for privacy protection.

 Use information from text, doorstep concerns, and other 

variables to estimate propensity to respond.
28 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

References

 Breiman, L., J. Friedman, R. Olshen, and C. Stone. 1984. Classification and 

Regression Trees. Boca Raton, FL: CRC Press.

 Solka, J. 1995. Matching Model Information Content to Data Information, PhD 

Dissertation, George Mason University.

 Tenenbaum, de Silva, & Langford. 2000. “A global geometric framework for 

nonlinear dimensionality reduction,” Science, 290:2318-2323.

 Martinez, A., 2002. A Framework for the Representation of Semantics, PhD 

Dissertation, George Mason University.

 Fraley & Raftery, 2002. “Model-based clustering, discriminant analysis, and density 

estimation: MCLUST,” Journal of the American Statistical Association, 97:611-631.

 MBC: https://www.stat.Washington,edu/rafter/Research/mbc.html

 Martinez W. and L. Tan, “Categorizing sentiment using unstructured text,” Joint 

Statistical Meetings, 2015.

 Savitsky, T.D. 2016. “Scalable Approximate Bayesian Inference for Outlier Detection 

under Informative Sampling,” Journal of Machine Learning Research, 17:1-49.
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Contact Information

Wendy Martinez

Bureau of Labor Statistics

Office of Survey Methods Research

202-691-7400

martinez.wendy@bls.gov
30 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Limitations

1. Limited access to interviewer notes due to PII concerns

a) No access to interviewers case level notes

b) No access to doorstep concern item “other-specify” description

2. Clustering method assigns a sample unit to membership in 1 

unique cluster, but more than one doorstep concerns may 

be observed for a sample unit member 

3. Text box for entering reason in SI is too small (usability 

perspective) resulting in short documents

30
31 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Box for Text Narrative

31