1 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

NADDI 2018

Implementing DDI to Document 

the Consumer Expenditure Surveys 

Reginald Noël

Parvati Krishnamurty

5 April 2018
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

The Consumer Expenditure Surveys 

National surveys on U.S. households’ spending

 Survey sponsor & data processing: BLS 

 Data Collection: Census Bureau

 Two independent survey instruments: Diary, Interview Survey 

 Data Processing: BLS – across 3 subsystems 

– Over 80 data processing jobs/edits for each survey of each year 

– Over 3,400 variables for each survey of each year 

 Data Products release schedule: BLS 

– Tabular data (rolling 4 quarters of data every 6 months)

– Public Use Microdata (annually)

– Input to CPI (quarterly)
3 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

This Presentation

The CE program is attempting to move towards a more efficient and 

comprehensive documentation of survey metadata. We provide 2 

illustrations of current inefficiencies, and how we are attempting to use 

DDI to address them:

1. Questionnaire development

2. Investigating data inconsistencies
5 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

1. Questionnaire Development
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CE Questionnaire Development Process

 The Interview Survey is long and contains complex skip patterns 

designed to reduce respondent burden.

 Changes to questionnaire items such as wording, order, skip patterns, or 

new questions are often needed. 

 CE provides detailed requirements, which Census then enters into their 

own specification system (SPIDER), and then programs the Blaise 

instrument. Changes are usually made every 2 years.

 CE Interview Survey instrument is being redesigned.
7 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Interview Survey Questionnaire Development

Challenges

 No question bank

 No integrated mapping of questionnaire metadata

Multiple documents in different locations 

 The changes document for each revision is in word/pdf/csv

Difficult to navigate: current full specifications ~ 4,000 pages

Iteratively making design changes is difficult

Solution: Testing the use of Colectica Questionnaires to create 

requirements for the new CAPI instrument, to be programmed into a Blaise 

instrument by Census Bureau. 
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Colectica Questionnaires: Advantages

 Creates DDI based survey specifications

 Surveys can be published in multiple formats including Blaise 5

 Produces machine readable DDI 3.2 format xml files (source code for 

CAI systems)

 Rapid design iterations: Easy to make changes in the software and then 

generate new specifications

 Uses a repository of questions so we can retrieve questions and blocks

 Integrates with the rest of the Colectica software suite including 

Repository, Portal, and Designer
9 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Some advantages: Output formats
10 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Some advantages: Repositories
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Colectica Questionnaire: Limitations

 Trial version cannot handle some of the more complex questionnaire 

design elements. 

 Some examples are: grid questions, reusable lists of response 

options, globally adding fields for specifications, setting don’t know 

and refusals globally, ability to store and cumulate display logic, 

rosters and loops, adding comments to questions.

 We got around the limitations by programming what we could in the 

software and providing a cover sheet for each section with additional 

information that could not be entered in the software.

e.g. text fills, refusal/don’t know options, instructions for looping, and 

grid questions.
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Cluster screener (grid question) in the CE
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Grid question in Colectica Questionnaires

Section:
14 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

2. Investigating Data Inconsistencies
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Fragmented metadata across time and processes

Challenges

 Maintained separately

by surveys (Diary or Interview) 

by years (... 2015, 2016, 2017 ...) 

by 3 data processing subsystems 

Initial Edit System (IES), Edit and Estimation System (EES), Microdata, and 

Tables

 Multiple points of manual input and alteration

 Many different folders, different file formats, separate database

Solution: Testing Colectica Designer, Repository, and Portal to put all 

metadata across time and subsystem onto one web-based portal
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

2. Investigating Data Inconsistencies

Inquiry Request: Research Universal Classification Code 580311 

 What variables are used to map to this UCC?

 How is the associated question asked in the survey?

 Get descriptive statistics, i.e. average, frequencies, etc.
17 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

2. Investigating Data Inconsistencies

 Current Production Method

1. Excel mapping file

2. Web-based P&C data dictionary

3. PDF public-use microdata dictionary

4. BLS website to get data

5. SAS enterprise to get stats

 Using Colectica and DDI

Colectica Repository and Portal 

through web-based prototype
18 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 1: Excel mapping file for UCC
19 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

2013 Interview Time Adjustment Expenditure Mapping file – (Excel file located in year-designated folder) 

File 1: Excel mapping file for UCC
20 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 2. Web-based P&C Data Dictionary 
21 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 2. Web-based P&C Data Dictionary 
22 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 2. Web-based P&C Data Dictionary
23 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 2. Web-based P&C Data Dictionary 
24 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 3. PDF Microdata Dictionary 
25 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 4. BLS website to download dataset 
26 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 4. BLS website to download dataset 
27 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

File 5. SAS to to get dataset descriptive statistics 
28 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

1. Colectica Portal: Prototype Home Page
29 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

2. Find and click the UCC in question
30 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

3. Obtain desired information
31 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

4. Get additional information
32 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

4A. Get additional info from different years
33 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

4B. Get additional info for different subsystems
34 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

A User’s Perspective

Colectica can help us by:

 Limiting the different folders and files need to research

 Saving on the amount of time to look up info

 Making all metadata readily accessible in one platform

 Facilitating easy access to descriptive data
35 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Consumer Expenditure Surveys

Contact Us:

krishnamurty.parvati@bls.gov

noel.reginald@bls.gov

CE DDI Team

Daniel Gillman

Evan Hubener

Parvati Krishnamurty

Reginald Noël

Bryan Rigg 

Arcenis Rojas

Lucilla Tan

Taylor Wilson
36 — U.S. BUREAU OF LABOR STATISTICS • bls.gov

Extra slides
37 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

DDI Structure for Consumer Expenditure Quarterly

Iteration I
38 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

DDI Structure for Consumer Expenditure Quarterly

Iteration I
39 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

DDI Structure for Consumer Expenditure Quarterly

Iteration I
40 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

CE DDI/Colectica Experience

Iteration 1 

Explored use of a few basic DDI constructs to describe the CE Interview Survey 

metadata. [Colectica Designer]

Colectica implements DDI

Iteration 2 

Proof-of-concept prototype [Colectica Portal and Repository] 

Eight survey questions across 2 years of the Interview Survey.

oEstablish and view the linkage of a survey question to its end as a variable in 

the public-use Microdata file

o Identify changes to a variable: for e.g., a change in the response options to the 

EDUCA (member education attainment) variable
41 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

What’s Next?

Iteration 3

 Will focus solely on Microdata and Tables (Ph4)

o Inputting metadata from 1996 to 2017

 Continue development work with Colectica 

 Scale up prototype, including Diary

 Incorporate the use of Colectica Questionnaire

 Create one portal for the public and another for interoffice use
42 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Potential Hurdles

Iteration 3

 Requires IT support

 Setting up a server

 Colectica programming language (C-sharp, .NET) 

 Equipment requests

 Limited funding
43 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Cover page

Household Operations: Cover Page

A. Section Instructions:

This section is a laundry list section that follows the laundry list standards. 

Design will be a table/grid format

Repeat OPSCSC through OPSOTH until OPSCSC = 99. 

After OPSOTH, the FR is expected to continue reading the list of expenditures from the cluster screener (OPSCSC)

B. Section Format:

Table with a maximum of 14 rows

C. Don’t knows and refusals:

1. Do not allow Don’t Know or Refusal on:

OPSCSC, OPSIDE, OPSECE, OPSCOM, OPSSEC

2. Do not display Don’t Know or Refusal options on screen

3. Where Don’t Know and Refusal are allowed, use keyboard shortcuts:

Don’t know = CTRL + D

Refusal = CTRL + R

Bookmark=CTRL + B
44 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bls.gov bls.gov

Cover page (continued)

D. Fills: 1.OPSCOM, OPSOUN and OPSOTH: 

OPSCSC

=

Category Name OPSIEX 

Less than

OPSIEX 

Greater 

than 

1 Gardening or lawn care services? 15 600

2 Housekeeping services? 20 1260

3 Termite or pest control services? 16 742

E. Fills: For OPSCOM, OPSOUN and OPSOTH: fill in...

F. Outlets eligibility: Outlets questions will be assigned based on a rotation matrix with POPS 

categories by PSU by quarter. The following categories are outlets eligible:

G. Item description: For OPSIDE: The text field should be a blank, non-required field to allow entry of the item 

description by the FR. After the FR enters a list number the instrument should place the cursor in the blank item 

description field. FRs can choose to hit enter to skip this. In case the item description is needed for a subsequent fill, the 

item code title will be used.