1 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Implementing the Data Documentation Initiative 

(DDI) for the Consumer Expenditure Surveys

Taylor J. Wilson

Bureau of Labor Statistics

Federal Committee on Statistical Methodology (FCSM)

Washington Statistical Society (WSS)

Washington, D.C.

14 September 2018
2 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Roadmap

 Survey challenges

 DDI in the Consumer Expenditure Surveys

 Implementation to date

 Technical requirements

 Going forward
3 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Survey Challenges
4 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Consumer Expenditure Surveys

 Complex surveys: massive amounts of data

 Data stored on servers going back to 1981

 Interview Survey: over 60 datasets; 1,800 variables

 Diary Survey: 10 datasets; about 1,600 expenditure 

items

 Over 70 data processing jobs or edits
5 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
6 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

One Survey, Two Dimensions

 Most end users are interested 

in changes across time. 

 Changes to a variable 

through the processing 

phases help users understand 

how the data are generated.

2013

2014

2015

2016

Phase II, III, IV 

Phase II, III, IV 

Phase II, III, IV 

Phase II, III, IV 
7 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

DDI in the 

Consumer Expenditure Surveys
8 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Data Documentation Initiative (DDI)

 International metadata standard useful for describing social 

science datasets.

 Tool box framework allows for an ad hoc solution to the 

interoperability problem between survey programs. 

“Enter once, use often.”

Effective Data Sharing

Standardized Terminology
9 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Who uses DDI?
10 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Problem: Current Metadata Structure

 Fragmentation among the subsystems

 Metadata exists in different folders and forms, for different 

surveys, subsystems, and years

 Many different points of manual input and alteration

 There is a need across all subsystems of CE to develop a 

standardized method for documenting and linking metadata
11 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Problem: Two Surveys Same Concept

 Diary and Interview both capture expenditures

Some expenditures are unique to interview

Some expenditures are unique to diary

Some expenditures are shared between interview and diary.

 Data are integrated in the table format at the end of 

processing.

Creating perfect interoperability between the surveys would be ideal. 

Document conceptual differences between survey items.
12 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Ideal Solution: DDI for the Entire Process

 DDI is implemented with the Colectica System

 Each step is cohesively described from survey questionnaire to 

public-use microdata (PUMD)

 This involves documenting the processing steps within the DDI 

framework across both surveys

 Documenting processing is useful internally but not externally 

to better understand what we do and how we do it

Continuity of operations (COOP)
13 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Interim Solution: Endpoints of the Survey Lifecycle

 Documenting the end state of the data across time using 

Colectica

Solves persistent user issues

Highlights discontinuities and adds context

 Documenting the beginning of the data at collection

Questions and responses

 Link these two things at a cursory level

Tells users what survey questions produce what PUMD variables

How that has changed over time
14 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Implementation to Date
15 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Prototype Goals

 Keyword search for variables

 Display lifecycle of a variable, from question text through Microdata, UCCs*

 Query relationships of variables across CE subsystems*

 Display changes in a variable across subsystems and/or time (e.g. change in 

response options to a question)*

 Display/link to reference documents (e.g. Information Booklet)*
16 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Questionnaire Development

 Question bank that allows us to retrieve questions or compare questions across 

years 

 Integrated mapping of questionnaire metadata

 Consolidate multiple documents in different locations 

 Streamline changes for each revision out of word/pdf/csv

 Make easier to navigate: current full specifications ~ 4,000 pages

 Iteratively making design changes is difficult
17 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

View of Prototype
18 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
19 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
20 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
21 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
22 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
23 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
24 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
25 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
26 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
27 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
28 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
29 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
30 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

?
31 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
32 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
33 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
34 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
35 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

View of Questionnaire
36 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov
37 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
38 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov
39 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Section:
40 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Technical Requirements
41 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Servers and C#

 SQL Server software necessary to 

handle Colectica materials

PostgreSQL 

 C# Programming using a Software 

Developer’s Kit (SDK)

We outsourced this to the Colectica 

developers
42 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

Going Forward
43 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Next Steps: Short Term

 Internal server being set up by BLS 

for us to use. 

Must meet the requirements of the 

Colectica software to be useful.

Prototype was built on Colectica local 

servers. 

 Team is placing as much withinscope metadata (1996-present) as 

possible into machine readable 

formats. 
44 — U.S. BUREAU OF LABOR STATISTICS TATISTICS • bl s.gov bl s.gov

Next Steps: Long Term

 The processing systems are being 

redesigned!

Once this process is complete the 

new system will be documented by 

our system. 

 Roll out a public-facing portal to 

make the user experience with the 

data more efficient and effective. 
Contact Information

45 — U.S. BUREAU OF LABOR STATISTICS • bl s.gov

CE_DDI@bls.gov

Taylor J. Wilson

(202) 691-6550

Wilson.Taylor@bls.gov