1

Crowdsourcing Codebook Enhancements

A DDI-based Approach

Lars Vilhuber (Cornell University)

Benjamin Perry (Cornell University)

Venkata Kambhampaty (Cornell University)

Kyle Brumsted (McGill University)

William Block (Cornell University)

FCSM, December 2nd 2015
• Data curators (Agencies) lack a mechanism to obtain 

structured feedback for their metadata

• Metadata standards for the social science community are 

difficult to navigate, even with complex tools

• Metadata curation is a labor intensive process

Issues

2
• Provide easy-to-use tools and interfaces to structured 

metadata

• Rely on open standards, namely the Data Documentation 

Initiative (DDI) schema

• Build infrastructure that enables data curators to leverage 

community-driven input to official documentation

Our Approach

3
The Comprehensive Extensible Data Documentation and Access 

Repository

How?

4

CED2AR
• Metadata curation software

• Designed for documenting existing datasets

• Funded by NSF grant #1131848

• Online at www2.ncrn.cornell.edu/ced2ar-web

What is CED2AR?

5
6

What is CED2AR?
Basic Information Flow

7

Official

Metadata

Crowdsourced

Metadata

Internal

Metadata Datasets

Staging Area Public Facing

User switches
Basic Information Flow

8

Official

Metadata

Crowdsourced

Metadata

Internal

Metadata Datasets

Staging Area Public Facing

User switches
Internal Processing

9

1. Creation of skeletal metadata

– Assuming data is already curated

– Converting data into standardized metadata

• Tools included (for SAS, Stata, SPSS, CSV), not discussed 

here [appendix slides]

2. Hand editing and subsetting

– Adding verbose descriptions

– Applying disclosure limitation
Internal Processing

10

• Simple editing interface

– Web-based, with limited rich text features

– Math allowed (LaTeX)

• Feedback

– Completeness of codebook? 

– Without technical jargon!

– Can be tuned
11

Internal Processing: Hand Editing
• Marking elements with different restrictions

Internal Processing: Access Control

12
• Provide feedback to improve sparse documentation

Internal Processing: Scoring

13
Workflow control

14

• Ability to view additions/subtractions

– Between versions

– Between crowd-sourced information and official 

information

• Ability to control access

– Editing versus viewing

– Authentication and reputation
• Uses Git, a distributed version control system

• Every aspect of the system is configurable

– Scheduled tasks check for changes

– Once changes exceed threshold, they are pushed

– Remote instances pull changes on demand

Versioning

15
Viewing changes made

Combining Knowledge

16
All changes are logged externally via Git

Versioning

17
Basic Information Flow

18

Official

Metadata

Crowdsourced

Metadata

Internal

Metadata Datasets

Staging Area Public Facing

User switches
19

Official view
20

Crowdsourced view
Basic Information Flow

21

Official

Metadata

Crowdsourced

Metadata

Internal

Metadata Datasets

Staging Area Public Facing

User switches
22

Authentication and Attribution

• When opening up contributions to a wide audience, 

how to triage between “rants” and meaningful 

contributions?

• Use of ORCID (academic network) for 

authentication

• Public attribution with link to (verified) academic ID 

is key for positive feedback (your effort is 

recognized) and prevention of negative contribution 

(your rant is traceable to you!) 
• Supports OpenID and OAuth2

– Currently using Google and ORCID with OAuth2

– Developing connectors to work with additional providers

• CED2AR handles identity management

Authentication

23
Basic Information Flow

24

Official

Metadata

Crowdsourced

Metadata

Internal

Metadata Datasets

Staging Area Public Facing

User switches
Combining Knowledge: Merging

25

• Curators are given an interface to merge crowdsourced 

documentation with official
Combining Knowledge: Merging

26
Combining Knowledge: Merging

27
Combining Knowledge: Citations

28

• Contributors can be tracked for each of their changes
29

Combining Knowledge: Citations
Try for yourself: http://demo.ncrn.cornell.edu

Demo

30
• Where are we taking the project?

• RePEC integrations?

– Additional authentication tied to existing (document) metadata

– “Claiming” user-created datasets

– Entry-point to discover datasets (data provenance) and 

therefore need for documentation

Future Directions

31
Thank you! 

Questions?

ced2ar-devs-l@cornell.edu 32
33

Extra Slides
What technologies are used to build CED2AR?

• Server: Apache/Tomcat

• Databases: BaseX and Neo4j

• Primary Languages/Frameworks:

– Java

– Spring MVC

– Bootstrap, JQuery, LESS

Technologies Used

34
• Convert data to metadata

• Online or offline conversion

• Can start with Stata, SPSS, SAS, R, ASCII or a 

relational DB

• We generate DDI 2.5 (codebook)

Generating Metadata

35