Federal Committee on

Statistical Methodology

Research Conference

Final Program and Abstracts

Walter E. Washington Convention Center

December 1-3, 2015

Major Sponsors:

Bureau of Economic Analysis

Bureau of Justice Statistics

Economic Research Service

Energy Information Administration

National Agricultural Statistics Service

National Center for Education Statistics

Statistics of Income Division, Internal Revenue

 Service

U.S. Census Bureau

U.S. Environmental Protection Agency

Also sponsored by:

American Institutes for Research

ICF International

IMPAQ International

Mathematica Policy Research

NORC at the University of Chicago

RTI International

Westat

Hosted by:

Council of Professional Associations on Federal Statistics
FCSM RESEARCH CONFERENCE PROGRAM COMMITTEE

Michael Planty, Co-Chair, Bureau of Justice 

Statistics

Andrew Zukerberg, Co-Chair, National Center 

for Education Statistics 

Wendy Barboza, National Agricultural Statistics 

Service

Stephen J. Blumberg, National Center for Health 

Statistics

Amy Branum, National Center for Health 

Statistics

Jesse Bricker, Federal Reserve Board

Benjamin Bridgman, Bureau of Economic 

Analysis

Margaret Conomos, Environmental Protection 

Agency

John Cromartie, Economic Research Service

Maria M. Garcia, U.S. Census Bureau

Julie Gershunskaya, Bureau of Labor Statistics

Jeffrey Gonzalez, Bureau of Labor Statistics

David Kashihara, Agency for Healthcare 

Research and Quality

Steven Klement, U.S. Census Bureau

Lynn Langton, Bureau of Justice Statistics

Rachel Lipari, Substance Abuse and Mental 

Health Services Administration

Yan Liu, Statistics of Income Division, Internal 

Revenue Service

Pamela McGovern, National Agricultural 

Statistics Service

Kristina Schafer, Department of Labor

Darius Singpurwalla, National Center for 

Science and Engineering Statistics

Erica Smith, Bureau of Justice Statistics

Ken Steve, Bureau of Transportation Statistics

Andrew White, National Center for Education 

Statistics

CONFERENCE COORDINATORS

Anna Holaus, U.S. Census Bureau

Stacia Martin, U.S. Census Bureau

Mae Pattison, Council of Professional Associations on Federal Statistics

LoWanda Rivers, U.S. Census Bureau

Elaine Russell, U.S. Census Bureau
FEDERAL COMMITTEE ON STATISTICAL METHODOLOGY

Jonaki Bose (Chair), Substance Abuse and 

Mental Health Services Administration

Nancy Bates, U.S. Census Bureau

Patrick Cantwell, U.S. Census Bureau

Chris Chapman, National Center for Education 

Statistics

Steven B. Cohen, Agency for Healthcare 

Research and Quality

John Eltinge, Bureau of Labor Statistics

Dennis Fixler, Bureau of Economic Analysis

Barry Graubard, National Cancer Institute

Arthur B. Kennickell, Federal Reserve Board

Jennifer Madans, National Center for Health 

Statistics

Rochelle (Shelly) Martinez, Office of 

Management and Budget

Jaki McCarthy, National Agricultural Statistics 

Service

Pamela McGovern (Webmaster), National 

Agriculture Statistics Service

Grace Medley (Secretary), Substance Abuse 

and Mental Health Services Administration

Peter Meyer (CDAC Chair), National Center for 

Health Statistics

Peter Miller, U.S. Census Bureau

Renee Miller, Energy Information Administration

Jeri Mulrow, National Center for Science and 

Engineering Statistics

Amy O’Hara, U.S. Census Bureau

Polly Phipps, Bureau of Labor Statistics

William Sabol, Bureau of Justice Statistics 

Nathaniel Schenker, National Center for Health 

Statistics

Rolf Schmitt, Bureau of Transportation Statistics

Marilyn Seastrom, National Center for Education 

Statistics 

Joy Sharp, Bureau of Transportation Statistics

Katherine K. Wallman (Champion), Office of 

Management and Budget

G. David Williamson, Agency for Toxic 

Substances and Disease Registry

Gordon Willis, National Institutes of Health

COUNCIL OF PROFESSIONAL ASSOCIATIONS

ON FEDERAL STATISTICS (COPAFS)

Katherine “Kitty” Smith, Executive Director

Mae Pattison, Executive Assistant
1

2015 FCSM Research Conference

The 2015 Federal Committee on Statistical Methodology (FCSM) Research Conference is one of the 

activities of the FCSM. The FCSM is a federal statistical interagency committee organized under the 

auspices of the Office of Management and Budget’s Statistical and Science Policy Office and is dedicated 

to improving the quality of federal statistics. The committee’s major goals are to:

 Communicate and disseminate information on statistical practice among all federal statistical 

agencies.

 Recommend the introduction of new methodologies in federal statistical programs to improve data 

quality.

 Provide a mechanism for statisticians in different federal agencies to meet and exchange ideas.

The 2015 FCSM Research Conference provides a forum for experts from around the world to discuss and 

exchange current research and methodological topics relevant to federal government statistical programs. 

Each day of the conference will offer papers on a wide range of topics including: adaptive survey design, 

address-based sampling, administrative records, Bayesian statistical methods, coding, confidentiality and 

disclosure, crowdsourcing, data quality, economic statistics, editing, frame development, imputation, 

mobile devices, multi-mode and web data collection, nonresponse and measurement error, paradata and 

metadata, questionnaire and survey design, record linkage, small area estimation, survey redesign, 

variance estimation, and weighting. 

Technical demonstrations run concurrently on the second day of the conference during the first morning 

session. This session includes demonstrations on “Data Visualization and Metadata Systems.”

Sessions feature papers and demonstrations by government, private sector, and academic researchers 

from three countries. All paper sessions include an open discussion; some sessions include a formal 

discussion. Papers will be made available on the FCSM Web site (http://fcsm.sites.usa.gov/) and 

presentations will be on the site of the Council of Professional Associations on Federal Statistics 

(http://copafs@copafs.org) following the conference.

KEYNOTE SPEAKER

Raj Chetty, Ph.D. is the keynote speaker in the opening plenary session.

Raj Chetty is a Professor of Economics at Stanford University, currently on leave as a Visiting Professor 

at Harvard University. Chetty's research combines empirical evidence and economic theory to help 

design more effective government policies. His work on tax policy, unemployment insurance, and 

education has been widely cited in media outlets and Congressional testimony. His current research 

focuses on equality of opportunity: how can we give children from disadvantaged backgrounds better 

chances of succeeding?

Chetty is a recipient of a MacArthur "Genius" Fellowship and the John Bates Clark medal, given by the 

American Economic Association to the best American economist under age 40. He received his Ph.D. 

from Harvard University in 2003 at the age of 23 and was a professor at UC-Berkeley until 2009, when he 

returned to Harvard as one of the youngest tenured professors in Harvard's history.
2

Final

Program
Federal Committee on Statistical Methodology Research Conference

Washington, DC – December 1-3, 2015

3

Tuesday, December 1 Wednesday, December 2 Thursday, December 3

7:30 a.m.

Registration

(Concourse)

Continental Breakfast

(Concourse)

8:30 - 9:40 a.m.

Welcoming Remarks and

PLENARY SESSION 

(Rooms 146A/B)

9:40 - 10:00 a.m.

Break 

(Concourse)

10:00 - 11:45 a.m.

CONCURRENT SESSION

A-1: Room 146A

A-2: Room 146B

A-3: Room 146C

A-4: Room 147B

11:45 a.m. - 1:15 p.m.

Lunch on Your Own

1:15 - 3:00 p.m.

CONCURRENT SESSION

B-1: Room 146A

B-2: Room 146B

B-3: Room 146C

B-4: Room 147B

3:00 – 3:15 p.m.

Break

(Concourse)

3:15 - 5:00 p.m.

CONCURRENT SESSION

C-1: Room 146A

C-2: Room 146B

C-3: Room 146C

C-4: Room 147B

7:30 a.m.

Registration

(Concourse)

Continental Breakfast

(Concourse)

8:30 - 10:15 a.m.

CONCURRENT SESSION

D-1: Room 146A

D-2: Room 146B

D-3: Room 146C

D-4: Room 147B

10:15 - 10:30 a.m.

Break

(Concourse)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION

E-1: Room 146A

E-2: Room 146B

E-3: Room 146C

E-4: Room 147B

12:15 p.m. - 1:45 p.m.

Lunch on Your Own

1:45 - 3:30 p.m.

CONCURRENT SESSION

F-1: Room 146A

F-2: Room 146B

F-3: Room 146C

F-4: Room 147B

3:30 - 3:45 p.m.

Break

(Concourse)

3:45 - 5:30 p.m.

CONCURRENT SESSION

G-1: Room 146A

G-2: Room 146B

G-3: Room 146C

G-4 Room 147B

7:30 a.m.

Registration

(Concourse)

Continental Breakfast

(Concourse)

8:30 - 10:15 a.m.

CONCURRENT SESSION

H-1: Room 146A

H-2: Room 146B

H-3: Room 146C

H-4: Room 147B

10:15 - 10:30 a.m.

Break 

(Concourse)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION

I-1: Room 146A

I-2: Room 146B

I-3: Room 146C

I-4: Room 147B

12:15 p.m. - 1:45 p.m.

Lunch on Your Own

1:45 - 3:30 p.m.

CONCURRENT SESSION

J-1: Room 146A

J-2: Room 146B

J-3: Room 146C

J-4 Room 147B

Meeting Rooms:

All rooms located on the L 

Street side of the Convention 

Center on the first floor.

Medical Assistance 

available in Room 149A/B. 
TUESDAY, DECEMBER 1 

4

7:30 a.m. - 5:00 p.m.

Registration

(Concourse)

7:30 - 8:30 a.m.

Continental Breakfast

(Concourse)

8:30 - 8:40 a.m.

Welcoming Remarks

Location: Rooms 146A and 146B

8:40 - 9:40 a.m.

PLENARY SESSION

Location: Rooms 146A and 146B

Keynote Speaker: Raj Chetty (Bloomberg Professor 

of Economics at Harvard University)

Using Federal Administrative Data to Evaluate 

and Improve Economic Policy

9:40 - 10:00 a.m.

Break

(Concourse)

10:00 - 11:45 a.m.

CONCURRENT SESSION A-1:

LINKING ADMINISTRATIVE RECORDS 

TO ENHANCE DATA

Location: Room 146A

Organizer and Chair: Lynn Langton (Bureau of Justice 

Statistics)

Linking Administrative BJS Data: 

Understanding Jail Mortality Through the 

Communities They Serve

Margaret Noonan (Bureau of Justice Statistics)

Susan Brumbaugh (RTI International)

Chris Ellis (RTI International)

Linking Administrative BJS Data: Better 

Understanding Prisoners’ Personal Histories by 

Linking the National Corrections Reporting 

Program and CARRA Data 

E. Ann Carson (Bureau of Justice Statistics)

Linking Administrative BJS Data: Linking the 

National Death Index and the Deaths in 

Custody Reporting Program to Better 

Understand Correctional Mortality

Zhen Zeng (Bureau of Justice Statistics)

Margaret Noonan (Bureau of Justice Statistics)

Discussant: William Sabol (Bureau of Justice 

Statistics)

10:00 - 11:45 a.m.

CONCURRENT SESSION A-2:

DESIGNING WEB SURVEYS 

Location: Room 146B

Organizer and Chair: Kenneth Steve (Bureau of 

Transportation Statistics)

Internet and Smartphone Coverage in a 

National Health Survey: Implications for 

Alternative Modes

Mick Couper (University of Michigan)

Jennifer Kelley (University of Michigan)

William Axinn (University of Michigan)

Heidi Guyer (University of Michigan)

James Wagner (University of Michigan)

Brady West (University of Michigan)

Designing Web Surveys for Different Screen 

Sizes and Operating Systems: What 

Respondents are Using

Robert Tortora (ICF International) 

Characteristics of the Population of Internet 

Panel Members

John Boyle (ICF International)

Naomi Freedner (ICF International)

Tala Fakhouri (ICF International)

Discussant: Chuck Shuttles (Symphony Advanced 

Media)
TUESDAY, DECEMBER 1 

5

10:00 - 11:45 a.m.

CONCURRENT SESSION A-3:

NONRESPONSE ADJUSTMENT AND 

VARIANCE ESTIMATION 

Location: Room 146C

Organizer: Julie Gershunskaya (Bureau of Labor 

Statistics)

Chair: Vladislav Beresovsky (National Center for Health 

Statistics)

JOLTS Variance Estimation to Account for 

JOLTS-CES Alignment

Mark Crankshaw (Bureau of Labor Statistics)

Michael Sverchkov (Bureau of Labor Statistics)

Linearization Variance Estimators for Mixedmode Survey Data When Response Indicators 

are Modeled as Discrete-time Survival

Abdellatif Demnati (Independent Consultant)

Weighting and Variance Estimation Plans for 

the 2016 National Household Survey

François Verret (Statistics Canada)

Arthur Goussanou (Statistics Canada)

Nancy Devin (Statistics Canada) 

Discussant: Phillip Kott (RTI International)

10:00 - 11:45 a.m.

CONCURRENT SESSION A-4:

IMPUTATION AND THE IMPACT OF 

NON-RESPONSE

Location: Room 147B

Organizer: Jesse Bricker (Federal Reserve Board)

Chair: Gary Benedetto (U.S. Census Bureau)

Bias From Unit Non-Response in the 

Measurement of Income in Household Surveys

C. Adam Bee (U.S. Census Bureau)

Graton Gathright (U.S. Census Bureau)

Bruce Meyer (University of Chicago) 

Do Imputed Earnings Earn Their Keep? 

Evaluating SIPP Earnings and Non-Response 

With Administrative Records 

Mark Klee (U.S. Census Bureau)

Rebecca Chenevert (U.S. Census Bureau)

Kelly Wilkin (U.S. Census Bureau) 

Bridging a Survey Redesign Using Multiple 

Imputation: An Application to the 2014 CPS 

ASEC

Jonathan Rothbaum (U.S. Census Bureau)

Discussant: John Czajka (Mathematica Policy 

Research)

11:45 a.m. - 1:15 p.m.

Lunch on Your Own

1:15 - 3:00 p.m.

CONCURRENT SESSION B-1:

ABSTRACTION TO EXTRACTION: 

EXPERIENCES FROM THE NATIONAL 

HOSPITAL CARE SURVEY IN USING 

ADMINISTRATIVE CLAIMS AND 

ELECTRONIC HEALTH RECORD DATA

Location: Room 146A

Organizer: Erica Smith (Bureau of Justice Statistics)

Chair: Carol DeFrances (National Center for Health 

Statistics)

Using Uniform Bill (UB)-04 Administrative 

Claims Data to Describe Hospital-Based Care 

Sonja Williams (National Center for Health Statistics)

Naga Shanmugam (National Center for Health Statistics)

Amy Brown (National Center for Health Statistics)

Carol DeFrances (National Center for Health Statistics)

Acquiring Survey Data From Electronic Health 

Record Systems: Experiences From the 

National Hospital Care Survey (NHCS)

Kristi Eckerson (Emory University IPA to National Center 

for Health Statistics)

Brian Gugerty (National Center for Health Statistics)

Anita Bercovitz (National Center for Health Statistics)

Carol DeFrances (National Center for Health Statistics)
TUESDAY, DECEMBER 1 

6

Matching the National Hospital Care Survey 

Records Across Settings and With Outside 

Data Sources

Shaleah Levant (National Center for Health Statistics)

Monica Wolford (National Center for Health Statistics)

Discussant: J. Neil Russell (Substance Abuse and 

Mental Health Services Administration)

1:15 - 3:00 p.m.

CONCURRENT SESSION B-2:

STUDYING NON-PROBABILITY 

STUDIES

Location: Room 146B

Organizer: Andrew Zukerberg (National Center for 

Education Statistics)

Chair: Sarah Grady (National Center for Education 

Statistics)

Evaluation of Strategies to Improve Utility of 

Estimates From a Non-probability Based 

Survey 

Benmei Liu (National Cancer Institute)

Sadeq Chowdhury (Agency for Healthcare Research and 

Quality)

Janet deMoor (National Cancer Institute)

Erin Kent (National Cancer Institute)

Steven Machlin (Agency for Healthcare Research and 

Quality)

Stephanie Nutt (LIVESTRONG Foundation)

Anita Soni (Agency for Healthcare Research and Quality)

Katherine Virgo (Emory University)

Gordon Willis (National Cancer Institute)

Maggie Wilson (National Cancer Institute)

K. Robin Yabroff (National Cancer Institute)

Interagency Consortium to Promote Health Economics 

Research on Cancer (HEROIC)

Response Rates and Response Bias in Web 

Panel Surveys 

John Boyle (ICF International)

Lew Berman (ICF International)

Jamie Dayton (ICF International)

Tala Fakhouri (ICF International)

Ronaldo Iachan (ICF International)

Melanie Courtright (Research Now)

Kartik Pashupati (Research Now)

Prediction-Enhanced Generalized Regression 

for Estimation from Purposive Supplements to 

Probability Samples

Avinash Singh (NORC at the University of Chicago)

Yongheng Lin (NORC at the University of Chicago)

Discussant: Jill Dever (RTI International)

1:15 - 3:00 p.m.

CONCURRENT SESSION B-3:

IMPROVEMENTS IN SURVEY 

ESTIMATION METHODOLOGY

Location: Room 146C

Organizer: Wendy Barboza

Chair: Tyler Wilson (National Agricultural Statistics 

Service)

Some Thoughts on Fitting Regression Models

to Complex Survey Data

Phillip Kott (RTI International)

Comparative Study of Confidence Intervals for 

Proportions in Complex Sample Surveys

Carolina Franco (U.S. Census Bureau)

Roderick Little (University of Michigan)

Thomas Louis (U.S. Census Bureau and John Hopkins 

University) 

Eric Slud (U.S. Census Bureau and University of 

Maryland)

Total Variability Measures for Selected 

Quarterly Workforce Indicators and 

LEHD Origin Destination Employment 

Statistics in OnTheMap

Kevin McKinney (U.S. Census Bureau)

Andrew Green (Cornell University and U.S. Census 

Bureau)

Lars Vilhuber (Cornell University and U.S. Census 

Bureau) John Abowd (Cornell University and U.S. Census 

Bureau)

Discussant: Carol Crawford (National Agricultural 

Statistics Service)
TUESDAY, DECEMBER 1 

7

1:15 - 3:00 p.m.

CONCURRENT SESSION B-4:

ECONOMIC STATISTICS

Location: Room 147B

Organizer: John Cromartie (Economic Research 

Service)

Chair: Jeffrey M. Gonzalez (Bureau of Labor Statistics)

Characteristics of Special Purpose Entities in 

Measures of U.S. Direct Investment Abroad

Dylan Rassier (U.S. Bureau of Economic Analysis)

Building a Taxonomy and Lexicon of Terms 

and Concepts at BLS

Daniel Gillman (Bureau of Labor Statistics)

Elizabeth Ashack (Bureau of Labor Statistics)

Daniel Chow (Bureau of Labor Statistics)

Ronald Johnson (Bureau of Labor Statistics)

Karen Kosanovich (Bureau of Labor Statistics)

Nicole Nestoriak (Bureau of Labor Statistics)

Ann Norris (Bureau of Labor Statistics)

Garrett Schmitt (Bureau of Labor Statistics)

Clayton Waring (Bureau of Labor Statistics)

New Technology Indicator for Technological 

Progress

Wendy Li (U.S. Bureau of Economic Analysis)

Discussant: Nikolas Zolas (U.S. Census Bureau)

3:00 - 3:15 p.m.

Break Hosted by

(Concourse) 

3:15 - 5:00 p.m.

CONCURRENT SESSION C-1:

LINKING DATA TO GENERATE NEW 

STATISTICS: JOBS, PATENTS, AND FDI

Location: Room 146A

Organizer and Chair: Rachel Lipari (Substance Abuse 

and Mental Health Services Administration)

Two Perspectives on Commuting and 

Workplace: A Microdata Comparison of Home 

to Work Flows Across Linked Survey and 

Administrative Files

Andrew Green (U.S. Census Bureau, Cornell University) 

Mark Kutzbach (U.S. Census Bureau)

Lars Vilhuber (U.S. Census Bureau, Cornell University)

An ‘Algorithmic Links with Probabilities’ 

Crosswalk for USPC Codes

Nikolas Zolas (U.S. Census Bureau)

Nathan Goldschlag (U.S. Census Bureau)

Travis Lybbert (University of California, Davis)

Linking BEA Surveys of Foreign Direct 

Investment in the United States with BLS 

Register and Survey Data: Twenty First 

Century Methods and Preliminary Findings

Elizabeth Weber Handwerker (Bureau of Labor Statistics) 

Mina Kim (Bureau of Labor Statistics)

Lowell Mason (Bureau of Labor Statistics)

David Talan (Bureau of Labor Statistics)

Discussant: Asrat Tesfayesus (U.S. Patent and

Trademark Office)

3:15 - 5:00 p.m.

CONCURRENT SESSION C-2:

EVALUATION OF SURVEYS THROUGH 

PRETESTING AND OTHER COGNITIVE 

METHODS

Location: Room 146B

Organizer: Jeffrey M. Gonzalez (Bureau of Labor 

Statistics)

Chair: Jennifer Edgar (Bureau of Labor Statistics)

Retrieval of Autobiographical Information

Erica Yu (Bureau of Labor Statistics)

Scott Fricker (Bureau of Labor Statistics)

The Science of Usability Testing

Jean Fox (Bureau of Labor Statistics) 

Using Graphics to Convey Complex Survey 

Concepts

Heidi St.Onge (U.S. Census Bureau)

Rebecca Morrison (National Science Foundation)

Using Eye-tracking to Understand how Fourth 

Grade Students Answer Matrix Items

Aaron Maitland (Westat)

Hanyu Sun (Westat)

Andrew Caporaso (Westat)

Roger Tourangeau (Westat)

Jonas Bertling (Educational Testing Service)

Debby Almonte (Educational Testing Service)
TUESDAY, DECEMBER 1 

8

Reliability Testing of the Walking Environment 

Module 

Stephanie Fowler (National Cancer Institute)

Gordon Willis (National Cancer Institute)

Rebecca Ferrer (National Cancer Institute)

David Berrigan (National Cancer Institute) 

3:15 - 5:00 p.m.

CONCURRENT SESSION C-3:

APPLICATION OF BAYESIAN METHODS

Location: Room 146C

Organizer: Yan Liu (Statistics of Income Division, 

Internal Revenue Service)

Chair: Kimberly Henry (Statistics of Income Division, 

Internal Revenue Service)

Evaluating the Alignment of Grant Allocation 

and Community Need Using Bayesian Spatial 

Probit Models: A Case Study of AmeriCorps

Robin Ghertner (Corporation for National and Community 

Service)

Katie Seely-Gant (Corporation for National and 

Community Service) 

Bayesian Estimation Under Informative 

Sampling

Terrance Savitsky (Bureau of Labor Statistics)

Daniel Toth (Bureau of Labor Statistics)

Statistical Considerations in Validating Clinical 

Quality Measures

Derekh Cornwell (Mathematica Policy Research)

Fei Zing (Mathematica Policy Research)

Juan-Diego Astudillo (Mathematica Policy Research)

Michael Sinclair (Mathematica Policy Research)

Nancy Sonnenfeld (Centers for Medicare and Medicaid 

Services)

Mihir Patel (Centers for Medicare and Medicaid Services) 

A Bayesian Hierarchical Model for Combining 

Several Crop Yield Indications

Nathan Cruze (National Agricultural Statistics Service) 

3:15 - 5:00 p.m.

CONCURRENT SESSION C-4:

ADAPTIVE DESIGN I

Location: Room 147B

Organizer: Steve Klement (U.S. Census Bureau)

Chair: Peter Miller (U.S. Census Bureau)

2015 National Survey of College Graduates: 

Enhancing the Use of Adaptive Design 

John Finamore (National Science Foundation)

Stephanie Coffey (U.S. Census Bureau)

Benjamin Reist (U.S. Census Bureau)

Investigating Nonresponse Subsampling in an 

Establishment Survey Through Embedded 

Experiments 

Katherine Jenny Thompson (U.S. Census Bureau)

Stephen Kaputa (U.S. Census Bureau)

Adept Adaptations: Developing Alternative 

Strategies in Adaptive Survey Designs 

Jaki McCarthy (National Agricultural Statistical Service)

Tyler Wilson (National Agricultural Statistical Service)

Adaptive Design Experimentation in the High 

School Longitudinal Study of 2009 Second 

Follow-up Field Test: Investigating Incentive 

Treatments

Elise Christopher (National Center for Education 

Statistics)
WEDNESDAY, DECEMBER 2 

9

7:30 a.m. – 5:00 p.m.

Registration

(Concourse)

7:30 - 8:30 a.m.

Continental Breakfast

(Concourse)

8:30 - 10:15 a.m.

CONCURRENT SESSION D-1:

IMPUTATION, MULTIPLE IMPUTATION 

AND ADMINISTRATIVE RECORDS

Location: Room 146A

Organizer: Maria Garcia (U.S. Census Bureau)

Chair: Darcy Steeg Morris (U.S. Census Bureau)

Treatment of Missing Data in HierarchicallyStructured Administrative Records: 

A Case Study in the Bakken Region Using FBI’s 

National Incident Based Reporting System

Dan Liao (RTI International)

Marcus Berzofsky (RTI International)

David Heller (RTI International)

Kelle Barrick (RTI International)

Matthew DeMichele (RTI International)

Kim Martin (Bureau of Justice Statistics)

Alexia Cooper (Bureau of Justice Statistics)

Introducing Parametric Models and 

Administrative Records Into 2014 SIPP 

Imputations

Gary Benedetto (U.S. Census Bureau)

Joanna Motro (U.S. Census Bureau)

Martha Stinson (U.S. Census Bureau)

Comparison of Modern Imputation 

Methodologies on Complex Data from 

Agricultural Operations

Darcy Miller (National Agricultural Statistics Service)

Andrew Dau (National Agricultural Statistics Service)

Discussant: Joseph Schafer (U.S. Census Bureau)

8:30 - 10:15 a.m.

CONCURRENT SESSION D-2:

DEMONSTRATIONS ON DATA 

VISUALIZATION AND METADATA 

SYSTEMS

Location: Room 146B

Organizer: Wendy Barboza (National Agricultural 

Statistics Service)

Chair: Michael Gerling (National Agricultural Statistics 

Service)

Census Bureau’s Microdata Analysis System 

(MAS) Pilot for the American Community 

Survey

Tiffany Julian (U.S. Census Bureau)

Amy Lauger (U.S. Census Bureau)

William Hazard (U.S. Census Bureau) 

Crowdsourcing Codebook Enhancements: A 

DDI-based Approach 

Lars Vilhuber (Cornell University)

Benjamin Perry (Cornell University)

Venkata Kambhampaty (Cornell University)

Kyle Brumsted (McGill University)

William Block (Cornell University)

Visualization of BLS Geospatial Data Using 

R/Shiny

Elizabeth Cross (Bureau of Labor Statistics)

Kenneth Cho (Bureau of Labor Statistics)

Randall Powers (Bureau of Labor Statistics)

Jeffrey M. Gonzalez (Bureau of Labor Statistics)

Wendy Martinez (Bureau of Labor Statistics))

Interactive Web Visualization of Birth and 

Pregnancy Trends by Maternal Age, 

Race/Ethnicity, and Marital Status: United 

States, 1909-2013

Li Lu (National Center for Health Statistics)

Yinong Chong (National Center for Health Statistics)

Sally Curtin (National Center for Health Statistics)

Betzaida Tejada Vera (National Center for Health 

Statistics)
WEDNESDAY, DECEMBER 2 

10

8:30 - 10:15 a.m.

CONCURRENT SESSION D-3:

INNOVATIVE RECRUITING AND 

QUESTIONNAIRE TESTING METHODS

Location: Room 146C

Organizer and Chair: Darius Singpurwalla (National 

Science Foundation)

How Do We Bring Them In?: A Case Study in 

Using Non-Traditional Approaches to 

Recruiting Focus Group Participants

Ashley Schaad (ICF International)

Rikki Welch (ICF International)

A Multi-Method Approach to Survey Pretesting

Joe Murphy (RTI International)

Ashley Richards (RTI International)

David Roe (RTI International)

Danni Mayclin (Energy Information Administration)

Cognitive Interviewing for Surveys on Sexual 

Assault With College Students

Sarah Cook (RTI International)

Michael Planty (Bureau of Justice Statistics)

Jessica Stroop (Bureau of Justice Statistics)

Chris Krebs (RTI International)

Christine Lindquist (RTI International)

Modeling the Effect of Diverse Communication 

Strategies on Decennial Census Test Response 

Rates 

Gina Walejko (U.S. Census Bureau)

Nancy Bates (U.S. Census Bureau)

Rachel Horwitz (U.S. Census Bureau)

Monica Vines (U.S. Census Bureau)

Respondent Driven Sampling With On-line 

Recruitment and Adaptive Follow-ups

Ronaldo Iachan (ICF International)

Naomi Freedner (ICF International)

Karen Trocki (Alcohol Research Group) 

Discussant: Rebecca Morrison (National Science 

Foundation)

8:30 - 10:15 a.m.

CONCURRENT SESSION D-4: 

TIME SERIES AND FORECASTING

Location: Room 147B

Organizer: Yan Liu (Statistics of Income Division, 

Internal Revenue Service)

Chair: Tamara Rib (Statistics of Income Division, Internal 

Revenue Service)

Structural Changes via Threshold Effects: 

Estimating U.S. Meat Demand Using Smooth 

Transition Functions

Nestor Rodriguez (Economic Research Service)

James Eales (Purdue University) 

Measuring Single-Year Poverty Transitions: 

Opportunities and Limitations

Ashley Edwards (U.S. Census Bureau)

A Tale of Two Series: Examining Seasonality in 

Changing Time Series

John Stewart (Bureau of Labor Statistics)

Steve Mance (Bureau of Labor Statistics)

Improving NASS’s Crop Yield Forecasts 

Noemi Guindin-Garcia (National Agricultural Statistics 

Service)

10:15 - 10:30 a.m.

Break

(Concourse)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION E-1:

METHODS FOR MISSING DATA 

IMPUTATION

Location: Room 146A

Organizer and Chair: Maria Garcia (U.S. Census 

Bureau)

Patterns of Item Nonresponse for Income in 

the National Survey on Drug Use and Health

Peter Frechtel (RTI International)

Kristen Gulledge Brown (RTI International) 

Jonaki Bose (Substance Abuse and Mental Health 

Services Administration)
WEDNESDAY, DECEMBER 2 

11

Missing Data Methods for Regression Analysis

Susan Edwards (RTI International)

Phillip Kott (RTI International)

Rachel Harter (RTI International)

Peter Frechtel (RTI International)

Stephen Tueller (RTI International)

Jiantong Wang (RTI International)

Jonaki Bose (Substance Abuse and Mental Health 

Services Administration)

Sarra Hedden (Substance Abuse and Mental Health 

Services Administration) 

Using Classification Trees to Recommend Hot 

Deck Imputation Methods: A Case Study 

Laura Bechtel (U.S. Census Bureau)

Darcy Steeg Morris (U.S. Census Bureau)

Katherine Jenny Thompson (U.S. Census Bureau) 

Within-Industry Productivity Dispersion and 

Imputation for Missing Data

T. Kirk White (U.S. Census Bureau)

Jerome P. Reiter (Duke University)

Amil Petrin (University of Minnesota and NBER) 

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION E-2:

INNOVATIONS IN SAMPLE DESIGN

Location: Room 146B

Organizer and Chair: David Kashihara (Agency for 

Healthcare Research and Quality)

Optimal Stratification and Allocation for the 

June Area Survey

Jonathan Lisic (National Agricultural Statistics Service)

Hejian Sang (Iowa State University)

Zhengyuan Zhu (Iowa State University)

Stephanie Zimmer (Iowa State University)

NHTSA’s Data Modernization Project

Fan Zhang (National Highway Traffic Safety 

Administration)

Chou-Lin Chen (National Highway Traffic Safety 

Administration)

Eun Young Noh (National Highway Traffic Safety 

Administration)

Rajesh.Subramanian (National Highway Traffic Safety 

Administration)

Discussant: Robert Baskin (Retired)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION E-3:

INCREASING RESPONSE RATES

Location: Room 146C

Organizer and Chair: Kenneth Steve (U.S. 

Department of Transportation)

Getting a Boost: Improving Cooperation Rates 

in Youth and Young Adults

Jennifer Cantrell (Truth Initiative) 

Randall Thomas (GfK Custom Research)

Elizabeth Hair (Truth Initiative)

Donna Vallone (Truth Initiative)

Designing a Multipurpose Longitudinal 

Incentives Experiment for the Survey of 

Income and Program Participation

Ashley Westra (U.S. Census Bureau)

Mahdi Sundukchi (U.S. Census Bureau)

Tracy Mattingly (U.S. Census Bureau)

Incentive Types and Amounts in a Web-based 

Survey of College Students

Chris Krebs (RTI International)

Michael Planty (Bureau of Justice Statistics)

Jessica Stroop (Bureau of Justice Statistics)

Marcus Berzofsky (RTI International)

Christine Lindquist (RTI International)

Efforts to Address Respondent Concerns in the 

American Community Survey

Todd Hughes (U.S. Census Bureau) 
WEDNESDAY, DECEMBER 2 

12

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION E-4:

DEMOGRAPHICS IN LARGE-SCALE 

FEDERAL SURVEYS: THE IMPACT OF 

MEASUREMENT AND CLASSIFICATION 

CHOICES ON KEY SOCIAL INDICATORS

Location: Room 147B

Organizer and Chair: Erica Smith (Bureau of Justice 

Statistics)

As Time Goes By: How Period Data Influence 

the Estimates of Recently Arrived Immigrants 

in the American Community Survey

Elizabeth Grieco (U.S. Census Bureau)

Luke Larsen (U.S. Census Bureau)

Howard Hogan (U.S. Census Bureau)

Comparison of Migration Data: 2013 American 

Community Survey and Annual Social and 

Economic Supplement of the Current 

Population Survey

David Ihrke (U.S. Census Bureau)

William Koerber (U.S. Census Bureau)

Alison Fields (U.S. Census Bureau) 

A New Approach to Classify and Analyze 

Changing Social Structure by Race/Ethnicity 

Across Censuses: Complex Households

Laurel Schwede (U.S. Census Bureau) 

Creating the American Opportunity Study 

(AOS): A New Tool to Study Trends in Social 

Mobility

Carol House (National Academy of Sciences)

J. Trent Alexander (U.S. Census Bureau)

Jonathan Fisher (Stanford University)

David Grusky (Standord University)

Michael Hout (New York University)

Amy O’Hara (U.S. Census Bureau)

Aliya Saperstein (Stanford University)

Matthew Snipp (Stanford University)

Travel Time Use Over Five Decades

Chao Wei (George Washington University)

Chen Song (George Washington University)

12:15 - 1:45 p.m.

Lunch on Your Own

1:45 - 3:30 p.m.

CONCURRENT SESSION F-1:

COMPARING AND UNDERSTANDING 

DIFFERENCES IN ESTIMATES ACROSS 

SURVEYS

Location: Room 146A

Organizer : Jeffrey M. Gonzalez (Bureau of Labor 

Statistics)

Chair: Wendy Martinez (Bureau of Labor Statistics)

Health Care Use and Expenditure Data for 

Elderly Medicare Beneficiaries: A Comparison 

of Two Surveys

Lisa Mirel (Centers for Medicare and Medicaid Services)

Steven Machlin (Agency for Healthcare Research and 

Quality)

Comparability in Race Reporting Between 

Cancer Registry Data and the National 

Surveys: Results From SEER-NLMS Linked Data

Mandi Yu (National Cancer Institute)

Sean Altekruse (National Cancer Institute)

Kathleen Cronin (National Cancer Institute)

Self-Reported vs. Administrative 

Race/Ethnicity Data and Racial Disparities in 

Criminal Justice System

Zhen Zeng (Bureau of Justice Statistics)

E. Ann Carson (Bureau of Justice Statistics)

Discussant: Paul Guerino (Centers for Medicare and 

Medicaid Services)
WEDNESDAY, DECEMBER 2 

13

1:45 - 3:30 p.m.

CONCURRENT SESSION F-2:

LINGUISTIC AND CULTURAL ISSUES

IN SURVEY DESIGN AND EVALUATION 

RESEARCH

Location: Room 146B

Organizer: John Cromartie (Economic Research 

Service)

Chair: Susan Jenkins (U.S. Department of Health and 

Human Services)

Multilingual, Multicultural, Multi-mode Testing 

of Questions for U.S. Census 2020

Alisú Schoua-Glusberg (Research Support Services)

Patricia Goerman (U.S. Census Bureau)

Mandy Sha (RTI International)

Does a Spanish Translation of USDA’s U.S. 

Household Food Security Survey Module 

Reduce Differential Item Functioning Between

Hispanic and White non-Hispanic Households?

Matthew Rabbitt (Economic Research Service)

Alisha Coleman-Jensen (Economic Research Service)

Adapting the Medicine Wheel Model to Extend 

the Applicability of the Traditional Logic Model 

in Evaluation Research

Susan Jenkins (U.S. Department of Health and Human 

Services)

Kristen Robinson (Social & Scientific Systems, Inc.)

Discussant: Mark Nord (UN Food and Agriculture 

Organization)

1:45 - 3:30 p.m.

CONCURRENT SESSION F-3:

PREDICTIVE ANALYTICS IN A RANGE 

OF CIRCUMSTANCES

Location: Room 146C

Organizer: Margaret Conomos (U.S. Environmental 

Protection Agency)

Chair: Barry Nussbaum (U.S. Environmental Protection 

Agency)

A Predictive Model of Patient Readmission 

Using Combined ICD-9 Codes as Engineered 

Features

Robert P. Yerex (University of Virginia Medical System)

Predictive Analytics with Administrative Data 

from the Mine Safety and Health 

Administration

Yuwen Dai (Summit Consulting, LLC)

Natalie Patten (Summit Consulting, LLC)

Albert Lee (Summit Consulting, LLC)

George Cave (Summit Consulting, LLC) 

Responsive Design in Suicide Prevention 

Program Data Collection

Aruna Rikhi (Headquarters Marine Corps Behavioral 

Health)

Jessica Jagger (Headquarters Marine Corps Behavioral 

Health)

Adam Walsh (Headquarters Marine Corps Behavioral 

Health)

Scenario Forecasting of Insurance Claims 

Based on the Representative Concentration 

Pathways (RCPs)

Vyacheslav Lyubchich (University of Maryland Center for 

Environmental Science)

Discussant: Michael Messner (U.S. Environmental 

Protection Agency)
WEDNESDAY, DECEMBER 2 

14

1:45 - 3:30 p.m.

CONCURRENT SESSION F-4:

GEOSPATIAL TECHNIQUES

Location: Room 147B

Organizer and Chair: Darius Singpurwalla (National 

Science Foundation)

Spatial Data Analyses, Visualization, and 

Research Opportunities in the Division of 

Cancer Control and Populations Sciences at 

the US National Cancer Institute

Zaria Tatalovich (National Institute of Health)

Li Zhu (National Institute of Health)

David Berrigan (National Institute of Health)

Integrating Geospatial and Public Health Data: 

Examples Using National Center for Health 

Statistics Data Systems

Lauren Rossen (National Center for Health Statistics) 

Patsy Lloyd (National Center for Health Statistics)

Geospatial Data for Statistical Use From the 

U.S. Department of Transportation

Edward Strocko (Bureau of Transportation Statistics)

Stratification of an Agricultural Area Sampling 

Frame Using Geospatial Cultivation and Crop 

Planting Frequency Data Layers

Claire Boryan (National Agricultural Statistics Service)

Zhengwei Yang (National Agricultural Statistics Service)

Patrick Willis (National Agricultural Statistics Service)

Discussant: Rolf Schmitt (Bureau of Transportation 

Statistics)

3:30 - 3:45 p.m.

Break

(Concourse)

3:45 - 5:30 p.m.

CONCURRENT SESSION G-1:

REDUCING MEASUREMENT ERROR: 

EDITING TECHNIQUES AND USE OF 

ADMINISTRATIVE DATA

Location: Room 146A

Organizer and Chair: Kristina Schafer (U.S. 

Department of Labor)

Editing Index – A New Methodology of 

Identifying Response Error

Chin-Fang Weng (U.S. Census Bureau) 

Using Administrative Data to Ensure Data 

Quality in Anonymous Surveys

Marina Murray (Peace Corps)

Angel Velarde (Peace Corps)

Exploring the Use of External Data in the 

American Community Survey: Opportunities 

and Challenges

Amy O’Hara (U.S. Census Bureau)

Deborah Stempowski (U.S. Census Bureau)

What Do We Know About the Presence of 

Young Children in Administrative Records

William O’Hare (The Annie E. Casey Foundation)

3:45 - 5:30 p.m.

CONCURRENT SESSION G-2:

RESPONDENT CONTACT TECHNIQUES 

AND RESPONSE RATES

Location: Room 146B

Organizer and Chair: Lynn Langton (Bureau of Justice 

Statistics)

Mandatory Messaging in the American 

Community Survey

Elizabeth Poehler (U.S. Census Bureau)

Dorothy Barth (U.S. Census Bureau)

Todd Hughes (U.S. Census Bureau)
WEDNESDAY, DECEMBER 2 

15

The Impact of Greeting Personalization on 

Estimates of Sexual Assault Victimization

Ashley Richards (RTI International)

Marcus Berzofsky (RTI International)

Kimberly Peterson (RTI International)

Christine Lindquist (RTI International)

Christopher Krebs (RTI International)

Michael Planty (Bureau of Justice Statistics)

Lynn Langton (Bureau of Justice Statistics)

Jessica Stroop (Bureau of Justice Statistics)

 

Testing a Model-Directed, Mixed Mode Protocol 

in the RECS Pilot Study

Stephanie Zimmer (RTI International)

Paul Biemer (RTI International)

Phillip Kott (RTI International)

Chip Berry (Energy Information Administration)

An Experiment Testing Alternative Email 

Contact Timing Strategies in a Web-Based 

Survey of Federal Personnel

Taylor Lewis (U.S. Office of Personnel Management)

Karl Hess (U.S. Office of Personnel Management)

3:45 - 5:30 p.m.

CONCURRENT SESSION G-3:

HEALTH CARE: MEASURING QUALITY, 

SPENDING AND REFORM

Location: Room 146C

Organizer and Chair: Amy Branum (National Center 

for Health Statistics)

Adjusting the Measurement of the Output of 

the Medical Sector for Quality: A Review of the 

Literature

Anne Hall (Bureau of Economic Analysis)

What Medical Conditions Are Driving the 

Spending Slowdown? An Analysis Using the 

New Health Care Account from the Bureau of 

Economic Analysis

Abe Dunn (Bureau of Economic Analysis)

Lindsey Rittmueller (Bureau of Economic Analysis) 

Bryn Whitmire (Bureau of Economic Analysis) 

State-Level Estimates from the NHIS 

Restricted Data: Analysis to support states 

implementation and evaluation of the ACA

Joanna Turner (University of Minnesota - State Health 

Access Data Assistance Center)

Discussant: Anjel Vahratian (National Center for Health 

Statistics)

3:45 - 5:30 p.m.

CONCURRENT SESSION G-4:

WHAT’S NEW WITH LONGITUDINAL 

SURVEYS?

Location: Room 147B

Organizer: Amy Branum (National Center for Health 

Statistics)

Chair: Sandra Decker (Agency for Healthcare Research 

and Quality)

An Early Look at the Consumer Expenditure 

Survey Gemini Redesign Proof of Concept Test 

Laura Erhard (Bureau of Labor Statistics)

Jennifer Edgar (Bureau of Labor Statistics)

Nhien To (Bureau of Labor Statistics)

An Overview of the New Survey of Income and

Program Participation

Jason Fields (U.S. Census Bureau)

Matthew Marlay (U.S. Census Bureau)

Patrick Campanello (U.S. Census Bureau)

Design of a National Longitudinal Survey of 

Small Businesses to Assess the Early Impact 

of Healthcare Reform

David Kashihara (Agency for Healthcare Research and 

Quality)

Robert Baskin (Retired)

Movement and Changing Environment on the 

National Children’s Study: How can we 

Understand the Impact on a Longitudinal 

Population Survey

Ned English (National Opinion Research Center at the 

University of Chicago)

Edward Mulrow (National Opinion Research Center at the 

University of Chicago)

Jennifer Hasche (National Opinion Research Center at the 

University of Chicago)

Katie Dekker (National Opinion Research Center at the 

University of Chicago)

Christina Park (National Institutes of Health)

Discussant: Chris Chapman (National Center for 

Education Statistics)
THURSDAY, DECEMBER 3 

16

7:30 a.m. - 3:30 p.m.

Registration

(Concourse)

7:30 - 8:30 a.m.

Continental Breakfast

(Concourse)

8:30 - 10:15 a.m.

CONCURRENT SESSION H-1: 

CHALLENGES AND BENEFITS FROM 

LINKING ADMINISTRATIVE AND 

SURVEY DATA

Location: Room 146A

Organizer: Pamela McGovern (National Agricultural 

Statistics Service)

Chair: Daniel Cork (Committee on National Statistics, 

The National Academies of Sciences, Engineering, and 

Medicine)

Creating Improved Survey Data Products Using 

Linked Administrative-Survey Data

Michael Davern (National Opinion Research Center at the 

University of Chicago)

Bruce Meyer (University of Chicago)

Developing a Residence Candidate File for Use 

With Employer-Employee Matched Data

Matthew Graham (U.S. Census Bureau)

Mark Kutzbach (U.S. Census Bureau)

Danielle Sandler (U.S. Census Bureau)

Using the Census to Evaluate Administrative 

Records and Vice Versa

J. David Brown (U.S. Census Bureau)

Jennifer Childs (U.S. Census Bureau)

Amy O’Hara (U.S. Census Bureau)

Discussant: Katharine Abraham (University of 

Maryland)

8:30 - 10:15 a.m.

CONCURRENT SESSION H-2:

MULTIMODE DATA COLLECTION 

Location: Room 146B

Organizer: Michael Planty (Bureau of Justice Statistics)

Chair: Cynthia Clark (Clark Survey Consulting)

Data-Driven Decision Making and the Design of 

Economic Census Data Collection Instruments

Eric Fink (U.S. Census Bureau)

Jennifer Beck (U.S. Census Bureau)

Diane Willimack (U.S. Census Bureau)

Integrated Management of Survey Modes

Jerome Wernimont (Westat)

Ray Snowden (Westat)

Evaluating Collection Mode Effect with Web 

Data Collection in the Canadian Labor Force 

Survey

Justin Francis (Statistics Canada)

Guy Laflamme (Statistics Canada)

Agreement Across Modes of Collection in the 

Occupational Requirements Survey: Results 

From a Pilot Job Observation Test

Kristen Monaco (Bureau of Labor Statistics)

Kristin Smyth (Bureau of Labor Statistics)

Tiffany Chang (Bureau of Labor Statistics)

8:30 - 10:15 a.m.

CONCURRENT SESSION H-3:

FRAME DEVELOPMENT AND 

RESPONDENT SELECTION 

Location: Room 146C

Organizer: Yan Liu (Statistics of Income Division, 

Internal Revenue Service)

Chair: Tracy Haines (Statistics of Income Division, 

Internal Revenue Service)

Vaping the Web: Crowdsourcing and Web 

Scraping for Establishment Survey Frame 

Generation

Bryan Rhodes (RTI International)

Annice Kim (RTI International)

Brett Loomis (RTI International)
THURSDAY, DECEMBER 3 

17

Using a Mobile Mapping Instrument to Evaluate 

a Permanent Grid Sampling Frame

Denise Abreu (National Agricultural Statistics Service)

Linda Lawson (National Agricultural Statistics Service)

Claire Boryan (National Agricultural Statistics Service)

Michael Gerling (National Agricultural Statistics Service)

Rick Hardin (National Agricultural Statistics Service)

Quantifying Urban Agriculture: A Case Study 

from Baltimore

Linda Young (National Agricultural Statistics Service)

A Meta-analysis of Within-Household 

Respondent Selection Methods 

Ting Yan (Westat)

Roger Tourangeau (Westat)

Rose McAloon (University of Maryland) 

8:30 - 10:15 a.m.

CONCURRENT SESSION H-4: 

HEALTH INSURANCE ESTIMATES: 

ASSESSING THE IMPACT OF A 

REDESIGN 

Location: Room 147B

Organizer and Chair: Stephen Blumberg (National 

Center for Health Statistics)

How Did the Method Change in the CPS ASEC 

Affect Health Insurance Estimates?

Carla Medalia (U.S. Census Bureau)

Brett O’Hara (U.S. Census Bureau) 

Estimating a Time-Trend of the Uninsured Rate 

for the CPS Parallel ASEC

Brett O’Hara (U.S. Census Bureau)

Carla Medalia (U.S. Census Bureau)

Comparing 2014 SIPP and 2014 CPS ASEC 

Employer-Sponsored Insurance Offer and Takeup Rates 

Joelle Abramowitz (U.S. Census Bureau)

Medicare Coverage and Reporting of the 

Elderly Population: A Comparison of CPS and 

Administrative Records

Renuka Bhaskar (U.S. Census Bureau)

James Noon (U.S. Census Bureau)

Sonya Rastogi (U.S. Census Bureau)

Brett O’Hara (U.S. Census Bureau)

Victoria Velkoff (U.S. Census Bureau)

10:15 - 10:30 a.m.

Break

(Concourse)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION I-1:

EXPERIENCES AND EVALUATIONS IN 

USING RECORD LINKAGE TO IMPROVE 

THE QUALITY OF SURVEY PRODUCTS 

AND DATA 

Location: Room 146A

Organizer: Pamela McGovern (National Agricultural 

Statistics Service)

Chair: Mark Gorsak (National Agricultural Statistics 

Service)

Simulated Versus Actual SNAP Unit 

Composition in Survey Households in Two 

States 

Karen Cunnyngham (Mathematica Policy Research)

John Czajka (Mathematica Policy Research)

Randy Rosso (Mathematica Policy Research)

Program Confusion in the 2014 SIPP: Using

Administrative Records to Correct False 

Positive SSI Reports

Katherine Giefer (U.S. Census Bureau)

Abby Williams (U.S. Census Bureau)

Gary Benedetto (U.S. Census Bureau)

Joanna Motro (U.S. Census Bureau)
THURSDAY, DECEMBER 3 

18

Evaluation of Record Linkage for SEER Breast 

Cancer Registries to Oncotype DX Assays

Michael Larsen (George Washington University)

Will Howe (Information Management Services)

Nicola Schussler (Information Management Services)

Benmei Liu (National Cancer Institute)

Valentina Petkov (National Cancer Institute)

Mandi Yu (National Cancer Institute)

Discussant: Amy O’Hara (U.S. Census Bureau)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION I-2:

MOBILE SURVEY DESIGN

Location: Room 146B

Organizer and Chair: Stephen Blumberg (National 

Center for Health Statistics)

Optimizing the Decennial Census for Mobile – A 

Case Study

Elizabeth Nichols (U.S. Census Bureau)

Erica Olmsted Hawala (U.S. Census Bureau)

Rachel Horwitz (U.S. Census Bureau)

Michael Bentley (U.S. Census Bureau)

Testing the Feasibility of Collecting 

Expenditure Data via Individual Mobile and 

Web Diaries 

Ian Elkin (Bureau of Labor Statistics)

Brett McBride (Bureau of Labor Statistics)

Lucilla Tan (Bureau of Labor Statistics) 

Gridlock: Improving Questionnaire Design for 

Mobile Devices

Randall Thomas (GfK Custom Research) 

Frances Barlas (GfK Custom Research) 

Measurement on the Move: Designing 

Questionnaires for Mobile First

Frances Barlas (GfK Custom Research)

Randall Thomas (GfK Custom Research)

Patricia Graham (GfK Custom Research)

A Summary of Issues Associated With the 

Modernization of Data Collection

Linda Lawson (National Agricultural Statistics Service)

Denise Abreu (National Agricultural Statistics Service)

Claire Boryan (National Agricultural Statistics Service)

Michael Gerling (National Agricultural Statistics Service)

Rick Hardin (National Agricultural Statistics Service)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION I-3:

ISSUES WITH DISCLOSURE AND DATA 

CONFIDENTIALITY

Location: Room 146C

Organizer and Chair: Andrew White (National Center 

for Education Statistics)

Formal Privacy Protection for Data Products 

Combining Individual and Employer Frames

Ashwin Machanavajjhala (Duke University)

Samuel Haney (Duke University)

Matthew Graham (U.S. Census Bureau)

Mark Kutzbach (U.S. Census Bureau)

John Abowd (U.S. Census Bureau, Cornell University)

Methods of Selecting Variables in Data 

Disclosure 

Charles Lin (Department of Veterans Affairs)

Hwai-Tai Lam (Department of Veterans Affairs)

Lijia Guo (Department of Veterans Affairs)

Tom Garin (Department of Veterans Affairs)

Considering the Usefulness of Standard FOIA 

Provisions in Interagency Agreements

Jennifer Goode, PhD (U.S. Census Bureau)

Michael Toland, PhD (U.S. Census Bureau)

Hampton Wilson (U.S. Census Bureau)

Rana Wahdan (U.S. Census Bureau)

Discussant: Alan Karr (RTI International)

10:30 a.m. - 12:15 p.m.

CONCURRENT SESSION I-4:

MODELING PRELIMINARY ESTIMATES 

Location: Room 147B

Organizer: Benjamin Bridgman (Bureau of Economic 

Analysis)

Chair: Ryan Howley (Bureau of Economic Analysis)

Early Estimates of Annual Manufacturing 

Industry Output

Peter Meyer (Bureau of Labor Statistics)

Jason McClellan (Bureau of Labor Statistics)

Jennifer Price (Bureau of Labor Statistics)

Sam Rowe (Bureau of Labor Statistics)
THURSDAY, DECEMBER 3 

19

Early Monthly Estimates for the Services 

Sector

Pieter Vlag (Statistics Netherlands) 

Ton de Waal (Statistics Netherlands) 

Cagan Kaya (Turkish National Statistical Institute)

Medicaid and CHIP Data Methodology for 

SAHIE Models

David Powers (U.S. Census Bureau)

Lauren Bowers (U.S. Census Bureau)

Wesley Basel (U.S. Census Bureau)

Samuel Szelepka (U.S. Census Bureau)

Discussant: Sarah Konya (U.S. Census Bureau)

12:15 p.m. - 1:45 p.m.

Lunch on Your Own

1:45 - 3:30 p.m.

CONCURRENT SESSION J-1:

ADAPTIVE DESIGN II

Location: Room 146A

Organizer and Chair: Steve Klement (U.S. Census 

Bureau)

Managing Survey Costs: Using Mobile 

Technology Coupled With Performance 

Dashboards to Improve Field Operations

Brad Edwards (Westat)

Richard Dulaney (Westat)

Abie Reifer (Westat)

Tamara Bruce (Westat)

Adaptive and Responsive Survey Designs: 

A Review and Assessment

Roger Tourangeau (Westat)

J. Michael Brick (Westat)

Sharon Lohr (Westat)

Jane Li (Westat)

Innovative Uses of Paradata in the Survey of 

Income and Program Participation

Matthew Marlay (U.S. Census Bureau)

Holly Fee (U.S. Census Bureau)

Jason Fields (U.S. Census Bureau)

Using Computer-Assisted Recorded 

Interviewing to Enhance Field Monitoring and 

Improve Data Quality 

Holly Fee (U.S. Census Bureau)

Matthew Marlay (U.S. Census Bureau)

Jason Fields (U.S. Census Bureau)

1:45 - 3:30 p.m.

CONCURRENT SESSION J-2:

WEALTH DATA AND WEALTH DATA 

QUALITY

Location: Room 146B

Organizer: Jesse Bricker (Federal Reserve Board)

Chair: Mark Klee (U.S. Census Bureau)

Changes in the Distribution of After-Tax 

Wealth: Has Income Tax Policy Increased 

Wealth Inequality?

Kevin Moore (Federal Reserve Board)

Adam Looney (Brookings Institution)

Rounding in Household Financial Surveys: The 

Roles of Survey Design and Individual 

Characteristics 

Michael Gideon (U.S. Census Bureau)

Joanne Hsu (Federal Reserve Board of Governors)

Brooke Helppie McFall (Institute for Social Research, 

University of Michigan) 

Reassessing Wealth Data Quality in the Survey 

of Income and Program Participation

Jonathan Eggleston (U.S. Census Bureau)

Mark A. Klee (U.S. Census Bureau)

Measuring the Wealth of Hard-to-Find U.S. 

Minorities: Sampling, Recruiting and Weighting 

Issues

Thomas Guterbock (University of Virginia)

James Ellis (University of Virginia)

Deborah Rexrode (University of Virginia)

Casey Eggleston (U.S. Census Bureau)

Darrick Hamilton (The New School)

William Darity, Jr. (Duke University)
THURSDAY, DECEMBER 3 

20

1:45 - 3:30 p.m.

CONCURRENT SESSION J-3:

REMOTE ACCESS AND ANALYSIS

TOOLS FOR WORKING WITH 

GOVERNMENT DATA

Location: Room 146C

Organizer: Michael Freiman (U.S. Census Bureau)

Chair: Bryan Schar (U.S. Census Bureau)

Education Data in the Cloud

Shelley Burns (National Center for Education Statistics) 

Brian Taylor (National Center for Education Statistics)

Marilyn Seastrom (National Center for Education 

Statistics)

Ted Socha (National Center for Education Statistics)

Research Using Remote Access or Remote 

Execution Systems

Peter Meyer (National Center for Health Statistics)

Vijay Gambhir (National Center for Health Statistics)

Corey Decker (National Center for Health Statistics)

James Craver (National Center for Health Statistics)

SAMHSA Data Portal

Brooklyn Lupari (Substance Abuse and Mental Health 

Services Administration)

Developing and Testing the Microdata Analysis 

System at the US Census Bureau

Bryan Schar (U.S. Census Bureau)

Michael Freiman (U.S. Census Bureau)

Amy Lauger (U.S. Census Bureau)

1:45 - 3:30 p.m.

CONCURRENT SESSION J-4:

JPSM SESSION AT 2015 FCSM 

CONFERENCE 

Location: Room 147B

Organizer: Andrew Zukerberg (National Center for 

Education Statistics)

Chair: Richard Valliant (Joint Program on Survey 

Methodology)

New Developments at JPSM

Richard Valliant (Joint Program in Survey Methodology)

Frauke Kreuter (Joint Program in Survey Methodology)

A Comparison of Ex-Ante, Laboratory, and Field 

Methods for Evaluating Survey Questions

Aaron Maitland (Westat)

The Impact of Rapport on Data Quality in CAPI 

and Video-mediated Interviews: Disclosure of 

Sensitive Information and Item Nonresponse

Hanyu Sun (Westat)

Topics in Model-Assisted Point and Variance 

Estimation in Clustered Samples

Timothy Kennel (U.S. Census Bureau)

The Use of Responsive Split Questionnaires in 

a Panel Survey

Jeffrey M. Gonzalez (Bureau of Labor Statistics)

Testing the Relationship Between Social 

Integration and Nonresponse

Ashley Amaya (Joint Program in Survey Methodology)

Sample Design in Three-stage Household 

Surveys Supplemented by Commercial Address 

Lists

Alena Maze (Joint Program in Survey Methodology)
21

This page intentionally left blank
22

Abstract

Booklet

This section represents abstracts received as of September 1, 2015. 

The following abstracts have not been edited for content.
Concurrent Session A Abstracts

23

CONCURRENT SESSION A-1

LINKING ADMINISTRATIVE RECORDS TO ENHANCE DATA

Linking Administrative BJS Data: Understanding Jail Mortality Through the 

Communities They Serve

Margaret Noonan (Bureau of Justice Statistics), Susan Brumbaugh (RTI International), and Chris Ellis (RTI 

International)

BJS has been collecting individual-level death data on jail inmates dying in custody through the Deaths in 

Custody Reporting Program (DCRP) since 2000. From the start of the collection, BJS has been interested in 

how the characteristics of the community a jail serves affects jail mortality outcomes. In addition to death data, 

the DCRP collects limited data, e.g. average daily population, annual admissions and a confinement count, on 

all jails in the United States via its annual summary form (ASF). 

While the ASF data is sufficient to calculate jail mortality rates, it is by design limited in its scope. In order to fully 

understand the link between jail and community characteristics, BJS tasked its data collection agent, RTI 

International to merge the DCRP death files with the 2013 Census of Jails (‘census’). Unlike the ASF, the 

infrequently fielded census collects data on the administration on jails and the inmates they serve. 

The paper will discuss the process through which BJS and RTI identified more than a dozen potential 

community files eligible for linking to the merged jail death-census files. Measures included indicators of poverty, 

crime, social disorder, substance abuse, mental and physical health as well as demographic and economic 

information. Ultimately, nine community-level files, such as the American Community Survey, the Wide-ranging 

Online Data for Epidemiologic Research, the Uniform Crime Report and the Area Health Resource File among 

others, created a comprehensive community-level file that will allow BJS to group jails on community and jail 

characteristics, with the ultimate goal of understand how these factors affect jail inmate mortality.

Linking Administrative BJS Data: Better Understanding Prisoners’ Personal 

Histories by Linking the National Corrections Reporting Program and CARRA Data 

E. Ann Carson, Ph.D. (Bureau of Justice Statistics)

BJS is committed to leveraging administrative data to better understand external factors that could contribute to 

successful reentry into the community by former inmates. During 2014, BJS executed and funded an 

interagency agreement (IAA) with the Center for Administrative Records Research and Applications (CARRA) at 

the U.S. Census Bureau’s Center for Economic Studies (CES). The main goal of this IAA is to link the NCRP 

data to the Social Security Administration’s (SSA) Numident file behind the U.S. Census’ secure firewall, which 

then allows analysts in the CARRA group to assign a personal identification key (PIK) to each inmate and delete 

all personal information. 

The PIK lets the NCRP data to be linked to a number of federal datatsets including: data on receipt of 

supplemental security income; Temporary Assistance for Needy Families (TANF); public housing and rental 

assistance history; Department of Housing and Urban Development-insured mortgage loans; SSA’s Death 

Master File; enrollment in Medicare; and any listing in Census’ decennial census or American Community 

Survey (ACS). With additional approval, NCRP data could be PIK’d to unemployment insurance (UI) wage data 

collected by CES through the Longitudinal Employer-Household Dynamics (LEHD), as well as to tax returns 

from the Internal Revenue Service (IRS). Linking NCRP and CARRA data allows BJS to make statements about 

the experiences of inmates before and after imprisonment without adding additional burden to our respondents.
Concurrent Session A Abstracts

24

Linking Administrative BJS Data: Linking the National Death Index and the Deaths 

in Custody Reporting Program to Better Understand Correctional Mortality

Zhen Zeng, Ph.D. (Bureau of Justice Statistics) and Margaret Noonan (Bureau of Justice Statistics)

This report evaluates the quality of record matching between the Death in Custody Reporting Program (DCRP) 

and the National Death Index (NDI) database. The goal is to assess whether the NDI provides more complete 

and accurate information on cause of death (COD) than the DCRP data. The records of 17,358 inmates 

reported to have died in local jails and state prisons from 2007 to 2010 in the DCRP were matched to NDI 

database. Ninety-four percent of the death records found matches in the NDI. For matched records, the NDI 

provides significantly more International Classification Codes (ICD-10) per death record than DCRP data (2.5 

vs. 1.1 codes per death). However, due to unmatched records, the overall rate of missing data on cause of 

death is higher in the NDI than in the DCRP (6.3% vs. 1.9%). Overall, 70% of matched records fall into the same 

broad categories of cause of death in NDI and DCRP, e.g. a DCRP cancer death was also listed as a cancer 

death in NDI. The agreement rate is higher for jail deaths than for prison deaths; for accidental deaths than for 

illness deaths, for single-cause deaths than for multiple-cause deaths; and for records with higher probabilistic 

match scores. Evidence suggests that differences in cause of death between the DCRP and NDI are due partly 

to record mismatches, partly to differences in COD categorization, as well as data and procedural differences 

between the DCRP and the NDI.

CONCURRENT SESSION A-2

DESIGNING WEB SURVEYS

Internet and Smartphone Coverage in a National Health Survey: Implications for 

Alternative Modes

Mick P. Couper, Jennifer Kelley, William Axinn, Heidi Guyer, James Wagner, and Brady West (Survey Research 

Center, University of Michigan)

The rise of Internet-enabled smartphones presents an opportunity to re-examine the issue of Internet coverage 

and its implications for coverage bias. While a number of papers have examined cell phone coverage and 

Internet coverage separately, few have looked at the possible compensatory effects of joint coverage. We 

added two interviewer observations (one on Internet access and the other on smartphone ownership among 

respondents) to the National Survey of Family Growth (NSFG) with a view to exploring the feasibility of Internetbased follow-up surveys. NSFG is a national probability survey of women and men age 15-44, using a 

continuous design. We examine 8 quarters (2 years) of data, from September 2012 to August 2014.

Overall, we find that 82.2% of respondent report Internet access and 76.1% report having a smartphone 

(estimates weighted for differential selection and nonresponse). Combined, this means that 89.4% have access 

to the Internet, either via traditional devices or via a smartphone. We also find some evidence of compensatory 

coverage when looking at key gender/race/age subgroups. For instance, while Black male teens (15-17) have 

the lowest rate of Internet access (74.9%) and the lowest rate of smartphone usage (58.9%), when combined 

82.6% have some form of Internet access.

We propose to examine the socio-demographic correlates of Internet and smartphone (and combined) coverage 

(access) in this population. In addition, we propose to look at the effect of differential coverage on key estimates 

produced by the NSFG, related to fertility, family formation, and sexual activity. While this does not address 

nonresponse bias issues related to alternative modes, our paper has implications for possible coverage biases 

that may arise in switching to a Web-based mode of data collection, either for follow-up surveys or to replace the 

main face-to-face data collection.
Concurrent Session A Abstracts

25

Designing Web Surveys for Different Screen Sizes and Operating Systems: What 

Respondents are Using. 

Robert D. Tortora (ICF International) 

An overall principle of sound questionnaire design is to present the same stimulus to respondents to minimize 

the chance of introducing measurement error. With the increasing use of a variety of devices including Iphones, 

androids, tablets, and desktop and laptop computers that respondents can use to complete web surveys 

questionnaire designers face new challenges in designing web questionnaires to present the same stimulus. In 

order to make good design decisions ICF is collecting data on the frequency of use of these types of devices as 

well as on the type of operating system used. This data are collected for those that complete an ICF web 

survey and for those that login but either do not attempt the survey or breakoff before completing the web 

questionnaire. Paradata and metadata will be presented with the goal of aiding design decisions for web 

surveys to reduce potential measurement error.

Characteristics of the Population of Internet Panel Members

John M. Boyle, Naomi Freedner, and Tala Fakhouri (ICF International)

Despite concerns about the non-probability basis for web surveys, an increasing number of studies have found 

that sample estimates from web surveys compare favorably to those from probability surveys. Aside from the 

inability to apply probability statistics to these web surveys, potential users worry about how reliable are these 

estimates. 

This paper examines one element that may contribute to the representativeness of web surveys, at least in 

certain circumstances. Who are the participants and non-participants of these now ubiquitous sources used in 

population estimates and how similar are they to the population they are meant to represent? 

The purpose of this paper is to investigate the actual coverage of Internet panels, generally, among American

adults. Rather than look at the composition of any one online survey, we consider the characteristics of the 

population who participate in any web based panel surveys. Since web panels frequently “partner” with other 

web panel organizations in order to generate samples that are larger, more diverse or more specific than what is 

available within their panel members, the population who participate in any web panels is more critical to 

evaluating coverage error and bias of this form of survey than specific coverage and bias errors in individual 

panels.

A national probability survey of adults was conducted using a dual frame, random digit dialing sample 

determined participation in Internet survey panels and frequency of participation. Using several classifications 

for web panelist, we compare the population characteristics of the universe of web panelists compared to the 

general public. These findings allow us to characterize the degree of coverage and bias associated with the 

general population of web panelists. We believe that this is the first step in understanding the issues associated 

with the representativeness and non-representativeness of web survey findings.
Concurrent Session A Abstracts

26

CONCURRENT SESSION A-3

NONRESPONSE ADJUSTMENT AND VARIANCE ESTIMATION

JOLTS Variance Estimation to Account for JOLTS-CES Alignment

Mark Crankshaw (Bureau of Labor Statistics) and Michael Sverchkov (Bureau of Labor Statistics)

The Bureau of Labor Statistics produces a monthly measure of labor demand and employment dynamics called 

the Job Opening and Labor Turnover survey (JOLTS). The JOLTS survey has aligned its implied employment 

change (hires minus separations) to the published CES employment change to enhance the reliability of JOLTS 

estimates. This change in estimation methodology, however, is not presently accounted for in JOLTS variance 

estimation.

To address this shortcoming, an extension of the current Balanced-Half Sample replication technique has been 

devised that will attempt to fully account for the JOLTS-CES alignment in JOLTS variance estimation. In 

essence, the extension consists of applying the JOLTS-CES alignment procedure to each replicate estimate 

and then building an estimate of variance from the aligned replicate estimates.

The proposed extension has been applied to historical JOLTS micro-data thus allowing a comparison to be 

made between the estimates of variation without accounting for JOLTS-CES alignment and the proposed 

extension of the balanced half sample that does.

Linearization Variance Estimators for Mixed-mode Survey Data when Response 

Indicators are Modeled as Discrete-time Survival

A. Demnati (Independent Consultant)

Collecting information from sampled units over the Internet or by mail is much more cost-efficient than 

conducting interviews. These methods make self-enumeration an attractive data-collection method for surveys 

and censuses. Despite the benefits associated with self-enumeration data collection— in particular Internetbased data collection— self-enumeration can produce low response rates compared to interviews. To increase 

response rates, non-respondents are subject to a mixed mode of follow-up treatments, which influence the 

resulting probability of response, to encourage them to participate. Because response occurrence is intrinsically 

conditional, I record response occurrence in discrete intervals, and I characterize the probability of response by 

a discrete time hazard. This approach facilitates examining when a response is most likely to occur and how the 

probability of responding varies over both time and follow-up treatments. I use regression analysis to investigate 

the effect of mixed-mode on the response probability. Factors and interactions are commonly treated in 

regression analyses, and have important implications for the interpretation of statistical models. The 

nonresponse bias can be avoided by multiplying the sampling weight of respondents by the inverse of an 

estimate of the response probability. Estimators and associated variance estimators of model parameters as 

well as of parameters of interest are studied. I take into account correlation over time for the same unit in 

variance estimation. As intensive follow-up is expensive, optimal resources allocation within stages of the survey 

design is a key issue. The proportion of resources that should be allocated to sampling, follow up, and data 

collection stages is determined by minimizing the variance subject to constraint on the survey global cost. 

Simulation results on the performance of the proposed estimators and associated variance estimators are also 

presented.
Concurrent Session A Abstracts

27

Weighting and Variance Estimation Plans for the 2016 National Household Survey

François Verret (Statistics Canada), Arthur Goussanou (Statistics Canada), and Nancy Devin (Statistics 

Canada) 

In 2011, the quinquennial mandatory Canadian census long-form was replaced by the voluntary National 

Household Survey (NHS). As voluntary surveys typically have lower unit response rates than mandatory 

surveys, the census long-form methodologies were adapted in 2011 in order to reduce the risk of non-response 

bias. While every one in five household received the census long-form prior to 2011, one third of households are

sampled in the NHS. To mitigate the risk of non-response bias the method of Hansen & Hurwitz (1946) for 

selecting a follow-up sub-sample of non-respondents is used. 

The NHS will again be conducted in 2016. Methodologies will be adapted further, in particular estimation 

methodologies. The objectives in developing these are to improve upon the 2011 estimation methodologies, to 

adapt to the 2016 sample design and response/non-response patterns and to meet two new objectives related 

to variance estimation and dissemination. Planned research on the 2016 NHS estimation methodologies will be 

presented, as well as the foreseen methodologies at this stage of the research process. The following topics will 

be explored: 

1. Handling departures from the Hansen & Hurwitz setting;

2. Calibration on a large number of census calibration totals without adversely affecting variance estimators;

3. Efficient estimation of the sampling and non-response variance while meeting the dissemination objectives 

and while respecting the operational constraints of this large-scale survey. 

CONCURRENT SESSION A-4

IMPUTATION AND THE IMPACT OF NON-RESPONSE

Bias from Unit Non-Response in the Measurement of Income in Household Surveys

C. Adam Bee (U.S. Census Bureau), Graton M.R. Gathright (U.S. Census Bureau), and Bruce Meyer (University 

of Chicago) 

Declining response rates to surveys is a widespread and troubling problem. Unit non-response (when a 

household is not interviewed at all) has been rising in most surveys. For example, unit non-response rates rose 

by 3-12 percentage points over the 1990s for six U.S. Census Bureau surveys (Atrostic et al. 2001). Many 

recent papers have raised the concern that this increased non-response has led to bias in key statistics. In light 

of this pattern, we propose a new method to evaluate and correct bias from unit non-response. Specifically, we 

use addresses to link the 2011 Current Population Survey Annual Social and Economic Supplement (CPS 

ASEC) to IRS Form 1040 records. This link allows us to compare several characteristics of respondents and 

non-respondents, including income, self-employment status, marital status, number of children, and the receipt 

of pensions and certain government benefits. We then compare our new methods of assessing bias to prior 

methods. Finally, we evaluate existing weighting methods and consider alternative weighting methods and other 

corrections.
Concurrent Session A Abstracts

28

Do Imputed Earnings Earn Their Keep? Evaluating SIPP Earnings and NonResponse with Administrative Records 

Mark Klee (U.S. Census Bureau), Rebecca Chenevert (U.S. Census Bureau), and Kelly Wilkin (U.S. Census 

Bureau) 

Recent evidence suggests that labor earnings reported in household surveys compare favorably with labor 

earnings in administrative records. On the other hand, imputed labor earnings in household surveys seem to 

match labor earnings in administrative records less closely. This finding has led many researchers to question 

the reliability of imputed labor earnings and to exclude these observations from wage analyses. However, this 

strategy might result in sample selection bias if labor earnings are not missing at random. In this paper, we 

compare reported and imputed labor earnings from the 2008 and 2014 panels of the Survey of Income and 

Program Participation (SIPP) to labor earnings from the Social Security Administration’s Detailed Earnings 

Record. We examine how the relationship between survey data and administrative records varies across 

demographic groups. Finally, we characterize survey non-respondents in order to improve our understanding of 

whether and how individuals select out of response on observable dimensions.

Bridging a Survey Redesign Using Multiple Imputation: An Application to the 2014 

CPS ASEC

Jonathan Rothbaum (U.S. Census Bureau)

The Current Population Survey Annual Social and Economic Supplement serves as the data source for income, 

inequality, and official source of poverty estimates in the United States. In 2014, the CPS ASEC questionnaire 

was redesigned to improve data quality and to reduce misreporting, item nonresponse, and errors resulting from 

respondent fatigue. The sample was split into two groups, with nearly 70% receiving the traditional instrument 

and 30% receiving the redesigned instrument. Due to the relatively small redesign sample, analyses of changes 

in income and poverty between this and future years may lack sufficient power. In this paper, I explore the 

possibility of using multiple imputation techniques to combine the two subsamples into a single sample that we 

can use to estimate income and poverty statistics with greater power and smaller standard errors. Multiple 

imputation is a general approach to analyzing data with missing values. I can treat the traditional sample as if 

the responses were missing for income sources targeted by the redesign and use multiple imputation to 

generate plausible responses. I use a flexible semiparametric imputation technique to place individuals into 

strata along two dimensions: 1) their probability of income recipiency and 2) their expected income conditional 

on recipiency for each income source. Within each stratum, I randomly select redesign individuals and donate 

their income recipiency, source, and value information to individuals in the traditional sample as the imputed 

data. By matching on these two dimensions this approach combines the ideas of propensity score matching 

(from the probability of recipiency strata) and predictive means matching (from expected income strata). In this 

paper, I implement this approach, use diagnostics to evaluate the matching models, and analyze the results.
Concurrent Session B Abstracts

29

CONCURRENT SESSION B-1

ABSTRACTION TO EXTRACTION: EXPERIENCES FROM THE NATIONAL 

HOSPITAL CARE SURVEY IN USING ADMINISTRATIVE CLAIMS AND 

ELECTRONIC HEALTH RECORD DATA

Using Uniform Bill (UB)-04 Administrative Claims Data to Describe Hospital-Based 

Care 

Sonja Williams, Naga Shanmugam, Amy Brown, and Carol DeFrances (National Center for Health Statistics)

The National Hospital Care Survey (NHCS) is designed to describe national patterns of health care encounters 

in hospital-based settings. The goal of the NHCS is to move toward the collection of all inpatient and 

ambulatory data electronically over time. Hospitals can transmit either UB-04 administrative claims or electronic 

health record (EHR) data. Since EHR adoption and implementation is still in various stages, most hospitals 

participating in the NHCS send UB-04 claims data. This presentation focuses on the benefits and challenges 

encountered with the collection of UB-04 data. Benefits include the collection of personally identifiable 

information (PII) that allows for data linkages across hospital settings and to outside data sources such as the 

National Death Index; inclusion of data for outpatient settings that were previously excluded, and information on 

intensive care use and revisits. However, the collection of the UB-04 data has posed some challenges. Many 

hospitals could not send the UB-04 data in the desired file format so the data format specifications had to be 

expanded. Since more than one claim can be generated for a single episode of care, methods had to be 

developed to consolidate the data the multiple claims into one record. The identification of substance-involved 

emergency department visits from the UB-04 claims data has proved to be difficult so algorithms are being 

created utilizing diagnoses, procedure and symptom information to yield better results. In conclusion, session 

participants will gain an understanding of the benefits and challenges posed by the collection UB-04 

administrative claims data from hospitals.

Acquiring survey data from Electronic Health Record Systems: Experiences from 

the National Hospital Care Survey (NHCS)

Kristi R. Eckerson (Emory University IPA to NCHS), Brian Gugerty, Anita Bercovitz, and Carol J. DeFrances 

(National Center for Health Statistics)

The NHCS provides accurate, objective, and timely data used to inform health policy and research about care 

provided by hospitals in inpatient, emergency, and outpatient departments. NHCS is designed to collect all data 

electronically. Initially, NCHS explored the use of hospital’s UB-04 administrative claims, since it was felt the 

burden on hospitals would be minimal. More recently, hospitals are being asked to send EHR extracts as these 

data are expected to include clinical data which will present a more complete picture of health care utilization in 

the United States. This presentation discusses the multiple approaches that are being used to collect EHRbased data, including development of a standard for data reporting, working with hospitals and EHR vendors to 

extract data, and pulling data from existing hospital reports or discharge summaries. To achieve consistent and 

efficient acquisition of these data, NCHS developed an HL7 CDA® Implementation Guide which provides 

specifications for how each data element should be reported. NCHS is working directly with participating 

hospitals to create their own EHR extraction process and also working with leading EHR vendors to develop 

reusable software interfaces that extract the required data according to HL7 CDA standards. The variety of 

EHR approaches as well as the strengths and challenges posed by each will be presented and discussed.
Concurrent Session B Abstracts

30

Matching the National Hospital Care Survey records across settings and with 

outside data sources

Shaleah Levant and Monica L. Wolford (National Center for Health Statistics)

The collection of personally identifiable information (PII) in the National Hospital Care Survey (NHCS) allows for 

the creation of patient identifiers. NHCS employs a probability-based record linkage method to identify patients 

using incomplete, encounter-level PII. Using name (first, last, middle initial), date of birth, sex, hospital identifier, 

medical record number, and ZIP code, patients are identified and their records are linked. This presentation will 

explore two analyses that use the probability-based patient identifiers in order to begin to describe the 

continuum of care. First, we will describe the demographics of patients with multiple encounters, within and 

between inpatient, emergency department, and other outpatient department settings for the 2013 NHCS. 

Second, we will describe the preliminary 2011 NHCS data that have been linked to the National Death Index 

(NDI). A validation test has assessed a 94 percent match rate between NHCS in-hospital deaths and NDI death 

records. We will explore the demographics of patients with a 30-day post-discharge mortality. These analyses 

are a first step in describing the incredible potential that patient identifiers have to aid in the understanding of 

health care across hospital settings—emergency department, other outpatient departments, and inpatient 

care—as well as allow for the analysis of individual and provider characteristics associated with in-hospital and 

post-discharge mortality. The ability to match data across settings and with death records may be used to 

improve hospital care decisions in the future. 

CONCURRENT SESSION B-2

STUDYING NON-PROBABILITY STUDIES

Evaluation of Strategies to Improve Utility of Estimates from a Non-probability 

Based Survey 

Benmei Liu (National Cancer Institute), Sadeq R. Chowdhury (Agency for Healthcare Research and Quality), 

Janet deMoor (National Cancer Institute), Erin Kent (National Cancer Institute), Steven Machlin (Agency for 

Healthcare Research and Quality), Stephanie Nutt (LIVESTRONG Foundation), Anita Soni (Agency for 

Healthcare Research and Quality), Katherine S. Virgo (Emory University), Gordon B. Willis (National Cancer 

Institute), Maggie Wilson (National Cancer Institute), K. Robin Yabroff (National Cancer Institute), and the 

Interagency Consortium to Promote Health Economics Research on Cancer (HEROIC)

Response rates for probability-based sample surveys have been dropping precipitously. In addition, these 

surveys are generally expensive to conduct. These two facts have led to substantial interest in the use of nonprobability-based surveys as an alternative. What is unclear, however, is the degree of bias and resultant effects 

on data quality introduced through use of a non-random selection approach. Poststratification/raking or 

propensity score adjustment weighting approaches are increasingly used in the literature as an attempt to 

reduce the bias of non-probability samples. Nonetheless, the findings are not consistent. In this research, we 

empirically examine the impact of applying weight adjustments to non-probability-based survey data and 

compare results to those obtained from a probability-based survey based on a similar questionnaire. 

We compare the results obtained from a nationally representative probability-based mail survey: the 2011 

Medical Expenditure Panel Survey (MEPS) Experiences with Cancer survey, with a non-probability-based opt-in 

internet survey: the 2012 LIVESTRONG Survey, in which the same questionnaire items were administered. We 

examine the extent to which the LIVESTRONG data, subsequent to the application of several different weighting 

strategies, are consistent with estimates obtained from the MEPS Experiences with Cancer survey with respect 

to population parameter estimation and measurement of key associations (through statistical regression 

analysis). 
Concurrent Session B Abstracts

31

Results of both descriptive statistics and regression analyses for key outcomes obtained from both surveys will 

be compared. With appropriate caveats, the feasibility and implications of using non-probability-based surveys 

will be discussed. This research emerges concurrently with other ongoing research involving this topic. The 

findings from this study are expected to contribute to the survey methods field from a practical point of view, by 

indicating methods and conditions under which non-probability-based surveys may appropriately be utilized. 

Response Rates and Response Bias in Web Panel Surveys 

John M. Boyle (ICF International), Lew Berman (ICF International), Jamie Dayton (ICF International), Tala 

Fakhouri (ICF International), Ronaldo Iachan (ICF Interntional), Melanie Courtright (Research Now), and Kartik 

Pashupati (Research Now)

Non-probability samples, such as online panels, are increasingly accepted as “fit for purpose” for low incidence 

populations (e.g., pregnant women), difficult to reach populations (e.g., health care workers) and other special 

populations, particularly when time or cost make probability surveys infeasible. However, there is much less 

enthusiasm for the application of these methods in social science research for general populations. Aside from 

the issue of statistical generalizability, low response rates within the panel and demographic biases in the 

achieved samples are often cited (AAPOR 2010). 

Are low response rates and demographic biases endemic to population surveys using web panels, or do they 

reflect the methods of particular surveys? Many web panel surveys are conducted in such a way that response 

rate cannot be calculated. In other cases, response rate is not reported. Further, most web surveys are not 

conducted to optimize response rate since sample is nearly unlimited and speed is often critically important to 

the client. In addition, biases in web surveys are usually identified by comparing the characteristics of the 

achieved sample to the population, which does not address the source of the error as the frame or the survey 

procedures.

This paper examines the application of two survey protocols in a general population survey conducted in the 

same community using a national web panel. Invitations will be sent to two Census balanced samples of 5,000 

from the master panel, with the goal of achieving at least 500 completes in each sample. For the first protocol, 

invitations will be followed by a single reminder, an industry standard. For the second protocol, a robust 

reminder schedule including up to 4 reminders will be fielded over a three week period. Response rate is 

calculated as the proportion of invited respondents who complete the interview. Non-response bias is 

calculated by comparing the characteristics of responders and non-responders from their panel profile. Findings 

are compared across the two samples from the same community in the experiment.
Concurrent Session B Abstracts

32

Prediction-Enhanced Generalized Regression for Estimation from Purposive 

Supplements to Probability Samples

Avinash C. Singh (NORC at the University of Chicago) and Yongheng Lin (NORC at the University of Chicago)

With the growing demand for Fit-For-Purpose Surveys to save cost by not using rigorous data collection 

methods, old issues of whether suitable inferences can be made from purposive or nonprobability samples are 

again at the forefront. For purposive samples, design-based methods are clearly not suitable. There is, however, 

the possibility of using model-based methods but they assume the design to be non-informative; i.e., the model 

is assumed to hold for the sample so that an asymptotically optimal and consistent prediction of the unseen total 

from the seen can be made under the model. In addition to the above concern, a second concern with any 

model-based method is that it is subject to potential mis-specification of the model mean resulting in bias even if 

the design is non-informative. To overcome these concerns, an alternative approach termed Model-OverDesign (MOD)-Integration for a simplified problem is proposed under the joint design-model randomization 

where the availability of a core probability sample for which the purposive sample acts as a supplement is 

assumed. In MOD-Integration method, a regression model is first fit using the probability sample accounting for 

the possibility of an informative design. A design-based estimate such as GREG is constructed which uses a 

correction for model-error to the mean predictor (i.e., the synthetic estimator). Next, the above model-error 

correction is enhanced by using a supplementary prediction from the seen in the purposive sample assuming 

that the model mean is correctly specified and the model errors are uncorrelated with the conceptual selection 

probabilities of units from the population that could be selected in the purposive sample. Following the 

simulation experiment of Hansen, Madow and Tepping (1983), the proposed method is compared with 

alternative pure design and model-based methods. An application to real data from NORC Technology survey 

(ABS supplemented with email blast) is also considered.

CONCURRENT SESSION B-3

IMPROVEMENTS IN SURVEY ESTIMATION METHODOLOGY

Some Thoughts on Fitting Regression Models to Complex Survey Data

Phillip S. Kott (RTI International)

A question often asked when fitting a regression model to complex survey data is whether the sampling weights 

need to be incorporated into the estimation. A less often asked, but equally important, question is whether the 

model fits the data at all. To answer to second question, one need to define what “fitting the data” means. 

Survey statisticians conventionally unask that question by treating what would be the maximum-likelihood 

estimates were complete information from the full population available as the targets of estimation. They 

generally ignore the question of whether such estimates have any intrinsic meaning. 

A more sensible approach begins by assuming some form of the standard model under which the expected 

difference between the model’s dependent variable and its predicted value in the population is zero no matter 

what values the model’s covariates take. No further assumption about the distributions of this difference need be 

made. Since this difference can depend on the weights, the weights – or some function of the weights – may 

need to be incorporated when estimating the parameters of the standard model with complex survey data. 

The standard model, although quite general, may nonetheless fail. When it does, it can be replaced by an 

extended model in which the difference between the dependent variable and its predicted value need only be 

uncorrelated with the covariates. Assuming only an extended model produces the same estimates as the 

conventional quasi-maximum-likelihood approach when fitting a linear or binary logistical regression equation to 

complex survey data. The two method diverge, however, when fitting a cumulative logistic (proportional-odds) 

model. The repercussions of this divergence are explored. It turns out that most “design-based” software can fit 

and evaluate a generalized cumulative logistic model under the standard-model/extended-model framework. 

Tests for model and weight selection using such software are also discussed. 
Concurrent Session B Abstracts

33

Comparative Study of Confidence Intervals for Proportions in Complex

Sample Surveys

Carolina Franco (U.S. Census Bureau), Roderick J. A. Little (University of Michigan), Thomas A. Louis, (U.S. 

Census Bureau and John Hopkins University), and Eric V. Slud, (U.S. Census Bureau and University of 

Maryland). 

Perhaps the most widely used method of computing confidence intervals in complex surveys is to add and 

subtract the margin of error (MOE) from the point estimator, where the MOE is computed as an estimate of the 

sampling error times the appropriate quantile of the normal distribution. Such a method is used, for instance, by 

the American Community Survey (ACS), the largest US household survey. For inference on proportions with 

moderate sample sizes this method often results in marked under-coverage. For example, the Wald interval is 

of this form and applies to the special case of a Simple Random Sample (SRS) with replacement. The erratic 

coverage properties of the Wald interval have been well documented in the literature (Brown et al, 2001, 2002). 

In the complex survey case, the problem is further compounded by possible dependence among observations 

and unequal weighting. A common strategy for constructing confidence intervals for proportions using data from 

complex surveys is to adjust intervals meant for simple binomial proportions by the design effect. We adopt this 

strategy and apply it to seven intervals, including the Wald interval. We design and implement a simulation that 

controls and measures the impact of many features thought to affect the performance of confidence intervals for 

proportions, including clustering, stratification, misspecification of variances, and patterns of variation of stratum 

sampling fractions and stratum survey-attribute proportions. We compare the intervals focusing on properties 

such as coverage, length and symmetry of coverage. Since the MOE-based interval so often under-covers, 

there is great potential for improved reporting of uncertainty in surveys like the ACS, which generates millions of 

estimates of proportions each year. 

Total Variability Measures for Selected Quarterly Workforce Indicators and LEHD 

Origin Destination Employment Statistics in OnTheMap

Kevin L. McKinney (U.S. Census Bureau), Andrew S. Green (Cornell University and U.S. Census Bureau), Lars 

Vilhuber (Cornell University and U.S. Census Bureau), and John M. Abowd (Cornell University and U.S. Census 

Bureau)

We report results from the first comprehensive total quality evaluation of five major indicators in the U.S. Census 

Bureau’s Longitudinal Employer-Household Dynamics (LEHD) Program Quarterly Workforce Indicators (QWI): 

beginning-of-quarter employment, full-quarter employment, average monthly earnings of full-quarter employees, 

total quarterly flow employment, and total quarterly payroll. Beginning-of-quarter employment is also the main 

tabulation variable in the LEHD Origin-Destination Employment Statistics (LODES) workplace reports as 

displayed in the Census Bureau’s interactive data feature OnTheMap (OTM). The evaluation is conducted using 

the multiple threads (implicates) generated by the edit and imputation models used in the LEHD Infrastructure 

File System. These threads conform to the Rubin (1987) multiple imputation model. Each implicate is the output 

of formal probability models that address coverage, edit and imputation errors. Design-based sampling 

variability and finite population corrections are also included in the evaluation. We derive special formulas for the 

Rubin total variability and its components that are consistent with the disclosure avoidance system used for QWI 

and LODES/OTM workplace reports. These formulas allow us to publish the complete set of detailed total 

quality measures for QWI and LODES. The analysis reveals that the five publication variables under study are 

estimated very accurately for tabulations involving at least 10 jobs. Tabulations involving three to nine jobs have 

acceptable quality. Tabulations involving one or two jobs, which are generally suppressed in the QWI, have 

substantial total variability but their publication in LODES allows the formation of larger custom aggregations, 

which will in general have the accuracy estimated for tabulations in the QWI of similar magnitude. 
Concurrent Session B Abstracts

34

CONCURRENT SESSION B-4

ECONOMIC STATISTICS

Characteristics of Special Purpose Entities in Measures of U.S. Direct Investment 

Abroad

Dylan G. Rassier (Bureau of Economic Analysis)

Under international guidelines, official statistics on international trade and investment include transactions and 

positions within multinational enterprises (MNEs). A complicating factor in the interpretation and understanding 

of the official statistics is the role of transactions and positions for MNEs structured with one or more special 

purpose entities (SPEs). SPEs generally have little or no physical presence and little or no economic activity. 

Examples include finance and holding companies, royalty and licensing companies, and leasing companies. 

The international guidelines recommend transactions with SPEs and positions in SPEs be included in official 

statistics, but the guidelines also recommend supplemental information be provided to better understand the 

role of SPEs. 

While recent research explores the effects of SPEs on some U.S. statistics, very little is known about the 

underlying characteristics of SPEs included in transactions and positions on U.S. direct investment abroad 

(USDIA). Thus, this paper provides an empirical look at SPEs included in transactions and positions on USDIA. 

In particular, the paper presents financial and operating characteristics of non-resident SPEs that are sponsored 

by U.S. MNEs using data collected by the U.S. Bureau of Economic Analysis on U.S. parents and their foreign 

affiliates. The analysis includes mean comparison tests of non-resident SPEs with their non-resident, non-SPE 

counterparts and includes estimates of the dual cost functions for SPEs and non-SPEs based on reported sales 

as a proxy for costs and measured value-added as a proxy for output.

Preliminary results reveal a large number of non-resident SPEs sponsored by U.S. MNEs. In addition, 

preliminary results indicate significant differences between reported balance sheet components and reported 

income statement components of non-resident SPEs and their non-resident, non-SPE counterparts. Significant 

differences also result for measured value-added attributed to SPEs and non-SPEs. However, the effects of 

measured value-added on reported sales are mixed for SPEs and non-SPEs.

Building a Taxonomy and Lexicon of Terms and Concepts at BLS

Daniel Gillman (Bureau of Labor Statistics), Elizabeth Ashack (Bureau of Labor Statistics), Daniel Chow (Bureau 

of Labor Statistics), Ronald Johnson (Bureau of Labor Statistics), Karen Kosanovich (Bureau of Labor 

Statistics), Nicole Nestoriak (Bureau of Labor Statistics), Ann Norris (Bureau of Labor Statistics), Garrett Schmitt 

(Bureau of Labor Statistics), and Clayton Waring (Bureau of Labor Statistics)

The taxonomy and lexicon project at the U.S. Bureau of Labor Statistics was started in summer 2013 with the 

aim to provide consistent access to BLS data and documents. The goals are to ensure each search criterion 

provide data and documents that are related and to provide non-technical (plain) English words to less 

experienced users to help guide them to the data and documents that meet their needs. The taxonomy portion 

of the work aims to improve searching for data, and the lexicon portion is intended to improve tagging, 

cataloging, and searching for documents. The work has advanced significantly since it was initiated, through 

discrete phases. The third phase is ongoing.

Phase 1 was focused on requirements gathering, identification of useful plain English words, and collecting 

technical terms and definitions. In Phase 2, the work to identify plain English words was completed, and all 

terms and words were organized into a multi-hierarchy taxonomy. Phase 3 includes work to improve the quality 

of the taxonomy and produce the lexicon. Use of focus groups and other techniques for cognitive testing of the 

taxonomy is planned.
Concurrent Session B Abstracts

35

The taxonomy is divided into facets – separate, independent hierarchies – for both measures and characteristics 

(the classifications used as dimensions in cross-tabulated data). Every underlying series can be reached by 

paths through the hierarchies of each facet. Now, the taxonomy is being used to improve the user interface for 

the new DataFinder series dissemination tool under development. Other uses are planned.

As in any development project, the taxonomy produced so far is not error-free. Many areas for improvement 

were identified at the end of Phase 2. We anticipate the work on quality in Phase 3 will address these concerns.

The paper will contain descriptions of all the work so far and plans for the future.

New Technology Indicator for Technological Progress

Wendy Li (Bureau of Economic Analysis)

Technological progress is the main driver of long-run economic growth. Economists have relied on the growth 

accounting framework to measure the multifactor productivity (MFP) growth, which serves as an indicator of a 

country’s pace of technology progress. However, Griliches (1996) argues that the measurement of MFP growth 

is just a measurement of ignorance and economists have criticized that the MFP measurement is just a point 

estimate without a robustness check (Manski, 2014). Furthermore, as OECD countries have started capitalizing 

research and development (R&D) in national accounts, the change of the MFP growth after capitalizing R&D 

may cause more puzzles than reflect the true nature of technological progress across countries and across 

industries. For example, the change of U.S. new MFP growth between the period of 1998 to 2007 and the 

period of 2009 to 2012 are -0.20% for information technology (IT) producing industries, 0.05% for IT using 

industries, 0.51% for non-IT industries (Rosenthal et al., 2014). The latest result from the U.S. indicates that 

during the period of 1998 to 2012, the non-IT industries have experienced a faster pace of technological growth 

than the IT industries, which contradicts the general consensus. As pointed out by OECD productivity 

measurement manual, there is no straightforward link between MFP and technical changes, and R&D 

investments only explain a small portion of overall annual movements in MFP.

This paper proposes a new technology indicator, the R&D depreciation rate by country and by industry. I 

conduct the study on four high-tech intensive industries over five different countries. The estimates of the R&D 

depreciate rates using these data provide several important results. First, in each industry, the country ranking 

in technological competitiveness, measured by industry-level R&D depreciation rate, is consistent to the country 

ranking on Forbes’ global 2000 list. Second, when I compare the results between the U.S. and Japan for four 

industries, the country with a lower industry-level R&D depreciation rate has a higher industry-level MFP. Third, 

the information derived from the time-varying R&D depreciation rates for the motors and the computer, 

electronic, and optical products industries in Japan and the U.S. is consistent with the historic studies in these 

two industries over the decade of the 2000s. The importance of the R&D depreciation rate is beyond the 

calculation of capital service flow and the construction of capital stock. The R&D depreciation rate can provide 

valuable information about the pace of technological progress and relative technological competitiveness across 

countries.
Concurrent Session C Abstracts

36

CONCURRENT SESSION C-1

LINKING DATA TO GENERATE NEW STATISTICS: JOBS, PATENTS, 

AND FDI

Two Perspectives on Commuting and Workplace: A Microdata Comparison of 

Home to Work Flows Across Linked Survey and Administrative Files

Andrew Green (U.S. Census Bureau, Cornell University), Mark J. Kutzbach (U.S. Census Bureau), and Lars 

Vilhuber (U.S. Census Bureau, Cornell University)

Commuting flows and workplace employment data have a wide constituency of users including urban and 

regional planners, social science and transportation researchers, and businesses. The U.S. Census Bureau 

releases two, national data products that give the magnitude and characteristics of home to work flows. The 

American Community Survey (ACS) tabulates employment responses and commuting statistics. The 

Longitudinal Employer-Household Dynamics (LEHD) program tabulates job records in the LEHD OriginDestination Employment Statistics (LODES). Differences in the domains, job definitions, and confidentiality 

protection measures across the datasets lead to divergence in comparable statistics, such as county-to-county 

aggregate commute flows. To understand these differences, this study compares ACS and LEHD source files, 

using probabilistic matching to join survey responses with job records. In our assessment, we compare 

employment status, home and work locations, commute distance, and other job characteristics at a person level, 

and identify person or job characteristics that are associated with or explain these differences. We discuss 

implications of match quality and the matching methodology for our comparison results. 

An ‘Algorithmic Links with Probabilities’ Crosswalk for USPC Codes

Nikolas J. Zolas (U.S. Census Bureau), Nathan Goldschlag (U.S. Census Bureau), and Travis J. Lybbert

(University of California, Davis) 

Patents are a useful proxy for innovation and technological change and diffusion. However, fully exploiting 

patent data for such economic analyses requires patents be tied to measures of economic activity, which has 

proven to be difficult. Recently, Lybbert and Zolas (2014) have constructed an International Patent Classification 

(IPC) to industry data crosswalk that incorporates several ‘Big Data’ elements and was dubbed an ‘Algorithmic 

Links with Probabilities’ approach. In this paper, we utilize a similar approach and apply it to a new patent 

classification index, the U.S. Patent Classification (USPC) system. The result is a USPC-Industry concordance 

that links U.S. patents to the North American Industrial Classification System (NAICS), International Standard 

Industrial Classification (ISIC), Harmonized System (HS) and Standard International Trade Classification (SITC). 

As a result of this linkage, it is possible to cross-analyze the USPC-IPC link for patents and allows researchers 

using U.S. patents to link them with a host of economic data that varies at the industry level. We demonstrate 

potential uses of the crosswalk and how it can be applied to measure patenting intensity across different sectors 

of the economy and patenting trends.
Concurrent Session C Abstracts

37

Linking BEA Surveys of Foreign Direct Investment in the United States with BLS 

Register and Survey Data: Twenty First Century Methods and Preliminary 

Findings

Elizabeth Weber Handwerker (Bureau of Labor Statistics), Mina Kim (Bureau of Labor Statistics), Lowell Mason 

(Bureau of Labor Statistics), and David Talan (Bureau of Labor Statistics)

One data gap that users mention often is the need to measure the employment effects from inward foreign 

direct investment. Since building new surveys can be expensive and can take several years to implement, we 

propose to create these measures by linking existing survey data on foreign direct investment with business 

register data. 

The U.S. Bureau of Labor Statistics (BLS) uses its Business Register to publish measures of employment 

dynamics and entrepreneurship which are released quarterly by its Business Employment Dynamics (BED) 

program. The U.S. Bureau of Economic Analysis (BEA) collects information on foreign direct investment through 

annual surveys. BEA conducts these surveys to estimate U.S. assets overseas and foreign ownership of U.S. 

assets. 

This paper lays out the conceptual framework for developing indicators of employment and wages related to FDI 

by linking survey micro-data from BEA with BLS business register micro-data. This paper describes the 

methodology for generating these measures, the linking process and results, as well as some of the challenges 

and caveats associated with this approach. The paper then presents preliminary results and opportunities for 

future work.

CONCURRENT SESSION C-2

EVALUATION OF SURVEYS THROUGH PRETESTING AND OTHER 

COGNITIVE METHODS

Retrieval of Autobiographical Information

Erica Yu (Bureau of Labor Statistics) and Scott Fricker (Bureau of Labor Statistics) 

Questions about past behavior often require respondents to retrieve information from autobiographical memory 

– a combination of personal experiences (episodic memory) and general knowledge (semantic). An example 

would be recalling whether you recently purchased any sweaters and, if so, the price that you paid. The retrieval 

of autobiographical information is fundamental to the design of survey questions and yet remains poorly 

understood. The design of recall questions and how they facilitate or inhibit recall may affect data quality and 

respondent burden. We explore this issue with the Consumer Expenditure Survey. 

Participants in an online free recall study reported five recent expenditures in a sequence. Participants were 

then asked to describe why each reported item in the sequence came to mind. Example reasons include: the 

expenditures being by or for the same person, purchased from the same store, or purchased for a shared 

purpose. The expenditures and reasons were then coded by two independent coders. These codes were 

analyzed to understand whether systematic retrieval strategies exist across individuals, what factors influence 

the use of different strategies, and whether different retrieval strategies are associated with data quality and 

respondent burden.

The main takeaway of this presentation is about how researchers can utilize these findings to optimize retrieval 

of autobiographical information in surveys. The audience will hear about why consideration of respondent 

response formulation processes such as retrieval of information are important and which retrieval strategies 

individuals may be using in surveys. Audience members may consider incorporating retrieval strategies into 

instrument design decisions and survey instructions.
Concurrent Session C Abstracts

38

The Science of Usability Testing

Jean E. Fox (Bureau of Labor Statistics) 

Usability testing has evolved from the stringent methods of experimental psychology, to less controlled, more 

qualitative tests, to the wide variety of methods used today. During that time, researchers have studied many 

aspects of usability testing with the goal of better understanding how they work. In this presentation, I will 

discuss the methods researchers have used to study usability testing. I will then present some of this research 

and what it means for conducting usability tests.

Specifically, I will explore research that addresses the following usability test design choices:

• With federal usability tests, we often run fewer than 10 participants. How does the sample size impact 

the results? How many participants should we have?

• Usability tests are often run by just one person. How can this practice influence the findings? What 

impact does the “Evaluator Effect” have on the results and recommendations?

• We often encourage participants to think aloud during usability tests. How does this practice affect the 

test results?

• Many practitioners are now using remote, unmoderated tests, where participants complete the tests on 

their own over the internet. This approach is attractive because it can provide input from a diverse 

sample quickly. But how does it compare to standard in-person usability tests?

Understanding this research is critical to ensure that we run sound usability tests and are unbiased in our 

interpretation of the results. The material covered in this presentation will help attendees run better usability 

tests and be more aware of factors that can influence their findings.

Using Graphics to Convey Complex Survey Concepts

Heidi M. St. Onge (U.S. Census Bureau) and Rebecca L. Morrison (National Science Foundation)

The State Government Research and Development Survey (SGRD) is conducted biennially, and collects 

information from state government agencies concerning their expenditures for research and development (R&D) 

via a self-administered web instrument. The survey collects data on both R&D that is performed by the state 

agency itself, as well as the R&D it funds outside the agency. The current survey universe is comprised of 

approximately 400 state-level agencies in the United States; some states have only one agency involved in 

R&D, while other states have many.

The survey has been in existence since 2006. Due to the complexity of the survey content, the current web 

instrument relies on an abundance of words in order to convey critical concepts necessary for accurate 

reporting. However, recent cognitive interviews with respondents indicated the number of words may be 

detrimental to accurate responses. Respondents often skipped over key words and phrases that are intended to 

help them determine a) which expenditures should be reported on the survey, and which should not, and b) 

which expenditure types should be reported in each survey question. Cognitive interview respondents also 

indicated that discussions with survey staff were useful in sorting out these details; unfortunately, interviews with 

a small number of agencies that are widely geographically dispersed are cost-prohibitive.

Instead of employing interviewers to assist respondents with survey completion, researchers developed simple 

diagrams that would accompany selected questions found in the web instrument. The presentation will show 

examples of the diagrams and text tested in cognitive interviews, provide feedback from respondents on their 

utility, and describe how the diagrams will be displayed in the upcoming deployment of the survey. It will also 

describe some of the challenges in implementing unique features for one survey within a system designed for 

multiple surveys.
Concurrent Session C Abstracts

39

Using eye-tracking to understand how fourth grade students answer matrix items

Aaron Maitland (Westat), Hanyu Sun (Westat), Andrew Caporaso (Westat), Roger Tourangeau (Westat), Jonas 

Bertling (Education Testing Service), and Debby Almonte (Education Testing Service) 

Eye-tracking has been used to better understand the survey response process. For instance, eye-tracking has 

been used to identify questions that are difficult to comprehend, how to present long lists of response options, 

and to measure the length of fixation on definitions in Web surveys. Matrix questions have been commonly used 

in Web surveys as well as in other types of surveys. The literature shows respondents took less time to answer 

questions when they were presented in a matrix than when they were presented individually across separate 

pages or screens. The use of matrix questions, however, may also be associated with several undesirable 

outcomes, including higher breakoff rates, higher missing data rates, and straightlining. Relatively little is known 

about how children answer matrix questions. This paper demonstrates how eye-tracking was used to determine 

the feasibility of using matrix questions to measure the background characteristics of students in the National 

Assessment of Educational Progress (NAEP) questionnaire. Fourth grade students were answered both matrix 

and discrete versions of questions on tablet computers while wearing real-world eye tracking glasses. This study 

addresses four research questions related to the use of eye-tracking to test survey questions. First, we examine 

whether matrix items require more effort to answer than discrete items for fourth grade students. Second, we 

investigate how the processing of sub items change within a matrix. Third, we examine how the processing of 

questions change over time. In order to address these research questions, we examine difference in the mean 

number of fixations per word and the mean duration per word for matrix and discrete questions. Overall, the 

study finds support for the use of matrix questions with fourth grade students in the NAEP. Implications for the 

use of eye-tracking equipment to evaluate survey questions are also discussed.

Reliability Testing of the Walking Environment Module

Stephanie L. Fowler (National Cancer Institute), Gordon Willis (National Cancer Institute), Rebecca Ferrer 

(National Cancer Institute), and David Berrigan (National Cancer Institute) 

Many data elements in the Federal Statistical System are obtained via standardized survey questions. It is well 

established that such questions and groups of questions addressing more complex constructs may be improved 

through the use of reliability and validity studies, cognitive testing, and analyses of the psychometric properties 

of alternative versions. Additionally, a growing body of evidence has identified contextual factors that influence 

responses such as question order and mode of administration. However, testing of Federal survey questions, 

especially those in health surveys, has sometimes been limited because of cost and difficulties in obtaining 

samples for studies. 

In this study we will describe our experiences using Amazon’s Mechanical Turk (mTurk) to measure reliability, 

obtain responses to a module of questions for psychometric analyses, examine question order effects, and 

explore the utility of a web-based platform for implementing structured cognitive probes concerning question 

meaning. The survey questions we used concern perceptions of the environment for walking and the frequency 

and duration of walking for transportation and leisure. These questions have already undergone cognitive 

testing and have been fielded in the 2015 Cancer Control Supplement of the National Health Interview Survey 

(NHIS).The cognitive probes we are implementing in the online platform were based on those administered in 

person during pretesting of the survey module. We intend to contrast the response to in-person cognitive 

interview questions with responses received in free text fields in the mTurk platform.

To date, we have received OMB approval for our study design and anticipate fielding the questions in mTurk in 

June of 2015. Similar studies in our Program have obtained 1000+ samples in less than one day. Thus we 

anticipate sharing results as well as information about the study design at FSCSM 2015. 
Concurrent Session C Abstracts

40

CONCURRENT SESSION C-3

APPLICATION OF BAYESIAN METHODS

Evaluating the alignment of grant allocation and community need using Bayesian 

spatial probit models: A case study of AmeriCorps

Robin Ghertner (Corporation for National and Community Service) and Katie Seely-Gant (Corporation for 

National and Community Service) 

Many federal grants are intended to be allocated based on community need. Evaluating this premise is 

challenging because of complex spatial relationships among communities and grantees. Both communities in 

need and grantees tend to cluster geographically, which must be properly accounted for. Additionally, spatial 

algorithms can be computationally burdensome. We estimated a spatial autoregressive probit model using 

Bayesian MCMC methods via open source packages in R, and sped up data management and estimation using 

sparse matrices and parallel processing. Data were drawn from ACS, County Health Status Indicators, Common 

Core of Data, and other federal sources, to create indicators of need as well as control variables to account for 

competing explanations. Results show significant spatial dependency, and controlling for this dependency 

attenuated the relationships between need and allocation. In general, the location of AmeriCorps grantees have 

a moderate alignment with economic and educational need, but are poorly aligned with health needs. 

Implications for AmeriCorps and other federal grant programs are discussed, both with respect to the study 

findings and the analytical methods.

Bayesian Estimation Under Informative Sampling

Terrance D. Savitsky (Bureau of Labor Statistics)

An informative sampling design assigns probabilities of inclusion that are correlatedwith the response of interest 

and often induces a dependence among sampled observations. It is well-known that model-based inference 

performed on data acquired under an informative sampling design will be biased for estimation of the joint 

distribution of model parameters supposed to generate the population from which the sample was drawn. 

Known marginal inclusion probabilities assigned through the sampling design may be used to weight the

likelihood contribution of each observed unit to form a “pseudo” posterior distribution with the practical intent to 

“undo” the design for inference about the population. This paper extends a theoretical result on consistency of 

the posterior distribution, defined on an analyst-specified model space, at the true generating distribution to the 

sampling-weighted pseudo posterior distribution used to account for an informative sampling design. We 

construct conditions on known marginal and pairwise inclusion probabilities that define a class of sampling

designs where consistency of the pseudo posterior is achieved, inprobability. We perform a simulation study and 

provide an application for data acquired under the allowable class of sampling designs that demonstrate the 

result.

Statistical Considerations in Validating Clinical Quality Measures

Derekh Cornwell (Mathematica Policy Research), Fei Xing (Mathematica Policy Research), Juan-Diego 

Astudillo (Mathematica Policy Research), Michael Sinclair (Mathematica Policy Research), Nancy Sonnenfeld 

(Centers for Medicare and Medicaid Services), and Mihir Patel (Centers for Medicare and Medicaid Services)

The Centers for Medicare & Medicaid Services (CMS) conducts voluntary Hospital Inpatient Quality Reporting 

(IQR) to improve information about hospital quality and performance. As a reporting incentive, annual payment 

updates (APUs) in Medicare depend on these hospital performance measures. To maintain accurate reporting 

of hospital performance measures, CMS conducts validation of hospital reported data on a sample of cases. 

CMS compares the medical records to the hospital reports on the sample cases and examines the 

discrepancies.
Concurrent Session C Abstracts

41

We will explore properties of standard parametric statistical tests relying on the Normal or Student’s t 

distribution. We will then compare results for confidence intervals using (1) nonparametric tests in the presence 

of smaller samples; (2) Empirical Bayes estimation, which may be particularly attractive when a program has a 

rich history of data and population covariates available; and (3) model-based estimation methods. We discuss 

the tradeoffs of these methods and offer guidance for different methods. These results will inform future 

validation designs.

A Bayesian Hierarchical Model for Combining Several Crop Yield Indications

Nathan Cruze (National Agricultural Statistics Service) 

The USDA National Agricultural Statistics Service conducts multiple surveys over the course of a growing 

season. Each of these surveys reflects current growing conditions and provides a prediction of end-of-season 

crop yield. In particular NASS conducts two interview-based surveys and one field measurement survey from 

which indications of crop yield may be obtained. It is also known that a number of weather conditions during the 

growing season may contribute to changes in crop yield. This talk describes a Bayesian hierarchical model 

which improves end-of-season crop yield predictions by combining these several disparate sources of 

information. The model incorporates benchmarking of state-level forecasts with regional forecasts of crop yield 

and gives rise to rigorous measures of uncertainty. It also permits a useful decomposition with respect to the 

emphasis placed on each information source. Several aspects of covariates selection and model performance 

are discussed.

CONCURRENT SESSION C-4

ADAPTIVE DESIGN I

2015 National Survey of College Graduates: Enhancing the Use of Adaptive Design 

John Finamore (National Science Foundation), Stephanie Coffey (U.S. Census Bureau), and Benjamin Reist

(U.S. Census Bureau)

 

A major goal of adaptive design is to increase the efficiency of survey operations while maintaining or improving 

survey data quality. The 2013 NSCG adaptive design study showed that it is possible to monitor data as it is 

being collected and use the monitoring results to inform data collection interventions with potential to improve 

data quality and reduce survey costs.

Now that our ability incorporate adaptive design techniques in a large-scale national survey has been 

established, our current goal is to enhance the use of adaptive design. The 2015 NSCG will include an 

expanded adaptive design methodology study that includes a larger sample and, taking advantage of the 

longitudinal nature of the NSCG data, explores the use of adaptive design techniques on both new and returning 

sample members. The 2015 NSCG study will explore monitoring metrics and data collection interventions in an 

effort to address the survey issues most important for the NSCG, including nonresponse bias, data timeliness, 

and survey costs.

For adaptive design methodology to advance, studies that result in definitive statements about the impact of 

various data monitoring techniques and data collection interventions must be conducted and documented. The 

2015 NSCG adaptive design study will use a sample size that provides the statistical power necessary to make 

more definitive statements about statistical differences between the study group and the control group on 

various measures, including response rates, R-indicators, cost, and effect on key estimates. The hope is that 

the 2015 NSCG study will add significantly to the existing survey methodology literature on adaptive design. 

We will discuss developing the 2015 NSCG adaptive design study and the data monitoring metrics and data 

collection interventions within the study. We will present initial study results and potential implications for using 

adaptive design techniques in future survey cycles. 
Concurrent Session C Abstracts

42

Investigating Nonresponse Subsampling in an Establishment Survey Through 

Embedded Experiments 

Katherine Jenny Thompson (U.S. Census Bureau) and Stephen J. Kaputa (U.S. Census Bureau)

Adaptive design strategies for data collection are intended to increase the quality of response data even under a 

reduced survey budget. In this framework, the U.S. Census Bureau is investigating nonresponse subsampling 

strategies, including a systematic sample of nonrespondents sorted by a measure of size, for use in the 2017 

Economic Census. Our previous research has explored this design change using an optimally allocated 

subsample of nonrespondents with the objective of selecting systematic samples in domains that have lower 

initial response. By concentrating nonresponse follow-up efforts in domains that have low response we attempt 

to balance response rates to reduce the effect of nonresponse bias. However, we found subsampling 

nonrespondents this way without changing the data collection procedure may have minimal tangible benefits 

besides cost reduction. Improving the data collection procedure to target “hard to reach” establishments that 

were selected for nonresponse subsampling is likely to improve estimates. In this presentation, we present the 

results of a field experiment, embedded in the 2014 Annual Survey of Manufactures (ASM), designed to test 

contact strategies for selected small units Then, we present the design and discuss the proposed analysis 

strategy for a subsequent embedded experiment in the 2015 ASM, which pairs our proposed nonrespondent 

subsampling design with the most effective follow-up procedures determined from the 2014 test.

Adept Adaptations: Developing Alternative Strategies in Adaptive Survey Designs 

Jaki McCarthy (National Agricultural Statistical Service) and Tyler Wilson (National Agricultural Statistical 

Service)

Many surveys are turning to adaptive survey designs in their data collection to minimize costs, reduce nonresponse bias or improve data quality. By identifying key design features affecting survey costs and errors, 

program managers can either proactively adapt or dynamically respond by altering those features in the survey 

data collection. Thus multiple “phases” of a survey with differing conditions are combined in 

estimation. Common examples of alternative procedures include increasing or capping phone call attempts or 

prioritizing cases for interviewers so targeted cases are attempted before other cases. However, for some 

surveys it may not be easy to identify alternative procedures to be employed in data collection. NASS has been 

testing the use of adaptive design in the Agricultural Resource Management Survey (ARMS). This long and 

challenging survey uses a mail questionnaire with in-person follow up for all initial nonrespondents. ARMS is an 

establishment survey with rich frame data. We used this information to develop response propensity models 

that predict, before data collection begins, the sampled units’ likelihood of response. We could use this 

information to prioritize cases for follow-up. But because this survey already features an in-person follow-up 

interview over a relatively long field period, it was not obvious what alternative data collection procedures we 

could use in an adaptive design. This talk will discuss how we identified a subset of units to target and 

developed the alternative procedures for those units. We developed these procedures based on feedback from 

field enumerators, input from survey program managers and principles of social psychology. How these new 

procedures were received by NASS staff as well as whether these procedures were effective for increasing 

response in our target records will be discussed. This case study may provide an effective method of developing 

alternative procedures in adaptive designs for other survey organizations. 
Concurrent Session C Abstracts

43

Adaptive Design Experimentation in the High School Longitudinal Study of 2009 

Second Follow-up Field Test: Investigating Incentive Treatments

Elise M. Christopher (National Center for Education Statistics)

The National Center for Education Statistics (NCES) has been working in recent years to improve declining 

response rates in its sample surveys. Across multiple longitudinal and cross-sectional surveys, several methods 

of adaptive design have been tested. Using models to determine likelihood to introduce bias should sample 

members not respond, NCES has effectively discerned which sample members to target for special intervention, 

in order to boost response rates. However, it is clear that sample members’ propensity to respond is not always 

strongly correlated with their likelihood to introduce bias if they do not respond, necessitating more research on 

tailored ways to encourage response. Incentives and other treatments thus become an important area of 

investigation. This presentation describes the adaptive design approach that NCES used in the High School 

Longitudinal Study of 2009 (HSLS:09) Second Follow-up Field Test, conducted in spring and summer, 2015. 

This field test examined the effectiveness of several types of incentive treatments to encourage sample member 

response. The comparative effectiveness of four treatment methods will be discussed: 1) prepaid incentives, 2) 

response-contingent incentives, 3) incentive boosts, and 4) an abbreviated interview. The paper will describe 

the results from the adaptive design experiment and the plans for implementation of the findings in full-scale 

data collection in 2016. 
Concurrent Session D Abstracts

44

CONCURRENT SESSION D-1

IMPUTATION, MULTIPLE IMPUTATION AND ADMINISTRATIVE RECORDS

Treatment of Missing Data in Hierarchically-Structured Administrative Records: 

A Case Study in the Bakken Region Using FBI’s National Incident Based Reporting 

System

Dan Liao (RTI International), Marcus Berzofsky (RTI International), David Heller (RTI International), Kelle 

Barrick (RTI International), Matthew DeMichele (RTI International) Kim Martin (Bureau of Justice Statistics), and 

Alexia Cooper (Bureau of Justice Statistics)

In response to the growing reliance upon administrative records to generate national estimates of key indicators 

of interest, federal statistical agencies have been expanding and enhancing activities to assess and improve the 

quality of data collected through administrative records systems. Given that administrative records are often 

collected across different agencies or local reporting units, encountering missing data at both the individual and 

aggregated levels is inevitable and must be addressed when developing estimates. 

The National Incident-Based Reporting System (NIBRS) is a system designed to collect data from administrative 

records to be used for research and statistical purposes. It was developed by the FBI as part of its Uniform 

Crime Reporting (UCR) Program to present comprehensive information about crime incidents. In this paper, we 

present an imputation method developed to treat missing data in NIBRS by leveraging other relevant external 

data sources. Given the hierarchical structure of NIBRS, this particular method addresses missing data 

occurring at multiple levels, including: a) incident level, due to item missing within each incident; b) agency level 

by month, due to some agencies reporting only for a partial year; c) agency level by year, due to some agencies 

not submitting data to NIBRS for an entire year. The proposed method will be applied to a study in the Bakken 

region of the US that utilizes the NIBRS data to examine how crime and law enforcement changed in the region 

as oil production increased from 2006 to 2012. A variance estimation method will also be presented to evaluate 

the uncertainties in our estimates introduced by this imputation technique. In a broad sense, this research also 

can be viewed as an example that demonstrates how to handle missing data in hierarchically-structured 

administrative records. 

Introducing Parametric Models and Administrative Records Into 2014 SIPP 

Imputations

Gary Benedetto (U.S. Census Bureau), Joanna Motro (U.S. Census Bureau), and Martha Stinson (U.S. Census 

Bureau) 

As part of the redesign of the Survey of Income and Program Participation (SIPP), we have developed a new 

imputation process for handling respondents who are missing entire sections of data. This new process creates 

indicator variables called topic flags that determine whether each section of questions was relevant for a 

respondent (e.g., receipt of Food Stamps). We model the joint distribution of these variables and covariates 

from the survey and from independent data sources using a parametric method called Sequential Regression 

Multiple Imputation (SRMI). The resulting, approximate distribution is used to impute missing values for the 

topic flags.

Modeling topic flags in this manner is an alternative imputation method to hot-deck imputation or whole-record 

donation for cases where respondents did not complete the majority of the survey. As opposed to hot-deck 

imputation that can only control for a limited number of characteristics, the SRMI approach is able to control for 

many more variables, including household, parent, and spouse characteristics.
Concurrent Session D Abstracts

45

Moreover, our process incorporates administrative records for the first time into SIPP production. These data 

offer a valuable set of covariates whose availability is unrelated to survey non-response and, as a result, helps 

to mitigate problems caused when survey data are not “missing at random”. This paper describes our modeling 

process, its advantages over more traditional imputation methods like hot-deck imputation, and demonstrates 

the usefulness of linking administrative data into the models. Lastly, we show preliminary results for the SIPP 

2014 panel topic flags.

Comparison of Modern Imputation Methodologies on Complex Data from 

Agricultural Operations

Darcy Miller (National Agricultural Statistics Service) and Andrew Dau (National Agricultural Statistics Sevice)

The National Agricultural Statistics Service (NASS), in conjunction with the Economic Research Service (ERS), 

conducts the Agricultural Resource Management Survey (ARMS) to study the well-being of farm households. 

ARMS has a complex survey design, collects personal household information, and has a burden exceeding 

ninety minutes. Due to item nonresponse, some of the ARMS data are missing. Prior to 2015, a complete data 

set for use by NASS was formed by a mixture of machine imputation (conditional mean) and manual imputation. 

Recently, Sequential Iterative Regression (ISR), a multivariate imputation methodology, was implemented 

through a cooperative agreement between the National Institute of Statistical Sciences and NASS (see Robbins 

et al. 2013), and it better preserves relationships and the distribution of the data. ISR has been noted to be a 

blend of the popular data augmentation described and fully conditional specification (FCS) methods, allowing for 

some flexibility of the conditional models while providing a valid joint distribution. IVEware, a product of the 

University of Michigan, utilizes Sequential Regression Multiple Imputation (SRMI; see Ragunathan et al. 2001), 

which utilizes FCS methodology for mixed data types and allows for more flexibility in terms of variable type and 

the incorporation of edit logic. In this study, the empirical performance of IVEware and ISR for use in ARMS are 

compared through simulation.

CONCURRENT SESSION D-2

DEMONSTRATIONS ON DATA VISUALIZATION AND METADATA SYSTEMS

Census Bureau’s Microdata Analysis System (MAS) Pilot for the American 

Community Survey

Tiffany Julian (U.S. Census Bureau), Amy Lauger (U.S. Census Bureau), and William Hazard (U.S. Census 

Bureau) 

The Census Bureau’s Microdata Analysis System (MAS) for the American Community Survey allows public 

access to confidential microdata while applying disclosure avoidance rules to the results. The system will be free 

to data users, and will be useful to those unable to do analysis using other more expensive and involved Census 

Bureau data options, such as requesting a special tabulation or using a Research Data Center.

A user will select variables for the analysis, then will select a universe (domain) on which the analysis is 

performed. The universe may be defined on a custom geographic region and may be restricted based on values 

of non-geographic variables. For a survey such as the American Community Survey, we anticipate allowing 

geographic regions defined as low as individual tracts, either singly or in combination with each other to form a 

larger area.

Other than geography, the core features that the system provides are twofold. First, it allows customized queries 

on the full microdata file. Second, and possibly more innovatively, the MAS performs real-time disclosure 

avoidance procedures on each request when it is submitted. When a table is requested, the system does a 

number of checks on-the-fly to make sure the data are safe to release. In the case of a table of counts, most of 

these checks are for excessive sparsity in the cell values. If the table passes disclosure review, then it is 

released to the user (perhaps with some rounding of cell sizes), whereas if the table does not pass, then the 

user receives an error message indicating that the table was not output for disclosure reasons. We are still 

testing and modifying the system, but at this point our hope is that the user will receive either a table or a denial 

within a minute.
Concurrent Session D Abstracts

46

Crowdsourcing Codebook Enhancements: A DDI-based approach

Lars Vilhuber (Cornell University), Benjamin Perry (Cornell University), Venkata Kambhampaty (Cornell 

University), Kyle Brumsted (McGill University), and William C. Block (Cornell University)

Recent years have shown the power of user-sourced information evidenced by the success of Wikipedia and its 

many emulators. Agencies on the other hand publish metadata on their data products, but have little opportunity 

to get structured feedback on that metadata (codebooks) from their users. Creating and augmenting metadata is 

a labor-intensive endeavor. Harnessing collective knowledge from actual data users can supplement officially 

generated metadata. As part of our Comprehensive Extensible Data Documentation and Access Repository 

(CED2AR) infrastructure, we demonstrate a prototype of crowdsourced metadata (codebooks), using DDI-C and 

supplemental XML. The system allows for any number of network connected instances (web or desktop 

deployments) of the CED2AR DDI editor to concurrently create and modify metadata. The backend transparently 

handles changes, and frontend has the ability to separate official edits (by designated curators of the data and 

the metadata) from crowd-sourced content. Features of the software allow official curators to merge in select 

enhancements into revisions of official documentation. CED2AR ingests and publishes widely-used DDI 

metadata standards, and can thus easily augment existing metadata publishing systems. Using standardscompliant ingest and publication methods, our proposed crowd-sourced metadata system can greatly enhance 

user acceptance of documentation, user-engagement in the documentation process, and reduce agency costs 

while improving the quality of data documentation. This development is part of the NSF-Census Research 

Network node at Cornell University.

Visualization of BLS Geospatial Data Using R/Shiny

Elizabeth Cross (U.S. Bureau of Labor Statistics), Kenneth Cho (Bureau of Labor Statistics), Randall Powers 

(U.S. Bureau of Labor Statistics), Jeffrey M. Gonzalez (U.S. Bureau of Labor Statistics), and Wendy L. Martinez 

(U.S. Bureau of Labor Statistics) 

Several U.S. Bureau of Labor Statistics (BLS) surveys have data with geospatial characteristics. For example, 

the Occupational Employment Statistics (OES) program publishes employment numbers for occupations at 

various geographic levels. Additionally, there is an on-line tool, hosted at www.bls.gov, which shows maps of 

different statistics (e.g., mean wages, location quotient) for these occupations for states and Metropolitan 

Statistical Areas (MSAs). Currently, these maps are created annually and saved as static files. When interacting 

with the current on-line mapping tool, users select the desired occupation, statistics, and geographic level and 

then the relevant map is displayed. The mapping tool appears to be dynamic (i.e., the user observes changes, 

in real-time, in the visualization based on options he/she chooses), but the user is, in fact, limited to only a predefined set of maps whose features and layout have been determined by BLS staff. 

In order to enhance functionality of the existing product by providing users with dynamic visualizations, 

researchers at BLS created a dynamic on-line mapping tool for OES data using R, an open-source computing 

environment for data analysis; its various spatial packages (e.g., choroplethr, leaflet); and Shiny, a web 

application framework for R. This presentation will demonstrate this dynamic mapping tool and comment on the 

lessons learned from converting a static mapping tool to a dynamic tool.
Concurrent Session D Abstracts

47

Interactive Web Visualization of Birth and Pregnancy Trends by Maternal Age, 

Race/Ethnicity, and Marital Status: United States, 1909-2013

Li Lu (CDC NCHS), Yinong Chong (CDC NCHS), Sally C. Curtin (CDC NCHS), and Betzaida Tejada Vera 

(CDC, NCHS)

Vital statistics numbers and rates are widely used by the research community as well as policy makers and the 

general public. Aggregate data, particularly rates, have traditionally been published in reports with static tables 

and figures, most often in PDF format. Improving the access of these data with an interactive graphical 

interface and providing downloadable Excel tables could enhance the usefulness of these data. Interactive 

graphs can make the data “come alive”, showcasing relationships and trends that may be difficult to detect 

otherwise. 

We developed a new data visualization approach for vital statistics data. Working with 

demographers/statisticians in the Division of Vital Statistics/NCHS, we designed dashboards and storyboards 

for the data visualization based on birth and pregnancy data. The storyboards were designed to present the 

data in the order and manner that are most useful and understandable. Historical data begin in 1909 (or earliest 

available) and go through 2013, and were compiled from a variety of CDC/NCHS resources. These data were 

exported into Excel workbooks and interactive charts were created using the Tableau professional desktop 

software. 

To date, two storyboards have been created and published by maternal age and race/ethnicity (1) general 

trends in the number, rate, and percent distribution of births (2) pregnancy, births rates, and percent distributions 

of births to unmarried women. These have been published in Tableau Public for users to access and 

manipulate. All data are provided in exportable Excel format.

Tableau Public can be a powerful tool in visualizing and understanding vital statistics aggregate data. It also 

provides a repository for these data in one location and Excel-exportable tables for all the data. The 

visualization can be updated quickly and easily as new data become available. This approach can be an 

important adjunct in the dissemination of vital statistics data 

CONCURRENT SESSION D-3

INNOVATIVE RECRUITING AND QUESTIONNAIRE TESTING METHODS

How Do We Bring Them In?: A Case Study in Using Non-Traditional Approaches to 

Recruiting Focus Group Participants

Ashley Schaad (ICF International) and Rikki Welch (ICF International)

This paper focuses on the effectiveness of several non-traditional strategies used to recruit focus group 

participants. The effectiveness of each strategy – including a highly targeted pay-per-click ad campaign through 

Facebook, advertisements on Craigslist, and partnerships with local organizations likely to be affiliated with the 

populations of interest, are compared in relation to one another, as well as in relation to the more traditional 

approach of using a professional recruitment firm. The data come from a project that included focus groups with 

participants from Greater Houston fitting one or more of the following demographic categories as part of a 

disaster preparedness research project: individuals with ambulatory or transportation difficulties, seniors 65 and 

older living independently, caregivers for seniors or individuals with cognitive impairments, individuals who are 

deaf or hard of hearing, individuals who are blind or have low vision, and monolingual Spanish speakers fitting 

the above categories. The Facebook and Craigslist ads were posted in English and Spanish. Separate groups 

of Facebook ads were developed for each population of interest. Local organizations that agreed to assist in the 

recruitment efforts were provided full-page flyers in English and Spanish to display in prominent locations and/or 

distribute to their membership. Potential focus group participants were asked to complete an online screener to 

determine their eligibility for the focus groups. The screener also asked participants how they heard about the 

groups. This paper presents an analysis of the effectiveness of each non-traditional method, including how 
Concurrent Session D Abstracts

48

many people completed the screener and how many actually participated in a focus group, broken out by 

recruitment method. We also discuss the relative effectiveness of the various approaches in garnering 

participants in relation to the time and monetary costs involved with each strategy.

A Multi-Method Approach to Survey Pretesting

Joe Murphy (RTI International), Ashley Richards (RTI International), David Roe (RTI International), and Danni 

Mayclin (Energy Information Administration)

There are several options for evaluating survey questions and questionnaires prior to data collection. 

Traditionally, expert review and cognitive interviewing have been used to identify potential issues with item 

wording to be addressed prior to finalizing the instrument. Recently, a third option has emerged in the form of 

online pretesting via crowdsourcing. With this approach, ready and willing participants are rapidly recruited and 

asked to complete a self-administered web version of a cognitive pretest. This presentation discusses the 

coordinated use of all three methods in the development of a national survey. We present findings on how the 

survey was improved and fitness of use for each method during the pretesting phase.

Our design employed the Question Appraisal System (Dean et al., 2007; Willis & Lessler, 1999) for the expert 

review component of the evaluation. After updates to the questionnaire, we conducted 15 cognitive interviews 

across 3 U.S. cities to learn about respondents’ understanding and strategies in completing the survey. After 

subsequent revision, we conducted a second round of 15 cognitive interviews along with 57 self-administered 

web interviews with participants recruited via Amazon Mechanical Turk. Finally, after a first pilot survey, we 

conducted a second round of self-administered web interviews with another 93 participants via Mechanical Turk.

This presentation will summarize the range of options available for pretesting survey instruments; introduce the 

new options for evaluation with a specific focus on online pretesting via crowdsourcing; summarize findings from 

a survey employing the three pretesting methods in an iterative fashion; and discuss applicability of the different 

methods for investigating particular types of questionnaire issues prior to finalizing a survey instrument.

Cognitive Interviewing for Surveys on Sexual Assault with College Students

Sarah Cook (RTI International), Michael Planty (Bureau of Justice Statistics), Jessica Stroop (Bureau of Justice 

Statistics), Chris Krebs (RTI International), and Christine Lindquist (RTI International)

This paper focuses on the cognitive interviewing methods that were used to test and improve a survey 

instrument that was ultimately used for the Campus Climate Survey Validation Study (CCSVS) Pilot Test, which 

involved using a web-based survey to collect data from approximately 22,000 undergraduate students across 

nine universities. Cognitive testing was done using two approaches: 1) via crowd-sourcing using an online 

opinion hub, and 2) via in-person interviews in three geographical areas (Raleigh, Durham, and Chapel Hill, NC; 

Washington, DC; and Portland, OR). Male and female college students were included in cognitive testing, and 

targeted recruitment and collaborations with crisis centers and universities enabled the inclusion of survivors of 

sexual assault in the in-person interviews. Through the cognitive testing process, feedback was obtained on key 

sections and items within the CCSVS instrument from 240 online respondents in our desired demographic 

(college students between the ages of 18-24), and from 36 college students who participated in the in-person 

interviews, 20 of whom were survivors of sexual assault. The audience will learn about how cognitive 

interviewing can be done different ways, and why doing cognitive interviewing is such an important component 

of research when studying sensitive topics, such as sexual assault, and when surveying college students. 

Although all respondents were college students, there is tremendous diversity among this population across a 

number of dimensions, including geography and socioeconomic status. Cognitive interviewing can help identify 

questions or terms that are not universally understood, and result in important improvements to survey 

instruments.
Concurrent Session D Abstracts

49

Modeling the Effect of Diverse Communication Strategies on Decennial Census 

Test Response Rates 

Gina Walejko (U.S. Census Bureau), Nancy Bates (U.S. Census Bureau), Rachel Horwitz (U.S. Census 

Bureau), and Monica Vines (U.S. Census Bureau)

To conserve decennial census costs, the Census Bureau has focused several research efforts on ways to maintain 

and increase self-response rates. Although used less during other Census Bureau data collections, one 

component of a successful decennial self-response operation has been an integrated communications campaign

Past decennial censuses used an array of advertising to build awareness and promote participation. The 2000 

Census was the first to benefit from paid advertising. The media campaign included television, radio, Internet, 

print, and out-of-home advertising, and messages targeting both the general public and select populations 

encouraged audiences to fill out their census forms. The 2010 Census campaign employed a bigger and expanded 

paid advertising operation. 

Since then, the use of digital advertisements has increased, and the Census Bureau is interested in how to take 

advantage of this medium. The 2020 Census will likely include an integrated communications campaign with more 

digital advertisements to complement an Internet self-response option and ability to process non-ID households 

in real-time.

In preparation for 2020, the Census Bureau tested several advertising and communication strategies in 

experimental panels across the Savannah, Georgia designated market area (DMA) in the spring of 2015. The 

Census Bureau served varied mixes of digital media to five segmented test panels and several other DMA-wide 

experimental treatments. 

Using advertising data, paradata, and response data, this presentation discusses the effectiveness of targeted 

digital advertisements, saturated advertising buys, telephone reminder or “influencer” calls, and mail pieces on 

response rates across self-response modes. Specifically, we discuss the results of a model-based approach that 

measured the effectiveness of an integrated communications campaign on self-response – an approach that 

controls for a diverse set of communication activities and their interactions. Findings will inform policy decisions 

for increasing census response rates, including how to use digital advertising to promote the 2020 Census and 

beyond.

Respondent Driven Sampling with On-line Recruitment and Adaptive Follow-ups

Ronaldo Iachan (ICF), Naomi Freedner (ICF), and Karen Trocki (ARG) 

This paper describes a methodology that ICF developed for on-line respondent driven sampling (RDS) sample 

recruitment and telephone data collection. The methodology includes novel methods for selecting both seeds 

for the RDS sample from a national sample, and a comparison sample matched to the seeds subsampled from 

the same national sample. We provide some background for the national sample survey, the National Alcohol 

Survey, and the design of the study of sexual minority women that is the subject of the paper. We discuss 

alternative methods for matching including stratified random subsampling and propensity score matching. The 

paper also describes the adaptive approaches planned for controlling the RDS sample network structure. 

These approaches use network data and para-data to control network parameters such as chain growth, or 

chain length, and intra-chain correlations. We discuss the importance of these parameters, and their control, for 

effectiveness of the RDS sample and for ensuring the quality of the data. In particular, these methods help 

prevent any possible scams that might arise from the online nature of the coupon distribution and the associated 

incentives. 
Concurrent Session D Abstracts

50

CONCURRENT SESSION D-4

TIME SERIES AND FORECASTING

Structural Changes via Threshold Effects: Estimating U.S. Meat Demand Using 

Smooth Transition Functions

Nestor M. Rodriguez (Economic Research Service-USDA) and James Eales (Purdue University) 

Structural change has been a focus in the food demand literature. Studies have analyzed structural change 

using a variety of different approaches and methodologies. This paper uses macroeconomic economic factors 

as a motivation to examine structural change under a new context. Because of the “Great Recession”, 

economic activity in the U.S., as measured by GDP, contracted from the last quarter of 2007 until the second 

quarter of 2009 resulting in acute economic effects like higher unemployment. Policymakers instituted 

expansionary monetary and fiscal policies to address the downturn in the economy. However, these policies 

may be potentially inflationary in the long run. 

This study will look at the potential effects of both unemployment and inflation on food demand, specifically meat 

demand. Quarterly meat data from the USDA including data on beef, pork, and poultry is used from 1960 -

2013. An Inverse Almost Ideal Demand System (IAIDS) model is used in a Seemingly Unrelated Regression 

(SUR) framework. These methods are combined with more contemporary time series modeling techniques, 

specifically Smooth Transition Autoregression (STAR) modeling. The resultant Smooth Transition Regression 

(STR) model can be used to examine the effects of transition variables on U.S. meat demand. 

Results include that expanding the IAIDS model by using unemployment and inflation as transition variables 

provide models that have more explanatory power based on in-sample performance and standard statistical 

criterion compared to the basic IAIDS model. Consumer preferences for meat are sensitive to the 

unemployment rate and its effects resulted in transitions between own-price and scale flexibility regimes. These 

transitions coincided with periods of high and low unemployment. Consumer preferences for meat are sensitive 

to inflation and its effects resulted in transitions between own-price and scale flexibility regimes. These 

transitions coincided with periods of high and low inflation. Given recent economic events, understanding the 

impact of these macroeconomic events on consumer food demand seems essential and can be informative to 

industry, policy makers, and researchers who examine and analyze this particular market. 

Measuring Single-Year Poverty Transitions: Opportunities and Limitations

Ashley Edwards (U.S. Census Bureau)

Previous research has shown that over the period from 2009 to 2012, the median length of a poverty spell 

was 6.2 months. However, while longitudinal data over multiple years has the benefit of capturing a greater 

number of poverty spells and reducing the proportion of spells affected by left or right-censoring, 

researchers and policymakers must wait years for data to be released, and the impact of sharp swings in 

the economy is masked over extended periods. 

The biggest limitation in providing single-year estimates of poverty spell dynamics relate to issues of spell 

capture and censoring. As researchers shorten the observation period, they capture fewer poverty spells 

and a greater proportion of spells are unable to be assigned “true” start and end dates as spells begin prior 

to the start of the calendar year or continue beyond the end of the calendar year. 

Analysis within single calendar years from 2009 to 2012 captures approximately 20 to 35 percent fewer 

spells when compared to the 4-year period, and left-censoring increases by approximately 240 to 265 

percent. Further, the traditional practice of dropping left-censored poverty spells from duration estimates 

introduces an unexplored amount of bias, as approximately 55 to 60 percent of spells are excluded due to 

left-censoring. 
Concurrent Session D Abstracts

51

This research presents estimates of single-year poverty dynamics across various demographic groups 

over the period from 2009 to 2012. Findings suggest that small differences in medians across calendar 

years combined with relatively high standard errors limit the conclusions that can be drawn by comparing 

year-to-year changes. Additionally, while the characteristics of left-censored spells captured within a single 

calendar year are fairly comparable with left-censored spells observed over the 4-year period, the 

increased capture of left-censored spells within single calendar years leads to greater bias when excluding 

left-censored spells, and often prohibits the assignment of median spell durations.

A Tale of Two Series: Examining Seasonality In Changing Time Series

John Stewart (Economist, Bureau of Labor Statistics) and Steve Mance (Economist, Bureau of Labor Statistics)

The Current Employment Statistics – State and Area (SA) program at the U.S. Bureau of Labor Statistics 

publishes sample-based employment estimates for over 400 metropolitan areas in the United States. These 

areas were redefined by the Office of Management and Budget and this caused breaks in the SA time series. 

Potentially, these changes could have prevented the publication of seasonally adjusted data for those areas due 

to a reliance on unavailable sample-based histories to forecast factors for the coming year. Series constructed 

using administrative data showed statistically significant seasonal changes, even for areas that experienced 

small level changes, although areas undergoing smaller compositional changes tended to show less severe 

breaks. This research used several measures to identify changing areas with potential breaks in seasonality and 

derived a threshold for evaluation. Areas whose change was greater than that threshold would have historical 

sample-based data deemed likely discontinuous; other areas would remain eligible for publication. This allowed 

the SA program to provide as much continuity in its publication of seasonally adjusted data as possible.

Improving NASS’s Crop Yield Forecasts 

Noemi Guindin-Garcia (United States Department of Agriculture, National Agricultural Statistics Service)

Monitoring crop condition and predicting yield on regional and national scales is becoming increasingly important 

in developing countries and has sustained importance for government agencies, private industry, and researchers 

under current weather and climate change conditions. Improving the crop yield forecast is the subject of continual 

study in the United States Department of Agriculture’s National Agricultural Statistics Service (USDA-NASS). In 

recent years, crop yields forecasting has become more complex and challenging since anomalous weather 

conditions (e.g., droughts, heat waves, freezes, and floods) have been observed in major United States cropproducing regions.

One of the main challenges in forecasting crop yields is to develop meaningful statistical indicators about crop 

condition during anomalous conditions due to the ability of plants to adapt and compensate during short period of 

stress. USDA-NASS is developing a Decision Support System (DSS) to monitor the effect of climate variability 

and extreme weather events on crop yields during the growing season. Agronomic parameters, soil, weather, 

and remote sensing data can be combined into a decision support system to develop meaningful statistical 

indicators about crop condition to adjust crop yield forecasts during anomalous weather conditions. The integration 

and adaptation of these products would greatly enhance NASS’s decision-making capability. The main goal is to 

detect areas under potential crop risk based on weather and crop condition indicators. 
Concurrent Session E Abstracts

52

CONCURRENT SESSION E-1

METHODS FOR MISSING DATA IMPUTATION

Patterns of Item Nonresponse for Income in the National Survey on Drug Use and 

Health

Peter Frechtel (RTI International), Kristen Gulledge Brown (RTI International), and Jonaki Bose (Substance 

Abuse and Mental Health Services Administration) 

On the National Survey on Drug Use and Health (NSDUH), family income is used in many analyses. The item 

response rate for family income tends to be around 90%, which is relatively low for this survey. Since the 2002 

survey, missing values in this variable have been replaced with valid values using an imputation method called 

Predictive Mean Neighborhoods (PMN).

This study analyzes the patterns of nonresponse for the family income variable, with an eye towards improving 

the imputation method for it. Some questions that are answered are: (1) Is the item response rate lower for youths 

than for adults? (2) Among the item nonrespondents, do the youths tend not to know the family income, while the 

adults tend to refuse to answer the question? (3) All respondents have the option to select a proxy respondent 

among the adult relatives in their household. How often do respondents select proxies? Is the item response rate 

higher for the proxy respondents than for the self-respondents? (4) Is the item response rate lower among highincome respondents than it is among low-income respondents? This can be tested indirectly using the predicted 

values from the imputation models.

The results of these tests are used to make recommendations for improving the imputation method. The current 

method does not distinguish between proxy respondents and self-respondents, nor does it distinguish between 

"don't know" responses and "refuse" responses.

Missing Data Methods for Regression Analysis

Susan Edwards (RTI International), Phillip Kott (RTI International), Rachel Harter (RTI International), Peter 

Frechtel (RTI International), Stephen Tueller (RTI International), Jiantong Wang (RTI International), Jonaki Bose 

(Substance Abuse and Mental Health Services Administration), and Sarra Hedden (Substance Abuse and Mental 

Health Services Administration) 

This paper presents various methods for handling missing data available to analysts interested in fitting 

regression models involving data from the National Survey on Drug Use and Health (NSDUH), where some of the 

variables in the models have missing values. Motivation for this work stems from the desire to standardize the 

treatment of missing data for official government analyses.

The NSDUH is an annual national and state level survey that collects information on the use of tobacco, alcohol, 

illicit drugs, and mental health in the U.S. The design of the NSDUH sample involves weighting, stratification, and 

clustering. Many commonly used NSDUH variables undergo a weighted sequential hot deck imputation procedure 

and therefore have no missing values. Item response rates on the NSDUH tend to be high; the vast majority of 

variables have item response rates greater than 95 percent. 

This work seeks to address particular challenges encountered when dealing with missing data on the NSDUH 

survey. Many popular missing data methods for regression models are not ideal for handling data collected via a 

complex sample design. Regression analyses for NSDUH are conducted by a highly diverse group of analysts 

and for a wide variety of purposes.
Concurrent Session E Abstracts

53

Using regression models from earlier NSDUH studies, the typical amount of missing data across survey years 

was assessed. The results from a simulation study using these same regression models to evaluate several 

missing data methods are presented along with guidance on which missing data method to use evaluated by the 

extent of missing data, the ease of implementation, and the anticipated effect on bias and variance. Missing data 

methods considered in the simulation study include listwise deletion, listwise deletion with additional weighting, 

maximum-likelihood estimation, and single imputation using the WSHD method. An overview of software capable 

of performing these methods is also addressed.

Using Classification Trees to Recommend Hot Deck Imputation Methods: A Case 

Study 

Laura T Bechtel (U.S. Census Bureau), Darcy Steeg Morris (U.S. Census Bureau), and Katherine J Thompson 

(US Census Bureau) 

The U.S. Census Bureau conducts an Economic Census every five years. Besides collecting a set of common 

items from all eligible establishments, the Economic Census collects detailed information on each establishment’s 

products. Beginning in 2017, the Economic Census will use the North American Product Classification System to 

produce economy-wide product tabulations from cross-sector collections. This marks a major departure from the 

current collection – which explicitly links products to industry – and makes the trade-area specific missing data 

adjustment practices impossible. An interdisciplinary research team was established to address this, and the 

outcome was the recommendation of a single (unified) methodology for the treatment of missing product data. 

The team conducted a comprehensive evaluation study using empirical and simulated data from the 2012 and 

2007 Economic Census. In all of the studied industries, a form of hot deck imputation appeared to be the best 

compromise of the considered methods. However, the recommended variation (nearest neighbor or random) was 

split between trade areas. Consequently, the team investigated whether certain properties of establishments or 

products might be predictive of one method having better statistical properties than another across different 

subdomains. 

Such exploratory analyses readily lend themselves to classification tree analysis. A classification tree starts with 

a categorical outcome (often binary) and grows branches that represent an explanatory variable. The nodes 

(splits) at the top of the tree represent the covariates that are most strongly related to predicting outcome. In our 

application, the outcome was the choice of hot deck method, and the predictors were characteristics of 

establishment data within the imputation cells. In this paper, we describe how we used classification trees to 

develop an understanding the underlying causes that led to one method preferable to another and provide some 

limited guidance for implementation in the 2017 Economic Census and future analyses.

Within-Industry Productivity Dispersion and Imputation for Missing Data

T. Kirk White (U.S. Census Bureau), Jerome P. Reiter (Duke University), and Amil Petrin (University of Minnesota 

and NBER) 

Within-industry differences in measured plant-level productivity are large. A large literature has been devoted to 

explaining the causes and consequences of these differences. In the U.S. Census Bureau's Census of 

Manufactures, the Bureau imputes for missing values using methods known to result in under-estimation of 

variability and potential bias in multivariate inferences. We present an alternative strategy for handling the missing 

data based on multiple imputation via sequences of classification and regression trees (CART). We use our 

imputations and the Bureau's imputations to estimate within-industry productivity dispersions. The results suggest 

that there is more within-industry productivity dispersion than previous research has indicated. We also estimate 

relationships between plant exit, productivity, prices, and demand shocks. For these estimands, we find the 

results are robust to using an alternative imputation strategy.
Concurrent Session E Abstracts

54

CONCURRENT SESSION E-2

INNOVATIONS IN SAMPLE DESIGN

Optimal Stratification and Allocation for the June Area Survey

Jonathan Lisic (National Agricultural Statistics Service/USDA), Hejian Sang (Iowa State University), Zhengyuan 

Zhu (Iowa State University), and Stephanie Zimmer (Iowa State University) 

A computational approach to optimal multivariate designs with respect to stratification and allocation is 

investigated under the assumptions of fixed total allocation, known number of strata, and the availability of 

administrative data correlated with the characteristics of interest under coefficient of variation constraints. This 

approach uses a penalized objective function that is optimized by simulated annealing through exchanging 

sampling units and sample allocations between strata. Computational speed is improved through the use of a 

computationally efficient machine learning method such as K-means to create an initial stratification close to the 

optimal stratification. To increase the acceptance rate of the simulated annealing algorithm, sampling units near 

initial strata boundaries are selected at a higher rate for potential exchanges. The numeric stability of the 

algorithm has been investigated and parallel processing has been employed where appropriate. Results are 

presented for both simulated data and USDA’s June Area Survey. An R package has also been made available 

for evaluation.

NHTSA’s Data Modernization Project

Fan Zhang (National Highway Traffic Safety Administration), Chou-Lin Chen (National Highway Traffic Safety 

Administration), Eun Young Noh (National Highway Traffic Safety Administration) and Rajesh Subramanian 

(National Highway Traffic Safety Administration). 

The National Highway Traffic Safety Administration (NHTSA) has embarked on a multi-year, multi-disciplinary 

endeavor to modernize its main data systems that sample and collect information on motor vehicle traffic crashes 

to generate nationally representative estimates. A major aspect of this effort is to redesign the survey samples. It 

has been thirty years since these systems were originally designed and the landscape has changed in terms of 

the population, vehicle fleet and driver behaviors and the data needs of the highway safety community have 

changed significantly.

The U.S. Congress authorized NHTSA to reassess the data needs of its internal and external stakeholders and 

use that assessment in redesigning the survey samples. NHTSA has designed new data collection systems that 

are aligned with the data needs and optimized to get information of interest. In addition, NHTSA was asked to 

design a scalable survey that would provide flexibility to address fluctuating budgetary scenarios or nonresponse 

with cooperating entities such as police jurisdictions.

With all these requirements in mind, NHTSA has designed two independent surveys that will provide nationally 

representative estimates of motor vehicle traffic crashes that cover different scopes. These are both three-stage 

sample designs. They are also scalable samples so NHTSA can change the sample size with minimum change 

to the existing sample. NHTSA obtained a sequence of nested PSU samples so that any one of them can be 

used for data collection based on prevailing budget and the selection probabilities can be calculated.

Finally, to guide sample allocation of this complex, multistage design, NHTSA built a mathematical optimization 

model to allocate the sample for various budgetary scenarios while minimizing variance of the key estimates 

generated from the survey.
Concurrent Session E Abstracts

55

CONCURRENT SESSION E-3

INCREASING RESPONSE RATES

Getting a Boost: Improving Cooperation Rates in Youth and Young Adults

Jennifer Cantrell (Truth Initiative) & Randall K. Thomas (GfK Custom Research), Elizabeth Hair (Truth Initiative),

and Donna Vallone (Truth Initiative)

Beginning in May, 2014, two probability methods (ABS and dual-frame phone) were used to recruit a panel of 

12,000 15 to 21 year old participants for an online panel to assess smoking-related behaviors and attitudes for 6 

survey waves. Wave 2 began in January, 2015. After obtaining a 40% cooperation rate in 5 weeks using only 

email invitations and reminders, we increased incentives from a base of $10 to a base of $20, resulting in 

cooperation rates near 60%. 

We implemented an experiment that tested two primary manipulations – increase minimum base incentive from 

$20 to a randomly assigned $30, $40, or $50 and randomly assign to 1 of 6 multiple enhancements:

1. Control (only an email that indicated the base incentive increase)

2. Provision of new sweepstakes [$500 prize] 

3. Interactive Voice Response (IVR) reminder calls 

4. Live phone prompting calls

5. Large postcard reminder

6. All of the above

There were 4,467 participants as of March 20 who had not responded at all to any prior communications efforts 

that formed the base for assignment to conditions. The cooperation rate for the ABS sample prior to the 

experimental implementation was 68.9% and for telephone recruit sample it was 58.8%. The increase in total 

cooperation rate due to the experimental implementation was 8.6% for ABS and 5.8% for telephone sample, with 

699 of the 4,467 participants assigned to the experiment complete the Wave 2 study for a 15.6% overall 

cooperation rate across conditions. In our preliminary analyses, the independent influence of ‘Large postcard’ 

had the largest effect on cooperation, with Sweepstakes and Live phone prompt having lower, but still significant 

positive effects on response. IVR did not have a significant impact on response. We discuss these results in the 

context of motivating the hardest to reach of low participating samples.

Designing a Multipurpose Longitudinal Incentives Experiment for the Survey of Income and 

Program Participation

Ashley Westra (U.S. Census Bureau), Mahdi Sundukchi (U.S. Census Bureau), and Tracy Mattingly (U.S. Census 

Bureau)

The U.S. Census Bureau has experimented with the use of monetary incentives in the Survey of Income and 

Program Participation (SIPP), a demographic longitudinal survey, since the 1996 Panel. As with most surveys, the 

main goal of using incentives is to increase response rates, especially when facing a steady increase in 

nonresponse over the course of a panel. For the most recent SIPP panel, the 2014 Panel, the survey has been 

extensively redesigned, with households being interviewed only once a year instead of every four months. Since 

this redesign could have an impact on the effect of incentives, a new incentives experiment is introduced for 

Waves 1 and 2 of the SIPP 2014 Panel. In addition to investigating the effect of incentives on response rate, we 

design a way of assigning incentives using a response propensity model with the purpose of reducing 

nonresponse bias. This new methodology is made possible due to the longitudinal design of the SIPP. We will 

outline the design of the multipurpose incentives experiment for Waves 1 and 2 of the SIPP 2014 Panel and 

provide preliminary results.
Concurrent Session E Abstracts

56

Incentive Types and Amounts in a Web-based Survey of College Students

Chris Krebs (RTI International), Michael Planty (Bureau of Justice Statistics), Jessica Stroop (Bureau of Justice 

Statistics), Marcus Berzofsky (RTI International), and Christine Lindquist (RTI International)

For the Campus Climate Survey Validation Study (CCSVS) Pilot Test, two incentive amount experiments were 

conducted to determine the optimal dollar amount for increasing response rates while balancing costs to 

implement, and a new incentive type was tested with a large sample of college students. The CCSVS Pilot Test 

involved using a web-based survey to collect data from approximately 22,000 undergraduate students across nine 

universities. Understanding what types and amounts of incentives are effective for encouraging survey 

participation among college students, especially for surveys on sensitive topics such as sexual assault, is 

important given the recent push for more surveys of this kind. Incentive-related results from the CCSVS Pilot Test 

will be discussed. Students were randomized within schools to receive gift cards of different values, and CCSVS 

respondents were able to choose among nine online and in-store gift card options. Students at two schools were 

randomized to receive either a $10 or $25 gift card, and students at two other schools were randomized to receive 

either a $25 or $40 gift card. All students at the five remaining schools were offered a $25 gift card. A distribution 

of the online and in-store gift cards chosen by the CCSVS respondents will also be presented. The audience will 

learn which incentive types are most effective for encouraging college students to participate in surveys; and what 

impact different incentive amounts have on response rates. The CCSVS Pilot Test results will help universities, 

researchers, and the government plan and budget for what is necessary to implement methodologically rigorous 

studies of college students on sensitive topics such as sexual assault.

Efforts to Address Respondent Concerns in the American Community Survey

Todd Hughes (U.S. Census Bureau) 

The American Community Survey (ACS) provides vital and high-quality information about the population in our 

Nation’s communities on important social, demographic, and housing topics including the age distribution, the 

composition of families, the number of veterans, the disabled, incomes, employment, and so on. However, the 

ACS places a burden on those who fill out the survey, and parts of the survey cause concern among survey 

respondents. It takes an average of 40 minutes to answer the 72 questions on the survey. Some of the ACS 

questions are sensitive, and sometimes people are understandably hesitant to answer them even though their 

information is kept confidential by law.

Multiple research projects are underway at the U.S. Census Bureau to address these respondent concerns. 

Research into including some questions on the survey only periodically, or asking questions of a smaller subset of 

ACS respondents in cases where those agencies do not need certain data annually could lead to reducing 

respondent burden. Other changes to survey design (such as matrix sampling) could provide additional options 

to reduce the number of questions asked of individual households. Research into reducing the number of 

contacts attempted to households, and revising the messages in materials sent to households may help to 

address some respondents’ concerns with intrusiveness. Research into the use of information that households 

have already provided to other federal agencies could lead to reductions to the number of questions asked on the 

ACS. An examination of questions perceived as difficult or sensitive could identify changes in question wording to 

address these concerns. This presentation will provide an overview of these various research projects, and some 

preliminary results from each.
Concurrent Session E Abstracts

57

CONCURRENT SESSION E-4

DEMOGRAPHICS IN LARGE-SCALE FEDERAL SURVEYS: THE IMPACT OF 

MEASUREMENT AND CLASSIFICATION CHOICES ON KEY SOCIAL 

INDICATORS

As Time Goes By: How Period Data Influence the Estimates of Recently Arrived 

Immigrants in the American Community Survey

Elizabeth M. Grieco (U.S. Census Bureau), Luke J. Larsen (U.S. Census Bureau), and Howard Hogan (U.S. 

Census Bureau)

The American Community Survey (ACS) includes a question on “year of entry” that asks household members 

who are foreign born which calendar year they came “to live in the United States.” Researchers have used the 

year of entry data to analyze “recent” immigrants – usually defined as those who arrived five to ten years before 

the survey. In doing this, they have noted that the number of new arrivals is considerably lower than those who 

arrived in prior years, suggesting the size of the recent immigrant population may be underestimated. The cause 

often is attributed to survey error – sampling, nonresponse, measurement, or coverage – although exactly how 

these errors influence the estimates is rarely demonstrated. This paper shows that the primary cause of the lowerthan-expected estimates is not survey error but the ACS data collection methodology, which causes estimates of 

survey-year arrivals to be downward biased. The chance that a new immigrant will enter the country during the 

survey year increases through time. However, the possibility that a recent arrival is included in sample depends 

on the month of arrival being before the month of interview. This results in an estimated number of new 

immigrants for the survey year that represents about half of expected arrivals for that entire year. In addition, 

when the annual data are aggregated into multi-year period estimates, the inclusion of years in which immigration 

could not occur introduces a strong bias in the calculated estimates for the more recent years of entry in the 

period considered.

Comparison of Migration Data: 2013 American Community Survey and Annual 

Social and Economic Supplement of the Current Population Survey

David Ihrke (U.S. Census Bureau), William Koerber (U.S. Census Bureau), and Alison Fields (U.S. Census 

Bureau) 

Geographic mobility is measured by several surveys conducted by the United States Census Bureau. This paper 

focuses on migration data collected through two different surveys, the American Community Survey (ACS) and 

the Annual Social and Economic Supplement (ASEC) of the Current Population Survey. It is the first in a series of 

evaluations planned to help users understand the Census Bureau surveys, migration estimates, and how they 

differ.

The ASEC offers a historical perspective on annual migration that is unparalleled. It first asked respondents about 

their movement in 1948. Data are available at the national and regional levels. There are two important measures 

offered by ASEC, reason for move and distance moved. Neither are currently available through the ACS.

While the ACS cannot highlight long-term trends, its strengths include the ability to focus on smaller geographies 

and populations. Migration data are available to the city/town level. There are also ACS flow products available 

down to the county level. This level of detail is too refined for ASEC migration data. 
Concurrent Session E Abstracts

58

Even though the two surveys have similar migration questions, design and methodological differences result in 

different estimates. Previous work by Koerber (2004) explained many of these differences. They include: how 

people are surveyed, when they are surveyed, and differences in the population universe. This paper revisits 

these differences and notes changes since full ACS implementation. The analysis plans to evaluate national 

mover rates for each survey. The mover rates will be deconstructed by selected characteristics, such as age 

groups and collection mode. Limited comparisons at the state and county level will also occur. 

General guidance is provided regarding when to use the ACS or ASEC for migration estimates. The paper 

concludes with plans for a series of evaluations of Census migration data compared to other surveys, population 

estimates, and administrative records. 

A New Approach to Classify and Analyze Changing Social Structure by 

Race/Ethnicity across Censuses: Complex Households

Laurel Schwede (U.S. Census Bureau)

The 2013 CNSTAT Seminar, “Changing Social Structures and the Meaning of Household in Federal Surveys” 

identified overall household type trends. Noting single-person household increases and household size declines, 

presenters suggested declining household complexity trends.

Other research associated household complexity with census omissions; unemployment; increased young adult 

residential dependency, family instability, and food insecurity.

One problem is that social structure research is hampered by the lack of an accepted standardized definition and 

typology for classifying complex households. Further, most analyses cannot disaggregate beyond Whites, Blacks 

and Hispanics; survey samples are too small to provide reliable statistics for smaller minorities.

The author developed a complex household typology for classifying and analyzing changing household 

structures. This typology extends classification beyond noncomplex single and nuclear families to differentiate 

blended; lineally extended; laterally extended; include nonrelatives; or multiple families. 

Using recoded census data, the author presents the typology, documenting complex household growth from 18% 

(1990) through 21% (2000), to 23% (2010). 

With 2010 Census data, she compares the overall and non-Hispanic White complex household distributions to 

those for Blacks, Hispanics, American Indians/ Alaska Natives, Asians, Native Hawaiians/Other Pacific Islanders, 

other race and multiple races, documenting wide differences, such as non-Hispanic White alone (19%) and 

Hispanic of any race (39%).

This research has four takeaways. First, it provides a new tool: an alternative typology for classifying and 

analyzing household structure. Second, using this tool with past census data enables us to track trends; it may be 

useful in improving household complexity measurement in the Census Bureau’s Planning Database and in 

adaptive design. Third, it enables researchers, policy makers, and minority leaders to identify and compare 

household complexity patterns across all OMB racial/ethnic groups. Fourth, it documents that changing overall 

and non-Hispanic household patterns mask changing subpopulation patterns, especially as minority populations 

grow faster than non-Whites.
Concurrent Session E Abstracts

59

Creating the American Opportunity Study (AOS): A New Tool to Study Trends in 

Social Mobility

Carol C. House (Committee on National Statistics, National Academy of Sciences), J. Trent Alexander (U.S. 

Census Bureau), Jonathan Fisher (Stanford University), David Grusky (Standord University), Michael Hout (New 

York University), Amy O’Hara (U.S. Census Bureau), Aliya Saperstein (Stanford University), and Matthew Snipp 

(Stanford University)

The National Academy of Sciences is overseeing the creation of a novel tool to use existing data sources to 

facilitate research into educational, occupational, and economic mobility across and within generations. 

Ultimately, the tool will permit on-demand links and extracts of existing decennial censuses, administrative data 

(e.g., Social Security earnings reports), and other survey data (e.g., the American Community Survey (ACS)) to 

generate reliable longitudinal measurements for the entire population and important subpopulations. 

This paper describes the first phase of a multi-phase activity termed the American Opportunity Study (AOS). This 

phase will be a “proof of concept” and produce six preliminary products: (1) a statistical methodology to digitize 

household members’ names and addresses from 1990 census microfilm records accurately and at low cost and 

append them to the 1990 electronic records; (2) a statistical methodology to assign identifiers to the electronic 

1990 records and link them to later census and ACS electronic records ; (3) a statistical methodology to establish 

intergenerational “relationship pointers;” (4) negotiated arrangements to append further information to these linked 

records from tax data, earnings reports, other administrative sources, and surveys; (5) a statistical methodology to 

estimate errors associated with linking data from multiple sources; and (6) a tool, with appropriate documentation, 

to enable on-demand links of 1990 census data with more recent censuses and selected administrative and/or 

survey data. 

Future phases will build on this initial work to enhance the tool and the methods for error estimation, to capture 

the information from the entire 1990 census and earlier censuses that is necessary for linking, and to develop 

methods for imputing or filling in gaps in the information. The data provided by the AOS will enable researchers to 

illuminate contemporary levels, differentials, and trends in mobility, and also provide valid comparisons with past 

decades.

Travel Time Use Over Five Decades

Chao Wei (George Washington University) and Chen Song (George Washington University)

In this paper, we use five decades of time-use surveys, including the annual American Time Use Survey between 

2003 and 2013, to document travel time uses in the aggregate and across demographic groups. We find that total 

travel time features an inverted-U shape over time, registering around 20 percent increase from 1975 to 1993, but 

18 percent decline from 1993 to 2013. 

We use the Blinder-Oaxaca method to decompose changes in the unconditional mean of total travel time into the 

portion explained by demographic shifts and the portion explained by changes in within-cell means and time 

effects. We find that demographic shifts explain around 45 percent of the increase in total travel time from 1975 to 

1993. Increases in educational attainment alone contribute to around 28 percent of the increases. Demographic 

shifts play a much smaller role in the evolution of total travel time afterwards.

We find that variations in total travel time from 2003 to 2013 are dominated by time effects that are common to all 

demographic groups. In particular, the shift of time allocation from travel-intensive non-market work to travel-nonintensive leisure accounts for around 50 percent of the decline in total travel time. There is no strong evidence for 

economizing on travel during the recent decade. 

We also examine how travel time covaries with other time use categories. We find that the decline in travel time is 

associated with reduction in time spent on leisure outside home, time spent on obtaining goods and services, and

time spent on civil activities. Leisure at home, especially entertainment using computer and TV and sleeping, are 

the major alternative time uses coupled with a decline in total travel time. Effectively, there is an increasing 

allocation of time to leisure at home compared to leisure outside.
Concurrent Session F Abstracts

60

CONCURRENT SESSION F-1

COMPARING AND UNDERSTANDING DIFFERENCES IN ESTIMATES 

ACROSS SURVEYS

Health Care Use And Expenditure Data For Elderly Medicare Beneficiaries: A 

Comparison Of Two Surveys

Lisa B. Mirel (Centers for Medicare and Medicaid Services) and Steven R. Machlin (Agency for Healthcare 

Research and Quality)

The population aged 65 and older is predicted to almost double in 2050 compared to 2012. This population’s 

health care use and expenditures is important to understand for health care policy. Two nationally representative 

surveys, the Medical Expenditure Panel Survey (MEPS) and the Medicare Current Beneficiary Survey (MCBS), 

capture this age group and their health care use and expenditures. The goal of this paper is to compare and 

contrast the surveys and, to the extent possible, compare use and expenditure estimates for select types of health 

care, using similar analytic samples. In terms of similarities, both surveys have panel designs and the overlapping 

panels are used to create yearly files and estimates. Use and expenditure information is collected in a similar 

manner for both surveys. However, MEPS enhances its expenditure estimates with data collected from a sample 

of medical providers who provided care to the household respondents while MCBS enhances its estimates with 

Medicare claims data. The main difference between the two surveys is their target population. In MEPS the 

sample frame is the US civilian, non-institutionalized population; for MCBS it’s the Medicare population, which 

includes both institutional and non-institutionalized persons. In this paper we show that despite attempts to derive 

similar analytic samples for comparison, differences still exist between the surveys’ results. These differing results 

are both important and relevant to consider when interpreting data from either source from a policy perspective. 

Comparability In Race Reporting Between Cancer Registry Data And The National 

Surveys: Results From SEER-NLMS Linked Data

Mandi Yu (National Cancer Institute), Sean Altekruse (National Cancer Institute), and Kathleen Cronin (National 

Cancer Institute)

The cancer surveillance data collected by the NCI’s SEER has been used extensively in monitoring cancer health 

disparities (HD) in the United States. Very often, the study of race/ethnic disparities involves comparing cancer 

incidence rates across population subgroups. It is important that the rates are uniformly measured and estimated 

for all subgroups. However, concerns remain over the differential comparability of race data collected in the 

cancer surveillance system (numerator) and the census population estimates (denominator) across racial/ethnic 

groups, and more importantly, whether such differences lead to invalid conclusions about racial/ethnic HD.

Research published in the Cancer Causes Control in 2007 found mild to moderate differences in race and 

Hispanic ethnicity reports between SEER and the National Longitudinal Mortality Study (NLMS) for cancer cases 

diagnosed from 1973 to 2001. This difference is mainly attributable to the use of different data source because 

SEER heavily relies on information abstracted from hospital’s medical records and administrative databases, 

whereas survey data is self-reported. Since 2000, the collection of denominator race data has switched to the 

new race classification system to comply with the 1997 OMB’s recommendations. Despite the compliance effort, 

SEER continues to use the old system, thus creating a potential mismatch and source of error. 

This study will extend this 1973-2001 analysis to 2011 and evaluate the comparability in race/ethnic reports due 

to differential reporting sources and classification systems. The results from this study provide guidance to 

researchers as to using population based cancer registry data to ascertain disparities in cancer incidence across 

racial and Hispanic ethnicity groups.
Concurrent Session F Abstracts

61

Self-Reported vs. Administrative Race/Ethnicity Data and Racial Disparities in 

Criminal Justice System

Zhen Zeng (Bureau of Justice Statistics) and E. Ann Carson ((Bureau of Justice Statistics) 

As a federal statistical agency, the BJS must adhere to the 1997 Office of Management and Budget (OMB) 

standards on race and ethnicity in its data collection efforts. A principle of the OMB standards is that an 

individual’s response to the race question should be based upon self-identification. However, the BJS routinely 

receives administrative record data of correctional populations, where the classification of an individual’s race and 

Hispanic origin is not based upon self-identification. 

This study compares aggregate counts of prisoners by self-identified race/Hispanic origin categories from BJS’ 

establishment surveys (i.e., Survey of Prison Inmates 2004 and National Inmate Survey 2007, 2008-9, 2011-12) 

to those from state administrative records (i.e., National Corrections Reporting Program and National Prisoner 

Statistics Program). Compared to self-reports, administrative records tend to overestimate the percentages of 

prisoners who are white or black, while underestimating the percentages of prisoners who are Hispanic, Native 

American, Asian, or multi-racial. The disparity in measurement varies significantly by a combination of factors, 

including race, sex, age, and collection year. 

Using Poisson regressions, we predict prisoner counts by self-identified race from administrative records. The 

results allow us to estimate the distribution of self-identified race for the years when only administrative race data 

is available.

Finally, we explore how the different measurements of race affect estimates of racial disparity in the criminal 

justice system. The ratio of Hispanic-to-white imprisonment rate increases by 68% when the measurement of race 

is based on self-identification than obtained from administrative data. And the ratio of black-to-white imprisonment 

rate is only 7% higher when race is measured by self-identification. 

CONCURRENT SESSION F-2

LINGUISTIC AND CULTURAL ISSUES IN SURVEY DESIGN AND 

EVALUATION RESEARCH

Multilingual, Multicultural, Multi-mode Testing of Questions for U.S. Census 2020

Alisú Schoua-Glusberg (Research Support Services), Patricia Goerman (U.S. Census Bureau), and Mandy Sha 

(RTI International)

As part of the research program for the population census in 2020, the U.S. Census Bureau is currently 

conducting a set of multilingual studies to refine and improve the census form in Spanish, Arabic, Chinese, 

Vietnamese, Korean, and Russian. A multi-mode approach to pretesting is being implemented. Expert review 

panels were assembled to weigh in on translation and cultural issues in the form and ancillary materials. 

Following their recommendations, revisions were made to the form in each language. Cognitive testing of the 

revised forms in each language is being carried out to identify patterns of interpretation of questions and 

terminology, as well as uncover cultural issues that may be impairing question equivalence across languages and 

target populations. Cognitive testing is being conducted in different administration modes: interviewer 

administered and self-administered, on paper and electronic devices. Focus groups will also be convened this 

summer to test acceptance and adequacy of messages for each of the target populations.

This presentation will discuss the types of findings from each study component, the kinds of issues uncovered, 

and the different possible strategies we are adopting to solve the translation and/or cultural issues found. 

Strategies for resolving translation issues in the questions and materials tested will be presented. Ongoing 

analysis with the additional languages will be discussed.
Concurrent Session F Abstracts

62

Does a Spanish Translation of USDA’s U.S. Household Food Security Survey Module 

Reduce Differential Item Functioning Between Hispanic and White non-Hispanic 

Households?

Matthew P. Rabbitt (USDA-ERS) and Alisha Coleman-Jensen (USDA-ERS)

USDA measures food security with the Household Food Security Survey Module (HFSSM), consisting of 

questions about conditions characterizing households having difficulty meeting basic food needs. Prior research 

indicates there may be differences in how Hispanic and White-non-Hispanic reference persons respond to items. 

A key assumption of the Rasch model, a type of Item Response Theory model on which the HFSSM is based, is 

that the severity and discrimination of items are invariant across subpopulations. Violation of this assumption is 

differential item function (DIF). There is modest DIF between Hispanic and White-non-Hispanic households with 

children which may arise from linguistic and cultural differences. In order to standardize Spanish-language 

interviews, USDA implemented a Spanish translation of the HFSSM in the Current Population Survey Food 

Security Supplement (CPS-FSS) in 2012.

We use data from the 2012-2014 CPS-FSS to estimate Rasch models of food security. The Rasch model 

assumes responses to the HFSSM items may be modeled as indicators of a single underlying index of food 

security. Separate Rasch models are estimated for households interviewed with the Spanish versus English 

HFSSM. Model parameters will be compared using tables and plots to assess the extent of DIF. To assess the 

role of acculturation in the observed DIF between Hispanics and non-Hispanics, we will estimate behavioral 

Rasch models that control for key households characteristics, such as country of origin, and year of entry into the 

U.S., and citizenship status. We also compare responses of Hispanic-headed households interviewed in English 

versus Spanish.

Adapting the Medicine Wheel Model to Extend the Applicability of the Traditional 

Logic Model in Evaluation Research

Susan Jenkins (U.S. Department of Health and Hunan Services, Administration for Community Living) and Kristen 

Robinson (Social & Scientific Systems, Inc.)

Evaluation is a systematic process for obtaining information on program activities, outcomes, and effectiveness. 

Performance measurement tracks how/whether program activities accomplish specific goals, objectives, and 

outcomes. Both require stakeholder agreement about inputs and activities/outputs, goals, objectives, and 

outcomes. A logic model is a plausible diagram of the sequence of causes (inputs, activities/outputs) that produce 

outcomes. Because a logic model illustrates a program’s logic/theory and focuses attention on the important 

connections between actions and results, it can build a common ground among stakeholders and establish a 

framework for evaluation and measurement. 

Traditional evaluation models organize projects linearly. Results that are consistent, and which can be isolated 

and measured are given precedence over more variant and interconnected outcomes. Such models can, 

however, neglect to involve the project’s various stakeholders in the evaluation process and in the development of 

the evaluation framework, leaving many of the meaningful outcomes unreported. Conversely, a Medicine Wheel 

model is a more holistic concept of evaluation/measurement. Its circular form and all‐encompassing categories 

help to expand an evaluation to capture outcomes and elicit responses that are otherwise overlooked. Developed 

within a Tribal context, the Medicine Wheel is meant to make sense of the world and bring order to it, without 

isolating or compartmentalizing various program components and intended individual, community, or systems 

outcomes. 

This paper describes the Administration for Community Living’s use of both the traditional logic model and 

Medicine Wheel model to develop a more robust evaluation of its Older Americans Act Title VI Tribal Grants 

Program. Evaluation designs using both models will be discussed as well as the implications each has on 

potential evaluations.
Concurrent Session F Abstracts

63

CONCURRENT SESSION F-3

PREDICTIVE ANALYTICS IN A RANGE OF CIRCUMSTANCES

A Predictive Model of Patient Readmission Using Combined ICD-9 Codes as 

Engineered Features 

Robert P. Yerex, UVA Medical System

The timing of post-discharge care is a significant factor in reducing unplanned hospital readmission. Statistical 

learning techniques can be applied to the development of models that predict the likelihood of patient readmission 

during the critical 30 day post-discharge period. The accuracy of these models is dependent on the quantity and 

quality of data used for training and validation. For Medicare and Medicaid patients who are members of an 

Accountable Care Organization (ACO), the Centers for Medicare and Medicaid (CMS) provides detailed claims 

based data that can be used, if appropriately collated, and transformed. This involves the identification and 

creation of useful features, which when included in the model, increases its predictive strength. Creation of 

derived features (feature engineering) is a process in which a large number of base dimensions (n) are combined 

to create a smaller features set (n* << n), reducing the complexity of the model, while retaining its inherent 

information value. The inpatient admission diagnoses, in the form of ICD9 codes, are an example of high 

dimensionality attributes found in the CMS claims data. Patterns inherent in the combinations of these codes can 

be used to create an engineered feature. In this study, a taxonomy of patterns of patient diagnoses was 

developed that was then used as a feature within a random survival forest model that predicts the hazard function 

(where the hazard event is unplanned readmission) of an individual patient for the first 30 days post discharge. 

Over the ensuing 30 days after release from hospital, a patient’s likelihood of readmission can be dynamically 

estimated based on the remaining portion of the hazard curve. Inclusion of the multiple diagnoses feature 

increased model accuracy to the point where it could be effectively used as a tool for targeting post-discharge 

patient care.

Predictive Analytics with Administrative Data from the Mine Safety and Health 

Administration

Yuwen Dai (Summit Consulting, LLC), Natalie Patten (Summit Consulting, LLC), Albert Lee, Ph.D. (Summit 

Consulting, LLC), and George Cave, Ph.D. (Summit Consulting, LLC) 

The U.S. Department of Labor (DOL), Mine Safety and Health Administration (MSHA) would like to predict at an 

early stage which mine operating firms with violations are at risk of failing to pay their fines on time. Summit 

Consulting, LLC (Summit), in collaboration with the DOL Chief Evaluation Office (CEO), used MSHA internal 

administrative data to develop predictive models that identify mine operators who are at high risk of failing to 

make timely payments to MSHA. Summit also created a dynamic, user-friendly Microsoft Excel tool (the Early 

Detection tool) based on the delinquency risk scores calculated by the predictive models. 

The early identification of delinquent operators has three components: 1) identification of the probability to be 

delinquent, and 2) the severity of delinquency in terms of duration of delinquency, and 3) the compounding effect 

of delinquency probability and severity. Summit used logistic regression to predict whether a firm would be over 

90 days delinquent, survival analysis to determine how long a firm would be in delinquency, and developed a 

composite risk score to rank the delinquency risk on violations based on both the predicted probability of 

delinquency and the duration of the delinquency. The early detection model has demonstrated good predictive 

power to discriminate operators with high delinquency risk and has uncovered key relationships in the 

delinquency behavior in a statistical and systematic way. Summit implemented the early detection model in a 

Microsoft Excel application so that MSHA can periodically use in-house resources to conduct the analysis and 

use the results for early enforcement activities. The highly automated tool requires minimum inputs and 

operations from users and has a user-friendly, dynamic interface. 

This project increases MSHA’s internal capacity for conducting and implementing program-enhancing data 

analytics and serves as a proof of concept for using predictive analytics to improve agency performance.
Concurrent Session F Abstracts

64

Responsive Design in Suicide Prevention Program Data Collection

Aruna Rikhi, MPH (HQMC Behavioral Health), Jessica Jagger, PhD, MSW (HQMC Behavioral Health), and Adam 

Walsh, PhD, MSW (HQMC Behavioral Health)

Timely, high-quality data is needed to monitor suicide trends, guide evidence-based suicide prevention and 

clinical efforts, and identify those service members at an increased risk of dying by suicide. In late 2013, United 

States Marine Corps (USMC) Behavioral Health implemented the Marine Intercept Program (MIP) to provide care 

coordination and regular follow up for Marines with reported suicidal ideations or attempts. To effectively 

implement the program, an IT tool was developed to allow for case entry and assignment, and to document 

services provided in accordance with program guidance. 

Public health data surveillance efforts are increasingly tied closely with information technology systems. The MIP 

Tracker was a unique IT tool that was developed through collaboration with stakeholders. These collaborations 

determined the technical requirements and created a flexible IT tool that is able to evolve with changing needs. 

Due to the responsive and adaptive design of this secure database both internal and external stakeholders have 

timely reports to guide their suicide prevention efforts. Marine Expeditionary Force (MEF) Prevention Directors, 

installation CCP individuals, and Quality Assurance are among the stakeholders to benefit from the adaptable 

secure database.

Data submitters can have difficulties with stagnant data entry systems and questionnaires. Surveillance 

personnel also benefit from an adaptable tool. As needs arise, especially for newer programs, a responsive 

system allows for growth and expansion of an evolving public health intervention. These potential changes do not 

affect the long term tracking of at risk Marines and attached Sailors at the population level rather these changes 

can provide additional insight to an individual Marine’s circumstances.

Scenario Forecasting of Insurance Claims Based on the Representative 

Concentration Pathways (RCPs)

Vyacheslav Lyubchich (University of Maryland Center for Environmental Science) 

The adverse effects of climate change bring increasingly more alterations into all aspects of human life and 

welfare, and one of the sectors that is particularly affected is insurance industry. Indeed, the year 2013 brought a 

record amount of claims and losses due to weather related damages, which in the USA and Canada alone cost to 

insurers more than $3 billion. We aim to provide a statistical data-driven insight into (non)linear relationship 

between weather-related house insurance claims and atmospheric variables, and to predict future claim dynamics 

accounting for the changes in extreme weather suggested by representative concentration pathways scenarios. 

We propose to employ a flexible thresholded time series models for count time series of insurance claims, and 

develop a new method to compare tails of the observed and projected extreme precipitation and evaluate its 

impact on number of claims in the modeling framework. We use alternative scenarios of future climate and 

infrastructure development to forecast the potential increase in the number of claims in certain geographic areas 

and advise the insurance industry and policy makers.
Concurrent Session F Abstracts

65

CONCURRENT SESSION F-4

GEOSPATIAL TECHNIQUES

Spatial Data Analyses, Visualization, and Research Opportunities in the Division of 

Cancer Control and Populations Sciences at the US National Cancer Institute

Zaria Tatalovich (National Cancer Institute), Li Zhu (National Cancer Institute), and David Berrigan (National 

Cancer Institute)

The National Cancer Institute has a long history of developing, sharing, analyzing and visualizing spatial data 

related to cancer incidence, mortality, prevalence, and survival, alongside with potential risk factors including 

disparities in health services, environmental factors, and behavior. This work sprouted within The Surveillance, 

Epidemiology, and End Results (SEER) Program of the Division of Cancer Control and Population Sciences, at 

the National Cancer Institute. SEER collects and reports cancer statistics at several levels of spatial aggregation. 

Interactive mapping and statistical tools are available to explore these data: NCI Geo-Viewer, Animated Historical 

Cancer Atlas, State Cancer Profiles, and SEER*Stat are among the most frequently used tools. 

The Division supports a portfolio of intra and extramural research concerning spatial statistics, geospatial 

modeling, small area estimation, health disparities calculation, and environmental determinants of cancer. The 

Division is also supporting substantial research concerning policy influences on cancer risk factors related to 

energy balance such as obesity and physical activity. This work frequently involves spatial approaches because 

exposure and access are critical elements of understanding policy effects on behavior. Finally, we are just 

beginning to invest in mobile Health technology for behavior monitoring and change that integrates spatial data 

into cancer control activities.

This presentation will include handouts summarizing data availability, shared software/tools, funding opportunities 

across the division as well as discussion and slides of major conceptual and analytical issues facing the further 

use of spatial data in the analysis of cancer, cancer risk factors, and cancer control.

Integrating Geospatial and Public Health Data: Examples Using National Center for 

Health Statistics Data Systems

Lauren Rossen (National Center for Health Statistics) and Patsy Lloyd (National Center for Health Statistics)

Previous research has shown that health is related to where people live, work, and play. The National Center for 

Health Statistics (NCHS) collects data from birth and death records, medical records, and health surveys. 

Although the publicly available data files do not typically include geographic data, geocoded data from many of 

the NCHS data systems is accessible through the Research Data Center (RDC). Integrating geospatial data with 

NCHS data systems provides researchers the opportunity to examine how various social and environmental 

determinants of health relate to specific health outcomes, behaviors, risk factors or disparities. Additionally, the 

linkage of geospatial data can augment NCHS data systems by providing multi-level or temporal information. 
Concurrent Session F Abstracts

66

The objective of this presentation is to illustrate how geospatial data can be used in public health research and 

highlight key considerations for researchers conducting geospatial analyses. We will present examples of 

research using geospatial data to examine a broad range of public health topics, including food insecurity and diet 

among children, and associations between air pollution and health outcomes. These examples, among others, will 

illustrate key considerations that researchers should be aware of when using geospatial data for public health 

research purposes, including: limitations related to confidentiality and disclosure risk; selection bias due to linkage 

refusal, or failure to geocode; measurement error or bias due to temporal inconsistencies and/or misalignment of 

geographic boundaries; and issues related to ecological fallacies. Additional methodological concerns may 

include multi-level modeling with complex survey data, combining multiple sources of uncertainty, and appropriate 

methods for smoothing.

Ultimately, integrating geospatial data with NCHS data systems enhances our ability to investigate broader social, 

economic and environmental determinants of health and disparities. Researchers should be aware of the various 

strengths and limitations of using geospatial data in their analyses.

Geospatial Data for Statistical Use from the U.S. Department of Transportation

Edward Strocko (Bureau of Transportation Statistics)

The Department of Transportation collects and compiles geospatial data on the extent, use, condition, 

performance, and consequences of the transportation system. The Bureau of Transportation Statistics compiles 

these data into the National Transportation Atlas Database, the National Commodity Origin Destination Accounts, 

and other products that he research community uses to measure economic and social interactions, transportation 

safety, and energy consumption and environmental impacts. The presentation identifies these geospatial data 

resources and applications, and highlights challenges in developing and using geospatial transportation data.

Stratification Of An Agricultural Area Sampling Frame Using Geospatial Cultivation 

And Crop Planting Frequency Data Layers

Claire G. Boryan (National Agricultural Statistics Service), Zhengwei Yang (National Agricultural Statistics 

Service), and Patrick Willis (National Agricultural Statistics Service)

Area Sampling Frames (ASFs) are the basis of the USDA National Agricultural Statistics Service (NASS) 

agricultural statistics program and many agricultural statistical programs around the world. The NASS ASFs are 

stratified based on percent cultivation. To improve the accuracy, objectivity, and efficiency of the NASS ASFs, this 

paper presents three land cover stratifications created using geospatial corn, soybean and wheat planting 

frequency data layers and a geospatial cultivation layer, which are derived from historical Cropland Data Layer 

(CDL) data. Geospatial crop frequency layers were created to provide predictive crop planting information, aiming 

to improve survey sampling efficiency, estimate accuracy and coefficients of variation (CVs). South Dakota (SD) 

U.S. is selected as the study area for this investigation as it is one of the major crop production states in the U.S. 

and has nearly 100% ground reference coverage to assess test results. Consequently, the corn, soybean and 

wheat crops are the primary focus in this study. The crop planting frequency and cultivation data layers are first 

derived from historical NASS CDL data and validated with Farm Service Agency (FSA) Common Land Unit (CLU) 

data. Three crop specific frames are derived and refined further by percent cultivation based on the NASS 

cultivation layer. These refined crop specific frames based on corn, soybean and wheat mean frequency 

statistics and percent cultivation are evaluated using 2014 SD FSA CLU data. Preliminary results indicate that the 

new method has potential for yielding a more accurate, objective, efficient and lower cost SD Area Sampling 

Frame design, which would be expected to result in improved survey estimates.
Concurrent Session G Abstracts

67

CONCURRENT SESSION G-1

REDUCING MEASUREMENT ERROR: EDITING TECHNIQUES AND USE OF 

ADMINISTRATIVE DATA

Editing Index – A New Methodology of Identifying Response Error

Chin-Fang Weng (U.S. Census Bureau) 

Data editing is a process to identify and correct potential respondent errors in a survey. Ideally, the errors list 

should contain few unimportant errors in order to minimize the subject matter specialists’ work. At the same time 

the listing should lead to maximizing the data quality of a survey. Unlike the current editing methodology, the 

editing index does not involve defining parameters in detecting potential respondent errors. The editing index 

utilizes two criteria: the relative difference (RD), which identifies the potential respondent errors that have higher 

probability of being a true error, while relative importance (RI) identifies the potential respondent errors that have 

the most impact on the final estimates. The editing index is efficient in that it identifies those errors that are both 

non-trivial and are highly likely to be true respondent errors. The editing index affords the analysts flexibility of 

prioritizing the errors to correct by utilizing the results of the RD and RI. This paper will show how the editing 

index compares to the Hidiroglou and Bertholot editing methodology in identifying potential respondent errors in a 

survey. The editing index is illustrated using the data from the Local Government Finance survey. 

Using Administrative Data to Ensure Data Quality in Anonymous Surveys

Marina Murray (Peace Corps) and Angel Velarde (Peace Corps)

When surveys are designed to gather sensitive information, researchers may opt not to collect names, assign 

questionnaire identification numbers, or enable identifiable metadata in online surveys. The Peace Corps Annual 

Volunteer Survey, an attempted census of currently serving volunteers, is administered anonymously to ensure 

that respondents, who may provide candid feedback about the Peace Corps program or staff, are not at risk of 

potential negative consequences.

With an advantage of protecting respondents’ identities and encouraging them to share information honestly, 

anonymous surveys have specific data quality issues. One issue that surveying organizations need to take into 

account the possibility that someone other than a member of the sampling frame responds, or that one 

respondent completes the survey more than once. Researchers are also unable to verify missing or inaccurate 

data with respondents.

Since its inception in 1961, the Peace Corps has maintained administrative data with comprehensive information 

about its volunteers, including demographic characteristics as well as details about their service and work, such 

as length of service, country and region, training hours, primary projects, etc. Peace Corps administrative data are 

the principal source of the agency’s official statistics. In the recent years, the quantity of administrative data has 

grown significantly, and researchers at Peace Corps use these data to strengthen data quality in the anonymous 

Annual Volunteer Survey. 

This paper describes how Peace Corps uses its administrative data to increase the quality of survey data by using 

known population characteristics in verifying respondent eligibility; setting upper and lower limits for numeric 

responses; identifying duplicate responses; checking inconsistency of individual records; and replacing missing 

values. It concludes with a discussion about remaining data quality issues and implications of cross-referencing 

administrative and survey data to respondent anonymity.
Concurrent Session G Abstracts

68

Exploring the Use of External Data in the American Community Survey: 

Opportunities and Challenges

Amy O’Hara (U.S. Census Bureau) and Deborah Stempowski (U.S. Census Bureau) 

The Census Bureau is exploring whether administrative records and third party data can supplement or supplant 

information currently collected in the American Community Survey (ACS). We are analyzing certain data 

elements from the ACS with external data sources, focusing on evaluating data quality and reducing respondent 

concerns. This paper illustrates the challenges and opportunities of using administrative records and third party 

data by discussing preliminary results on key variables including income and flush toilets.

Efforts to integrate external data into ACS are described, including the identification, acquisition, and preparation 

of administrative records and third party data. We discuss measurement issues, particularly definitional 

differences and reference period alignment. We compare the distribution of income reported to ACS and income 

reported to the Internal Revenue Service.

We discuss matching issues, and the challenge of securing external data for all cases. Direct data collection and 

imputation are necessary when external data are unavailable for a portion of the sample. The expected stability 

of data elements is discussed: if an apartment has a flush toilet in 2013, can we assume that the flush toilet 

remains in future years? We discuss ongoing efforts to evaluate such questions, and to document the coverage, 

quality, and availability of external data sources for ACS integration.

What Do We know About the Presence of Young Children in Administrative Records

William P. O’Hare (The Annie E. Casey Foundation)

The net undercount of young children (under age 5) in the2010 U.S Decennial Census was twice as high as any

other age group. The Census Bureau’s Demographic Analysis found a net undercount of nearly a million young

children in the 2010 Census which amounts to 4.6 percent of this age group. Young children also had the lowest 

net coverage rates of any age groups in the Census Bureau’s American Community Survey, Current Population 

Survey and Survey of Income and Program Participation. Use of administrative records is being discussed as a 

way to reduce respondent burden and costs in the 2020 census without compromising data quality. This article 

reviews what we know about the presence of young children in the set of administrative records often used by the 

Census Bureau. Implications for use for administrative records in the Census and major Census Bureau surveys 

are discussed.
Concurrent Session G Abstracts

69

CONCURRENT SESSION G-2

RESPONDENT CONTACT TECHNIQUES AND RESPONSE RATES

Mandatory Messaging in the American Community Survey

Elizabeth Poehler, Dorothy Barth, and Todd Hughes (U.S. Census Bureau)

The American Community Survey (ACS) is a mandatory national household survey. Previous research has 

indicated that the use of mandatory language on mail materials can improve response rates for the ACS (Dillman, 

1996). Therefore, materials mailed to respondents include messages about the mandatory nature of the survey. 

The envelopes for two of the mailings include the statement, “YOUR RESPONSE IS REQUIRED BY LAW” in bold 

font. In response to respondent concerns about the prominence of this message the Census Bureau conducted 

an experiment in May of 2015 to determine the impact of removing this mandatory reference from the outside of 

the envelopes. Several other references to the mandatory nature of participation appear throughout the materials 

mailed to respondents, such as letters, brochures, and reminder postcards, but these materials were not modified 

for the May test. Response rates were compared at various points of data collection to determine the impact of 

the change to the envelopes. A second experiment, conducted in August 2015, tested more comprehensive 

changes throughout the mail materials to soften the references to the mandatory nature of participation in the 

survey. The design and results of the May test and the design of the August test will be discussed.

The Impact of Greeting Personalization on Estimates of Sexual Assault 

Victimization

Ashley Richards (RTI International), Marcus Berzofsky (RTI International), Kimberly Peterson (RTI International), 

Christine Lindquist (RTI International), Christopher Krebs (RTI International), Michael Planty (Bureau of Justice 

Statistics), Lynn Langton (Bureau of Justice Statistics), and Jessica Stroop (Bureau of Justice Statistics) 

Evidence suggests that personalized invitations tend to increase response rates in web surveys (Cook et al., 

2000). However, personalization may have an unintended impact on survey estimates. There is some evidence 

that personalization may reduce self-disclosure on sensitive items (Joinson, Woodley, & Reips, 2007) or increase 

socially desirable responding (Heerwegh et al. 2005), but other studies have been unable to replicate these 

findings (e.g. Heerwegh, 2005; Heerwegh & Loosveldt, 2006). To evaluate the impact of personalization on 

response rates and survey estimates of sensitive items, we compared the effects of personalized and generic 

greetings in a survey on an extremely sensitive topic: sexual experiences, including sexual assault victimization.

We conducted a Web survey with students at five universities. Sample members were randomly assigned to 

receive either a personalized greeting (“Dear John”) or a generic greeting (“Dear [Fill: School Name] Student”) in 

their survey invitation and reminders. Despite the literature suggesting personalization increases response rates,

we hypothesized the personalized greeting would result in a lower response rate because our survey focused on 

such a highly sensitive topic. We also predicted personalization would result in lower rates of self-reported sexual 

assault victimization compared to a generic greeting. This is because we assumed sample members receiving the 

personalized greeting would perceive the survey as less anonymous, making them less likely to participate if they 

had experienced sexual assault victimization, or less likely to report their victimization experiences if they did 

participate.

We compared the effect of greeting on response rates and reported victimization. The personalized greeting 

resulted in a significantly higher response rate, but the generic greeting resulted in higher rates of sexual assault 

victimization; this difference is statistically significant for females. This experiment adds evidence to the divided 

literature on the effect of personalization on self-disclosure on sensitive items. Our findings suggest 

personalization increased response rates but decreased reported victimization, emphasizing that a higher 

response rate is not necessarily indicative of more accurate data.
Concurrent Session G Abstracts

70

Testing a Model-Directed, Mixed Mode Protocol in the RECS Pilot Study

Stephanie Zimmer (RTI International), Paul Biemer (RTI International), Phil Kott (RTI International), and Chip 

Berry (Energy Information Administration)

In designing web/paper mixed mode general population surveys, a typical aim is to maximize response by the 

web while achieving acceptable response rates. To increase web response, the web option is initially presented to 

a sample member and response by paper questionnaire is withheld until later in the nonresponse followup phase 

(the so-called web-first protocol). Evidence in the literature suggests that initially offering the web-only option to 

sample members who cannot or will not complete the questionnaire by web will suppress their overall 

propensities to respond. Ideally, then, one should implement a paper-first protocol to sample members who will 

only complete the questionnaire by paper and the web-first protocol for all others. The paper reports on an 

experiment conducted for the Residential Energy Consumption Survey (RECS) Pilot Study to test this approach 

using a model to identify sample members who have lower propensity to respond by the web. 

In this experiment, half the sample of approximately 5,000 households was randomly assigned to the modeldirected protocol treatment and the remaining half received the web-first protocol. The model used for protocol 

assignment was based on ACS data where the outcome model was “web-access in the home.” In the modeldirected treatment, the 25% lowest propensity cases received an initial invitation to complete a paper 

questionnaire and the remainder received an invitation to complete the questionnaire by web. In the web-first 

treatment, all sample members received an initial invitation to complete the questionnaire by web. 

This paper examines two key hypotheses: (a) no difference in the proportion of the sample responding by web 

and (b) higher response rates in the model-directed treatment. Other issues that will be explored include the 

effects of choice of protocol on: response rates by short vs. long questionnaire; sample representativity; and 

survey costs.

An Experiment Testing Alternative Email Contact Timing Strategies in a Web-Based 

Survey of Federal Personnel

Taylor Lewis (U.S. Office of Personnel Management) and Karl Hess (U.S. Office of Personnel Management)

The Federal Employee Viewpoint Survey (FEVS) is an annual survey of approximately 900,000 permanently 

employed civilian personnel from over 80 agencies. First administered in 2002, the Web-based survey measures 

a wide range of employee perceptions, attitudes, and behaviors, serving as a valuable tool for human resources 

managers to determine which aspects of the organization are working well and which may require intervention. 

Like many other surveys, however, the FEVS has experienced a gradual response rate decline in recent years. A 

response rate surpassing 50% was once routinely attainable, but that threshold has not been met since 2010.

The current contact protocol commences by sending all sampled individuals an initial invitation to participate. 

Five weekly reminder emails are sent to nonrespondents thereafter, generally on the same day and time during 

the work week, and then a final reminder is sent on the Friday of the sixth field period week with a message 

indicating the survey closes today. In this paper, we discuss results from an experiment in which two alternative 

contact protocols were investigated. In the first, individuals were randomly assigned to one of six predetermined 

time blocks for receiving the initial survey invitation. The six cohorts were then rotated through each of the other 

five time blocks for receiving subsequent reminders. In the second, individuals were also randomly assigned to 

one of six predetermined time blocks for the initial invitation, but subsequent weeks’ reminders were scheduled 

adaptively, contingent upon response patterns observed up to that point in the field period. Using survey 

timestamp information and auxiliary information from the sampling frame, response patterns were modeled and 

nonrespondents were sent a follow-up reminder at the time block deemed most likely to elicit a survey complete.
Concurrent Session G Abstracts

71

CONCURRENT SESSION G-3

HEALTH CARE: MEASURING QUALITY, SPENDING AND REFORM

Adjusting the Measurement of the Output of the Medical Sector for Quality: A 

Review of the Literature

Anne E. Hall (Bureau of Economic Analysis)

In January 2015, BEA released the first version of the health-care satellite account, which redefines the good 

being measured in health care output from a single service to an episode of treatment of a specific medical 

condition. This change follows multiple recommendations by the Committee on National Statistics and by 

international authorities on national accounting as applied to medical care. BEA now faces the intensely difficult 

problem of how to adjust the price indexes for the quality of health care. In this paper, I review and summarize a 

number of previous papers that created quality-adjusted price indexes for individual medical conditions. It divides 

them into those that use primarily outcomes-based adjustments and those that use only process-based 

adjustments. Outcomes-based adjustments adjust the indexes based on observed aggregate health outcomes, 

usually mortality. They usually do so by calculating a concept called net value, which is the monetized value of the 

improved health outcome minus the increased spending on the condition. Process-based adjustments adjust the 

indexes based on the treatments provided and medical knowledge of their effectiveness. Outcomes-based 

adjustments are easier to implement while process-based adjustments are more demanding in terms of data and 

medical knowledge. I then calculate outcomes-based adjustments using the net value method for the indexes in 

the health-care satellite account with mortality by cause of death with data from the Centers for Disease Control 

and Prevention. They show that improved outcomes in diseases of the circulatory system created positive net 

value and declining inflation for those conditions but most other categories of diseases exhibit increasing inflation 

because spending on them is higher than the value of the improved outcomes.

What Medical Conditions Are Driving the Spending Slowdown? An Analysis Using 

the New Health Care Account from the Bureau of Economic Analysis

Abe Dunn (Bureau of Economic Analysis), Lindsey Rittmueller (Bureau of Economic Analysis), and Bryn Whitmire 

(Bureau of Economic Analysis)

This year the Bureau of Economic Analysis (BEA) released a new experimental account that tracks national 

health care spending by medical condition. These statistics improve upon our understanding of the health sector 

by blending together both medical claims data and survey data to present new measures of national spending 

and the cost of treatment by condition. This paper introduces key aspects of this account and uses it to study the 

spending slowdown over the 2000 to 2010 period. The account reveals that the slowdown is primarily driven by a 

reduction in cost per case growth, but spending trends can vary greatly across conditions and have differing 

effects on the slowdown. Over half of the overall slowdown can be accounted for by circulatory conditions. 

However, disease categories that include conditions such as diabetes, back problems, and preventative services 

slow more rapidly. In contrast, categories that include conditions such as migraines, depression, and trauma 

actually show acceleration in spending.
Concurrent Session G Abstracts

72

State-Level Estimates from the NHIS Restricted Data: Analysis to support states 

implementation and evaluation of the ACA

Joanna Turner (University of Minnesota - SHADAC)

The Patient Protection and Affordable Care Act (ACA) involves significant changes in health insurance coverage 

and health care systems across the United States. Many of the provisions of the ACA are being implemented by 

states and state specific factors such as health insurance coverage, access, use, and affordability will influence 

both implementation and impact of the law. As a result, a full understanding of the impacts of the ACA over time 

will require assessing the impacts of reform at the state level.

The National Health Interview Survey (NHIS), the nation’s primary survey for monitoring the nation’s health, 

provides a strong base for informing both the implementation and evaluation of the ACA. The NHIS provides 

national estimates for a variety of health indicators, but only selected estimates are available for states with 

sufficient sample size. This study expands the use of the NHIS for state-level analysis of key health care 

outcomes. 

We submitted a proposal to the National Center for Health Statistics to work in a Census Bureau Research Data 

Center (RDC), for access to state identifiers, to develop state-level estimates for ACA relevant indicators and to 

make these publicly available. We discuss the benefits and challenges of working in an RDC, as well as the 

disclosure process. We demonstrate how to access the 2011, 2012, and 2013 NHIS state-level estimates through 

an online interactive data tool. We plan to create an on-going series to support the tracking of health outcomes for 

states over time.

CONCURRENT SESSION G-4

WHAT’S NEW WITH LONGITUDINAL SURVEYS?

An Early Look at the Consumer Expenditure Survey Gemini Redesign Proof of 

Concept Test 

Laura Erhard (Bureau of Labor Statistics), Jennifer Edgar (Bureau of Labor Statistics), and Nhien To (Bureau of 

Labor Statistics)

The U.S. Bureau of Labor Statistics (BLS) began the Gemini Project in 2009 with a goal of redesigning the 

Consumer Expenditure Survey (CE) as a response to increasing evidence in measurement error, declining 

response rates, the emergence of new data collection technologies, and need for flexibility in addressing changes 

in the interviewing environment. In 2013, a redesign plan was approved. In order to test the basic underlying 

structure and components of the new design, a proof of concept test was developed and will be fielded from July 

2015 – October 2015. The main objective of the test is to assess the feasibility of completing one wave of the 

survey redesign--whether a single-sample expenditure survey design consisting of two personal interviews, 

individual diaries, incentives, record usage, and technology usage (electronic diaries) is achievable. The survey 

redesign’s “concept” will be proved if no major methodological, operational, or experience issues are discovered. 

This presentation will provide an overview of the POC test design, a discussion of lessons learned to date, 

highlights from Field Representative feedback on the test, and a preliminary analysis of the test results. 

Preliminary analysis will include completion rates for each component of the test (recall interview, individual 

diaries, and records interview) and sample performance compared to the CE production sample. This 

presentation will be of interest to survey methodologists, data producers, and data users.
Concurrent Session G Abstracts

73

An Overview of the New Survey of Income and Program Participation

Jason Fields (U.S. Census Bureau) , Matthew Marlay (U.S. Census Bureau), and Patrick Campanello (U.S. 

Census Bureau)

The Survey of Income and Program Participation (SIPP) is now halfway through the 2014 panel, having wrapped 

up Wave 2 in mid-2015. This panel began data collection in February 2014 and will conclude Wave 4 collection in 

mid-2017. This paper describes the redesign effort, discusses some challenges and accomplishments in the 

fielding of the survey, and presents some initial statistics from the Wave 1 data.

The Census Bureau spent over five years reengineering the SIPP. The reengineering focused on reducing costs 

and modernizing the survey instrument and data processing system. A key component of the reengineering has 

been a transition to annual interviewing. Through this transition, we simplify the data structure, and are working to 

improve the ability of SIPP to support our data users. 

Fielding the SIPP is not a trivial undertaking. The interview period runs for four months, from February through 

May of each year. The initial 2014 sample consisted of 53,000 households. Of these, we were able to interview 

approximately 30,000, for a final response rate of just over 70%. To get these interviews, our six regional offices 

hired about 1,200 field representatives, who completed an average of 25 interviews each during the four-month 

interviewing period.

The SIPP is focused on collecting information about the economic and social characteristics of the civilian, 

noninstitutionalized population of the United States. To accomplish this, we ask questions about a variety of 

topics, including Demographics, Residency, Fertility History, Program Receipt, Employment and Unemployment, 

Program Receipt, Health Insurance, Assets, and Disability. This paper presents some initial statistics about SIPP 

respondents along these dimensions. These statistics will help other government agencies such as the Social 

Security Administration more efficiently run their programs and more accurately measure eligibility and predict 

future need.

Design of a National Longitudinal Survey of Small Businesses to Assess the Early 

Impact of Healthcare Reform

David Kashihara (Agency for Healthcare Research and Quality) and Robert Baskin (Retired)

Effective January 1, 2014, the Affordable Care Act (ACA) stated that all firms with 50 or fewer full-time equivalent 

(FTE) employees could purchase health insurance for their employees from health insurance marketplaces 

known as the Small Business Health Options Program (SHOP). This paper describes the sample design of the 

2013-2014 Medical Expenditure Panel Survey – Insurance Component (MEPS-IC) Longitudinal Survey and 

presents the new questions developed to measure the impact of the ACA. A longitudinal design was utilized to 

capture more detailed transitions that would be missed by a cross-sectional design. In 2014, this survey sampled 

responding business establishments to the 2013 MEPS-IC that were eligible to use SHOP marketplaces. The 

sample design was planned to balance precision considerations for two key variables (offer health insurance and 

number of enrolled employees) and ACA-relevant comparisons of the insurance characteristics of these firms 

before and after the implementation of the ACA. Relevant survey results will be provided along with planned 

changes for upcoming versions of this policy-relevant survey.
Concurrent Session G Abstracts

74

Movement and Changing Environment on the National Children’s Study: How Can 

We Understand the Impact on a Longitudinal Population Survey

Ned English (NORC at the University of Chicago) Edward Mulrow (NORC at the University of Chicago), Jennifer 

Hasche (NORC at the University of Chicago), Katie Dekker (NORC at the University of Chicago), and Christina 

Park (NICHD)

The National Children’s Study Vanguard Study was a pilot study for a large-scale epidemiological cohort study of 

children and their mothers and fathers. Measures were to be taken from pre-pregnancy until the children reached 

adulthood. The Vanguard Study tested a variety of sampling schemes, measurement forms and protocols, and 

information management systems. Ultimately, it was decided that moving forward with the large-scale study was 

not feasible and it was never launched. Nonetheless, data from the Vanguard Study can be used to address a 

number of research questions in pediatric and environmental health. Once women were enrolled in the study, the 

first participant interview was scheduled to occur before childbirth. Women were then interviewed at childbirth 

and again at regularly-spaced intervals every three to six months, including whether they stayed in place or 

moved.

National longitudinal studies depend on recontacting diverse populations across an extended period of time. 

Movement is not evenly-distributed across populations, however, with lower SES households tending to change 

address at higher-rates than others. Moreover, such households tend to be more challenging to find with 

standard locating methods using databases assuming credit and historical purchasing activity.

The NCS captured historical information about where all participants resided during the course of the study, 2009-

2012. We then geocoded each address in order to understand location and distance moved, in addition to linking 

extant information for analysis. Our analysis focuses on how movers experienced different environments, from 

both a socioeconomic and environmental perspective. We profile how the places in which NCS participants have 

resided in may be similar or different, as well as the types of people who tend to move to different locations. Our 

findings shed light on nature of movement in cohort studies and how the concept of environment can change. 

Such mover profiling will be useful not only in studying retention outcomes but in studying outcomes of child 

health and well-being.
Concurrent Session H Abstracts

75

CONCURRENT SESSION H-1

CHALLENGES AND BENEFITS FROM LINKING ADMINISTRATIVE AND 

SURVEY DATA

Creating Improved Survey Data Products Using Linked Administrative-Survey Data

Michael Davern (NORC at the University of Chicago) and Bruce D Meyer (University of Chicago)

The groundwork has been laid through past administrative to survey data linkage research to create 

improvements in survey data products (microdata, summary files, and report tabulations). However, due to a 

variety of constraints (including cost of altering existing processing systems and potentially introducing new error) 

the opportunities have not been fully exploited. In this paper we lay out a framework which examines the 

component of survey error (Non-response error, sampling error, coverage error, processing error and 

measurement error) to make the case that past research with linked data demonstrates that significant 

improvement could be made in policy relevant survey estimates using linked data products. We use a mean 

squared error metric to estimate the gains in estimate quality that would be possible using linked data. We also 

use findings from past administrative-survey linked research findings on policy relevant estimates of Medicaid, 

SNAP, Social Security, Public Assistance, and uninsurance to make the case that data producers should not only 

continue to invest in data linkage research, but that more importantly, there is an imperative to produce large 

measureable improvements in survey estimate quality by creating enhancements to existing survey data products 

(microdata, summary data and report tabulations). We conclude with an assessment of the costs (both in terms 

of money and timeliness) associated with integrating linkage into the estimation process to produce data 

products. We compare the costs to those often paid by surveys in other attempts to improve survey data quality 

such as reducing unit non-response and increasing response rates.

Developing a Residence Candidate File for use with Employer-Employee Matched 

Data

Matthew R. Graham (U.S. Census Bureau), Mark J. Kutzbach (U.S. Census Bureau), and Danielle H. Sandler 

(U.S. Census Bureau)

The Longitudinal Employer-Household Dynamics (LEHD) program at the U.S. Census Bureau is developing a 

candidate file of residences using federal administrative data sources. LEHD, an employer-employee jobs frame, 

uses place-of-residence information for several missing-data imputation procedures and as a residence 

characteristic in a public use data product. Prior methodologies simply de-duplicated the list of possibly 

disagreeing addresses that pertained to a person record in a given year, using a besting rule. Our proposed 

methodology will use survey response data, linking a person with his or her residence and job, as a truth set, in 

order to train a model to discriminate among a blocked candidate set of administrative residence locations. We 

will then use the estimates to predict probability weights for the entire set of persons and candidate residences. 

Downstream imputation processes and tabulations may draw from the resulting probability distribution, which we 

expect to improve data quality, better represent uncertainty in the data, and minimize exposure of any one source. 
Concurrent Session H Abstracts

76

Using the Census to Evaluate Administrative Records and Vice Versa

J. David Brown (U.S. Census Bureau), Jennifer H. Childs (U.S. Census Bureau), and Amy O’Hara (U.S. Census 

Bureau)

An ideal way of evaluating the accuracy and coverage of administrative records for use in census enumeration 

would be through a comparison to the actual occupancy and number of residents in each housing unit on April 1, 

2010. While the 2010 Census provides information about this, not all Census enumerations are equally reliable. 

Censuses, like surveys, have some level of unit and item nonresponse as well as measurement error.

A common way to evaluate the quality of survey response data is by comparing it to information from administrative 

records on the same people. Meyer and Goerge (2011), for example, compare responses on food stamp receipt 

from both the American Community Survey (ACS) and the Current Population Survey (CPS) to administrative data 

on food stamps. With such an approach, however, one must determine the direction of quality comparison. Is 

comparing the two sources a measure of administrative record quality, Census response quality, both, or neither?

Sources of error in survey data collection have been well documented in the literature (see Groves et al., 2009). 

More recently, researchers have started documenting systematic errors within administrative records sources as 

well (Groen, 2012). At the Census Bureau, researchers have been using administrative records as a research tool 

to assess survey responses, allowing for the possibility that neither the census nor the records are perfect (Mulry 

et al, 2006). This paper follows that vein.

This report posits that some census responses are likely of higher quality than a given administrative record, and 

others may be of worse quality. By exploring characteristics of census responses that we hypothesize are related 

to accuracy, we propose a way to use a combination of survey and administrative record sources to evaluate the 

quality of each, and to provide evidence in support of this method. 

CONCURRENT SESSION H-2

MULTIMODE DATA COLLECTION

Data-Driven Decision Making and the Design of Economic Census Data Collection 

Instruments

Eric B. Fink (U.S. Census Bureau), Jennifer L. Beck (U.S. Census Bureau), and Diane K. Willimack, (U.S. Census 

Bureau)

The economic census, which the U.S. Census Bureau conducts every 5 years, is undergoing a major reengineering effort. For the 2012 Economic Census, establishment-level data were collected using two-selfadministered modes: mail-out/mail-back and electronic. For the 2017 Economic Census, only electronic reporting 

will be available. This presents unique challenges and opportunities for designing electronic instruments. For 

example, in past censuses, paper questionnaires contained long, detailed lists of products that filled multiple 

pages; respondents had to search for goods and services in this list and enter revenue data. A Web instrument 

offers automated features that may reduce the burden of sifting through long product lists, and improve data 

quality. 

This paper focuses on the planned electronic collection of detailed product lines for implementing the North 

American Product Classification System (NAPCS), a comprehensive demand-oriented classification system 

launched among the statistical agencies of the United States, Canada, and Mexico. As part of this effort, we 

analyzed product information that businesses reported in the 2012 Economic Census to discern reporting patterns 

associated with industries in the economic census. These reporting patterns may be leveraged in 2017 

instrument design, providing a familiar starting point for respondents to begin supplying product-level receipts. We 

use the results from this analysis, in conjunction with usability testing, to aid development of an effective design 

for obtaining product-level detail. 
Concurrent Session H Abstracts

77

We demonstrate how analyzing respondent reported data, paired with instrument design testing, may be used to 

inform instrument-design decisions for business surveys. As many Federal surveys increasingly rely on 

electronic-based data collection, electronic instruments can provide many benefits over traditional paper forms. 

The electronic instrument need not have the look and feel of its paper-based counterpart to be effective, but can 

be uniquely customizable to the respondent to improve the survey experience, thus reducing respondent burden 

and increasing data quality. 

Integrated Management of Survey Modes

Jerome Wernimont (Westat) and Ray Snowden (Westat)

In efforts to improve response rates, reduce costs, and take advantage of emerging new technology, it is 

becoming increasingly common for surveys to utilize multiple survey modes to address different needs. Some 

studies use multiple modes at the beginning of data collection, while others adopt a phased approach or 

incorporate additional modes in an adaptive design. These multimode approaches present unique challenges in 

automating case workflow control, providing common services and supporting unified reporting and status 

monitoring. In order to be efficient, the management of the survey process requires integrated systems that 

support data transformation with a high degree of flexibility and real-time control.

This presentation will begin with a short literature review, then discuss the issues and explore examples and 

approaches to developing these capabilities, including solutions that have been adopted or proposed for this 

purpose by a number of survey data collection organizations. The presentation will also feature a case study of 

Westat’s multimode management platform that has been developed to meet many of these requirements, while 

providing core management functions and common services that are needed by many projects through their full 

survey life cycle. A review of several implementations of the multimode manager will be presented. We will 

conclude with an assessment of how well today’s multimode management systems in North America and Europe 

meet emerging survey methods requirements. The need to continue development and expand these capabilities 

will also be explored.

Evaluating Collection Mode Effect with Web Data Collection in the Canadian Labour 

Force Survey

Justin Francis (Statistics Canada) and Guy Laflamme (Statistics Canada)

The Canadian Labour Force Survey (LFS) is a monthly household survey measuring unemployment and other 

labour characteristics. The survey has a rotating panel design, where households remain in the sample for 6 

consecutive months, and has historically used both telephone and in-person collection. In 2015 LFS introduced 

the option for web data collection. At the end of each telephone or in-person interview, respondents can choose to 

complete an Electronic Questionnaire (EQ) online for the following month, instead of having interviewers contact 

them in person or by telephone. To evaluate the impact of the new strategy on main estimates, EQ was 

introduced with an embedded randomized experiment, offering the EQ option to half of each new sample in each 

region. 

Both design-based and model-based analyses were conducted to evaluate possible errors introduced by the 

change in collection mode. Direct comparisons between the two strategies were made for unemployment rate and 

other main survey estimates. Markov latent class analysis was used to evaluate changes to the rates of labour 

force misclassifications across repeated months of collection. Finally, propensity score matching was used to 

compare responses across collection modes while adjusting for self-selection of collection mode choice. Key 

findings as well as challenges encountered are discussed. 
Concurrent Session H Abstracts

78

Agreement across modes of collection in the Occupational Requirements Survey: 

Results from a Pilot Job Observation Test

Kristen Monaco (Bureau of Labor Statistics), Kristin Smyth (Bureau of Labor Statistics), and Tiffany Chang 

(Bureau of Labor Statistics)

The Occupational Requirements Survey (ORS) is an establishment survey conducted by the Bureau of Labor 

Statistics (BLS) for the Social Security Administration (SSA). The survey collects information on the vocational 

preparation and the cognitive and physical requirements of occupations in the U.S. economy, as well as the 

environmental conditions in which those occupations are performed. 

These data are collected by BLS Field Economists who conduct interviews with establishment representatives. A 

question that has been raised during the collection process is whether the data collected through this mode result 

in similar measurements as data that would be collected through direct job observation (which is more typical 

among small scale studies of job tasks). To answer this question, BLS will conduct a job observation pilot test 

during Summer 2015. Under this test, Field Economists will recontact establishments who responded to the ORS 

pre-production survey and collect data on the physical demands and environmental conditions of the job.

Our paper will present the results of an analysis that compares the data from direct collection to that gathered 

through interviewing an establishment representative. The comparisons will be performed at the occupational 

level (defined by eight digit Standard Occupational Classification code) and include visual comparison techniques 

(such as a Bland-Altman plot) and statistical comparisons (through a set of statistical tests, such as MannWhitney-Wilcoxon test).

CONCURRENT SESSION H-3

FRAME DEVELOPMENT AND RESPONDENT SELECTION

Vaping the Web: Crowdsourcing and Web Scraping for Establishment Survey Frame 

Generation

Bryan B. Rhodes (RTI International), Annice E. Kim (RTI International), and Brett R. Loomis (RTI International)

Traditionally sampling frames for establishment surveys have been based on one or more administrative or 

commercial data sources, such as state licensure lists, trade group registries, or Dun & Bradsteet listings. These 

sources, however, have several limitations. One limitation is that the business lists may be updated only 

periodically, and so may not include new businesses, reducing frame coverage. Another limitation is that the list 

may not provide enough information about the business for a particular survey’s sampling needs (e.g., products 

sold, website). By contrast, online business directories that cater to consumers (e.g., Yelp, Google, 

YellowPages.com) are constantly being updated by their user base, and may contain extensive information about 

their businesses. This paper will outline an innovative methodology that uses automated web scraping of these 

online directories along with crowdsourcing to build a sampling frame of specialty sellers of Electronic Nicotine 

Delivery Systems (ENDS), called vape shops, in Florida.

We first used web scraping software to conduct a series of automated searches of online retailer directories. 

These automated searches used combinations of keywords (e.g., vape, vapor, e-cig) and locations (e.g., Orlando, 

Miami) to search for vape shops. The results of these searches were collected and consolidated to create a 

potential list of all vape shops in Florida. Next, to confirm that the businesses identified were in fact vape shops, 

we used Amazon’s Mechanical Turk crowdsourcing platform to have workers call each establishment and verify 

that it met our definition of a vape shop. Through these methods we were able to create an up-to-date list of vape 

shops in the state of Florida in only a few days at fairly minimal costs. This paper will provide details on the web 

scraping and crowdsourcing methods as well as a comparison of the results to a state licensure list of tobacco 

retailers.
Concurrent Session H Abstracts

79

Using a Mobile Mapping Instrument to Evaluate a Permanent Grid Sampling Frame

Denise A. Abreu (National Agricultural Statistics Service), Linda A. Lawson (National Agricultural Statistics 

Service), Claire Boryan (National Agricultural Statistics Service), Michael Gerling (National Agricultural Statistics 

Service), and Rick Hardin (National Agricultural Statistics Service)

The National Agricultural Statistics Service (NASS) conducts the June Agricultural Survey (JAS), which is based 

on an area frame. Segments of land comprise the sampling units for the JAS. Every year, 20 percent of the 

sampled segments are rotated out and a new rotation of segments is introduced. Building and constructing an 

area frame is expensive and time consuming. The agency is evaluating the use of a permanent grid sampling 

frame as a cost saving initiative. 

NASS is also investigating use of mobile mapping technology during data collection. Currently, JAS sampled 

segment boundaries follow roads or other physical features to match the infrastructure on the ground. Field 

enumerators use an aerial photo to locate and interview all operators within the segment boundary. NASS is 

evaluating the modernization of its data collection effort through replacing the use of the aerial photo and paper 

questionnaire with the use of a mobile mapping instrument. 

To test this concept of a permanent grid segment in conjunction with the mobile mapping instrument, enumerators 

in North Carolina, Pennsylvania and South Dakota visited with farm operators during Summer 2014. This report 

documents the challenges faced as well as the lessons learned from this test and discusses future plans for 

NASS’ data collection activities, including the use of Farm Service Agency (FSA) common land units (CLUs). 

Quantifying Urban Agriculture: A Case Study from Baltimore

Linda Young (National Agricultural Statistics Service)

Increasingly agriculture in the urban areas of the US is perceived to be an important factor in food security, 

especially for the socially disadvantaged. As an example, an acre within an urban area may provide numerous 

families an opportunity to grow fresh fruits and/or vegetables. Some of the urban agriculture operations represent 

farms, an entity that sells, or has the potential to sell, a thousand dollars or more of agricultural produce. Others, 

such as a small plot within a community garden, do not. Yet, all can contribute to food security. The USDA’s 

National Agricultural Statistics Service (NASS) has been asked to propose a method for better quantifying urban 

agriculture. A pilot study of a new approach to quantifying urban agriculture is being conducted in two US cities, 

the first of which is Baltimore. Algorithms incorporating information from satellite imagery, social media, and 

permit lists are used to identify areas of potential agricultural activity. A survey is then conducted to assess 

whether each of the operations in a sample of the identified areas is non-agricultural, agricultural but not a farm, 

or an urban farm. In this paper, the study is outlined; the results from satellite imagery are discussed; and the 

results of the pilot study are presented. Challenges and lessons learned will be highlighted. The potential of 

scaling the pilot study up for national implementation is discussed.

A Meta-analysis of Within-Household Respondent Selection Methods 

Ting Yan (Westat), Roger Tourangeau (Westat), and Rose McAloon (Joint Program in Survey Methodology, 

University of Maryland) 

Random selection of a respondent within a sampled household is essential for maintaining the probability nature 

of the resulted sample and for making reference from the sample to the general population. The latest review on 

within-household selection methods lists at least 15 selection methods that have been used to generate a sample 

of persons from a probability sample of households. However, there is no clear guideline in the literature on how 

to select from the existing pool of respondent selection methods that is driven by empirical evidence. This paper 

conducts a meta-analysis on empirical research that experimentally compares two or more respondent selection 

methods. The goal of the meta-analysis is to generate quantitative effect sizes indicating the size of the impact of 

different selection methods in terms of cooperation rates and sample compositions. These meta-analytic results 

will be of practical significance to survey organizations and practitioners; the effect sizes will be used as 

systematic and accessible evidence to guide survey practitioners on their choice of respondent selection 

methods. 
Concurrent Session H Abstracts

80

CONCURRENT SESSION H-4

HEALTH INSURANCE ESTIMATES: ASSESSING THE IMPACT OF A 

REDESIGN

How Did the Method Change in the CPS ASEC Affect Health Insurance Estimates?

Carla Medalia (U.S. Census Bureau) and Brett O’Hara (U.S. Census Bureau)

In 2014, the redesigned questions replaced the traditional health insurance questions in the Current Population 

Survey Annual Social and Economic Supplement (CPS ASEC). In previous research, using survey and modeled 

data, the effect of the methods change in 2013 was estimated to be a 1.7 percentage point decrease in the 

uninsured rate (O’Hara and Medalia forthcoming). In order to better understand the implications of the methods 

change, Congress requested that the Census Bureau implement the traditional CPS ASEC questions as an 

additional supplement in 2015. This request resulted in two versions of the survey: the 2015 CPS ASEC, which 

included the redesigned health insurance and income questions, and the 2015 Parallel Supplement, which 

included the traditional questions on income and health insurance coverage. To examine the effect of changing 

the method of collecting health insurance data on estimates, this paper compares the 2015 Parallel Supplement 

to the 2015 CPS ASEC.

Estimating a Time-Trend of the Uninsured Rate for the CPS Parallel ASEC

Brett O’Hara (U.S. Census Bureau) and Carla Medalia (U.S. Census Bureau)

In 2014, redesigned questions replaced the traditional health insurance series in the CPS ASEC. Previous 

research modeling the change produced a reasonable estimate of the 2013 uninsured rate for the traditional 

questions using data from the American Community Survey and previous years of the Annual Social and 

Economic Supplement of the Current Population Survey (CPS ASEC) (O’Hara and Medalia forthcoming). In this 

paper, we take two approaches to produce model-based estimates of health insurance coverage. First, we take a 

backwards forecasting approach to estimate the uninsured rates in 2013 and 2012 using the 2015 CPS Parallel 

ASEC, which reused the traditional questions. We then use the forward forecasting approach used in O’Hara and 

Medalia 2015 to predict the uninsured rate in 2012. Finally, we make two comparisons of the model-based 

estimates: we compare the 2012 model-based uninsured rate versus a 2012 survey-based uninsured rate, and 

we compare the forwards and backwards model-based 2013 uninsured rates. 

Reference

O’Hara, B., and C. Medalia. "Health Insurance in the 2014 CPS ASEC: Examining the Effect of the Redesign." 

U.S. Census Bureau Working Papers, forthcoming.

Comparing 2014 SIPP and 2014 CPS ASEC Employer-Sponsored Insurance Offer and 

Take-up Rates 

Joelle Abramowitz (U.S. Census Bureau)

This paper compares offer and take-up rates of employer-sponsored insurance in the 2014 Current Population 

Survey Annual Social and Economic Supplement (CPS ASEC) to those found in Wave 1 of the 2014 Survey of 

Income and Program Participation (SIPP) Panel. We examine estimated offer and take-up rates as well response 

rates across the surveys. The analysis also compares the reasons reported for why individuals were ineligible to 

enroll in employer-sponsored insurance or were eligible but chose not to enroll. We compare outcomes for the 

full sample and by demographic characteristics, including educational attainment, sex, race/ethnicity, and age, as 

well as by employer characteristics, including firm size.

The CPS ASEC was recently redesigned to address concerns over validity of health insurance estimates. Among 

the changes to the survey content, the redesign introduced questions about employer-sponsored health insurance 

benefits and reasons for nonparticipation in an employment-based health insurance plan. These questions are 

the focus of this paper and are of particular interest given changes introduced through the 2010 Patient Protection 

and Affordable Care Act (ACA), which may lead to changes in the rates of offer and take-up. 
Concurrent Session H Abstracts

81

Comparing estimates of offer and take-up across surveys is important for understanding robustness of and 

discrepancies in alternate estimates and identifying any factors contributing to these discrepancies. In particular, 

key models predicting the impact of the ACA have used the 2005 and 2008 SIPP employer-provided health 

benefits topical modules (Congressional Budget Office Health Insurance Simulation Model, RAND COMPARE 

microsimulation module) and the 2005 CPS Contingent Work and Alternative Employment Supplement (Urban 

Institute Health Insurance Policy Simulation Model) for the take-up portion of their simulations. Understanding 

differences in estimates of offer and take-up rates in the CPS ASEC and SIPP could have important implications 

for using them to model and evaluate the effects of the ACA. 

Medicare Coverage and Reporting of the Elderly Population: A Comparison of CPS 

and Administrative Records

Renuka Bhaskar (U.S. Census Bureau), James Noon (U.S. Census Bureau), Sonya Rastogi (U.S. Census 

Bureau), Brett O’Hara (U.S. Census Bureau), and Victoria Velkoff (U.S. Census Bureau)

Medicare coverage of the elderly population in the United States is widely recognized as being nearly universal. 

Recent statistics from the Current Population Survey Annual Social and Economic Supplement (CPS ASEC) 

indicate that 93.1 percent of individuals ages 65 and older were covered by Medicare in 2013. Those without 

Medicare include those who are not eligible for the public health program, though the CPS ASEC estimate may 

also be impacted by misreporting. Using linked data from the CPS ASEC and Medicare Enrollment Database, we 

estimate the extent to which individuals misreport their Medicare coverage - including those who report having 

Medicare but are not enrolled (false positives) and those that do not report having Medicare but are enrolled (false 

negatives). We use regression analyses to evaluate factors associated with both types of misreporting including 

socioeconomic, demographic, and household characteristics. We then provide national and state-level estimates 

of the implied Medicare-covered, insured, and uninsured elderly population, taking into account misreporting in 

the CPS ASEC. Finally, we evaluate the characteristics of those who are not covered by Medicare and examine 

separately those who are uninsured and those who have only private insurance, Medicaid, or SSI. Our results will 

be useful to researchers studying insurance coverage and reporting and to policy makers aimed at improving 

health insurance coverage for the elderly population.
Concurrent Session I Abstracts

82

CONCURRENT SESSION I-1

EXPERIENCES AND EVALUATIONS IN USING RECORD LINKAGE TO 

IMPROVE THE QUALITY OF SURVEY PRODUCTS AND DATA

Simulated versus Actual SNAP Unit Composition in Survey Households in Two 

States 

Karen Cunnyngham (Mathematica Policy Research), John L. Czajka (Mathematica Policy Research), and Randy 

Rosso (Mathematica Policy Research)

National survey data are used to simulate Supplemental Nutrition Assistance Program (SNAP) eligibility in order 

to assess SNAP performance and to estimate the effects of proposed changes to the program on SNAP eligibility. 

An important part of simulating SNAP eligibility is forming potential SNAP filing units within survey households. A 

SNAP unit may contain only a subset of household members, and a household may contain more than one SNAP 

unit. Under a project cosponsored by the Census Bureau’s Center for Administrative Records Research and 

Applications (CARRA) and the Economic Research Service (ERS) of the U.S. Department of Agriculture, SNAP 

administrative data from two states, New York and Colorado, were linked to data from three surveys, the 

American Community Survey (ACS), Current Population Survey Annual Social and Economic Supplement (CPS 

ASEC), and Survey of Income and Program Participation (SIPP). SNAP unit membership as recorded in the 

administrative data was compared to simulated unit membership (ACS and CPS ASEC) and reported unit 

membership (SIPP). The analysis sample for each survey consisted of all households in which one or more 

members matched a SNAP administrative record. Unlike previous studies of linked survey and SNAP 

administrative records, this analysis took account of unmatched as well as matched SNAP records.

This paper describes how the linked files were constructed and presents key findings. For example, the analysis 

of ACS data linked to New York SNAP administrative data showed that in 50 percent of survey households, (1) all 

members of both the ACS survey household and the New York administrative unit were matched to the other 

dataset and (2) the numbers of simulated and administrative SNAP units were the same. The vast majority of 

these households contained only one SNAP unit. In addition, between 44 and 47 percent of the households 

receiving SNAP benefits included nonparticipants.

Program Confusion in the 2014 SIPP: Using Administrative Records to Correct 

False Positive SSI Reports

Katherine Giefer (U.S. Census Bureau), Abby Williams (U.S. Census Bureau), Gary Benedetto (U.S. Census 

Bureau), and Joanna Motro (U.S. Census Bureau)

The 2014 Survey of Income and Program Participation (SIPP) asks about two programs administered by the 

Social Security Administration (SSA): Supplemental Security Income (SSI) and Social Security (OASDI or OldAge, Survivors, and Disability Insurance). While the eligibility requirements for SSI and Social Security differ, both 

programs provide benefits to individuals who are blind, disabled, or aged. This similarity combined with the 

presence of false positive rates of SSI receipt in the 2008 SIPP led to changes aimed at clarifying which program 

was being asked about in the SSI section of the 2014 instrument. Despite these modifications, the rate of false 

positive SSI reports in the 2014 SIPP rose to 49% compared to 23% in the 2008 Panel. 

Matching 2014 SIPP Wave 1 data with administrative records from SSA revealed that many of the SSI false 

positives fell under a specific type of false positive reporting that can be described as “program confusion”. In 

these cases, respondents were either reporting both SSI and Social Security (double reporting) or only reporting 

SSI (program swapping), when administrative records indicated they were only receiving Social Security. 
Concurrent Session I Abstracts

83

The high rate of false positive reporting affected the quality of the Wave 1 SSI data to the extent that an edit 

correction was necessary to avoid suppressing the data on the public use file. Using administrative records, we 

were able to identify respondents who made false positive SSI reports because of program confusion and correct 

their survey reports, which will allow the data to be released for public use. This paper discusses the issue of 

program confusion and the method used to “correct” the data, as well as the distribution of SSI receipt in Wave 1 

before and after the correction. 

Evaluation of Record Linkage for SEER breast cancer registries to Oncotype DX 

assays

Michael D. Larsen (George Washington University), Will Howe (IMS), Nicola Schussler (IMS), Benmei Liu 

(National Cancer Institute), Valentina Petkov (National Cancer Institute), and Mandi Yu (National Cancer Institute)

The SEER cancer registries contain information on breast cancer cases in their registry areas. A diagnostic test 

based on the Oncotype DX assay, performed by Genomic Health, Inc. (GHI), is often performed for certain types 

of breast cancers. Record linkage using personal identifiable information was conducted to associate Oncotype 

DC assay results with SEER cancer registry information. The software Link Plus was used to generate a score 

describing the similarity of records and to identify the apparent best match of SEER cancer registry individuals to 

the GHI database. Clerical review was used to check samples of likely matches, possible matches, and unlikely 

matches. Performance of record linkage procedures, impact of blocking and comparison variables used in 

matching, and the impact of record linkage decision rules on substantive analyses will be reported. It is 

anticipated that this work will lead to improved methods with greater accuracy and reduced cost for linking SEER 

and GHI files in the future. Suggestions will be given for record linkage methods in other scenarios and for 

methods of evaluating record linkage performance.

CONCURRENT SESSION I-2

MOBILE SURVEY DESIGN

Optimizing the Decennial Census for Mobile – A Case Study

Elizabeth Nichols (U.S. Census Bureau), Erica Olmsted Hawala (U.S. Census Bureau), Rachel Horwitz (U.S. 

Census Bureau), and Michael Bentley (U.S. Census Bureau)

The U.S. Census Bureau is committed to offering an Internet response option for the 2020 Census. We expect 

most self-responses to come in online. Current surveys, such as the American Community Survey, have been 

online since 2013. Other decennial census field tests, such as the 2014 Census Test, have used an online 

instrument to collect data. However, as of 2014, the online instruments developed were designed for optimal view 

on a desktop or laptop. Although these surveys could be answered on tablets or smartphones, the screen display 

was very small and required the user to pinch to zoom or other manipulations to be able to clearly read and 

answer the questions. 

As of 2014, mobile-ownership statistics show 64% of adults own a smartphone and 7% of adults are dependent 

upon their smartphone for Internet access. The smartphone-dependent adults are also more likely to be lower 

income, younger, and minority (Pew Internet project, 2014). With the growth of mobile device ownership overall 

and the differences in device-dependent Internet access populations, the Census Bureau realized it must create a 

mobile-optimized design for the 2020 Census. For the 2015 Census Test, the Census Bureau developed a 

mobile-optimized online census, in addition to the traditional desktop/laptop design. A mobile style guide was 

developed for the programmers. In addition to user acceptance testing, usability testing was conducted prior to 

fielding the survey.

This presentation discusses the creation of a mobile-optimized style guide and the best practices which arose 

from the usability testing. Additionally, we compare usage and break-off data from the 2014 and 2015 Census 

Test online instruments. The 2014 instrument was not optimized and the 2015 instrument was. The goal of the 

comparison was to determine whether optimizing for mobile improved data quality and the user experience. 
Concurrent Session I Abstracts

84

Testing the Feasibility of Collecting Expenditure Data via Individual Mobile and 

Web Diaries 

Ian Elkin (Bureau of Labor Statistics), Brett McBride (Bureau of Labor Statistics), and Lucilla Tan (Bureau of Labor 

Statistics) 

The current Consumer Expenditure Diary Survey (CED) at the Bureau of Labor Statistics (BLS) uses a penciland-paper instrument (PAPI) to collect expenditure information from respondents. PAPI diary data collection has 

a number of inherent drawbacks, such as limiting entry to a single individual in a single location, and requiring that 

the main diary keeper carry the diary with them throughout the day in order to support in-the-moment diary 

keeping. Additionally, having one person in the household maintain the diary for all household members has 

drawbacks. The CE program leveraged new data collection technologies to present households with options for 

contemporaneous diary maintenance that included a mobile application for Smartphones as well as a desktopbased web diary. In addition to testing the use of an online mode to collect diary data, the CE program continued 

past research exploring the feasibility and effectiveness of providing a diary to all household members over a 

certain age. By providing individual diaries, the CE program hopes to address underreporting caused by proxy 

reporting, by allowing each household member to report for themselves. Beginning in August 2014, CE fielded a 

5-month test (n=600) to determine the feasibility of collecting expenditure data via mobile and web diaries 

provided to all eligible members of a household. In this presentation, we report on preliminary findings regarding 

sample performance, sample composition, diary maintenance patterns, data quality impact, and other associated 

operational issues as well as provide recommendations for implementation and future research. 

Gridlock: Improving Questionnaire Design for Mobile Devices

Randall K. Thomas (GfK Custom Research) and Frances M. Barlas (GfK Custom Research) 

Grid formats are commonly used for self-administered surveys, found frequently in both mail and online modes. 

They are the most efficient means of assessing a series of elements that can share a similar response scale. 

However, and partly because of their efficiency, many grids have become lengthy in terms of the numbers of 

responses used and number of elements presented. This explosion in grid length has been especially seen in 

online surveys. While grids with many columns and many rows may not be ideal for respondents on desktops or 

laptops, for the 20% to 30% of participants attempting to complete on a mobile device, these grids can become 

impossible to complete. In a series of 3 experiments, we examined how the number of responses presented 

could affect both the validity and differentiation ability of the scales. In the first experiment, we randomly assigned 

1,734 respondents to 3, 4, or 5 responses, comparing bipolar with unipolar variations across different types of 

content (behavioral and attitudinal). In the second experiment, we had 4,998 respondents randomly assigned to 

receive 3, 4, or 5 responses with a unipolar variation. In our third experiment, we compared several grid 

variations with fewer or more response options. This third experiment had 4,555 respondents who had been 

randomly assigned to complete the survey on a desktop/laptop, a tablet, or a smartphone. Across all 

experiments, scales with fewer response categories were found to be as highly differentiating as scales with more 

categories. In addition, in comparing the validity of the response formats, we found that scales with fewer 

response categories were as valid as scales with more response categories. This suggests that response scales 

with fewer categories are a viable and important consideration in designing online surveys to accommodate the 

large and growing share of respondents on mobile devices. 
Concurrent Session I Abstracts

85

Measurement on the Move: Designing Questionnaires for Mobile First

Frances M. Barlas (GfK Custom Research), Randall K. Thomas (GfK Custom Research), and Patricia Graham 

(GfK Custom Research)

The proportion of participants who take web-based surveys on mobile devices is increasing and may improve 

coverage and reduce non-response. We first review a number of studies we have done with both probability and 

non-probability samples to detail the demographic differences of participants who use mobile devices for online 

surveys as well as attitudinal differences after controlling for normal demographic factors. We next review studies

we have conducted to empirically determine best practices for mobile questionnaire design. A necessary, but not 

sufficient, method to adapt web surveys to mobile devices is to implement responsive design whereby the survey 

display is adjusted based on the screen size and orientation. Beyond this, evidence-based recommendations are 

needed to better inform design decisions around survey question and response format. As such, we utilize data 

from two different types of studies to inform our recommendations. The first was “accidental mobile” studies 

where respondents selected their survey completion device and the second was a “purposefully mobile” study 

where respondents were randomly assigned to a completion device. We examine how device affects responses 

to various question formats, including single and multiple choice, horizontal and vertical display and scrolling, 

open-ended numeric, sliders, and drop down formats over a variety of survey topics. We will discuss observed 

differences in responses that appear to occur because of device used and not due to sample differences as well 

as differences that appear to be due to differences in respondents across devices. We will summarize the 

empirical evidence for changing the very nature of questionnaire design to prepare simpler, easier to complete 

surveys with a mobile-first mentality. We will show that such an approach improves the survey experience and 

data collected with surveys completed on mobile devices, and also benefits all respondents including those 

completing studies on desktop and laptop devices.

A Summary of Issues Associated With the Modernization of Data Collection

Linda A. Lawson (NASS), Denise A. Abreu (NASS), Claire Boryan (NASS), Michael Gerling (NASS), and Rick 

Hardin (NASS)

The National Agricultural Statistics Service (NASS) is considering the use of a permanent grid frame instead of its 

area frame for the June Agricultural Survey (JAS). It has been proposed that a grid of one-square mile areas be 

used to establish the frame for each state. The grid areas would be stratified by agricultural intensity and then a 

stratified random sample drawn. As with the current area frame, every year 20 percent of the sampled segments 

would be rotated out and a new rotation of segments introduced. A challenge associated with this proposed 

approach is that only a portion of an agricultural field may lie within the selected grid whereas currently sampled 

areas have boundaries that follow roads or other physical features. 

Because of the presence of partial fields in a sample unit, the use of mobile mapping technology may be critical 

for proper identification of the area to be sampled. Currently, enumerators identify fields within a sample unit on 

an aerial map. The mobile mapping also allows electronic data entry. To test the concept of a grid segment in 

conjunction with the mobile mapping instrument enumerators in North Carolina, Pennsylvania and South Dakota 

visited with farm operators during summer 2014. A sample of grid segments was selected in each state. For 

each sampled segment, the enumerators completed a segment evaluation form that gathered information on 

issues associated with the grid concept, use of a mobile mapping instrument, connectivity and glare associated 

with the iPad. This report documents the results of both the mobile mapping as well as the permanent grid 

concept. 
Concurrent Session I Abstracts

86

CONCURRENT SESSION I-3

ISSUES WITH DISCLOSURE AND DATA CONFIDENTIALITY

Formal Privacy Protection for Data Products Combining Individual and Employer 

Frames

Ashwin Machanavajjhala (Duke University), Samuel Haney (Duke University), Matthew Graham (U.S. Census 

Bureau), Mark Kutzbach (U.S. Census Bureau), and John Abowd (U.S. Census Bureau, Cornell University)

Published tabular summaries of linked employer-employee data usually use a job frame (statutory employer 

linked to a specific employee) but include characteristics of both the individual (employee) and workplace 

(employer establishment). Formal privacy protection of these characteristics requires defining the sensitivity of the 

published statistic to the inclusion of an individual or to the variation in the size of a single workplace 

(establishment). We propose a model that simultaneously protects individuals and establishments using 

parameters that control the conventional differential privacy for individuals and a generalization that provides a 

similar privacy guarantee for the employment magnitudes associated with an employer establishment. We 

implement our model using three alternative noise distributions. We present results for cross-sectional 

employment summaries for combinations of employer industry, geography, and ownership; employee sex, age, 

race, ethnicity, and education. The system is illustrated using the LEHD Origin-Destination Employment Statistics 

(LODES) database displayed in the U.S. Census Bureau's OnTheMap application. 

Methods of Selecting Variables in Data Disclosure

Charles C Lin (National Center of Veterans Analysis and Statistics, Department of Veterans Affairs), Hwai-Tai 

Lam (Office of Actuary, Department of Veterans Affairs), Lijia Guo (Office of Actuary, Department of Veterans 

Affairs), and Tom Garin (National Center of Veterans Analysis and Statistics, Department of Veterans Affairs)

Data released for public use needs to be protected for personal identified information such as name and Social 

Security Number and any unusual pattern in data which could have a high risk of re-identification. All privacy 

protected information should be stripped from released variables in data according to a set of disclosure 

guidelines. Excessive restrictions of released data elements, however, could render the data useless. The task of 

proper data disclosure with minimum risk of re-identification can be overwhelming when the number of variables 

becomes large. The brute force approach of examining all becomes infeasible as the number of possible 

combinations of variables increases exponentially. To prepare a data for public use, this study proposes 

systematic methods examining each variable with respect to both risks and benefits. These methods compare a 

subset of variables to another and traverse back and forth from subsets to subsets in a step-by-step manner until 

a stopping criterion is met. The final subsets of variables will provide reasonable choices of variables for public 

release. The methodology with its implementation in SAS program will be discussed in details and demonstrated 

with examples.

Considering the Usefulness of Standard FOIA Provisions in Interagency 

Agreements

Jennifer Goode (U.S. Census Bureau), Michael Toland (U.S. Census Bureau), Hampton Wilson (U.S. Census 

Bureau), and Rana Wahdan (U.S. Census Bureau)

The Freedom of Information Act (FOIA), 5 U.S.C. §552, is a disclosure statute which requires federal agencies to 

disclose information unless such information is covered by one or more of the FOIA’s exemptions. Each agency 

is responsible for responding to requests related to their own data. However, to the extent that the agencies have 

jointly or collaboratively created or generated information, documents, or data under an Interagency Agreement 

(IAA), each agency has an interest in the information disclosed to a requester under the FOIA. This paper 

discusses the usefulness of the inclusion of standard FOIA language in applicable Interagency Agreements. 
Concurrent Session I Abstracts

87

CONCURRENT SESSION I-4

MODELING PRELIMINARY ESTIMATES

Early estimates of annual manufacturing industry output

Peter B. Meyer (Bureau of Labor Statistics), Jason McClellan (Bureau of Labor Statistics), Jennifer Price (Bureau 

of Labor Statistics), Sam Rowe (Bureau of Labor Statistics), and others in BLS-OPT

BLS estimates labor productivity of manufacturing industries using measures of output from the Annual Survey of 

Manufactures or the Economic Census. In this paper we use other data sources to make estimates of those 

output measures before they are released so that it is possible to construct preliminary productivity statistics for 

many manufacturing industries about four months after the end of the reference year.

The output measure used for these preliminary productivity statistics is called the “value of production.” This is a 

sectoral output measure, which is different from either gross output or value-added output, and excludes sales 

from one establishment in the industry to another. As computed from the underlying data it incorporates several 

elements for each industry: (1) the overall value of shipments, (2) inventory levels at the beginning and end of the 

year, (3) sales to other establishments in the same industry, and (4) resales, which are specially excluded from 

industry output.

These four components are predicted by the output and inventory measures in the M3 survey, the Federal 

Reserve’s Industrial Production indexes, imports, exports, employment, wages paid, rail carloads shipped, net 

sales in the Census’s Quarterly Financial Reports, and price indexes.

We examine the quality of the predictions relative to estimates previously released by the labor productivity 

program, and discuss the likely criteria for releasing preliminary estimates using this method in the future. 

Early Monthly Estimates for the Services Sector

Pieter Vlag (Statistics Netherlands), Ton de Waal (Statistics Netherlands), and Cagan Kaya (Turkish National 

Statistical Institute) 

The services sector contributes about 50 % to the Dutch economy. Statistics Netherlands uses a combination of 1) 

VAT (Value Added Tax) data and 2) a survey for the largest enterprises to produce a quarterly turnover estimate 

for the services. The quality of these quarterly estimates is good and can be used as a reference.

The demand for early available monthly information about the services sector is large Therefore, Statistics 

Netherlands intends to introduce a monthly survey for the services. However, the decision process is long and 

timeliness issues remain: a turnover estimate based on a regular survey cannot be published earlier than 40 – 45 

days after the month.

Therefore, alternatives to develop an early indicator for the services were investigated. Restrictions for these 

alternatives were: low production costs, based on existing data, based on existing methodologies. 

Three alternatives have been explored:

• a simple econometric model using information outside the services sector (order position, consumer 

confidence, bankruptcies, producer confidence).

• quantification of the results of the monthly Business Survey Netherlands (BSN) for enterprises active in the 

services sector. The BSN provides information on the opinions of Dutch producers about performance over 

the last three months and expectations over the next three months.

• using data from the 1600 traffic (induction) loops on high-ways in the Netherlands. These traffic loop data 

provide real-time information (per minute) about traffic intensities of small cars, large cars and trucks. Traffic 

intensities near ports, airports, industrial areas and border crossings related to growth of the transportation 

sector (= part of the services) and economic activity in general.
Concurrent Session I Abstracts

88

Time-series analyses have been used to compare the results of these early indicators with the quarterly estimates 

based on surveys. During the presentation the advantages and disadvantages of these three alternatives will be 

discussed, too. 

Medicaid and CHIP Data Methodology for SAHIE Models

David Powers (U.S. Census Bureau), Lauren Bowers (U.S. Census Bureau), Wesley Basel (U.S. Census 

Bureau), and Samuel Szelepka (U.S. Census Bureau)

This work seeks to identify and document the best available Medicaid and CHIP data sources, data 

transformations, and methodology for use in the U.S. Census Bureau’s Small Area Health Insurance Estimates 

(SAHIE) program. SAHIE is the only source of single-year health insurance coverage estimates for all counties in 

the United States. SAHIE are model-based estimates that enhance the American Community Survey (ACS) data 

by combining them with informative administrative records data for a given year. With state Medicaid expansions 

taking place in many states under the Affordable Care Act (ACA), the time lag of the Medicaid data historically 

used in the SAHIE models of ACS health insurance coverage may limit their usefulness going forward. In 

response, the SAHIE program currently seeks to reduce the Medicaid time lag by benchmarking its detailed 

Medicaid tallies (by age, sex, basis, county) from the Medicaid Statistical Information System (MSIS) with timely 

state-level Medicaid growth figures or numerical benchmarks from more rapidly published data series from 

Centers for Medicare and Medicaid Services (CMS) and/or Kaiser Family Foundation (KFF). In this work, we lay 

out the conceptual differences between various Medicaid and Children’s Health Insurance Program (CHIP) data 

sources, citing key assumptions and filters, and we study numerical differences between data series for like 

concepts and time periods. We describe a few candidate benchmarking approaches for combining the detailed 

MSIS and timely state-level data series, and we study the predictive accuracy of each approach by examining 

historical simulations (using old data to predict later-published actual data), discussing apparent advantages and 

disadvantages to each option. Finally, we compare summary results from provisional SAHIE models that utilize 

the different versions of the Medicaid and CHIP data and benchmarking approaches.
Concurrent Session J Abstracts

89

CONCURRENT SESSION J-1

ADAPTIVE DESIGN II

Managing Survey Costs: Using Mobile Technology Coupled with Performance 

Dashboards to Improve Field Operations

Brad Edwards (Westat), Richard Dulaney (Westat), Abie Reifer (Westat), and Tamara Bruce (Westat)

The cost of traditional household surveys continues to rise and challengers increasingly question their value in the 

face of nonprobability studies and big data. In face to face surveys, interviewer labor and expenses drive the 

project budget, yet little is known about how interviewers spend their time when they are not conducting 

interviews. Although CARI can provide some insight into interviewer behavior during interviews, it often reveals 

how poorly many interviewers ask questions, and little of their behavior when not conducting interviews (how they 

spend the majority of their time) has been observable. 

However, technology exists today that can remedy that. Smart phones can tell us where interviewers are and 

what they are doing. Speech to text, smart small screen design, and internet connectivity enable them to record 

and transmit outcomes of contact attempts on the doorstep. With text and instant messaging the supervisor and 

interviewer can remain in almost constant contact. Performance dashboards can give the supervisor at-a-glance 

views of the region’s current activity, with drill down and drill through capability, and with alerts based on statistical 

process control that highlight the problems of greatest concern each day. In the last decade we saw the concept 

of a virtual telephone center take shape, enabling telephone interviewers to work at home, but with the equivalent 

supervision and coaching provided in a brick and mortar center. The capability exists today to create a virtual 

field center, with levels of control that come close to those we have come to expect in CATI call centers. We 

used task analysis and qualitative research with this dispersed work force to inform an attempt to create a usercentered design for a real time field management system, deployed it on a large field survey, and conducted a 

cost-benefit analysis in evaluating its deployment. 

Adaptive and Responsive Survey Designs: A Review and Assessment

Roger Tourangeau (Westat), J. Michael Brick (Westat), Sharon Lohr (Westat), and Jane Li (Westat)

This talk reviews the growing literature on responsive and adaptive designs for surveys. These designs 

encompass a set of methods for managing data collection, including front-loading potentially difficult cases, 

tailoring the data collection strategy to different sample subgroups, prioritizing effort according to estimated 

response propensities, imposing stop rules for ending data collection, monitoring key survey estimates throughout 

the field period, using two- or multi-phase sampling for following up nonrespondents, and calculating indicators of 

non-response bias (such as the R-indicator) other than response rates to monitor and guide field work. We give 

particular attention to efforts to evaluate these strategies experimentally or via simulations. Although the field 

seems to have embraced the new tools for managing data collection, most of the evaluation studies suggest they 

produce marginal reductions in cost and nonresponse bias. Part of the problem is that it is generally easier to 

lower the response propensities of high-propensity cases than to raise the propensities of low-propensity cases. 

Other issues limiting the effectiveness of these designs include weakly predictive auxiliary variables, ineffective 

interventions, and slippage in the field in the implementation of interventions.
Concurrent Session J Abstracts

90

Innovative Uses of Paradata in the Survey of Income and Program Participation

Matthew Marlay (U.S. Census Bureau), Holly Fee (U.S. Census Bureau), and Jason Fields (U.S. Census Bureau)

The Census Bureau is fielding its redesigned Survey of Income and Program Participation (SIPP). As part of the 

redesign, SIPP has access to new paradata sources. This presentation discusses those sources, how paradata 

can inform field operations and monitoring, and some analytical results.

Our paradata come from sources including a field representative (FR) certification test, an FR debriefing 

instrument, keystroke files, our Unified Tracking System (UTS), our Contact History Instrument (CHI), and our 

Computer-Assisted Recorded Interview (CARI) system.

Paradata is useful during data collection to control costs, track interviewing progress, and monitor FR 

performance, among others. We track costs via UTS, which provides information such as hours, miles, and 

contact attempts per case. We can see whether certain cases, FRs, or entire regions are out of compliance. 

UTS lets us track interviewing progress – we know how many cases have been completed, started, and not even 

touched. Finally, we monitor FR performance through UTS and CARI. Our CARI system records interview items 

(with respondent consent), which can then be coded for whether FRs are reading questions as worded, whether 

respondents understand the questions, etc. This tool also allows us to detect possible falsification.

Paradata let us evaluate data quality and survey design. Keystroke files let us construct interviewing timing 

reports, useful for identifying where respondents might get bogged down, or providing evidence for ways to 

shorten the survey. We can use the certification test to determine where our training needs improvement. We 

can use debriefing results to determine where FRs encountered problems and their severity. And we can use 

CHI to see how easy or difficult respondents are to locate; these results can feed into analyses such as whether 

difficult-to-locate respondents produce more incomplete surveys, item nonresponse, overall refusals, etc.

Using Computer-Assisted Recorded Interviewing to Enhance Field Monitoring and 

Improve Data Quality 

Holly Fee (U.S. Census Bureau), Matthew Marlay (U.S. Census Bureau), and Jason Fields (U.S. Census Bureau) 

The reengineered Survey of Income and Program Participation (SIPP) entered production in 2014. A key goal for 

the reengineered SIPP is to improve field monitoring using paradata such as Computer-Assisted Recorded 

Interviewing (CARI). CARI is a Census Bureau technology used to monitor Field Representatives (FRs) by 

listening to recordings of interactions between FRs and respondents. FRs must obtain consent from each 

respondent to record the interview. The goal of CARI is twofold: to ensure the accuracy and quality of data 

collected by identifying opportunities to improve the FR’s performance and to improve the survey’s design by 

identifying difficult or problematic questions

The Quality Assurance (QA) Component in the CARI System ensures that the data collected have minimal errors 

and monitors FRs’ behavior. The QA program identifies falsification problems and errors introduced by the FRs 

during the interview. Falsification occurs when data collected did not come from the respondent or if an answer is 

different from what the respondent reported. Errors introduced by FRs occur when questions are not read 

consistently across FRs. During QA, for each completed interview, two coders review a subset of questions for 

up to three respondents per household. 

CARI is also useful in evaluating the quality of the survey itself. Approximately 66% of respondents agreed to be 

recorded for Wave 1 of the 2014 SIPP panel. One of our analyses is to examine how this consent rate differed 

along a number of dimensions, including respondent characteristics, household location, and incentive receipt. 

We can also improve data quality by analyzing interview techniques. The current study examines the difference 

in data quality indicators, such as Don’t Know or Refused responses, between CARI and non-CARI interviews by 

estimating a three-level organizational model using multilevel modeling.
Concurrent Session J Abstracts

91

CONCURRENT SESSION J-2

WEALTH DATA AND WEALTH DATA QUALITY

Changes in the Distribution of After-Tax Wealth: Has Income Tax Policy Increased 

Wealth Inequality?

Kevin Moore (Federal Reserve Board) and Adam Looney (Brookings Institution)

A substantial share of the wealth of Americans is held in tax-deferred form such as in retirement accounts or as 

unrealized capital gains. Most data and statistics on assets and wealth is reported on a pre-tax basis, but pre-tax 

values include an implicit tax liability and may not provide as accurate a measure of the financial position or 

material well-being of families. In this paper, we describe the distribution of tax-deferred assets in the SCF from 

1989 to 2013, provide new estimates of the income tax liabilities implicit in those assets, and present new 

statistics on the level and distribution of after-tax net worth. The results of our analysis suggest that, relative to 

published statistics on pre-tax net worth, the distribution of after-tax wealth is slightly less concentrated at each 

point in time and the effectiveness of the income tax system in reducing wealth inequality has decreased during 

the last decade. We find the reduction in the long-term capital gains rate is the primary reason for the muted 

effectiveness of the income tax system in reducing wealth inequality.

Rounding in household financial surveys: the roles of survey design and individual 

characteristics 

Michael Gideon (U.S. Census Bureau), Joanne W. Hsu (Federal Reserve Board of Governors), and Brooke 

Helppie McFall (Institute for Social Research, University of Michigan) 

This paper explores the prevalence of rounding, exact values, and item nonresponse in self-reported wealth data. 

The determinants of rounding behavior and how systematic it is have implications for economic research using 

survey data, particularly on income dynamics and inequality, as well as the measurement of key economic 

relationships, like savings rates. 

Three patterns emerge from our preliminary analyses of mortgage balances in HRS, SCF and CogEcon. First, 

respondents in the SCF report substantially more precise mortgage balances, and this is true across the 

distribution of their values. Second, there appears to be increasing rounding over the distribution of their values. 

However, this appears to be non-monotonic, in that rounding is larger near “focal points” such as $100,000. 

Therefore, exact value and range data should be combined with care, especially in the analysis of changes in 

values across waves in panel data. Third, we analyze the predictors of rounding, particularly the role of education, 

income, cognition/financial literacy, time to respond (using paradata), survey mode, and the characteristics of 

range alternatives offered. Respondents who report using records while completing the questionnaire provide 

more precise balances (less rounding, more significant figures). People in the top quintile of income report less 

precise mortgage balances (fewer significant figures), yet education itself does not appear to impact rounding. 

Comparing respondents who consulted financial records to answer the survey with respondents who did not, 

people using financial records were more likely to report exact dollars. Asking people to use their records could 

potentially lead to increased precision, especially for those who might otherwise round heavily, but could also 

introduce bias if take-up varies systematically by income or some other characteristic.
Concurrent Session J Abstracts

92

Reassessing Wealth Data Quality in the Survey of Income and Program 

Participation

Jonathan Eggleston (U.S. Census Bureau) and Mark A. Klee (U.S. Census Bureau)

The Survey of Income and Program Participation (SIPP) and the Survey of Consumer Finances (SCF) are two 

principal sources of wealth data for the U.S. population. The Social Security Administration sponsored 

Mathematica Policy Research to write a 2003 report that identified considerable discrepancies in wealth estimates 

across these surveys. While one might expect SIPP and SCF to deliver different estimates for a variety of 

reasons, the magnitude of differences in levels and trends across surveys fostered doubts about SIPP wealth 

data. To address these concerns, SIPP implemented various strategies that the report recommended to close the 

gaps between wealth estimates. We use 2008 SIPP and 2010 SCF data to assess for the first time the impact of 

these changes on comparisons of wealth estimates. We also employ 2014 SIPP and 2013 SCF data to 

investigate whether the re-engineering of SIPP has any implications for wealth estimate comparisons

Measuring the Wealth of Hard-to-Find U.S. Minorities: Sampling, Recruiting and 

Weighting Issues

Thomas M. Guterbock (University of Virginia), James M. Ellis (University of Virginia), Deborah L. Rexrode 

(University of Virginia), Casey M. Eggleston (U.S. Census Bureau), Darrick Hamilton (The New School), and 

William A. Darity, Jr. (Duke University).

The first phase of the National Asset Scorecard and Communities of Color project (NASCC) was a telephone 

survey conducted by the University of Virginia Center for Survey Research designed to better understand the 

asset and debt positions of various ethnic and racial groups whose wealth status is often inadequately measured. 

Nearly 3,000 interviews were collected in Boston, Los Angeles, Miami, Tulsa and Washington. Challenges 

included low incidence target populations tending to be wireless only, an intrusive survey instrument over 40 

minutes long, and 85 sample quota cells. 

The research goals dictated a disproportionate sample focused on each city’s unique demographic profile (e.g., 

Native Americans figured prominently in Tulsa, Haitians in Miami, etc.). We used 2010 Census and ACS public 

use data at the levels of tract and ZIP Code Tabulation Area (ZCTA) to inform sampling. Lorenz curves showing 

geographic concentration of groups helped us identify areas for targeted sampling. We used listed landline 

sample by census tract, random-digit dial wireless sampling by ZCTA, and listed landline sample from ethnic 

surname lists. Most groups were targeted via several sample types.

Initial attempts at geographically targeted samples produced low yields of the targeted groups. In general, 

directory-listed phone samples could be targeted geographically but tended to miss the groups of interest. Cell 

phone samples tended to reach the targeted groups, but lacked the geographic specificity we required. Targeting 

cell phone samples by ZCTA did not improve sampling as much as expected.

Overall response rates varied across samples but were in low single digits. Nevertheless, the completed 

interviews represented a usable cross-section of the populations of interest. We applied weights to each sample 

to match ACS estimates on household demographics. We examine the resulting design effects to assess the 

precision of the final samples as tools for examining wealth disparities between the groups.
Concurrent Session J Abstracts

93

CONCURRENT SESSION J-3

REMOTE ACCESS AND ANALYSIS TOOLS FOR WORKING WITH 

GOVERNMENT DATA

Education Data in the Cloud

Shelley Burns (National Center for Education Statistics), Brian Taylor (National Center for Education Statistics), 

Marilyn Seastrom (National Center for Education Statistics), and Ted Socha (National Center for Education 

Statistics)

The National Center for Education Statistics (NCES) provides multiple data access tools that range from simple 

information lookup (such as the College Navigator), to table generation/data analysis tools that allow for more 

sophisticated statistical analyses of NCES data, such as the NAEP Data Explorer (NDE) and International Data 

Explorer (IDE) for NCES assessment data. NCES has also developed PowerStats, a data analysis tool that 

provides public access to firewall-protected restricted-use NCES datasets, while building in security/suppression 

safeguards to prevent disclosure risk. PowerStats users can create tables and regression analyses with a dragand-drop interface, and can receive their data in Excel, PDF, and other formats. PowerStats also provides users 

the ability to search for variables, to save recoded variables for future use, and to build an online library of their 

work and share their work with other users. For the more in-depth data analyses (such as the creation of 

composite variables), NCES also provides restricted-use data to researchers through a licensing program that 

allows researchers to use NCES Restricted data at their institutions under strict security guidelines and 

inspection. Future directions for all NCES data access tools and the potential migration of the license program to 

remote access in a secure cloud environment will be presented. In-house testing has been proposed for allowing 

staff to access restricted-use data in a secure cloud environment. The benefits and obstacles to providing access 

to our restricted-use data under the evolving technical environment will be elaborated. For example, among the 

options being considered may be the use of remote, secure cloud-based desktops that would allow for all analysis 

of restricted-use data in the cloud and not on a researcher’s own equipment.

Research Using Remote Access or Remote Execution Systems

Peter Meyer (National Center for Health Statistics), Vijay Gambhir (National Center for Health Statistics), Corey 

Decker (National Center for Health Statistics), and James Craver (National Center for Health Statistics)

The Department of Health and Human Services (HHS) has been developing remote access, remote execution,

and Internet accessible systems that allow researchers access to health data. Some of these systems allow 

access to restricted use data that would not usually be made available but has high analytic utility. Other systems 

simply facilitate the research by providing access to public use data but not requiring statistical software. 

Hundreds of researchers from other Federal agencies, academic institutions, and non-profits access data through 

these systems annually. This presentation will provide an overview of how HHS has developed these systems 

and their analytic limitations. Also, we will discuss plans for developing the next generation of data access 

products and how they might assist the research community in the future.

SAMHSA Data Portal

Brooklyn Lupari (Substance Abuse and Mental Health Services Administration)

The Substance Abuse and Mental Health Services Administration (SAMHSA) hosts a Data Archive where all of 

the data collected by the Center for Behavioral Health Statistics and Quality (CBHSQ) is disseminated for public 

use. Public-use files are made readily available, along with access to a restricted-data analysis system (R-DAS) 

which allows for more in-depth online analysis on restricted mental health and substance use data.

The Data Portal has been developed by SAMHSA as a way for approved researchers to safely access microlevel, confidential data for research purposes. The system has been developed completely unique to the 

requirements and needs of SAMHSA, and has functionality and safeguards built in to ensure data security, while 

allowing for maximum access to the rich data CBHSQ collects.
Concurrent Session J Abstracts

94

The development and implementation of the Data Portal has brought about the opportunity for researchers to gain 

access to confidential data in a safe, manageable way - with access remotely from their own locations. It has also 

greatly increased the ways that CBHSQ/SAMHSA data can be used and analyzed, ensuring that the data is being 

utilized to its full capacity.

More information can be found at: http://www.icpsr.umich.edu/icpsrweb/content/SAMHDA/dataportal.html

Developing and Testing the Microdata Analysis System at the US Census Bureau

Bryan Schar (US Census Bureau), Michael Freiman (US Census Bureau), and Amy Lauger (US Census Bureau)

Data users in government, private industry, non-profits and academia have substantial demand for data from the 

Census Bureau’s surveys and censuses, such as the American Community Survey (ACS). Hence the Census 

Bureau aims to disseminate data widely and with as much detail as possible while keeping the pledge of 

confidentiality given to all respondents. Although the Census Bureau produces many estimates and tables, along 

with public use microdata for some surveys, existing data products may not meet the needs of some users. This 

presentation describes the development and capabilities of the Microdata Analysis System (MAS), an online 

remote access system being developed by the Census Bureau. The MAS will allow any user to create custom 

tables—and later, other analyses—from the underlying data. In many cases, the amount of detail in geographic 

and other variables available in the MAS will be greater than what is currently possible, with estimates for the 

ACS available down to low levels of geography. Tables and estimates will be produced quickly and interactively 

for the user’s specified geography and analysis variables. The system will also dynamically generate measures of 

variance for all estimates. Data will be protected through dynamic application of disclosure checks in response to 

the user’s query, and since calculations will take place behind two firewalls, the user will not be able to see the 

microdata directly. We describe the capabilities of the system and how we plan to incorporate user feedback in 

improving the system’s functionality. In addition, we discuss the testing performed to ensure the system’s utility 

and the planned disclosure avoidance methodology to protect the data.

CONCURRENT SESSION J-4

JPSM SESSION AT 2015 FCSM CONFERENCE

New Developments at JPSM

Richard Valliant (Joint Program in Survey Methodology) and Frauke Kreuter (Joint Program in Survey 

Methodology)

JPSM has revised its curriculum and plans to launch several new degree and certificate programs in the coming 

year. The curriculum is being broadened beyond just survey methodology to include a new emphasis on analysis 

of big data and data science. The amount of digital data generated as a by-product in society is growing fast, 

e.g., data from satellites, sensors, transactions, administrative processes, social media, and smartphones. These 

data are characterized by high volume, high velocity, and high variety. The hope is to gain insights from this data 

for different areas such as e.g., health and crime prevention, planning of infrastructures, and business decisions. 

In Spring 2015 a new course was taught on Big Data for Federal Agencies that covered sources of these data 

and techniques that can be used for analysis. Another, more advanced big data class will be taught in Fall 2015. 

JPSM now offers certificates in Survey Methodology and Survey Statistics that are taught completely online. A 

third certificate that combines survey methodology and data science will be added and offered in conjunction with 

other departments at Maryland. In Spring 2016 we also plan to begin offering an online 30-credit Masters of 

Professional Studies in which students take core courses in survey methodology but can choose electives in 

several areas, including data science.
Concurrent Session J Abstracts

95

A Comparison of Ex-Ante, Laboratory, and Field Methods for Evaluating Survey 

Questions

Aaron Maitland (Westat)

A diverse range of evaluation methods is available for detecting measurement error in survey questions. Ex-ante 

question evaluation methods are relatively inexpensive, because they do not require data collection from survey 

respondents. Other methods require data collection from respondents either in the laboratory or in the field 

setting. Research has explored how effective some of these methods are at identifying problems with respect to 

one another. However, a weakness of most of these studies is that they do not compare the range of question 

evaluation methods that are currently available to researchers. The purpose of this research is to understand 

how the methods researchers use to evaluate survey questions influence the conclusions they draw about the 

questions. In addition, we seek to identify more effective ways to use the methods together through three studies. 

The first study examines the extent of agreement between ex-ante and laboratory methods in identifying problems 

and compares the methods in how well they predict differences between questions whose validity has been 

estimated in record-check studies. The second study evaluates the extent to which ex-ante and laboratory 

methods predict the performance of questions in the field as measured by indirect assessments of data quality 

such as behavior coding, response latency and item nonresponse. The third study evaluates the extent to which 

ex-ante, laboratory, and field methods predict the reliability of answers to survey questions as measured by 

stability over time. The findings suggest (1) that a multiple method approach to question evaluation is the best 

strategy given differences in the ability to detect different types of problems between the methods and (2) how to 

combine methods more effectively in the future.

The Impact of Rapport on Data Quality in CAPI and Video-mediated Interviews: 

Disclosure of Sensitive Information and Item Nonresponse

Hanyu Sun (Westat)

Rapport is generally described as a sense of connection, mutual comfort and ease of conversational coordination 

during an interaction (Foucault, 2010). Although there is no universally accepted way to measure rapport, the 

general consensus is that it is good for survey interviews and may affect the quality of the responses obtained 

(e.g., Foucault, 2010; Cassell & Miller, 2007). Technological advances in recent years have made videomediated interviews more feasible and affordable; however, little attention has been paid to videoconferencing as 

a potential mode of data collection in survey interviews. In video-mediated interviews, the interviewer and the 

respondent can see and talk to each other via a video window. Although rapport-related verbal behaviors have 

been found to increase the disclosure of moderately sensitive information in face-to-face interactions (e.g., 

Dijkstra, 1987; van der Zouwen et al., 1991), it is unknown if rapport can be established to the same extent in 

video-mediated interviews, leading to similar levels of disclose.

I examined the impact of rapport on data quality in CAPI and video-mediated interviews with a laboratory 

experiment that varies the level of rapport, the mode of data collection, and the version of the questionnaire. Eight 

professional interviewers and 128 respondents participated. I found that (1) there was no significant difference in 

rapport ratings between video-mediated and CAPI interviews; that (2) the impact of rapport on disclosure depends 

on question sensitivity: when questions are moderately or less sensitive, rapport does not seem to affect 

disclosure; whereas when questions are highly sensitive, rapport leads to more rather than less honest 

responses; and that (3) the effects of rapport on item nonresponse depends on question sensitivity: when 

questions are slightly sensitive, rapport motivates respondents to invest more effort to respond, whereas when 

questions are moderately or more sensitive, rapport leads to more item nonresponse. 
Concurrent Session J Abstracts

96

Topics in Model-Assisted Point and Variance Estimation in Clustered Samples

Timothy Kennel (U.S. Census Bureau)

This presentation will address two topics related to model-assisted estimation in clustered samples. The first 

topic explores a new variance estimator that is motivated by sandwiches. New sandwich variance estimators are 

constructed to estimate the variance of calibrated estimators in two-stage sample designs. We find that including 

leverage adjustments into the sandwich variance estimators can improve estimated variances, especially in small 

to moderate sized samples. The second topic proposes a new method to produce calibrated point estimates of 

categorical variables in cluster samples. We highlight several advantages of this new method and provide several 

ways to estimate the variance of the new estimator.

The Use of Responsive Split Questionnaires in a Panel Survey

Jeffrey M. Gonzalez (Bureau of Labor Statistics)

Lengthy surveys may be associated with high respondent burden, low data quality, and high unit nonresponse. To 

address these concerns, survey designers may reduce the length of a survey by eliminating questions from the 

original questionnaire, but this means that some information would never get collected. An alternative may be to 

divide a lengthy questionnaire into subsets of survey items and then administer each subset to distinct 

subsamples of the full sample. This is referred to as a split questionnaire design and has the benefit of collecting 

all of the original survey information. We identify a significant deficiency in the current set of split questionnaire 

methods, namely, the incomplete use of prior information about the sample unit in the design. In most 

contemporary applications of split questionnaires, generally only characteristics of the survey items (e.g., content, 

cognitive burden) are used to inform the design; however, if joint consideration is given to characteristics on the 

survey items as well as the sample unit when designing a split questionnaire, then there may be the potential to 

improve the split questionnaire's utility. In this dissertation, we explore the extent to which, if any, jointly 

considering both types of information at the design stage will yield more efficient split questionnaires. We propose 

various methods for incorporating prior information about the sample unit into the split questionnaire using 

features of responsive design. We highlight how this specific application of a responsive split questionnaire can 

be used to address the concerns present in a major federal survey. Finally, we draw from the literature pertaining 

to survey design, experimental design, and epidemiology to develop and implement a framework for evaluating 

the proposed new elements of our split questionnaire design.

Testing the Relationship between Social Integration and Nonresponse

Ashley Amaya (Joint Program in Survey Methodology)

Differential nonresponse is an enigma researchers have attempted to understand for decades. Several theories 

have been introduced to offer an explanation for differential nonresponse. One such theory is the theory of social 

integration. Individuals who are more integrated (i.e., individuals who participate in a broad range of social 

relationships) are more likely to respond to a survey request because they feel compelled to adhere to their 

group(s) norms or perceive that their participation may result in benefits for their group(s). 

Despite a plethora of literature describing the theory of social integration, it is difficult to test. Integration is a latent 

construct that cannot be directly measured. Therefore, most researchers have either used a single variable (e.g., 

volunteerism) to measure integration or a series of variables that cluster around a single topic (e.g., political 

interest). This technique underestimates the integration-nonresponse relationship as individuals who are 

integrated in ways not accounted for by the chosen variable. This method may also result in spurious inference as 

significant findings may be due to a different latent construct, not integration.

Our goals for this research are to (re)test the social integration theory and more accurately quantify the magnitude 

of the effect integration has on differential nonresponse. While methodologists have traditionally used individual 

variables to measure integration, more holistic measures have been developed in other fields, namely indices 

using multiple measures of social roles (e.g., committee member) and social activities (e.g., attending church). 

We conduct latent class analysis using data available for all sampled individuals to create a social integration 

index. We then place the index in a regression model to predict response and test the theory.
Concurrent Session J Abstracts

97

Sample Design in Three-stage Household Surveys Supplemented by Commercial 

Address Lists

Alena Maze (Joint Program in Survey Methodology)

When using commercial address lists to sample households, investigators spend considerable time and money 

on screening households for eligibility as well as locating certain subpopulations (to achieve target sample sizes). 

Utilizing the demographic information on these lists to target eligible persons and subgroups has the potential to 

lower costs and field workers’ workload. However, this demographic information is often out of date or missing 

altogether. This research investigates how to use this information in screening of housing units for multistage 

samples. Among the goals of this study are

• Estimate the accuracy rates in which commercial lists can correctly identify households with certain 

characteristics (e.g. Hispanics, Non-Hispanic Blacks, Teens (15-19), Females, etc.)

• Determine how to allocate two and three stage samples supplemented with commercial lists accounting 

for: Inaccuracy of listings, costs at each stage of sampling, target sample sizes and coefficients of variation, for 

estimates of demographic subgroups, Stratification of secondary sample units by area characteristics (e.g. density 

of Blacks, Hispanics, Others), Stratification of housing units by commercial list characteristics (e.g. Race/Ethnicity, 

ages of persons). Theoretical variance formulas, including variance components, are derived for estimated totals 

and means with and without calibration to population controls. Several methods of variance component 

estimation are studied, including design based ANOVA, model-based Bayes, anticipated variance (model + 

design)). 
98

Author

Index
FCSM RESEARCH CONFERENCE AUTHOR AND COAUTHOR INDEX

NAME SESSION NAME SESSION

99

John Abowd B-3, I-3

Joelle Abramowitz H-4

Denise Abreu H-3, I-2

J. Trent Alexander E-4

Debby Almonte C-2

Sean Altekruse F-1

Ashley Amaya J-4

Elizabeth Ashack B-4

William Axinn A-2

Frances Barlas I-2

Kelle Barrick D-1

Dorothy Barth G-2

Wesley Basel I-4

Nancy Bates D-3

Laura Bechtel E-1

Jennifer Beck H-2

C. Adam Bee A-4

Gary Benedetto D-1, I-1

Michael Bentley I-2

Anita Bercovitz B-1

Lew Berman B-2

David Berrigan C-2, F-4

Chip Berry G-2

Jonas Bertling C-2

Marcus Berzofsky D-1, E-3, G-2

Renuka Bhaskar H-4

Paul Biemer G-2

William Block D-2

Claire Boryan F-4, H-3, I-2

Joanki Bose E-1

Lauren Bowers I-4

John Boyle A-2, B-2

J. Michael Brick J-1

Amy Brown B-1

J. David Brown H-1

Kristen Gulledge Brown E-1

Tamara Bruce J-1

Susan Brumbaugh A-1

Kyle Brumsted D-2

Shelley Burns J-3

Patrick Campanello G-4

Jennifer Cantrell E-3

Andrew Caporaso C-2

E. Ann Carson A-1, F-1

George Cave F-3

Tiffany Chang H-2

Chou-Lin Chen E-2

Rebecca Chenevert A-4

Raj Chetty Keynote

Jennifer Childs H-1

Kenneth Cho D-2

Yinong Chong D-2

Daniel Chow B-4

Sadeq Chowdhury B-2

Elise Christopher C-4

Stephanie Coffey C-4

Alisha Coleman-Jensen F-2

Sara Cook D-3

Alexia Cooper D-1

Derekh Cornwell C-3

Mick Couper A-2

Melanie Courtright B-2

Mark Crankshaw A-3

James Craver J-3

Kathleen Cronin F-1

Elizabeth Cross D-2

Nathan Cruze C-3

Karen Cunnyngham I-1

Sally Curtin D-2

John Czajka I-1

Yuwen Dai F-3

William Darity, Jr. J-2

Andrew Dau D-1

Michael Davern H-1

Jamie Dayton B-2

Ton de Waal I-4

Corey Decker J-3

Carol DeFrances B-1

Katie Dekker G-4

Matthew DeMichele D-1

Abdellatif Demnati A-3

Janet deMoor B-2

Nancy Devin A-3

Richard Dulaney J-1

Abe Dunn G-3

James Eales D-4

Kristi Eckerson B-1

Jennifer Edgar G-4

Ashley Edwards D-4

Brad Edwards J-1

Susan Edwards E-1

Casey Eggleston J-2

Jonathan Eggleston J-2

Ian Elkin I-2

Chris Ellis A-1

James Ellis J-2
FCSM RESEARCH CONFERENCE AUTHOR AND COAUTHOR INDEX

NAME SESSION NAME SESSION

100

Ned English G-4

Laura Erhard G-4

Tala Fakhouri A-2, B-2

Holly Fee J-1

Rebecca Ferrer C-2

Alison Fields E-4

Jason Fields G-4, J-1

John Finamore C-4

Eric Fink H-2

Jonathan Fisher E-4

Stephanie Fowler C-2

Jean Fox C-2

Justin Francis H-2

Carolina Franco B-3

Peter Frechtel E-1

Naomi Freedner A-2, D-3

Michael Freiman J-3

Scott Fricker C-2

Vijay Gambhir J-3

Tom Garin I-3

Graton Gathright C-3

Michael Gerling H-3, I-2

Robin Ghertner C-3

Michael Gideon J-2

Katherine Giefer I-1

Daniel Gillman B-4

Patricia Goerman F-2

Nathan Goldschlag C-1

Jeffrey Gonzalez D-2, J-4

Jennifer Goode I-3

Arthur Goussanou A-3

Matthew Graham H-1, I-3

Patricia Graham I-2

Andrew Green B-3, C-1

Elizabeth Grieco E-4

David Grusky E-4

Brian Gugerty B-1

Noemi Guindin-Garcia D-4

Lijia Guo I-3

Thomas Guterbock J-2

Heidi Guyer A-2

Elizabeth Hair E-3

Anne Hall G-3

Darrick Hamilton J-2

Samuel Haney I-3

Rick Harden H-3, I-2

Rachel Hartner E-1

Jennifer Hasche G-4

William Hazard D-2

Sarra Hedden E-1

David Heller D-1

Karl Hess G-2

Howard Hogan E-4

Rachel Horwitz D-3, I-2

Carol House E-4

Michael Hout E-4

Will Howe I-1

Joanne Hsu J-2

Todd Hughes E-3, G-2

Ronaldo Iachan B-2, D-3

David Ihrke E-4

Jessica Jagger F-3

Susan Jenkins F-2

Ronald Johnson B-4

Tiffany Julian D-2

Venkata Kambhampaty D-2

Stephen Kaputa C-4

David Kashihara G-4

Cagan Kaya I-4

Jennifer Kelley A-2

Timothy Kennel J-4

Erin Kent B-2

Annice Kim H-3

Mina Kim C-1

Mark Klee A-4, J-2

William Koerber E-4

Karen Kosanovich B-4

Phillip Kott B-3, E-1, G-2

Chris Krebs D-3, E-3, G-2

Frauke Kreuter J-4

Mark Kutzbach C-1, H-1, I-3

Guy Laflamme H-2

Hwai-Tai Lam I-3

Lynn Langton G-2

Luke Larsen E-4

Michael Larsen I-1

Amy Lauger D-2, J-3

Linda Lawson H-3

Albert Lee F-3

Shaleah Levant B-1

Taylor Lewis G-2

Jane Li J-1

Wendy Li B-4

Dan Liao D-1

Charles Lin I-3

Yongheng Lin B-2
FCSM RESEARCH CONFERENCE AUTHOR AND COAUTHOR INDEX

NAME SESSION NAME SESSION

101

Christine Lindquist D-3, E-3, G-2

Jonathan Lisic E-2

Roderick Little B-3

Benmei Liu B-2, I-1

Patsy Lloyd F-4

Sharon Lohr J-1

Brett Loomis H-3

Adam Looney J-2

Thomas Louis B-3

Li Lu D-2

Brooklyn Lupari J-3

Travis Lybbert C-1

Vyacheslav Lyubchich F-3

Ashwin Machanavajihala I-3

Steven Machlin B-2, F-1

Aaron Maitland C-2, J-4

Steve Mance D-4

Matthew Marlay G-4, J-1

Kim Martin D-1

Wendy Martinez D-2

Lowell Mason C-1

Tracy Mattingly E-3

Danni Mayclin D-3

Alena Maze J-4

Rose McAloon H-3

Brett McBride I-2

Jaki McCarthy C-4

Jason McClellan I-4

Brooke Helppie McFall J-2

Kevin McKinney B-3

Carla Medalia H-4

Bruce Meyer A-4, H-1

Peter Meyer I-4, J-3

Darcy Miller D-1

Lisa Mirel F-1

Kristen Monaco H-2

Kevin Moore J-2

Darcy Steeg Morris E-1

Rebecca Morrison C-2

Joanna Motro D-1, I-1

Edward Mulrow G-4

Joe Murphy D-3

Marina Murray G-1

Nicole Nestoriak B-4

Elizabeth Nichols I-2

Eun Young Noh E-2

James Noon H-4

Margaret Noonan A-1

Ann Norris B-4

Stephanie Nutt B-2

Amy O’Hara E-4,G-1, H-1

Brett O-Hara H-4

William O’Hare G-1

Erica Olmsted I-2

Christina Park G-4

Kartik Pashupati B-2

Mihir Patel C-3

Natalie Patten F-3

Benjamin Perry D-2

Kimberly Peterson G-2

Valentina Petkov I-1

Amil Petrin E-1

Michael Planty D-3, E-3, G-2

Elizabeth Poehler G-2

David Powers I-4

Randall Powers D-2

Jennifer Price I-4

Matthew Rabbitt F-2

Dylan Rassier B-4

Sonya Rastogi H-4

Abe Reifer J-1

Benjamin Reist C-4

Jerome Reiter E-1

Deborah Rexrode J-2

Bryan Rhodes H-3

Ashley Richards D-3, G-2

Aruna Rikhi F-3

Lindsey Rittmueller G-3

Kristen Robinson F-2

Nestor Rodriguez D-4

David Roe D-3

Lauren Rossen F-4

Randy Rosso I-1

Jonathan Rothbaum A-4

Sam Rowe H-1

Hejian Sang E-2

Aliya Saperstein E-4

Terrance Savitsky C-3

Ashley Schaad D-3

Bryan Schar J-3

Garrett Schmitt B-4

Alisú Schoua-Glusberg F-2

Nicola Schussler I-1

Laurel Schwede E-4

Marilyn Seastrom J-3

Katie Seely-Gant C-3
FCSM RESEARCH CONFERENCE AUTHOR AND COAUTHOR INDEX

NAME SESSION NAME SESSION

102

Mandy Sha F-2

Naga Shanmugam B-1

Michael Sinclair C-3

Avinash Singh B-2

Eric Slud B-3

Kristin Smyth H-2

Matthew Snipp E-4

Ray Snowden H-2

Ted Socha J-3

Chen Song E-4

Anita Soni B-2

Nancy Sonnenfeld C-3

Heidi St.Onge C-2

Deborah Stempowski G-1

John Stewart D-4

Martha Stinson D-1

Edward Strocko F-4

Jessica Stroop D-3, E-3, G-2

Rajesh Subramanian E-2

Hanyu Sun C-2, J-4

Mahdi Sundukchi E-3

Michael Sverchkov A-3

Samuel Szelepka I-4

David Talan C-1

Lucilla Tan I-2

Zaria Tatalovich F-4

Brian Taylor J-3

Betzaida Tejada Vera D-2

Randall Thomas E-3, I-2

Katherine Jenny Thompson C-4, E-1

Nhien To G-4

Michael Toland I-3

Robert Tortora A-2

Daniel Toth C-3

Roger Tourangeau C-2, H-3, J-1

Karen Trocki D-3

Stephen Tueller E-1

Joanna Turner G-3

Richard Valliant J-4

Donna Vallone E-3

Angel Velarde G-1

Victoria Velkoff H-4

François Verret A-3

Lars Vilhuber B-3, C-1, D-2

Monica Vines D-3

Katherine Virgo B-2

Pieter Vlag I-4

James Wagner A-2

Rana Wahdan I-3

Gina Walejko D-3

Adam Walsh F-3

Jiantong Wang E-1

Clayton Waring B-4

Elizabeth Weber Handwerker C-1

Chao Wei E-4

Rikki Welch D-3

Chin-Fang Weng G-1

Jerome Wernimont H-2

Brady West A-2

Ashley Westra E-3

T. Kirk White E-1

Bryn Whitmire G-3

Kelly Wilkin A-4

Abby Williams I-1

Sonja Williams B-1

Diane Willimack H-2

Gordon Willis B-2, C-2

Patrick Willis F-4

Hampton Wilson I-3

Maggie Wilson B-2

Tyler Wilson C-4

Monica Wolford B-1

K. Robin Yabroff B-2

Ting Yan H-3

Zhengwei Yang F-4

Robert Yerex F-3

Linda Young H-3

Erica Yu C-2

Mandi Yu F-1, I-1

Zhen Zeng A-1, F-1

Fan Zhang E-2

Li Zhu F-4

Zhengyuan Zhu E-2

Stephanie Zimmer E-2, G-2

Nikolas Zolas C-1
103

Reports Available in the Statistical Policy Working Paper Series

1. Report on Statistics for Allocation of Funds, 1978 (NTIS PB86-211521/AS)

2. Report on Statistical Disclosure and Disclosure-Avoidance Techniques, 1978 (NTIS PB86-211539/AS)

3. An Error Profile: Employment as Measured by the Current Population Survey, 1978 (NTIS PB86-214269/AS)

4. Glossary of Nonsampling Error Terms: An Illustration of a Semantic Problem in Statistics, 1978 (NTIS PB86-

211547/AS)

5. Report on Exact and Statistical Matching Techniques, 1980 (NTIS PB86-215829/AS)

6. Report on Statistical Uses of Administrative Records, 1980 (NTIS PB86-214285/AS)

7. An Interagency Review of Time-Series Revision Policies, 1982 (NTIS PB86-232451/AS)

8. Statistical Interagency Agreements, 1982 (NTIS PB86-230570/AS)

9. Contracting for Surveys, 1983 (NTIS PB83-233148)

10. Approaches to Developing Questionnaires, 1983 (NTIS PB84-105055)

11. A Review of Industry Coding Systems, 1984 (NTIS PB84-135276)

12. The Role of Telephone Data Collection in Federal Statistics, 1984 (NTIS PB85-105971)

13. Federal Longitudinal Surveys, 1986 (NTIS PB86-139730)

14. Workshop on Statistical Uses of Microcomputers in Federal Agencies, 1987 (NTIS PB87-166393)

15. Quality in Establishment Surveys, 1988 (NTIS PB88-232921)

16. A Comparative Study of Reporting Units in Selected Employer Data Systems, 1990 (NTIS PB90-205238)

17. Survey Coverage, 1990 (NTIS PB90-205246)

18. Data Editing in Federal Statistical Agencies, 1990 (NTIS PB90-205253)

19. Computer Assisted Survey Information Collection, 1990 (NTIS PB90-205261)

20. Seminar on Quality of Federal Data, 1991 (NTIS PB91-142414)

21. Indirect Estimators in Federal Programs, 1993 (NTIS PB93-209294)

22. Report on Statistical Disclosure Limitation Methodology, 1994 (NTIS PB94-165305)

23. Seminar on New Directions in Statistical Methodology, 1995 (NTIS PB95-182978)

24. Electronic Dissemination of Statistical Data, 1995 (NTIS PB96-121629)

25. Data Editing Workshop and Exposition, 1996 (NTIS PB97-104624)

26. Seminar on Statistical Methodology in the Public Service, 1997 (NTIS PB97-162580)

27. Training for the Future: Addressing Tomorrow=s Survey Tasks, 1998 (NTIS PB99-102576)

28. Seminar on Interagency Coordination and Cooperation, 1999 (NTIS PB99-132029)

29. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 1999 (NTIS 

PB99-166795)

30. 1999 Federal Committee on Statistical Methodology Research Conference: Complete Proceedings, 2000 

(NTIS PB2000-105886)

31. Measuring and Reporting Sources of Error in Surveys, 2001 (NTIS PB2001-104329)

32. Seminar on Integrating Federal Statistical Information and Processes, 2001 (NTIS PB2001-104626)

33. Seminar on the Funding Opportunity in Survey Research, 2001 (NTIS PB2001-108851)

34. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2001 (NTIS 

PB2002-100103)

35. Seminar on Challenges to the Federal Statistical System in Fostering Access to Statistics, 2002

36. Seminar on the Funding Opportunity in Survey Research, 2003

37. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2003

38. Summary Report of the FCSM-GSS Workshop on Web-Based Data Collection, 2004

39. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2005

40. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2007

41. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2009

42. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2012

43. Federal Committee on Statistical Methodology Research Conference (Conference Papers), 2013

Copies of these working papers may be ordered from NTIS Document Sales, 5285 Port Royal Road, Springfield, 

VA 22161; Telephone: 1-800-553-6847. The Statistical Policy Working Paper series is also available 

electronically from FCSM's Web site http://fcsm.sites.usa.gov/. 
NOTES

104
NOTES

105
NOTES

106
NOTES

107

The Federal Committee on Statistical Methodology and the Council of Professional Associations on

Federal Statistics recognize and are abundantly grateful to the following organizations for

sponsoring this year’s FCSM Research Conference. Their commitment and support help to ensure

the exchange of cutting edge statistical methods and findings among statisticians and other social 

scientists from the Federal government, academia, and the business community.

National Center for 

Education Statistics