www.rti.org RTI International is a trade name of Research Triangle Institute. RTI and the RTI logo are U.S. registered trademarks of Research Triangle Institute.

Developing and Evaluating Methodology 

for Split Questionnaire Design in the 

National Survey of College Graduates

Andy Peytchev

FCSM Conference

November, 2021
Acknowledgement

o Collaboration with Emilia Peytcheva, Darryl Cooney, Darryl Creel, Dave 

Wilson, and Jeremy Porter

o This research is supported by the National Center for Science and 

Engineering Statistics (NCSES) at NSF under a broad agency agreement

• Jennifer Sinibaldi and Matthew Williams

o The views expressed in this document are those of the authors and do not 

necessarily reflect the views of the National Center for Science and 

Engineering Statistics within the National Science Foundation

2
Objective

o Evaluate SQD to reduce burden

• How well is the data reproduced

• What methods perform best, specifically for NCSES data

o National Survey of College Graduates

• Fairly long (approximately half an hour to complete)

• Data not in restricted access, allowing flexibility in statistical tools

3
Survey Length

o Reduce burden on each respondent

• Sharp and Frankel (1983)

o Reduce nonresponse and potential nonresponse bias

• Long questionnaires can have higher nonresponse rates - e.g., Heberlein and 

Baumgartner (1978); Adams and Darwin (1982); Dillman, Sinclair and Clark 

(1993)

- Finding less consistent for interviewer-administered modes

- Lack of evidence for nonresponse bias

o Reduce measurement error

• Peytchev and Peytcheva (2017)
Split Questionnaire Design (Raghunathan and Grizzle, 1995)

o Main objective: shorten the survey instrument to reduce respondent burden 

while maintaining a rectangular dataset with all survey variables

o Extension of the multiple matrix sampling design (Shoemaker, 1973 and 

Munger and Lloyd, 1988)

5
Split Questionnaire Design

o Divide questionnaire into modules

o Administer a subset to each sampled individual, while observing all 

possible combinations of variables (i.e., bivariate associations)

o Multiply impute data for omitted module(s)

Core Module A Module B Module C

Full qnnre Group 0

Core Module A Module B Module C

Split qnnre Group 1

Group 2

Group 3
Key Factors to Evaluate Prior to Implementation

o How to create the splits

o How to impute the missing data

o What is the impact on:

• Nonresponse rates

• Nonresponse bias

• Measurement error bias and variance

Calls for an 

experimental 

design

Can be 

simulated on 

existing data
Creating the Splits

o The cognitive perspective

• Organize by topic

o The statistical perspective

• Maximize associations across modules

• Matrix sampling idea
Trying Marijuana in the National Survey for Drug Use and Health

9

2002 and Earlier

How do you feel about adults 

smoking one or more

packs of cigarettes per day?

How do you feel about adults 

trying marijuana or hashish 

once or twice?

2003 and After

Question on smoking was dropped:

During the past 12 months, how 

many times have you attacked 

someone with the intent to 

seriously injure them?

How do you feel about adults trying 

marijuana or hashish once or 

twice?
Attitudes Towards Trying Marijuana, 2000-2004 NHSDA/NSDUH

30.5% 27.9% 28.2%

43.1% 42.3%

55.7% 55.7% 54.7%

43.3% 44.2%

0.0%

10.0%

20.0%

30.0%

40.0%

50.0%

60.0%

2000 2001 2002 2003 2004

Percent

Year

Neither approve nor

disapprove

Somewhat disapprove

Strongly Disapprove

Data source: Wang, K., R. Baxter, and D. Painter. (2005). Modeling Context Effects in the National Survey of Drug Use and 

Health (NSDUH). Proceedings of the Joint Statistical Meetings.
Creating the Splits Revisited

o The cognitive perspective

• Organize by topic

o The statistical perspective

• Maximize associations across modules

• Matrix sampling idea

o Statistically informed splits

• Organize by topic and modify based on missing associations
Statistically Informed Splits: National Survey of College Graduates

o Correlations between all 

variables

o Ordered by sequence in 

the questionnaire

o Heatmap to identify groups 

of questions that lack 

associations with 

questions in other modules
Statistically Informed Splits: National Survey of College Graduates

13

Logical 

Split

Statistically 

Informed 

Split
Multiple Imputation

Two very different types of approaches with different strengths and 

weaknesses

o Regression-based imputation

o Weighted sequential hot-deck imputation

14
Multiple Imputation: National Survey of College Graduates, 2019

o Almost exclusively categorical variables

o Some variables with large number of categories

o Many variables (over 200)

o Many cases (almost 100,000)

o Identifying software and hardware limitations

• Breaking up processes

• Choice of software

• Both

15
Next Steps

o Complete imputation steps

• Improve models

• Finalize imputed datasets

o Evaluate and compare

• Approach to creating splits

• Imputation methods

o Offer recommendations

o Disseminate findings

16
Andy Peytchev

apeytchev@rti.org

17