The Use of Responsive Split Questionnaires in a Panel Survey

Title of dissertation: THE USE OF RESPONSIVE SPLIT QUESTIONNAIRES IN A PANEL SURVEY Jeffrey M. Gonzalez, Doctor of Philosophy, 2012 Dissertation directed by: Professor Richard Valliant The Joint Program in Survey Methodology Lengthy surveys may be associated with high respondent burden, low data quality, and high unit nonresponse. To address these concerns, survey designers may reduce the length of a survey by eliminating questions from the original questionnaire, but this means that some information would never get collected. An alternative may be to divide a lengthy questionnaire into subsets of survey items and then administer each subset to distinct subsamples of the full sample. This is referred to as a split questionnaire design and has the benefit of collecting all of the original survey information. We identify a significant deficiency in the current set of split questionnaire methods, namely, the incomplete use of prior information about the sample unit in the design. In most contemporary applications of split questionnaires, generally only characteristics of the survey items (e.g., content, cognitive burden) are used to inform the design; however, if joint consideration is given to characteristics on the survey items as well as the sample unit when designing a split questionnaire, then there may be the potential to improve the split questionnaire’s utility. In this dissertation, we explore the extent to which, if any, jointly considering both types of information at the design stage will yield more efficient split questionnaires. We propose various methods for incorporating prior information about the sample unit into the split questionnaire using features of responsive design. We highlight how this specific application of a responsive split questionnaire can be used to address the concerns present in a major federal survey. Finally, we draw from the literature pertaining to survey design, experimental design, and epidemiology to develop and implement a framework for evaluating the proposed new elements of our split questionnaire design. THE USE OF RESPONSIVE SPLIT QUESTIONNAIRES IN A PANEL SURVEY by Jeffrey Mark Gonzalez Dissertation submitted to the Faculty of the Graduate School of the University of Maryland, College Park in partial fulfillment of the requirements for the degree of Doctor of Philosophy 2012 Advisory Committee: Dr. Richard Valliant, Chair/Advisor Dr. Francis B. Alt Dr. John L. Eltinge Dr. Frauke Kreuter Dr. Nathaniel Schenker c © Copyright by Jeffrey Mark Gonzalez 2012 Preface This dissertation details and evaluates potential methods for designing a specific class of split questionnaire surveys. These methods will contribute to the ongoing research involving split questionnaire designs as well as the redesign of the U.S. Consumer Expenditure Quarterly Interview Survey (CE) at the U.S. Bureau of Labor Statistics (BLS). The primary objective of this dissertation is to develop and propose various extensions of split questionnaire methods and to examine their utility in addressing some of the challenges that the current CE presents. The current CE instrument is challenging for both the interviewers and respondents and the challenging nature of the instrument can adversely affect the survey products. The primary challenges are as follows. First, the interview is long. Depending on the amount and type of expenditures reported, it takes on average 65 minutes to complete (BLS Handbook of Methods, 2007). Lengthy surveys are often viewed as burdensome and subsequently hypothesized to be negatively correlated with response quality. Second, the questions are detailed. In particular, the respondent is asked to report information (e.g., what was purchased, the amount of the purchase, and when it was purchased) for about 60 to 70 percent of their household’s expenses made in the previous three months. These generally include significant purchases, such as those for property, automobiles, and other large durable goods, as well as recurring expenses, such as mortgage/rent and utility bill payments. Even though these are thought to be the types of expenditures that respondents can recount over a threemonth period or longer, the nature of the reporting task may still pose problems ii for respondents and thus undermine the quality of the data collected. Furthermore, there are external researchers who claim that CE estimates of consumer expenditure shares are biased (Garner et al., 2006). They base their claim on the disparity between expenditure shares calculated from CE data and those calculated from the Bureau of Economic Analysis’s Personal Consumption Expenditures (PCE) of the National Income and Product Accounts, a data source that is often cited as being the gold standard for comparison. Finally, there is also concern over declining unit response rates. Although this trend is not unique to the CE (de Leeuw and de Heer, 2002), the unit response rate for the CE was about 80 percent in the early 2000s but around 74 percent in 2009. The Office of Management and Budget (OMB) has emphasized that every attempt should be made to achieve and maintain an acceptable unit response rate (OMB Standards and Guidelines for Statistical Surveys, 2006). Therefore, to potentially reduce respondent burden while improving data quality and the unit response rate, this dissertation will examine extensions of the current set of split questionnaire methods and their feasibility for use as a redesign option for the CE. Two special features of the current CE (which are unlikely to change during the redesign effort) make it an appropriate data source for investigating extensions of split questionnaire methods. These features are (1) the panel survey design and (2) the measurement goal of the survey – collecting detailed household expenditures and information about the types of consumers who make those purchases. The CE employs a rotating panel survey design in which a particular consumer unit (CU), which can be thought of as being equivalent to a household, is interviewed once iii every quarter for five calendar quarters. Although the initial interview is a bounding interview and the data are not used in any official published estimates, similar categories of expenditures, in general, are asked in all five interviews. The implementation of split questionnaire designs that will be explored in this dissertation is for the first interview to remain as is, but the second would be subject to the split questionnaire design. The second feature is household expenditure patterns. We can leverage our understanding of purchase behavior across time and how some demographic characteristics are predictive of those purchases to collect those information from an initial data gathering effort (e.g., first interview). Once we have the necessary information, we can input it into a model that describes purchase behavior, and use the outputs from the model to design the split questionnaire for the second interview. Thus, we make explicit use of both the characteristics of the survey items and the sample units to design the split questionnaire. We refer to this type of split questionnaire as a responsive split questionnaire design because we draw on features of responsive designs (see Groves and Heeringa [2006] for a discussion of responsive designs). Responsive split questionnaire designs will be the focus of this dissertation research. This dissertation is organized into the following chapters. Chapter 1 briefly introduces the topic of study and provides some motivation for and applications of the dissertation research. Chapter 2 delves deeper into the motivation by providing a literature review of relevant research in survey methodology and other fields (e.g., epidemiology). Chapter 3 presents the results of preliminary data analyses. These highlight relationships in the data that could potentially be used in the development iv of decision rules (by a decision rule we mean whether or not to ask a particular question of a sample unit), and demonstrate the utility of extending the current set of split questionnaire methods. We also use these preliminary analyses to identify areas of the research problem that warrant particular focus. In Chapter 4, using historical CE data, we examine a variety of responsive split questionnaire designs based on different techniques for incorporating prior information on sample units to develop decision rules. We also employ mathematical programming methods to ensure that various survey constraints are met when a responsive split questionnaire design is utilized. We then evaluate the performance of our decision rules by exploring hypothetical scenarios of collected data had the methods been implemented. In Chapter 5, we conclude with a discussion of the lessons learned and some of the limitations of this research, offer guidelines for survey programs wishing to utilize the methods developed in this research, and identify areas for future investigation.