First Workshop of the Consumer Expenditure Surveys Redesign Paper, Split Questionnaire Page 1 of 5

Split Questionnaire Methods for the 

Consumer Expenditure Surveys Program 

1. Background Statement 

The CEQ presents a number of challenges for both interviewers and respondents. First, 

the interview is long. Depending on the amount and type of expenditures reported, it takes, on 

average, 65 minutes to complete (BLS Handbook of Methods, 2007). Second, some of the 

questions are detailed and include what was purchased, the amount of the purchase, and when it 

was purchased. In fact, the respondent is asked to report this information for about 60 to 70 

percent of the household’s expenses during the previous three months (BLS Handbook of 

Methods 2007). Finally, there is concern over declining response rates. The response rate for the 

CE was about 80 percent in the early 2000s, but now hovers around 74 percent. 

Survey methodological research suggests that there is a relationship between length of a 

survey and each of response rate, data quality, and burden. More specifically, several studies 

provide evidence that lengthy survey questionnaires tend to have lower response rates (Adams 

and Darwin 1982; Bogen 1996; Love and Turner 1975). There is also research suggesting that 

lengthy questionnaires can have adverse effects on data quality (Herzog and Bachman 1981; 

Johnson et al, 1974; Kraut et al, 1975). Finally, Bradburn (1978) identifies length of the 

interview as one of the key dimensions of respondent burden – suggesting that lengthier surveys 

are more burdensome. It is also worth noting that total time expended by the survey respondent 

is the primary component of the Office of Management and Budget’s (OMB) concept of 

respondent burden (OMB Standard and Guidelines for Statistical Surveys, 2006). 

To address these concerns, one approach may be to administer a shorter questionnaire to 

each sample unit. A challenge, however, is that we must decide which questions to eliminate 

from the survey, in an environment where users regularly ask for more data on more topics. An 

alternative to completely eliminating questions from the full survey is to divide the questionnaire 

into subsets of questions and then administer each subset to distinct subsamples of the full 

sample. This creates a shorter questionnaire that still collects the necessary information from at 

least some of the sample members. These designs are often referred to as split questionnaire 

(Raghunathan and Grizzle 1995) or multiple matrix sampling (Shoemaker 1973) designs. Opting 
First Workshop of the Consumer Expenditure Surveys Redesign Paper, Split Questionnaire Page 2 of 5

for this approach requires careful consideration of two key issues – design and estimation of the 

split questionnaire. 

Modifying the current CE instrument via split questionnaire methods requires a focused 

attention on the features of the CE that would most influence the design and estimation 

decisions. First, many of the expenditure items that CE collects information for are rarely or 

infrequently reported during the interview, such as expenditures for plumbing supplies, floor 

coverings, and refrigerators. The ability to compute precise estimates of characteristics of these 

items (e.g., purchase cost) would potentially be hampered by the reduced sample size receiving 

questions about those items. This is because the sampling variance would increase. Second, 

primary data users, such as the CPI, preparers of published tables, and microdata users, have 

specific needs and requirements regarding the CE data. By asking only a subset of questions of 

each household, there would be missing data on the remaining questions of the full questionnaire 

that are not asked of that household. An important consideration is the potential impact of this 

missing data on the key data users and how it is addressed (e.g., imputation, weighting 

adjustments). 

As CE considers a redesign, we are mindful of the quality-cost trade-offs that any 

redesign would require. We recognize that only shortening the length of the survey would likely 

not decrease data collection costs since we believer that costs are dominated by locating and 

contacting sample units and by the primary mode of data collection. However, research on split 

questionnaires may provide a framework for considering other design modifications that would 

result in a substantial cost savings. For instance, a shorter CE interview might be more amenable 

to being administered by telephone, which would likely lead to reduced field costs. 

2. Relevant Work 

Internal Research 

Researchers within the BLS have started to investigate the use of split questionnaire 

methods as one option for redesigning the CE. Their research has focused on some statistical 

issues related to these methods. Specific research endeavors are summarized below. 

 Gonzalez and Eltinge (2007a) offer a discussion of some of the previous research on matrix 

sampling methods, survey design considerations, and implementation trade-offs. They 

highlight three phases of the survey process that warrant consideration – development of the 
First Workshop of the Consumer Expenditure Surveys Redesign Paper, Split Questionnaire Page 3 of 5

split questionnaire, data collection, and processing and analysis.1 In the context of the CE, 

they also note: (1) if we modify the CE using split questionnaire methods, then the 

modification should be consistent with the primary objectives of the CE program; and, (2) 

utilization of these methods must be statistically defensible and operationally feasible. 

 Gonzalez and Eltinge (2007b) identified various statistical issues that need consideration if 

CE is modified via split questionnaire methods. Two key issures are: (1) adjustment of 

calibration weighting methods to account for subsampling; and, (2) adjustment of standard 

balanced repeated replication methods of variance estimation. They also provided notation 

for considering simple design-based and imputation-based estimators of population means 

under a split questionnaire design. 

 Gonzalez and Eltinge (2008) began to explore the use of prior information on a sample unit 

to inform the design of split questionnaire forms. Specifically, they based the subsampling 

probabilities for a sample unit (for matrix sampling in the second interview) on the 

expenditure values reported during the initial interview and evaluated the variance properties 

of point estimators based on simple weighting adjustments. 

 Gonzalez and Eltinge (2009) explored issues related to imputing expenditure values for 

various categories. Using a regression framework, the authors addressed problems 

encountered when imputing expenditures (e.g., the flexibility of the imputation procedure to 

impute “zero-dollar” amounts), model specification, and the potential effect on variance 

estimation. 

External Research 

Two key papers external to BLS address design and estimation issues related to split 

questionnaire methods. These are summarized below. 

 Raghunathan and Grizzle (1995) developed a survey design where a questionnaire was split 

into components and individuals were administered various subsets of the components. They 

developed a multiple imputation method for analyzing the data collected under this design. 

They concluded that little was lost by administering only parts of the questionnaire to each 

sample unit. 

 

1

 We take the perspective that design of a split questionnaire encompasses issues related to both development of the 

split questionnaire and data collection. 
First Workshop of the Consumer Expenditure Surveys Redesign Paper, Split Questionnaire Page 4 of 5

 Thomas et al. (2006) developed an algorithm for designing split questionnaire forms in 

which questions were allocated to different forms based on their correlation with the other 

items. The idea was that highly correlated items would be allocated to different forms so that 

imputation could be used more effectively to analyze the data. They also required a fixed 

“core” set of questions (that are asked of all sample units). This included questions that were 

highly predictive of responses to several other items, were of special interest, and/or had 

certain precision requirements. 

3. Key Issues 

We seek recommendations for alternative designs that could be implemented as part of a 

redesign of the CE surveys. However, if split questionnaire methods are adopted, then CE will 

have to determine the “most efficient” way to split the questionnaire while still collecting all of 

the necessary information. The “most efficient” design would be informed by both cognitive 

(e.g., flow of questionnaire) and statistical (e.g., impact on precision of certain estimates) issues. 

In addition, CE will have to explore different methods for meeting the statistical requirements of 

various data users. This would include appropriate methods for estimation (e.g., weighting, 

imputation). Finally, the cost of implementing a split questionnaire design must also be a key 

consideration since the cost of any redesign must be consistent with the current CE budget. 

4. Discussion Points 

1. What is the impact of breaking up the interview on data quality (e.g., reporting, response 

rates) and respondent burden? 

2. What are the cognitive aspects of breaking up the interview that CE should consider? 

3. What features of the current CE (e.g., types of expenditures being collected, panel) would 

have the greatest influence on design and estimation issues? 

4. What are the primary statistical issues, in addition to the ones cited above, that CE needs 

to address when investigating the utility of these methods? 

5. What are the implications for the primary CE data users (e.g., CPI, published tables, and 

academic community)? 

6. What are the operational challenges associated with implementing these types of designs? 

7. What should the next steps be to explore and research this issue? 
First Workshop of the Consumer Expenditure Surveys Redesign Paper, Split Questionnaire Page 5 of 5

5. References 

Adams, L. M. and Darwin, G. (1982). Solving the Quandary Between Questionnaire 

Length and Response Rate in Educational Research. Research in Higher Education, 17, 231–40. 

Bogen, K. (1996). The Effect of Questionnaire Length on Response Rates – A Review of 

the Literature. Proceedings of the Section on Survey Research Methods, American Statistical 

Association, 1020–5. 

Bradburn, N. M. (1978). Respondent Burden. Proceedings of the Section on Survey 

Research Methods, American Statistical Association, 35–40. 

Bureau of Labor Statistics, U.S. Department of Labor, Handbook of Methods, Chapter 16, 

April 2007 edition, Consumer Expenditures and Income. 

http://www.bls.gov/opub/hom/pdf/homch16.pdf (visited October 6, 2009). 

Gonzalez, J. M. and Eltinge, J. L. (2007a). Multiple Matrix Sampling: A Review. 

Proceedings of the Section on Survey Research Methods, American Statistical Association, 

3069–75. 

Gonzalez, J. M. and Eltinge, J. L. (2007b). Properties of Alternative Sample Design and 

Estimation Methods for the Consumer Expenditure Surveys. Paper presented at the 2007 

Research Conference of the Federal Committee on Statistical Methodology, Arlington, VA, 

November, 2007. 

Gonzalez, J. M. and Eltinge, J. L. (2008). Adaptive Matrix Sampling for the Consumer 

Expenditure Quarterly Interview Survey. Proceedings of the Section on Survey Research 

Methods, American Statistical Association, 2081–8. 

Gonzalez, J. M. and Eltinge, J. L. (2009). Imputation Methods for Adaptive Matrix 

Sampling. A poster presented at the 2009 Joint Statistical Meetings, Washington, DC, August, 

2009. 

 Herzog, A. R. and Bachman, J. G. (1981). Effects of Questionnaire Length on Response 

Quality. The Public Opinion Quarterly, 45, 549–59. 

 Johnson, W. R., Sieveking, N. A., and Clanton, E. S. (1974). Effects of Alternative 

Positioning of Open-Ended Questions in Multiple-Choice Questionnaires. Journal of Applied 

Psychology, 59, 776–8. 

Kraut, A. I., Wolfson, A. D., and Rothenberg, A. (1975). Some Effects of Position on 

Opinion Survey Items. Journal of Applied Pyschology, 60, 774–6. 

Love, L. T. and Turner, A. G. (1975). The Census Bureau Experience: Respondent 

Availability and Response Rates. Proceedings of the Business and Economics Section, American 

Statistical Association, 76–85. 

Munger, G. F. and Lloyd, B. H. (1988). The Use of Multiple Matrix Sampling for Survey 

Research. Journal of Experimental Education, 56, 187–91. 

Office of Management and Budget (2006). Standards and Guidelines for Statistical 

Surveys. Washington, DC. Retrieved October 28, 2009 from the Office of Management and 

Budget website www.whitehouse.gov/omb/inforeg/statpolicy/standards_stat_surveys.pdf. 

Raghunathan, T. E. and Grizzle, J. E. (1995). A Split Questionnaire Survey Design. 

Journal of the American Statistical Association, 90, 54–63. 

Shoemaker, D. M. (1973). Principles and Procedures of Multiple Matrix Sampling.

Cambridge, MA: Ballinger Publishing Company. 

Thomas, N., Raghunathan, T. E., Schenker, N., Katzoff, M. J., and Johnson, C. L. (2006). 

An Evaluation of Matrix Sampling Methods Using Data from the National Health and Nutrition 

Examination Survey. Survey Methodology, 32, 217–31.