Richard A. Kulka 

Consultant 

10/26/2011 
Background: Current Workshop 

 Builds on extensive ongoing program of both internal 

and external research, advice and review (since the 

mid-1980s). 

 A series of workshops, forums and expert panels, 

under the general aegis of the “Gemini Project”

 All focused on development of a fundamental redesign 

of the Consumer Expenditure Surveys 

 Major component of the charge and work of the 

CNSTAT Panel 

10/26/2011 CE Redesign Options Workshop 2 
CNSTAT Panel Charge: 

 Review output of other workshops and forums. 

 Convene household survey data producer workshop on 

experience of how other survey organizations address 

similar challenges. 

 Contract with experienced data collection organizations to: 

 Develop concrete design options and 

 A discussion of their relative merits. 

 Convene workshop based on results of this work. 

 Produce a consensus report of findings and 

recommendations on redesigned survey options. 

10/26/2011 CE Redesign Options Workshop 3 
Contractor Charge 

 Provide balanced evaluation of benefits, costs and 

risks of proposed options compared with current CE 

surveys. 

 Provide comprehensive proposal for survey design 

and/or process that collects required data. 

 Provide clear recommendations for evaluating 

proposed design in the CE context. 

 Focus in particular on: (1) measurement error; (2) 

changes in survey environment; and (3) flexibility in 

data collection modes. 

10/26/2011 CE Redesign Options Workshop 4 
Have they met this charge? 

 Overall, each has done so, and in several instances 

their reports exceed these requirements. 

 But: there are also several more implicit questions in 

this charge and the potential value that these reports 

may provide for the Panel and BLS. 

 Given the very brief time period available 

for this work, these reports are very rich in innovative 

ideas and insights. 

 However, is very much the case that the “devil is in the 

details.” 

10/26/2011 CE Redesign Options Workshop 5 
Discussion Outline 

 What are the key features of each proposed redesign? Are 

these well justified relative to other alternatives and their 

feasibility? Which proposed aspects of these designs merit 

some additional thought and clarification? 

 Are these two proposed designs fundamentally (or 

radically) different from the current CE surveys, and, in 

turn, from one another? 

 Are there some additional insights, questions or themes 

that merit careful consideration or serious attention 

independent of the specific designs proposed by each 

contractor? 

10/26/2011 CE Redesign Options Workshop 6 
Westat—Key Features 

 Maintains separate diary and quarterly interview 

surveys. 

 Both provide several alternative methods for reporting 

data: electronic or paper; strong emphasis on use of 

receipts; downloading electronic data files; scanning 

of receipts and other paper. 

 Both rely on submission of such data to a central 

repository, where data are received, extracted, 

translated, coded and evaluated with “smart” 

prompting by web survey or telephone to obtain 

additional information required by the CE program. 

10/26/2011 CE Redesign Options Workshop 7 
Westat—Key Features 

 Both involve intensive monitoring and prompting by 

interviewers through multiple scheduled contacts to 

stimulate complete and timely reporting. 

 Both are explicitly designed to reduce dependence on recall 

and recall interviews. 

 In diary component, all CU members aged 14+ would selfrespond by self-selected method of data collection. 

 Quarterly interviews would maintain a three-month 

reference period (with recall interviews minimized), but 

with only two waves of data collection 12 months apart). 

10/26/2011 CE Redesign Options Workshop 8 
Westat: Observations/Questions 

 An innovative, complex central depository is a particularly 

attractive feature of proposed design. 

 Burden on this system to receive and dynamically manage 

multiple types of data and provide real-time feedback and 

queries is considerable. 

 Detailed description of reading and interpreting various 

types of data submitted and generating queries in real time 

most impressive. 

 But feasibility, especially in the diary component, with oneweek data collection, seems problematic. 

 Need for web survey prompting after submission for many 

(if not most) of the expenditures reported could serve as a 

barrier for reporting (analogous to “panel conditioning”) . 

10/26/2011 CE Redesign Options Workshop 9 
Westat: Observations/Questions 

 Not clear whether actual or “objective” reduction in 

respondent burden will be perceived as real. 

 Downloading and submitting data daily in diary survey 

(especially banking and retail data) may be more 

burdensome. 

 Learning curve involved in electronic reporting of quarterly 

interview data may increase perceived burden with 12 months 

“downtime” between waves.

 Twelve months between quarterly interviews may also not 

be optimal for other reasons (e.g., significant changes in 

CU composition). 

 While an Event History Calendar will likely improve recall 

(when a recall interview is necessary) in quarterly 

interview, its value in a one week diary seems more limited. 

10/26/2011 CE Redesign Options Workshop 10
Westat: Observations/Questions 

 Data requirements for CE analysts mostly met, other 

than panel component. 

 Cost of proposed design substantially greater than 

current CE surveys, thereby requiring careful analysis 

of cost-benefit tradeoffs and priority setting. 

10/26/2011 CE Redesign Options Workshop 11
Westat: Summary 

 Proposed design focuses heavily on: 

 increased reporting flexibility, extensive use of monitoring, 

prompting and help to reduce respondent burden 

 greater use of personal electronic data, receipts and real time 

recording to reduce reliance on recall. 

 A sophisticated “data repository” (that serves as much more) 

to hold it all together. 

 Feasibility testing of all components and at all levels is 

critical. Detailed evaluation protocol and plan described 

may not be sufficient. 

 Most attractive features of design make it significantly 

more costly. 

10/26/2011 CE Redesign Options Workshop 12
UWM: Key Features 

 A key feature of the proposed design is a single sample that 

integrates diary and retrospective reports of expenditures. 

 Reduces redundancies between the CEI and CED surveys. 

 Single-sample design has two components: 

 A cross-sectional component, where each CU participates for 

one month. 

 A panel component, for which each CU participates three 

times (also for one month) at six month intervals. 

 Minimizes reliance on retrospective reporting, promoting 

real-time recording of all expenditures, emphasizing selfreporting for all CU members. 

10/26/2011 CE Redesign Options Workshop 13
UWM: Key Features 

 Reporting task is the same for each component: 

maintaining a diary for a one month period in which all CU 

members record all purchases and expenditures. 

 Use of shared computer tablets as primary “diary” 

recording device, with web-based data collection (or CASI 

and download). 

 Capture and recording of both paper and electronic 

receipts. 

 Use of portable diaries, cell and smartphones, and other 

technology (e.g., scanner, barcode and financial software) 

for real-time data capture and “memory triggers.”

10/26/2011 CE Redesign Options Workshop 14
UWM: Observations/Questions 

 In principle, a fully-integrated design that emphasizes 

prospective, “real-time” collection of expenditures and 

payments. 

 Takes full advantage of technology, broadly conceived, including 

a common platform for integrating input and information. 

 Data captured by a flexible array of tools serving either as 

memory or recording aids, or for direct data input. 

 In practice, however, there are some significant questions about 

the feasibility of this design. 

 Some involve the design and integration of the technology, 

including their support, maintenance and training. Some 

relevant experience available from Knowledge Networks, 

Stanford and Abt SRBI, Nielsen and USDA, but requirements are 

significantly different and greater. 

10/26/2011 CE Redesign Options Workshop 15
UWM: Observations/Questions 

 Of greater uncertainty, however, is one’s capacity to recruit and 

motivate CU members to maintain a diary for a full month rather 

than two weeks (three times for the panel component!), even 

with all of the tools to be leveraged in this effort. 

 In principle, the preferred requirement to use a computer tablet 

for reporting is both a strength and weakness of the design. 

 For example, the flexibility enhanced by use of the other 

technologies and methods proposed is undermined somewhat as 

they are relegated largely to “memory triggers” or aides.

 Use and integration of these various other technology- and 

paper-based methods as memory aids or triggers, versus direct 

sources for data input, requires significantly more thought and 

design work. 

10/26/2011 CE Redesign Options Workshop 16
UWM: Observations/Questions 

 Recognition that nonresponse is a key threat to this design, 

the need for a “responsive design” approach, and thoughts 

about how to carry this out is a major strength. 

 Most data requirements for CE analysts are met by this 

design (one exception being quarterly panel change 

estimates by a CU), but it explicitly gives highest priority to 

improvement of data for the CPI. 

 Costs of proposed design are substantially greater than 

current CE surveys, but since some of these (e.g., 

technology and development) would be amortized over 

multiple years, a more detailed analysis is necessary. 

10/26/2011 CE Redesign Options Workshop 17
UWM: Summary 

 A bold, creative design reflecting a number of the key concerns raised 

in previous workshops and forums of both methodologists and users. 

 Proposes to take maximum advantage of virtually all recent and future 

technological and cultural changes identified in the survey and 

consumer environments. 

 A clear focus on significantly, and perhaps dramatically, reducing our 

historical dependence and reliance on retrospection and recall for 

reporting expenditure data. 

 However, in spite of its considerable appeal, there is little empirical 

evidence to date to make one confident that such a design is feasible. 

 Achieving the full degree of flexibility envisioned requires more design 

thinking and work, likely beyond the types and sizes of evaluations 

proposed. 

 Its costs, at least at the outset, need to be more carefully evaluated. 

10/26/2011 CE Redesign Options Workshop 18
How Different Are These Designs? 

 Both of these proposed designs are clearly different in 

fundamental ways from the current CE surveys. 

 These differences clearly reflect insights and concerns from 

prior research, workshops and forums sponsored by BLS. 

 The UWM design is self-described as a “radical” departure 

from the current approach, which is certainly true. 

 The Westat design is somewhat less “radical,” maintaining 

more features of the current design, but with some 

important fundamental design changes as well. 

10/26/2011 CE Redesign Options Workshop 19
How Different Are These Designs? 

 And, at first blush, they are also quite different from one 

another. 

 While the differences are significant and important, 

however, these are in part somewhat illusory. 

 Philosophies driving both are to design a consumer 

expenditure survey that is more prospective (less 

dependent on recall), flexible, and technological. 

10/26/2011 CE Redesign Options Workshop 20
How Different Are These Designs? 

 One key difference is: 

 Westat’s proposed use of a central data depository as a major 

element to support these features, versus 

 UWM’s reliance on shared computer tablets as the primary “diary” 

recording device and platform. 

 Both were discussed as options in a previous Westat report for BLS. 

 A second key difference is UWM’s proposal to integrate the 

diary and interview surveys, an option also considered by 

Westat in its current report. 

 Thus, one can certainly envision deriving various “hybrid” 

approaches that take advantage of the desirable features of 

both designs, some of which overlap in any case. 

10/26/2011 CE Redesign Options Workshop 21
Additional Insights, Questions or 

Themes 

 In their reviews of factors driving these redesigns, two key 

factors seem especially significant and worthy of more 

detailed attention: 

 The Westat report in particular does an excellent job of 

showing that over 70 percent of consumer purchases are 

documented in electronic format. 

 Both reports clearly show, based on experiences across a 

diverse array of surveys, that use of records significantly 

improves the quality of data obtained from respondent 

reports. 

 While each proposed design encompasses key features that 

recognize and attempt to address and leverage these 

factors, these observations seem dramatic and significant 

enough to merit more detailed investigation. 

10/26/2011 CE Redesign Options Workshop 22
Over 70% of Consumer Purchases 

Are in Electronic Format 

 Encouraging respondents (1) to provide personal 

expenditure data by retrieving their information from these 

databases and then (2) obtaining their consent to seek such 

information from retailers and utility companies are sound 

ideas as far as they go. 

 Far bolder, however, would be for BLS to use its credibility 

and clout (likely to require new legislation)to obtain such 

information directly for all CE households (with their 

consent) from all holders of these electronic records (as 

suggested by Westat). 

 For retail purchases, creating a universal CE “loyalty card” 

issued in partnership by BLS and all retail vendors might 

also be explored. 

10/26/2011 CE Redesign Options Workshop 23
Use of Records Significantly 

Improves Data Quality 

 While both designs aggressively promote the use of 

receipts and records far beyond the current methodology 

in the CE surveys, this is such a significant factor for the 

likely success of any future design that it merits more direct 

and sustained attention. 

 A “crash” program of both qualitative and quantitative 

research using a classic “Dillman” approach, which 

explores ways of highlighting the critical importance of this 

behavior and minimizing the burden of doing so, seems 

essential to assure the success of either of these two designs 

(or any other that would meet the requirements of the CE 

redesign effort). 

10/26/2011 CE Redesign Options Workshop 24
Use of Records Significantly 

Improves Data Quality 

 Work on the SIPP, noted in both reports, would be a 

good starting point for such an effort. 

 Of particular note was their insistence that use of 

records was the “norm,” the usual, expected part of the 

respondent’s task.

 Similarly, the focus of respondent incentives would be 

on encouraging respondents to use records, rather 

than just responding to the survey in general. 

10/26/2011 CE Redesign Options Workshop 25
Conclusions 

 Both proposed redesigns provide significant, detailed 

advice, guidance and insights to the Panel and BLS. 

 Neither, however, provides a “turn-key” design package 

that could be implemented as proposed without significant 

testing, a critical examination of costs (and benefits), and 

(unfortunately, given the BLS charge) some substantial 

additional research. 

 Probably unrealistic at the outset to assume that they 

would. 

 Very significant progress in providing the elements and a 

concrete roadmap for how a specific, well-conceived and 

well-tested design can be achieved. 

 Both contractors are to be congratulated for their efforts. 

10/26/2011 CE Redesign Options Workshop 26