1 

Workshop on Redesigning the Consumer Expenditure Surveys 

Issues in Implementing Change in a Complex Survey 

Chet Bowie (NORC at the University of Chicago) – Discussant 

October 26, 2011 

Introduction

The ongoing Consumer Expenditure Surveys (CES) that began in 1980 consists of two separate 

components, each with its own questionnaire and sample: 1) an interview survey in which each 

of the consumer units (CUs) in the sample is interviewed every 3 months over a 12-month 

period, and 2) a diary survey in which CUs complete a diary of expenses for two consecutive 1-

week periods. Data collection for both survey components is done by the Census Bureau under 

contract with the Bureau of Labor Statistics (BLS). The two survey components are designed to 

capture different types of consumer expenditures – larger or regularly occurring expenditures 

that can be easily recalled over a period of time, such as car purchases and rent and utility 

payments, in the Interview Survey; and smaller, more frequent purchases that cannot be easily 

recalled, such as grocery items or housekeeping supplies, in the Diary Survey. Since CUs are 

asked to record all expenditures made during the reporting period in the Diary Survey, an overlap 

in coverage occurs between the two components. 

The BLS has begun a major project to investigate alternatives for redesigning the surveys. The 

primary objective of the redesign effort is to improve expenditure estimates through a reduction 

in measurement error, with a particular emphasis on underreporting. This effort is designed to 

maintain or improve response rates and keep survey costs at their current level. The project aims 

to address how survey estimates have been impacted by changes in society, new consumer 

products and spending methods, and to explore how new data collection technologies can 

improve expenditure reporting. 

As part of this process BLS has contracted with the National Academy of Sciences, Committee 

on National Statistics (CNSTAT) to conduct an independent review of the design options 

available and to make specific recommendations for redesign. CNSTAT has organized a Panel 

of experts to carry out that task. The Panel initiated a Request for Proposal (RFP) through the 

National Academy of Sciences (NAS) for design options and a discussion of the relative merits 

of the options. As a result of the RFP, the NAS awarded two subcontracts to come up with 

options to redesign the surveys. The subcontractors are WESTAT and a partnership between the 

University of Wisconsin-Milwaukee, University of Nebraska, and Abt SRBI. These two 

subcontractors presented their results in a workshop scheduled on October 26, 2011 in 

Washington, DC. 
2 

The NAS asked me to be a formal discussant at this workshop focusing on the issues of 

implementing change in a large ongoing survey, particularly the proposed redesigns offered by 

the subcontractors. This paper summarizes the comments I made at the workshop. 

The comments and views expressed in this paper and presented at the workshop are mine and do 

not represent those of NORC at the University of Chicago. 

Issues

I discussed the following issues at the workshop related to implementing change in a complex 

survey: 

Context 

Planning the change 

Stakeholders (internal and external) 

Staffing

Testing and Implementation 

Funding 

Cost 

Change in Contractor 

Context

I believe I was chosen to be the discussant for this session because of the experience and 

knowledge I have gained as a survey professional in both the federal government and the private 

sector. I spent 33 years at the Census Bureau, the last 9 of which I was the Director of the 

Demographic Surveys Division (DSD). In this position I was responsible for conducting many 

of the large scale national surveys for the federal statistical community. Three of these surveys – 

the Current Population Survey, the National Crime Victimization Survey, and the Consumer 

Expenditure Surveys – I will use as examples of change implementation in my discussion today. 

I have now been in the private sector for 6 years – 2 of which was with a marketing research 

company and the last 4 at NORC at the University of Chicago. So I have had an opportunity to 

look at survey operations from both perspectives. 

Current Population Survey Redesign – I was the senior executive at the Census Bureau 

responsible for this effort. At that time we called it a modernization effort. It began in 1986 and 

was implemented in January 1994. Among the overhaul efforts were to: 1) Revamp the 

questionnaire in its entirety, taking into account cognitive interviewing techniques: 2) conduct 
3 

the survey entirely by computer-assisted means (no longer could an interviewer just ask a 

respondent if anything changed since the last interview); and 3) develop a state-of-the-art 

processing system that would accommodate the first two. 

National Crime Victimization Survey Redesign – Although the NCVS underwent a redesign in 

1992 (while I was still at the Census Bureau), the basic design of the survey has remained 

constant through its almost four decades of existence. Revisions to the survey’s screening 

questions, because they would impact the amount and nature of the offenses measured by the 

survey were implemented in 1992. The revised crime screening questions were designed to 

better assist respondents in remembering events they had experienced by adding specific cues 

and prompts, targeting some offenses such as violence by intimates and rape, and varying the 

frames of reference. The redesign also added sexual assault other than rape as an offence 

measured by the survey. 

I was also involved in various phases of a later NCVS redesign effort since I joined NORC. The 

Bureau of Justice Statistics has initiated several research projects to support the NCVS redesign. 

Seven projects, in six subject areas began in fiscal years 2007, 2008 and 2009. A solicitation on 

sub-national estimates of criminal victimization was initiated in FY 2010. 

The subject areas covered by these projects are: 

1. Local area estimation 

2. Interview mode 

3. Sample design 

4. Survey reference period 

5. Non-response bias 

6. Crime screener analysis 

7. Sub-national estimates of criminal victimization 

NORC conducted an examination of a 12-month reference period and an analysis of possible 

non-response bias. 

Introduction of CAPI in the CE Interview Survey – The CE CAPI development project was an 

interagency effort, with management representatives from both BLS and the Census Bureau 

serving on the CAPI Steering Group. The steering group developed the strategic plan for the 

project and chartered numerous working teams that were then assigned to establish instrument 

design standards, write specifications, program and test the CAPI instrument and related 

systems, develop a new Case Management System, establish new interviewer training, and plan a 

large “dress rehearsal” to assess the impact of CAPI on CE estimates. 
4 

Planning the Change

In my opinion, this is the most critical aspect in implementing change in a complex survey 

because if you fail to plan, you plan to fail. As I noted previously, the planning for the CPS 

redesign started in 1986, 8 years before the redesign was to be implemented. In 1986, a highlevel working group that included the top leadership of both the Bureau of Labor Statistics and 

the Census Bureau convened. These experts met for 2 years, during which they formulated an 

extensive plan to modernize the CPS during the 1990’s. 

The planning for the introduction of CAPI in the CE Interview Survey was started years before 

the technology was introduced in the survey in April 2003. 

When these redesign planning groups are initiated, you have to decide on an organizational 

structure (who’s in charge of what aspects of the planning), the schedule and periodicity of 

meetings, and how to resolve conflicts when they arise which they will. 

Structure around the planning is important. You don’t want the structure so complex that 

nothing gets done, but you have to determine how to breakdown the redesign activities, what 

types and how many workgroups/subcommittees are needed, who heads up these groups and the 

process and periodicity for reporting back to the redesign leaders. The important point here is to 

ensure there is accountability and order for the planning that is taking place. 

Meetings are a time consuming and integral part of the planning process. For the CPS redesign 

effort, BLS and the Census Bureau staff met one day a week for 2 years to formulate an 

extensive plan to overhaul the CPS from top to bottom during the 1990’s. This is a big 

commitment for both the survey sponsor and the data collection organization. During this 2 year 

planning period a partnership was developed between the 2 agencies that served the redesign 

process better than a client/contractor relationship would have. Both groups had the same vision 

and a shared investment in the success of the redesign. Both groups were working toward the 

same goals and outcomes. There was a very collegial bond formed between the two agencies 

that carried forward in future years. 

There were times when conflicts arouse, however. And these conflicts were at times intense and 

passionate. As a result of the partnership that had been established and the mutual respect that 

was shared across the agencies, the conflicts were resolved. There were instances when the 

expertise of the individuals involved was in their opinion the overriding factor to be considered 

that agreement could not be reached. In those cases, BLS was the final decision maker. 

BLS has established a planning process – the Gemini Project – to plan the CE redesign. BLS 

initiated this survey redesign plan in February 2010 to account for the length of time needed to 

develop, evaluate, pilot test, and implement a large-scale redesign. If this process goes according 

to their original schedule, the Gemini Project will have a redesign roadmap in place by 2013. 
5 

Their timeline for development and pilot testing is not more than 5 years, with a new survey 

planned to be in the field within 10 years. 

The Gemini project is managed by a project leader and steering team, with guidance from an 

executive management group. The project steering team is seeking input from a panel of survey 

experts from outside BLS, who meets with the team periodically over the course of the project. 

I want to commend BLS for the thoroughness of their planning process for the redesign. 

Stakeholders

Wikipedia defines stakeholders as a person, group, organization, or system who affects or can be 

affected by an organization's actions. 

The stakeholders for the CE redesign are many and varied and include all groups involved in or 

concerned about the impact of survey design changes on the collection, processing, editing 

(including imputation and allocation), weighting, estimation, evaluation, quality, and use of the 

data. These groups include: 

BLS (Office of Prices and Living Conditions, Division of Consumer Expenditure 

Information Systems, Division of Consumer Prices and Price Indexes, Division of Price 

and Index Number Research, Division of Price Statistical Methods/Branch of Consumer 

Expenditures)

Census Bureau (Demographic Surveys Division, Demographic Statistical Methods 

Division, Field Operations, Population Division/construct experimental poverty 

measures) 

Other Federal Agencies (Bureau of Economic Analysis, the Internal Revenue Service, 

and the Department of Defense) 

Congressional committees request special tabulations on issues such as the potential 

impact of increases in the minimum wage 

Office of Management and Budget (OMB needs to be consulted early and throughout the 

planning process) 

Academic community and research organizations for analytical work 

Private companies for marketing their products and services 

Prospective entrepreneurs for developing their business plans 
6 

Once you identify the stakeholders that will influence (either positively or negatively) the survey 

redesign, you need to anticipate the kind of influence, again positive or negative, these groups 

will have on the initiative. Finally you need to develop strategies to get the most effective 

support possible for the redesign and reduce any obstacles to its successful implementation 

(managing expectations). 

BLS has already initiated or planned a series of information-gathering meetings, conference 

sessions, forums and workshops to address this whole stakeholder involvement. These events 

include: 

Conference on Research in Income and Wealth (CRIW) – National Bureau of Economic 

Research (NBER) Conference on Improving Consumption Measurement (July 2009) 

Survey Redesign Panel Discussion, co-sponsored by the Washington Chapter of the 

American Association for Public Opinion Research (DC-AAPOR) and the Washington 

Statistical Society (WSS) (January 2010) 

Data Capture Technology Forum (March 2010) 

AAPOR Panel Presentation on Utilizing Respondent Records (May 2010) 

Data Users’ Needs Forum (June 2010) 

CE Methods Workshop (December 2010) 

Household Survey Producers Workshop (June 2011) 

CE Redesign Options Workshop (October 2011) 

BLS has done a commendable job in stakeholder outreach and has gone beyond the outreach that 

was done for the CPS modernization effort. 

BLS will need to add more of these information sharing events in the future once they decide on 

a final design option and get the results from the redesign testing phase. 

Staffing

For the CPS, separate staffs were established to conduct the ongoing monthly survey and the 

survey redesign activities. Although senior staff was involved in both activities, the day-to-day 

operations of both were conducted by separate staffs. In this way there were no conflicts in 

priorities and the schedule and operational aspects of both operations were maintained. 

Testing and Implementation

The questions to address here involve: 
7 

What to test? 

Is each change evaluated separately? 

Is an overlap survey required? 

Providing new technology to survey respondents 

How to phase-in the redesign? 

Whenever significant changes are made in an ongoing survey operation, there is always the 

expectation that those changes will affect the characteristics of the data obtained. It is important 

to measure any such effects. A large number of survey design features are proposed to be 

changed in the CE redesign and a number of them, alone or in combination, could result in 

significant changes in the estimates. BLS should want to be able to explain to their stakeholders 

why differences between the new and old series occur and to comment on whether any changes 

reflect improvements in the quality of the data. They also need to understand from a scientific 

point of view the effect of different design features on consumer expenditure estimates. A third 

reason for wanting to know the reasons for the differences between the two surveys is to use the 

information diagnostically to ensure a smooth transition to the implementation of the redesigned 

CE in 2020. 

The RFP requested recommendations for the design of a study or studies to evaluate the 

feasibility and effectiveness of the proposed redesign option. Both options suggested further 

investigation and experimentation in areas such as: 

Software development 

Length of the diary period 

Effects of the use of technology 

The extent and level of incentives 

The level and frequency of interviewer-respondent interaction 

Collection of administrative data 

While the RFP did not request a test to compare the redesigned survey to the current production 

survey, one of the option papers suggested that such a test should be conducted. The field 

experiments mentioned above can help compare the redesigned methods with the current 

methods. However, a comparison using representative samples and a full data collection cycle 

would provide a more rigorous test of the new data collection methods, any improvements in 
8 

data quality and the effect on the survey estimates. These types of comparisons are expensive 

but if funding becomes available, I highly suggest that BLS consider such a test. 

One of the redesign options proposes to provide each household with a computer tablet, barcode 

reader, and a portable scanner for recording their information. They estimate that the data 

collection contractor would need 6,000 to 8,000 sets of these devices per month (this allows 

flexibility in the delivery and pick up of the units as well as replacement units and sufficient 

numbers of units to accommodate geographic dispersion of the sample across the U.S.). These 

units will require service warranties and/or insurance. Broadband internet access will be 

required for all the consumer units. Finally, the consumer units will need to be trained to use this 

technology. 

Providing these devises to respondents will add a level of complexity to the redesign 

implementation. An infrastructure will have to be developed to procure the devises, get them to 

interviewers to be placed in the consumer unit, train the respondent, arrange for insurance or a 

warranty and to provide broadband internet access. It’s not clear how payment for the internet 

access will be handled – direct payment to the provider or reimbursement to the respondent. An 

inventory control system will be required to track the devices over time as they move from 

respondent to respondent and through the maintenance system. All of this is doable; it just adds 

an extra level of complexity to an already complex survey operation. 

Another issue involved in using this new technology is the repository that will be required to 

collect, house and merge the information for each consumer unit. This repository will receive 

information in many different forms (paper/electronic/scanned) and a thorough test of the 

operation should be conducted to ensure the right information is connected to the appropriate 

consumer unit. 

Finally, the most concerning aspect for me in the use of technology for collecting the expenditure 

information is IT security. Consumer units will be transmitting personally identifiable 

information (PII) over unencrypted lines. The data repository will house a large amount of PII. 

There are physical as well as IT security issues. BLS pays a great deal of attention to protecting 

a person’s PII. They also require continuous monitoring and security requirements for NIST 800-

53 Rev. 3. If technological solutions are adapted for the redesign, much attention and resources 

will need to be devoted to IT security. 

Funding

A redesign effort that will extend over 10 years from planning to implementation will require a 

sustained funding stream. The current political landscape is uncertain and funding for new 

initiatives is unclear. A change in Administrations during this time period is possible and a new 

Administration may bring different priorities and visions for how the federal budget should be 
9 

allocated. It would be a travesty to begin the redesign effort and the required level of funding is 

dramatically reduced or eliminated during the lifecycle of the redesign process. 

Funding for the CE Survey modernization effort was included in the FY 2011 budget and the 

President’s request for FY 2012. It appears that $8.8 million is requested in the 2012 budget to 

modernize the CE but it’s unclear from the budget narrative how this funding is to be allocated. 

Cost

One of the Gemini Project objectives states that the costs of a redesigned survey cannot exceed 

current budget levels. To allow for an unpredictable budgetary environment, BLS is planning to 

address both a complete redesign of the CE surveys as well as more limited modifications to the 

current design. 

Both option papers provide cost estimate projections for a total redesign of both the quarterly and 

diary surveys. The cost projections provided are 2 to 3 times greater than the current ongoing 

program costs. Unless the budget climate changes in the future, the cost objective stated above is 

not attainable. BLS will have to make choices about which components of the two options are 

the most useful to achieve their objectives based on future budget availability. 

Change in Contractor

The Census Bureau has been conducting the CE surveys since 1980. One must assume that BLS 

has been satisfied with the responsiveness of the Census Bureau and the quality of the program 

deliverable for this relationship to continue for this long. However, is it time to evaluate whether 

a change in the data collection contractor is warranted given the changes that are being proposed 

for the surveys’ redesign? 

I would suggest the following scenario. Once a redesign option is chosen and the requirements 

for its design, testing and implementation are decided, BLS should request the Census Bureau to 

develop a technical approach and a cost estimate based on that plan. BLS should also release a 

competitive RFP with the same requirements and statement of work for the private sector to bid 

on.

BLS loses little following this approach other than the staff time required to develop and 

evaluate the proposals that are submitted through the RFP process. They will use the same 

quality metrics, proposed management plans, staff qualifications, and past experience to evaluate 

the technical feasibility and costs of the Census Bureau’s and private sector proposals. In this 

way they will have the information needed to make an informed decision.