other research that might be done, the cost-benefits of such an experiment, I

think, would be rather high.

Ambiguous Qu ons

The discussions we had at the conference suggest that a number of ques-
tions in both the CES and the CPS may be misunderstood by respondents. [In
fact, we really don't know much, if anything, about how the questions in these
surveys are interpreted. Here I would recommend, as others have, that the BLS
sponsor a series of laboratory and field experiments on how respondents
interpret the standard CES and CPS questions. At the Behavioral Sciences
Laboratory (BSI) in Cincinnati we have experimented with the use of "think-
aloud" (or "talk-aloud"), personal interviews to discover how respondents
interpret survey questions and how they arrive at their answers. [ think this

ful in revealing how respondents answer

intensive technique would be quite us

the CES and CPS interviews. Such research has, of course, already been

recommended for the C. the BLS-Census Bureau Questionnaire Design Task

Force. The same thing should be done with the CES.

Because the CES interview is rather lengthy, I would recommend breaking it
up into perhaps a half a dozen or so smaller interviews of no more than an hour
in length (each of which must be taped and transcribed). My experience with

doing such interviews here at the BSL is that they can be quite demanding of

respondents, and so an hour is about the upper 1 t on their patience and
concentration. or content analysis of these think-aloud protocols, I would

recommend at least 200--preferably 300 or 400--cases for each one of the mini-

interviews. You need this many cases to discover most of the variation in


As in the use of the quarterly panel interviews, I would recommend a field

experiment in which CES households are assigned to one of two conditions:

1. A standard diary interview with a responsible adult informant keeping

track of all the purchases for the CU, and

An interview in which every member of the CU, age 12 or over, keeps a

diary for a two week period.

Recent studies by Teen-Age Research Unlimited, Inc., a Lake Forest,
Illinois firm that monitors teen consumer behavior found that teenagers between

12 and 19 years old were directly responsible for roughly $40 billion of their

families' purchases in 1986 (data cited in a Cincinnati Enquirer story by
Patricia Gallagher, February 15, 1987, D-1). [ suspect that many of these
expenditures are not accurately reported in the CES and may account for a
significant part of the underreporting on such items as clothing and food at
home, as teenagers make many such purchases according to the recent surveys by
the Lake Forest firm. So having everyone age 12 or over keep a diary would
very likely increase the accuracy of CES reports, though it would certainly
increase the cost of data collection significantly. So I recommend a field

experiment to determine the cost-benefits of such a change in the standard

diary survey,

nk it would also be useful to experiment with reminder telephone

calls to diary respondents in the early evening hours (7-9 P.M.) each day or
every other day, The interviewer would call to find out if the respondent has
any questions about completing the diary. This should help maintain a respon-
dent's motivation to take the task seriously. Perhaps a phone call reminder

6


more accurate had my wife participated in the interview with me. I believe
this observation would hold true for the great majority of households with more
than one person over the age of 12. For this reason, I would recommend a field
experiment in which households for the CES are assigned to one of two condi~

tions:

l. A standard CES interview with a responsible adult informant reporting

for the entire household, and

2. <A CES interview with all household informants, age 12 or older--
preferably with all members of the CU present simultaneously, as this

would provide an immediate check for validity on each other's reports.

This is a relatively straightforward experiment that could tell us rather

quickly whether signi ant improvements in reporting on the CES can be
achieved by greatly reducing, or eliminating altogether, the use of proxy
reports. The Bureau would then have to judge whether this change would be

;

worth the cost--assuming there were significant improvements in reporting (as I

think there would be),

The experiment should be done in at least two locations with probably no
less than 1000 househoids at each site (300 in each experimental condition).
This should he sufficient to tell us whether we can expect significant improve-
ments on the preblems of underreporting (e.9., on apparel and household
furnishings), and in accuracy of reporting more generally. This is a very

feasible experiment, and I highly recommend that it be done, Compared to some

te


how respondents interpret the questions. The analysis of such protocols should
suggest various ways in which existing CES and CPS questions can be revised and
elaborated to produce more accurate reports. So I think it is well worth the

investment to pursue this research.

Recall Problems

The analysis of the think-aloud protocols should also yield some useful
insights into how respondents retrieve information from memory to answer the
CES and CPS interviews. Several panelists at the conference have, moreover,
proposed some ways of improving recall on difficult questions (e.g., the use of
birthdays, holidays, etc. as bounding events). Though I think such research
may be worth doing, I am not optimistic that it will lead to significant
improvements in reporting on the CFS because of the length of the recall

atest under-

period: three months, For the items that are subject to the
reporting (e.g., apparel and house furnishings) a three month recall period is
just too long and difficult for most respondents to handle. Adding new
questions or statements to aid their recall of this material (i.e., eliciting
landmark events) will only add to an already lengthy and burdensome interview
for perhaps only modest improvements in recall. I could very well be wrong,

S

but I think that significantly greater improvements in reporting in the
(and the CPS) can be achieved by reducing or eliminating proxy reports than by
focusing on ways to improve recall for a three month time period (or for even a

one-month period).

Furthermore, as I indicated at the conference, the BLS should seriously

consider eliminating the quarterly CES interviews and reallocating those


the first day or two of each week would be sufficient to accomplish this
purpose. This type of experiment can easily be done and should probably be
part of a larger factorial experiment on the effects of having all members of

the CU, age 12 or over, keep a diary.

Most of my recommendations have focused on the CES because it appears to
be subject to significantly greater response error than the CPS. The report of
the BLS-Census Dureau Questionnaire Design Task Force also seems to have
addressed rather well the cognitive and conceptual problems in the CPS ques~
tionnaire (a similar task force should do the same for the CES). In addition,
T would recommend--as with the CFS~-a controlled field experiment on the
effects of using proxy reports. Interviews with everyone age 16 or over in the
household should he compared with standard CPS interviews to assess potential
improvements in reporting from this straightforward, though costly, change in

the present mode of data collection.

I think many of the problems of accuracy in the CPS could be addressed by
having respondents xeep a diary of their work hours each day, reasons for not
working, svecific job search activities, etc. This mav seem like another
burdensome task, but I think it would be much easier to do than the CES diary.
A small-scale experiment with 500-600 respondents (half of whom are currently
unemployed) would tell us whether we might expect to find significant improve-
ments in reporting in the CPS, It is perhaps not as glamorous as some other
research we might do, but it is quite practical and could produce some rather
large payoffs in the measurement of the labor force in the near future.
Whereas other things we might do may take quite a few years to produce a

payoff, if any at all.


University of Cincinnati Behavioral Sciences Laboratory
Institute for Policy Research

Mail Location 132
Cincinnati, Ohio 45221-0132

February 23, 1987

Ms. Cathryn S. Dippo

Senior Mathematical Statistician
Office of Research and Evaluation
Bureau of Labor Statistics

U.S. Department of Labor

441 G Street, N.wW.

Washington, D.C. 20212

Dear Ms. Dippo:

Enclosed are two copies of a summary report containing my recommendations
for redesigning the CES and the CPS.

I enjoyed meeting you and your colleagues at the conference which I
thought was most worthwhile.

I look forward to hearing from you on your plans for the CES and the OPS,
If you have any questions, feel free to call me at 513-475-5028,

Sincerely,

Georg® Bishop
GB/k1lb

Enclosures


resources to the diary survey, as there is evidence that the reports in the \

latter are generally more accurate than those obtained in the quarterly panel

’

interviews. I fully realize that there are also problems with the quality of
the diary reports, but they seem to me to be far more tractable than those

associated with the recall problems in the quarterly panel interviews.

Respondent Motivation

Mere I think the use of techniques developed by Cannell and his associates
at Michigan--explicit instructions to the respondent on how to report, commit—
ment to being a 300d respondent, feedback and reinforcement procedures, etc.--
would be quite helpful in improving respondents’ reports in both the diary and
panel interviews of the CES, as well in the CPS interviews. Controlled

experiments with these procedures are well worth doing. And I leave it to

Cannell and his colleagues to propose that work.

o think serious consideration should be given to experimenting with

financial incentives for respondents to complete the diary survey, for example,
as accurately as possible. [It is a burdensome task for many respondents, and
so financial compensation might well motivate them to take it more seriously

than they now do. Specifically, I would suggest controlled experiments with

we
a

and without financial compensation for completing the diary (e.g., for » 510,
or $25) to determine whether it results in significant improvements in report-

ing This may not be as theoretically or methodologically exciting as some

other ways of improving reporting, but it may work much better than anything

else that we mi

ht try. So I think it is worth experimenting with, at least on

a small scale with perhaps 300-400 respondents.

wn



‘DATIONS ON REDESIGNING THE
CONSUMER EXPENDITURE SURVEYS
AND THE

XURRENT POPULATION SURVEY

by

Seorge F, Bishop
Pehavioral Sciences Laboratory

Institute for Policy Research
sniversity of Cincinnati


University of Cincinnati Behavioral Sciences Laboratory

institute for Policy Research

Mail Location 132
Cincinnati, Ohio 45221-0132

I think much of the systematic response error in these surveys stems from

four sources:

l. the use of proxy respondents to report for other members of the

household,

2 the ambiouity of various questions in both the CES and CPS,

3. the difficulty of recalling the information that is required by many

ture

of the questions, varticularly those in the Consumer =xpe

Survey, and

4, the problem of getting respondents sufficiently motivated to report

fully and accurately the information that is needed.

Proxy Reporting

Rased on the discussions we had at the conference and mv experience as an
interviewee, I think the use of proxy reports on the behavior of other members
of the household is probably the major source of inaccuracy, particularly in
the Consumer Expenditure Survey (CES). A number of the household purchases I

had to report on, for example, in my CES interview would have been noticeably