ou

effectiveness is measured by testing hypotheses of probable direction of bias. If evidence is
that the variable is likely to be under-reported, the criterion is “more is better... Two or
moe alternative procedures can be tested and compared. Experimental groups of 1-200
respondents are usually adequate if the variabie is gencral in the population. In our studies

\ We usually select blocks from Census tracts that are contrasting in variables which we think

may affect reporting — age, educatonal level. for example. Within each block, addresses are

jrandomly assigned to each treatment group. This is an efficient design for identifying major

differences. Based on this study, a questionnaire can be prepared for larger-scale field

testing.

3. Validity tests - While the criterion of “more is better” is reasonable for evaluating

newly developed experimental techniques. it is not adequate to assume validity. More may
not be better for some variables. Purchases of “prestige” items and behavior perceived as
desirable may be over-reported. Moreover. while tests may demonstrate one procedure as
being significantly better than onother. it says little about its validity compared with
“truth.” A procedure may produce an improvement in reporting of 20 peredent and still be
50 percent under the true figure

The best validation is of course a comparison of reports in surveys with some record
that can be presumed to be accurate. The purist will argue that in studies based on a
sample from records. only under-reports can be measured. For many variables it seems

‘probable that the direction of bias is failure to report rather than to over-report. While

* J studies cannot provide estimates of the net survey bias, the under-reporting bias at least can
OF AN : ; ;
.  \’be ‘measured. For some variables, especially in CPS. the under-reporting hypothesis is le:

acceptable and one needs to consider special kinds of records which permit an assessment of
both under- and over-reporting.

7 It is not feasible .o attempt validation for large numbers of variables in one study
One can, however, select some important variables for which records are available for
checking. The records may wel not be representative of a meaningful population but they
can produce useful estirnates.

Where not available, a record set might be generated. It may be possible to convince


questions. The effects are devastating to data quality. Respondents do not have adequate
time for recalling information and. perhaps more serious. the impression is communicated by
the interviewer that the goal is simply to complete the questions with little interest in
accuracy or completeness.

The responsibility for too rapid a pace rests partly with the interviewer but more with
the researcher. The interviewer must be provided with an instrument that can be
completed in a reasonable time in a thoughtful. relaxed atmosphere.

Developing, Testing. and Evaluating

Large and complex surveys such as CES and CPS present many research isues to
whet the appetite cf methodologists. Several topics were discussed at our meetings arising
from reports of field experiences; the reports raise the familiar issues of effect of lapse of time.
telescoping, bounding effectiveness. self proxy reports, etc. One can visualize a program of
methodological studies an question wording, questionnaire design, and interviewing
techniques directed at improving reporting of the major content variables

A testing program might focus first on a few topics of special importance that appear
to be pourly reported. Selecting variables that are likely to reflect a variety of response

errors will permit results to be generalized to other variables of like type. A sequence of

af \ctesearch states can be considered, beginning with some pilot studies to develop hypotheses
for effective techniques.

1. Pilot studies - As mentioned at the meetings. investigations that encourage
respondents to introspect and report their thoughts. frames of reference. perceptions. and
comprehension would provide insights to enable the researchers to develop questions that
reflect respondent's orientations. Through this procedure one may both simplify the
reporting task and improve the quality of information reported. The outcome of these
investigations is a set of questions or. more likely, more than one set of questions. reflecting
hypotheses of effective methods for measuring the variable. The second stage involves
experiments to test and compare the various procedures under field conditions.

2. Field experiments - The objective of these experiments is to test the methods and

procedures developed in the pilot studies. No measures of validity are usually available, and


a store to keep records that can be used as a sample frame. From such studies one can
make estimates of undet-reports for the entire sample and can examine differing response
biases by characteristics of respondents, self. proxy reporting, decay over time, and effects of

social desirability on reporting.

4. Behavior coding pretest interviews — Diagnosing problems with survey questions

by coding interviewer and respondent behavior is receiving growing attention. The
\ \.. technique is based on identifying behaviors that indicate the interviewer or the respondent is
\

having a problem. The technique also is used to monitor interviewer performance. providing

an opportunity for immediate feedback to correct interviewing errors.

Frequently difficulties with questions are manifested in the behavior of the interviewer
or the respondent. Respondents may ask what the question or a word means, or may
suggest an interpretation of a phrase. The problem may be signaled by a response which is

not relevant to the intent of the question or which provides only a partial response, or by

some other form of inadequate answer. Interviewers may need to engage in more “probin
activity to obtain an adequate response: they may repeat the question, explain, define. etc
Simply the number of exchanges between interviewer and respondent often indicates

difficulties. Several interviewer-respondent exchanges to a question requiring only a “yes” or

“no” response suggest confusion or misunderstanding. Awkward wording and complex
sentence structure cause problems for the interviewer as well as for the respondent

Coding systems exist that are readily adapted to any questionnaire. Numeric codes
i | bee are designed for all major behaviors: for example. codes for question-asking: 1. asked exactly
wv as written, 2. asked with minor changes, 3, asked with major changes. There are codes for
respondent's request for clarificaton. inadequate responses, “don’t know.” etc.

Coding 50-100 interviewers yields frequencies of behaviors by type for each question

Distributions are examined to identify questions with problems. The nature of the problem

tem is especially adapted to

is often identified by studying the behavior patterns. The sy

‘ telephone interviews which are monitored on line. Feedback can be given to interviewers on

their performance immediately following the interview. Analysis of problem questions are

based on the same (but expanded) code system.


Charles Cannell
January 1987

Some Comments on Methodological Issues
Bureau cf Labor Statistics Questionnaire Design Advisory Conference
January 15-16, 1987
Many topics reiating to the validity of the data from CES and CPS were raised during
our two-day conference. The major issue is the iength and complexity of the CES

questionnaire. In my opinion these present sufficient barriers to response validity that

methodological innovations can lead to only minor improvements without substantial

reduction in length. The questionnaire poses a very heavy task demand on the respondent.

and on the interviewers. Because of the perceived time pressure, both interviewer and
respondent tend toward a fast pace. with the respondent answering rapidly and often with
little thought. while the interviewer is poised to check a box and rush to the next question.

ven to shortening and simplifying the questionnaire. What

rious consideration must be

variable can be omitted. or what detail can be eliminated” Can a redesign be created with
central topics in all questionnaires and modules included in sub-samples?

The respondent's job in surveys often is more difficult than researchers recognize. To

respond adequately the respondent must

Understand the question and comprehend its meaning as the researcher
intended. For example: “Last week. did you have a job from which you
were temporarily absent or on layoff?” The respondent must remember
the definition of “last week:” the definition of “job” (which may be
difficult for some respondent); what is meant by “temporarily absent” —
took a couple of hours off to go to the doctor. does that count?” “The
boss said work was slack and if I wanted Friday off it was OK. {s that a
layoff?” The respondent needs to answer such questions him herself or
ask for clarificaton from the interviewer. Time and effort is required by
the respondent in trying to grasp what is wanted

Engage in cognitive processing of information. Recalling information.
checking one s recail tor accuracy. competence. and for the correct
reference periods require careful thought and effort. This requires time
and effort

For adequate performance. the respondent needs to understand what is wanted. have the

skills and procedures for retrieving and organizing the information, and be sufficiently

motivated to perform the task conscientiously


~

A Final Note

The material sent prior to the meeting included several methodological articles and
memos prepared by BLS staff members comparing estimates from CES and CPS with other
data sets. Other reports examined apparent under-reporting over time, and for a varity of
topics. These studies make a valuable contribution to the survey methodological literature
They both replicate findings from other content areas and add new information

In order for survey research to improve the quality of its data, methodologists need to
share in each other's findings and hypotheses. I urge that more of BLS methodological

studies be shared through journal publication


1

The papers sent to us in preparation for the conference demonstrated that these
surveys share general survey problems of the decay in reporting over time and apparent
failure to report some kinds of purchases. The simple explanation is that these events are
forgotten. Forgetting implies that the information is no longer accessible. But suppose that
respondents were promised $500 if they could report those events accurately. The
probability is that a large proportion would be reported and that the effect of time lapse on
reporting wouid be significantly reduced. The implication is that such events are accessible
if the individual puts forth the effort required. This leads to the conclusion (which is
supported by some of our research) that reporting can be improved by making the task
easier to perforin and by getting the respondent to exert greater effort. Research to improve

to attend to both reducing task difficulty and to motivating the

the quality of data needs

respondent to greater effort. The two are closely interrelated and interactive

Some Comments on the Questionnaire
and the Respondent Role

‘The present CES and CPS questionnaires presume a passive respondent role: that is.

the respondent is a repository of information and the interviewer's task is to dig it out.

There is little in this role prescription to excite or energize the respondent. To borrow from
computer jargon. the questions should be more “respondent-friendly.” sound less like an
abstract from a statistical table and more like a topic in which the respondent can get
involved. A productive role is that of a cooperative enterprise. The respondent and the
interviewer are cooperating in imortant research in which accuracy of information is
paramount and both share the responsibility for the success of the interview

The focus of the respondent's role (as well as that of the interviewer) is on task
performance, and techniques are designed to foster that role in preference to a personal
friendship pattern that sometimes characterizes survey interviews.

For most people. being a survey respondent is a new experience. Respondent
behavior can be improved by increasing their knowledge of the repoting task — what is
expected of them and how to behave. This can be accomplished by adding instructions to

the questionnaire. The following is a general description of the use of instructions to develop


the respondent role.

“The idea behind the use of instructions is simple. To perform a task adequately, one
needs to know what is expected: what the task is and how to perform it. If this is
not known already. an efficient way to transmit this information for many tasks is

\__ through explicit. verbal instructions that clarify goals by specifying what is to be

accomplished and by setting forth the criteria for good performance. Moreover.
instructions clarify the Specific actions involved in achieving the goals....( Another)
function of instructions in the interview is to detail information. As is true for the
goal of the interview. respondents typically are given very little information about
how to answer questions adequately. Instructions about the operations needed for

‘adequate performance include such things as carefully considering each queston, being

diligent in retrieving and organizing iinformation needed for responding, and

requesting clarification if a question is not understood. Instructions may also point
out the importance of considering various frames of reference and taking all the time
necessary to answer adequately....In addition to these genera! guides. instructions can
be tailored to the reeds of specific questions. Such instructions can clarify the

objective of a question or suggest efficient ways to retrieve relevant information for a

particular question.” (Cannell. Miller, and Oksenberg. “Research on interviewing

techniques.” Chapter 11 in Leinhardt (ed.}, Sociological Methodology, 1981. San

Francisco. Jossey-Bass, 1981.)

The questionnaire thus becomes both a set of questions and specifications for
respondent role behavior, and provides specific cues to assist the respondent in effective
reporting techniques. Not only do instructions clarify and direct respondent performance
but they appear to be important in motivating desirable respondent behavior

The Interviewer Role

The inclusion of instructions in the questionnaire focuses the interviewer's role directly
on the reporting task, Since both instructions and questions are to be asked verbatim, the
interviewing behavior is more nearly standardized among interviewers.

ial attention is paid to feedback procedures. The combination of instructions to

inform the respondent of what is expected and feedback to inform and reinforce good role
behavior teaches role behavior. Exampls of feedbacks that focus on the task include:

-*That’s useful information for our research.”
--Thanks, that’s helpful.”

Feedback also helps to motivate good performance through its reinforcing properties.
Pace

Regardless of how good the questionnaire and interviewing technique are, their
effectieness is lost if the interview pace is too fast. When the interviewer thinks that the

interview is longer than the respondent will tolerate, the tendency is to speed through the