aL

completed the questionnaire so we could visit. Other agencies
who conduct- business pretests go to the firm and have the
respondent fill out the questionnaire while the agency
representatives wait (some representatives leave and come back in
several hours). We did not want to use this approach because of

potential bias.

The ETJO staff considered the pretest useful as it identified
potential problems in the questionnaire which the interviewers
could be prepared to address during follow-up telephone data
collection, such as how to calculate average wages. Pretesting a
form of this scope was really a "dress rehearsal" of the
questionnaire. To pretest an establishment survey we needed a
questionnaire (not simply questions, as the survey was not of
that format) from the start. Therefore, we spent a great deal of
time constructing it before pretesting, and relied on past
literature and focus groups to circumvent potential problems. We
addressed some pretest problems through questionnaire redesign.
Other pretest problems were addressed through the supporting
survey materials mailed to respondents, interviewer training, and

the response analysis survey.

For the ETJO Survey, we also conducted an operations test of 100

units. The purpose of the operations test was a "dress


new hires in a month divided by the number of jobs open at the
end of that-month. Economic theory suggests that shortages in
competitive markets will be accompanied, at the margin, by rising
wages. Therefore, data were collected in the survey on the wages
of new hires--the wages most directly affected by market
conditions. Data on separations allow the data on job openings
to be understood more completely in relation to turnover in an
occupation. The ETJO survey team, therefore, decided on the
collection of five data items by detailed occupation: number of
job separations, number of new hires, average wage of new hires,
number of job openings, and the length of time the jobs had been

open.

The project staff conducted two focus groups to obtain expert
opinions on labor shortages and to assess tentative ETJO concepts
and definitions.5 The focus groups followed a specific protocol.
First, to start the discussion, group members were asked to think
of examples of occupations where there might be a labor shortage
and common characteristics of hard-to-fill jobs. Second, we
asked about concepts and indicators--what constituted a labor
shortage and what variables were necessary to measure shortages.
Third, we presented the tentative ETJO definitions and asked for

comments. Fourth, we inquired about availability of records--we


25

appeared to correctly report employees who were laid off. This
suggests that instructions for inter-establishment transfers need
to be included in the column heading, just as with layoffs. A
future RAS should investigate the number of employees involved to
see if the omissions would significantly affect the survey
estimates before adding it to the column headings. Third,
difficulties in reporting the average hourly wage of new hires
arose when establishments paid employees by the mile, percent of
load, or percent of revenue or commission. The motor freight
transportation and warehousing industry had the most difficulty
with the reporting of the average hourly wage. Our
recommendation was that a future survey provide specific

instructions for each industry.

Finally, establishment respondents were asked how much time they
had spent to complete the survey and how likely they would be to
participate in a future survey. On average, the large
establishments spent five times (61 minutes) as long as the small
establishments (12 minutes), and about two times as long as the
medium-sized establishments (27 minutes). Those establishment
respondents who chose "not very likely to participate in the
future survey" (12 percent of respondents) had a higher average
time of completion, 34 minutes, compared to 29 and 25 minutes for

those responding somewhat likely to participate (23 percent of


literature on household surveys is not always applicable to

establishment surveys since the two differ in several respects.

First, respondents in establishment surveys answer in their role
as employers or employees of an establishment, primarily during
working hours. Since we interrupt their working day, the issue
of response buxden is, therefore, of critical importance. Large?
establishments may have employees whose responsibilities include
completing forms for government and other surveys. However,
respondent burden is still very much a concern to large
establishments, as they are selected into many surveys. Response
burden is also a concern for respondents in small establishments,
such as the owner or office manager, who tend to have less
flexibility in their schedule due to many competing

responsibilities.

Second, professional terminology with qualified definitions is
more likely to be used in establishment survey questionnaires.
For example, the term "internal transfer" is used to describe the
transfer of employees from one establishment to another within
the same firm. Dillman (1978) cautiously supports the use of
precise language for respondents who share a particular
vocabulary. He notes that when surveying physicians it is

clearer to talk about "pharmaceutical companies,” instead of


33

Konschnik, C.A., Monsour, Nash J., and Detlefsen, R.E. (1985),
"Constructing and Maintaining Frames and Samples for
Business Surveys," Proceedings of the Section on Survey
Research Methods, American Statistical Association, 113-122.

Krueger, R.A. (1988), Focus Groups: A practical Guide for Applied
Research, Newbury Park, CA: Sage Publications.

Merton, R.K., Fiske, M., and Kendall, D.L. (1956), The Focused
Interview, New York: Free Press.

Morgan, David L. (1988), Focus Groups as Qualitative Research,
Newbury Park, CA: Sage Publications.

Plamisano, M. (1988), "The Application of Cognitive Survey
Methodology to an Establishment Survey Field Test,"
Proceedings of the Section on Survey Research Methods,
American Statistical Association, 179-184.

Plunkert, L. (1981), Job Openings Pilot Program: Final Report,
Internal Report in the Bureau of Labor Statistics.

Scott, S. (1980), "Reinterview Methods in an Establishment Survey
of Job Openings," Proceedings of the Section on Survey
Research Methods, American Statistical Association, 445-449.

Sergean, R. (1958), "The Response of Industrial Firms to an
Approach by Letter Questionnaire," Occupational Psychology,
32, 73-85.

Sharp, L. and Frankel, J. (1983), “Respondent Burden: A Test of
Some Common Assumptions," Public Opinion Quarterly 47, 36-
Sos

Stewart, D.W., and Shamdasani, Prem N. (1990), Focus Groups:
Theory and Practice, Newbury Park, CA: Sage Publications.

Sudman, S., and Bradburn, N. (1973), "Effects of Time and Memory
Factors on Response in Surveys," Journal of the American
Statistical Association, Vol. 68, No. 344, 805-815.

Templeton, Jane F.(1987), Focus Groups: A Guide for Marketing &
Advertising Professionals, Chicago, IL: Probus.

Werking, G, Tupek, A, and Clayton, R. (1988), "CATI and Touchtone
Self-Response Applications for Establishment Surveys,"
Journal of Official Statistics, Vol 4, No. 4, 349-362.


rehearsal" of the data collection system. The five-member
operations test staff - college students, professional, and
clerical support staff - collected data from September 24, 1990,
through October 26, 1990 from 100 randomly selected units from
the eight industries. The test resulted in a usable response
rate of 73 percent. The test findings suggested the strong need
for highly skilled, well-trained interviewers. The range of
occupational definitions, industry coverage, and the several
categories of information asked for, required a great degree of
interviewer judgment. The majority of the survey forms received
by the mail did not have the totals line filled out. The test
also revealed that more respondents had to be contacted by phone
than anticipated, both to collect initial data and to reconcile
or complete data reported by mail. Based on these findings, many
refinements to the processing programs as well as screen design,
system flow, and scripting were made. The data collection staff
streamlined the collection procedures and gained experience with

the industries and occupations respondents.

Response Analysis Survey

The data generated by the ETJO survey were evaluated by a
response analysis survey (RAS) conducted by telephone from April
through June 1991. The purpose of the RAS was to assess the

quality of the ETJO survey data, including several types of


28

helps the survey researcher identify difficulties in respondent
comprehension and recall, unclear terminology and instructions in
the questionnaire, and incompatibilities between records systems
and survey definitions. The response analysis survey identifies
problems similar to those of the pretest, but on a larger scale
so the possible impact on survey estimates can be assessed.
Additionally, the RAS measures the response burden in terms of

time.

To increase our understanding of establishment survey error, more
research needs to be conducted to determine the exact source of
measurement error and the interrelationships among the sources.
Pretests are helpful in this matter, as are response analysis
surveys. For example, after identifying a particular error such
as underreporting in a response analysis survey, BLS researchers
use a probe method to determine if the respondent did not recall
certain items, if the respondent thought they should be excluded
rather than included, or if the record system did not include the
items. If we know the error and can isolate reasons for it, we

can implement appropriate solutions.

More research also needs to be conducted on understanding

participation in establishment surveys. The pattern of lack of


24

records, combined with memory or another source, were likely for
respondents using three sources of information. The respondents
who relied exclusively on memory were concentrated in the small
establishments (1-49 employees) that could more easily follow

day-to-day business operations.

Interviewers then asked questions about respondents’
comprehension and interpretation of instructions and definitions.
In general, respondents followed instructions and definitions,
although further investigation was suggested in three areas:
adherence to two reference periods, the treatment of inter-
establishment transfers, and the calculation of wages under non-
standard pay schemes. First, respondents had the most difficulty
with the reference period of the last business day of the month
for job openings, even though we added this information to the
column heading of the questionnaire after the pretest. Nearly
all respondents indicated that they correctly reported
separations and new hires for the entire reference month. Ina
future questionnaire, we recommended using a consistent reference
period, as we felt that respondents would continue to overlook
the dual time frames. Second, respondents undercounted internal
transfers when calculating employee separations and new hires.

Of the 22 establishments which had transfers within the month, 12

did not include them in their figures. In contrast, they


34

Wright, Patricia (1980), "Strategy and Tactics in the Design of
Forms," Visible Language, 14, 151-193.

Wright, P., and Barnard, P. (1975), “Just Fill in this Form -- a
Review for Designers," Applied Ergonomics, 6.4, 213-220.

(1978), "Asking Multiple Questions
about Several Items: the Design of Matrix Structures on
Application Forms," Applied Ergonomics, 9.1, 7-14.



19

All respondents indicated they thought that survey instructions
were clear and concise, and the form was straightforward, i.e.,
they understood what we were requesting. Yet, when we probed
further as part of our protocol procedures, we isolated problems.
One respondent reported job openings for the entire month, rather
than the last business day of the month, a mistake that might
cause job openings to be inflated. That prompted us to put
explicit reference times in the column headings of the

questionnaire.

Respondents commented that they felt the questionnaire was well-
documented and had few difficulties adhering to the definitions.
Again, probing further, we uncovered certain difficulties. One
establishment respondent indicated he posted salaried job-vacancy
announcements before an employee leaves. Since the establishment
did not have any salaried jobs open at the time of the pretest,
there was no error. However, it raised the possibility of error,
as our definitions instructed respondents not to include the job
as an opening until the employee leaves. Another respondent
noted difficulties reporting the average hourly wage of new hires
since drivers were paid by the mile. We decided to address these
and a few other potential problems in the response analysis

survey to assess their scope.


11

Pretesting of OEUS questionnaires generally involves traveling to
establishments to conduct in-depth interviews with respondents.
Cognitive research methods, such as retrospective and concurrent
think-aloud, open-ended questioning of the respondent about the
completed survey responses may be used. Also, respondent
debriefings after the questionnaire is completed provide
information on the ways respondents reach an answer. The areas
of questioning for the pretest include respondent comprehension
of instructions and definitions; problems with records retrieval,
including reasons for item nonresponse and potential errors of

omission and commission; records availability; and time burden.

For new surveys, OEUS also conducts an operations test consisting
of about 100 randomly selected units prior to the actual survey.
The purpose of the operations test is to assess the nonresponse,
especially item nonresponse, in an operational mode and to debug

data collection systems and procedures.

The final component of questionnaire design is a response
analysis survey (RAS), which is a large-scale retrospective
analysis of data quality. It is used to identify respondent
error which is addressed in subsequent questionnaire design
activities for the survey. The response analysis survey involves

selecting a stratified random sample of establishments


Bibliography

Aday, Lu Ann (1989), Designing and Conducting Health Surveys, San
Francisco, CA: Jossey-Bass Publishers

Bradburn, N.M., Rips, L.d., and Shevell, S.K. (1987), “Answering
Autobiographical Questions: the Impact of Memory and
Inference on Survey," Science, Vol. 236, 157-161.

Bureau, M. (in press), "Experience with the Use of Cognitive
Methods in Designing Business Survey Questionnaires,"
Proceedings of the Section on Survey Research Methods,
American Statistical Association, 546-550.

Cannell, C.F., Miller, P.V., and Oksenberg, L. (1981), Research
on Interviewing Techniques, in S. Leinhardt (Ed.),
Sociological Methodology, San Francisco: Jossey-Bass, 389-
437.

Clickner, R.P. and Craig, J. (1987), "Using Business
Establishment Size in an Environmental Survey," Proceedings
of the Section on Survey Research Methods, American
Statistical Association, 541-545.

Cox, Brenda G., Elliehausen, G.E., and Wolken, J.D. (1989),
"Surveying Small Businesses about Their Finances,"
Proceedings of the Section on Survey Research Methods,
American Statistical Association, 553-557.

Demaio, T. (1983), “Approaches to Developing Questionnaires,"
Statistical Policy Working Paper 10, Office of Management
and Budget.

Demaio, T., and Jenkins, C. (1991), "Questionnaire Research in
the Census of Construction Industries," Proceedings of the
Section on Survey Research Methods, American Statistical
Association, 453-457.

Dillman, D. (1978), Mail and Telephone Surveys: the Total Design
Method, New York, John Wiley & Sons.

Fecso, R., and Pafford, B. (1988), "Response Errors in
Establishment Surveys with an Example from an Agribusiness
Survey," Proceedings of the Section on Survey Research
Methods, American Statistical Association, 315-320.


FIGURE 1. BLS Questionnaire Design Research Methods

FOCUS GROUPS

*Concepts “Definitions
“Instructions *Records Availability

!

Pre-Data DOCUMENT-DESIGN ANALYSIS
Collection *Comprehension
*Readability

Y

COGNITIVE PRETEST

*Dress Rehearsal *Comprehension
*Records Availability

a

RESPONSE ANALYSIS SURVEY

Post-Data

Collection *Records Use *Comprehension

*Response Burden



Response errors arise when definitions used in survey questions
do not match the record-keeping practices of establishments or
when data are not easily retrievable in the required format. For
example, to measure productivity, the survey questionnaire may
request data on the number of hours that employees have worked.
The employer may only have records on number of hours that
employees have been paid including sick leave and vacation hours.
If the respondent is to provide the correct data, response burden
may be substantially increased; if not, response errors are

generated.

Respondents from small establishments may use records or recall
the requested information from memory. When using memory the
tasks of establishment respondents are similar to those of
household respondents. Research on household respondent's
cognitive recall that has been addressed in various household
survey studies is generally applicable to establishment surveys
(Bradburn et al., 1987; Cannell et al., 1981; Sudman and
Bradburn, 1973). In summary, among others, the three aspects of
establishment surveys -- burden, professional terminology and
records use -- are important considerations when designing

questionnaires.


"companies that sell medicines," and when surveying city planners
“annexation” is a clearer term than ."an addition." Since
establishment surveys are often conducted by mail, precise
definitions and concepts are usually placed on the questionnaire
to prevent terms from being misunderstood. Definitions are not
usually included in household surveys.

A third difference between household and establishment surveys
also involves the use of records (Werking, et al., 1988).
Establishment survey respondents, especially in large companies,
often need to refer to written or automated sources of
information when completing a questionnaire. Household surveys

primarily rely on respondent recall without the use of records.

As previously mentioned, our primary focus is on establishment
surveys conducted by mail with telephone follow-up to
nonrespondents. Since these surveys are self-administered,
additional differences into the data collection process are
introduced. For example, the interviewer is not present to
assist with questions the respondent may have. Also, the agency
performing the survey has little or no control over who answers
the questionnaire. Another concern is that ongoing surveys may
not be answered by the same respondent during different survey

periods.


20

Most respondents said that the survey data were available with
minimal difficulty, regardless of whether the business had
computerized or manual records. But one respondent used memory
rather than records in completing the survey. In addition, the
construction industry respondent had difficulty completing the
form as multiple site and division supervisors had to be

contacted to obtain the data.

Respondents mentioned that reporting monthly rather than annual
data made the survey much easier. The respondents' estimates of
completion time were from 20 minutes to an hour; the latter was
the mode. Several respondents indicated that their first
impression of the form was that it looked time-consuming because
of the number of occupations listed on it, which concerned us
greatly. We attempted to reduce the number of nonresponses by
stating in the cover letter that respondents only had to report

data for occupations where there was turnover.

One drawback of the pretest is the small sample size of nine
units or less, a restriction imposed by the Office of Management
and Budget. Another drawback is the lengthy time necessary to
set up the interviews. The pretest involved multiple calls to

respondents to get their cooperation and then to see if they had


10

loosely constructed set of relevant questions. Group members
influence each other by responding to ideas and comments
generated by the group dynamics. As a group, they identify
problems or difficulties associated with survey definitions,

variables or concepts.

Focus groups offer a useful tool that has only recently been
discovered (or rediscovered) by quantitatively-oriented social
scientists. Scholars in marketing research have extensively used
focus groups for the past forty years (Greenbaum, 1987;
Higginbotham and Cox, 1979; Krueger, 1988; Morgan, 1988;

Templeton, 1987).

The second step in the OEUS questionnaire design process is
document design analysis. In most OEUS establishment surveys,
the scope of information requested lends itself to form-based,
rather than question-by-question, questionnaires. Thus, we
consult the literature on document design (e.g., DeMaio, 1983;
Dillman, 1978; Felker et al., 1980; Wright, 1980) in order to
develop a questionnaire that generates high respondent
comprehension and readability. Sometimes alternative designs are
used in pilot surveys to see which format elicits the most

accurate and complete data.


26

respondents) and very likely to participate (65 percent of
respondents), respectively. Thus, the more time the interview
takes, the lower the respondent's motivation to participate in
the future survey. The following table gives a breakdown of the
likelihood of future participation in the survey by size and

establishment.

Table of Establishment Size by Likelihood
of Future Participation

Size Likelihood of Future Participation
Frequency
Row Pct
Col Pct NOT VERY SOMEWHAT VERY
21 46 1193 186
11.3 24.7 64.0 44.8
1 42.0 48.0 44.2
16 Ze 84 122
13.1 18.0 68.9 29.4
2 32.0 22.9 SL2
as 28 66 107
tena Fae 615.7 25.8
6) 26.0 2o.8 24.5
50 96 269 415

12.1 Zao 64.8 100


Phipps, Polly A., Shail Butani, and Young I. Chun. “Designing Establishment Survey
Questionnaires.” The Annual Meeting of the American Association for Public Opinion
Research, St. Petersburg, FL, May 1992.

Abstract
In this paper, we demonstrate how establishment survey questionnaire design can be
improved by utilizing different research methods such as focus groups, document design
analyses, pretests, and response analysis surveys. We discuss the lack of research on
establishment survey methods in the past, uncover differences between establishment and
household surveys, and highlight cognitively-oriented Bureau of Labor Statistics’ work
on designing establishment survey questionnaires. The results indicate that each research
method contributes to improving establishment survey questionnaire design by reducing
various types of measurement errors.


13

demand-side data combined with the supply-side indicators provide

amore reliable set of measures to identify labor shortages.

The objectives of the ETJO pilot survey were to assess the
feasibility of collecting job vacancy and turnover data by
occupation on a regular basis and to provide the cost estimates
for implementing a full-scale, ongoing survey. Estimates of
separations, new hires, wages of new hires, and job openings were
to be produced by major occupational group. However, it should
be noted that the restricted universe and small sample size of
the ETJO pilot survey limited the ability to generalize from the

data obtained.

The ETJO Survey employed a stratified probability sampling plan
based on industry and size class.4 The sample consisted of
approximately 3300 units selected from an universe of about 1.5
million establishments. The universe covered private
establishments with one or more employees during the first
quarter of 1989 in the 50 States and the District of Columbia.
The sample was divided into three equal monthly panels. Each
panel was surveyed 1 month, out of survey 2 months, and then back
in the survey 1 month (i.e. 1-2-1). The data-collection period
for the survey began in November 1990 and ended in April 1991.

Panel 1 was surveyed in November and February, Panel 2 in


12

(approximately 150-500) and conducting a brief telephone
interview with respondents on their responses shortly after they
have completed the survey. The interview is standardized and
focuses on potential errors pertaining to respondent
comprehension, recall, records incompatibility, questionnaire

format, and respondent burden.
4. Application to Employee Turnover and Job Openings Survey

Depending on time and budget resources, OEUS surveys may go
through all or selected steps of the questionnaire design process
related above. In discussing design issues, we will use as our
main example the Employee Turnover and Job Openings (ETJO) pilot
survey, which has utilized all the procedures. The ETJO pilot
survey was conducted in response to a Congressional mandate to
the Department of Labor to develop a methodology to measure
national labor shortages. Without job openings and turnover
data, there is a serious gap in our knowledge of labor shortages.
While the unemployment rate in an occupation is a measure of how
difficult it is for workers with previous experience in the
occupation to find employment, it tells little about the
difficulty employers have in finding employees. Job openings

data provide a more direct measure of this difficulty. These


SECTION Ii: EMPLOYEE TURNOVER AND JOB OPENINGS SURVEY

Figure 3
Special trade contractors
Report for the Relerence Month (Cols. 1 + 3) Report for Last Business Day of Ref. Month (Cols. 4 - 7)
(1) (2) (3) (4) (5) (6
Jecupational Number of Number of Average Number of Number of oetings wild te:
Dictionary Separations new hires hourly wage job Tess than 210 More than
No. Occupations. (exclude temp. layofls) | (exclude recalls) |__of new hires ings 2 weeks 4 weeks 4 weeks
TOTAL ALL OCCUPATIONS: ie

10000 | Managerial and administrative occupations

13002 Financial managers

13008 Purchasing managers

15017 Construction managers

General managers and top
bet execulives ‘

9999 All other managers and
‘ administrators

20000 Professional, paraprofessional, and technical occupations

21114 Accountants and auditors

21902 Cost estimators

22100 Engineers

22514 Drafters

DO NOT COMPLETE SHADED AREAS.


Figure 2 .

SECTION Il — INSTRUCTIONS Column 3. Average Hourly Wage of New Hires -- For enzh occupation
Please complete Section lil using the following Instructions, wi new bes cid Be ¢ ee de ee

wage al which new employees were hired. I there was only one new hire
Galen’. = Estar th lolad rus of separations during during the month, enter the wage for that person. If the new hires are not
the reference month in ‘he fisst.zow. Then, enter the number of separations

paid hourly, (e.g., weekly, biweekly, monthly, or annually), please calculate
by occupation. Separations are terminations of employment of permanent of

an hourly rate by dividing the salary of the new hires during a pay period by
temporary workers initiated by either the employee or employer. vs Lod,
Include: Exchude:

© Quits @ Temporary layoffs
e Layoffs over 30 days (under 30 days)
e Discharges for cause @ Workers on strike

@ Retirements @ Outside consultants
@ Unauthorized absences and contractors

over 1 week Workers from temporary -
e Deaths help agencies

e Within - establishment
transfers, promotions,
or demotions

e Transfers to other
establishments of the company
@ Permanent separations
due to disability

Column 2. New Hires — Enter the total number of new hires during
the reference month in the first row. Then, enter the number of new
hires by occupation. New hires include all permanent or temporary
additions to the workforce of the establishment.

Include: Exchide:

@ Transfers from other @ Recalls from temporary
establishments of the company layotts

@ Workers hired and separated @ New hires who have not

during the month . yet reported
@ Retums from strikes

@ Outside consultants
and contractors

@ Workers from temporary -
help agencies

NOTE:
I your establishment had no reportable separations, new hires, or job
openings, completed only the cover page and retum the questionnaire.

Include: Exclude:
@ Straight-time wages or e Tips
salary @ Premium pay for overtime,

@ Incentive payments, 6.9.,
@ Cost of living pay

@ Performance bonuses
e Lump-sum payments

Column 4. Job Openings -- Enter the total number of job openings as of

the last business day of the reference month in the first row. Then, enter the

number of job openings by occupation. Jobs are open if work would have
started immediately or during the next pay period and if active recruiting of
workers from outside the establishment took place. “Active recruiting”
means elforts during June to {ill openings through means such as: listings
with private or public agencies or school placement offices, help wanted
advertising, recruitment programs, or interview of applicants.

Include: Exclude:

 Full- and part-time e Jobs to be filled by recall from
positions temporary layoffs

@ Permanent and temporary @ Jobs which can only be filled by

Positions within - establishment transfers,

Promotions, and demotions
@ Jobs where workers are hired,

and are scheduled to start in
the next pay period

Jobs which can only be filled by
workers from temporary - help
agencies

Columns 5 - 7. Of the job openings reported in Column 4, enter the
number of openings by length of time they have been unlilled. Use the
following categories: less than 2 weeks, 2 to 4 weeks, and more than 4

weeks. The sum of the categories should equal the number of job
openings reported in Column 4

holidays, weekend, shift work


3. OEUS Questionnaire Design Activities

The OEUS conducts pilot surveys for potential new programs and
one-time special surveys. As indicated in Figure 1, OEUS
questionnaire design activities include: (1) focus groups to
assess survey concepts and indicators, definitions and
availability of records, (2) document design analysis to create
an optimal questionnaire format, (3) pretests to determine
respondent comprehension difficulties in completing the
questionnaire and to identify records availability, and (4) a
response analysis survey, to assess comprehension, records use,

and data quality.

Concept development is an early stage in questionnaire design,
when economists, statisticians and survey methodologists
tentatively decide what is to be measured and what can be
measured. This process involves reviewing the literature on the
subject and an analysis of possible components to be measured.
Once these components have been identified, OEUS conducts focus
groups with individuals familiar with the subject matter in order
to assess the proposed concepts, definitions and terminology, and
to determine the content and availability of relevant records.
Members of a focus group participate in nondirective interactive

communication (Krueger, 1988; Merton, 1956; Morgan, 1988) using a


23

actual and potential response and nonresponse errors. The
standardized RAS questions focused on the following areas: the
sources of information used in answering the ETJO questionnaire,
the extent of adherence to instructions and definitions, and the
level of respondent burden in terms of time and effort involved
in. completing the ETJO questionnaire. To conduct the RAS, we
selected a stratified random sample of 510 respondents. We
interviewed a total of 420 units, yielding a RAS response rate of
83 percent. The RAS establishments were distributed across all

eight industries surveyed.

To evaluate what types of records were used, interviewers first
read a short list of possible data sources to respondents,
including memory, personnel records, payroll records, and other
employees. A respondent could choose as many categories as were
applicable. The question was repeated for each of the ETJO data
items: separations, new hires, average wage of new hires, job
openings, and duration of job openings. For all items,
approximately one-third of the respondents reported using one
data source in completing the survey, one-half reported using two
sources, and one-sixth reported using three sources. Personnel
records were used most frequently as an exclusive source of
information; personnel and payroll records were most likely when

respondents reported two sources; and personnel and payroll


29

motivation to participate needs to be understood and addressed,

or the result will be lower quality data.

In-depth investigation of error and its sources and motivational
processes, however, is constrained by establishment survey costs,
which is an area where future research should also be
concentrated. We believe that establishment survey collection
methodology meade further large-scale research which is developed
more rigorously. This will result in the higher quality data for
survey researchers, policy makers, and other users of

establishment survey data.


follows. In Section 2, we compare the tasks of respondents in
establishment and household surveys. Section 3 includes a
description of establishment survey questionnaire design
activities in the Office of Employment and Unemployment
Statistics (OEUS) of the Bureau of Labor Statistics (BLS). A
specific case study in OEUS is highlighted in Section 4.
Finally, in Section 5, we discuss overall design issues in
establishment questionnaires and end with a proposed research

agenda.
2. Special Issues in Establishment Surveys

As with household surveys, issues in establishment surveys
include frame coverage, the need for random sampling, mode of
data collection, questionnaire design, and the response
formulation process. Respondents in both establishment and
household surveys read or listen, as well as answer, with varying
degrees of comprehension, retrieval, judgment and communication.
Groves (1989) discusses several sources of survey measurement
error in the context of household surveys including the
questionnaire, respondent, interviewer, and mode of data
collection. Establishment surveys are similar in that all of

these sources of error may be present. However, the rich


16

asked focus group members if establishment data were available

and how bes€ to collect it.

Focus group members independently generated demand-side variables
similar to those tentatively decided upon by the ETJO staff,
including separations, new hires, and job openings. Both focus
groups discussed supply-side issues, such as quality, skills, and
qualifications of workers, reflecting the complexity of current
labor-shortage issues. Although we could not address supply-side
issues directly in a survey of employers, focus-group members
felt that detailed occupational categories helped in assessing
shortages by skill. The focus groups gave us confidence that our
survey concepts and definitions were appropriate for measuring
labor shortages. Participants indicated that most firms kept the
data being requested for ETJO purposes, so we expected data to be
available and the burden to not be excessive.

Based on our experience with the focus groups, we recommend that
future focus groups be conducted even earlier in the survey
design process. We think they would be useful in developing as

well as in assessing concepts.


18

Pretest Interviews

In conducting the pretest, we developed requirements for pretest
units, which were based on industry and establishment size
specifications. In order to identify either industry or size-
class specific problems, sample units from each industry and each
size class were represented in the pretest.6 We conducted seven
pretest interviews: four by site visits and three by telephone.
Initial phone contacts and advance notification of a survey have
been found to be useful to establish the legitimacy of a survey
and obtain respondent cooperation (Fox et al., 1988). For these
reasons, we contacted potential establishment respondents by
telephone and obtained their agreement to complete the pretest
questionnaire. We then mailed the survey materials to the
respondent, which were to be completed before our visit. We
decided to conduct interviews after materials were completed
because we felt that, as BLS representatives, our presence could

bias the response process.

We prepared a debriefing protocol to assess respondent
understanding of and adherence to survey instructions, terms, and
definitions. The Baobacor included questions on the sources of
information used to complete the survey and the use of supporting

survey materials.


Researchers-have just begun to pay rigorous attention to
establishment survey questionnaire design and quality issues. A
number of recent studies use “cognitive” methods, e.g., think-
aloud and in-depth pretests and focus groups, as a way to assess
and improve Sababaianmene questionnaires (Bureau, 1992, Cox et
al., 1989; DeMaio and Jenkins, 1991, Fecso et al., 1988; Goss,
1989; Gower and Zylstra, 1990; Palmisano, 1988). For example,
Cox and colleagues (1989) test the National Survey of Small
Business Finances and find unclear financial terminology,
concepts, and reference periods. They reduce these problems by
clarifying terminologies and instructions and changing the order
of questions. In redesign work on the Survey of Employment,
Payrolls and Hours for Statistics Canada, Goss and associates
(1989) test for respondent comprehension including understanding
of terminology and reference periods, respondent burden, and
record-keeping compatibility. Their recommendations for
improvement include the rewording and deletion of text, question
reordering, color coding of sections, and an increase in print

size.

This paper focuses on questionnaire design for establishment
surveys, especially those conducted by mail with telephone

follow-up to nonrespondents. The structure of the paper is as


27

5. Overall Design Issues and Research Agenda

The working paper by the Federal Committee on Statistical
Methodology (1988) entitled "Quality in Establishment Surveys"
discusses the limitations of our understanding of establishment
survey error and recommends more work to improve and document
establishment survey quality. Given the limitations of
individual research methods, it is best to utilize multiple
methods to identify errors in questionnaire design. All of the
methods -- focus groups, document design analysis, pretests and
the response analysis survey -- are found to be useful in
uncovering both survey weaknesses and strengths, but they are
useful at different stages in the survey process to identify
survey errors associated with the questionnaire, respondent, or

the records.

As the literatures shows (Stewart and Shamdasani, 1990), focus
groups are useful in assessing and clarifying concepts and
definitions at the beginning stages of the questionnaire design.
Focus groups are also successful in identifying errors associated
with comprehension and records availability. Document design
analysis reveals how business surveys can be formatted to

maximize overall comprehension and readability. The pretest


1. Introduction

Establishment surveys, where the unit of analysis is a business
or organization rather than an individual or household, have been
conducted since the beginning of this century and provide
critical information about the economy (Federal Committee on
Statistical Methodology, 1988). Information from establishment
surveys is not only needed by business firms for marketing,
planning and development, but is utilized by policy makers for
evaluating state and national economies and making future
economic plans. However, unlike their household counterparts,
surveys of establishments have not received much attention from
survey methodologists. The dearth of early research on
establishment surveys becomes very clear when searching for
relevant literature on the topic. For example, Dillman's (1978)
comprehensive bibliography of available material on the design
and implementation of mail and telephone surveys (1930s-1970s)
includes only one article on establishment surveys (Sergean,
1958). The area of establishment surveys is not addressed by
Groves (1989) in his recent work, Survey Errors and Survey Costs.
As is the case with earlier research on household surveys, most
of the recent work on establishment survey methods concentrates
on survey coverage and sample design (e.g., Konschnik et al.,

1985; Fecso et al., 1985).2


32

Fecso, R., Tortora, R.D. and Vogel, F.A. (1985), "Recent and
Future Activities to Improve Sampling Frames for
Agriculture," Proceedings of the Section on Survey Research
Methods, American Statistical Association, 132-141.

Federal Committee on Statistical Methodology (1988), "Quality in
Establishment Surveys," Statistical Policy Working Paper 15,
Washington, DC: Office of Management and Budget.

Felker, Daniel B, Pickering, Frances, Charrow, Veda, Holland,
V.M., and Redish, J.C. (1980), Guidelines for Document
Design, Washington D.C.: American Institutes for Research.

Fox, R., Crask, Melvin and Kim, Jonghoon -(1988), "Mail Survey
Response Rate: A Meta-Analysis of Selected Techniques for
Inducing Response," Public Opinion Quarterly, Vol. 52, 467-
491.

Freedman, S., "Control and Measurement of Specification Error in
Establishment Surveys," Proceedings of the Section on Survey
Research Methods, American Statistical Association, 304-308.

Goss, G. (1989), Qualitative Research to Evaluate the
Questionnaire of the Survey of Employment Payrolls and Hours
(SEPH): Final Report, Report prepared for Statistics
Canada.

Gower, A., and Zylstra, P. (1990), "The Use of Qualitative
Methods in the Design of a Business Survey Questionnaire,”
Paper presented at the International Conference on
Measurement Errors in Surveys, Tucson, AZ.

Greenbaum, T.L. (1987), The Practical Handbook and Guide to Focus
Group Research, Lexington, MA, Lexington Books.

Groves, Robert M (1989), Survey Errors and Survey Costs, New
York, John Wiley & Sons.

Higginbotham, J.B. and Cox, K.K. (Eds.) (1979), Focus Group
Interviews: A Reader, Chicago: American Marketing
Association.

Hirschberg, D. (1987), "Some Economic and Accounting Measures of
Business Activity by Size of Firm," Proceedings of the
Section on Survey Research Methods, American Statistical
Association, 525-528.


30

FOOTNOTES

1 Plunkert (1981) and Scott (1980) are the exceptions. They
addressed the issues of data quality for job openings by
conducting response analysis surveys at the Bureau of Labor
Statistics.

2 We define large establishments as those with 250 or more
employees, medium with 50 to 249, and small with 49 or fewer
employees. Literature shows breakpoints of establishment size
vary by the nature of surveyed establishments. See Clickner and
Craig (1987) and Hirschberg (1987) for additional references.

3 Perhaps more emphasis should be put on household respondents'
use of records. In a study by Sharp and Frankel (1983),
respondents found it no more burdensome to refer to records--
their check books--for expenditure items during household
interviews than to recall the information from memory.
Furthermore, Aday (1989) encourages the use of records, such as
checkbooks, doctor or hospital bills, or appointment books to
improve the accuracy of factual data in household health-related
surveys.

4 For the pilot survey, the scope was limited to only one
industry from each of the major industry groups. The size
classes were small (1-49 employees), medium (50-249 employees),
and large (250 or more employees).

5 Member of the BLS Business Research Advisory Council
participated in the first focus group. Members of the
Philadelphia Chapter of the National Association of Corporated
and Professional Recruiters participated in the second focus

group.

6 An oil and gas industry establishment was not included because
the pretest was limited to the Washington D.C. region.


14

December and March, and Panel 3 in January and April 1992. Data

were requested for the specific survey month.

Since some occupational categories vary by industry, a separate
questionnaire was developed for each industry. Generally, data
collection operations were as follows: 1) establishments were
sent an initial package consisting of questionnaire,
instructions, occupational definition booklet, and other
miscellaneous supporting materials; 2) about two weeks after the
initial mailing, the nonrespondents were sent a follow-up
package; and 3) about one month after the initial mailing, the
remaining nonrespondents were contacted using a Computer-Assisted
Telephone Interview (CATI) instrument for data collection. The
CATI instrument was designed to closely resemble the mail
questionnaire. The overall usable response rate for the first
and second collection periods, respectively, was 70 and 75

percent.

Concepts and Definitions--Focus Group Analysis

The first several months of the survey were spent reviewing the

literature on labor shortages and determining what data we would
need to collect. The ETJO survey developed two direct measures

of the difficulty employers found in ming othe duration of

existing job openings and the "vacancy fill rate," or number of


ad

Document Design Analysis

Given the nécessity of collecting similar data for numerous
occupations, we considered a matrix-based design to be the best
available questionnaire format (Wright and Bernard, 1978). Thus,
the questionnaire took on the appearance of a matrix, with
occupations listed vertically on the side and data item headings
running horizontally across the top, rather than a question-by-
question format. We used the literature on document design to
construct a questionnaire for optimum respondent comprehension.
This included using very specific instructions and definitions
and readable print (Felker et al., 1980; Wright, 1980). As much
as possible, we placed definitional information in lists (see
Figure 2), rather than imbedding it in paragraphs, where it was
less likely to be read by respondents (Wright and Bernard, 1975).
Instructions were numbered and identically corresponded to
numbered column headings in the matrix. Instructions were also
placed on the top of a 11 1/2 x 17 inch page with the beginning
of the questionnaire matrix on the bottom (see Figures 2 and 3).
This was done so that respondents would not have to refer to a
separate page of instructions when beginning the questionnaire.
We included a few instructions in the matrix column headings
(See Figure 3). For example, we noted what to do with employees
who were laid off or recalled to work in order to reduce

potential sources of response error.