Survey Methodology, June 1991 7

As a natural follow-up to TDE, the Bureau is currently conducting several small research
tests of anew Voice Recognition reporting system. Preliminary results for VR reporting have
replicated the same high monthly response rates achieved under TDE, but with the important
advantage that respondents find VR reporting more natural and generally prefer it over TDE.
At this time, the cost of the VR hardware is approximately 15 times that of TDE; however,
within several years as the initial costs of VR drop, this collection method should become a
viable replacement for TDE.

2.4 Research Results

Over the past 7 years, the Bureau has been able to establish that payroll data is available
in most firms prior to the publication deadline for the preliminary estimates and that CATI
collection has the ability to take traditionally late mail responders (i.e., 0-20% response rate
for preliminary estimates) and within 6 months turn them into timely responders with response
rates of 82-84% (Figure 1). These response rates have been remarkably stable over the years
as the CATI sample has been expanded from 400 units to 5,000 units and the number of
participating States increased from 2 States to 14 States. The research results indicate that the
data do exist at most establishments in time to meet the publication deadline and that CATI
collection can raise mail response rates by 60-80% for these late respondents and this rate can
be maintained in the targeted range of 80-90% over long periods of time. The principal limiting
factor in a respondent’s ability to make the publication deadline was found to be the length
of the firm’s pay period (Figure 3). Employer pay periods are generally weekly, biweekly, semi-
monthly or monthly. Weekly and semi-monthly payrolls can almost always be collected in time
for publication with biweekly pay periods available most of the time; however, most monthly
payroll systems close out well after the publication deadline. Monthly payrolls have been one
of the largest factors in limiting the CATI response rates to the 82-84% range.

Several other important results have come out of the CATI research. Under CATI, approx-
imately 60% of the respondents will have their data available on the prearranged date for the
first call with the remaining 40% using the first call as a prompt call (Figure 1). This rate has
varied little across States or over the years of testing. A small test is scheduled to be conducted
to see if an advance postcard notice to the respondent shortly before the prearranged CATI
contact date will significantly limit the number of callbacks required.

1984 1985 1986 1987 1988 1989 1990
Mail Resp. Rates 47% 47% 48% 49% 49% 51% 52%
Units 400 400 2000 3000 5000 5000 5000
Resp. Rates 83% 84% 82% 84% 83% 84% 82%
can % Call Back 44% 42% 40% 41% 42% 41% 41%
Av. Minutes: 5.6 5.6 5.0 48 44 3.6 3.8
Units 400 600 2000 5000
Resp. Rates 78% 80% 84% 82%
TDE & Voice
‘% Call Back 45% 45% 43% 40%
Av. Minutes 1.8 1.8 1.7 1.7

Figure 1. Research Summary


12 Werking and Clayton: Data Quality Through Mixed Mode Collection

September 1989 — August 1990

May 1988 — February 1989

Percent Percent
100 20
oo + =e va
80
: 15
70 .
$60
; A
5 50 * 10
& \ ; 4
Bao ! \_y
\}
a» Y J
Sy -
Wee
20 TF Semivmonthly
0 Be
0 o

1989

‘Sep Oct Nov Dec Jan Feb Mar Apr May Jun Jul Aug
1990

Figure 3. CES CATI First Closing Performance

Figure 4. Touchtone Data Entry Distribution

by Length of Pay Period of TDE Calls by Time of Day
Index Value Percent
220 100
90
200
80
Mail
180
7 70
160 60
140 50
43%
40
120
Potential
Savings 30
100 +
{ TOE 20 7% — 17%
13%
80 10%
—_— 0
60 T 7 — °

90 91 92 93 94 95 96 97 98
Year

99 2000

Figure 5. Estimated Unit Costs by Mode: 1990-2000

Note: Not official BLS projections

0-25 26-50 51-75 76-99 100+
Size of Monthly Revisions (000)

Figure 6. Distribution of Magnitude of
Revisions (1985-1989)



Survey Methodology, June 1991 9

Self-response

Cost Category Mail CaTI (TDE & VR)

LABOR
mail out

mail return

data entry

NIN

edit and edit reconciliation

nonresponse followup

NON-LABOR
postage /

telephones.

ZIZINININ

Area

microcomputers

Recent Annual Price Change Factors
Labor +5.7% ECI, State and Local Government

Postage +5.0% U.S. Postal Service

Telephone -1.7% CPI-U, Intrastate toll calls

Microcomputers 19.5% PPI Experimental Price Indexes (16 bit computer)

Figure 2. Data Collection Costs (arrows show direction of recent price change)

For the labor categories, the monthly mailout, mailback, check-in and forms control opera-
tions of mail were replaced by a single annual mailout-only operation under TDE, thus
eliminating a large monthly clerical operation in the States. The batch keypunching, keypunch
validation, and forms control operations under mail were completely eliminated under TDE,
where the respondent touchtone enters the individual firm’s data and validates each entry.
Another major quality and cost-efficiency advantage of TDE was that procedures for telephone
nonresponse follow-up became far more feasible under TDE than mail. Under TDE, an
accurate up-to-date list can be generated of respondents who have not yet called in their data,
this list can then be used to conduct brief telephone prompting calls. Under mail, telephone
follow-up activities of ‘‘apparent’’ nonrespondents were awkward since the State staff did not
know whether the respondent’s form was not yet completed, currently in the mail, in the State
check-in process or at keypunch; in addition, respondents who had recently sent their form
tended to resent the additional reminder for an activity that they perceived as completed. Due
to the voluntary nature of the program and the uncertainty of a respondent’s response status,
telephone prompting under mail was only used for critical (large employers) units.

There were no significant cost savings made for edit reconciliation as the number of edit
failures under TDE remained at about the same level as under mail. This was also true for
postcard reminders where the number of postcards used under mail collection for late
respondents was approximately the same as the number used under TDE, where respondents
received an ‘‘advance’’ postcard notice to touchtone their data by the due date.


8 Werking and Clayton: Data Quality Through Mixed Mode Collection

The average time for a CATI interview depends on the number of items to be collected, the
time efficiency of the interview instrument, and the experience of the data collector. The average
time for a CATI call (Figure 1) was reduced by one-third as the CATI instrument was
streamlined and interviewers became more experienced. Another very important concern in
the testing was the effect of CATI on sample attrition. There was some concern that employers
would not want to be constantly bothered by telephone contacts and would drop out of the
program. However, the sample attrition rate for CATI was about one-third of that for mail
with almost no loss of large reporters under CATI. In summary, CATI appears to have come
close to maximizing the achievable response rate for the preliminary estimates while also
enjoying broad support from the respondents.

Due to the increased cost associated with CATI collection, the Bureau initiated research
into touchtone collection. During the 4 years of testing, TDE has demonstrated the ability to
take timely CATI reporters having 82-84% response rates and maintain these high rates under
completely automated TDE reporting (Figure 1). The importance of this result lies in the cost
savings under TDE collection versus CATI collection. One of the major concerns for TDE
collection was that, unlike CATI where respondent contacts are scheduled throughout the day,
TDE respondents might tend to call during the same time period thus generating busy signals
and require an excessive number of touchtone PCs to handle peak load reporting. Fortunately,
this was not the case, and while the touchtone PCs are on-line 24 hours a day, most calls are
relatively uniformly distributed between 8am and Spm (Figure 4). TDE respondents tend to
require the same proportion of prompt calls as CATI respondents - approximately 40%.
Methods are currently also being tested to reduce the TDE prompting workload. One major
advantage for the respondent is that TDE collection requires only one-half of the time of a
CATI interview with the average TDE interview lasting only 1 minute and 45 seconds. Addi-
tionally, touchtone phones are widely available at most establishments; current estimates
indicate that over 80% of employers could report under touchtone data collection. While TDE
reporting offers many advantages to the survey agency, its strongest feature is respondent accep-
tance; respondent reaction to touchtone reporting has been very positive due to its speed and
convenience for the respondent.

One general observation concerning the development of a CATI research program is that
it is not critical which CATI hardware or software system is used during the research phase
as longas it is reasonably flexible for change. The final results from testing may suggest very
different CATI requirements for production implementation than those originally required
for the research program. The most important and time consuming activity is the development
and refinement of the methods and procedures for respondent contact. Once effective methods
and procedures are developed, the requirements for the “‘right’’ system become more obvious.

2.5 Cost Analysis

With the performance testing and respondent acceptance for CATI and TDE proving to
be highly successful, the final phase of research shifted to analyzing the transitional costs of
CATI and the ongoing costs of TDE collection.

The major “‘labor’’ and ‘‘non-labor”’ cost categories were studied for mail, CATI, and TDE
collection (Figure 2). The study looked not only at estimates of current cost, but also at projected
costs over the next 10 years using the current rate of increase for the major cost items. Since
CATI was to play only a 6 month transitional role (.e., moving late responding mail units to
on-time responding CATI units) prior to conversion to ongoing TDE, the major focus of the
cost analysis was on the cost tradeoffs between ongoing mail versus ongoing TDE data
collection.


Survey Methodology, June 1991 1

monitored for signs of wage-induced inflationary pressures resulting from strong job growth
during a period of low unemployment. With this greatly increased use and visibility of the mon-
thly data, came a corresponding user frustration with the periodic large revisions to the
preliminary estimates.

While monthly revisions to the preliminary estimates had always been a part of the CES
survey process and even though the size of these revisions had been reduced over the years,
large revisions of over 100,000 in the preliminary monthly employment estimates of over-the-
month change were now being viewed as unacceptable. These user demands for greater accuracy
in the preliminary estimates would lead the Bureau to develop proposals for the implementation
of automated CATI and TDE collection methods into the U.S. government’s largest monthly
survey. While user demand is critical, major changes of this magnitude could not be under-
taken without full support at the State level where data collection actually occurs. One guideline
which remained constant throughout the research program was that the collection system was
ultimately the States’ data collection system and therefore must be designed to integrate well
into their survey environment and create as minimal an organizational impact as possible. To
that end, the CATI and TDE systems remained open for change throughout the research
program. As many State suggestions and requirements as possible were taken into account with
each new release of the systems. The success of much of the development work can be credited
to the resiliency and endurance of the 14 research States as they made constant recommendations
for improvements in systems and procedures. In the end, the CATI interview instrument had
moved from an awkward simulated household survey type interview approach to a fast and
efficient ‘“‘screens”’ and ‘‘windows”’ approach well suited for capturing and editing longitudinal
economic data. Thus, at the conclusion of the research phase, the systems and procedures were
well tested and refined across a wide range of States. This approach to testing brought with
it a strong sense of confidence in the methods and the systems at the State level. This would
prove to be essential for the Bureau’s proposed quick production implementation timetable
of these state-of-the-art collection methods.

3.2 Approach and Impact

The main focus of the implementation proposal was the control of revisions in the
preliminary estimates. Over the past 5 years, approximately 40% of all revisions were over
50,000 with 13% of the revisions exceeding 100,000 (i.e., large revisions) (Figure 6). The goal
of the implementation study was to identify a minimum set of late responders which, if obtained
by the publication deadline, would control the size of revisions to what was considered to be
in an acceptable level (under 50,000 revision in the over-the-month employment change). While
one obvious approach was to convert all 175,000 late respondents to the new collection methods,
this approach was considered to be lengthy and costly. While there was a need to responsibly
control the size of revisions (i.e., not necessarily completely eliminate all revisions), there was
also a corresponding need to resolve this problem in as timely a fashion as possible (i.e., convert
the smallest number of units necessary to control revisions to under the 50,000 level).

Establishment surveys, unlike household surveys, generally have differential weighting for
individual units with very large units being ‘‘certainty’’ units in the sample design. In the CES
sample design, units with 100+ employees make up only 20% of the sample (i.e. , 75,000 units),
but account for over 83% of the unweighted sample employment. These units tend to have
a much lower response rate for the preliminary estimates so that if the late respondents’
employment trend differs from the early respondents, these units can create a substantial
revision in the sample estimates. Revision impact studies were conducted to assess the affect
of large employers 100+ on the preliminary estimates. To test the impact of large employers,


Survey Methodology, June 1991 3
Vol. 17, No. 1, pp. 3-14
Statistics Canada

Enhancing Data Quality Through the Use of
Mixed Mode Collection

GEORGE S. WERKING and RICHARD L. CLAYTON!

ABSTRACT

A chronic problem in the preparation of time critical estimates is the significant limitations inherent in
data collection by mail. To address this issue, the U.S. Bureau of Labor Statistics has conducted an
extensive 7 year research effort into the use of computer assisted telephone interviewing (CATI) and the
computer assisted self interviewing (CASI) methods of touchtone and voice recognition self-response.
This paper will summarize some of the significant results of this research covering both performance
and cost data. The paper will conclude with a discussion of a large scale implementation program of these
techniques for a monthly sample of 350,000 establishments.

KEY WORDS: Employment statistics; Revisions; CATI; Touchtone collection; Voice collection; Cost
analysis.

1, INTRODUCTION

1.1 Employment Statistics in the United States

On the first Friday of each month, the U.S. Bureau of Labor Statistics releases data on the
United States’ employment situation for the previous month. On release day, the Commissioner
of Labor Statistics appears before the Joint Economic Committee of Congress and provides
adetailed analysis of the current month’s data and trends; at the same time, the data are made
available to the news media and the financial and business communities. This closely watched
set of statistics is the earliest indicator available on the previous month’s economic activity
and is used as a major gauge of the health of the U.S. economy. The data in the release cover
employment, hours and earnings by detailed industry which are derived from the Bureau’s
350,000 unit monthly establishment survey - the Current Employment Statistics (CES) survey -
along with labor force and unemployment data which are derived from the Bureau’s 60,000
unit household survey - the Current Population Survey (CPS).

The establishment survey data have many important economic uses. Due to the CES survey’s
size and timeliness in conjunction with the importance of the basic payroll statistics which it
collects, the CES monthly estimates are not only used as principal economic indicators by
themselves but they are also included in the development of many of the Nation’s other major
economic indicators including: Personal Income for the Gross National Product, the Index
of Leading Economic Indicators, the Index of Coincident Indicators, the Industrial Produc-
tion Index, Real Earnings measures and Productivity measures. While the timeliness and
accuracy of the CES statistics are essential in analyzing the current economic conditions in the
United States, the CES survey has had to rely on a mail data collection process since its incep-
tion in the early 1900s. This collection process results initially in the publication of
“preliminary”’ estimates for higher level aggregates using only the sample returns received to
date followed by ‘‘final’’ estimates two months later which use the full sample. The process
of producing both preliminary and final estimates for a given month periodically yields a

1 G, Werking and R. Clayton, Monthly Industry Employment Statistics Division, Bureau of Labor Statistics, Room
2089, 441 G Street, N.W., Washington, D.C., USA, 20212.


14 Werking and Clayton: Data Quality Through Mixed Mode Collection

REFERENCES

CLAYTON, R.L., and HARRELL, L., Jr. (1989). Developing a cost model of alternative data collection
methods: MAIL, CATI and TDE. Proceedings of the Section on Survey Research Methods, American
Statistical Association.

CLAYTON, R.L., and WINTER, D.L.S. (1990). Speech data entry: Results of the first test of voice
recognition for data collection. Proceedings of the Section on Survey Research Methods, American
Statistical Association.

JURAN, J.M., GRYNA, F.N., Jr., and BINGHAM, R.S., Jr., eds. (1979). Quality Control Handbook,
Third Edition. McGraw Hill.

GROVES, R.M.1J., et al., eds. (1988). Telephone Survey Methodology. New York: John Wiley and Sons.

OFFICE OF MANAGEMENT AND BUDGET (1988). Quality in Establishment Surveys. Statistical
Policy Working Paper 15.

OFFICE OF MANAGEMENT AND BUDGET (1990). Computer Assisted Survey Information
Collection. Statistical Policy Working Paper 19.

PONIKOWSKI, C., and MEILY, S. (1988). Use of touchtone recognition technology in establishment
survey data collection. Presented at the First Annual Field Technologies Conference, St. Petersburg,
Florida.

WERKING, G.S., TUPEK, A.R., PONIKOWSKI, C., and ROSEN, R. (1986). A CATI feasibility study
for a monthly establishment survey. Proceedings of the Section on Survey Research Methods, American
Statistical Association, 639-643.

WERKING, G.S., and TUPEK, A.R. (1987). Modernizing the Current Employment Statistics Program.
Proceedings of the Business and Economic Statistics Section, American Statistical Association, 122-130.

WERKING, G., TUPEK, A., and CLAYTON, R. (1988). CATI and touchtone self-response applications
for establishment surveys. Journal of Official Statistics, 349-362.


Survey Methodology, June 1991 13

late respondents in size class 100 + were included in the original sample used for the preliminary
estimates and the estimates were recalculated. These new estimates were then compared to the
original preliminary estimates to determine the impact of 100+ employers on revisions. The
results indicated that from one-half to two-thirds of the revision was attributable to these units.
These studies were repeated over several months with similar results. Applying these projected
reduction rates in revision size to revisions over the past 5 years resulted in over 97% of all
revisions being below the 50,000 level as compared to the current level of only 60%. This greatly
reduced targeted sample size for conversion to CATI/TDE provided for an accelerated
implementation time schedule consistent with controlling conversion costs to the minimum
level necessary to protect against large revisions in the preliminary estimates. The Bureau will
begin the implementation of CATI/TDE collection methods in 25 States in 1991 with planned
expansion to all States in 1992. With the implementation of the new collection methods, the
Bureau will be able to help resolve one of the most difficult and visible quality issues affecting
the CES user community.

4. SUMMARY

The decade of the 1980s has brought about many changes for survey agencies. Some of the
changes can be viewed in terms of our accomplishments made over the decade while others
are more subtle and need to be viewed in terms of the changes in the survey environment in
which we operate.

The 1980s created a far more quality conscious user constituency which is quick to identify
and point out our product limitations. While our products may not have deteriorated, our users’
expectations on quality and ‘‘fitness for use’’ have greatly increased. This is an issue which
we as statistical agencies must be able to respond to in order to maintain our credibility with
the user community. The 1980s also ushered in dramatic new technological breakthroughs most
notably in microcomputers. The new technology has offered survey agencies many new oppor-
tunities for improving data collection control and quality including: CATI, CAPI, TDE, VR
and FAX. Some of these options offer improved quality and control at lower ongoing costs.
The decade of the 1990s may well offer even greater opportunities for using technology to
improve our data collection timeliness and quality at lower costs.

As we look at the status of our statistical programs, we often find very rigid environments.
The data collection approach for our surveys often date back to their inception. Our data
collection cost assumptions and cost studies are usually well outdated and often simplistic in
approach. Since data collection generally represents the largest part of a survey’s cost, it is
usually well entrenched in the agency’s organizational structure and can be quite difficult to
restructure in order to accommodate large scale change. It is within this survey environment
that we will face the major challenges and opportunities of the 1990s.

The challenge for statistical agencies in the 1990s will be threefold:
© to be responsive to the changing quality needs of our users;

© to attempt to have our research stay up with the rapid change of technology and automated
data collection approaches; and perhaps more importantly

* to continue to find ways to incorporate successful research into our ongoing programs.

These challenges will determine the cost and quality competitiveness of our programs and our
agencies in the future.


10 Werking and Clayton: Data Quality Through Mixed Mode Collection

In the non-labor categories, the cost of postage under mail (currently 58 cents per unit) is
replaced by the cost of a telephone call and the amortized cost of the TDE machine (together
currently 46 cents per unit). Postage is a continually increasing cost with an annual price increase
of approximately 5% (Figure 2). The rising cost of postage is driven by annual labor cost
increases (+ 5.7%) and by fuel costs (also generally increasing) with labor accounting for over
80% of total postage costs. In contrast, under TDE the cost of telephone calls has been
decreasing in recent years (— 1.7%), along with the cost of microcomputers (— 19.5%).

Excluding the additional new requirement of full telephone nonresponse prompting for TDE,
there are demonstrable cost savings in shifting from mail to TDE. Perhaps more importantly,
under a 10 year projection of future costs of these two collection methods (Figure 5), these
savings grow substantially. Attempts will be made to redirect future cost savings from TDE
to help offset the full nonresponse prompting activities.

There are several major conclusions concerning TDE reporting which have emerged from
the performance and cost analysis review. (1) The traditional view that mail is the least expen-
sive collection option available to statistical agencies is no longer true. The major technological
breakthroughs of the 1980s in automated telephone collection can not only reduce the collection
cost below that of mail but can also improve timeliness and control over the collection process.
Additionally, over the next 5-10 years, the cost of mail will become even less cost competitive
with these high-technology/low-labor collection approaches due to the increasing labor and
postage costs associated with mail. (2) The transition of respondents from mail to TDE appears
to cause very little disruption to monthly reporting. In the Bureau’s follow-up interviews with
respondents who were converted to TDE, results have shown that respondents have very little
trouble adapting to this new method of reporting. Virtually all respondents completed their
first month TDE report accurately and without assistance, with many respondents commenting
on the ease of reporting under TDE. (3) TDE can be viewed as a reliable replacement method
for survey data collection; over the past 4 years of collection there have been no major equip-
ment failure problems or disruptions of the collection process. Minor equipment problems have
been easily resolved using a back-up PC when required. In addition, future back-up protection
for the State TDE collection process will involve the use of a call forwarding option to reroute
calls to a central site should major problems occur at the State.

3. IMPLEMENTATION

3.1 Major Issues

By the end of 1989, the Bureau had completed a very successful research program and had
sustained the high performance levels over 7 years. However, there is a significant difference
between the completion of successful research and full scale implementation of new methods.
While over 10,000 units were being actively collected under these new techniques, these units
represented under 3% of the CES sample. Proposed collection changes for a monthly sample
of 350,000 units which has been collected for well over half a century under a decentralized
State mail collection environment requires not only a very strong demonstrable user need but
also broad-based support at national, regional and State levels.

As it turned out, the user need had begun to change in the early 1980s. During the 1980s,
the U.S. economy experienced the longest sustained peace-time growth period in its history,
with over 19 million jobs created, and unemployment rates at their lowest levels since the early
1970s. By the mid 1980s, economic policy was firmly focused on establishing non-inflationary
economic growth. The monthly CES employment growth and wage data were being closely


6 Werking and Clayton: Data Quality Through Mixed Mode Collection

in the data items for that month and waits for the prearranged CATI call from the State agency.
When the State agency calls, the data are collected under CATI, edited and a time for the next
month’s collection call is arranged.

Touchtone Date Entry - Under TDE reporting, the employer does the same activities as under
CATI except instead of waiting for the State agency’s CATI call, the employer now calls an
800 telephone number connected to the touchtone PC located at the State agency. The employer
then touchtone enters the data items following the prompts in the automated CES interview.
As each data item is entered by the employer, it is read back for respondent verification.
Voice Recognition - VR data reporting is identical to touchtone collection except the employer
no longer needs to have a touchtone phone. The employer now reads the data as they appear
on the form and the voice PC translates and reads back the data to the employer for verification.
The VR system is speaker independent and accepts continuous speech; it recognizes the digits
0 through 9 and ‘‘yes”’ and ‘“‘no”’.

2.3. Research Tests

The Bureau began developing a PC-based CATI system in 1983 for use in a two State test that
began in 1984 (Figure 1). The CATI system developed by the University of California at Berkeley
was selected for the test and was subsequently used throughout the research effort. A small
random sample of 200 units was selected in each State and collection procedures and systems
were refined over the next 7 years. The initial research tests were highly successful in the response
rates they achieved and the tests were expanded to 9 States in 1986 and then to a total of 14
States in 1988. The composition of the test sample was also changed in 1986. Instead of selecting
random samples of the full CES sample, the subsequent research tests focused only on random
samples of habitually late CES respondents (i.e., those units which had a response rate of under
20% for the preliminary estimates publication deadline). Thus, the success of the new collection
methods of CATI and TDE was measured in terms of their ability to move samples of reporting
units with a 0-20% preliminary estimates response rate to a stable ongoing 80-90% response
rate. By the end of the CATI research phase in 1990, the Bureau was collecting over 5,000 units
monthly under CATI and had conducted well over a quarter of a million CATI interviews.

While CATI was proving to be highly successful in improving response rates, it also became
clear by 1985 that ongoing CATI collection would be more expensive than the existing mail
collection. At this time, a separate path of research was begun on how to reduce the cost of
CATI, while still maintaining the high monthly response rates which it was achieving. While
improvements were made in reducing the length of time required for a CATI interview, it was
anew alternative PC-based telephone reporting method which would offer dramatic reductions
in the collection costs of CATI.

By 1985, many U.S. banks were operating a version of touchtone entry verification for check
cashing at drive-in windows. The Bureau identified a PC-based touchtone reporting system.
suitable for survey research testing and by 1986 was conducting a small two State test of this
technique for collecting data. TDE was not viewed as a direct replacement for mail nor as a
competitive method to be tested against CATI. CATI’s role was to take habitually late
responders and turn them into timely responders through personal contact and an educational
process, while TDE’s role was to take these timely CATI responders and maintain their response
rates at the same high level, but at a greatly reduced unit cost. Over the 5 years of data collec-
tion, TDE has also proven to be a very successful and reliable method of telephone data
collection. The research phase for TDE is now also being concluded with over 5,000 units
continuing to report monthly under TDE across 14 States; in total the Bureau has collected
over 100,000 schedules using this new automated reporting method.


Survey Methodology, June 1991 5

The 1980s saw the Bureau shift from experimental research in the CES survey to full
production testing of some of the most advanced state-of-the-art automated collection tech-
niques then available, with major implementation of these techniques scheduled for 1991.

2, CES RESEARCH PROGRAM

2.1 Research Goals

In the early 1980s, the Bureau began an extensive 7 year research effort into the causes of
late response and alternative collection methods which could significantly increase response
rates for the preliminary estimates. The focus of the survey research centered around obtaining
answers to three basic questions:

¢ Are data available at the establishment in time to respond by the publication deadline for
the preliminary estimates?

Are there data collection methods which can ensure an 80-90% response rate under these
tight time constraints?

* Can the cost of these data collection methods be controlled at about the same level as the
current mail collection costs?

At the conclusion of the research program, a mixed mode CATI/TDE collection approach
emerged which satisfied the response rate and cost constraints for the survey. The following
sections provide a brief description of these personal computer (PC) based data collection
methods, the research tests, the response rate results and the cost analysis. Further details on
these tests are documented in the research papers listed in the references. Additionally, recent
results on measurement error for Touchtone collection are presented in a paper by Phipps and
Tupek, this issue.

2.2 Data Collection Methods

The CES survey has a very limited data collection time period available to meet the
preliminary estimates publication deadline. The CES survey’s reference date is the payroll period
containing the 12th of the month; thus, there are only 2 1/2 weeks available to collect, keypunch,
edit, tabulate, validate and publish the data. In order to meet these tight time constraints, a
collection method must be able to obtain the required data as soon as they become available
within the establishment. The four data collection methods studied are described in turn below.

Mail - The CES questionnaire is a single page mail-shuttle form which provides space for the
employer to record 12 months of data. The employer receives the questionnaire in the mail
each month on or about the 12th of the month (i.e., the survey reference date) and subsequently
fills in the row of data items corresponding to the current month. There are five basic data
items collected: all employees, women worker employment and production (or nonsupervisory)
worker employment, hours and earnings. Once completed, the employer mails the form back
to the State agency where it is keypunched and edited. The form is then filed so that it can
be mailed back to the employer for the collection of the next month’s report. As indicated
earlier, this process currently yields a 50% response rate in the 2 1/2 weeks available for the
preliminary estimates.

Computer Assisted Telephone Interviewing ~ Under CATI collection, the employer is mailed
the CES questionnaire once at the beginning of the year and retains it for recording each month’s
data throughout the year. Each month as the payroll data become available, the employer fills


4 Werking and Clayton: Data Quality Through Mixed Mode Collection

substantial revision to the initial estimates. These revisions not only affect the basic CES
statistics but also the other statistics which make use of CES estimates as input. To address
this issue, the Bureau initiated a research program into automated telephone data collection
approaches with the objective of substantially reducing the size and frequency of large revisions
in the preliminary estimates.

This paper provides an overview of the Bureau’s 7 year research program into automated
telephone collection techniques and summarizes some of the most significant results. The
following sections describe the CES survey process; discuss the research program evaluating
Computer Assisted Telephone Interviewing (CATI), Touchtone Data Entry (TDE) and Voice
Recognition (VR) data collection methods; detail some of the significant research results
covering both performance and cost data; and conclude with a discussion of the large scale
implementation program of these methods in the CES survey.

1.2. Current Employment Statistics Survey

The CES survey, with 350,000 units, is the largest monthly sample survey in the United States.
It is conducted by the Bureau as a Federal-State cooperative program under which the Bureau
specifies the survey’s sample design and operational procedures while each State conducts all
data collection and edit reconciliation activities. The Bureau produces and publishes exten-
sive monthly industry detail at the 2, 3 and 4 digit industry levels for the Nation as a whole
while each State produces monthly State and area (270 Metropolitan Statistical Areas) estimates.

The CES estimates are widely regarded as highly accurate economic statistics. Once each
year, complete (or universe) employment counts for the previous year become available from
the Unemployment Insurance tax records; these counts are used to annually benchmark (realign)
the CES sample estimates to these universe counts. The annual benchmark process yields more
accurate current monthly estimates along with providing an annual estimate of overall survey
error. The average difference in the CES final sample estimate versus the complete universe
count over the past 5 years is under 0.2% with 4 years in the 1980s when the difference was
approximately zero. While the CES final monthly estimates are regarded as highly accurate
relative to the universe counts; the preliminary monthly estimates, which are based on approx-
imately 50% of the mail sample returns, have been periodically subject to large revisions when
compared to the final estimate that is available 2 months later. Over the years, some improvement
in reducing the size of the monthly revisions has been made; however, periodic large revisions
have been viewed as a byproduct of conducting a large decentralized mail data collection process.

The decade of the 1980s brought about a number of changes for the CES program which
would significantly alter the urgency and options for resolving the monthly revision issue. The
1980s created a far more quality conscious user constituency and while the CES products had
not necessarily deteriorated, the CES users’ expectations on quality and ‘‘fitness for use’”’ had
greatly increased. Much of this new way of thinking is directly attributable to the efforts of
Deming, Juran and others on the subject of quality management. The 1980s also saw a much
greater focus on the uses and the importance of the CES payroll statistics in assessing the current
health of the U.S. economy, but with the rise in the use and the visibility of the CES statistics
came a corresponding user frustration with monthly revisions. The 1980s also ushered in some
dramatic new technological breakthroughs, most notably in microcomputers. This new
technology offered survey agencies many new opportunities for improving data collection
control and quality that included: Computer Assisted Telephone Interviewing, Touchtone Data
Entry, Voice Recognition, Computer Assisted Personal Interviewing and FAX. Several of these
methods would ultimately offer options to significantly improve timeliness and quality at an
equivalent or reduced ongoing program cost.