ACKNOWLEDGEMENTS
The authors thank Henry Chiang, Athar Malik and Darrell Philpot for their

assistance with this research.
REFERENCES

Cox, Anthony C. and Martin B. Coope. (1981). Selecting a Voice for a Specified
Task: The Example of Telephone Announcements. Language and Speech 24,
233-243.

Marics, Monica A. and Beverly H. Williges. (1988). The Intelligibility of
Synthesized Speech in Data Inquiry Systems. Human Factors 30, 719-732.

Roelofs, J.C. Synthetic Speech in Practice: Acceptance and Efficiency (1987).
Behaviour and Information Technology 6, 403-410.

Schwab, Eileen C., Howard C. Nusbaum and David B. Pisoni. (1985). Some
Effects of Training on the Perception of Synthetic Speech, Human Factors 27,
395-408.

Waterworth, J.A. Interaction with Machines by Voice: A Telecommunications

Perspective. (1984). Behaviour and Information Technology 3, 163-177.

1990 Wendler, Eric R. "Respondent-Initiated Computer-Directed surveys."
Presented at the Annual Conference of the American Association of Public

Opinion Research, Lancaster, Pennsylvania, May.


14

of the identical data, respondents may have forgotten whether or not they had
previously reported their data. The system currently accepts the data with the
latest date and time, although analysts are provided a list of respondents with
duplicate records for review, and if necessary, correction.

A common reason for callback seemed to be related to the "enter 1" to confirm
after each data item is entered. Many of the respondents who corrected their data
had "1" in the data field prior to the callback, and some other response afterwards.
Callbacks were twice as common with firs‘ .ime respondents on the touchtone
system than for respondents who were "experienced" users.

A few respondents called in their data three times for a given month, and one
respondent called in data four times. These respondents seemed to be having
difficulties with the system, but finally reported all of their data correctly.

Overall, these data suggest that respondents are having some difficulties with
the system (more than they admit to during the respondent debriefing interview).
Some steps could be taken to help alleviate some of the problems. These include
providing more time for respondents to answer, providing better instructions, and
trying to improve the confirmation of data entry method. In addition, being able
to go back to a data item might solve some of the problems.

4.3 Respondent Debriefing Survey

Interviewers at the ACT Lab conducted a telephone debriefing survey with
TDE respondents during 1989. Given the human-factors literature discussed
earlier, some of the questions focused on understanding and pace of the digitized
voice. Early results of the machine-recorded data showed a substantial number of

repeats and retries of questions, thus, questions were developed to address that


Table 3. Results of Debriefing Survey*

Voice understandable

Easy to understand #'s read back
Never reentered numbers

Never repeated questions (N=209)
Used speed enhancement feature
Instructions adequate

Never had poor telephone connection
TDE experience very favorable

*N=411

97%

100%

60%

83%

63%

98%

93%

93%


While not directly related to surveys, the human-factors literature suggests
several interrelated factors that may contribute to performance errors in a human-
machine interface. First, respondents may not be familiar or comfortable with the
technology. Waterworth (1984) suggests that the language of human-machine
interface is different than human communication as actions are performed in an
order reflecting computer program logic. Since the ability to think in a way that
parallels the logic is not a trivial exercise, those with limited experience may have
difficulty understanding the task and using the system. Second, synthetic speech
is more difficult to understand than natural speech and places greater processing
demands on working memory (Schwab et al., 1985). Thus, comprehension and
memory problems associated with the mode may cause errors.

Synthetic speech includes both digitized speech, where a human voice is
sampled, digitally encoded, and stored, and rule-based synthesized speech,
generated using text as input (Marics and Williges, 1988). The TDE system
utilizes digitized speech, which is less difficult to understand than rule-based
synthetic speech. However, comprehension problems occur with digitizing, as it
introduces distortion into original speech (Cox and Coope, 1981). Research
shows that the understanding of synthetic speech may improve with training. In
an experiment on perception of synthetic speech, Schwab and colleagues (1985)
found training with synthetic speech increases perception performance. Thus,
comprehension may improve with exposure to and experience with the system.
Another factor that may affect comprehension is the pace of the system. Results
are mixed on how the rate of speech affects error in human-machine interactions.

A study by Roelofs (1987) found that the rate of speech did not significantly


21

Werking, George, Alan Tupek and Richard Clayton. (1988). CATI and
Touchtone Self-Response Applications for Establishment Surveys, Journal of
Official Statistics 4, 349-362.

Winter, Debbie L.S. and Richard L. Clayton. (1990). Speech Data Entry:
Results of the First Test of Voice Recognition for Data Collection. Bureau of

Labor Statistics, Washington, D.C.


23

Table 2. Record-Check Data--TDE Mode Error for CES Data Items Before Edit
Corrections

All Wom. Prd. Pay- Prd.
Emp. Wrk. Wrk. toll Hrs.
% Mode error 0.0 2 73 73 44

(SE) (4) (3) (5.2) (3.8) (3.7)


18

calls requesting data. However, for some surveys, self initiation and the lack of
human contact may be problems which would contribute to nonresponse error.
Although respondent acceptance of touchtone collection is very favorable,
there are some steps which can be taken to make the system better. These
include:
- giving respondents enough time to key enter their data, especially for
the first few questions and those which have a long string of digits,
- investigating ways to improve the confirmation of data items, and
- providing longitudinal edit checks to detect reporting of dollars and
cents and other gross errors. The edits could be built into the TDE system with
appropriate questions/probes to respondents to correct or confirm their answers.
BLS has used touchtone collection with one other survey. This survey was a
small sample followup of businesses who had participated in a Survey of
Employer Drug Assistance Programs in 1988. The followup survey in 1990 was
intended to determine if any substantial changes had occurred in the percentage of
businesses providing employer drug assistance programs over the past two years.
These businesses were mailed a short survey questionnaire requesting numeric or
yes/no answers and encouraged to report their data by touchtone. About 20
percent of the businesses reported by touchtone and an equal amount by mail.
Followup activities began about two weeks after the initial mailout, with the
remaining data collected by telephone (CATI).
We believe that other surveys with time dependent data can take advantage of

the time and keypunch savings of touchtone data collection. The mode may


10

over on the keypad to the number directly on the side or below the correct digit.
This accounted for 17 discrepancies.

The fourth type of discrepancy occurred primarily for the all employee
question. There were eight establishments who had a "1" entered for this item in
the TDE data, but had a larger employment number on their survey form. We
speculate that respondents entered "1" twice when confirming the previous
question on month.

Finally, there were a few respondents who had corrected data on their survey
form, but not on TDE. There were other discrepancies which we could not
explain. In addition, several respondents transposed their numbers or were off
one category, accounting for the "other" reasons. For most of the errors, it was
difficult to ascertain if they were caused by the task, or not clearly comprehending
the question or numbers being verified. We suspected the former, but only for the
second discrepancy, adding too many digits, could we really rule out
comprehension problems.

At the bottom of Table 1 are the error rates for the survey questions, ranging
from 1.2 to 2.5. The all-employee, women-worker and production-worker
questions have a lower percentage of errors than payroll and hours. This is not
surprising since payroll and hours worked are usually four to six digits, compared
to two to three digits for the other items. Thus, longer strings of numbers cause
more difficulty. This may be related to difficulties inputing the data, lack of
repondent motivation in correction, or problems remembering longer strings of

numbers during validation.


The first source of data has two components. One is the TDE data recorded by
machine from April to December, 1989. This includes 1,930 observations for the
nine-month period. Establishments were phased into TDE slowly; the number of
establishments peaked in November at 465, with the addition of a large group.
The other component is the same data recorded by establishments on a survey
form. All respondents receive a yearly survey form on which they are requested
to record their data for each month. Mail respondents fill in the form each month
and mail it to the state employment security agency. The agency records the data,
then returns the form by mail for next month's collection. CATI and TDE
respondents are sent the survey form, but they do not return it. However, we sent
a request to the 465 TDE respondents to return their 1989 survey form, and
received 446 forms, a 96 percent return rate. We then compared the TDE and
form data, identifying discrepancies between the two. We refer to these data as
the record-check data.

The second source includes machine recorded data on respondent performance
during the TDE telephone call. The TDE instrument was reprogrammed in
January 1990 to automatically count and record the number of times a question
was repeated due to nonresponse (question repeat), the number of times a
respondent reentered data (data reentry) for each question, and the number of
times an establishment called and hung up before entering data. Unfortunately,
only the questions asking for the month and all employees total could be
explicitly separated into question repeat and data reentry. For the women and

production workers, payroll, hours, and overtime/commissions items, we had to


affect errors. But Marics and Williges (1988) found that speech rate affects
speech intelligibility, measured by transcription errors and response latency.

Thus, potential errors in the human-machine interface can occur from lack of
experience with the technology and task, and from comprehension and memory
problems associated with voice clarity and pace. Yet these problems are
surmountable, as the evidence indicates that experience and training can increase
performance.

3. Data

There were several objectives we considered in measuring TDE problems and
mode error, and determining what data to use or collect. First, it was necessary to
identify if and where problems were occurring. Second, we felt respondents
should identify and interpret problems, but we also wanted measures independent
of respondent assessment. Third, we needed to address problems and errors
associated with the task and comprehension, including the possible improvement
of respondent performance over time.

We decided to assess TDE problems and mode error using three different data
sources, which have in common approximately 465 Pennsylvania business
establishments. These establishments report their monthly survey data by TDE to
the Automated Collection Techniques (ACT) Laboratory at the BLS national
office in Washington, D.C. A small number of the establishments began
reporting by TDE to the ACT Lab in April, 1988 (N=26), and others were added
monthly through November. The majority of establishments moved from mail to

TDE reporting.


19

communicate the importance of timeliness to the respondent. This paper indicates
that measurement errors are controllable using touchtone collection.

Given the timeliness and lower costs of touchtone data collection, we expect it
will be used more extensively in the future. We know of two other applications
of touchtone recognition in a survey setting. Statistics Canada is now developing
a touchtone data collection system for a retail trade survey. In addition, a
touchtone system for a survey of AT&T customers is being developed at Bell
Laboratories (Wendler, 1990).

BLS is also experimenting with the use of voice recognition technology for
data collection in the CES survey (see Winter and Clayton, 1990). While
touchtone telephones are increasingly available, we estimate that only about 70
percent of our respondents have touchtone telephones today. Once speaker-
independent voice recognition technology reaches an acceptable level for the ten
digits needed to report CES data, we expect users will prefer it over touchtone
collection. Further work on measurement errors associated with voice

recognition technology needs to be undertaken.


the TDE data item, but the question was completed on the survey form. This
nonresponse accounted for the greatest number of discrepancies, 82 out of 177.
The item nonresponse rate was 40 percent higher in the first month the
establishment reported by TDE. This indicated problems existed with first time
use of TDE that might decrease with experience. The rest of the item
nonresponse had no identifiable pattern; our suspicion was that some
establishments missed the item, possibly due to distractions, and continued on
with the next question. There were some establishments that had more difficulty
than others, indicated by two or more nonresponses. Nearly half of the item
nonresponse occurred in 18 establishments; in seven of the 18, two or more
nonresponses happened during first-time use. It was difficult to determine the
reasons for item nonresponse. There could have been problems comprehending
the question or answer, or problems associated with carrying out the task,
including data entry and distractions associated with an office setting.

[Table 1 about here]

The second type of discrepancy was entering extra digits or, in a few cases,
entering too few digits, which accounted for 18 of the 177 errors. This was
specifically a problem associated with carrying out the task for payroll data,
where four respondents tried to enter cents instead of rounding to the nearest
dollar. Several of the same respondents appeared to enter a half hour, 50, for
production-worker hours rather than rounding.

For the third type of discrepancy, the TDE numbers were nearly the same as
those on the form, but one number off. The number entered incorrectly indicated

a potential task problem in that the respondent may have had their fingers slide


FIGURE 1 - MACHINE RECORDED DATA
PERCENT OF QUESTIONS REPEATED/REENTERED

14
7 vere eeee retire rote PPPrer ney rhe pave Sm per Eve ANE ae ore eco U
2 a ene
[Rane Ree oO Re erceeae | mmarree. | anmnrrrrnre
Ss weeny ©
eae coors goer toveatitd a ete aeeeee tat Luca ts Rh keh vals onsareeetscone me eecaecte tite} aeaet TA

| °Nonth AE WW. PW. Payroll Hours OT/Comm

CES DATA ITEMSIN=1203), EXCEPT OT (N=899)
i @ 90% Confid. int. 4 Point Estimate
| Prvproduction workers, OT -Overtime >
Comm=Commissions



ASSESSING MEASUREMENT ERRORS IN A
TOUCHTONE RECOGNITION SURVEY

POLLY A. PHIPPS AND ALAN R. TUPEK!

ABSTRACT
Electronic data collection utilizing touchtone recognition is in place for a monthly
establishment survey at the Bureau of Labor Statistics (BLS). The Touchtone
Data Entry (TDE) system features digitized phrases requesting respondents to
answer questions using the numeric keypad of a touchtone telephone. TDE has
substantial implications for ering survey costs; many labor intensive activities
are eliminated. However, little is known about measurement errors associated
with this mode of data collection. This study assesses TDE mode error using
three sources of data, which allow for analyses of errors associated with selected
aspects of the human-machine interface. In addition, instrument design issues
associated with mode error are addressed. We conclude by extending the
implications of our findings to other surveys.
KEYWORDS: mode of data collection; human-machine interface.
1. INTRODUCTION
The U.S. Bureau of Labor Statistics (BLS) issues monthly employment
estimates for the United States from a survey of 300,000 business establishments.
This survey, the Current Employment Statistics (CES) survey, provides one of the
earliest monthly measures of U.S. economic health. However, the preliminary
estimates from the survey are released with data from only about one-half of the
business establishments in the survey. Revised estimates are produced one and
two months after the initial press release. The low response rate for the initial

press release can result in large revisions to the estimates. The BLS began

1 Polly Phipps, Collection Procedures Research Laboratory, Bureau of Labor
Statistics, Room 2821, 441 G Street N.W., Washington D.C. 20212; Alan R.
Tupek, Office of Employment and Unemployment Statistics, Bureau of Labor
Statistics, Room 2919, 441 G Street N.W., Washington, D.C. 20212.


12

times respondents reenter their data, and how often the question is read back a
second or third time before they respond. Information can also be kept on those
respondents who hangup before entering data.

The machine-recorded data was collected for a total sample of 1,232
observations over a three-month period. There were approximately 474 unique
respondents, many of whom provided data for two or three months. There were
few differences by month, and hence, all data are presented for three months
combined.

Figure 1 provides, for each data item, the percent of calls for which the
question was stated to the respondent more than once. The question could be
stated a second time if the respondent does not answer in two seconds (repeat), or
if the respondent fails to confirm his or her answer by entering "1," after it is read
back (reenter). The table indicates that the first two questions on the month and
all employees, and the payroll and hours questions have a higher rate than other
data items. The higher rates for the first two questions may be due to respondents
needing a few questions to orient themselves to the system. The earnings and
hours questions generally have the greatest number of digits, so we suspect that
data entry errors are more likely to occur, causing the question to be reread and
the answer to be reentered.

[Figure 1 about here]

Figure 2 provides data for the first two questions on month and employment
for repeated questions after no answer and after lack of confirmation, separately.
It was not possible to acquire data separately for the other data items. The CES

touchtone system requires respondents to enter at least their report identification


11

Table 2 shows the potential effect of the discrepancies on the CES data items,
calculated by taking the sum of the difference between the values in the TDE
system and the form, then dividing by the sum of the values on the form. The
CES Survey uses a link-relative estimator for published estimates. The estimates
in Table 2 do not take into consideration this estimator. However, the estimates
in Table 2 provide an indirect measure of TDE mode error on survey estimates.
None of the error is significantly different from zero at the five percent level.
However, the potential for mode error appears to be more serious for production
workers, payroll and production-workers hours. In the sample used for this study,
the number of production workers are overestimated by 7.3 percent, payroll by
7.3 percent, and hours by 4.4 percent.

[Table 2 about here]

Nearly all the discrepancies would have failed the edit parameters used in the
CES survey and been corrected. The resultant effect of the discrepancies after
edit corrections is nearly zero for all data items. Examples of the discrepancies
include a respondent who incorrectly meres payroll for the number of
production-workers item, increasing production workers by over ten thousand,
and two respondents who incorrectly entered the number of production-worker
hours for number of production workers, raising the latter by several thousand.
Payroll and hours have similar gross discrepancies, including the establishments
who put in cents instead of rounding for payroll.

4.2 Machine-Recorded Data
The touchtone system provides a tool for assessing difficulties which

respondents have with this mode of collection. TDE can record the number of


combine repeats and reentrys, due to the structure of the original computer
program. We refer to these data as the machine-recorded data.

The third source of data is a telephone debriefing survey, conducted from
January to April of 1990 with the Pennsylvania establishments on their
experiences with the TDE system. Approximately 411 business establishment
respondents completed the interview, an 88 percent response rate. The questions
covered such topics as voice quality, pace of interview, task problems, use of
systems features, adequacy of instructional materials, and a system rating.

4. Results
4.1 Record-Check Data

When we requested TDE establishments to return their survey forms, our first
question was: how many really used the forms? We speculated that one source
of mode error was respondents who did not complete the form for use when
entering their TDE data, which would increase demands on their memory. Those
who did not complete the form might be more likely to enter and/or verify
incorrect data. Thus, the request for the survey forms indicated that respondents
were to return the form regardless of whether they completed it or not. However,
only one establishment mailed in a blank form; all others sent in completed forms.
While nonrespondents may work from memory, most of the respondents had
completed their forms, giving us reason to believe memory problems due to lack
of form use were not a major source of errors.

When comparing the data received by TDE with that on survey forms, we
identified and coded discrepancies. The results are shown in Table 1. The first

type of discrepancy occurred when there was a symbol indicating nonresponse for


22

Table 1. Record-Check Data--Number and Type of TDE Discrepancies by Item,
and Percent Error by Item

All Wom. Prd. Pay- = Prd.
Emp. Wrk. Wrk. roll Hrs. _— Total

TDE item

nonresponse a 19 11 23 29 82
1-2 few/too

many digits 3 1 S) 7 z 18
Slipped on

keypad 2 ] 2 ) 5 17
Dis/confirm

"1", "0" exxor 8 3 1 1 1 14
Form corrected,

not TDE ] 1 1 4 2 12
No apparent

error reason 6 1 sl 6 6 26
Other reasons 1 1 1 2 a 8
Total 23 29 28 48 49 177
% Exror 1.2 15 15 25 2.5 18
(SE) (2) (3) (3) (4) (4) (3)

N=1930 for each item
4 If this item is not completed, the respondent cannot proceed.


collection at a considerably reduced cost per unit. Collection by TDE is now
expanding in the CES, beginning with the establishments with the greatest
employment. Research continues in determining ways to improve the touchtone
system, even though respondents’ acceptance of touchtone collection is highly
favorable.

The purpose of the current research is to identify respondents’ problems using
the touchtone system and to measure errors associated with the mode. Results
indicate where improvements are needed to reduce respondent probleii.s as well
as errors. In section 2 we address the potential errors due to the TDE mode of
data collection, which requires a human-machine interface. In the third section
we describe the three sources of data collected for the study. A record-check
survey, machine-recorded data and a respondent debriefing survey are used in the
analysis of problems and errors. Section 4 includes the methods, analyses and
findings from each of the data sources. Finally, in section 5 we provide an overall
assessment of measurement errors due to this mode of collection, suggestions for
improving the system, and implications for other surveys.

2. Measurement Error in a Human-Machine Interface

Respondent use of a system such as TDE to answer survey questions has little
precedent. However, touchtone recognition is widely used in such procedures as
electronic banking and customer-controlled telephone services. While these
services may save time and expense, they have a potential to alienate users.
Problems and errors can originate with the system, task, or respondent. System
problems primarily generate nonresponse error, while measurement error is

related to the task and respondent performance.


15

topic. In addition, respondents were asked to rate the TDE system and answer
questions relating to systems design.

Table 3 presents the results of the survey. Respondents expressed little
difficulty in comprehending the digitized voice. About 97 percent said the voice
was very understandable, three percent said it was somewhat understandable, and
no respondents chose the not very understandable category. All respondents
indicated that it was easy to understand the numbers as the voice read them back
for confirmation. During the first two months of the survéy, 177 respondents
were asked about the pace of the interview. Most of the respondents said the pace
was about right (88%), two percent said it was too fast, and about ten percent felt
it was too slow. Our suspicion throughout the study was that comprehension was
much less a problem than difficulties carrying out the task. While it was difficult
to separate out the two in the records-check data, the debriefing interviews lend
support to our suspicion.

[Table 3 about here]

For task difficulties, 60 percent of respondents said they never had to reenter
numbers, while 39 percent said sometimes, and one percent indicated they had to
reenter numbers often. Interviewers asked respondents the reasons for this, and a
majority indicated they accidently entered a wrong number. Others said they did
not have enough time, were distracted, entered numbers too fast, and a few other
miscellaneous reasons. In the latter several months of interviewing, respondents
were asked about the repeating of questions (without reentering data). About 83
percent of the 209 respondents asked this question said they never found it

necessary to repeat questions, while 17 percent said they repeated sometimes.


17

good, i.e., respondents said they easily understood numbers being read back for
confirmation.

Record-check data show that establishments may have carried over their
confirmation of the month into the all-employee questions. In addition, the
machine-recorded data indicate respondents often do not respond to the month
question the first time it is asked. If respondents are using their survey forms, it is
likely that moving from the identification number at the top of the form, to the
month and data items further down, they require extra time to locate themselves.
This problem could be solved by locating all information that needs to be entered
in one location. This could reduce the number of question repeats for the "month"
item and potentially lower costs by reducing the length of calls. Question repeats
for other items might be reduced by giving respondents more time to respond,
since they report they were distracted from the task. However, since most
respondents feel the pace of the system is about right, and many are using the
speed enhancement feature, adding more. time could cause frustration. Probably
little can be done to reduce the number of reentries, as respondents indicate they
have entered a wrong number and need to correct it.

The data show that errors are reduced with experience. This indicates that a
panel survey may be best for this mode of data collection. For surveys requiring
numeric or yes/no responses, we believe touchtone also has great potential. The
errors are not extremely serious, and respondents rate their experiences using
TDE very favorably. TDE may be particularly attractive to business respondents,

who can call at times convenient for them, rather than be interrupted by telephone


The features of the BLS touchtone system include the ability to:

- detect legitimate establishment respondents based on a match to a file of
establishment numbers;

- vary the set of questions depending on the industry of the establishment,

- read back all responses for respondent confirmation using a computer
simulated (digitized) voice (respondents are requested to enter "1" to confirm
their answer or "0" to reenter their answer);

- wait two seconds for respondents to begin entre .u.2 their answer, and
wait two seconds between digits before interpreting data entry as complete (data
entry is also assumed complete if the entire field length is filled -- example: the
data item "month" has a two-digit field);

- repeat each data item question up to three times (for identification
number, month and all employees) or request the respondent to confirm that they
have no answer for the question (for all other data items), if a respondent does not
confirm their answer or if no answer is provided in the two seconds after the
question is read;

- store the date, start and end time of each call, and all data items.

In addition, the panel of touchtone respondents receive a postcard reminder
each month and a prompt call if they do not self-report by a specified date. The
prompt call asks the respondent to telephone into the touchtone system as soon as
possible. Data are not usually collected during the prompt call.

Earlier tests of the TDE system provided data on the timeliness of response,
the cost of collection, and edit failure rates. Werking and colleagues (1988)
found the timeliness of TDE collection with CATI back-up as effective as CATI


FIGURE. 2 - MACHINE RECORDED DATA

PERCENT OF QUESTIONS REPEATED
VERSUS REENTERED FOR {ST TWO DATA ITEMS

Month-Repeated Month-Reentered AE-Pep eated AE-Peentered
CES DATA ITEM & PROBLEM TYPE (N=1203}

BM 90% Confid. Int. + Point Estimate



investigating the use of automated collection techniques in the mid 1980s to
increase the timeliness of response and reduce the potential for large revisions.

The CES survey has traditionally been collected by mail through state
employment security agencies. Research tests conducted between 1984 and 1986,
involving the replacement of mail collection with computer-assisted telephone
interviewing (CATI), have shown CATI to be an effective means for improving
the timeliness of response. Response rates under CATI collection have been
between 85 and 90 percent for preliminar” .stimates, compared to 45 to 50
percent with mail collection. While CATI collection has been effective in
improving the timeliness of response, the cost of full CATI collection in the CES
survey can not be absorbed within the survey's current budget. Research has been
initiated into the area of touchtone data entry (TDE) to develop an alternative
collection method with the performance gains of CATI, but with a lower unit
cost.

The primary reason to consider collection by touchtone self-response is to
reduce the cost of collecting data by CATI, while maintaining the timeliness and
quality of CATI collection. The CES survey seemed to be a good candidate for
touchtone collection, since only about six numeric data items are collected each
month, The data items include: all employees, women workers, production
workers, production-worker payroll, production-worker hours, and for some
industries, over-time hours or employee commissions. Establishments are asked
to report totals for each data item for the pay period which includes the 12th of
the month. TDE respondents are also asked to report their establishment
identification number and the month for which they are providing data.


13

number, the month, and employment. The system will accept item nonresponse
for the other data items. The mandatory entering of month and employment
allowed the separation of repeated questions after no answer versus after a
respondents lack of confirmation of the answer provided. Of the 10.1 percent of
respondents with problems on the month question, almost all were due to repeats,
that is, two seconds passed without a response. On the other hand, problems with
the employment question were almost evenly split between repeats after no
previous answers and reentrics after !ack of confirmation of the previous answer.
[Figure 2 about here]

Only two percent of the calls received by TDE each month included just a
report identification number. During these calls, respondents had simply hung up,
or could have been cut off the system.

In addition, the TDE system records all calls received that include at least the
report identification number, month and employment. Using the TDE component
of the record-check data discussed earlier, we identified respondents with more
than one call during a month and coded reasons for the call-backs. In all, about
four percent of the respondents called the system more than once in a given
month. Most of these respondents provided data items which were not supplied
on the initial call (2%). An additional one percent provided corrections to some
data items in addition to new data items. Many of these respondents appeared to
have had problems with entering the data the first time. Another one percent of
the respondents called back only to provide corrections to data items previously
supplied or provided identical data. These calls were often several days later,
possibly implying that new data had been obtained from their records. In the case


16

The majority of those who repeated said they were distracted, while others said
there was not enough time, and a few other miscellaneous reasons.

Most respondents had little difficulty with telecommunications failure, as 93
percent said they never experienced a poor telephone connection. Of the 33
respondents who did get a poor connection, most said it happened only once. A
large number of the respondents, 63 percent, used the pound sign, a feature of the
system designed for speeding up the reporting of data.

Nearly all respondents said the instructions sent to them as they bey ui TDE
were adequate. Interviewers asked respondents to rate their experience with
TDE. Approximately 93 percent rated it very favorable, while six percent said
their experience was somewhat favorable, and three respondents indicated a not
very favorable experience.

5. Discussion

The data show few serious problems with the TDE mode of data collection.
Record-check data indicate some item nonresponse error, which is associated
with first-time users. Entering additional or incorrect digits appears to be the
most serious problem affecting the data items. However, in a panel survey,
longitudinal edit checks can reduce this error, as could logical edit checks in all
surveys. In addition, the rounding of data needs to be addressed in respondent
instructions. Both the record-check and machine-recorded data show that there
are more difficulties with longer strings of numbers, probably in both entering
data and verifying incorrect data. The latter could indicate difficultly

remembering longer number chains, as comprehension of numbers appeared to be