RESPONDENT MOTIVATION, RESPONSE BURDEN, AND DATA QUALITY 

IN THE SURVEY OF EMPLOYER-PROVIDED TRAINING 

James Kennedy and Polly Phipps 

Bureau of Labor Statistics 

Annual Meeting of the American Association for Public Opinion Research, May, 1995 

Ft. Lauderdale, FL. 

 The Survey of Employer-Provided Training (SEPT2) has been designed to provide 

information on the nature of training that is provided or financed by employers, including the 

type, intensity, and cost of training activities. The types of training covered by the survey are 

orientation, safety and health, apprenticeship, basic skills, workplace-related, and seven types 

of job-skills training. The intensity and cost measures include the length of the training 

activity, the number of individuals involved, and the wages of participants. 

 In this paper, we report on a multivariate experiment conducted by the Bureau of Labor 

Statistics (BLS) to test many aspects of an employer's training log, including cognitive and 

motivational variables as well as variations on the collection procedure. Sixty employers, or 

individuals who are knowledgeable about training at their establishments, were asked to keep 

a record, over a two-week period, of all formal training activities that occurred each day. The 

sample comprised 20 small (50-100 employees), 20 medium (250-600) and 20 large (1000+) 

establishments, selected for comparisons among different-sized businesses. 

Research Questions 

 An experimental question had to do with the effects of respondent's attitudes and beliefs 

about the survey in predicting attrition and deterioration of data quality. BLS field 

economists (FEs), who collected the survey data, judged respondents' understanding of the 

survey and their attitudes toward it on the first, third, seventh, and fourteenth days, and 

interviewed respondents on those days about such things as how interesting and effortful the 

survey was for them. Further, at the end of the first week respondents reported their attitudes 

about SEPT2. Correlation between these reports and diligence in maintaining the log 

provided important information regarding respondent attitudes, motivation and data quality. 
 Another experimental question for SEPT2 research concerned the efficacy of a personal 

visit midway through the data collection interval. While such as visit has been shown to 

motivate respondents to continue responding diligently, it was not known what the 

comparative effect of a phone call would be. On one hand, during a personal visit midway 

through the survey, the field economist might be able to catch errors which were being made 

before it was too late, and further motivate the respondent who may have been forgetful or 

had not taken the time to complete the log. On the other hand, a phone call may be a better 

procedure for an establishment survey, as respondents are busy and may not appreciate the 

interruption, especially when they are being diligent about keeping the log. Thus after a week 

of reporting training, the respondent received either a personal visit or a phone call. If no 

difference in data quality was found, BLS might find it more economical during the field of 

SEPT2 to implement phone call contact after the first week, as opposing to sending someone 

to the worksite. 

 We anticipated data differences between large, medium, and small establishments, as a 

result of record-keeping practices and the sheer amount of information to be handled. Large 

companies were expected to provide more formal training, while smaller ones would provide 

more informal types of training. On one hand, the larger establishments should have easier 

access to records, especially when they have a department dedicated to training; on the other 

hand, larger companies, because they would have more activities to enter into the log, were 

expected to be more burdened. Thus, we anticipated problems with data quality due to large 

establishments' respondents taking shortcuts. 

Procedure 

 The procedure called for initial prescreening of establishments to select those that 

expected to offer training during the data-collection period. Following selection screening, a 

Bureau of Labor Statistics field economist contacted the establishments by telephone to 

identify a knowledgeable respondent and schedule an initial meeting. The phone call was 

followed up with a confirmation letter. 

 At the initial meeting the FE instructed the respondent in the procedure, i.e., maintaining 

the log, and answered the respondent's questions. The log was designed so that one page was 

filled out for each training activity, i.e., each class or session was recorded separately. 
Respondents were asked to report the dates and times of training, the names or identifiers of 

those who attended--both trainees and trainers--and the wages of those involved. On the third 

day of data collection, the FE called the respondents on the telephone to check on their 

progress and answer their questions. 

 Half the respondents received a personal visit from the FE on the seventh day of data 

collection, while half received a phone call. All respondents completed a short attitude scale 

on the seventh day, describing their thoughts and feelings about the procedure. 

 On the fourteenth day of data collection the FE debriefed the respondent. This occasion 

provided information to BLS about respondent burden, including a description of the process 

by which the respondent acquired the training information, information about any problems 

that were encountered, and observations and suggestions that respondents had for further 

development of SEPT2. 

 On the first, third, seventh, and fourteenth days of the procedure the FE interviewed the 

respondent to assess their understanding of various aspects of the procedure and their 

attitudes and opinions about participating in the survey. FEs also kept a record of the 

respondents' perceptions, attitudes and understanding at each of these meeting dates. Thus, a 

large amount of psychometric information was gathered throughout the interview process. 

Results

 The log collected two kinds of data: reports of training activities that occurred, and 

information about the wages of training participants. The attitude scale and the many 

interviews also provided a rich base of information about the response process. Field 

economists' reports informed us about the administration of the survey and issues that might 

have otherwise gone undetected. In this paper, we focus on wages, and how wage data 

quality was affected by respondent attitudes, motivation, and response burden. 

Attitudes 

 Field economists rated respondents' attitudes about supplying wages as moderately 

positive on average. After the initial interview, field economists reported on respondents' 

"attitude about supplying wage data," on a 6-point scale running from 0, "Refuses," to 5, 

"Willing to supply data." The same rating was made after the final meeting. While the 

means lie above the midpoint of the scale at 3.0 for the initial interview and 3.6 at the 14-day 
visit, they certainly do not suggest that respondents expressed enthusiasm for supplying 

wages, and do not suggest that cooperation would come readily from them. The field 

economist rating first and second rating correlated significantly but weakly, r(52)=.289, 

p=.040, suggesting that field economists' perceptions of respondents' attitudes were slightly 

consistent over the two-week period. 

Quality of Wage Data 

 Respondents were asked to supply names or identifiers of trainees and in-house trainers 

and their wages: this part of the survey was not very successful, with about half of 

respondents failing to report wages with sufficient accuracy. 

 The quality of wage data was rated for each establishment. The categories included: 

actual wages, averages for a class or group of trainees, wages rounded off to the nearest 

round number ($15,860 reported as $16,000), wage ranges (i.e., $10,000-30,000), incomplete 

wages and wages missing entirely. These ratings were then further dichotomized into 

Good/Not-Good numerical ratings for correlation with other scales. Several hypotheses were 

suggested: that wage quality would decay as burden increased, that respondents with more 

positive attitudes would provide higher quality wage data, and that respondents who had 

difficulty understanding the procedure would provide lower-quality data. In order to seek 

evidence for these hypotheses, correlations were calculated. 

 The most significant correlation suggests that respondents who reported on the seventhday attitude scale that participation in the survey was "Interesting" were more likely to 

provide wage data correctly, r (52) =.361, p=.007. Interestingly, field economists' first-day 

ratings of the respondents' attitudes about reporting wage correlated very weakly with wage 

data quality, r(52)=.109, p=.432, while the fourteenth-day ratings correlated significantly, 

r(52)=.347, p=.010. The only other correlate which was significant at α=.05 was the field 

economists' assessment of respondents' attitudes toward the survey on the fourteenth day. 

 Several variables approached significance. The amount of reported time spent, as 

recorded in the seventh-day interview and in the attitude scale, correlated inversely with wage 

data quality, r(45)=-.283, p=.053, and r(47)=-269, p=.061, respectively, as did the report of 

total time spent collected on the last day, r(51)=-.261, p=.059.. Respondents who spent more 

time on the log gave poorer quality data. Further, the self-reported effort required in keeping 
the log correlated negatively with wage data quality, r (53) =-.249, p=.067. These findings 

suggest that the burdensomeness of the task might result in poorer data quality, as harried 

respondents scramble to amass information for the survey. But also note that the number of 

training activities reported at the seventh day did not correlate significantly with data quality. 

Field economists' predictions, after the initial interview, that respondents would be 

conscientious the entire time correlated marginally with wage data quality, r (53) =.226, 

p=.098, as did the field economists' estimates of the respondents' motivation after seven days, 

r (47) =.270, p=.061. 

 In analysis of categorical variables, no significant effect upon wage data quality was found 

for most variables. A significant effect on wage data quality was found, however, for 

establishment size. Further, the kinds of unusable data differed significantly among size 

classes. Large establishments were much more likely to provide poor quality data, and small 

businesses were more likely to provide good data, χ2(2) =10.72, p, .005. Analysis of the 

distributions of the various kinds of not-good wage data, by establishment size, found that 

large establishments were significantly more likely than others to provide averaged, 

incomplete, and missing wage data, and were less likely to provide good data, χ2(10)=22.31, 

p,.014 (also, see table below). 

Types of data produced by respondents from various sized establishments. Good Averaged Ranges Rounded Incomplete None

Small

Medium

Large

0

2

4

6

8

10

12

14
 It was hypothesized that an effect of the personal visit on the seventh day would be to 

motivate respondents to provide better data. No such effect was seen in the wage data (see 

table below). Wage data quality was equal for respondents receiving personal visits and 

phone calls. Analysis of the various categories of wage data shows that establishments 

receiving personal visits were more likely to give averaged wages, while those receiving 

phone calls were twice as likely to return logs with missing wage data. As an explanation, it 

might be proposed that averaged wages look like good data, as a self-presentational device; 

respondents can deliver logs that look correct in the face-to-face setting, without 

embarrassing themselves and also without going to the trouble of responding rigorously. 

Respondents who communicate with the field economist over the telephone, on the other 

hand, are less bound by interpersonal concerns, and perhaps as a consequence of this feel less 

pressure to deliver data at all. Effects were not statistically significant in this test. 

Types of data produced by respondents who received personal visits vs. those who received 

phone calls after one week Good Averaged Ranges Rounded

Incomplete

None

Visit

Phone Call 0

2

4

6

8

10

12

14

 In sum, it is interesting to note that the best predictors of wage data quality are the 

respondents' attitudes about participation, especially their expression that they find 

participation interesting and the field economists' ratings of their attitudes toward the end of 
the study, and establishment size. Two very different lessons can be learned from these 

results. First, it may be important to select a respondent who is motivated to cooperate: 

perhaps the selection process needs to be modified to allow the field economist to find the 

person who will participate most adequately. Note that the knowledgeability of the 

respondent produced a correlation coefficient with wage data quality which was essentially 

zero. Second, response burden needs to be carefully assessed in establishment surveys, as 

there are clear trade-offs between burden and quality. It appears that the employer log is not 

well suited to the situations found in large establishments: when training is very plentiful and 

records are decentralized, it may be too burdensome for respondents to provide complete 

wage data.