APPLYING COGNITIVE THEORY TO AN ESTABLISHMENT MAIL SURVEY 

Polly A. Phipps, Bureau of Labor Statistics 

Room 2821, GAO Bldg., 441 G Street NW, Washington, DC 20212 

Introduction 

This paper reports the results of a study using 

cognitive theory to assess an establishment mail 

survey. Two methods of evaluation are used in the 

study, a survey pretest and a Response-Analysis 

Survey (RAS). The paper is divided into four 

sections. First, background information on the mail 

survey is provided. Second, background information 

on and findings from the pretest and RAS are 

detailed. Third, recommendations for the survey are 

offered using information processing theory from 

cognitive psychology, as it has been applied to 

surveys (Tourangeau 1984). Three of the four stages 

of information processing most applicable to this 

research are highlighted, including respondent 

comprehension, retrieval and response. Fourth, the 

methods used in the study are discussed and 

conclusions from the study are drawn. 

Survey Background 

The BLS, i-n cooperation with State employment 

security agencies, collects monthly survey data on 

employment, hours, and payroll from a sample of 

300,000 nonagricultural establishments. In March 

1989, a subsample of 3,600 monthly respondents was 

mailed a supplemental survey on annual nonwage 

cash payments and payrolls. The mail survey was a 

pilot test conducted to determine the feasibility of 

collecting nonwage cash payment and payroll data on 

an annual basis. Nonwage cash payments to 

employees, not captured in the monthly payroll data, 

were considered a growing part of total employee 

compensation and important to measure. 

The mail survey requested information on nonwage 

cash payments made to production workers and all 

employees during the calendar year 1988. 

Respondents were asked to indicate if their 

establishment gave certain types of nonwage cash 

payments and to report their monetary amounts. 

Nonwage cash payments consisted of six categories: 

bonuses and awards, lump-sum payments, 

commissions paid less often than monthly, cash profit 

sharing, severance pay, and other. Respondents were 

also asked to report the amount of 1988 payrolls for 

production workers and all employees. 

Two survey designs were used in the pilot, a long 

and short form. Each survey was on a single sheet of 

paper, with both sides utilized. However, the layout 

of the front page of the two forms differed. The short 

form requested four pieces of information near the 

top of the front page. They included the nonwage 

cash payment monetary totals for all employees and 

production workers, and the payroll totals for all 

employees and production workers. There were four 

boxes for a respondent to place the totals. Further 

down the front page of the short form, respondents 

were to check a grid-like series of yes/no questions 

on the six categories of nonwage cash payments. The 

questions included: did the establishment make the 

payment type, was the payment included in the allemployee box, was the payment made to production 

workers, and was the payment included in the 

production-worker box? 

The long form differed from the short as the totals 

boxes were placed at the bottom of the front page, 

rather than near the top. Near the top of the front 

page of the long form was a grid with the six 

categories of nonwage cash payments on the side and 

several questions across the top. Questions included: 

did the establishment make the payment type, and 

were any payments made to production workers? 

After the respondents checked answers to the 

questions they were to report the monetary totals for 

all employees and production workers for each 

payment category. Then they were to go on to fill out 

the nonwage cash payment and annual payroll totals 

at the bottom. 

The back of the short and long form was identical. 

It included examples of three types of employer 

payments in a list format. The employer payments 

included nonwage cash payments, payroll, and other 

payments, the latter were to be excluded from the 

report. Further down the back page were item 

instructions and survey definitions. The survey was 

primarily printed in a size eight Helvetica font. 

pretest Results 

The nine-unit pretest procedures included initial 

telephone contact with establishments to elicit 

respondents. Following the telephone contact, 

respondents were mailed a survey to complete. Then 

a personal interview was conducted with six of the 

respondents at their establishments, and three 

respondents were interviewed by telephone. A 

retrospective think-aloud protocol was used. The 

procedure involved open ended questioning of the 

respondent on completed survey responses. The areas 

of questioning for the pretest focused on respondent 

comprehension of instructions and definitions and 

problems retrieving the data, including reasons for 

item nonresponse and potential errors of omission. 

The pretest interviews indicated that all but one of 

the nine respondents thought they had a good 

understanding of the survey. Respondents tended to 

focus on the unavailability of records. They attributed 

item nonresponse to retrieval problems with records, 

i.e., the company's records did not break out the 

specific categories the survey requested. This was 

true for both payments and payrolls. For example, in 

four of the nine establishments, severance payments 

were not identified as such, and were included as part 

of the regular payroll. Thus, respondents were unable 

to generate a monetary amount for severance 

payments. In the case of payrolls, the majority of 

establishment records did not discriminate between 

production workers and all employees. Totals for the 

production-worker category were either unavailable 

or very time consuming to calculate. 

Pretest questions on the understanding of payment 

types led to some very specific discussions on what 

should be included and excluded. Since the survey 

defined payment types primarily by examples, the 

discussion of specifics was understandable. But 

608 
during the discussions it became clear that 

respondents did not have a global understanding of 

what was encompassed by the term "nonwage cash 

payment." For example, one respondent could not 

understand why an employer contribution to a thrift 

plan was not a nonwage cash payment. She knew the 

payment was in a list of example exclusions, but she 

did not see the reasoning. When she was told it was 

not immediately available from the employer as cash, 

it became clear. However, that global definition was 

not part of the survey. 

One of the pretest questions involved reading off a 

list of specific bonuses and awards types and asking 

if the establishment made the payment during the 

year. This question elicited errors of omission from 

three of the pretest respondents. The errors were 

primarily omission of small awards, such as safety 

and employee-recognition awards. Even though the 

monetary value of the payments was not great, the 

omissions clarified that we were dependent on the 

respondent's memory in item reporting. 

Response Analysis Survey (RAS) Res.ults 

In the RAS, a subsample of establishments 

responding to the mail survey were subsequently 

surveyed on the data they had provided. Questions 

covered the same major areas as the pretest, however, 

the RAS was a structured as opposed to an openended instrument. It was designed to be carried out 

over the telephone in a short period of time. 

Establishments were selected into the RAS sample 

on the basis of several types of actual and possible 

response and nonresponse errors listed below (1-5). 

The sample design was chosen to study potential 

problem areas for future survey improvement, not to 

measure error in the survey population. 

1. An establishment indicated they made a nonwage 

cash payment to all employees but gave no monetary 

amount. 

2. An establishment indicated they made a nonwage 

cash payment to production workers but gave no 

amount. 

3. A nonwage cash payment was indicated for all 

employees, and the production-worker amount was 

equal to that of all employees. 

4. No amount was given for the all-employee annual 

payroll. 

5. No amount was given for the production-worker 

payroll. 

Establishments who had not responded to payroll 

items (4 and 5) accounted for about 60 percent of the 

sample. The correspondence between production 

worker and all employee amounts for nonwage cash 

payments (3) accounted for about 30 percent of the 

sample. After the fact, this turned out not to be a true 

error. A RAS question was asked concerning the 

figures. Most respondents said that the figures were 

truly equal: payments were only given to production 

workers. Not providing a monetary amount for 

nonwage cash payments for all employees (1) was 

the only other category accounting for a noticeable 

amount of error. 

The RAS was conducted by telephone interviewers 

in the Automated Collection Techniques Laboratory 

of the BLS. The sample included 350 establishments; 

a 96 percent response rate was achieved. All 

establishments received a prenotification letter 

explaining the RAS and a xeroxed copy of their 

original form for reference purposes. 

Similar to the pretest, certain questions in the RAS 

interviews indicated respondents thought they had an 

understanding of the survey content. Most 

respondents said they did not leave out any payments 

due to uncertainty of whether a payment was to be 

included or not. In addition, establishments who 

checked on the survey form that they made nonwage 

cash payments, but gave no amounts (errors 1 and 2), 

stated difficulties with records as reasons for the 

nonresponse, rather than any misunderstanding or 

overlooking of what was to be provided. This 

included 66 omissions from approximately 35 

establishments. 

However, questions were asked of all RAS 

respondents to elicit possible "errors of omission," 

i.e., payments that should have been included, but 

were not. All respondents were asked if their 

establishment gave Christmas or executive bonuses, 

or merit, incentive or employee recognition awards 

during calendar year 1988. If they gave any of the 

payments, they were asked whether they included it 

in the figures they reported, and if not, the reasons for 

any omissions. 

The reporting errors are shown by payment type in 

Table 1. The questioning did elicit errors: (A) 115 

establishments reported they gave Christmas 

bonuses, while 37 (32 percent) did not include them 

as nonwage cash payments; (B) 70 reported executive 

bonuses, and 30 (43 percent) did not include them; 

(C) 25 reported merit awards, and 11 (44 percent) did 

not include them; (D) 40 reported incentive awards, 

but 16 (40 percent) did not include them; and (E) 22 

reported employee-recognition awards, and 12 (54 

percent) did not include them. Some of the 

establishments are included in more than one of the 

payment categories. Approximately 13 

establishments did not report their payments in two 

of the categories; four establishments in three of the 

categories, and five establishments in four of the 

categories. Thus, 22 respondents accounted for half 

of the 106 unreported payments. 

Table 1. Payment Reporting Errors 

PAYMENT GAVE DID NOT 

TYPE PYMT. INCL. 

Xmas bonuses 115 37 

Executive bonuses 70 30 

Merit Awards 25 11 

Incentive Awards 40 16 

Employeerecognition awards 22 12 

TOTAL 272 106 

Table 2 shows the reasons for payment omissions. 

Since the "all-payments" category includes some 

employers more than once, a separate column is 

included for one category, Christmas bonuses. 

Respondent difficulty reporting the payments due to 

the unavailability or limitations of records (categories 

1, 5, 6 ,7) accounts for the largest share of 

underreporting. Second, approximately 15 of the 106 

omissions (category 2) are due to reporters simply 

forgetting the payment, a retrieval problem. In 

addition, of the 34 respondents who may have 

609 
understood that they should include the payments 

they left out (categories 1, 3, 5, 7, 8, 10), over a third 

checked the bonus and awards box no on the original 

survey form, indicating they did not make the 

payment. 

Table 2. Reasons for Payment Omissions 

REASONS 

1. No separate records for payment 29 

2. Forgot it 15 

3. Too time consuming 

4. Did not understand survey 

5. Too difficult to prepare 

6. Included in payroll2 

7. Unavailable from records 

8. Confidential 

9. Thought it meant cash/$bills 

10. Unavailable from this office 

11. Other 

12. Missing 

TOTAL 

ALL XMAS 

PYMTS. BONUS 

13 

7 

10 2 

8 3 

6 0 

4 3 

4 0 

4 0 

4 1 

3 1 

19 5 

4 2 

106 37 

As discussed earlier, establishments not reporting 

payroll items accounted for the largest proportion of 

those selected into the RAS sample. Table 3 shows 

the reasons given by respondents for not reporting the 

annual-payroll totals. RAS respondents were not at 

all clear that they were to provide payroll totals, 

shown by the overlooked or did not understand 

categories. Additionally, those reporters omitting 

production worker payroll had more difficulties with 

records, shown by the 21 reporting no separate 

records for production workers. 

Table 3. Reasons for Payroll Omissions 

ALL PROD. 

REASONS EMP. WRKR. 

1. Overlooked it 37 59 

2. Did not understand they were 

to provide it 17 24 

3. Not available from this office 6 4 

4. Too time consuming 4 9 

5. Same as production workers/ 

all employees 3 8 

6. Unavailable from records 3 5 

7. Don't know why 3 4 

8. Too difficult to prepare 2 5 

9. Not available at the time 2 5 

10. Confidential 1 0 

11. No separate records for PW -- 21 

12. Other 5 9 

13. Missing 0 6 

TOTAL 83 159 

Since the first two categories of overlooking and 

lack of understanding overlap, there was a great deal 

of confusion about providing the payroll totals. This 

is partly due to a form difference, shown in Table 4. 

The short form had the all-employee and productionworker payroll near the top, while the long form had 

it at the bottom. Of the RAS respondents who 

received the short form, 82 percent completed the allemployee payroll, as opposed to 72 percent of long 

form recipients. For the production worker payroll, 

61 percent of short-form recipients completed the 

item, compared to 49 percent of long-form 

respondents. Of the 33 respondents who reported 

overlooking both annual and production-worker 

payroll, 24 received the long form. 

Table 4. Percent Completing Payrolls by Form Type 

Long Short 

Form Form 

All-Employee 

Payroll 72% 82% 

Production-Worker 

Payroll 49% 61% 

In summary, the main problem was not reporting 

payroll items. Payroll nonresponse was primarily 

caused by the respondent overlooking the item or not 

understanding they were to provide it. For 

production workers only, records unavailability was 

also a reason for payroll nonresponse. Item 

nonresponse for nonwage cash payments was less of 

a problem. Respondents indicated the primary reason 

for not reporting payments was records 

unavailability. Finally, errors of omission for bonuses 

and awards were uncovered by the RAS. Reasons for 

payment omissions included both records 

unavailability and respondents lack of retrieval of the 

payment. 

Survey Recommendation8 

Comprehension 

1. Instructions should be made clearer. Abbreviated 

instructions should be located next to answer spaces. 

2. The request for annual payroll from all reporters, 

regardless of whether they had payments,.needs to be 

made clear by putting the request m a more 

rominent location. 

• The term "nonwage cash payment" should be 

defined clearly, and put in a prominent location on 

the survey form. 

Mail and other self-administered surveys are 

heavily dependent on a respondent's reading and 

language abilities. However, there is research on 

format, language, and printtype, that identifies factors 

which can improve respondent comprehension. When 

designing forms, Wright (1980) recommends using a 

format similar to a question and answer dialogue, 

finding it is easier for the respondent. Redish (1986) 

suggests reducing the number of steps that a 

respondent has to take in filling out the form, which 

reduces burden and decreases nonresponse. Research 

on both household and establishment surveys 

indicates that questions and instructions are best 

placed together, as respondents immediately begin to 

fill in a form and do not refer to separate instructions 

(DeMaio and Bates 1989; Gower 1989). Along with 

other researchers, Wright (1980) points out that 

!anguage should be easy to understand and free of 

jargon, and print should be legibile. 

These findings are applicable to the problems with 

comprehension found in the pretest and RAS. The 

survey format included instructions and definitions 

located on the back of the form, a standard practice in 

BLS establishment surveys. Given the complexity of 

what is requested, the details on nonwage cash 

610 
payments and payrolls are on the back, including a 

few questions, and category headings which suffice 

for questions. Further down the back page are 

instructions on how to complete the form. Since the 

instructions and question details are separated from 

the front-page answer spaces, comprehension 

problems in the survey have clearly been increased. 

For example, to follow survey instructions or know 

what is included in certain categories, the respondent 

has to xerox the back page of the survey to refer to, 

or flip back and forth as necessary. As another 

example, definitions for bonuses and awards are at 

the bottom of the back page. It is unlikely that a 

respondent will read and remember that information, 

given the location and the smallness of print. 

Increasing the size of the text would have enhanced 

its readability. 

Since respondents do not readily refer to separate 

instructions, abbreviated instructions placed by the 

questions themselves might have solved some of the 

nonresponse problem. RAS respondents receiving the 

long form were less likely to provide the annualpayroll totals, stating that they overlooked it or did 

not understand they were to provide it. Many of these 

respondents had indicated in the yes/no questions that 

they did not have have nonwage cash payments. 

Without any further instructions, it appears that 

respondents simply stopped at that point, rather than 

go on to the payroll section. It is likely that they 

thought since they had no payments, the rest of the 

form was not necessary to complete. The short form, 

requiring the totals at first, appears to have caught the 

respondent's attention, pointing to the importance of 

format in obtaining complete data. 

The survey language appeared to be a problem in 

the pretests, in that respondents did not have general 

knowledge of what was encompassed by a "nonwage 

cash payment." Some RAS respondents reported an 

extremely narrow interpretation of nonwage cash 

payments, one of which was to include only cash 

bills. Another group of respondents did not report 

certain payments when in fact they gave them, saying 

they did not understand the survey. A global 

definition of nonwage cash payments, rather than 

only using specific examples, may have increased 

survey understanding. 

Retrieval 

4. Examples of payments should be included on the 

front of the form. 

During the pretest, a number of respondents when 

cued stated that they simply forgot to report certain 

payment types. This finding was repeated in the RAS. 

Establishment surveys, assumed to be based 

exclusively on "hard" records data, had been 

exempted from concerns with respondent memory. 

However, the pretest and RAS results indicated that 

respondent memory was a factor contributing to 

nonresponse. 

Certain changes to the survey format have the 

potential to aid retrieval. First, the definitions of 

nonwage cash payments are imbedded in text on the 

back page of the form. Placing the examples in the 

lists of nonwage cash payments on the back page 

could have decreased nonresponse, as persons have a 

greater recall when items are placed in a list, rather 

than text (Wright and Reid 1973). Second, memory 

cues, i.e., examples of awards and bonuses could be 

added to the front of the survey form. The examples 

should probably be based on the most frequently 

Ven or largest monetary awards or bonuses, such as 

liday awards or executive bonuses. While there 

may be a concern that only example payments will be 

elicited, site visits have not substantiated that. Asking 

a respondent if they have a particular type of payment 

seems to jar respondents' memories, and payments of 

the same category besides the examples are recalled. 

Response 

5. Experimentation using persuasive-communication 

techniques should be considered to increase 

respondent motivation. 

In selecting a survey response, satisficing can 

occur: choosing a sufficient rather than ideal, 

thought-out response (Krosnick 1990). In a voluntary 

mail survey, motivation rests primarily with the 

respondent. In this paper, item nonresponse has been 

primarily interpreted as due to problems in 

comprehension and retrieval. An alternative 

possibility is that a number of RAS respondents 

reporting "no" to payments and leaving items blank 

were communicating that they did not wish to 

respond to the survey. Whether that is the case or not, 

the number of omissions suggest a potential problem 

with respondent motivation. 

Conclusion 

Given the limitations of individual research 

methods, it is best to utilize multiple methods to 

identify error in mail surveys. Both the pretest and 

the RAS were useful in finding survey weaknesses, 

but they identified different problems. Pretest 

findings indicated that respondents lacked a global 

understanding of survey terms, while the RAS 

identified more errors of omissions. The detailed 

questions from site visit respondents on what 

constituted a nonwage cash payment stand in contrast 

to the errors of omission of many RAS respondents. 

Pretest respondents had probably read the definitions 

quite carefully in preparation for the site visit; they 

may not have been so careful when completing the 

survey under real conditions. Given the pretest 

results, we did not think omission of payments would 

be a problem, but the RAS indicated otherwise. The 

two methods together proved useful. 

Wright (1980) discusses four types of research 

methods that can be used for quality control checks 

on written information. They include: observational 

field studies, survey procedures, in-depth interviews, 

and laboratory comparisons. According to Wright, 

observational studies are important in seeing how 

forms are used, while surveys of users or records 

checks (RAS included here) and in-depth interviews 

(such as the pretest) isolate unclear terms and 

concepts. Observational studies could have been 

utilized in this research, but as with in-depth 

interviews, they must be limited in number due to 

their time and expense. Wright points to laboratory 

comparisons as potentially useful in measuring the 

effect of a change in language or layout. 

Comparisons are frequently used outside of a 

laboratory in survey field experiments, where 

question wording and position are varied to see the 

effect on responses. Although the purpose of the two 

611 
forms in the survey was not to measure format 

effects, they were valuable in identifying 

comprehension problems. 

In conclusion, this study provided an opportunity to 

use a theory from cognitive psychology outside of a 

laboratory setting to improve survey quality. Through 

the use of such methods as pretesting and a response 

analysis survey, problems with comprehension, 

retrieval and judgment were identified, and solutions 

were recommended. Information processing theory 

and retrospective think-aloud techniques were found 

to be useful in guiding the study, and further BLS 

work on establishment mail surveys is now 

underway. 

End Notes 

The author wishes to thank Darrell Philpot for his 

assistance with this research. 

References 

DeMaio, Theresa J. and Nancy A. Bates. (1989). 

Using Cognitive Research to Improve the Design of 

the Decennial Census Form. Presented at the Fifth 

Annual Conference (ARC-V), Arlington, Virginia. 

Gower, Allen R. and Ruth Dibbs. (1989). Cognitive 

Research: Designing A Respondent-Friendly 

Questionnaire for the 1991 Census. Presented at the 

Fifth Annual Conference (ARC-V), Arlington, 

Virginia. 

Krosnick, Jon A. (1990). The Impact of Satisficing 

on Survey Data Quality. Presented at the 1990 

Census Bureau Annual Research Conference, 

Arlington, Virginia. 

Tourangeau, Roger. (1984). Cognitive Sciences and 

Survey Methods. Pp. 73-100 in Thomas B. Jabine, 

Miron L. S traf, Judith M. Tanur and Roger 

Tourangeau (eds.), Cognitive Aspects of Survey 

.Methodology: Buildi.ng a Bridge Between 

Disciolines. Washington, D.C.: National Academy 

Press. 

Wright, Patricia. (1978) Feeding the Information 

Eaters: Suggestions for Integrating Pure and Applied 

Research on Language Comprehension. Instructional 

Science 7:249-312. 

Wright, Patricia. (1980). Strategy and Tactics in the 

Design of Forms. Visible Language 14:151-193. 

Wright, Patricia and Fraser Reid. (1973). Written 

Information: Some Alternatives to Prose for 

Expressing the Outcomes of Complex Contingencies. 

Journal of Aoolied Psychology 57:160-166. 

Wright, Patricia and P. Barnard. (1975). 'Just fill in 

this form'-a review for designers. Applied 

Ergonomics 6:213-220. 

612