THE EFFECTS OF ADVANCE LETTERS AND REMINDER/THANK YOU LETTERS ON

REDUCING NONRESPONSE IN AN ESTABLISHMENT SURVEY:

AN EXPERIMENTAL STUDY

Young I. Chun and Kenneth W. Robertson, U.S. Bureau of Labor Statistics

Young I. Chun, 2 Massachusetts Avenue, N.E. Washington, DC 20212

Key Words: Nonresponse errors, organizational

behavior, informants

1. INTRODUCTION

An increased nonresponse rate is a major problem not

only in household surveys in the U.S. and abroad but

also in establishment surveys (Christianson and

Tortora, 1995) and economic censuses (Ambler and

Mesenbourg, 1992). An increasing nonresponse rate

in establishment surveys has been of primary concern

to government bureaus collecting data from businesses

or firms, as a high response rate is considered an

important component of data quality, and the effort to

reduce nonresponse mounts the survey cost. It is

important for survey researchers to reduce as much

nonresponse as possible, because nonresponse affects

the reliability of statistical estimates by introducing

bias.

This paper reports the findings from a recent field

experiment conducted by the Bureau of Labor Statistics

(BLS) that was designed to evaluate a set of

nonresponse-reducing techniques for an establishment

mail survey. The nonresponse reducers investigated

are advance letters and reminder/thank you letters.

Among various design techniques (e.g.,

personalization, stamped return envelope, first

outgoing postage, sponsorship, financial incentives,

questionnaire color, and etc.), the combined use of

advance and reminder/thank you letters have been

found most effective in reducing nonresponse rates for

voluntary household surveys and censuses (Dillman et

al., 1993), but have not been tested yet for

establishment surveys.

2. RESEARCH DOMAINS AND HYPOTHESES

Reducing and measuring nonresponse errors requires

an understanding of the sources of nonresponse errors.

Sources of nonresponse errors are the societal

environment, the situational context, and the variables

involved in the brief survey interaction such as the

respondent, the interviewer, the questionnaire, and the

mode of data collection. The theoretical concepts

relevant to household survey nonresponse (Groves,

Cialdini, and Couper, 1992) may be applicable to

understanding the nonresponse process in

establishment surveys. At the societal level, those

informants reporting for small firms tend to view the

economic survey as data gathering tools of the large

firms and refuse to participate. Borrowing from

Cialdini’s (1984) concept of authority, a mandatory

establishment survey is likely to reduce nonresponse as

compared to a voluntary establishment survey because

of the social psychological pressure of obeying legal

authority. Petty and Cacioppo’s (1986) theory of

attitude change offers an explanation that an

establishment survey questionnaire, which looks very

informant friendly (i.e., a peripheral cue), may reduce

the informant’s opt-out.

The reality that the informant’s behavior takes place in

organizational context of the establishment survey,

however, demands theoretical approaches which take

into account the organizational behavior of the

informant. Research in organizational behavior

(Taylor, 1911; Likert, 1967; March and Simon, 1957;

Katz and Kahn, 1966; and Weick, 1969) and concepts

specific to surveys of facts (Martin, 1993) bear on

theorizing the informant’s nonresponse process in

establishment surveys. The informant behaves under

authority within the organizational system and his

reporting task primarily depends on the use of records.

Organizational theory of communication is a good

starting point to clarify our thinking about noncomplying behavior in organizations.

We allude to four schools of organizational

communication which Porter and Roberts (1983)

classify: classical-structuralists, human relationists,

decision theorists, and process or systems viewers. The

classical structuralists (Taylor, 1911) view

organizations as closed, hierarchical, and static

systems while stressing authority, control, coordination

and other internal structural relationships. In contrast,

the human relations theorists (Likert, 1967) focus on

informal interpersonal communication systems and

group interactions inside organizations. The behavioral

decision theorists (March and Simon, 1957) describe

organizations as decision-making structures, and view

that individuals in organizations make rational

decisions while taking into consideration of inherent

constraints in the organization. They indicate that

decision making is hindered by the information
available being incomplete and thus “uncertainty

absorption” takes place as information fails to fit the

extant classification schemes. Their discussion offers

an explanation of how information distortion and

gatekeeping take place and influences nonresponding

behavior in organizations. Process or systems theorists

(Weick, 1969) move us to multivariate views of

organizational communications where the larger

environment of organizations is the important

determinant of behavior. Each school of

organizational communication theory has different

degrees of merit in our search for ways to understand

nonresponding behavior of the informant in

organizations. From the classical structuralists, we

learn that the informant’s reporting behavior is to some

extent controlled or coordinated by authority. The

decision to participate in a survey is in part a function

of the extent to which the informant responds to

authority within the organization. The human

relationists offers an explanation of how the request for

surveys is routed within organizations, given the active

nature of all potential informants. The decision

theorists helps us understand the relation of

information to the decision-making process, how this

relates to nonresponse, and how such information is

distributed, altered, and absorbed. Finally, the systems

theorists direct our attention to forces outside the

organization which influence internal nonresponding

communication behavior. In sum, the merits of the

decision theorists would be most obvious helping us

understand the informant’s nonresponse behavior

within organizations. The systems theorists helps to

identify factors external to organizations. The

structuralists moves us to pay attention to authority

based hierarchical structural components through

which nonresponse can take place. The inherent

shortcomings of organizational communication

theories is that they do not pay attention to understand

the nature of tasks given in establishment surveys,

namely fact reporting as opposed to attitude response.

Three primary response functions that Martin (1993)

describes as important in measuring facts in a survey

are directly relevant to understanding the nonresponse

behavior of the informant in fact-gathering

establishment surveys. This is because the response

burden perceived in response functions would also

drive the informant not to comply with the survey

request. Elicitation, classification, and enumeration

(or quantification) are three major response operations.

Elicitation is to call forth respondents’ reports that

involve comprehension, recall, judgment, and

communication. Classification is to categorize facts

according to the survey definition. Enumeration or

quantification is accomplished by counting or

estimating the events or persons. All these three

functions are performed in establishment surveys.

Informants who are asked to elicit, for example,

reports of injuries of employees at work places in a

calendar year should go through all cognitive

information process to reach an answer. Informants

who are asked to provide the number of nonsupervisory

employees should classify data in their information

system according to the survey definition to reach an

answer, and enumerate the total. When any of these

response functions are perceived to be difficult,

informants are more likely to resist engaging the

response process and thus instead opt out.

Nonresponse reducers we see in literature have not

necessarily derived from the theoretical considerations

we have described above. Among other procedures,

requiring mandatory response, making multiple

contacts, and identifying the informant have been

suggested or tested as most effective. A change to

mandatory reporting was the most common reason for

unchanged nonresponse rates over the 10-year period

in the international survey of business surveys

(Christianson and Tortora, 1995). A similar finding

was documented in the 1987 U.S. economic census

where respondents attributed their filing mostly to the

survey being required by law (Ambler and

Mesenbourg, 1992). Making multiple contacts

including those with telephone follow-up, and sending

the survey to a named individual have also been shown

to significantly increase the response rate (Paxon,

Dillman and Tarnai, 1995). To date, establishment

survey research has not provided us with insight about

the relative effects of advance letters and

reminder/thank you letters on nonresponse reduction.

In contrast to household surveys, where it is usually

clear who may serve as the respondent, establishment

surveys are problematic because it is not obvious (1)

who within the establishment is qualified to respond,

and (2) how well the informant is able to use the

information system that is usually necessary to provide

data about employees. In such regards, advance letters

and reminder/thank you letters are used to help the

firm identify a correct informant and prepare the

informant for the upcoming records-based

establishment survey, and to remind the informant

about the survey or to thank him/her for participating.

During this test, the selected establishments received

an advance letter that describes several aspects of the

survey. The importance of the survey was stressed, the

uses of the data were explained, and the fact that the

establishment was randomly selected to represent many
other firms in the industry was indicated. The letter

indicates that the establishment should expect to

receive the Hours at Work Report form requesting the

total number of hours employees are paid including

hours of paid leave, and the total number of hours they

were actually on the job. The letter notes a distinction

between production/nonsupervisory and

nonproduction/supervisory workers. Confidentiality of

data is promised in that only ratios of hours at work to

hours paid are estimated for each of the major

industrial groups. The Reminder/thank you letter is

written to appreciate those who returned the form or

reminding those who did not to complete it. Just like

the advance letter, the reminder letter described the

importance of the survey, and the importance of the

respondent’s answer due to representation of the

reporting firm for many others. Confidentiality of data

is again promised as well.

Several hypotheses can be tested. We suspect that the

effect of the advance and reminder letters is short term,

(perhaps a few days when received, but not more than

a week) such that informants induced by these

additional contacts are likely to behave quickly. The

longer they wait after receiving these letters the less

effect the letters will have on their willingness to

respond. The most important hypothesis to test,

however, in this experiment is that those firms

receiving both advance and reminder/thank you letters

should generate the highest response rate. We also

expect that the nonresponse reduction by the advance

and reminder letter is significant when compared to the

control group.

It is also hypothesized that advance letters in

establishment surveys are more effective in reducing

nonresponse than reminder/thank you letters. The

reason is that advance letters motivate the recipients to

expect the upcoming survey of business facts and be

prepared with the relevant data before receiving the

first survey questionnaire. On the other hand,

reminder/thank you letters remind nonrespondents of

completing the survey questionnaire after they received

the initial survey packet. Early warnings are generally

better than late reminders. Sending advance letters

also help the survey sponsor receive Post Office

Returns (PORs) and other returns earlier. This allows

the survey processors to correct the addresses and to

allocate resources in a more informative manner.

At the industry level, we expect that the effect of

advance and reminder letter is larger among

manufacturing industries than among

nonmanufacturing industries due to the perceived

response burden related to the availability of hard data.

Manufacturing industries are requested to supply data

about production employees whose records are usually

available in the information system; some

nonmanufacturing industries find it difficult to locate

the required information as hours data may not be

available for commissioned workers or piece workers.

Nonmanufacturing industries are also less likely than

manufacturing industries to have individual or

summary data, and are more likely to have

employment mixes of part time, full time and

temporary workers. This perceived response burden

among nonmanufacturing industries may therefore

cause informants to refuse to respond the survey.

3. RESEARCH DESIGN

Experiment: The experiment (n = 400) was embedded

in the Hours at Work Survey (HWS) (n = 6000) which

primarily collected employees’ hours paid and hours at

work for 1993. Three treatment groups were tested

against a control group which received neither advance

letters nor reminder/thank you letters: (1) a survey with

advance letters and no reminder letters, (2) a survey

with reminder/thank you letters and no advance letters,

and (3) a survey with advance letters and

reminder/thank you letters. Treatment and control

groups all received an initial and two replacement

survey packets.

The HWS is a national annual survey of 6,000

establishments conducted by the BLS since 1981. The

HWS primarily collects data on both the total number

of hours nonsupervisory or production employees were

on the job (i.e., hours paid minus paid leave) and the

total number of hours for which these employees were

paid. The information obtained is used as one factor in

the estimation of national productivity by industry.

The HWS, which involves a self-administered mail

survey with follow-up mailings, facsimile and

computer-assisted telephone interview, provided the

researchers with an opportunity to analyze the effect of

sending advance letters and reminder/thank you letters

in establishment surveys. The HWS is designed to

construct ratios of hours at work to hours paid for each

of the major industrial sectors of the U.S. economy, on

a yearly basis. Ratios are also produced by quarter, and

for four employment-size classes. These ratios are

used to adjust the presently published hours paid

measures. The resulting hours at work measures are

incorporated into the labor productivity series as the

revised labor input. The survey, conducted by mail and

followed up by telephone, request that each selected
establishment report the total hours at work and the

total hours paid for all production and nonsupervisory

workers for each quarter of the previous year. Data

collected will provide estimates for each of the major

sectors (1-digit SIC) of the nonagricultural economy

plus additional industry subdivisions (2-digit SIC)

within the manufacturing sector.

Sample: The sample for the experiment (n = 400) was

stratified by two major industrial divisions

(manufacturing and nonmanufacturing) by three

employment size classes (< 50, < 250, and 250 +).

Within these strata additional sorting was done by

industry and size to provide an implicit stratification by

their industry and size classifications. Within the

major stratifications (cell), units were systematically

selected after a random start. Within each cell, units

were systematically assigned to four treatment and

control groups. The Hours at Work Survey is based on

a probability sample taken from most of the 5.2 million

nonagricultural establishments which report

employment and earnings to State unemployment

insurance programs. Approximately 6000

establishments are randomly selected after

stratification by industry and employment sizes, using

standard probability sampling methods. The sample is

selected primarily from the most recent Universe

Database (UDB). The UDB is the BLS sampling frame

for all establishment surveys. It is comprised of

administrative files produced by State agencies for the

Unemployment Insurance (UI) programs. Information

obtained from the UDB Frame usually include an

establishment employment figure, a UI account

number, a four-digit SIC code, a county code, a name

and address for the establishment, a plant location, and

an area code and telephone number, if available.

Analysis plan: Following Dillman and his colleague’s

(1993) approach, we report findings from the six pairwise comparisons among treatment and control groups,

and logistic regression analyses. The pair-wise

comparisons of each of the treatments with the control

group and with each other helps to determine the level

of nonresponse reduction. The response rate is

computed as follows:

 R = Usable Units/Viable Units, where

• Usable units are defined as those which have

provided data which passed all edit checks,

• Viable units are those eligible units such as usable

units, nonrespondent, refusal, unsuccessful

mailout, data unusable, and eligibility not

determined when employment >=50. Excluded in

the viable units are those units such as out of

scope (e.g., a change in industry to a nonsampled

industry), out of business, duplicate units, and

eligibility not determined when employment < 50.

The hypothesis that the multiple contacts including

advance letters and reminder/thank you letters would

significantly reduce nonresponse in establishment

surveys was tested after computing one-tailed t-test

values.

Results from the logistic regression analysis indicate

whether or not each component makes significant

impact on the response rate. Its analytical limitation is

that the parameter estimates themselves cannot be

easily interpreted in terms of the size of the effect on

the response rates. The first model includes all of the

main effects (i.e., industry stratum, size stratum,

advance, and reminder/thank you) and interaction

terms. After observing which terms would turn out to

be significant, the reduced model with main effects and

a lesser number of interaction terms will be tested.

When making pair-wise comparisons and evaluating

parameter estimates of the logistic regression analyses,

we test the results at the alpha = 0.05 level. A

significant t-test value computed for pair-wise

comparisons indicates that the difference is due to a

real effect and not just sampling variation.

4. FINDINGS AND DISCUSSION

The major results from this experiment are presented

through descriptive and inferential statistical methods.

The overall response rates for each of the treatments at

the national, industry, and size levels are presented in

Table 1.

Table 1. National & Stratum Level Response Estimates*

(A=Advance, R=Reminder, C=Control)

Contact n Response Rate (%)

Type National Industry Size

M NM S1 S2 S3

A&R 94 57.4 56.3 58.7 53.3 66.7 51.6

A 98 52.0 64.6 40.0 58.8 59.4 37.5

R 98 46.9 54.2 40.0 54.5 52.9 32.3

C 98 40.8 50.0 31.3 60.6 32.4 29.0

*Note: Industry level response rates are categorized by M (manufacturing) and

NM (nonmanufacturing). Size level response rates are categorized by ‘S1’ where

employment is less than 50, ‘S2’ where employment is greater than or equal to 50

and less than 250, and ‘S3’ where employment is greater than or equal to 250.

As expected, the response rate from the advance and

reminder group obtained the highest response rate of

57.4 percentage points, followed by the advance group

(52.0%), the reminder group (46.9%), and the control
group (40.8%). (Note that these are not the final

response rates from the survey. Due to the

confounding effect of CATI follow-up, we ended our

test at this point. However, data collection continued

via CATI.) An intriguing finding is that the

nonmanufacturing group that received both advance

and reminder letters had a higher response rate than

the manufacturing group while all other

nonmanufacturing groups gained lower response rates

than the manufacturing groups. Within the

manufacturing industry, the response rate of the

advance group surpassed even that of the advance and

reminder group.

Within each employment size group, no consistent

pattern appeared. Within Size 1, the advance and

reminder group had the lowest response rate. With

Size 2 and 3, the advance and reminder group gained

the highest response rate. Within each treatment

group, no consistent pattern emerged either. Within

the advance and reminder group and the advance

group, the response rate was highest in Size 2. Within

the reminder and control group, the response rate was

highest in Size 1.

Table 2 below presents six comparisons corresponding

to all possible comparisons among three treatments and

one control group.

Table 2. Pair-wise comparison of response Estimates*

(AR=Advance+Reminder, A=Advance, R=Reminder, C=Control)

 Pair Response Rate Difference (%)

National Industry Size

M NM S1 S2 S3

A-C 11.2 14.6 8.8 -1.8 27.0* 8.5

R-C 6.1 4.2 8.8 -6.1 20.6* 3.2

AR-C 16.6* 6.3 27.4* -7.3 34.3* 22.6

*

A-R 5.1 10.4 0.0 4.3 6.4 5.2

AR-R 10.5 2.1 18.7* -1.2 13.7 19.4

AR-A 5.4 -8.3 18.7* -5.5 7.3 14.1

* See the note in Table 1. A single asterisk indicates a significant difference at

the alpha = 0.05 level for a one-tailed t-test.

The most important finding in this table is that the

group that received both advance and reminder letters

gained the statistically significant response rate

compared to the control group, and this difference of

16.6 percentage points mostly came from the gain in

the nonmanufacturing industries. Differences from all

other comparisons in the national level did not appear

significant, although most of them were in the positive

direction. Overall, nonresponse reduction was 11.2

percent due to advance letters; 6.1 percent due to

reminder letters. (Recognize that the effect of advance

letters was 10.5 percent when added after the reminder

letter, and the effect of the reminder letter was 5.4

percent when added after the advance letter.) The

pattern disappeared when viewed at the industry level.

Within the manufacturing industry, none of the

differences were significant, and the highest difference

came from advance letters compared to the control

group. Within the nonmanufacturing industry, we still

find the advance and reminder letter combination made

the significant nonresponse reduction effect when

compared to the control group, and the difference from

all other comparisons did not appear to be significant.

These findings suggest that the effect of different

treatment vary depending upon which industry units

receive the treatment, which strongly indicates an

interaction effect between industry types and response

rates. At the size level, it was found in Size 2 and

Size 3 that nonresponse reduction was significant and

highest when the advance and reminder groups were

compared against the control group. Size 1 did not

conform to the pattern. Within Size 2, it is worth

noting that the advance letter and the reminder letter

made the significant nonresponse reduction against the

control group. Size specific variations of treatment

effects suggest an interaction effect between size and

response rate. In sum, it appeared that the effect of the

advance and reminder letter is most obvious among

nonmanufacturing industries (as compared to

manufacturing industries) and among firms hiring 50

or more employees.

Using logistic regression analysis, we observed

parameter estimates in the full model that includes all

main effects industry and size strata, and all possible

interaction terms. The significant terms were intercept

and size 1 (firms with less than 50 employees), and the

interaction term of industry and size 1. We

subsequently tested a reduced model that includes all

main effects, and an interaction term only. The results

are presented in Table 3.

Table 3. Logistic Regression Modeling* (n=388)

Variable Parameter Estimate

Intercept -1.5

Industry 1.1

Size 1 1.6

Size 2 0.7

Advance 0.5

Reminder 0.2*

Industry*Size 1 -1.5

*Note: All parameter estimates are significant at the alpha = 0.05

level unless indicated by a single asterisk.

Findings from the reduced model indicate that

significant (at alpha = .05 level) improvements were
realized from the advance letter, industry, size 1, and

size 2 , but not from the reminder letter. The

interaction term of industry and size 1 was statistically

significant. Substantively speaking, nonmanufacturing

industries with less than 50 employees appeared to be

more cooperative across all treatment and control

groups than manufacturing industries. This finding

puzzled the researchers because manufacturing

industries have been conventionally more cooperative

than nonmanufacturing industries in part because

manufacturing industries appeared to have more record

based information systems or individual/summary

reports, thus allowing the informant to easily comply

with the request. The reliability of this intriguing

finding should be subjected to further testing with a

larger sample as each cell (industry-size for treatments

and control groups) in the current test includes less

than 20 units and this small sample might have

confounded the results.

5. CONCLUSIONS AND FURTHER RESEARCH

The experiment evaluated the separate and combined

effects of advance letters, and reminder letters on

response rate. A statistically significant result was

found for the combined use of advance letters and

reminder/thank you letters, increasing the response

rate by about 16.6 (plus or minus 14.5) percentage

points when compared to the control group that

received neither contact. The findings here in the

establishment survey complement those based on the

household survey. The results helped us redesign the

data collection process for the following year's HWS.

We conducted a full split sample test (n = 6000) for the

1994 survey, comparing the effect of the combined use

of advance and reminder letters against the control

group. The preliminary results confirm the major

findings from the current small scale test, indicating

the advance and reminder combination significantly

reduces nonresponse in the establishment survey. The

more complete reports at the industry and size level,

the implications of nonresponse reduction in its

interaction with measurement errors, and cost

modeling will all be available from the full split sample

test.

BIBLIOGRAPHY

Ambler, C.A., and Mesenbourg, T.L. (1992) “Response

Improvement Initiatives in the 1992 Economic

Censuses,” Paper presented in the International

Conference on Establishment Surveys, Buffalo, NY.

Christianson, A. and Tortora, R. (1995), “Issues in

Surveying Businesses: an International Survey” in

Business Survey Methods, (eds.) B.G. Cox, D.A.

Binder, B.N. Chinnappa, A. Christianson, M.J.

Colledge, and P.S. Kotts, New York: John Wiley.

Cialdini, R.B. (1984) Influence: The New Psychology

of Modern Persuasion, New York, Quill.

Dillman, D.A., Clark, J.R., and Sinclair, M.D. (1993),

“How Prenotice Letters, Stamped Return Envelopes

and Reminder Postcards Affect Mailback Response

Rates for Census Questionnaires,” Paper presented

at the U.S. Bureau of Census Annual Research

Conference.

Groves, R.M. Cialdini, R.B., and Couper, M.P.

(1992), “Understanding the Decision to Participate

in a Survey” in Public Opinion Quarterly, 56.

Katz, D., and Kahn, R. (1966) The Social Psychology

of Organizations, New York: Wiley.

Likert, R. (1967) The Human Organization, New York:

McGraw-Hill

March, J.G. and Simon, H.A. (1957) Organizations

New York: Wiley

Martin, E. (1993) “Response Errors in Survey

Measurements of Facts” Paper presented in the

ISI’s 49th Session

Paxon, M.C., Dillman, D.A. and Tarnai, J. (1995)

“Improving Response to Business Mail Surveys” in

Business Survey Methods, (eds.) B.G. Cox, D.A.

Binder, B.N. Chinnappa, A. Christianson, M.J.

Colledge, and P.S. Kotts, New York: John Wiley

Petty, R.E., and Cacioppo, J.T. (1986) Attitudes and

Persuasion: Classic and Contemporary

Approaches, Dubuque, IA, W.C. Brown.

Porter, L.W., and Roberts, K.H. (1983)

“Communication in Organizations” in Handbook

of Industrial and Organizational Psychology, (eds)

Marvin Dunnette John Wiley and Sons.

Taylor, F.W. (1911) The Principles of Scientific

Management. New York: Harper and Brothers.

Weick, K. (1969) The Social Psychology of

Organizaing, Reading, Mass.: Addison-Wesley.