RESEARCH INVESTIGATIONS IN GAINING PARTICIPATION FROM SAMPLE FIRMS IN THE 

CURRENT EMPLOYMENT STATISTICS PROGRAM October 1997

Robert M. Groves 1, David Cantor2

, Mick Couper1

, Kerry Levin2

, Kate McGonagle3

, Eleanor Singer3

, 

and John VanHoewyk3 

1218 Lefrak Hall, College Park, MD 20742 

Key Words: Establishment surveys, response rates 

I. Introduction

This paper reviews a set of randomized experiments

designed to identify causes of nonresponse in

establishment surveys. It draws heavily on the experience

of the authors with the U.S. Current Employment

Statistics (CES) program, a monthly survey of about

400,000 employers. The purpose of the survey is to

provide monthly estimates of jobs created in the economy

and of changes in payroll levels. The experience of the

authors is limited to experimental research to increase the

cooperation rate of sample employers. The paper is

organized by stages of the survey process. For some of

these stages the discussion begins with some theoretical

constructs that appear to be relevant to the stage.

2. Frame Development and Updating

Establishment survey sampling frames (especially those

conducted by government agencies) are often constructed

from an administrative record system, designed not for

survey purposes but for program purposes. Such

administrative record systems might contain estimates of

numbers of employees, an industry classification, reports

of income or production counts, as well as address and

telephone information. Whether the content of the

sampling frame record is relevant to the survey depends

on the relationship between the administrative record

system producing the frame and the survey in question.

2.1 Births in Establishment Frames 

A critical step in frame evaluation is the 

determination of new units coming into the population 

("births") and units that have ceased existence ("deaths"). 

The first experiment describes reviews some insights into 

this process. 

Experiment 1. CES Enrollment for Business Births. 

Data collection for the CES is extremely critical for 

decision makers in a variety of areas, and the data 

collected should be an accurate reflection of the economic 

situation in the United States. To ensure that all business 

establishments are represented, Westat conducted a study 

to specifically examine the most effective procedures for 

soliciting new businesses (i.e., business births) into the 

CES. Business births are establishments who recently 

applied for a new unemployment insurance (UI) account 

number. These businesses generate between 2 and 2.5 

million jobs a year and represent 750,000 establishments. 

Including these businesses in the CES is vital since they 

represent a large share of the fluctuation in employment 

each year. 

Sample design. The sample for the study included 

establishments selected by BLS from six states. 

Establishments in these states recently applied for a UI 

number. The majority of establishments were businesses 

that began operations within the last year and were very 

small (e.g., 5 or fewer employees). 

Treatments. Once the sample was received, 

refinement calls were made to identify the most 

knowledgeable respondent and update information such 

as address, phone number, fax number, etc. 

"Knowledgeable" was defined as the individual within 

the establishment who was responsible for payroll 

activities. Next, the sample was assigned to one of two 

experimental conditions. The experiment varied the 

amount of information the respondent received in the 

mail prior to the interview. In condition 1, respondents 

received an advance letter which described the purposes 

of the study and urged their participation. In condition 2, 

respondents received the advance letter as well as 

materials describing the CES and the 790 form used to 

collect the CES data. 

After refinement, the enrollment process began. 

During enrollment, interviewers read a verbatim script 

that introduced the study and briefly described the 

purpose. During this call, interviewers attempted to 

collect CES data. 

All respondents who provided the first month of CES 

data were conducted five months thereafter to collect the 

same CES information. Prior to the second CES data 

1 University of Michigan and Joint Program in Survey Methodology

2 Westat, Inc.

3 University of Michigan; names in alphabetical order 

289 
Results. The refinement call yielded a response rate 

of 94%. Ineligible cases included establishments with no 

employees, establishments with no location in the 

sampled state, and establishments that went out of 

business. A total of 372 businesses were eligible for CES 

enrollment. 

A total of 282 establishments were successfully 

enrolled into the CES. The remaining cases were 

determined ineligible, unavailable, or refused to 

participate. Enrollment response rates did not differ 

between conditions 1 and 2. Thus, it appears that sending 

additional materials did not affect enrollment rates. 

In over 50% of the cases, the interviews were 

completed with the owners of the establishment. 80% of 

the respondents were located at the sampled address 

identified during the refinement call. 

About 6% of the sample included establishments who 

applied for a UI number but currently had no employees. 

In fact, businesses in this situation fell into several 

categories-- seasonal; just opened haven't hired yet; and 

probably would never hire employees in the future. 

Of the refusals, workload was primary reason 

reported. Owners of the establishments could not assign 

the task to another division, location, etc. 

2.1 The Sample Refinement Step 

A common first step in an establishment survey is a 

frame cleaning and enrichment step, sometimes called 

"sample refinement." This often involves a first contact 

with the sample establishment to determine whether it is 

still in business, has merged with another firm, or has 

changed its identity in some other way important to the 

survey (e.g., industry classification). 

Experiment 2. Using Initial Contacts to Gather 

Information on the Firm. The aim of this study was to 

assess whether or not the likelihood of successfully 

obtaining the name of a contact person during sample 

refinement varied as a function of the number of 

questions asked of the respondent. These questions were 

designed to be of value to the interviewer at the time of 

the first enrollment call in anticipating concerns of the 

employer about the survey request. A sample of 320 

employers of less than 100 employees from 4 states and 

four SIC codes in retail and services sectors was used. 

Statistical analyses revealed no differences in the 

likelihood of successfully obtaining the name of a contact 

person as a function of using a "short" (98% refinement 

rate) versus "long" questionnaire (95% refinement rate). 

On average, the results showed no effect of questionnaire 

length on likelihood of obtaining information, regardless 

of firm size or SIC category. Based on the results of this 

experiment, the decision was made to continue seeking 

information about the firm beyond the name of the 

3. Advance Communication with the Sample Unit 

Since some business establishments have hundreds or 

thousands of employees, the chances are small that 

advance letters addressed merely to the firm would be 

read by an appropriate staff member. Thus, in some 

establishment surveys, during the sample refinement step, 

the name and address of either the preferred informant or 

the person with authority to commit the establishment to 

the survey participation is obtained. This person is then 

mailed a letter addressed to him/her personally, which 

sometimes has content tailored to their role in the 

company. 

Experiment 3. Alternative Mailings to the Contact 

Person. The purpose of this experiment was to determine 

whether differences in the appearance, volume, or 

frequency of mailings describing the CES program 

affected participation rates among newly solicited 

establishments. Four different advance mailings were 

randomly assigned to subsets of a sample of employers. 

The mailings varied on the amount of descriptive content 

about BLS programs, of information about the CES, and 

of information about the data items requested. The 

mailings also varied on their use of modem graphics and 

presentation style. It was hypothesized that more 

information about BLS and more modern-appearing 

advance mailings would improve cooperation rates. A 

sample of 1379 employers of less than 100 employees 

from 4 states and four SIC codes in retail and services 

sectors was used. 

Results. Approximately 48% of all cases were 

successfully enrolled, and enrollment rates were 

comparable across the treatment conditions. In general, 

enrollment rates were consistent both between and within 

firm size categories across the treatment conditions; 

however, there was some indication that the mailing 

strategy that had the greatest amount of information about 

BLS and the survey achieved higher cooperation rates as 

the firm sizes increased. 

4. Initial Contact to Deliver the Survey Request 

This section first reviews a set of organization attributes 

that may affect a firm' s likelihood of cooperation with a 

survey request, then describes relevant experimental 

studies. 

GOAL STRUCTURE. The goals of many establishments 

are more narrowly defined than those of most households. 

Most business establishments have goals of profit for the 

owners and shareholders as well as employment stability 

for their staff. The restricted number of goals of 

establishments could affect their initial perspectives on 

requests from the external environment. Their views of 

the external environment are heavily shaped by their goal 

290 
employees, their external environments are hostile ones. 

Most businesses have competitors, who seek to increase 

their own market share at their expense. In doing so, they 

seek intelligence about the operations of their 

competitors, to assess their plans for new products or 

services and to judge their strength for innovation. 

To some extent all requests from the outside might be 

seen through the filter of the goal of profit and 

employment. Hence, the most salient aspects of the 

request often involve the amount of time the survey will 

take away from profit activities or the threat that releasing 

the information will have on actions of competitions or 

government regulators (an indirect threat to profit). 

The fact that business organizations appear to be 

subject to more "form-filling" requirements than 

households intensifies the focusing effect of restricted 

goals of organizations. That is, when there is a request 

for information from a survey organization, it appears 

likely that firms are associating that request with past 

form-filling requests. Most of these requests in the U.S. 

are made by govemment agencies, either for tax 

collection, personnel policy requirements, environmental 

regulation compliance, job safety regulation compliance, 

building code regulations, wholesale and retail licensing, 

or health requirements. Sometimes, establishment survey 

interviewers appear to develop techniques that distinguish 

their request from those other requests. If it appears that 

the respondent is associating the request with a mandatory 

form, the interviewers do not make distinctions, because 

the analogue is helpful to the likelihood of cooperation. 

AUTHORITY HIERARCHIES. Most business 

establishments with more than a few employees contain 

a hierarchy that gives to some staff members more 

authority to make decisions than to others. These 

distinctions are made real by systems of job titles, formal 

rules, written job descriptions, and well-defined 

supervisory relationships. Of great relevance to survey 

requests is that not all staff have the authority to interact 

freely with those outside the organization, especially 

regarding the provision of information about the firm. 

Permission to obtain information about the firm is 

generally controlled at higher levels of the authority 

chain. 

ROLE DIFFERENTIATION. Consistent with authority 

hierarchies, staff differ in their role definitions. As 

organizations grow, role specialization seems to increase, 

with each member being assigned the responsibility for a 

smaller number of areas. (Downsizing of organizations 

and attempts to flatten the structure of organizations are 

acting against this tendency.) This is relevant to survey 

requests because it is common that only some roles in the 

organization have access to information being sought by 

a survey. With role differentiation comes information 

personnel counts; others, to production figures. Some 

roles know about financial affairs; others, about 

environmental compliance experiences. Further some 

roles have direct access to the public outside the firm; 

others, do not. 

There are three role related duties of particular 

interest to survey takers: 1) receptionist to the public, 2) 

decision-makers for release of information about the firm, 

and 3) roles compiling and updating the information 

being sought by the survey. The receptionist role is one 

that exists in organizations that judge it useful to screen 

and disperse outside contacts in a manner to minimize 

disruptions of staff involve in the production processes of 

the firm. Firms appear to differ on what rights and 

responsibilities are given to the receptionist role. 

Sometimes they are trained only in telephone or greeting 

skills and instructed in what queries are sent to which 

other individuals in the firm. Other times they are 

experienced employees with deep knowledge of the firm, 

and thus possess information about authority structures 

and information resources within the firm. Some 

organizations have units specializing in dealing the with 

requests form the public (e.g., public relations divisions 

or legal affairs divisions). Interviewers find themselves 

routed to these units when presenting survey requests to 

the firm. When information needed by the survey is 

isolated into a single unit, sometimes the interviewer is 

referred to that unit. In many organizations, however, 

that unit does not have the authority to provide the 

information to outsiders. 

The existence of role differentiation concurrent with 

authority chains produces a dilemma for interviewers in 

some cases. The role with authority to permit the 

interviewer to obtain information may have very little 

knowledge about whether the information exists within 

the firm or whether it exists in a form desired by the 

survey. This sometimes means that the target person 

makes judgements about the difficulty of complying with 

the survey request that are wildly incorrect. A top 

manager, unaware that the reporting requirement for 

unemployment insurance is similar to that of the survey 

requirement, may judge that to assemble the information 

required by the survey would require many more staff 

hours than are actually required. In this case, the 

ignorance by the top manager leads to reluctance to 

cooperate. The opposite can also occur, sometimes to the 

advantage of the survey requestor. 

Interviewers in establishment surveys appear 

sensitive to disceming the extents of the role of the 

gatekeepers. Indeed, the sample refinement step permits 

the interviewer to learn what information is possessed by 

the gatekeeper, what authority they have to provide 

information, and what knowledge they have of the 

291 
establishment interviewers tend to be flexible about what 

information is sought from the gatekeeper. The stakes of 

this strategy are reduced by the fact that the purpose of 

the sample refinement call is merely obtaining 

information about the contact person for later mailing. In 

most cases refusal to provide this information might be 

seen as a restriction on the freedom of the contact person 

to make decisions in the interests of the firm. 

Experiment 4. Level of Entry into the Firm. The aim 

of this study was to assess whether the likelihood of 

successfully obtaining the name of a contact person 

during sample refinement and collecting data during 

enrollment varied as a function of the organizational level 

at which the request was made. This study was motivated 

by both qualitative observations of CES respondents and 

theories about organizational structure and decisionmaking. 

In the sample refinement stage, interviewers were 

asked to request the name, address, telephone number, 

and title of the "Top Manager" in Treatment 1, the "Head 

of Payroll" in Treatment 2, and the "Unemployment 

Insurance (UI) Reporter" in Treatment 3. A sample of 

646 employers of less than 100 employees from 4 states 

and four SIC codes in retail and services sectors was 

used. 

Results. Approximately 59% of all refined cases 

agreed to participate during the enrollment call. There 

was a marginally statistically significant effect of entry 

level such that rates of successful enrollment were higher 

in the Head of Payroll condition (61.5%, s.e.=4.2) than 

the Top Manager condition (54.1%, s.e.=4.7). Moreover, 

there was some suggestion that enrollment rates for the 

Top Manager and Head of Payroll conditions are lower in 

larger firms than smaller firms, and in the UI Reporter 

condition, higher in larger firms than in smaller firms. It 

is possible that this finding can be explained by 

differences between large and small firms in the 

organization of labor category roles. 

Experiment 5. Manipulating the Saliency of 

Mandatory Participation in the Survey. The CES data 

request is mandatory in some states and nonmandatory in 

others. The BLS was interested in discovering whether 

the mandatory/nonmandatory nature of the data request 

affected cooperation rates. Thus, the specific aim of this 

study was to assess the impact of varying the emphasis on 

the mandatory nature of CES on the likelihood of 

obtaining agreement to participate during enrollment. 

The experiment was conducted on a sample of cases in 

two states whose laws require compliance with the CES 

program. 

Experimental Treatment Conditions. The 

experimental variation across treatment groups was 

introduced at the mailing stage and the enrollment stage. 

from North Carolina and Florida (states with mandatory 

compliance laws) and four SIC codes in retail and 

services sectors was used. 

Results. Approximately 78% of all eligible cases 

agreed to participate during the enrollment call, with 85% 

(s.e.=2.6) of cases in the mandatory condition agreeing to 

participate, compared to 72% (s.e.=3.2) of cases in the 

nonmandatory condition. 

The experimental design did not allow us to test 

hypotheses about how strong the mandatory appeal 

needed to be to improve response rates. There is some 

evidence on this issue. For example, Tulp et al. (1991) 

found that a mandatory statement on an outer envelope 

raised response rates by businesses to Census Bureau 

surveys by 20 percentage points or more. In a national 

probability study, Dillman, Singer, Clark and Treat 

(1996) found that a mandatory appeal using a statement 

on the outer envelope was as effective as an appeal using 

both an insert and the envelope statement. However, 

when evaluating this evidence, an important design 

difference to weigh is the fact that the CES data request 

is not a one-time request and may, in fact, need a 

particularly strong appeal. 

Experiment 6. The Effect of Requested Length of 

Participation on Cooperation. The specific aim of this 

study was to assess the impact of describing the length of 

program participation as two years versus three years on 

the likelihood of obtaining agreement to participate 

during enrollment. The study was motivated by a 

concern at the BLS about whether cooperation rates were 

negatively affected by a requested length of participation 

(three years), compared to some other shorter period of 

time. A sample of 740 employers of less than 250 

employees from 4 states and four SIC codes in retail and 

services sectors was used. 

Experimental Treatment Conditions. The 

experimental variation across treatment groups was 

introduced at the mailing stage and the enrollment stage. 

Following sample refinement and assignment to the 

experimental treatment condition, interviewers mailed 

program materials that varied by whether the data request 

was made for "24 months or 2 years" or "36 months or 3 

years" to the contact person. 

Results. Approximately 53% of all eligible cases 

agreed to participate during the enrollment call, with 

54.8% of cases in the two-year condition agreeing to 

participate, compared to 52.8% of cases in the three-year 

condition. There were no significant effects of panel 

length on cooperation rates during enrollment, regardless 

of firm size or SIC category. It is noteworthy that more 

elaborate logistic regression models were estimated to 

explore the possibility that the effects of treatment 

condition depend on firm size or SIC category. No 

292 
These results suggest that a three-year panel length is 

the most cost effective choice. Respondents who were 

told that participation would last for two years were no 

more likely to agree to participate than those who were 

told that participation would last for three years. 

Moreover, the three-year condition was no more costly 

than the two-year condition in terms of number of calls or 

number of person contacts. In fact, the three-year 

condition is less costly than the two-year condition in 

terms of numbers of panels to be rolled into the sample. 

5. Handling the Reluctant Respondent 

5.1 Perceived Burden as a Function of Information 

Access 

It appears common in establishment surveys that 

reluctant respondents will reveal that their lack of access 

to the desired information complicates providing the 

survey interview. Establishment survey protocols thus 

invest more time in getting the name of a contact person 

with intimate knowledge of the information being sought. 

5.2 Experience with Using Information to Obtain 

Goals 

It is a common interviewing technique in establishment 

surveys to couch arguments for cooperation in terms 

related to the organizational goals of the sample unit. For 

example, when the unit is affected by interest rates and 

competitive pressures for staff salaries, a survey of 

payroll and employment can be presented as providing 

information of use to the firm for its own decisionmaking on future activities. Similarly, a farm survey of 

costs and production can be portrayed as valuable to the 

sample farmers in their own work. It seems safe to 

observe that business establishments more readily tend to 

make decisions regarding their goal attainment using 

objective information than do households. There appear 

to be two levels of this phenomenon. First, since many 

business persons use quantitative information in their day 

to day work, they appreciate the utility of such 

information for other purposes, as well. Second, when 

the survey collects data from their unit that they use for 

their own decision-making, the interviewer can attempt to 

link their personal value of that information to the value 

of statistics on larger population, based on those data. 

5.3 The Existence of Policies Guiding Survey 

Participation 

Some businesses have formal policies about what types of 

information can be released outside of the firm. These 

policies are rules of the company that constrain the 

behavior of all employees, unless exceptions are made or 

the rules changed. Interviewers report that this form of 

reluctance is the most difficult for interviewers to 

interviewers is an appeal to a level of authority that can 

label the case as an exception to the rule. In 

organizations with such rules formally codified in written 

procedures, the likelihood is small for such decisions. In 

less formally documented cases, the odds increase. 

For any of the above reasons, it appeared likely that 

CES interviewers needed a set of flexible techniques to 

address the variety of criteria used by contact persons in 

making decisions regarding survey cooperation. 

Experiment 7. Interviewer Training to Handle 

Reluctant Respondents. The goal of this experiment was 

to test the impact of a training protocol specifically 

developed to increase interviewers' skills in overcoming 

respondent reluctance during solicitation. Response rates 

achieved by interviewers prior to the reluctance training 

were compared to post-training response rates. A sample 

of 320 employers of less than 100 employees from 4 

states and four SIC codes in retail and services sectors 

was used in a "pre-" phase and 329, in a "post-" phase. 

Seventeen interviewers from the Kansas City data 

collection center (DCC) participated in both phases of the 

sample refinement and solicitation stages. Work-histories 

supplied by the DCC manager indicated that all had been 

employed primarily as CATI data collectors and lacked 

experience with solicitation prior to this study. Their 

average tenure as BLS interviewers was nearly 2 years, 

ranging from approximately 1 year to nearly 5 years. 

Prior to the start of Phase 1, interviewers received a 

general training on data solicitation that entailed a review 

of a sample solicitation script, multi-establishment 

classification rules and the respondent packet materials. 

The Reluctance Workshop took place between the 

completion of Phase 1 and the start of Phase 2. 

The Reluctance Workshop. Each interviewer was 

provided with a workshop manual including notes which 

paralleled the training. Specific topics covered in the 

workshop were: general principles of refusal aversion; 

non-specific respondent concerns; diagnosing the 

concern; time and burden concerns; government 

concerns; dealing with hostile respondents; company 

policy concerns; confidentiality concerns; and the "passoff." 

Workshop participants learned that refusal aversion 

is a two-step process: once reluctance is encountered, the 

interviewers' task is to quickly and accurately 

identify/diagnose the source of the reluctance and 

appropriately match a rebuttal to that source. The 

workshop interspersed mini-lectures on each topic with 

written and oral practice exercises, including role-playing 

in pairs (interviewer-respondent) and small groups. An 

important goal of the role-playing was to help the 

interviewer provide quick and accurate diagnoses of 

respondent concerns and, in a seamless fashion, provide 

293 
encouraged to practice rebuttals using their own words. 

Interviewer evaluations indicated that they did not 

particularly enjoy the role-playing but nearly all of them 

rated it as one of the most helpful aspects of the training, 

both because of the practice it provided and because of 

the exposure to the rebuttals of other interviewers. 

Results. There was a statistically significant increase 

in enrollment rates in the post-workshop phase. 

Cooperation was obtained from 62.9% (s.e.=3.0%) of 

cases worked before the Reluctance Workshop (i.e., 

Phase 1 respondents), compared to 73.3% (s.e.-2.8%) of 

cases worked following the Reluctance Workshop (i.e., 

Phase 2 respondents). This increase between the phases 

of slightly more than 10 percentage points is statistically 

significant (t=2.53, p<.05, one-tailed). Examination of 

enrollment rates by firm size showed a pattern of higher 

rates in Phase 2 compared to Phase 1 for fn'ms of all sizes 

except those with 20-49 employees. The largest 

differences were found in the smallest firms, suggesting 

that the reluctance training was particularly helpful for 

dealing with small firms. Interviewer-level analyses 

revealed that the workshop may be most cost effectively 

targeted to the lowest performing interviewers. 

Interviewers with Phase 1 enrollment rates below the 

median were compared with interviewers above the 

median on their Phase 2 enrollment rates and on their 

change in rates between the phases. The two sets of 

interviewers had similar Phase 2 enrollment rates. As 

would be expected, low performers showed the most 

improvement between the phases, while high performers 

changed very little. These results indicate that the 

workshop had little impact for the high performing 

interviewers, but was effective in leveling the playing 

field for the lower performing interviewers. 

6. Summary 

This set of experiments was designed to systematically 

probe various stages of the enrollment protocol of an 

establishment survey, with results contributing to design 

decisions. At the conclusion of the studies some 

decisions were taken; others were postponed to await 

further assessments. The head of payroll was the chosen 

contact person; advance mailings consist of the new 

folder with a trifold and a short description of data items 

sought; and the reluctance training has been implemented. 

The results of the mandatory/voluntary experiment, the 

study of births, and the panel length experiments have not 

been implemented, awaiting meshing with other design 

features. 

References 

Couper, M.P., "The Impact of Survey Introductions on 

Data Quality," paper presented at the International 

Quality, Bristol, England, April, 1995. 

Couper, M.P. and Groves, R.M. (1996), "Household 

Level Determinants of Survey Nonresponse," 

Chapter Five in Braverman, M.T. and Slater, J.K. 

(eds.) Advances in Survey Research, 70, Summer, 

1996, pp. 63-80. 

Dillman, Don, Mail and Telephone Surveys: the Total 

Design Method, New York: John Wiley and Sons, 

1978. 

Freedman, J.L. and Fraser, S.C., "Compliance without 

Pressure: the foot-in-the-door Technique," Journal of 

Personality and Social Psychology, 1966, 4:195-202. 

Groves, R.M., Cialdini, R.B., and Couper, M.P., 

"Understanding the decision to participate in a 

survey." Public Opinion Quarterly, 56 (4): 475-495, 

1992. 

Groves, R.M. and Couper, M.P. (1995), "Theoretical 

motivation for post-survey nonresponse adjustment 

in household surveys." Journal of Official Statistics, 

11:1, pp. 93-106. 

O'Neil, M., Groves, R. M., and Cannell, C. C., 

"Telephone Interview Introduction and Refusal 

Rates: Experiment in Increasing Respondent 

Cooperation," Proceedings of the Survey Research 

Methods Section of the American Statistical 

Association, 1979. 

Reingen, P.H. and Kernan, J.B., "Compliance with an 

Interview Request: a Foot-in-the-door, SelfPerception Interpretation," Journal of Marketing 

Research, 1977, 14:365-369. 

Survey Methodology Program, University of Michigan, 

Report on Sample Refinement Experiment, 1996. 

294