Data Collection Methods in Establishment Surveys 

Brad Edwards and David DesRoches 

Brad Edwards is at Westat, 1600 Research Blvd., Rockville, MD 20850 (email BradEdwards@westat.com). 

David DesRoches is at Mathematica Policy Research, P.O. Box 2393, Princeton, NJ 08543 (email 

DDesRoches@mathematica-mpr.com 

1. Introduction 

Survey methodologists often choose surveys of people or households as their primary research interest. However, a 

considerable literature has developed over the past two decades on methods for surveys of establishments. 

Establishments present a number of survey challenges today. Some are shared with all surveys, and others are 

unique to establishments. 

2. Cognitive Model for Response in Establishment Surveys 

The process of responding to a survey question is currently understood as taking the form of four distinct tasks: 

comprehending the question, retrieving an answer from memory, making a judgment about how well the answer fits 

the question, and providing a response. For establishments, the process is mediated by people, but in some sense the 

establishment is more properly the research subject and the respondent. Institutional memory may be shared (and 

shared unequally) by a number of people, and much of it is likely to be codified in records. Each step in the response 

process can be viewed as taking place at the intersection of an establishment’s employees or members and its 

records. Response quality is dependent, not just on instrument design and respondent ability, but also on choosing 

the best respondent who can access appropriate records and make judgments for the establishment. 

3. Aligning Questions with Respondents and Data 

The best respondent may be the head of the establishment or a department within it, or may be someone in a lesser 

position who deals with the public or who fills a specific function (e.g., a quality assurance nurse in a nursing home; 

a building manager in a survey of commercial energy consumption; a receptionist; a records or accounting clerk). 

Given a specific analytic construct or survey objective, the methodologist can define the respondent requirements, 

perhaps with the help of some cognitive testing. Who has the data? Who has ready access to the data? Whose 

opinion is most relevant for the question? Are the data in records? Often the best answers to these questions, across 

many survey constructs or objectives, result in criteria that are a compromise, and it may not fit some people or 

establishments very well. 

4. Mode Choices 

At least as many modes are available for establishment surveys as for household surveys. Web, paper (mailed, 

faxed, or scanned), telephone (voice/keypad/interactive voice recognition or IVR), face-to-face (one-on-one or 

group administration) have all been used to collect data from many establishments. Factors that affect the decision 

about the most appropriate mode for a given survey include the nature, quantity, and quality of the data that are to be 

collected, the resources available, the burden on the establishment, and the capacity of the establishment and 

respondents. 

Multiple modes are often used for a given survey. The most common reasons for choosing multiple modes are 

minimizing cost without diminishing quality, and maximizing response rates. Modes may be used sequentially or 

simultaneously. In a cross-sectional survey, one mode (mail, face-to-face or telephone) might be used for the initial 

approach, and another mode (e.g., Web) might be used for follow-up. In a panel design, one mode might be used for 

collecting data at time 1, and other modes brought into play in time 2 or 3. Different modes might be suitable for 

different respondents in the same establishment, or for establishments that vary in size or on other dimensions. 

Multi-mode may reduce total burden on the establishment but increase resource demands in the survey organization. 
Multiple modes introduce complexity in data collection management and processing, and may produce effects that 

are mode-specific (i.e., mode effects). 

5. Current Practice: Choosing Mode(s) 

The most common current practice is to assign one mode to all establishments. The mode might be chosen based on 

what similar data collection efforts have employed with these establishment types. Web is an attractive mode 

because most establishments have an internet connection and staff who are “Web savvy.” If the establishments are 

sampled from a list that contains e-mail addresses, a message with a link to the survey can easily be sent. The mode 

is inexpensive and produces results quickly. 

As costs have risen and more (and less expensive) modes have become feasible, surveys have increasingly tried 

offering more than one mode, and ask the respondents to choose which mode they prefer. Research on household 

surveys suggests that offering a choice can have the counter-intuitive effect of depressing response rates. It has also 

become common to begin with one mode for all, and then switch to other modes when the first mode does not 

produce a response (e.g., moving from mail to telephone, or from telephone to face-to-face). 

6. Current Practice: Selecting Respondent(s) 

Most establishment surveys select one respondent per establishment. Respondent selection is often based on title or 

position, with the assumption that all or almost all sampled establishments have someone with that title or function. 

Alternatively, one might start with a topic (e.g., policies regarding disposal of hazardous waste) and ask the 

establishment to identify the best respondent to answer questions about the topic. Ultimately, the establishment 

determines who (if anyone) will answer the questions. Survey requests addressed to the owner or manager might be 

received by an assistant and delegated to someone else. 

7. Current Practice: Minimizing Nonresponse 

Unit nonresponse rates are not a concern for mandatory surveys, but for all other establishment surveys minimizing 

nonresponse can be a major objective. A mixed mode design can help. Other common tools include promotional 

material (including recruitment material on a Web site), endorsements from organizations (e.g., trade or professional 

associations) or prominent individuals, and offering monetary or in-kind incentives to the establishment or the 

individual respondent. 

Incentives highlight the blur between establishments and individuals within the establishment who answer the 

survey questions. Is it better to offer an incentive to the establishment or to the respondent? The answer may 

depend on the burden level and whether the respondent’s time answering the survey questions is paid by the 

establishment. The intended effect of the incentive should also be kept in mind: the goal of the incentive is to 

increase the response rate, so that could determine the appropriate recipient. 

Item nonresponse rates are another concern that arises from the cognitive model of survey response in 

establishments. In situations where the individual respondent does not know the answer or refuses to provide an 

answer, it can be unclear whether the information exists in the establishment. Perhaps it resides in records the 

respondent did not or could not access, or perhaps another employee in the establishment knows the answer. One 

solution some surveys have tried is to flag an item as missing during the interview, identify where the information is 

or who has it, and pursue the record location or the other individual at the end of the interview. Alternatively, the 

missing information might be retrievable in a follow-up contact such as a phone call. 

8. Current Practice: Using Administrative Data 

Establishment surveys are especially well suited to link survey data with data from administrative records. Most 

establishments maintain many records, typically in electronic form that can be accessed easily by an employee. The 

administrative data can be used to validate survey responses, or to augment the survey data. It may be possible to 

link the administrative records with survey data after data collection is complete. Some surveys match data at the 

individual level. For example, a survey of jails and prisoners within jails might match survey data and criminal 
justice administrative data at the prisoner level. Or a survey of businesses might match each business with its Dunn 

and Bradstreet data. 

Some establishment surveys use administrative data to substitute for missing or conflicting data in the survey. 

Administrative data are also commonly used for benchmarking survey estimates. 

9. Total Survey Error in Establishment Surveys 

The Total Survey Error (TSE) framework is useful in designing and managing establishment surveys, especially for 

nonsampling error. TSE encourages identification of errors and error sources, determining the direction of errors 

within a given source, and quantifying error. It emphasizes close examination of tradeoffs between two error 

sources. Methodologists and survey managers seek to minimize TSE, and understand and address major error 

sources. But they also seek to control or reduce costs. Tradeoffs between cost and quality are common in all 

surveys, but take on special significance in establishment settings, because of the opportunities to extract data from 

records. Other tradeoffs common in establishment surveys are: 

 Item nonresponse bias vs. unit nonresponse bias 

 Coverage vs. effective sample size 

 Data quality vs. timeliness of release 

Unit nonresponse is associated with respondent/establishment burden and other barriers to responding. Level of 

burden varies widely in establishment surveys, from a few items reported by Web or telephone to many hours of 

face-to-face data collection from a number of employees in an establishment, repeated again and again over months 

or years. Perceived burden may also vary widely, and may be related to establishment size and complexity. Very 

small or very large establishment size can be a barrier to responding. 

Organizational structure and gatekeepers who protect key employees can also be barriers. Chain structures (such as 

grocery stores or pharmacies) can be problematic because it is often unclear where authority to participate in a 

survey effort lies, and the locus of control may vary from person to person and from one establishment to another in 

a chain. Many establishment surveys have chosen to begin at the establishment level and work up the corporate 

ladder if necessary, because that is viewed as posing the least risk to unit nonresponse. However, patterns of 

information flow within the chain and the degree of local autonomy have implications for the data collection 

approach. In some chains, a more efficient strategy might be to begin at the top of the chain, and seek to obtain 

central clearance for all sampled establishments to participate. This is another example of a tradeoff between cost 

and quality. 

Pre-loading information about the establishment into the data collection instrument, and using that information to 

construct more meaningful, less redundant questions can reduce burden at the respondent level. Conversely, 

collecting data in the interview that are readily available from public sources can increase perceived burden. 

10. Comparative Error and Conditioning Error 

Comparative error has been identified recently as a component of TSE. The potential for its presence lies in any 

study that compares groups of respondents, but it is commonly found in multinational and multicultural surveys. 

Many methodologists begin instrument design by drafting questions, but an important prior step is to define the 

analytic constructs for the study (e.g., define home ownership before drafting “Do you rent or own your home?”). In 

comparative surveys, the existence of the constructs must be validated in all groups or cultures that are targeted for 

comparison. Furthermore, the instrument’s ability to identify true differences at the individual level, as distinct from 

the different ways groups may comprehend the instrument, must be established (through DIF analysis or other 

means). If these steps are ignored, the risk of comparative error is high. Such error can also arise from using 

different survey infrastructures, data collection processes or data processing protocols, common practice in 

multinational surveys. With increasing globalization, surveys that collect data to compare establishments by ethnic 

group, culture, nation, and region fill an important demand for data and are a fast-growing share of the establishment 

survey sector. 
Conditioning error is another type of error that may exist in establishment surveys with longitudinal designs, but it 

has not been identified as a component of TSE. It arises when a respondent learns from participating in a survey 

once, or from answering a series of questions in a long, complex instrument, that the next interview or next question 

is likely to be in the same format. Thereafter, the respondent can predict the response(s) that might result in the 

shortest interview time or the smallest cognitive burden. Any establishment survey that goes back to the same 

establishments to collect data at different points in time cannot ignore the possibility of conditioning error. Preloading later instruments with previous responses (i.e., dependent interviewing) can bound current questions and 

yield greater consistency in response. However, in the TSE framework, the first step in understanding this error type 

is to identify it in the context of other types of errors. 

In establishments, conditioning error may arise because of low sensitivity to change. The establishment may have 

made an initial decision about the resources or staff required to complete the survey request, and never revisit that 

decision, even in the face of major change in the establishment or in the survey request. And because the 

establishment -- not the respondent -- is the unit of analysis, changes in respondents can create seam effects, related 

entirely to the phenomenon of respondent A answering questions at Time 1, and respondent B answering questions 

at time 2. 

11. Toward Guidelines for Collecting Data in Establishment Surveys 

After reviewing current practice in several aspects of establishment survey data collection, it is possible to suggest a 

few guidelines for the field: 

 Mode choice – tailor the mode choice based on the respondent’s observed or reported communication 

behavior, but don’t ask the respondent to choose; 

 Multi-mode – use multi-mode designs to suit data needs and respondent needs, but mount experiments on 

mode effects; 

 Select the optimal level in establishments for data reporting, based on empirical and cognitive data on roles 

and records in the establishment population; and 

 Incorporate administrative data in survey design. 

These suggested guidelines are at a high level, and could be more helpful if elaborated in some detail and illustrated 

with examples. 

12. Future Research 

Suggestions for future research include studies of: 

 Respondent behavior and optimal mode 

 Techniques to minimize conditioning effects 

 Establishment culture, industry culture and jargon as factors affecting survey design and survey response 

 Cognitive methods to inform matching the data request with the best respondent 

 The potential for new and emerging data collection modes 

 The application of informatics to establishment surveys: studying how to design a data collection system 

that delivers the right question, to the right person in the right place and time, in the right way 

 The potential for analytics to enrich the relationship between survey responses and administrative data in 

establishment surveys 

Applications of analytics include the study of business data (e.g., paradata, “big data,” unstructured data) using 

statistical analysis to discover and understand historical patterns with an eye to predicting and improving business 

performance in the future. Its application to establishment surveys could produce better data collection designs, 

integrating business information from a variety of sources (people, records) and producing higher quality data at less 

cost.