Does it Pay to Try Again? Using Production Metrics from the 

Recruiting Process on an Establishment Survey to Design 

Recruiting Protocols

Sara Zuckerbraun, Patricia LeBaron, John Loft

RTI International

szuckerbraun@rti.org, plebaron@rti.org, jloft@rti.org

Manisha Sengupta

National Center for Health Statistics

msengupta@cdc.gov

Introduction and Past Literature

Early survey literature states that the number of contact attempts is one of the most powerful predictors of response 

rate (Dillman, Christenson, Carpenter, & Brooks, 1974; Goyder, 1985, 1987; Heberlein & Baumgartner, 1978). 

Nevertheless, repeatedly contacting sample members to gain their agreement to participate often fails to produce an 

increase in survey completions. More recently, survey literature has focused on this concept of diminishing returns 

in household surveys. Findings from the Behavioral Risk Factor Surveillance System Survey show that after 31 days 

in the field, the number of cases receiving a final disposition and the number completing interviews declines 

drastically. This decline was such that continuing data collection beyond 31 days was deemed inefficient (Qayad, 

Balluz, & Garvin, 2009). The Survey of Income and Program Participation Methods Panel also documents 

diminishing returns from repeated attempts. Contact logs from this study show that 24% of cases were finalized at 

the first contact. After only three attempts, 60% of cases were finalized. This number increases to 90% upon eight 

attempts (Bates, 2003). Similarly, Tancreto and Bentley (2005) reported findings that tell a similar story. Using the 

experience of nonresponse follow-up efforts in the 2000 census, the authors showed that additional attempts, 

whether face to face or over the telephone, achieved smaller gains in survey completions relative to earlier attempts. 

The cumulative gains in response rates after the fourth attempt were half of the gains after the third attempt. 

In addition to looking at the relationship between number of contact attempts and gains in response rates, some 

literature also examines the relationship between the length of the field period and gains in response rates. Using a 

10-day effort within the Current Population Survey, Groves and Couper (1998) found diminishing returns in the 

latter days of the field period. Specifically, the first 30% of the time was highly productive and yielded a 43% 

cumulative resolution rate. This productivity waned, and by the close of day 7, 70% of the time had elapsed and 

79% of cases had been resolved. The last 30% of the time needed to be devoted to only 21% of the cases. In addition 

to increasing completion rates, researchers may decide to extend a data collection period to improve the 

representativeness of the completed cases and reduce nonresponse bias. Teitler, Reichman, and Sprachman (2003) 

document a study demonstrating that as the data collection period progressed, respondent sample characteristics 

more closely resembled those of the population. However, the authors also found this increased representativeness 

quickly diminished as levels of effort increased. 

Problem

The concept of diminishing returns on establishment surveys has not received significant attention in the survey 

research literature. Some notable exceptions are Messeri et al., 1995; Edwards, Long, Hogue, and O’Neill, 2009; and 

Britt and Featherston, 2007. Messeri et al. offer a straightforward discussion of the steps taken and personnel 

involved when patient care agencies decide whether to participate in a study and state the average length of time 

(16.4 weeks) needed for recruiting and collecting data from these agencies. The calendar time needed for 

organizations to complete surveys is also a focus of the Edwards et al. and Britt and Featherston papers; Edwards et 

al. found that that it took 154 days to reach the targeted response rate, and Britt and Featherston reported an average 

completion time of 40 weeks. Apart from these papers, there are few published metrics or guidance for 

establishment surveys regarding the length of data collection and the persistence of contact attempts. Although 

1
 

 

 

 

 

 

 

respecting that organizations need time to process the survey request, survey researchers nevertheless face the 

question of “how much is enough?” 

Although our research cannot answer this question for all types of establishments, in this paper we examine the 

relationship between level of effort and completion rates on the National Survey of Residential Care Facilities

(NSRCF). 

Research Questions 

We developed four research questions about the relationship between level of effort and completion rates. In 

developing and answering these, we found the response process model for establishment surveys outlined by 

Sudman, Willimack, Nichols, and Mesenbourg (2000) and expanded upon by Willimack, Nichols, and Sudman

(2002) and Willimack (2007) to be useful in providing a theoretical basis for our findings. Summarized briefly, what 

we will call the “Sudman-Willimack model” depicts how organizations reach their decision about whether to 

participate in a survey. The model recognizes that organizations proceed through steps of understanding the nature 

of the survey questions, determining the appropriate respondent(s) and/or administrative records possessing this 

information, assessing the priority of responding, and finally, responding. To the organization, these steps are work 

that does not produce organizational benefits. The steps and decisions will vary based on attributes of the 

organization, such as its structure, differentiation of functions, and the way authority is structured. The model 

explains a number of points that we found in our research and that practitioners of establishment surveys observe in 

general: why establishment surveys tend to have longer field periods than household surveys, and why some 

establishments seem to be harder to contact and take longer to agree to participate than other establishments.

Our first research question was: what is the relationship between level of effort made on a given case and the rate at 

which that case completes the interview? We operationalized level of effort in two ways: the number of contact 

attempts made by interviewers, and the elapsed calendar time from when the interviewer began working the case 

until the case communicated its final decision. Our second research question examined how the level of effort 

required to complete cases varied by organizational characteristics of the facility, namely size, ownership, and 

geographic location. Specifically, we looked at how small versus very large establishments and those independently 

owned versus owned by a larger organization differ in terms of the level of effort required to achieve a completed 

interview. Third, we asked: do late responders differ significantly from others on key items in the instrument? Late 

responder analysis has been recognized and widely used as a form of nonresponse analysis (Ramirez, 2007). It 

assesses whether late responders (who proxy for the nonresponders) differ from those who respond earlier in ways 

that systematically introduce bias into the parameter estimates collected by the survey. We were interested in the late

responder analysis since it directly told us whether the additional level of effort and calendar time yielded additional 

completed cases, without which our survey estimates for key variables would have been biased. Finally, our fourth 

research question essentially paraphrased the title of our paper: “Does it pay to try again?” We assessed the 

likelihood that an additional contact attempt will result in the establishment agreeing to the interview. We combined 

this assessment with the findings of the late responder analysis to answer if and when it paid to try again on the 

NSRCF. We use the terms establishment, organization, respondent and case interchangeably in our analyses. 

Because the NSRCF was surveying residential care facilities, we often use the term facilities as well.

Data Source

Our data come from the 2010 NSRCF, a national study of residential care facilities providing long-term care to 

primarily older adult populations conducted by the National Center for Health Statistics (NCHS), Centers for 

Disease Control and Prevention (CDC). The 2010 NSRCF used a stratified two-stage probability sample design. The 

first stage was the selection of residential care facilities (RCFs) from the sampling frame representing the universe 

of RCFs. To be eligible for the study, RCFs were selected that are licensed, registered, listed, certified, or otherwise 

regulated by the state and that have four or more licensed, certified, or registered beds; provide room and board with 

at least two meals a day; provide around-the-clock on-site supervision; and help with personal care, such as bathing 

and dressing, or health-related services, such as medication management. Facilities licensed to serve the mentally ill 

or the developmentally disabled populations exclusively were excluded. The primary sampling strata of RCFs were 

2
 

 

 

 

 

 

 

 

 

defined by facility size and region. Within these sampling strata of RCFs, facilities were sorted by metropolitan 

statistical area status and selected by using systematic random sampling. There were 3,605 RCFs randomly selected 

to become the survey sample. NCHS contacted the 3,605 facilities by mail and telephone and asked the facilities to 

complete a short telephone screener. Of these 3,605 facilities, 2,644 were determined eligible while eligibility could 

not be ascertained for 325 facilities. The unweighted response rate for the screener was 90.5%. Facilities that 

screened as eligible facilities were asked to make an appointment to complete an in-person survey. Of the 325 

facilities with unknown eligibility, 262 were estimated to be inscope using the percent of eligible facilities in the 

group that could be contacted, bringing the total number of estimated inscope facilities to 2,906. Interviews were 

completed with 2,302 facilities, for a first-stage facility unweighted response rate of 79% and a weighted (for 

differential probabilities of selection) response rate of 81%. In the second stage of sampling, depending on facility 

size, three to six residents were sampled from each participating facility. Data were collected using two separate 

questionnaires: the facility questionnaire included questions about the sampled facility, and individual resident 

questionnaires were used to collect information about sampled residents. All data collected on sampled residents 

came from interviews with facility staff who answered questions based on referring to the residents’ records or their 

own knowledge of the residents; residents were never interviewed. More information on the NSRCF design may be 

found in the NCHS Series 1 Methods Report (Moss et.al, 2011).

For this paper, we analyzed only the 2,644 eligible facilities. Of these, 2,302 later completed the in-person survey, 

and the balance of 342 refused. We used the NSRCF facility questionnaire data, sample frame data, and paradata. 

Paradata capture data about the survey process, including type of contact made, date of contact, and outcomes, while 

sample frame data capture characteristics known about the establishments. 

Measures and Methods of Analysis

The analysis is divided into two broad sections: the first section addresses the relationship between level of effort 

made on a given case and the case completion rates (i.e., the first two research questions), and the second section 

examines whether late responders differ significantly from non-late responders on key items in the instrument (third 

research question). In the first section of the analysis, we operationalized level of effort in two ways, using two 

metrics.

1. Contact attempts. We counted the number of contact attempts made by interviewers over the course of working 

a case. These included making phone calls, leaving voicemail messages, sending letters or e-mails, making in-person 

visits, speaking to personnel at the establishment, and setting interview appointments. This metric is designed to

measure interviewer effort. On some projects, this metric is called “events” or “attempts.”

2. Elapsed time. We calculated the elapsed days from when the interviewer first began working the case to when 

the establishment either completed the in-person survey or conveyed to the interviewer their final decision not to 

complete. We conceptualized five points in the lifecycle of each case: (a) start, (b) complete screener, (c) make 

appointment for interview, (d) complete interview, and (e) refuse. A case can take one of three potential paths:

Screen, schedule, then complete: a-b-c-d 

Screen, schedule, then refuse, a-b-c-e

Screen, refuse: a-b-e

Our metric of elapsed days for the full survey begins counting from point a and ends at what we consider to be the 

establishment making their final decision. In the case of a refusal, this is point e. In the case of a complete, this is 

point d. The argument could be made that point c better represents the final decision point among establishments 

which complete, but we felt that since establishments can (and frequently do) decide at the last minute to cancel, the 

time between c and d also represents deliberation time during the establishment’s decision process. Those 

deliberations end when the establishment finally completes the in-person survey.

In the late-responder section of the analysis, we used these same two metrics - elapsed time for completion, number 

of contacts attempts - to identify late responders. We defined late responders in two ways: facilities whose elapsed 

time to complete the survey exceeded 120 days, and facilities that completed the survey after more than 20 contact 

attempts. Using the number of elapsed time criterion, there were 436 late responders among completed cases 

(18.9%). There were 243 late responders (10.6%) using the number of contact attempts criterion.

3
 

 

 

 

 

 

 

 

 

 

 

The late responder analysis was divided into 2 parts. In the first part we compared the size distribution of completed 

facilities to the size distribution of facilities on the sample frame, and examined how the inclusion and exclusion of 

the late responders impacted the distribution. This gave us a picture of how late responders might impact the survey 

results from a nonresponse bias perspective. In the second part, we compared late responders with non-late

responders on key survey estimates to examine whether there were significant differences between them. The 

NSRCF collected information on a host of facility and resident characteristics. T-test and Chi-square tests were used 

to identify significant differences between non-late and late responders. We took into account the probability of 

selection by using base weights in the analyses.

Findings

To begin, we present the basic metrics we found for elapsed time and number of contact attempts. Table 1 describes 

these two metrics for all cases, completed cases, and refusal cases. Because a high response rate was needed on this 

survey, interviewers expended a great effort over an extended period of time to gain cooperation among 

establishments. As shown, the average completed case required 77.42 days and 11.01 contact attempts to complete. 

Interviewers spent more time working on the harder cases, only some of whom ultimately completed the survey. 

Among cases that ultimately refused, the interviewers spent, on average, 164.57 days, or about 5 and a half months, 

attempting to convince these facilities to participate. The mean number of contact attempts was 20.73.The bottom 

row for “all cases” shows findings that fall between the findings for completes and refusals, reflecting the means of 

all cases until they reached their final outcome, whether that outcome was a complete or a refusal. 

Table 1. Descriptive Statistics for Elapsed Time and Number of Contact Attempts by Different Case 

Outcomes

Type of Cases

Elapsed Time (Days) Number of Contact Attempts

Mean SD Min, Max Mean SD Min, Max

Completed Cases 77.42 51.40 10, 253 11.01 7.36 2, 57

Refusal Cases 164.57 56.81 9, 252 20.73 11.80 4, 66

All Cases 88.75 59.78 9, 253 12.28 8.71 2, 66

I. Addressing Research Question 1: What is the relationship between level of effort made on a given case and 

the rate at which that case completes the interview?

To address the first research question, we charted the percentage of completed cases along the continuum of days in 

the field period. In other words, Figure 1 shows the percentage of ultimately complete cases that were complete on a 

given day in that case’s field period. As is typically the situation, interviewers do not begin contacting all cases on 

the same day. Therefore, the x-axis displays the number of days in an individual case’s field period, as opposed to an 

overall period. The graph normalizes the life cycle of the case, simulating a situation where all of the cases were 

started at the same time. All cases that completed the interview are included (n=2,302). This graph shows that the 

relationship between length of the field period and the number of completes is consistent with the relationship 

typically found in household studies. As the field period progresses fewer completes are obtained. The most difficult 

cases are those that are shown in the upper right hand corner of the graph, in Quadrant I. These cases required the 

longest amount of time in the field and were completed after almost all of the other cases. Conversely, the “lowhanging fruit” are shown in Quadrant III of the graph. These cases were completed both early in the field period and 

were among the first to be completed overall. One case, which was completed in the shortest amount of time, was 

completed in only 10 days. The longest case took 253 days to complete. These durations are lengthier than one 

would typically find on a household survey. They complement earlier findings (Willimack et al., 2002; Willimack, 

2007) that there is a response process in organizations that is different from, and longer than, that among household 

respondents. 

4
 

 

 

 

Figure 1. Percent Complete Cases by Elapsed Time

100

80

60

40

20

0

Quadrant 2

Quadrant 3

Quadrant 1

Quadrant 4

0 50 100 150 200 250 300

Elapsed Time in Days

In addition to the percent of completed cases by duration in the field, we operationalized level of effort by 

measuring number of attempted contacts with sample members throughout the data collection period. Figure2,

depicting completed cases by contact attempt, shows a less curvilinear distribution. This graph shows us that some 

cases were completed with few attempts, but at the end of the data collection period, when a large percentage of 

cases had already been completed. These cases populate Quadrant II of the graph. This could be explained by 

contacts at the sampled establishment indicating that appropriate staff would not be available for a number of weeks, 

and thus making appointments for recontact far into the future. These cases would have a longer field period with 

few contact attempts. Notice that there are no cases in the fourth quadrant of the graph. For a case to be depicted 

here, more than 30 contact attempts would have had to occur before half of the sample was completed—an 

impossibility.

Overall, this scatter plot graph reveals a similar pattern as the previous line graph. The highest numbers of contact 

attempts were made after the majority of cases were completed. The scatter plot also provides a view of the outliers

and illustrates, for instance, how quite a few cases among the first 50% to be completed necessitated twenty or more 

attempts. The scatter plot also shows how cases overall follow a trend but are not clustered tightly around a central 

line.

Figure 2. Percent Complete Cases by Number of Contact Attempts

5
100

80

Quadrant 2 Quadrant 1

60

40

% Complete

Quadrant 3 Quadrant 4

20

0

0 10 20 30 40 50 60

# of Contact Attempts

II. Addressing Research Question 2: How does the level of effort required to complete cases vary by 

organizational characteristics of size and ownership and geographic characteristics?

To investigate this question we looked at our two metrics of elapsed time and contact attempts for different 

subgroups of sample members. The NSRCF sample specified whether a facility was independently owned or part of 

a chain; it also specified its geographic census region and the number of licensed beds in the facility. Small facilities 

were defined as having between four and 10 beds, medium facilities had 11 to 25 beds, large facilities had 25 to 100 

beds, and very large facilities had over 100 beds. One could theorize that differences in characteristics of the 

facilities could affect their response patterns and timing of decisions to participate. Larger facilities and those that 

are a part of a chain may take more time to gain necessary permissions to participate in the survey. Similarly, 

gaining access to the necessary data and identifying appropriate staff to provide these data may take longer in larger 

facilities, resulting in longer field periods. The next four graphs display these relationships. 

Figure 3 shows the number of days the case had been worked versus the percentage of chain-affiliated facilities that 

had been completed, project wide, by that day. The other line displays this for independent facilities. The graph 

shows that while these two lines do not diverge widely and certainly become equal at the end of the field period, 

there is a point when independent facilities appear to complete quicker than chain-affiliated facilities. Shortly after 

day 80, the completion rate among chain-affiliated facilities is 60%, while independent facilities are closer to 70%. 

This differential continues until approximately day 225 when the two lines converge. In addition to graphing, we 

also conducted several t-tests of the difference of means. The mean elapsed completion time for chain-affiliated 

facilities was 82.2 days and for independent facilities was 73.9 days; the t-tests revealed that the difference in means 

was statistically significant (p-value <0.01). 

6 
 

 

 

Figure 3. Percent Complete Cases by Elapsed Time by Facility Type

Figure 4 charts the relationship between the number of contact attempts for each case and the percent of completes 

that had been obtained by that point in the project’s field, broken out by ownership type. Looking at the means, we 

see that the mean number of contact attempts for chains to reach completion is 11.47 and for independent facilities it 

is 10.67. The means are quite close to one another, yet the t-test of difference of means did prove that the means of 

the two facility types are statistically different from one another (p <0.01).

Figure 4. Percent Complete Cases by Number of Contact Attempts by Facility Type

7
 

 

We now turn to a discussion of facility size. The following two graphs (Figures 5 and 6) display two curves, one for 

very large facilities and one for small-sized facilities. Our t-tests of the difference of means between the very large 

(100+ beds) and the small (4-10 beds) facilities showed that the mean elapsed time (very large: 88.1 days versus 

small: 75.8 days), and mean number of contact attempts (very large: 13.1 attempts versus small: 10.9 attempts) , 

differed significantly (p <0.01). There were no statistically significant differences between any other combinations 

of sizes. Therefore, our graphs focus only on small facilities versus very large facilities.

Figure 5 clearly shows that small facilities reach completion sooner than very large facilities—the curve

representing small facilities is almost always above the curve representing the very large facilities, until the very end 

of the field period. This graph also shows how both sizes of facilities experience diminishing returns in response 

rates as the field period continues. At day 175, about 90% of cases were completed. To complete the final 10% of 

cases, another 75 days were required. 

Figure 5. Percent Complete Cases by Elapsed Time by Size

Finally, Figure 6 tells a similar story. Relative to the total number of completed cases, small facilities require fewer 

attempts before completing the interview. 

8
 

 

 

 

 

 

Figure 6. Percent Complete Cases by Number of Contact Attempts by Size

We examined the differences in these two metrics between facilities in the four census regions of Northeast, South, 

Midwest, and West. There was no clear direction of difference among facilities in the South, Northeast, Midwest,

and West regions. We then grouped the non-South regions together and performed t-tests on the difference of means 

between South and non-South metrics. The mean elapsed time was 83.7 days for the South and 71.7 for the nonSouth, and the mean number of contact attempts was 12 for the 10 for the non-South. The t-tests tests confirmed that 

the South was indeed statistically higher (p <0.01) compared to the non-South. We do, however, lack a theory as to 

why this is the case.

III. Addressing Research Question 3: Do late responders differ significantly from non-late responders on key 

items in the instrument?

Before addressing this question directly, we decided to assess whether excluding the late responders would have 

impacted the distribution of small, medium, large and very large facilities among the completed facilities. If the 

distribution of the completed cases differed significantly from that of the full in-scope sample, this would have 

signaled nonresponse bias. We compared the size distributions among the entire selected sample (which we consider 

to be the gold standard - if we had 100% participation the distribution of the completed cases would have matched

the distribution of the sample) to the size distribution of all responding facilities, and then to the non-late responders 

and late responders in turn. Table 2 shows these distributions. Differences between non-late and late responders are 

evident in the very large strata. Overall, 7% of the sampled cases belonged to the very large stratum and among 

facilities completing the survey, a slightly lower percentage - 6.5% - belonged to the very large strata. Had there 

been only up to 20 contact attempts and only the non-late responders included, only 6.1% would have belonged to 

the very large stratum. Among the responders responding after more than 20 contact attempts,10.35% belonged to 

the very large stratum. Similarly, using the elapsed time criterion, 6.0% and 8.4% of the non-late and late 

responders, respectively, belong to the very large strata. This shows that later cases are disproportionately weighted 

toward very large facilities, a pattern which was also evident in Figures 5 and 6. 

Table 2. Distribution of facilities by facility size

In-scope All Non-late Late responders

9
 

 

 

 

 

 

Facility size sample responding

Facilities Ā20

attempts

Ā 120 

days

time

>20

attempts

>120 

days

time

Small 49.9 51.4 52.0 51.5 46.1 50.7

Medium 15.9 15.5 15.6 15.6 14.9 15.1

Large 27.2 26.6 26.4 26.8 28.7 25.8

Very large* 7.0 6.5 6.1 6.0 10.3 8.4

*Differences between late and non-late responders by contact attempts and elapsed time significant at p<0.01 based on chisquare test.

We then continued with Research Question 3 by examining whether late responders differed significantly from nonlate responders on survey estimates. Results from this analysis show that late and non-late responders according to 

the criterion of >20 contact attempts differed significantly from one another in: the mean number of apartments, the 

mean number of residents, chain ownership, whether respite care is offered, whether the facility serves Medicaid 

patients, various resident demographic measures, and the number of hours worked by aides and LPNs. Many of 

these are related to size and could be a reflection of the fact that late responders are disproportionately weighted 

towards the very large stratum, as shown in Table 2.

Statistically significant differences between late and non-late responders are shown in Figures 7-9. Only the >20 

contact attempts criterion is shown. 

Figure 7. Selected Facility Size Indicators by Response Status

Contact attempts criterion late Contact attempts criterion non-late

29.5 29.7

23.0 22.2

0.0

5.0

10.0

15.0

20.0

25.0

30.0

35.0

Mean number of apartments Mean number of residents

(1) (1)

1 Difference between late and non-late responders significant (p<.01)

10
Figure 8. Selected Facility Characteristics by Response Status

44.9

32.5

40.1

36.3

25.4

51.3

0.0

10.0

20.0

30.0

40.0

50.0

60.0

Contact attempts criterion late Contact attempts criterion non-late

Percent affiliated with a chain Percent providing respite care Percent serving residents using 

(1) (1) Medicaid-paid LTC services

(2)

1 Difference between late and non-late responders significant (p<.05)

2 Difference between late and non-late responders significant (p<.01)

11
 

 

 

 

 

 

 

 

Figure 9. Selected Caregiver and Resident Characteristics by Response Status

Contact attempts criterion late Contact attempts criterion non-late

38.2

361.2

27 9

23.9

281.6

20 6

Total weekly hours Total weekly hours Mean number of White Mean number of 

worked by LPN worked by Aide residents residents aged 75 - 84 

(1) (1) (1) years (1)

1 Difference between late and non-late responders significant (p<.01)

Results from this analysis show that non-late and late responders to the NSRCF do differ in their responses to 

selected key survey questions, including facility size and ownership characteristics.. One of the limitations of this 

analysis relates to the fact that the analysis is done at the facility level but does not look at what the effects of nonresponse would be at the resident level. Since large facilities account for a large percent of residents, it is likely that

having a lower response rate for larger facilities may affect estimates of resident characteristics more than they 

might of facility characteristics.

IV. Discussion and Research Question 4: Does it pay to try again?

This paper began with the question “Does it pay to try again?” The evidence from the NSRCF suggests an 

affirmative answer in several respects.

First, in terms of completed cases, additional contact attempts produced data from additional facilities. Table 4

shows the distribution of facilities by the total number of contact attempts they had. The number and percentage of 

these facilities that had the desired outcome of complete (rather than a refusal) is also shown. 

Table 4. Distribution of Total Number of Contact Attempts by Facility, and Completion/Noncompletion 

Outcome

# of Contact 

Attempts # of Facilities Complete %

1-5 451 443 98.23%

6-10 1,040 991 95.29%

11-15 505 422 83.56%

16-20 263 203 77.19%

12
 

 

 

 

 

 

 

 

21-25 173 121 69.94%

26-30 96 63 65.63%

31-35 37 21 56.76%

36-40 35 18 51.43%

41-45 18 10 55.56%

46-50 14 7 50.00%

>50 12 3 25.00%

TOTAL 2,644 2,302

Although a completion was most likely from the first attempts, there was a better-than-ever chance of a positive 

outcome event after 30 or more attempts because 21 out of the 37 (56.76%) of the respondents who received 31 to 

35 contact attempts elected to complete the survey. Even as many as 40 to 50 attempts were still more likely than 

not to yield additional completed cases among various facility types (size, ownership). The outcome after 51 or more 

attempts, however, was not very positive since only 25% of those cases elected to complete the survey. The extent to 

which other establishments would behave like the NSRCF respondents is not known. However, it is important to 

link the findings about contact attempts presented in this paper to the Sudman-Willimack response process model. 

Multiple contact attempts in establishment surveys function toward a variety of purposes. There are likely to be 

multiple points of contact within a single establishment. Establishments are bureaucratic entities with both formal 

and informal internal divisions of responsibility. The decision to participate in a survey may require input from the 

director of the organization, from a legal department, and from the individuals who supply the requested 

information. There may not be a single custodian of information. The NSRCF requested information as varied as 

counts of residents, services used by residents, employees functioning in various capacities, and the physical 

properties of the facilities. No single individual in the facility may be the best source about these varied features. 

Establishment surveys may attempt to identify a single point of contact in the responding organization to assemble 

the requested responses. However, this may contribute to additional respondent burden to an individual. Also, a

single respondent may not have information to provide the full array of data requested in the survey. In addition, 

contact attempts may be made as a courtesy to verify receipt of information about the survey or a reminder of a 

pending appointment. 

Second, in terms of nonresponse bias on NSRCF, if the data from facilities completed after >20 contact attempts had 

not been added to the sample of completed facilities, the resulting data set would be deficient in terms of survey 

estimates which were key for our study, such as facility size, facility ownership, resident demographics, staffing of 

aides and LPNs, and Medicaid patients. Relying only on non-late responders, the picture of residential care would 

have under-represented large facilities, facilities affiliated with chains, and facilities offering more hours of LPNs 

and aides. Without these additional data points, the final data set would have presented an inaccurate picture of 

residential care facilities and the services they offer. 

Information about calling patterns among facilities of varying characteristics may be useful in planning the 

recruitment protocol for establishment surveys. 

References

Bates, N. (2003). Contact histories in personal visit surveys: The Survey of Income and Program Participation 

(SIPP) methods panel. Proceedings of the American Association for Public Opinion Research – Section on 

Survey Research Methods (pp. 7-14). Retrieved from

http://www.amstat.org/sections/srms/proceedings/y2003/Files/JSM2003-000437.pdf

Britt, R., & Featherston, F. (2007, June). Increasing survey cooperation: Motivating chronic late responders to an 

annual survey. Paper presented at International Conference on Establishment Surveys (ICES III), Montreal, 

Quebec, Canada.

13
 

Cox, B. G., & Chinnappa, B. N. (1995) Unique features of business surveys. In B. G. Cox, D. A. Binder, B. N. 

Chinnappa, A. Christianson, M. J. Colledge, & Kott, P. S. (Eds.), Business Survey Methods (pp.1-17). New 

York: John Wiley and Sons.

Dillman, D., Christenson, J. A., Carpenter, E. H., & Brooks, R. M. (1974). Increasing mail questionnaire response: 

A four state comparison. American Sociological Review, 39, 744-756.

Edwards, K., Long, K., Hogue, C., & O’Neill, G. (2009). Improving Response Rates for the 2007 Census of 

Government Employment. Government Division Report Series. (Research Report #2009-8). Washington, DC: 

U.S. Census Bureau. Retrieved from http://www.census.gov/prod/2009pubs/govsrr2009-8.pdf

Goyder, J. (1985). Face-to-face interviews and mailed questionnaires: The net difference in response rate. Public 

Opinion Quarterly, 49(2), 234-252.

Goyder, J. (1987). The silent majority: Nonrespondents on sample surveys. Boulder, CO: Westview Press. 

Groves, R. M., & Couper, M. P. (1998). Nonresponse in household interview surveys. New York: Wiley.

Heberlein, T. A., &Baumgartner, R. (1978). Factors affecting response rates to mailed questionnaires: A quantitative 

analysis of the published literature. American Sociological Review, 43, 447-462.

Messeri, P., Aidala, A., Abramson, D., Healton, C., Jones-Jessop, D., & Jetter, D. (1995). Recruiting rare & hard to 

reach populations: A sampling strategy for surveying NYC residents living with HIV/AIDS, using agency 

recruiters. Proceedings of the American Statistical Association Section on Survey Research Methods (pp. 1064-

1068). Retrieved from http://www.amstat.org/sections/srms/Proceedings/papers/1995_185.pdf

Moss AJ, Harris-Kojetin LD, Sengupta M, et al. Design and operation of the 2010 National Survey of Residential 

Care Facilities. National Center for Health Statistics. Vital Health Stat 1(54). 2011.

Qayad, M. G., Balluz, L., & Garvin, W. (2009, February). Does continuing data collection beyond one month 

improve the completion and response rates in behavioral risk factor surveillance system survey? Survey 

Practice. Retrieved from http://surveypractice.wordpress.com/2009/02/25/extra-calls-and-response-rates/.

Ramirez, C. (2007, June). Nonresponse bias analysis in a survey of banks. Paper presented at International 

Conference on Establishment Surveys (ICES III), June 2007.

Sudman, S., Willimack, D., Nichols, E., & Mesenbourg, T. (2000, June). Exploratory research at the U.S. Census 

Bureau on the survey response process in large companies. Paper presented at International Conference on 

Establishment Surveys (ICES II), Buffalo, New York.

Tancreto, J. G., & Bentley, M. (2005). Determining the effectiveness of multiple nonresponse followup contact 

attempts on response and data quality. Proceedings of the American Statistical Association Section of Survey 

Research Methods (pp. 3626-3632). Retrieved from 

http://www.amstat.org/sections/srms/proceedings/y2005/Files/JSM2005-000340.pdf

Teitler, J. O., Reichman, N. E., & Sprachman, S. (2003). Costs and benefits of improving response rates for a hardto-reach population. The Public Opinion Quarterly, 67(1), 126-138.

Willimack, D. K. (2007, June). Considering the establishment survey response process model in the context of the 

administrative sciences. Paper presented at International Conference on Establishment Surveys (ICES III), 

Montreal, Quebec, Canada.

Willimack, D. K., Nichols, E., & Sudman, S. (2002). Understanding Unit and Item Nonresponse in Business 

Surveys. In R. M. Groves, D. A. Dillman, J. L. Eltinge, & R. J. A. Little (Eds.), Survey Nonresponse. New

York: Wiley.

14