# #

The Impact of Efforts to Increase Response Rates on Survey Estimates 

The Impact of Efforts to Increase Response Rates on Survey 

Estimates 

Karen K. Wessels and Frances M. Barlas 

ICF International 

9300 Lee Highway 

Fairfax, VA 22031 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Introduction 

Survey response rates have steadily declined over the last several years, bringing into question the validity of the 

data collected. Oftentimes, researchers use multiple survey modes in an effort to increase their study’s overall 

response rate, resulting in what is referred to as a mixed-mode data collection. However, research suggests that an 

increase in nonresponse does not necessarily produce nonresponse bias, typically a top concern when response rates 

are low (Groves et al., 2006). 

Groves et al. (2006) suggested that nonresponse bias is more likely to occur when the survey title, introduction, etc. 

emphasize the topic of the survey, thus stimulating interest and participation among those interested in the topic and 

decreasing the likelihood of response by those not interested in the topic. This is especially likely to cause 

nonresponse bias when the survey variables are correlated with the interests perceived to be covered in the survey; 

those who are interested in a topic are likely to have different opinions than those who are not interested in the topic. 

Thus, nonresponse bias may be influenced more so by the title of the survey and the information provided in the 

introduction than the response rate itself; the more objective the survey title, the more likely the respondent interest 

in the survey will be mixed, thus decreasing nonresponse bias. 

The Pew Research Center has conducted extensive research on the impact of declining response rates on survey 

estimates and argues that the time and money spent to increase response rates may produce little to no difference in 

the survey results. Three studies were conducted at the Pew Research Center that compare survey estimates from a 

5-day survey using their “standard” methodology with survey estimates using a “rigorous” methodology with a 

much longer field period producing a significantly higher response rate. Both the 1997 and 2003 studies reveal 

differences in survey estimates that were small and “statistically indistinguishable,” indicating that although the 

more rigorous methodology resulted in higher response rates, the survey estimates were about the same (Keeter et 

al., 2006). Specifically, in the 2003 study, estimates for 77 out of 84 comparable items from the standard and 

rigorous methodology were not statistically different. Thus, using more resources in an attempt to increase response 

rates did not impact the quality of the data for the majority of the comparable items. However, it is important to note 

that the rigorous methodology resulted in a response rate of 50 percent, thus the opinions of half of the population 

were still not observed. In addition, the results of this study were based on a political survey and may not be 

generalized to surveys on other topics. These authors, similar to Groves et al. (2006), recommend that a rigorous 

approach be taken when the level of interest in the survey topic may lead to nonresponse by certain groups and thus 

cause nonresponse bias. 

As in the 1997 and 2003 Pew Research Center studies, the 2012 study found few significant differences in estimates 

between the standard survey (yielding a lower response rate) and the more rigorous survey which employed 

techniques aimed at increasing response rates (e.g., longer fielding period, monetary incentives) (Kohut, A, 2012). 

Although the more rigorous survey methodology resulted in a higher response rate (22% v. 9% for the standard 

survey), 28 of the 40 estimates showed differences of two percentage points or less; there were three point 

differences on seven items and four point differences on the remaining five items. 

Groves and Peytcheva (2008) conducted a meta-analysis of 59 studies that were designed to estimate the magnitude 

of nonresponse bias on statistics of interest; using these 59 studies, methodological design factors, including features 

of the survey itself, characteristics of the sample, and survey statistics used to analyze results that produce a 

relationship between nonresponse rates and nonresponse bias, were identified. Groves and Peytcheva’s strongest 

finding was that, although high response rates can reduce the risk of bias, as previously discussed, high response 

rates are less likely to reduce the risk of bias when the motivation to participate is highly correlated with the survey 

variables. Similarly, a secondary finding was that survey estimates most likely to show nonresponse bias are those 

assessing the cause of survey participation (e.g., Why did you choose to participate in this survey?). 

Some nonresponse research has focused on early versus late responders to determine whether these two groups are 

statistically different, under the assumption that later responders are similar to non-responders (Irani, T. A. et al., 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

2004, Paganini-Hill et al., 1993). Other studies focus on comparing early responders with all respondents to 

determine whether additional efforts to increase response rates (after early responders submitted their data) led to 

statistically different results (Groves, R. M., 2006 and Millar, M. M. & Dillman, D. A., 2011). The current study 

uses this methodology to determine whether additional efforts to increase response rates to a dual-mode Army 

survey (web and paper) impacted the survey results. Specifically, the current research investigates whether the 

survey estimates would be statistically different if the survey had been closed earlier, thus not including late 

responders. The current study expands on the findings from the Pew Research Center’s studies by investigating 

estimates from data collected via web and paper, whereas the Pew Research Center’s analyzed data collected via 

telephone surveys. 

As discussed above, the current study uses survey data from a mixed-mode survey, web and paper. There are several 

advantages of employing a mixed-mode data collection, including a decrease in non-response and the potential bias 

associated with non-response due to the respondent having more than one choice of mode, and a reduction in cost 

when a lower cost survey mode is used for a substantial portion of the sample (VanNieuwenhuyze, J. et al., 2010). 

However, there are disadvantages to using a mixed-mode data collection as well, the most significant being the 

potential for the introduction of bias, including selection and measurement effects. Selection effects occur when 

different types of respondents choose different survey modes. However, this same bias of self-selection into one 

survey mode over another can contribute to increased response rates compared to a single-mode survey. 

Measurement effects result from a respondent’s answer being influenced by the survey mode they are using (i.e., the 

participant would respond differently if they answered the same survey using a different mode). Given possible 

selection and measurement effects, the current study analyzes results for the web and paper version of the survey 

separately. We investigate whether or not estimates from data collected via web and paper modes of survey 

administration are differentially impacted by a shorter, less rigorous data collection period. 

Response Rates and Nonresponse Bias of Web Surveys Compared to Other Survey Modes 

Research shows that response rates of web surveys are typically lower than other survey modes. In a meta-analysis 

of 45 published and unpublished experimental comparisons between web and other survey modes, Manfreda et al. 

(2008) found that web surveys yield, on average, an 11% lower response rate than other survey methods. Several 

reasons for this difference were identified, including security and privacy concerns; a lack of new techniques to 

decrease non-response rates in web surveys (e.g., prepaid monetary incentives do not work for web surveys when 

the incentive is transferred electronically); limited web literacy among certain segments of internet users (e.g., lack 

of knowledge of how to access and fill out a web-based survey); low frequency of computer use by some 

individuals; and technical limitations, such as software incompatibilities and long or irregular loading times. When 

specifically comparing web surveys to paper surveys, web surveys may yield lower response rates due to 1) the lack 

of a continuous reminder (e.g., there is no hard copy of the survey sitting on a respondent’s desk); 2) an invitation to 

participate via email is more likely to be overlooked than an invitation via postal mail; and 3) potential respondents 

are more likely to perceive email invitations as spam and less legitimate than postal mail invitations. 

One possible solution to low web survey response rates is to use a multi-mode approach. In a study by Beebe, T. J. 

et al. (2012), web survey non-respondents were contacted by phone after two mailings for follow-up. Response rates 

and nonresponse bias were studied to determine the impact of the multi-mode and follow-up approach. Although 

response rates increased, nonresponse bias did not decrease. 

In a study by Shin, E. et al. (2012), overall unit and item response rates were compared for Gallup panel members 

who were sent web surveys and members who were sent mail surveys. Overall, the web survey yielded significantly 

lower response rates compared to the mail survey (72.9% v. 67.0%, or 9% lower for the web survey). However, it is 

important to note that sample members were not randomly assigned to receive the web or the mail survey; instead, 

they were strategically placed into one group or the other based on their frequency of internet use and whether they 

provided an e-mail address; one may hypothesize that the difference in response rates may have been more dramatic 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

if sample members had been randomly assigned to the two groups. Item nonresponse rates were also investigated; 

results indicated item nonresponse was significantly lower on the web survey than on the mail survey (an average of 

0.3 items on the web survey v. 2.2 on the mail survey). In addition, web respondents were more likely to have no 

missing items (85% for the web survey v. 63% for the mail survey). Of particular importance was the lower item 

nonresponse rate to the height (0.6% v. 7.5%) and weight (1.2% v. 8.2%) questions, characteristically perceived as 

sensitive information, on the web survey compared to the mail survey, indicating respondents may be more willing 

to provide sensitive information on a web survey. The web survey also elicited more open-ended comments at the 

end of the survey compared to the mail survey (item/comment response rate of 19.6% web vs. 15.1% mail). These 

findings suggest that respondents who make the decision to take a survey are more likely to provide complete 

information on a web survey than on a mail survey. However, future research may investigate the quality of the data 

provided using each survey mode. 

Borkan, B. (2010) also studied mode effects for web versus paper surveys. Again, overall unit response rate was 

lower for the web survey than for the paper survey. Contrary to Shin et al.’s (2012) findings, item non response rates 

did not differ by mode, except for a question on respondent age. However, it is important to note that nonrespondents were followed up with the opposite survey mode (e.g., those that did not respond to the web survey 

were sent a paper survey). By gathering additional data in the mode the respondent may have been more comfortable 

with, a potential item nonresponse effect may have been eliminated. In addition to higher response rates to the mail 

survey, Borkan, B. (2010) studied how the opposite mode follow-up impacted response rates. The overall response 

rate increased by 16% when mail survey non-respondents were followed up with web surveys and 87% when web 

survey non-respondents were followed up with paper surveys. This may have implications for cost savings in multimode studies as a researcher can attempt to get a response using a web survey first and only follow up with a paper 

survey for those who do not respond by web. However, it is important to note that overall response rates were higher 

when the paper survey was followed up by a web survey than vice versa (51% v. 40%). Borkan, B. (2010) also 

found no mode effect on the psychometric quality of the items with a rating scale; however, findings support 

previous studies that participants may respond differently to background questions (e.g., race, gender) on a webbased versus a paper-based survey. 

Due to the mode differences identified in previous research, the current study analyzes data collected via web 

separately from data collected via paper-and-pencil surveys received through postal mail. 

Method 

The 2012 Army Morale Welfare and Recreation (MWR) Survey was a mixed-mode survey (web and paper) 

administered in the spring of 2012, and used a stratified random sample by patron group (i.e., active duty Army, 

spouses, retirees, and civilians) and Army Garrisons (75 Army sites) to ensure data were captured on the groups of 

interest to the Army Morale, Welfare, and Recreation office. 

The web survey was in field for approximately 10 weeks, launching on March 12th and closing on May 24th; the 

paper survey was in field for about eight weeks, with the first of two Mail Out Packages (MOPs), including a letter 

about the survey and a copy of the paper survey, being sent between March 23 and April 4, 2012 (see Table 1). In an 

effort to encourage web participation, email reminders were sent between March 19th and March 23rd; a second set 

of email reminders were sent between March 29 and April 4. To ensure that all sample members were contacted, 

USPS reminders were also utilized, with the first USPS reminders being sent between April 9th and April 19th; a 

second set of USPS reminders were sent between May 2nd and May 11th. Both sets of USPS reminders included the 

web address and login information to enable online participation; the instructions also included information on how 

to submit a paper survey (included in the MOPs). In the intervening time (between the first and second USPS 

reminders), a second set of MOPs were sent to encourage survey participation via paper-and-pencil. 
 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 1. 

Survey Administration Schedule 

Pre-notification email February 29-March 6 

Email announcement March 12 – March 16 

Email reminder March 19 – March 23 

Mail Out Packages (MOPs)1 

2

nd email reminder 

March 23 – April 4 

March 29 – April 4 

USPS reminder2 

2

nd Mail Out Packages (MOPs)1 

2

nd USPS reminder2 

April 9 – April 19 

April 18 – April 24 

May 2 – May 11 

End of survey administration May 24 

1 Included URL and password and a paper copy of the survey 

2 Included URL and password 

The data collection time periods for analysis were based on the survey schedule to most closely capture which event 

led to participation. Since the web survey was in the field two weeks longer than the paper survey, the web analysis 

was broken into three time periods instead of two as in the case of the paper survey (see Table 2). 

Web survey data were analyzed by the following time periods: 

ク Data collected during the first two weeks of the web survey administration (following the pre-notification 

email, email announcement, and 1st email reminder) 

ク Data collected during the first four weeks of the web survey administration (following the above 

notifications, and the 1st Mail Out Package (MOP) and 2nd email reminder) 

ク Data collected during the first seven weeks of the web survey administration (following the above 

notifications, and the 1st USPS reminder and 2nd MOPs) 

ク Data collected during the entire web administration (following the above notifications, and the 2nd USPS 

reminder) 

Paper survey data were analyzed by the following time periods: 

ク Data collected during the first two weeks of the paper survey administration (following the pre-notification 

email, email announcement, 1st email reminder, 1st Mail Out Package (MOP), and 2nd email reminder) 

ク Data collected during the first five weeks of the paper survey administration (following the above 

notifications, and the 1st USPS reminder and 2nd MOPs) 

ク Data collected during the entire paper administration (following the above notifications, and the 2nd USPS 

reminder) 
 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 2. 

Count of Eligible Surveys Received by Survey Type and Time Period 

Web Paper 

N % of web N % of paper Period of data collection Tools to obtain responses 

responses responses 

Email announcement 

0 to 2 weeks st 10,765 33% 

1 email reminder 

st

1 Mail Out Packages 2 to 4 weeks nd 11,039 34% 10,772 45% 

2 email reminder 

st

1 USPS reminder 4 to 7 weeks nd 7,657 23% 8,406 35% 

2 Mail Out Packages 

7 to 10 weeks nd

3,329 10% 4,725 20% 2 USPS reminder 

1 23,903 32,790 

1

58% responded using the web survey. 

Note: The email announcement with instructions on how to complete the web survey began on March 12, 2012; mailing of the first Mail 

Out Packages (MOPs) for the paper survey began on March 23, 2012. Thus, data were collected via web ~2 weeks longer than via paper. 

One-third of web respondents completed the questionnaire during the first two weeks of the web survey 

administration, during which a pre-notification, the email announcement, and an email reminder were sent. Twothirds (67%) of web respondents completed the questionnaire during the first four weeks of the web administration, 

which included the notifications sent during the first two weeks of the web administration and the first Mail Out 

Packages (MOPs) and 2nd email reminder. Another quarter of web respondents completed the survey during the next 

three weeks of the web administration, which included the 1st USPS reminder and the 2nd MOPs; the final 10% of 

web respondents completed the questionnaire during the last three weeks of the web administration, which included 

the 2nd and last USPS reminder. 

Almost half (45%) of paper respondents completed the questionnaire during the first two weeks of the paper survey 

becoming available, during which, in addition to electronic notifications (email announcement, 1st email reminder, 

and 2nd email reminder), the 1st MOPs were sent out. Four-fifths (80%) of paper respondents completed the 

questionnaire during the first five weeks of the paper survey becoming available which, in addition to the 

notifications above, included the 1st USPS reminder and the 2nd MOPs. The final 20% of paper respondents 

completed the survey during the last three weeks of the administration, which included the 2nd USPS reminder. 

The current study compares early responders with all respondents to determine whether additional efforts (e.g., 

email and USPS reminders) to increase response rates to a mixed-mode Army survey (web and paper) impacted 

survey estimates. Specifically, the current research investigates whether survey estimates would have been 

statistically different if the survey had been closed earlier with a lower response rate, and some of the notifications, 

such as the 2nd MOPs, which included a paper copy of the survey, and the two USPS reminders had not been sent. 

Results may be considered in the future to potentially lower the costs of future survey administration. 

Results are analyzed by the four patron groups of interest to the Army: active duty members, spouses of active duty 

members, retirees, and civilians. Twenty survey items were analyzed to test for differences in survey estimates based 

on the various data collection cut-off dates. 

Analysis 

The majority of results of the 2012 Army Morale, Welfare, and Recreation (MWR) Services survey were reported in 

terms of the percent satisfied or agreement with positive statements regarding MWR programs and services, by 

patron group (i.e., spouse, active duty, retiree, or civilian). The percent ‘satisfied’ or ‘very satisfied’ were combined, 
The Impact of Efforts to Increase Response Rates on Survey Estimates 

as were the percent who ‘agreed’ or ‘strongly agreed’ with statements. We selected a set of questions that capture 

questions that were asked of all respondents and questions that were only asked of a subset of respondents (e.g., 

some questions were only asked of those who indicated using a specific program or service). 

Figures 1 through 3 are provided to visually display changes in the survey estimates over the web survey 

administration period by patron group. Figure 1 presents the percent of web respondents who indicated they were 

satisfied with Army Family and MWR programs (item asked of all respondents) by patron group for web 

respondents as additional data were collected over the survey administration. If the web survey had been closed after 

just two weeks in the field, survey estimates (for percent satisfied) would have differed by up to two percentage 

points depending on patron group, whereas if the survey had closed at either the four or seven week mark, survey 

estimates would have differed by up to just one percentage point (none of these differences were statistically 

significant). Table 3 presents total differences for 20 measures analyzed for web respondents. Results by patron 

group are presented in Tables 4-7. 

Figure 1. 

Percent Satisfied With Army Family and MWR Programs at Installation by Patron Group – Web Respondents 

 

100 

95 

90 

85 

80 

75 

64 64 64 

63 

62 62 

63 63 

62 

63

62

61 

57 60 

58 

61 

58 

60 

59 

Spouse 

70 Active duty 

65 Retiree 

60 Civilian 

Total 55 

50 

Email 1st MOPs & 2nd 1st USPS reminder 2nd USPS 

announcement & email reminder & 2nd MOPs reminder 

1st email reminder 

0-2 weeks 0-4 weeks 0-7 weeks 0-10 weeks 

Figures 2 and 3 present the same information as Figure 1 for two additional survey items based on web respondents 

- satisfaction with Family and MWR Army entertainment (item asked of all respondents) and agreement that Army 

child, youth, and school age programs are a valuable benefit to them (item asked only of respondents who indicated 

they have children under 18 that live in their home or stay with them often). If the web survey had been closed after 

just two weeks in the field, survey estimates for the percent satisfied with Family and MWR Army entertainment 

would have differed by up to four percentage points depending on patron group (statistically significant difference 

for spouses, p<.05), whereas if the survey had closed at either the four or seven week mark, survey estimates would 

have differed by up to just one percentage point (none of these differences were statistically significant). Similarly, 

if the survey had been closed after two weeks in the field, survey estimates for the percent who agreed Army child, 

youth, and school age programs are a valuable benefit to them would have differed by up to five percentage points 

depending on patron group (statistically significant difference for spouses, p<.05), whereas if the survey had closed 

at the four week mark, survey estimates would have differed by up to just two percentage points; at the seven week 

mark, by up to just one percentage point. 
The Impact of Efforts to Increase Response Rates on Survey Estimates 

Figure 2. 

Percent Satisfied With Family and MWR Army Entertainment by Patron Group – Web Respondents 

 

 

80 

75 

70 

65 

60 

55 

46 

44 44 

46 46 46 

42 

43 44 

43 

44 

43

39 

37 

38 

37 38 

36 

Spouse 

50 Active duty 

45 Retiree 

Civilian 40 

Total 35 

30 

Email 1st MOPs & 1st USPS reminder 2nd USPS reminder 

announcement & 2nd email reminder & 2nd MOPs 

1st email reminder 

0-2 weeks 0-4 weeks 0-7 weeks 0-10 weeks 

Figure 3. 

Percent Agree Army Child, Youth, and School Age Programs are a Valuable Benefit to me by Patron Group – Web 

Respondents 

 

100 

95 

90 

85 

80 

75 73 

68 

64 65 

68 

63 

68 

60 62 

58 62 

64 

62 

58 57 

58 

57 

58 

52 55 

Spouse 

70 Active duty 

65 Retiree 

60 Civilian 

55 Total 

50 

Email 1st MOPs & 2nd 1st USPS reminder 2nd USPS 

announcement & email reminder & 2nd MOPs reminder 

1st email reminder 

0-2 weeks 0-4 weeks 0-7 weeks 0-10 weeks 

Similar to Figures 1-3, Figures 4-6 presents data for paper respondents for the same three survey items to visually 

display changes in the survey estimates over the paper survey administration period by patron group. Results 

indicate a more dramatic change in paper survey estimates over time than in the web survey estimates, with an 

implication that the paper survey would need to be in the field for at least five weeks before survey estimates 

stabilize. 
############################################################

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Figure 4 presents the percent of paper respondents who indicated they were satisfied with Army Family and MWR 

programs (item asked of all respondents) by patron group for paper respondents as additional data were collected 

over the survey administration. If the paper survey had been closed after just two weeks in the field, survey 

estimates (for percent satisfied) would have differed by up to five percentage points statistically significant 

difference for active duty members, p<.05) depending on patron group, whereas if the survey had closed at the five 

week mark, survey estimates would have been the same, regardless of patron group. Table 8 presents total 

differences for 20 measures analyzed for paper respondents. Results by patron group are presented in Tables 9-12. 

Figure 4. 

Percent Satisfied With Army Family and MWR Programs at Installation by Patron Group – Paper Respondents 

 

100 

95 

90 

85 

80 

Spouse 

75 

63 64 64 

62 62 

61 61 

58 

61 

58 

53 

55 55 

Active duty 

70 Retiree 

65 Civilian 

60 Total 

55 

50 

1st MOPs 1st USPS reminder 2nd USPS 

& 2nd email reminder & 2nd MOPs reminder 

0-2 weeks 0-5 weeks 0-8 weeks 

Figures 5 and 6 present the same information as Figure 4 for two additional survey items based on paper respondents 

- satisfaction with Family and MWR Army entertainment (item asked of all respondents) and agreement that Army 

child, youth, and school age programs are a valuable benefit to them (item asked only of respondents who indicated 

they have children under 18 that live in their home or stay with them often). If the paper survey had been closed 

after just two weeks in the field, survey estimates for the percent satisfied with Family and MWR Army 

entertainment would have differed by up to eight percentage points depending on patron group (statistically 

significant difference for civilians, p<.05), whereas if the survey had closed at the five week mark, survey estimates 

would have differed by up to just one percentage point (none of these differences were statistically significant).1 

 

Similarly, if the survey had been closed after two weeks in the field, survey estimates for the percent who agreed 

Army child, youth, and school age programs are a valuable benefit to them would have differed by up to twelve 

percentage points depending on patron group (statistically significant difference for active duty members and 

civilians, p<.05), whereas if the survey had closed at the five week mark, survey estimates would have differed by 

up to three percentage points (none of these differences were statistically significant). (See Figure 6) 

 

1

 Note that estimates presented in the figure have been rounded, thus some differences appear one percentage point more or less than the 

information reported in the text. 
 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Figure 5. 

Percent Satisfied With Family and MWR Army Entertainment by Patron Group – Paper Respondents 

 

80 

75 

70 

65 

60 

Spouse 

55 

Active duty 

50 

57 

50 49 

46 47 47 

44 

39 

45 

41 

45 

41 

38 38 

36 

Retiree 

45 Civilian 

40 Total 

35 

30 

1st MOPs 1st USPS reminder 2nd USPS 

& 2nd email reminder & 2nd MOPs reminder 

0-2 weeks 0-5 weeks 0-8 weeks 

Figure 6. 

Percent Agree Army Child, Youth, and School Age Programs are a Valuable Benefit to me by Patron Group – Paper 

Respondents 

 

100 

95 

90 

85 

80 

Spouse 

75 

Active duty 

70 

80 

77 

70 

68 

66 68 

67 

67 

65 64 

65 

62 

Retiree 

65 Civilian 

60 Total 

55 

50 

1st MOPs 1st USPS reminder 2nd USPS 

& 2nd email reminder & 2nd MOPs reminder 

0-2 weeks 0-5 weeks 0-8 weeks 
 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Findings and Conclusions 

Web 

Across the 20 measures analyzed, web survey estimates by patron group differed by 0 to 2 percentage points 

between estimates from data collected during the first four weeks of data collection and the full administration. 

However, these differences were not statistically significant (p<.05). Differences were between 0 and 1 percentage 

points between estimates from data collected during the first seven weeks of data collection and the full 

administration by patron group; none of these differences were statistically significant (p<.05). Differences were 

much larger, up to 13 percentage points, between estimates from data collected during the first two weeks of data 

collection and the full administration, several of these differences being statistically significant (p<.05) (see Tables 

3-7). Differences tended to be largest for active duty spouse and retiree respondents for the two week to full 

administration comparison. 

Web survey estimates stabilized after four weeks into the web administration and a 15% response rate. Thus, a web 

administration of this population that achieves a response rate of at least 15% is necessary (given the same sample 

design); fewer email and USPS reminders may be used to achieve this. Data collected beyond the first four weeks of 

the web survey administration had little to no impact on survey results (i.e., differences were not statistically 

significant), though the collection of additional data could be necessary for minimizing variance. 

Paper 

Across the 20 measures analyzed, paper survey estimates by patron group differed by 0 to 4 percentage points 

between estimates from data collected during the first five weeks of data collection and the full administration; five 

of the 20 estimates were statistically different (p<.05) (see Tables 8-12). Differences were much larger, up to 16 

percentage points, between estimates from data collected during the first two weeks of data collection and the full 

administration. Differences tended to be largest for active duty and civilian respondents for the two week to full 

administration comparison. 

Most of the paper survey estimates stabilized after five weeks into the paper administration and a 15% response rate. 

However, as discussed above, five of the 20 measures analyzed showed statistically significant differences between 

estimates obtained during the first five weeks of the paper administration and the full paper administration. Thus, 

evidence suggests that a paper administration of this population would need to achieve a higher response rate (higher 

than 15% given the same sample design) for survey estimates to stabilize. 

Limitations 

This study had more power than the typical survey, with a total sample of over 250,000 Army members. The 

findings of this study may differ with smaller sample sizes. Even after just two weeks into the administration of each 

mode, approximately 22,000 respondents had submitted a survey in one of the two modes (i.e., about 11,000 

respondents in each mode). A survey with a smaller sample would have less power, thus potentially suffering from 

larger margins of error with lower response rates. Smaller sample sizes can also yield more unstable results. In 

addition, respondents could choose between modes in the current study, thus there was no way of knowing what 

motivated their participation (e.g., a mail out package with a paper copy of the survey may have reminded a 

participant to fill out the survey online). 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 3. 

Differences in Web Survey Estimates by Survey Close Date – Total Army 

Estimate - Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 4 0 to 7 0 to 10 

Responding weeks ME weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

99% 62.2 0.9 62.1 0.6 62.2 0.6 62.2 0.5 installation 

Rated current installation’s Army Family and MWR 

83% 42.0 1.0 43.1 0.7 43.5 0.6 43.4 0.6 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 79% 44.0 1.0 43.1 0.7 42.8 0.6 42.9 0.6 

Agreement that Army child, youth, and school age 

27% 60.3 1.8 61.6 1.3 62.1 1.1 62.4 1.0 programs are a valuable benefit 

Use Army Child Care services 39% 20.0 1.2 20.4 0.9 21.1 0.8 21.2 0.7 

Agree that participating in MWR programs and 

85% 71.9 0.9 70.9 0.7 70.7 0.6 70.8 0.5 services provides a healthy way to relieve stress 

Satisfied with military life 30% 81.1 1.5 80.9 1.0 80.5 0.8 80.2 0.8 

Plan to stay in the Army beyond my present 

15% 88.3 1.4 87.3 1.2 86.7 1.0 86.2 1.0 obligation or longer 

Increase in use of Family and MWR programs and 

18% 46.7 2.6 42.2 1.7 42.3 1.4 42.4 1.3 services during a deployment 

Used on post outdoor recreation center in the past 

97% 39.6 0.9 36.9 0.6 36.4 0.6 36.7 0.5 12 months 

Participated in activities at the on post bowling 

96% 37.9 0.9 35.4 0.6 35.1 0.6 35.3 0.5 center in the past 12 months 

Interested in golf 96% 38.9 0.9 38.9 0.7 38.9 0.6 38.9 0.5 

Interested in weight/strength training 95% 72.2 0.9 70.0 0.6 70.1 0.5 70.6 0.5 

Interested in cardiovascular training 96% 83.5 0.7 82.1 0.5 82.2 0.4 82.4 0.4 

Satisfied with on post fitness facilities 56% 87.5 0.8 88.3 0.6 88.6 0.5 88.6 0.5 

Interested in going to the movies 95% 78.4 0.8 78.0 0.6 78.4 0.5 78.9 0.5 

Satisfied with the on post Leisure Travel/Ticket 

29% 81.1 1.3 81.7 1.0 82.0 0.8 81.6 0.8 Office 

Children interested in bowling 38% 64.5 1.5 64.5 1.0 64.4 0.9 64.3 0.8 

Satisfied with food and beverage service 39% 50.6 1.5 50.2 1.1 50.3 0.9 50.4 0.9 

Eat out for dinner off post at least once per month 93% 72.7 0.9 73.3 0.6 72.8 0.5 72.8 0.5 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full web survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st four weeks of data collection and full administration 

3

 Difference in survey estimates between 1st seven weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 4. 

Differences in Web Survey Estimates by Survey Close Date – Active Duty Army Spouses 

Estimate - Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 4 0 to 7 0 to 10 

Responding weeks ME weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

99% 61 5.2 60 2.0 61 1.5 60 1.4 installation 

Rated current installation’s Army Family and MWR 

83% 33 5.2 36 2.1 37 1.6 37 1.5 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 82% 42 5.6 39 2.2 38 1.7 38 1.5 

Agreement that Army child, youth, and school age 

52% 73 6.4 68 2.6 68 2.0 68 1.8 programs are a valuable benefit 

Use Army Child Care services 67% 45 6.6 32 2.3 33 1.8 33 1.6 

Agree that participating in MWR programs and 

90% 71 5.1 67 2.0 67 1.5 67 1.4 services provides a healthy way to relieve stress 

Satisfied with military life 99% 81 4.1 80 1.6 80 1.2 80 1.2 

Plan to stay in the Army beyond my present 0% N/A N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 

66% 30 5.8 32 2.3 34 1.8 34 1.7 services during a deployment 

Used on post outdoor recreation center in the past 

98% 48 5.3 39 2.0 40 1.5 40 1.4 12 months 

Participated in activities at the on post bowling 

97% 61 5.3 50 2.0 51 1.6 50 1.5 center in the past 12 months 

Interested in golf 97% 36 5.2 30 1.9 30 1.4 30 1.3 

Interested in weight/strength training 97% 82 4.2 77 1.7 77 1.3 78 1.2 

Interested in cardiovascular training 97% 93 2.8 90 1.2 91 0.9 91 0.8 

Satisfied with on post fitness facilities 62% 88 4.1 88 1.6 89 1.2 89 1.2 

Interested in going to the movies 97% 91 3.1 92 1.1 92 0.9 92 0.8 

Satisfied with the on post Leisure Travel/Ticket 

32% 70 7.3 78 2.9 78 2.3 77 2.1 Office 

Children interested in bowling 66% 69 6.3 63 2.4 64 1.8 65 1.7 

Satisfied with food and beverage service 32% 45 8.0 47 3.5 46 2.7 47 2.5 

Eat out for dinner off post at least once per month 95% 83 4.2 79 1.7 78 1.3 78 1.2 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full web survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st four weeks of data collection and full administration 

3

 Difference in survey estimates between 1st seven weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 5. 

Differences in Web Survey Estimates by Survey Close Date – Active Duty Army 

Estimate - Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 4 0 to 7 0 to 10 

Responding weeks ME weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

99% 57 2.1 58 1.7 58 1.4 59 1.3 installation 

Rated current installation’s Army Family and MWR 

92% 36 2.1 36 1.7 36 1.5 37 1.4 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 80% 36 2.3 37 1.8 37 1.6 38 1.5 

Agreement that Army child, youth, and school age 

43% 64 3.1 65 2.4 63 2.1 64 2.0 programs are a valuable benefit 

Use Army Child Care services 53% 26 2.5 28 2.0 28 1.8 28 1.7 

Agree that participating in MWR programs and 

90% 76 1.9 78 1.5 76 1.3 76 1.2 services provides a healthy way to relieve stress 

Satisfied with military life 99% 81 1.7 81 1.3 81 1.2 80 1.1 

Plan to stay in the Army beyond my present 

92% 88 1.4 87 1.2 87 1.0 86 1.0 obligation or longer 

Increase in use of Family and MWR programs and 

54% 50 2.9 51 2.3 52 2.0 52 1.8 services during a deployment 

Used on post outdoor recreation center in the past 

97% 47 2.2 46 1.7 45 1.5 45 1.4 12 months 

Participated in activities at the on post bowling 

94% 53 2.2 53 1.7 51 1.5 51 1.4 center in the past 12 months 

Interested in golf 95% 45 2.2 45 1.7 45 1.5 45 1.4 

Interested in weight/strength training 95% 85 1.5 86 1.2 85 1.1 85 1.0 

Interested in cardiovascular training 95% 90 1.3 90 1.0 90 0.9 90 0.8 

Satisfied with on post fitness facilities 78% 88 1.6 88 1.2 88 1.1 88 1.0 

Interested in going to the movies 94% 85 1.6 86 1.2 86 1.0 86 0.9 

Satisfied with the on post Leisure Travel/Ticket 

29% 79 3.2 79 2.5 79 2.2 79 2.0 Office 

Children interested in bowling 53% 65 2.7 66 2.2 65 1.9 65 1.8 

Satisfied with food and beverage service 41% 51 3.4 50 2.7 50 2.3 51 2.1 

Eat out for dinner off post at least once per month 91% 76 1.9 75 1.5 74 1.3 74 1.2 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full web survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st four weeks of data collection and full administration 

3

 Difference in survey estimates between 1st seven weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 6. 

Differences in Web Survey Estimates by Survey Close Date – Army Retirees 

Estimate - Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 4 0 to 7 0 to 10 

Responding weeks ME weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

99% 63 2.1 63 1.1 64 1.0 64 0.9 installation 

Rated current installation’s Army Family and MWR 

89% 41 2.2 47 1.3 48 1.1 48 1.0 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 79% 46 2.4 44 1.3 44 1.1 44 1.1 

Agreement that Army child, youth, and school age 

13% 52 5.3 55 3.3 57 2.8 57 2.6 programs are a valuable benefit 

Use Army Child Care services 24% 9 2.2 7 1.2 7 1.1 7 1.0 

Agree that participating in MWR programs and 

82% 73 2.0 70 1.2 71 1.0 71 1.0 services provides a healthy way to relieve stress 

Satisfied with military life 0% N/A N/A N/A N/A 

Plan to stay in the Army beyond my present 0% N/A N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 0% N/A N/A N/A N/A services during a deployment 

Used on post outdoor recreation center in the past 

98% 43 2.2 33 1.1 32 0.9 33 0.9 12 months 

Participated in activities at the on post bowling 

97% 41 2.2 28 1.1 27 0.9 28 0.9 center in the past 12 months 

Interested in golf 97% 43 2.2 43 1.2 43 1.0 43 1.0 

Interested in weight/strength training 96% 69 2.1 62 1.2 62 1.0 62 1.0 

Interested in cardiovascular training 97% 82 1.7 76 1.0 77 0.9 77 0.8 

Satisfied with on post fitness facilities 47% 86 2.0 89 1.1 90 0.9 90 0.8 

Interested in going to the movies 97% 76 1.9 72 1.1 72 0.9 72 0.9 

Satisfied with the on post Leisure Travel/Ticket 

27% 81 2.8 83 1.7 84 1.4 84 1.4 Office 

Children interested in bowling 23% 65 3.9 65 2.4 65 2.0 64 1.9 

Satisfied with food and beverage service 35% 48 3.4 49 2.0 50 1.7 49 1.6 

Eat out for dinner off post at least once per month 95% 74 2.0 74 1.1 73 0.9 73 0.9 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full web survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st four weeks of data collection and full administration 

3

 Difference in survey estimates between 1st seven weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 7. 

Differences in Web Survey Estimates by Survey Close Date – Army Civilians 

Estimate - Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 4 0 to 7 0 to 10 

Responding weeks ME weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

98% 64 1.2 63 1.0 63 0.9 63 0.9 installation 

Rated current installation’s Army Family and MWR 

73% 46 1.4 45 1.2 45 1.1 45 1.0 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 77% 46 1.4 46 1.2 46 1.0 46 1.0 

Agreement that Army child, youth, and school age 

22% 58 2.6 58 2.2 58 2.0 58 1.9 programs are a valuable benefit 

Use Army Child Care services 35% 17 1.6 17 1.3 17 1.2 16 1.1 

Agree that participating in MWR programs and 

83% 70 1.2 70 1.0 69 0.9 69 0.9 services provides a healthy way to relieve stress 

Satisfied with military life 0% N/A N/A N/A N/A 

Plan to stay in the Army beyond my present 0% N/A N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 0% N/A N/A N/A N/A services during a deployment 

Used on post outdoor recreation center in the past 

97% 36 1.2 36 1.0 35 0.9 35 0.9 12 months 

Participated in activities at the on post bowling 

95% 31 1.2 31 1.0 30 0.9 29 0.8 center in the past 12 months 

Interested in golf 95% 36 1.2 36 1.0 36 0.9 36 0.9 

Interested in weight/strength training 94% 68 1.2 68 1.0 68 0.9 69 0.8 

Interested in cardiovascular training 95% 81 1.0 81 0.8 81 0.7 81 0.7 

Satisfied with on post fitness facilities 52% 88 1.1 88 0.9 88 0.8 88 0.8 

Interested in going to the movies 94% 76 1.1 76 0.9 76 0.8 76 0.8 

Satisfied with the on post Leisure Travel/Ticket 

28% 83 1.7 83 1.4 83 1.3 83 1.2 Office 

Children interested in bowling 34% 64 2.1 64 1.7 64 1.5 63 1.4 

Satisfied with food and beverage service 43% 51 1.9 52 1.5 52 1.4 52 1.3 

Eat out for dinner off post at least once per month 91% 71 1.2 71 1.0 70 0.9 70 0.8 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full web survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st four weeks of data collection and full administration 

3

 Difference in survey estimates between 1st seven weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 8. 

Differences in Paper Survey Estimates by Survey Close Date – Total Army 

Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 5 0 to 8 

Responding weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

96% 61 0.9 62 0.7 62 0.6 installation 

Rated current installation’s Army Family and MWR 

76% 51 1.1 50 0.8 49 0.7 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 70% 44 1.1 45 0.8 45 0.8 

Agreement that Army child, youth, and school age 

18% 65 2.4 67 1.6 67 1.4 programs are a valuable benefit 

Use Army Child Care services 30% 17 1.4 18 1.0 19 0.9 

Agree that participating in MWR programs and 

72% 71 1.0 72 0.8 72 0.7 services provides a healthy way to relieve stress 

Satisfied with military life 22% 75 2.0 76 1.3 76 1.2 

Plan to stay in the Army beyond my present 

5% 87 7.3 82 3.5 82 2.3 obligation or longer 

Increase in use of Family and MWR programs and 

14% 36 2.7 37 1.9 38 1.6 services during a deployment 

Used on post outdoor recreation center in the past 

97% 19 0.7 21 0.6 22 0.5 12 months 

Participated in activities at the on post bowling 

97% 18 0.7 21 0.6 22 0.5 center in the past 12 months 

Interested in golf 95% 34 0.9 35 0.7 35 0.6 

Interested in weight/strength training 94% 53 1.0 57 0.7 59 0.6 

Interested in cardiovascular training 96% 69 0.9 72 0.6 74 0.6 

Satisfied with on post fitness facilities 38% 93 0.9 92 0.6 92 0.6 

Interested in going to the movies 95% 70 0.9 72 0.7 74 0.6 

Satisfied with the on post Leisure Travel/Ticket 0% N/A N/A N/A Office 

Children interested in bowling 29% 59 1.9 60 1.3 60 1.2 

Satisfied with food and beverage service 32% 57 1.7 58 1.3 57 1.1 

Eat out for dinner off post at least once per month 92% 68 0.9 68 0.7 68 0.6 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full paper survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st five weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 9. 

Differences in Paper Survey Estimates by Survey Close Date –Active Duty Army Spouses 

Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 5 0 to 8 

Responding weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

99% 53 2.2 55 1.6 55 1.4 installation 

Rated current installation’s Army Family and MWR 

78% 32 2.3 33 1.7 33 1.5 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 82% 36 2.3 38 1.7 38 1.5 

Agreement that Army child, youth, and school age 

53% 66 2.9 67 2.0 68 1.8 programs are a valuable benefit 

Use Army Child Care services 74% 24 2.1 26 1.6 27 1.5 

Agree that participating in MWR programs and 

88% 64 2.2 66 1.6 67 1.4 services provides a healthy way to relieve stress 

Satisfied with military life 86% 74 2.0 75 1.4 75 1.3 

Plan to stay in the Army beyond my present 0% N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 

57% 37 2.8 36 2.0 37 1.8 services during a deployment 

Used on post outdoor recreation center in the past 

99% 29 2.0 31 1.4 32 1.3 12 months 

Participated in activities at the on post bowling 

99% 42 2.2 45 1.6 46 1.4 center in the past 12 months 

Interested in golf 98% 29 2.0 30 1.4 30 1.3 

Interested in weight/strength training 99% 74 1.9 75 1.4 75 1.2 

Interested in cardiovascular training 99% 89 1.4 90 0.9 90 0.9 

Satisfied with on post fitness facilities 58% 90 1.8 90 1.2 91 1.1 

Interested in going to the movies 99% 92 1.2 93 0.8 92 0.8 

Satisfied with the on post Leisure Travel/Ticket 0% N/A N/A N/A Office 

Children interested in bowling 72% 58 2.5 59 1.8 59 1.7 

Satisfied with food and beverage service 33% 42 3.8 46 2.7 45 2.5 

Eat out for dinner off post at least once per month 97% 80 1.8 78 1.3 78 1.2 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full paper survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st five weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 10. 

Differences in Paper Survey Estimates by Survey Close Date – Active Duty Army 

Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 5 0 to 8 

Responding weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

98% 63 9.4 58 4.2 58 2.7 installation 

Rated current installation’s Army Family and MWR 

83% 39 10.2 36 4.5 36 2.9 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 82% 41 10.4 39 4.6 41 2.9 

Agreement that Army child, youth, and school age 

44% 77 12.0 70 5.7 67 3.9 programs are a valuable benefit 

Use Army Child Care services 59% 45 12.2 29 4.8 28 3.2 

Agree that participating in MWR programs and 

92% 79 8.3 75 3.9 73 2.5 services provides a healthy way to relieve stress 

Satisfied with military life 94% 85 7.5 78 3.7 77 2.4 

Plan to stay in the Army beyond my present 

87% 87 7.3 82 3.5 82 2.3 obligation or longer 

Increase in use of Family and MWR programs and 

51% 33 12.9 43 5.7 42 3.8 services during a deployment 

Used on post outdoor recreation center in the past 

99% 32 9.2 31 4.0 34 2.6 12 months 

Participated in activities at the on post bowling 

99% 31 9.2 38 4.2 40 2.7 center in the past 12 months 

Interested in golf 97% 43 10.0 43 4.3 43 2.7 

Interested in weight/strength training 99% 70 9.1 78 3.6 82 2.1 

Interested in cardiovascular training 99% 79 8.0 88 2.8 88 1.8 

Satisfied with on post fitness facilities 77% 95 5.2 92 2.7 90 1.9 

Interested in going to the movies 98% 85 7.2 87 2.9 88 1.8 

Satisfied with the on post Leisure Travel/Ticket 0% N/A N/A N/A Office 

Children interested in bowling 57% 54 12.2 55 5.3 57 3.6 

Satisfied with food and beverage service 44% 68 14.4 51 6.6 53 4.1 

Eat out for dinner off post at least once per month 96% 72 9.1 76 3.7 76 2.4 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full paper survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st five weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 11. 

Differences in Paper Survey Estimates by Survey Close Date – Army Retirees 

Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 5 0 to 8 

Responding weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

94% 63 1.1 64 0.8 64 0.8 installation 

Rated current installation’s Army Family and MWR 

78% 56 1.2 56 0.9 55 0.9 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 65% 46 1.3 47 1.0 47 1.0 

Agreement that Army child, youth, and school age 

6% 62 4.6 64 3.4 65 3.1 programs are a valuable benefit 

Use Army Child Care services 14% 4 1.2 5 1.1 6 1.0 

Agree that participating in MWR programs and 

65% 73 1.2 74 0.9 74 0.9 services provides a healthy way to relieve stress 

Satisfied with military life 0% N/A N/A N/A 

Plan to stay in the Army beyond my present 0% N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 0% N/A N/A N/A services during a deployment 

Used on post outdoor recreation center in the past 

96% 16 0.8 17 0.6 18 0.6 12 months 

Participated in activities at the on post bowling 

96% 12 0.7 13 0.6 14 0.6 center in the past 12 months 

Interested in golf 94% 35 1.0 36 0.8 36 0.8 

Interested in weight/strength training 92% 47 1.1 50 0.9 51 0.8 

Interested in cardiovascular training 95% 64 1.0 66 0.8 67 0.8 

Satisfied with on post fitness facilities 28% 94 1.1 93 0.8 93 0.8 

Interested in going to the movies 93% 64 1.1 65 0.8 66 0.8 

Satisfied with the on post Leisure Travel/Ticket 0% N/A N/A N/A Office 

Children interested in bowling 13% 62 3.1 64 2.3 63 2.2 

Satisfied with food and beverage service 29% 61 2.0 62 1.5 62 1.4 

Eat out for dinner off post at least once per month 90% 65 1.1 64 0.8 64 0.8 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full paper survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st five weeks of data collection and full administration 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Table 12. 

Differences in Paper Survey Estimates by Survey Close Date – Army Civilians 

Estimate - Estimate - Estimate -

Percent 0 to 2 0 to 5 0 to 8 

Responding weeks ME weeks ME weeks ME 

Satisfied with Army Family and MWR programs at 

97% 61 9.5 61 2.9 61 1.8 installation 

Rated current installation’s Army Family and MWR 

64% 44 10.1 45 3.6 46 2.3 programs higher compared to other places stationed 

Satisfied with Family and MWR Army Entertainment 73% 57 10.9 50 3.4 49 2.1 

Agreement that Army child, youth, and school age 

15% 80 21.0 68 7.3 68 4.4 programs are a valuable benefit 

Use Army Child Care services 30% 19 14.1 10 3.3 13 2.2 

Agree that participating in MWR programs and 

76% 72 9.6 72 3.1 71 1.9 services provides a healthy way to relieve stress 

Satisfied with military life 0% N/A N/A N/A 

Plan to stay in the Army beyond my present 0% N/A N/A N/A obligation or longer 

Increase in use of Family and MWR programs and 0% N/A N/A N/A services during a deployment 

Used on post outdoor recreation center in the past 

98% 20 7.8 23 2.5 23 1.6 12 months 

Participated in activities at the on post bowling 

97% 21 7.9 20 2.4 19 1.4 center in the past 12 months 

Interested in golf 95% 38 9.6 35 2.9 35 1.8 

Interested in weight/strength training 96% 66 9.4 63 2.9 63 1.8 

Interested in cardiovascular training 97% 78 8.0 75 2.6 75 1.6 

Satisfied with on post fitness facilities 43% 92 7.6 92 2.4 90 1.6 

Interested in going to the movies 95% 70 9.0 74 2.6 75 1.6 

Satisfied with the on post Leisure Travel/Ticket 0% N/A N/A N/A Office 

Children interested in bowling 29% 47 18.2 58 5.5 60 3.3 

Satisfied with food and beverage service 41% 74 13.2 61 4.6 60 2.8 

Eat out for dinner off post at least once per month 93% 66 9.6 62 3.0 64 1.8 

Note: Some item response rates are low due to skip patterns (i.e., not all respondents eligible for the survey item). 

 Bolded estimates indicate a statistically significant difference from the estimate obtained from the full paper survey administration at p<.05. 

1

 Difference in survey estimates between 1st two weeks of data collection and full administration 

2

 Difference in survey estimates between 1st five weeks of data collection and full administration 
 

 

 

 

 

 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

References 

Beebe, T. J., McAlpine, D. D., Ziegenfuss, J. Y., Jenkins, S., Haas, L., and Davern, M. E. (2012). Deployment of a 

Mixed-Mode Data Collection Strategy Does Not Reduce Nonresponse Bias in a General Population Health 

Survey. Health Services Research, 47(4), 1739-1754. 

Borkan, B. (2010). The Mode Effect in Mixed-Mode Surveys: Mail and Web Surveys. Social Science Computer 

Review, 28(3), 371-380. 

Groves, R. M. (2006). Nonresponse rates and nonresponse bias in household surveys. Public Opinion Quarterly, 

70(5), 646-675. 

Groves, R. M. and Peytcheva, E. (2008). The Impact of Nonresponse Rates on Nonresponse Bias: A Meta-Analysis. 

Public Opinion Quarterly, 72(2), 167-189. 

Groves, R. M., Couper, M. P., Presser, S., Singer, E., Tourangeau, R., Acosta, G. P., and Nelson, L. (2006). 

Experiments in Producing Nonresponse Bias. Public Opinion Quarterly, 70(5), 720-736. 

Irani, T., A., Gregg, J. A., and Telg, R. (2004). Choosing to Use the Web: Comparing Early and Late Respondents to 

an Online Web-based Survey Designed to Assess IT Computer Skills Perceptions of County Extension 

Agents. Journal of Southern Agricultural Education Research 54(1), 168-179. 

Keeter, S., Kennedy, C., Dimock, M., Best, J., and Craighill, P. (2006). Gauging the Impact of Growing 

Nonresponse on Estimates from a National RDD Telephone Survey. Public Opinion Quarterly, 70(5), 759-779. 

Kohut, A. Keeter, S., Doherty, C. Dimock, M., and Christian, L. (2012, May 15). Assessing the Representativeness 

of Public Opinion Surveys. The Pew Research Center For The People & The Press. Retrieved February 25, 2013 

from http://www.people-press.org/files/legacypdf/Assessing%20the%20Representativeness%20of%20Public%20Opinion%20Surveys.pdf. 

Manfreda, K. L., Bosnjak, M., Berzelak, J., Haas, I., and Vehovar, V. (2008). Web surveys versus other survey 

modes: A meta-analysis comparing response rates. International Journal of Market Research, 50(1), 79-104. 

Millar, M. M. & Dillman, D. A. (2011). Improving Response to Web and Mixed-Mode Surveys. Public Opinion 

Quarterly, 75(2), 249-269. 
 

 

The Impact of Efforts to Increase Response Rates on Survey Estimates 

Paganini-Hill, A., Hsu, G., Chao, A., & Ross, R. K. (1993). Comparison of Early and Late Respondents to a Postal 

Health Survey Questionnaire. Epidemiology, 3, 375-379. 

Shin, E., Johnson, T. P., and Rao, K. (2012). Survey Mode Effects on Data Quality: Comparison of Web and Mail 

Modes in a U.S. National Panel Survey. Social Science Computer Review, 30(2), 212-228. 

VanNieuwenhuyze, J., Loosveldt, G., and Molenberghs, G. (2011). A Method for Evaluation Mode Effects in 

Mixed-Mode Surveys. Public Opinion Quarterly, 74(5), 1027-1045.