E-mail and Web-Based Survey Response Rates:
Considerations for Disaster, First Responder and Military Populations
I. General Response Rate Issues
Many issues impact survey response rates. Broadly, these include availability at time of receiptof the survey, inclusion criteria, ability to complete the survey, and willingness to complete thesurvey. While it is important to account for all of these issues, response rates are typicallycalculated by simply dividing the number of surveys received by the number of surveys
distributed. This approach tacitly assumes that all persons receiving the survey met inclusion
criteria, that all surveys reached their destination, and that all potential participants were able tocomplete the survey.
II. Postal Mail Response Rate
Let’s consider each of the issues mentioned above for a typical postal mail survey.
A. Inclusion Criteria: Depending on the nature of the survey, all individuals on amailing list from a group or organization may not meet inclusion criteria based on theparticular aims of the study. As such, a list must be created which is subject toinaccuracies by identifying some as meeting inclusion criteria who ought not have been
included, and excluding some who would have met criteria. Therefore, one is often
mailing surveys to individuals who will not meet the inclusion criteria. Also, anindividual’s status regarding inclusion criteria can change between the time of selection
to participate in the survey and the time of survey administration. As an example, asurvey intended to be completed only by “registered students” may reach some students
who recently dropped out, transferred, or graduated. Such students would no longer meetthe inclusion criteria to complete the survey. Further, potential participants may not be
inclined to state why they fail to meet inclusion criteria on the survey,
B. Accessibility: Accessibility refers to whether the person is available at the time ofsurvey administration. A person’s accessibility may change during the course of thesurvey. For example, when sampling from a military population, one may select asample to survey at a given installation with “presence at the installation on a particulardate when an event occurred” as being a criterion for inclusion. In the ensuing months
between the event and survey distribution, military personnel may be deployed ortransferred, temporarily relocated for TDY (temporary duty), or even leave the military.
Even though all of the persons meet the inclusion criteria, since these persons are notpresent at the installation to receive a survey, they would no longer be accessible. As
another example, individuals who are on a long vacation or business trip or an extended
hospital stay due to illness or rehabilitation may not be accessible at the time of thesurvey.
C. Receipt of Survey: Did the survey reach the intended participant? This is easily
discerned when a survey is returned to the sender. However, not all undeliverable mail isreturned to the sender. Further, if a family member still lives at the residence, it is likely

that the mail will be delivered to that residence even if the intended recipient no longer
resides there.
D. Ability to Complete the Survey: A number of issues could impact this factor. For
example, the ability to read and write is typically required for most surveys. If therecipient does not speak the same language used in the survey, then he or she might beunable to complete the survey. Suffering from serious illness can impede one’s effort tocomplete a survey. Clearly, a person in a coma would be unable to complete the survey.
For work-related surveys, the amount of time the recipient is available at work can be afactor, especially if his or her job entails extensive travel or off-site field work. In
addition, whether or not time is provided during work to complete the survey could be afactor in the individual’s ability to complete the survey.
E. Willingness to Complete the Survey: While this is the most straightforward category
in terms of definition and classification, there are still issues worth consideration. We
may wish to differentiate “true willingness” from issues such as survey overload andburnout.
III. E-mail and Web-based Response Rate Issues
E-mail and web-based surveys tend to yield lower response rates than postal mail surveys. In
addition to the items that apply to mail, those items could also pertain to email and web surveys
We reconsider the same four issues, now from the perspective of e-mail and web-based surveys,
to better understand this trend.
A. Inclusion Criteria: The issues surrounding inclusion may be very similar to thosearising with postal surveys. However, what can be different is that it might be even moredifficult for a participant to explain the nature of his or her exclusion criteria if an e-mailor web-based survey does not have an allotted space on the electronically-completed form
(e.g., only checkboxes) for such explanation. Email recipients are often likely to deletean e-mail rather than reply and provide information as to why they do not meet inclusion
criteria.
B. Accessibility: The issues for accessibility are essentially the same as those arisingwith postal surveys.
C. Receipt of Survey: This issue constitutes the greatest source of deviation from thepostal mail survey. E-mail addresses are likely to change more often than postal mailaddresses. Since e-mail addresses are usually individually-based, whereas postal mailaddresses may be occupied by more than one person, this raises additional issues related
to receipt. Due to the increased use of anti-virus and anti-spam protection software andthe use of e-mail filters, the number of undeliverable e-mails continues to increase.
Recent literature indicates that between 5.3% and 25.5% of all e-mail surveys are
undeliverable. Other examples of undeliverable surveys could be due to a person’s e-mailinbox may be full. The e-mail service may not be functional for the duration of thesurvey. Even if a survey actually reaches the intended person’s inbox, the recipient may
not open the e-mail, due to concerns about spam or viruses, not having available time tocomplete the survey, not recognizing the sender, or issues of survey overload and burnout.
As such, the person may delete the e-mail without ever opening it.
Draft
9/23/2021

Page 2

D. Ability to Complete the Survey: The ability to complete an e-mail or web-based
survey requires access to the Internet. This is referred to as “noncoverage.” For example,
older persons and persons with a lower socio-economic status may have limited access tothe Internet. Persons with non-office-based jobs may have limited access to their workprovided Internet service and e-mail, and at times, such access may even be blocked.
This is also an issue when surveying persons in countries without easy access to theInternet.
E. Willingness to Complete the Survey: This is the most straightforward category. The
issues remain similar to those arising in postal surveys.
IV. Military, Disaster, and Public Health Responder Population
Military, disaster and public health responder populations are particularly challenging withrespect to response rates for web-based and e-mail surveys. The nature of the pace and duties ofthis population would have significant impacts. For example, it is very common that persons aredeployed, on TDY at a different location, detailed on a specific duty, in addition to usual illness
or leave. As such, they are not available to receive the survey and may no longer be eligible toparticipate (for example, if the survey was to be administered pre-deployment and the person isnow deployed). While all organizations experience some level of turnover, the rates may bemuch higher in these populations, particularly in the military. For example, approximately onethird of the Army leaves the Army every year. Also, this population, due to training, may be
extremely wary of spam and viruses and therefore may be less likely to open an “unofficial” email.
Military, disaster and public health responder populations’ ability to complete a survey may also
be significantly hindered by being at a location or on an assignment that does not allow for email/Internet access. Further, their personal time may be quite limited. Depending on theircurrent assignment they may not have sufficient time to complete a survey. In addition, these
populations may have limited personnel, time and resources to cover the duties of individuals
attempting to take the survey.
An advantage in surveys with these populations is that their willingness to complete the survey
may actually be higher than in other groups due to their commitment to the populations theyserve.
V. Calculating Response and Other Rates
According to the American Association of Public Opinion Research (AAPOR), standard
response rates include the number of surveys completed (partially completed surveys may beincluded) divided by the number of surveys completed + the number refusing to complete thesurvey + the number of non-contacts + the number of persons of unknown eligibility. If an
estimate of the percentage of unknown eligibility cases that are eligible is known, this may beused instead.
According to the AAPOR, a cooperation rate includes the number of surveys completed(partially completed surveys may be included) divided by the number of surveys completed + thenumber refusing to complete the survey even though eligible + the number of other eligible
participants who did not complete the survey. In a modified version of this rate, the AAPOR
removes those unable to complete an interview from the denominator.
Draft
9/23/2021

Page 3

In the Department of Defense Survey of Health Related Behaviors Among Military Personnel(HRB), an accessibility rate is used. The accessibility rate is the percentage of persons selectedthat were still available to be surveyed at the time of the survey. The response rate is calculated
as the number of persons completing the survey divided by the number of accessible persons at
the time of the survey.
The cooperation rate, as well as a response rate that accounts for accessibility, may offer amethod for calculating a performance rate for e-mail and web-based surveys. This would allow
for the calculation of a performance rate that captures those who completed the survey amongst
those that were eligible, accessible, received the survey, had the ability to complete it, andconsented to do so.
VI. Conclusion
Numerous elements affect the meaning and calculation of response rates, particularly in military,
disaster and first responder populations. Only when the response rate is quite high can it be anindicator of representativeness, but even then it is only one index for assessing representativeness.
Response rates represent a group of concepts ranging from accessibility rates to completion rates.
Better describing how a calculation is made and what it represents will facilitate its
understanding and use, particularly among mail and e-mail surveys.
In order to best describe a response rate, a number of persons may need to be removed from thedenominator. Unfortunately, it may not always be feasible for researchers to obtain exact
numbers of persons who were unavailable, ineligible, or unable to complete a survey.
Researchers may need to estimate some of these numbers based on the available empiricalliterature or through the use of key informant surveys.

Draft
9/23/2021

Page 4

Table 1: Items to Remove from Denominator for E-mail / Web-based Survey
Performance Rates
A.

Inclusion Criteria
1.
Person meeting inclusion is excluded from list
2.
Person not meeting criteria is included on list
3.
4.
5.

Person’s status regarding inclusion criteria changes between time of selection toparticipate and time of survey administration
Person reticent to respond to survey to explain why he or she is ineligible
Person cannot explain why he or she is ineligible due to limitation of electronic form.

B.

Accessibility
1.
Extended absence from work (deployed or transferred, temporarily relocated, oreven leave the job/military).
2.
Extended absence from home (long vacation or business trip).
3.
IllnessC.

Receipt of the Survey
1.
An undeliverable survey is returned to sender2.
Survey arrives at the correct address, but it is no longer the intended recipient’spermanent residence.
3.
E-mail survey is undeliverable due to an error in the e-mail address.
4.
Anti-virus and anti-spam software prevents delivery of e-mail survey
5.
Recipient’s e-mail inbox is full6.
E-mail server is not functional for duration of survey
7.
Recipient deletes the e-mail survey without opening it.

D.

Ability to Complete the Survey
1.
Recipient does not understand the language in which the survey is written.
2.
Serious illness impedes a recipient’s ability to complete the survey.
3.
Amount of time the recipient has to complete the survey is severely limited.
4.
Recipient has limited access to the Internet.

Draft
9/23/2021

Page 5

Table 2: Operational Calculation of Postal/E-mail / Web-based Survey Performance Rates
Postal Mail
Total Number of Surveys Distributed
A. Inclusion Criteria
1.

2.

Subtract

Subtracta1

a2

Subtract

Subtractb1

b2

Subtract

Subtractc1

c2

Subtract

Subtract

d1

d2

NPOSTAL-final

NEMAIL-final

Does not meet inclusion criteria
No longer meets inclusion criteria due to criterion changes

Subtotal for Eligibility Criterion
B.

NPOSTAL-initial

E-mail andWeb-Based
NEMAIL-initial

Accessibility

1. Absent from work (deployed or transferred, left job).
2. Absence from home (long vacation or business trip).
3. IllSubtotal for Accessibility Criterion
C.

Receipt of the Survey

1. Undeliverable surveys
2. Incorrect address
3. Anti-virus and anti-spam software prevents delivery of e-mailsurvey.

4. E-mail service is not functioning
5. Recipient deletes the e-mail survey without opening.
Subtotal for Receipt of the Survey Criterion
D.

Ability to Complete the Survey

1.
2.
3.
4.

Recipient does not understand the language.
Illness impedes the ability to complete the survey.
Limited amount of time to complete the survey.
Limited access to the Internet or email.

Subtotal for Ability to Complete the Survey Criterion
Number of Surveys Used in Denominator of Performance Rate

NPOSTAL-final = NPOSTAL-initial – (a1 + b1 + c1 + d1)
NEMAIL-final = NEMAIL-initial – (a2 + b2 + c2 + d2)

Draft
9/23/2021

Page 6