1 The DCRP study design allows for late data submissions. Therefore, the 2009 response rates reported here are ones

to date. They are higher than the ones available for use at the time of randomizing the 2010 sample to treatment

groups discussed further below.

2 Response rates (AAPOR RR5) are specific to the DCRP Annual Summary Form, which is requested of each

agency regardless of incidence or number of inmate deaths. RR5 is applicable to the 2009 DCRP because of a frame

verification effort that preceded data collection and, thus, concluded eligibility status for each agency.

A Comparison of Establishment Collection Mailing Methods

Chris Ellis1, Margaret Noonan2, Tim Flanigan3, Scott Ginder4, Hope Smiley McDonald5

1, 3, 4, 5 RTI International, 2 U.S. Department of Justice, Bureau of Justice Statistics

1 ellis@rti.org, 

2 margaret.noonan@usdoj.gov, 3

tsf@rti.org, 

4 sginder@rti.org,

5

smiley@rti.org

Introduction

Each year, approximately 4,000 inmates die while under the custody of the American correctional system (Noonan

& Carson, 2011). Effectively tracking mortality statistics among this population necessitates complete coverage of 

the nation’s prisons and jails. Thus, in 2000, the Bureau of Justice Statistics (BJS) established the Deaths in Custody

Reporting Program (DCRP) to collect mortality statistics and publish detailed analyses of comparative death rates

across demographic categories, offense types, and facility/agency characteristics. Policymakers, correctional

administrators, and government officials use DCRP products to maintain critical oversight and inform new policies,

procedures, and budgets.

Since 2009, RTI International has served as BJS’s DCRP data collection agent. Using a multimode approach 

designed to minimize respondent burden, RTI’s role is to collect inmate death data from the 50 state prison systems

(Department of Corrections [DOCs]) and across approximately 3,000 local jail jurisdictions. The inclusion of jails in 

the DCRP study population is critical in gaining a comprehensive understanding of the issue of mortality in 

corrections. First, jails are a necessary and routine point in the correctional process. In particular, although not all

arrestees proceed to prison, all prison inmates have served time in one of the nation’s jail jurisdictions. Second, jails

account for a substantial number of inmates, with between 12 and 13 million admissions per year and an average

daily population of approximately 750,000. Third, jails represent the most transitional of correctional settings, often

holding inmates for only hours, days, and weeks (versus months and years). In fact, the average length of a jail stay

is approximately 21 days. Jails account for 65% of the unconvicted inmate population.

RTI offers respondents the opportunity to complete the forms on the DCRP Web site (https://bjsdcrp.rti.org/), via

hard copy (i.e., facsimile or mail), or by electronic (i.e., bulk file) submission. Additionally, RTI includes e-mail,

mail, and telephone prompting steps into the data collection approach for nonresponding agencies and, in some

cases, data are collected via telephone to further reduce nonresponse. To further ensure high-quality data, such as

low item nonresponse, RTI implements a rigorous data quality follow-up process. In addition to soft and hard 

prompts within the Web forms, this process uses machine and interactive (i.e., statistical) edits to identify

inconsistent or missing data within or across death records. Agencies with resultant data quality issues are contacted

by telephone to resolve any discrepancies.

One of the biggest challenges associated with collecting these data is the reality that correctional administrators

manage overcrowded facilities that are understaffed and insufficiently funded, and routinely balance inmate and

staff safety. In short, DCRP respondents must increasingly ―do more with less,‖ which among other things means

response time to data requests is limited. Despite this challenge, in 2009,1 RTI achieved a 100% response rate across

the 50 state DOCs, a 100% response rate across the 150 largest jail jurisdictions, and a 96.7% response rate across

the remaining jail jurisdictions.2

(Identification and tracking of the 150 largest jail jurisdictions is a construct BJS

and RTI use to facilitate data collection. Along with the DOCs, these ―top 150‖ jail jurisdictions account for

approximately 80% of all inmate deaths and, thus, are tracked separately from the general jail jurisdictions during

the data collection.)

1
 

3

For the 2000–2008 DCRP collections, forms were mailed in a U.S. Department of Justice envelope.

Background

Choosing the most appropriate mailing method (e.g., Federal Express [FedEx]) is an important aspect of any mail or

mixed-mode data collection protocol, especially in the context of minimizing nonresponse (Dillman, 2000; Fowler,

1993). Typically, when mailing method is shown to positively affect response rate, it is in the context of what

method was used for nonresponse follow-up prompting. This is often attributed to use of a ―special‖ mailing

method, such as United States Postal Service (USPS) Priority Mail or FedEx. For example, Moore and An (2001)

describe a survey of physicians, in which mail response rates were significantly improved by the use of a USPS

Priority Mail contact. Similarly, in an experiment that examined refusal conversions, the FedEx cohort resulted in a

higher response rate when compared with the USPS Priority Mail cohort (Hagedorn et al., 2006).

Although there is agreement that varying mailing methods within a single study protocol increases response

propensity (Dillman, 2007), the impact of alternate initial mailing methods (those inviting participation in the study) 

is less clear. In other words, what impact does using FedEx or USPS Priority Mail have on eventual response rates,

especially when compared to USPS First Class? Moreover, there appears to be a relative dearth in the survey

literature on studying the inverse of this special mailing principle: whether some study populations would respond

equally as well or even better to a simple USPS First Class mailing.

Additionally, the choice of mailing method necessarily should also take into consideration study- and samplespecific factors, such as whether a signature from the recipient is required; whether mailing addresses may contain 

post office boxes (thus eliminating FedEx as a viable option); whether delivery directly to the potential participant is

important (thus suggesting FedEx as a primary method); and whether the methods under consideration will provide

delivery in a timely and cost-efficient manner across all geographic regions.

Finally, and consistent with the concept of a tailored design for each study, costs and respondent perceptions

associated with a mailing method(s) should be considered. In other words, if applicable, how does the study

population view the government using perceivably ―more expensive‖ methods, such as FedEx?

DCRP Mailing Experiment

Regular mailings to the entire population of jail jurisdictions and state DOCs provided the DCRP team with insight

into the mail delivery environment for the study. Other research has shown that the use of Priority Mail did not

improve response rates across two mailing cohorts (Messer & Dillman, 2011). Given the lack of clear guidance from

the extant literature and seeing a need and an opportunity for an experiment, RTI and BJS determined to test which 

mailing method would work best for the DCRP respondents.

The DCRP Data Collection Protocol

The routine data collection protocol for DCRP involves several mailings and prompts to each agency throughout and 

following the reference year period. The DCRP protocol implemented for 2010 data collection involved a

multimode design, involving original and replacement mailings, various reminders or nonresponse prompts, and an 

extensive data quality follow-up effort. Exhibit 1 provides an overview of that protocol. Exhibit 2 details the data

collection schedule starting with the mailing of the 2010 Annual Summary Forms through data collection close-out. 

Genesis of the DCRP Mailing Experiment

The initial DCRP protocol (2009) involved sending all correspondence to agencies, other than simple reminders or

prompts, via FedEx.3 The draw of using FedEx for these mailings was the prominence of the packaging and the

ability to track shipments, both of which were believed to facilitate data collection overall. This approach seemed 

especially appropriate given the experience on two of RTI’s prominent establishment/employee data collections. 

2
Exhibit 1. 2010 Data Collection Protocol

2010 Annual Summary Forms 

Original Mailing

(including individual death reports)

2010 Annual Summary Form Reminder

(E-mail or Letter)

Data Processing

Annual Summary Form

Replacement

Mailing

Telephone

Nonresponse Calls

Data Processing

Data Processing

Final Nonresponse

Prompt

(E-mail or Letter)

Data Quality Follow-Up

Calling

Yes

Yes

Yes

No

No

No

Annual 

Summary

Form

Received?

Annual 

Summary

Form

Received?

Annual 

Summary

Form

Received?

3
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Exhibit 2. 2010 Data Collection Schedule

Data Collection Activity Approximate Date

2010 Annual Summary Forms Week of January 20, 2011

–Sent by FedEx, USPS Priority, and USPS First Class

First Reminder by E-mail/Letter February 28, 2011

Verification Calls to Agencies March–April 2011

2010 Nonresponse Prompt by E-mail/Letter February 2011

2010 Replacement Forms Mailing May 24, 2011

2010 Telephone Nonresponse Calls August–September 2011

2010 Final Nonresponse E-mail/Letter Prompt September 2011

2010 Data Quality Follow-up Calls August–October 2011

Specifically, in Year 1 of the National Inmate Survey (NIS), funded by BJS, project staff found that the letters sent

to busy law enforcement heads and facility administrators through USPS were often lost or quickly forgotten. Thus,

a substantial amount of staff time was dedicated to resending letters via the postal service, fax, or e-mail. Once NIS

staff began to use FedEx to deliver letters, the initial costs of sending the letters overnight were quickly offset by the

increased attention the letters received upon delivery, resulting in reduced staff time. Additionally, the use of express

mail (FedEx) proved to be a highly effective strategy for the National Sample Survey of Registered Nurses

(NSSRN) 2000. This strategy increased the response rate nearly three-fold between the 1996 survey and the 2000 

survey, thus reducing the number of cases that required costly telephone follow-up 

(http://datawarehouse.hrsa.gov/nssrn.aspx).

Once the mailings were delivered, RTI received several telephone calls from respondents questioning the use of

what was perceived as an expensive delivery carrier. Respondents were informed that the mailing was done under a

less expensive, negotiated rate RTI has with FedEx that was lower than the typical package costs. This

notwithstanding, the perception of wasteful spending, particularly when state and county budgets were tight because

of the recession, was difficult to ameliorate.

There was some truth to the respondents’ concerns. Although the average cost to ship each package was $7.00—in

line with the project budget and, again, anticipated to ―pay for itself‖ through lessened nonresponse follow-up—RTI

experienced a changing landscape with FedEx variable or supplemental charges. There were also additional costs

associated with some FedEx packages, such as ones that had to be resent because of address changes, were deemed 

nondeliverable, were rejected by the recipient, or were sent to Alaska and Hawaii. These added surcharges ranged 

from $10 to $20 per package. With a sample of more than 3,000 agencies, agency address changes or delivery

refusals were an unavoidable occurrence, even if limited only to instances when the agency point of contact leaves

or changes positions. Finally, FedEx has character limits on its packaging labels. For a study like DCRP, where

agency names or destination fields can be lengthy, this required a detailed review to truncate longer than acceptable

fields. Ultimately, this resulted in additional unexpected costs. These extra costs reinforced the decision to

reevaluate FedEx as the primary mailing method.

As a result of these findings, respondent feedback, and a desire to contribute to the methods literature concerning 

establishment mailing methods, the DCRP team decided to test a hypothesis that other, less expensive forms of

mailing may be used without jeopardizing DCRP response rates.

Mailing Experiment Methodology

In consultation with BJS, RTI embedded an experiment into the routine mailout process for 2010 Annual Summary

Forms, the mailing of which occurred in January 2011. This involved mailing all routine forms to all agencies, but

varying the mail carrier. The goal was to determine whether there were timing and rate of response benefits

associated with different mailing options. In addition, the experiment would analyze cumulative costs associated 

with the number of nonresponse prompting mailings (i.e., whether additional mailings offset any savings associated 

with a lower cost mailing method). This would allow for a scientific analysis of these mailing options and,

hopefully, confirm the hypothesis that less expensive shipping methods are equally suitable for DCRP.

4
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

The following procedures were employed for the mailing experiment. First, because of the critical nature of their 

participation and data, the 50 state DOCs, the largest 150 jail jurisdictions, and an additional 28 ―special case‖

general jail jurisdictions were excluded from the experiment (i.e., randomization and analysis, such that they would 

continue to receive materials via FedEx). This exclusion was in an effort to not unpredictably affect the success of

the surrounding and ongoing data collection. Of the 2,717 remaining agencies, RTI randomly assigned the sample to

one of three mailing method cohorts: FedEx, USPS Priority Mail, or USPS First Class Mail. Because the default

data collection protocol utilized FedEx, the vast majority of agencies eligible for the experiment (1,717) were

randomly assigned to a control group, or Cohort 1. The remaining 1,000 agencies were randomly assigned to one of

two treatment groups—Cohorts 2 or 3 (500 each) (see Exhibit 3). For mailings scheduled to occur later in the data

collection protocol, agency cohort affiliation was noted and the appropriate mailing method (i.e., FedEx, USPS

Priority Mail, or USPS First Class Mail) was applied to maintain the integrity of the experiment.

Exhibit 3. Experimental Cohorts

Cohort Mode of Mailing Sample Size

1 FedEx 1,717

2 USPS Priority Mail 500

3 USPS First Class Mail 500

Total 2,717

The randomization procedure controlled for 2009 response status, ensuring that approximately the same proportion

of nonrespondents were assigned to each of the three cohorts. The randomization also controlled for the speed with 

which a jurisdiction responded in 2009 to ensure a good mix of prompt versus slower responders in each cohort.

Subsequent to randomization, several jurisdictions were later deemed ineligible for analysis because they merged

with another jurisdiction for DCRP reporting or because they responded to the 2009 collection prior to receiving the

experimental mailing. A comparison of the randomized and subsequently analyzed cohorts, including sizes and

count/percentage of 2009 nonrespondents, is provided in Exhibit 4.

Exhibit 4. Comparison of Randomized and Analyzed Cohorts

Cohort

Mode of 

Mailing

Randomized Analyzed

Total

2009 

Nonrespondents Percent Total

2009 

Nonrespondents Percent

1 Fed Ex 1,717 91 5.3 1,672 91 5.4

2 USPS

Priority

Mail

500 27 5.4 488 27 5.5

3 USPS First

Class

500 26 5.2 484 26 5.4

Total 2,717 144 5.3 2,644 144 5.4

Other details concerning the experiment related to package size and mailing dates. In particular, and regarding

Cohort 3, the First Class Mail envelope used was a 9‖ x 12‖ white Tyvek®

envelope, specially designed for DCRP.

In addition to the standard RTI return address in the top left corner, and the recipient’s affixed mailing address in the

center, a study identifier (―Deaths in Custody Reporting Program [DCRP]‖) was printed in the bottom left quadrant

of the mailing envelope. To ensure comparability with the other two cohorts, the same study identifier was added in 

sticker format to the FedEx and USPS Priority envelopes. Additionally, considering the varying delivery times

associated with each mailing method, RTI attempted to obviate potential systematic differences in response time by

staggering the mailing date of each cohort’s packages. Therefore, Cohort 3 (USPS First Class) was mailed a few

days before Cohort 2 (USPS Priority Mail), followed by Cohort 1 (FedEx) in an attempt to ensure that agencies

received their forms on approximately the same day.

5
Results

Analysis of the 2,644 eligible cases was conducted to answer research questions classified into two groups: (1) those

primarily relevant to DCRP, and (2) those intended to generalize to establishment survey research at large.

The questions primarily relevant to the DCRP were:

1a. While maintaining the rest of the protocol, can the data collection forms mailing method be changed

without jeopardizing response rates?

1b. Which mailing method is the most cost-effective over the course of the entire protocol?

The questions intended to be generalized to establishment surveys were, ―Prior to the first nonresponse prompt (the

period most transferrable to other establishment data collections)...‖

2a. Do response rates vary by mailing cohort?

2b. Do response rates vary among prior-round (2009) nonrespondents by mailing cohort?

2c. Does the speed with which we receive a response differ by mailing cohort, and what role does mode of

response play?

Questions Relevant Primarily to the DCRP

Focusing on question 1a, Exhibit 5 displays the response rate by mailing cohort over time for the period from initial

mailing to the end of nonresponse follow-up. Delineated on the graph are the points in time when jurisdictions were

contacted per the overarching data collection protocol—an e-mail prompt 34 days after initial mailing, a frame

verification call approximately 30 days after the e-mail prompt, a replacement mailing of data collection materials

56 days after the verification call, and nonresponse follow-up calls beginning 92 days after the second forms

mailing. The frame verification call was primarily intended to gather information relevant to the next year’s (2011)

DCRP data collection, but RTI took advantage of the contact to either thank the respondent for his or her 2010

response or remind him or her of the request for 2010 data. Of note in Exhibit 4 is the generally parallel track that

the individual mailing cohorts follow, with FedEx and Priority Mail merging together relatively quickly. First Class

Mail seemed to follow the same trajectory, but at a slightly lower response rate. The three cohorts all merge to the

mid-90% range at the end of the data collection period: 97.0% for FedEx, 96.3% for USPS Priority Mail, and 95.5% 

for USPS First Class Mail. Although not identical, the differences among these response rates are not statistically

significant (χ2 = 2.9, p = 0.23, 2 degrees of freedom [df]).

Focusing on question 1b, the cost of conducting data collection, it is important to compare the cost of applying each 

method to the entire DCRP universe of 2,954 entities. To do so, RTI calculated a cost per case for each mailing

method that included both postage and the labor involved in preparing the mailing. RTI assumed that the cost per 

case for each cohort was a reasonable estimate of what the cost per case would be if a particular method would be

deployed into full-scale production. Costs were calculated based on typical rates for survey support staff that have

been used to deploy DCRP forms mailings in the past.

In addition to the cost per case for mailing, there are also costs associated with sending the e-mail prompt, making

the verification calls, and conducting the nonresponse follow-up. The cost of sending an e-mail prompt does not

vary based on the number of e-mails being sent; the only significant expense is in developing the actual message. As

such, the cost of the e-mail prompt does not vary between the cohorts and was not considered in this comparison.

Likewise, the verification call must be done for all entities in the DCRP universe, so there is no verification call

differential based on mailing method to include in the comparison. Nonresponse follow-up costs do differ between

the mailing cohorts because the response rates (and hence the number of nonrespondents to be contacted) may be

different when the calls commence. An estimate of the labor required to make a nonresponse call and a typical

hourly rate for the individuals making the calls were used to construct the dollar amount included in the comparison.

6
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

-

-

Exhibit 5. Response Rate by Mailing Cohort, With Indicators of Prompting Contact

Exhibit 6 contains the cumulative response rate by mailing cohort at each of the data collection contact points.

These rates are the basis for the estimates in Exhibit 7, which shows the projected number of nonrespondents

remaining at each of the data collection contact points for the entire DCRP universe, by mailing cohort. That is, if

each mailing method was applied to the total DCRP universe, these are the counts of nonrespondents that the results

of the experiment would predict at each data collection contact point. These predictions were used to calculate the

estimated total comparison cost for each of the mailing methods. The e-mail prompt and verification call counts

were not used, for the reasons mentioned above, but are included for completeness. Exhibit 8 displays the individual

component and total comparison costs for each mailing method examined. Notably, despite being less expensive

than FedEx shipping rates, the labor involved in assembling the USPS Priority Mail package led to this method 

having the highest comparison cost—$2,136 higher than FedEx and $11,453 higher than USPS First Class Mail.

Despite having slightly higher costs at the nonresponse follow-up stage of data collection, the USPS First Class Mail

method was the least expensive, $9,317 less than FedEx.

Exhibit 6. Cumulative Response Rates at Each Data Collection Contact Point, by Mailing Cohort

Cumulative Response Rate Prior to:

E mail Verification Forms Nonresponse

Cohort Mailing Method Prompt Call Remailing Follow Up

1 FedEx 64.5 76.9 83.4 91.8

2 USPS Priority Mail 62.5 77.0 83.6 91.4

3 USPS First Class 59.1 71.5 78.7 89.7

Total 63.2 75.9 82.6 91.3

7
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

-

-

Exhibit 7. Estimated Number of Nonrespondents at Each Data Collection Contact Point, by Mailing 

Cohort

Projected Number of Nonrespondents Remaining at the Time

of:

E mail Verification Forms Nonresponse

Cohort Mailing Method Prompt Call Remailing Follow Up

1 FedEx 1,048 682 491 242

2 USPS Priority Mail 1,108 678 484 254

3 USPS First Class 1,208 842 629 305

NOTE: The projected number of nonrespondents by mailing cohort assumes a total DCRP universe of 2,954. Varying rates of

estimated nonresponse are calculated according to the achieved response rates shown in Exhibit 6.

Drawing together the results of investigating questions 1a and 1b, it seems reasonable to conclude that the USPS

First Class Mail method should be applied to DCRP to minimize costs without jeopardizing response rates.

Questions Relevant To Establishment Surveys in General

The length of the DCRP data collection period is longer than that of the typical establishment survey, and contacting

protocols can differ widely across survey efforts. In an attempt to account for these differences in the generalization

of results from this experiment, analysis was conducted on response status prior to any contacts beyond the initial

mailing (i.e., if the DCRP data collection period were shorter and, thus, more similar to other establishment surveys,

how would mailing method affect response status?).

Exhibit 8. Component and Total Comparison Costs, by Mailing Cohort

Focusing on question 2a, variance of response rates by mailing method, Exhibit 9 displays the response rate by

mailing cohort over time for the period corresponding only to the initial mailing. In particular, it depicts the response

rate progression since the estimated receipt date for the original mailing. Of note in Exhibit 9 is that the response

trajectories over time seem very similar across mailing cohorts, although the FedEx cohort consistently maintains a

slightly higher response rate than the Priority or First Class cohorts. Statistically speaking, the mailing cohort effect

on response rates is only marginally significant (χ2 = 4.9, p = 0.09, 2 df). Looking at specific differences among the

8
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

cohorts, the difference between FedEx and First Class Mail is the only significant difference, with the odds of

receiving a response from the First Class Mail cohort being 0.79 times those of the FedEx cohort (χ2 = 4.8, p = 0.03,

1 df), with a 95% confidence interval of (0.65, 0.98). Thus, it appears that use of FedEx may have some ability to

motivate response that USPS First Class Mail does not have, although such a conclusion is tenuous given the initial

marginal significance.

Exhibit 9. Response Rate by Mailing Cohort Prior to the First Prompting Contact

Considering question 2b, variance of response rates among prior-round (2009) nonrespondents by mailing method,

RTI noted (unsurprisingly) that response rates in 2010 among 2009 nonresponders were lower than response rates

among 2009 responders (χ2 = 78.8, p < 0.01, 1 df). This result did not vary by mailing cohort. Of more interest for

this experiment, however, were potential mailing cohort differences within the 2009 nonrespondent subgroup.

Exhibit 10 displays 2010 response rates by 2009 response subgroup and by mailing cohort. FedEx again appears to

have a slightly higher response rate (29.7% among 2009 nonrespondents) than Priority Mail (25.9%) and First Class

Mail (26.9%). However, these difference are within the boundary of random variation (χ2 = 0.2, p = 0.91, 2 df), so

RTI cannot conclude that a mailing effect exists. Thus, RTI concluded that mailing method does not impact the

propensity of a nonrespondent to respond in the following year of a longitudinal or recurring establishment survey.

Looking to the speed with which RTI received a response to our initial mailing (question 2c), we noted that among

those cases that provided data, the mean number of days that passed before data were provided (response time) did

vary by mailing cohort, with USPS Priority Mail and USPS First Class Mail both differing from FedEx, but not

differing from one another. It is worth noting that the response time was calculated from the estimated date of

receipt, not the date of mailing. This means that observed differences in response time by mailing cohort controlled 

for differing speeds with which the mailing methods made delivery. Mean response time for the FedEx cohort was

2.5 days faster than Priority Mail (t = 3.59, p < .01, 1,667 df) and was 2.4 days faster than First Class Mail (t = 3.37,

p < .01, 1,667 df).

9
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Exhibit 10. Response Rate by 2009 Response Status Subgroup and Mailing Cohort

2009 Nonrespondents 2009 Respondents

Subgroup Subgroup

Cohort Mailing Method Size Respondents Rate Size Respondents Rate

1 FedEx 91 27 29.7 1,581 1,052 66.5

2 Priority Mail 27 7 25.9 461 298 64.6

3 First Class Mail 26 7 26.9 458 279 60.9

Total 144 41 28.5 2,500 1,629 65.2

In addition to the mailing cohort effect on response time noted above, RTI also hypothesized that response time

could vary with use of the web response mode. Investigating mode (Web only, Mixed [with web], No web) usage

among the mailing cohorts showed that mode of response was not independent of mailing cohort (χ2 = 16.8, p < .01,

4 df).

As a result of the significance between mailing cohort and mode, a regression analysis was conducted on response

time to allow the inclusion of a mailing cohort by mode interaction. Similar to the initial analysis above, the model

indicated that response time varied significantly by mailing cohort (F = 3.6; p = .03; 2 numerator df; 1,661

denominator df). RTI also found a significant difference in response times by mode of response (F = 73.1; p < .01; 2 

numerator df; 1,661 denominator df). Despite the significant relationship between mailing cohort and mode of 

response, the interaction term in the model was not significant (F = 1.3; p = .28; 4 numerator df; 1,661 denominator 

df). In the absence of a significant interaction between mode of response and mailing cohort, a final model was

specified without the interaction term to draw conclusions about response time differences by mailing cohort and

mode of response.

Operating under the final, no interaction model, pairwise comparisons of mean response time were conducted for the

mailing cohorts and for the mode categories to pinpoint the differences and estimate the effect that these two

characteristics had on response time. The model estimated mean response times, accounting for the other terms in 

the model, were compared. Exhibit 11 displays the estimated mean response times, differences, and associated 

significance tests. Looking first to mailing cohort, FedEx had a faster response time than both USPS Priority Mail

(by 2.0 days) and USPS First Class Mail (by 1.6 days). Regarding mode of response, those responding only via the

Web exhibited response times 8.2 days faster than those not using the Web at all—and 5.5 days faster than those

who used a mixed mode of response including the Web. The group not using the Web at all responded 2.7 days

slower, on average, than the mixed-mode group. Because the interaction term was not significant, the mailing cohort

effect appears to be constant across the modes of response, and vice versa. For example, a FedEx respondent is

expected to respond 2 days faster than a Priority Mail respondent, on average, regardless of the mode of response.

Given the magnitude of the difference in response times by mode, it is clear that every effort should be made to

promote Web response, regardless of mailing method used to solicit information.

Discussion and Conclusions

Although prior research has well addressed the topic of altering mailing methods within a data collection and tested 

alternatives of those latter-stage methods, this study contributes to a topic with less representation in the survey

literature—that of initial mailing methods among establishments and their effect on total survey cost, time of

response, and mode of response. Although it is important to note that this experiment involved a topic known to

most of the contacted agencies and one done on an annual basis, there are nonetheless some very important findings.

First, the overall results show that at the end of the data collection period, the response rates across the two treatment

mailing cohorts (i.e., USPS Priority Mail and USPS First Class Mail) and the control group cohort (i.e., FedEx) do

not vary statistically from one another. Thus, use of a more traditional mailing method (e.g., USPS First Class Mail) 

in a study similar to DCRP may be entertained without compromise to response rates. Second, when costs over

time—including the costs associated with possibly higher interim rates of nonresponse—are taken into

consideration, the results from this experiment underscore that savings can be realized by using the least expensive

shipping option in terms of shipping and handling rates (i.e., USPS First Class Mail). Analyzing the time it takes for

10
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

  

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Exhibit 11. Model-Based Mean Response Times, Group Differences, and Tests of Significance for Mailing

Cohort and Mode of Response

Group 1 Group 2

Group 1 

Mean

Response

Time

Group 2 

Mean

Response

Time

Difference

(Group 2 − Group 1) Significance

Mailing Cohort

FedEx Priority Mail 14.2 16.2 2.0 Yes: t = 3.08,

p < .01, 1,665 df

FedEx First Class Mail 14.2 15.8 1.6 Yes: t = 2.38,

p = .02, 1,665 df

Priority Mail First Class Mail 16.2 15.8 −0.4 No: t = -0.5,

p <.62, 1,665 df

Mode of Response

NOTE: Based on the final model of Response Timei = β0 + β1 x Mailing Cohorti+ β2 x Modei+ εi.

Web Only No Web 10.8 19.0 8.2 Yes: t = 14.51,

p < .01, 1,665 df

Web Only Mixed with Web 10.8 16.4 5.5 Yes: t = 5.48,

p < .01, 1,665 df

No Web Mixed with Web 19.0 16.4 −2.7 Yes: t = -2.49,

p = .01, 1,665 df

a respondent to submit his or her data, FedEx appears to yield the fastest response (assuming materials are received 

at the same point in time across mailing methods). Finally, FedEx recipients are more likely to choose to respond via

the Web than other mailing methods.

However, although USPS First Class Mail does not appear to impede response rates, it should be noted that many

establishment data collections do not enjoy a long duration of active data collection such as is the case for DCRP.

Thus, when time is constrained and costs are not prohibitive, using overnight delivery services such as FedEx may

provide some modest benefit in terms of time to response. Additional improvements in time to response appear to be

linked to the use of a Web response option.

Although this study contributes to the methodological literature, there are some limitations that bear mentioning. 

First, the DCRP survey was established under the 2000 Deaths in Custody Reporting Act (DCRA; Public Law 106-

297). Before DCRA expired in 2006, prison and jail administrators were federally mandated to report inmate

mortality data to BJS. Given this mandate, it is likely that the DCRP respondents were and are accustomed to

participating despite the law having expired and regardless of how they received their forms. Moreover, the DCRP

data collection period4

is likely longer than most surveys (6 months) with regimented and recurring follow-up 

prompts built into the protocol. These distinctions may separate DCRP from other establishment surveys and limit

the generalizeability of these findings to other surveys. Despite these limitations, this research contributes to the

greater body of best survey practices and may serve as an important stepping stone to further research that evaluates

differences in mailing types across different types of surveys and audiences. Finally, more research is needed to

assess whether these findings may be generalized to other establishment survey populations (e.g., private sector

4

Specifically, the period during which an agency’s Annual Summary Forms can be received, and is actively

prompted for, occurs between January and June of the year following the reference period.

11
respondents, medical communities) and for establishment surveys that (1) are not as well recognized to the field, and 

(2) do not have as extensive a follow-up period.

References

Dillman, D. A. (2000). Mail and Internet surveys: The tailored design method (2nd ed.). New York: John Wiley &

Sons.

Dillman, D. A. (2007 updated). Mail and Internet surveys: The tailored design method (2nd ed.) Hoboken, NJ: John 

Wiley Co.

Fowler, F. J., Jr. (1993). Survey research methods (2nd ed.). Newbury Park, CA: Sage Publications, Inc.

Hagedorn, M. C., Montaquila, J., Carver, P., O’Donnell, K., Chapman, & C. (2006). National Household Education 

Surveys Program of 2005: Public-use data file user’s manual, volume I, study overview and methodology.

(NCES 2006-078). U.S. Department of Education. Washington, DC: National Center for Education 

Statistics. http://nces.ed.gov/nhes/dataproducts.asp#2005dp.

Health Resources and Services Administration (HRSA). (no date). Data warehouse. Available at

http://datawarehouse.hrsa.gov/nssrn.aspx.

Messer, B. L., & Dillman, D. A. (2011). Surveying the general public over the internet using address-based 

sampling and mail contact procedures. Public Opinion Quarterly, 75(3), 429-457. doi: 10.1093/poq/nfr021.

Moore, D. L., & An, L. (2001). The effect of repetitive token incentives and priority mail on response to physician 

surveys. In Proceedings of the American Statistical Association. Government Section on Survey Methods. 

Atlanta, GA.

Noonan, M. E., & Carson, E. A. (2011, December). Prison and jail deaths in custody, 2000-2009 – Statistical tables. 

Bureau of Justice Statistics, Washington DC. Available at

http://bjs.ojp.usdoj.gov/content/pub/pdf/pjdc0009st.pdf .

12