Impact of Monetary 

Incentives and 

Mailing Procedures 

An Experiment in a 

Federally Sponsored 

Telephone Survey 

Methodology Report 

U.S. Department of Education 

Institute of Education Sciences 

NCES 2006-066 

Impact of Monetary 

Incentives and 

Mailing Procedures 

An Experiment in a 

Federally Sponsored 

Telephone Survey 

Methodology Report 

March 2006 

J. Michael Brick 

Mary Collins Hagedorn 

Jill Montaquila 

Shelley Brock Roth

Westat 

Christopher Chapman 

National Center for 

Education Statistics 

U.S. Department of Education 

Institute of Education Sciences 

NCES 2006-066 
U.S. Department of Education 

Margaret Spellings 

Secretary 

Institute of Education Sciences

Grover J. Whitehurst 

Director

National Center for Education Statistics

Mark Schneider 

Commissioner

The National Center for Education Statistics (NCES) is the primary federal entity for collecting, analyzing, 

and reporting data related to education in the United States and other nations. It fulfills a congressional 

mandate to collect, collate, analyze, and report full and complete statistics on the condition of education in 

the United States; conduct and publish reports and specialized analyses of the meaning and significance 

of such statistics; assist state and local education agencies in improving their statistical systems; and 

review and report on education activities in foreign countries. 

NCES activities are designed to address high priority education data needs; provide consistent, reliable, 

complete, and accurate indicators of education status and trends; and report timely, useful, and high 

quality data to the U.S. Department of Education, the Congress, the states, other education policymakers, 

practitioners, data users, and the general public. 

We strive to make our products available in a variety of formats and in language that is appropriate to a 

variety of audiences. You, as our customer, are the best judge of our success in communicating 

information effectively. If you have any comments or suggestions about this or any other NCES product or 

report, we would like to hear from you. Please direct your comments to: 

National Center for Education Statistics 

Institute of Education Sciences 

U.S. Department of Education 

1990 K Street, NW 

Washington, DC 20006-5650 

March 2006

The NHES World Wide Web Home Page is http://nces.ed.gov/nhes. 

The NCES World Wide Web Electronic Catalog is http://nces.ed.gov/pubsearch. 

Suggested Citation

Brick, J.M., Hagedorn, M.C., Montaquila, J., Brock Roth, S., Chapman, C. (2006). Impact of Monetary 

Incentives and Mailing Procedures: An Experiment in a Federally Sponsored Telephone Survey (NCES 

2006-066). U.S. Department of Education. Washington, DC: National Center for Education Statistics. 

Contact:

Chris Chapman 

202-502-7414 

nhes@ed.gov
Monetary Incentives and Mailing Procedures 

iii 

Contents 

Page 

List of Tables .......................................................................................... iv 

List of Figures ......................................................................................... v 

List of Exhibits ........................................................................................ v 

1. Introduction ...................................................................................................................... 1 

2. Incentives in Surveys ....................................................................................................... 5 

3. Screener Experiment......................................................................................................... 6 

4. Adult Education for Work-Related Reasons Survey Experiment..................................... 28 

5. Summary ........................................................................................... 33 

References .... ....................................................................................................................... 35 

Appendix A: Response Rate Calculation Methods............................................................... A-1 

Appendix B: Logistic Regression Analysis .......................................................................... B-1
Monetary Incentives and Mailing Procedures 

iv 

List of Tables 

Table 

1 Weighted unit response rates and percentage distribution of reasons for unit 

 nonresponse for the NHES Screener: Selected years 1991–2003........................... 3

2 Sample sizes for Screener experimental conditions: 2003...................................... 8

3 Weighted percentage distributions of final disposition codes for numbers 

 sampled for the Screener, by incentive group: 2003............................................... 11

4 Mean number of Screener call attempts for mailable cases and for nonmailable 

 cases, by final disposition: 2003 ............................................................................. 12 

5 Screener response rates and ever refusal rates, by incentive group: 2003.............. 14

6 Screener response rates and ever refusal rates, by advance letter incentive: 

 2003 ........................................................................................................................ 16

7 Percentage distribution of Screener completes for mailable cases that completed 

 the Screener, by refusal conversion stage: 2003..................................................... 18

8 Percentage of cases with completed Screeners that never refused, by advance 

 letter incentive: 2003 .............................................................................................. 18 

9 Percentage of cases that ever refused to complete the Screener and final 

 disposition, by incentive group: 2003..................................................................... 19

10 Mean number of calls to final status for Screeners that ever refused, 

 by incentive group: 2003 ........................................................................................ 20

11 Mean number of calls for initial refusal Screener cases, by outcome 

 of call and incentive group: 2003............................................................................ 20

12 Screener response rates after first conversion attempt, by incentive 

 group: 2003............................................................................................................. 21

13 Percentage of cases that completed Screeners after second refusal conversion, 

 by Priority Mail and incentive group: 2003............................................................ 23

14 Percentage of Screeners that refused twice, by final disposition and incentive 

 group: 2003............................................................................................................. 24

15 Mean number of calls between refusals for second Screener refusal cases, 

 by incentive group: 2003 ........................................................................................ 25

16 Percentage of completed PFI and AEWR interviews, by incentive 

 group: 2003............................................................................................................. 26
Monetary Incentives and Mailing Procedures 

v 

List of Tables (continued) 

Table Page

17 Sample sizes and weighted percents for Adult Education for Work-Related 

 Reasons for experimental conditions: 2003............................................................ 29

18 Percentage distribution of AEWR final dispositions, by type of 

 nonresponse and incentive group: 2003.................................................................. 30

19 Number of calls to final disposition for AEWR initial refusal and 

 maximum call cases, by type of nonresponse and incentive group: 2003 .............. 31

20 Number of AEWR second refusal cases, by final disposition, incentive 

 group, and Priority Mail: 2003................................................................................ 32 

B-1 Logistic regression estimates for initial cooperation .............................................. B-3 

B-2 Logistic regression estimates for first refusal conversion....................................... B-4 

B-3 Logistic regression estimates for second refusal conversion .................................. B-5

List of Figures 

Figure Page

1 Weighted unit response rates for the NHES Screener: Selected years, 

 1991–2003 .............................................................................................................. 4

2 Screener response rates and ever refusal rates, by incentive group: 2003.............. 15

3 Screener response rates by total incentive amount and incentive 

 group: 2003............................................................................................................. 17

4 Percentage of completed PFI and AEWR interviews, by incentive 

 group: 2003............................................................................................................. 27

List of Exhibits 

Exhibit 

1 Summary of major study comparisons ................................................................... 10

Monetary Incentives and Mailing Procedures 

1 

1. Introduction 

Achieving high response rates in sample surveys has become increasingly more difficult in recent 

years. Atrostic, Bates, Burt, and Silberstein (2001) show that at the end of the twentieth century, the rates 

of nonresponse were increasing for in-person household surveys conducted by the U.S. Census Bureau. 

The issue is of even greater concern in random digit dial (RDD) telephone surveys. Curtin, Presser, and 

Singer (2000) and Steeh et al. (2001) examine changes in response rates in RDD surveys, but the Survey 

of Consumer Attitudes (SCA) is the only national RDD survey they review. No response rates for RDD 

surveys conducted after 1999 are given in these articles. However, steep declines in RDD survey response 

rates after 1999 have been reported by Curtin, Presser, and Singer (2005). 

Groves and Couper (1998) describe a host of factors that may affect response rates in household 

surveys. For RDD surveys, the most important of these factors are those related to the study 

characteristics (e.g., burden, salience, sponsorship, and content), the general survey environment (e.g., 

call screening technology, level of telemarketing and other types of calls to the household, and the 

economy), and the survey methodology (e.g., use of advance letters, refusal conversion attempts, and 

monetary incentives). Curtin, Presser, and Singer (2000) present evidence that the level of effort used to 

gain survey participation in the SCA increased over approximately 18 years. More specifically, they show 

that the mean number of call attempts to complete an interview more than doubled and that there was an 

increase in refusal conversions, resulting in an increase of about 25 percent in interview time to complete 

a survey.1

 By 1996, monetary incentives were introduced experimentally in the SCA and by 1999, 

incentives were used for all possible respondents in an effort to keep response rates from falling to 

unacceptable levels. 

The National Household Education Surveys Program 

The National Household Education Surveys Program (NHES) is an RDD survey developed by the 

National Center for Education Statistics (NCES) in the Institute of Education Sciences, U.S. Department 

of Education. It is designed to collect information on important educational issues through telephone 

surveys of households in the United States. NHES has enabled NCES to gather data on a wide range of 

issues, such as early childhood care and education, children’s readiness for school, parent perceptions of 

school safety and discipline, before- and after-school activities of school-age children, participation in 

adult and continuing education, parent involvement in education, school choice, homeschooling, and civic 

involvement. NHES uses computer-assisted telephone interviews (CATI) and has been conducted by 

Westat in 1991, 1993, 1995, 1996, 1999, 2001, and 2003. 

NHES provides data on the populations of special interest to NCES and education researchers as 

defined by age and/or grade in school for each child survey and for persons ages 16 and older who are not 

enrolled in grade 12 or below for each adult survey. It targets these populations using specific screening 

and sampling procedures. Specific age or grade ranges for a given survey are determined by the survey 

topic and the research questions formulated for the specific survey administration. 

NHES provides national cross-sectional estimates for the 50 states and the District of Columbia. 

The NHES design also yields estimates for subgroups of interest for each survey, as defined by age or 

grade for children, educational participation status for adults, and Black and Hispanic origin for all 

populations of interest. In addition to providing cross-sectional estimates, NHES is also designed to 

provide estimates of change over time in key statistics. The survey instruments are designed to address 

 

1

 Curtin, Presser, and Singer (2000) cite an increase from 2.1 hours in 1981 to 2.7 in 1996; these 2 years were used for this comparison because 

the average interview lengths for these two administrations were comparable. 
Monetary Incentives and Mailing Procedures 

2 

the selected issues in sufficient detail so that analyses can be performed to help explain the phenomena of 

interest. 

This report describes an experiment conducted in NHES:2003 that was designed to examine the 

effect of different mailings and monetary incentives. The goal of the experiment was to determine if 

monetary incentives should be used in future surveys, and to identify the effectiveness of various levels of 

incentives in gaining increased initial cooperation, refusal conversion, and overall unit response rates. The 

2003 administration was conducted from January 2 through April 13, 2003. Data were collected using a two 

stage sampling procedure, where households were selected and then screened for eligible subjects for the 

topical surveys. Then, within each household, eligible subjects were sampled and topical interviews were 

conducted about them. In the NHES:2003 Screener, household members were enumerated and 

demographic and educational information that determined eligibility for the two topical surveys was 

collected. The NHES:2003 topical surveys were the Parent and Family Involvement in Education Survey 

(PFI) and the Adult Education for Work-Related Reasons Survey (AEWR). The populations of interest in 

the surveys were children enrolled in kindergarten through 12th grade for PFI and persons age 16 or older 

who were not enrolled in grade 12 or below, not institutionalized, and not on active duty in the U.S. armed 

forces for the adult education survey. The Screener was brief and took an average of 3.5 minutes to 

complete, while the PFI interview took an average of 24.1 minutes to complete and the adult education 

interview took an average of 16.8 minutes to complete (data not shown). 

The monetary incentive experiment (detailed later) focused primarily on the initial contact, or 

Screener, because gaining participation at this initial stage is the most difficult component of RDD 

surveys (Groves and Couper 1996; Maynard and Schaeffer 1997). Table 1 shows the number of 

completed screener interviews and the response rates for NHES data collections from 1991 to 2003. A 

graphical depiction of the NHES Screener response rates between 1991 and 2003 is given in Figure 1. 

The response rates are weighted by the base weights of the sampled telephone numbers to account for 

differential selection probabilities used in the surveys. 

The Screener unit response rates in the first two administrations (1991 and 1993) were more than 

80 percent, but then fell in 1995 and 1996. Brick and Collins (1997a) show that much of this decrease is 

due to changes that increased the length and content of the screening interview. Brick and Collins (1997b) 

also report on an experiment that confirmed these findings.2

 In 1999, the length and content of the 

Screener were revised to be more consistent with the earlier surveys and the response rates rose to 74 

percent. The Screener response rates for 2001 and 2003 declined despite the fact that the length and 

content of the Screener were very similar to 1999. 

This decrease in Screener response rates occurred despite increases in the levels of effort used in 

NHES over the years. These increases were in the number of call attempts, the number of refusal 

conversions, and the use of mailings. The levels of effort for the earlier surveys (1991, 1993, 1995, and 

1996) are summarized in Brick and Collins (1997a), Nolin et al. (2000) for the 1999 survey, Hagedorn et 

al. (2003) for the 2001 survey, and Hagedorn et al. (forthcoming) for the 2003 survey. These reports 

discuss methods to encourage participation in NHES such as: use of letters, messages left on answering 

machines, refusal conversion attempts, additional call attempts for different types of outcomes, special 

interviewer training, and monetary incentives. 

In NHES:2001, addresses that could be used for mailing purposes (referred to as mailable 

addresses) were obtained for the sampled telephone numbers from either of two commercial firms 

(referred to as a mailable address). Addresses were obtained for about 48 percent of sampled telephone 

numbers (about 85 percent of known residential telephone numbers in the sample). A first class letter 

2 The Brick and Collins (1997b) study included a comparison of response rates for a brief Screener containing a “screen-out” question with an 

expanded Screener that included full enumeration.
Monetary Incentives and Mailing Procedures 

3 

explaining the purpose of the survey and encouraging participation in the study was mailed to a random 

sample of one-half of the households with mailable addresses. The weighted unit response rate for 

numbers with addresses to which letters were mailed was 75 percent and for those numbers with addresses 

to which letters were not sent the rate was 70 percent. (By comparison, the unit response rate for numbers 

without addresses was 55 percent.) 

Table 1. Weighted unit response rates and percentage distribution of reasons for unit 

nonresponse for the NHES Screener: Selected years, 1991–2003 

Reasons for unit nonresponse (percentage distribution) 

Year of survey 

Number of 

completed 

screeners 

Overall unit 

response rate 

(percent) Refusals

Maximum 

calls1

Other 

nonresponse2

1991 .................. 60,322 81.0 84 7 9 

1993 .................. 63,844 82.1 68 15 183

1995 .................. 45,465 73.3 84 9 7 

1996 .................. 55,838 69.9 83 10 7 

1999 .................. 55,929 74.1 76 17 7 

2001 .................. 48,385 67.5 74 18 8 

2003 .................. 32,049 61.7 76 16 8 

1 “Maximum call” cases are those that received at least eight call attempts during which contact was made with a person yet the 

Screener was not completed. 

2 “Other nonresponse” includes language problems, a portion of residential no-answer and answering machine calls, and other 

forms of nonresponse. 

3 The NHES:1993 percent of “other nonresponse” cases is higher than that in other surveys. The lower rate of refusals in 

NHES:1993 and the generally higher response rate are indicative of the fact that less refielding of other nonresponse cases was

needed prior to ending data collection with an acceptable Screener response rate. 

NOTE: To avoid any differences in rates that might be attributable to the calculation method, all unit response rates given here 

were calculated using the business office method. The official rates for 2001 and 2003 use the survival method (see appendix A 

for details on the methods for computing response rates). The screening procedure (i.e., number of household members 

enumerated) in each data collection differed according to the sample requirements of the topical surveys conducted in the specific 

year. Percents may not sum to 100 because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, Selected years, 1991–2003. 
Monetary Incentives and Mailing Procedures 

4 

Figure 1. Weighted unit response rates for the NHES Screener: Selected years, 1991–2003 

45

50

55

60

65

70

75

80

85

90

95

100

1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004

Year of survey

Unit response

rate (%)

 

NOTE: To avoid any differences in rates that might be attributable to the calculation method, all unit response rates given here 

were calculated using the business office method. The official rates for 2001 and 2003 use the survival method (see appendix A 

for details on the methods for computing response rates). The screening procedure (i.e., number of household members 

enumerated) in each data collection differed according to the sample requirements of the topical surveys conducted in the specific 

year. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, Selected years, 1991–2003. 

0 
Monetary Incentives and Mailing Procedures 

5 

The results of the advance mailing in NHES:2001 provided further evidence that sending an 

advance letter increased Screener response rates in the survey. Previously, Brick and Collins (1997b) 

reported this result from an experiment conducted in a pretest for NHES:1996. This finding is not 

consistent with experiments in some other RDD surveys (e.g., Singer, Van Hoewyk, and Maher, 2000) 

where an advance letter did not increase response rates. The NHES advance letter is sent in an official 

U.S. Department of Education envelope, which may increase the likelihood that respondents open the 

mail and give greater legitimacy to the survey, as noted in Dillman (2000). This issue will be addressed 

later in the analysis of the incentive experiment findings. 

A primary focus of the current experiment, the effectiveness of mailings to households that 

initially refuse to participate, was also examined in the 2001 NHES survey. For the 2001 survey, refusal 

conversion letters (i.e., letters intended to persuade respondents to change their minds and complete the 

survey) were sent via FedEx or Priority Mail to 40 percent of the eligible telephone numbers for which 

addresses were available. Of those that received refusal conversion letters, 55 percent completed the 

Screener, compared to 38 percent of those that were not sent a letter. This difference must be interpreted 

with caution, however, as the receipt of a refusal conversion letter is confounded with the availability of 

address information. In the 2003 NHES experiment discussed in this report, a more systematic approach 

was taken to examining the effectiveness of sending a letter in increasing the probability of refusal 

conversion. 

In the next section, some of the literature on the use of incentives in surveys is reviewed, 

concentrating on RDD surveys to the extent possible; the design of the experiment in NHES:2003 is then 

described. Next the findings of the experiment are presented, including information on the effects of 

varying incentive and mailing conditions on response rates, costs, and quality. The conclusion provides 

some discussion of the implications of these findings for future RDD surveys, and NHES in particular. 

2. Incentives in Surveys 

Incentives have been used as a way of improving response rates in surveys for decades; yet the 

theory supporting the use of incentives is still not fully established. One theory proposed to explain the 

effectiveness of incentives is social exchange (Dillman, 2000), which suggests that giving an incentive to 

a household is considered by the household as an act that must be reciprocated. Responding to the survey 

is the reciprocal act. Monetary exchange theory is also sometimes invoked to explain responses when 

incentives are used. This theory suggests the household may be more likely to respond to the survey 

because the incentive is payment for this response. Groves, Singer, and Corning (2000) suggested a more 

complex model in which incentives may be used to compensate for other survey characteristics that may 

diminish likelihood of responding, such as long length (i.e., high respondent burden), content that lacks 

relevance to a respondent, or sponsorship by an organization that may be unknown to a respondent. 

Singer (2002) summarized these theories and reviewed the intended and unintended consequences of 

using incentives in surveys. Some of this literature is reviewed briefly below, focusing primarily on topics 

relevant to incentives in telephone surveys. 

A meta-analysis by Church (1993) shows that the payment of incentives in mail surveys is one 

method that consistently results in higher response rates. A similar finding was reported in a study by 

Trussell and Lavrakas (2004). They illustrated that for households that did not have a prior agreement to 

complete a survey, incremental increases in cash incentives were associated with increased response. 

Shettle and Mooney (1999) reported on an experiment in using monetary incentives that further supports 

the cost-effectiveness of incentives in mail surveys that are conducted under the aegis of the U.S. Federal 

Government. The studies of incentives nearly always find that prepaid incentives (those offered without 
Monetary Incentives and Mailing Procedures 

6 

requiring the recipient to respond) are more effective than promised incentives that are contingent on 

response. Berk et al. (1987) found that for an in-person survey prepaid incentives were effective, but 

promised incentives were not. Church (1993) reported a similar finding for mail surveys. Singer et al. 

(1999) and Gelman, Stevens, and Chan (2003) use different meta-analysis models for telephone surveys 

and do not agree that promised incentives are ineffective, but do agree that prepaid incentives are more 

effective than promised ones. 

The evidence on the utility of incentives in RDD surveys is more limited and more recent. One of 

the reasons for the delay in the use of incentives in telephone surveys is that it is more difficult to send 

prepaid incentives because addresses cannot be obtained for some sampled telephone numbers. Recently, 

the proportion of residential telephone numbers that can be linked to addresses using inexpensive 

methods has increased. This development has made sending advance mailings and prepaid incentives 

more attractive for RDD surveys. Both Singer et al. (1999) and Gelman, Stevens, and Chan (2003) have 

found that monetary incentives increase response rates in RDD surveys. 

Another way of using incentives is to provide an incentive only to those units that refuse to 

complete the interview in the initial attempt. However, the published literature on incentives for refusal 

conversion is limited. Martin, Abreu, and Winters (2001) found that response rates increased when 

monetary incentives were used to convert refusers in the Survey of Income and Program Participation, a 

Census Bureau survey. Groves et al. (1999) studied the effect of the differential use of incentives (i.e., 

only for refusal conversion) and found that most respondents already believed incentives were commonly 

used by survey organizations. Knowledge of the fact that other respondents were given incentives or 

different levels of incentives did not have a statistically significant effect on the participation of sampled 

persons for future surveys. Less is known about the effectiveness of incentives at the refusal conversion 

stage for RDD surveys. In one published report addressing this issue, Singer, van Hoewyk, and Maher 

(2000) found that prepaying incentives was effective in increasing survey participation among those who 

had initially refused to participate. 

In addition to the initial contact with an adult in a sampled household, called the Screener 

interview in NHES, obtaining the cooperation of persons sampled in the Screener for extended interviews 

is very important. However, no reports on the effectiveness of incentives for improving extended 

interview response rates exist. Generally, those sampled for extended interviews for NHES topical 

surveys have been willing to participate at relatively high levels, especially when a parent is responding 

about a sampled child. Response rates for surveys of adults about their adult education experiences have 

traditionally been lower. For example, in NHES:2001, the weighted unit response rate for the surveys of 

parents about their children was 87 percent while the corresponding adult education survey unit response 

rate was 77 percent (these are rates computed for sampled persons in households that completed the 

Screener). As a result, there was much interest in the potential of incentives to increase the response rates 

for the AEWR-NHES:2003 and a test of incentives for this survey was included in the experiment. 

3. Screener Experiment 

The overall objective of the incentive experiment in 2003 was to determine if there were 

economical alternatives for improving unit response rates in NHES. In light of the results of studies 

comparing the effectiveness of prepaid and promised incentives (see section 2), only prepaid incentives 

were used in this experiment. The experiment focused primarily on the Screener because this is where 

most nonresponse occurs, but an additional experiment for the adult education survey was also included. 

The experiment at the Screener level examined the effect of different treatments in the initial contact, at 

the first refusal conversion stage, and at the second refusal conversion stage. The adult education survey 

experiment (discussed in the next section) was only at the first refusal conversion stage. 
Monetary Incentives and Mailing Procedures 

7 

Since advance first class letters had been shown to be effective in improving response rates in 

previous administrations of the NHES, a first class letter in a U.S. Department of Education envelope and 

on official stationery was sent to every sampled telephone number for which an address could be 

identified. Alternatives such as sending the advance letter by Priority Mail were considered to be too 

costly. 

The treatments for the Screener that were subject to manipulation were 

 the amount of money included with the advance letter; 

 the type of mail used for converting those that refused the initial contact; and, 

 the amount of money included with the refusal mailing. 

In addition, mailing an official NHES brochure has been considered a possible method of 

improving response rates. To test this possibility, one of the experimental groups that received no 

monetary incentive was sent a brochure in the initial mailing.

Two types of mailings were used in this experiment to examine their effects on refusal conversion; 

these were first class mail sent in U. S. Department of Education letterhead envelopes (with a mailing cost 

of $0.37 per letter), and Priority Mail sent in U. S. Postal Service Priority Mail envelopes (with a mailing 

cost of $3.85 per letter). The experimental groups were developed to permit the examination of the 

following effects on response rates: 

 the effect of varied incentives in the advance letter ($0, $2, and $5); 

 the effect of varied incentives in the refusal conversion letter ($0, $2, and $5); and 

 the effect of Priority Mail versus first class mail for the refusal conversion letter. 

Table 2 shows the 10 experimental conditions included in the Screener experiment. The sample 

sizes for each of the conditions and for the nonmailable3

 cases (which were excluded from the 

experiment) are also given in the table. Addresses were identified for 59,365 of the telephone numbers in 

the experiment. Because the experimental treatments all involve mailings, telephone numbers for which 

an address could not be found were not included in the experiment. The sample also included 50,435 

telephone numbers for which addresses could not be identified. The percentage of cases for the 

experiment is affected by differential sampling of cases with addresses and differential sampling of 

telephone numbers in areas with a high concentration of minorities. Hagedorn et al. (forthcoming) 

provide the details on the sample design for NHES:2003. The fact that all numbers cannot be treated will 

depress the effectiveness of any strategy that involves mailing letters, but the effect is not as great as 

might be expected. In NHES:2003, 90 percent of all numbers that could be determined to be residential 

numbers had addresses (85 percent when the numbers are appropriately weighted by the inverse of the 

probability of selection). For most of this report, the focus is on the cases with addresses that were 

included in the experiment. Where relevant, cases without addresses are included in tabulations. 

 

3 Nonmailable cases are the cases for which an address could not be obtained. These cases were not sent an advance letter or incentive. This 

subgroup was not treated as a control group for this experiment because there is a considerable literature, including results from previous NHES 

surveys, that shows that this subgroup has lower response rates in RDD surveys than mailable cases. 
Monetary Incentives and Mailing Procedures 

8 

Table 2. Sample sizes for Screener experimental conditions: 2003 

Advance letter 

Refusal conversion letter 

plus brochures 

Group Mailing method Incentive Mailing method Incentive

Sample 

size1

 Total ............................. † † † † 109,800

Total mailable cases............... † † † † 59,365

 1 ......................................... First class/brochure $0 First class $0 5,765

 2 ......................................... First class $0 First class $2 5,700

 3 ......................................... First class $0 First class $5 5,700

 4 ......................................... First class $0 Priority $0 6,850

 5 ......................................... First class $0 Priority $2 5,700

 6 ......................................... First class $2 First class $0 6,850

 7 ......................................... First class $2 Priority $0 5,700

 8 ......................................... First class $2 Priority $2 5,700

 9 ......................................... First class $5 First class $0 5,700

 10 ....................................... First class $2 First class $2 5,700

Total nonmailable cases......... † † † † 50,435

† Not applicable.

1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly 

allocated to the experimental conditions. For nonmailable cases, the sample size is the number of telephone numbers. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Group 1 is essentially a control group, but every case in this group contained a treatment that was 

not included in any other group. Along with the advance letter, the group 1 cases were sent a short color 

brochure describing NHES to determine the possible effectiveness of a brochure for increasing response 

rates. The brochure was not included in any other advance mailings. Thus, the experimental design allows 

for a test of the effect of the brochure in the $0 advance incentive condition on whether potential 

participants will ever refuse to respond to the Screener survey. A brochure was included with an 

informational letter if a respondent specifically asked to be mailed more information on the study before 

agreeing to be interviewed. Refusal conversion mailings included brochures. 

The sample sizes in the table are the number of telephone numbers with addresses assigned to 

each group. Group 4 (no incentive in either the advance letter or the refusal conversion letter, and a 

refusal conversion letter sent by Priority Mail) is similar to the condition that was used in NHES:2001, 

although in 2001, FedEx was used rather than Priority Mail.4

 Larger sample sizes were assigned to groups 

4 and 6 because they were identified as the treatments that might increase response rates and be 

acceptable approaches in a federal government survey in that advance incentives have some advantages 

over refusal conversion incentives. 

Once sample sizes for the Screener experimental groups were determined and the cases were 

randomly assigned to the groups, advance letters were mailed with or without incentives according to the 

 

4 In NHES:2001, FedEx was used for refusal letters, with the exception of rural route and Post Office box addresses, to which FedEx cannot be 

sent. For these addresses, Priority Mail was used. 
Monetary Incentives and Mailing Procedures 

9 

plan. If the household refused to participate, refusal conversion letters were sent with the appropriate 

incentives. If a refusal was hostile (profane or abusive), no further efforts were made to contact the 

household. In addition, a small number of cases that directly contacted the U.S. Department of Education 

to refuse received no further contact. 

For analyses of the effects of the treatments on refusal conversion, the sample sizes in each group 

are considerably smaller, because only the subset of the telephone numbers with addresses for which 

Screener refusal conversion is necessary are included. Of the 59,365 telephone numbers in the 

experiment, only 20,259 cases required and received Screener refusal conversion letters (see table 10 for 

the total number of cases in each incentive group that ever refused). 

If the household refused a second time, they were subject to a second refusal conversion call. To 

boost response rates, a sample of 75 percent of the households that had not received a Priority Mail letter 

previously (groups 1, 2, 3, 6, 9, and 10) was sent a Priority Mail letter prior to the second refusal 

conversion call. Only 75 percent were sent by Priority Mail so that the effectiveness of the special mailing 

could be evaluated. The sampling to determine which numbers were assigned to the special mailing was 

done randomly within the experimental groups. The findings section of this report focuses on the initial 

and first refusal conversion rates to avoid the effect of the Priority Mailing at second refusal. The results 

of the Priority Mail at the second refusal conversion are then analyzed separately. 

An examination of the treatments applied to each experimental group (shown in table 2) reveals 

that a full factorial design was not used for this experiment. One reason for this is that, while testing each 

level of advance incentive ($0, $2, and $5), each level of refusal conversion incentive ($0, $2, and $5), 

and each mailing method for first refusal letters (first class and Priority Mail), the sample sizes in 

NHES:2003 were not sufficient to support a full factorial design. A second reason is that some treatments 

in a full factorial design (such as $5 advance incentive, $5 refusal conversion incentive, and Priority Mail 

for the refusal mailing) were deemed cost-prohibitive for NHES; therefore, even if such a treatment was 

found to be effective, there is little chance that it would ever be used in NHES. A third reason is that by 

balancing the experimental sample with respect to prior treatments or conditioning on the prior treatments 

during analysis, it is possible to minimize confounding effects. Exhibit 1 summarizes the key 

experimental comparisons that can be made with respect to the Screener treatments used in the 

experiment. 

Table 3 shows the final disposition code for all the telephone numbers included in the experiment. 

In addition, the numbers without addresses are included for comparison. This table gives an overall 

assessment of the effects of the various treatment groups. There are essentially five dispositions: an adult 

in the household completed the Screener interview, the number was determined to be nonresidential, all 

the calls to the household resulted in either a ring no-answer or an answering machine, the household 

refused to participate, or the household did not respond for some other reason (such as a language barrier 

or it was not available during the field period). The percentages in the table are weighted by the 

household base weight to take account of the differential sampling of minority areas and telephone 

numbers with addresses. All other analyses in this paper also use the household base weights. 

Table 4 gives the mean number of calls required to finalize the experimental Screener cases. This 

table gives a measure of the workload required on the part of the interviewer in order to finalize a 

Screener. On average, it required seven calls to finalize a Screener. This mean includes calls made to 

telephone numbers that were not resolved as residential or nonresidential. The table also gives the mean 

numbers of call attempts by final disposition code of the Screener. There is significant variation in the 

mean number of call attempts to finalize a Screener, ranging from three calls for nonresidential numbers 

to 22 calls for never-answered numbers. For nonmailable cases, an average of six calls were needed to 
Monetary Incentives and Mailing Procedures 

10 

finalize a Screener, but nearly 60 percent of these numbers were identified as nonworking or 

nonresidential before any call attempts were made by the interviewers. 

Exhibit 1. Summary of major study comparisons 

Experimental 

process Characteristic(s) Aspect(s) compared Groups compared 

 

Overall Unit response rate No monetary incentive vs. 

any monetary incentive 

Groups 1, 4 vs. groups 2, 3, 5, 

6, 7, 8, 9, 10 

 

Advance 

mailing 

Ever refused rate Brochure vs. no brochure Group 1 vs. groups 2, 3, 4, 5 

Advance 

mailing 

Ever refused rate $0 advance incentive vs. $2 

advance incentive vs. $5 

advance incentive 

Groups 1, 2, 3, 4, 5 vs. groups 

6, 7, 8, 10 vs. group 9 

 

Advance 

mailing 

Percent completed for 

cases that never refused 

$0 advance incentive vs. $2 

advance incentive vs. $5 

advance incentive 

Groups 1, 2, 3, 4, 5 vs. groups 

6, 7, 8, 10 vs. group 9 

 

First refusal 

mailing 

First refusal conversion 

rate 

$0 refusal conversion 

incentive vs. $2 refusal 

conversion incentive1

Groups 1, 4, 6, 7 vs. groups 2, 

5, 8, 10 

 

First refusal 

mailing 

Unit response rate after 

first refusal conversion,2

first refusal conversion 

rate 

Priority Mail vs. first class 

for first refusal mailing, for 

groups with the same 

advance and refusal 

conversion monetary 

incentives 

Group 1 vs. group 4; group 2 

vs. group 5; group 6 vs. group 

7; group 8 vs. group 10 

 

Second 

refusal 

mailing 

Percent completed after 

second refusal conversion 

Priority Mail vs. no mailing 

for second refusal 

conversion 

Group 1 Priority Mail vs. 

Group 1 no mailing, and 

similar comparisons for 

groups 2, 3, 6, 9, 10 1

 The balance of the design allows for first refusal conversion comparisons without considering the advanced incentive, since half 

of the four groups with an initial incentive of $0 had a $2 conversion incentive and half of the four groups with an initial incentive 

of $2 had a $2 conversion incentive, and the sample sizes in each group were approximately equal. 2

 Screeners that were converted on second refusal conversion attempts are counted as nonrespondents in this comparison. 

A primary goal of the Screener experiment was to determine if the different incentive and mailing 

conditions would have variable effects on response rates. Therefore, much of the analysis focuses on 

whether interviews were completed and related estimates. One of the statistics used is the unit response 

rate. The definition of the unit response rate used is explained in detail in appendix A, but is briefly 

described here. The overall unit response rate is the percentage of interviews completed taking all survey 

stages into account. For the Screener interview, there is only one survey stage. To compute the Screener 

response rate, it is assumed that 19.7 percent of the sampled numbers classified as never answered were 

residential.
11

Table 3. Weighted percentage distributions of final disposition codes for numbers sampled for the Screener, by inc

Final disposition 

Incentive group 

Sample 

size1 Percent complete

Percent 

nonresidential

Percent 

never answered Percent ref

 Total .......................... 109,800 ———Total mailable cases............ 59,365 53.3 (0.20) 21.2 (0.19) 2.0 (0.04) 17.9 (0 1 – ($0 brochure/1st $0).. 5,765 50.5 (0.65) 21.3 (0.55) 2.2 (0.14) 20.4 (0 2 – ($0/1st $2) ................. 5,700 53.0 (0.63) 21.2 (0.51) 2.0 (0.14) 17.8 (0 3 – ($0/1st $5) ................. 5,700 54.7 (0.62) 21.3 (0.57) 2.0 (0.13) 16.5 (0 4 – ($0/Priority $0).......... 6,850 50.0 (0.64) 21.4 (0.54) 2.1 (0.13) 20.7 (0 5 – ($0/Priority $2) ......... 5,700 52.7 (0.68) 21.0 (0.59) 2.0 (0.13) 18.3 (0 6 – ($2/1st $0) ................. 6,850 53.2 (0.54) 21.6 (0.50) 2.0 (0.10) 17.6 (0 7 – ($2/Priority $0).......... 5,700 54.8 (0.78) 20.5 (0.68) 1.9 (0.12) 17.3 (0 8 – ($2/Priority $2).......... 5,700 54.2 (0.75) 21.6 (0.61) 1.8 (0.11) 17.1 (0 9 – ($5/1st $0) ................. 5,700 54.8 (0.60) 21.3 (0.52) 1.9 (0.15) 16.6 (0 10 – ($2/1st $2) ............... 5,700 55.2 (0.58) 21.0 (0.61) 2.1 (0.12) 16.6 (0Total nonmailable cases...... 50,435 6.0 (0.12) 87.8 (0.15) 1.7 (0.03) 3.5 (0— Not reported. 1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly allocated to the experim

nonmailable cases, the sample size is the total number of telephone numbers; this total includes 30,725 telephone numbers that were identified as busin

through pre-field procedures and were never dialed. 

2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent (maximum call status), o

unavailable during the field period. NOTE: Standard error shown in parenthesis. Percents may not sum to 100 because of rounding. The parenthetical descriptions of the experimental con

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first class mail in a U.S. Depa

envelope, as were initial refusal letters in conditions not specifying Priority Mail. The Priority Mail letters were sent in the U.S. Postal Service Priority 

letters included a colored NHES project brochure. Final dispositions include the results of the second refusal conversion attempts. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys Program, 2003. 
Monetary Incentives and Mailing Procedures 

12 

Table 4. Mean number of Screener call attempts for mailable cases and for nonmailable cases, by 

final disposition: 2003 

Final Screener disposition Sample size1 Mean screening call attempts

 Total ....................................... 79,075 —

Total mailable cases......................... 59,365 7.0 (0.04)

 Completed.................................... 29,196 4.7 (0.03)

 Nonresidential.............................. 12,259 3.4 (0.04)

 Never answered............................ 3,385 22.3 (0.10)

 Refused ........................................ 10,102 12.9 (0.09)

 Other nonresponse2 ...................... 4,423 18.7 (0.18)

Total nonmailable cases................... 19,710 5.9 (0.05)

 Completed.................................... 2,853 5.5 (0.09)

 Nonresidential.............................. 9,968 3.2 (0.03)

 Never answered............................ 4,730 16.8 (0.07)

 Refused ........................................ 1,659 13.8 (0.18)

 Other nonresponse2 ...................... 500 19.9 (0.41)

— Not reported. 

1 For mailable cases, the sample size is the number of telephone numbers for which an address match was obtained, randomly 

allocated to the experimental conditions. For nonmailable cases, the sample size of 19,710 is the total number of telephone 

numbers that were ever dialed; this total excludes 30,725 telephone numbers that were identified as business or nonworking 

numbers through pre-field procedures and were never dialed. These pre-field procedures only identify a subset of business and 

nonworking numbers. 

2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent 

(maximum call status), or the respondent being unavailable during the field period. 

NOTE: Detail may not sum to totals because of rounding. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

For some analyses the unit response rates are not the most relevant measures. A reason noted 

earlier for focusing on other statistics, in addition to response rates, is that the response rates include the 

effect of the Priority Mailing at second refusal conversion. This mailing was differential and not part of 

the main Screener incentive experiment. It only affected groups without any previous Priority Mailing 

(groups 1, 2, 3, 6, 9, and 10). A second reason that other statistics may be more pertinent is that 

experimental groups (e.g., all groups sent a $2 incentive in the advance mailing) are sometimes combined 

to increase the precision of the estimates of the effect of the treatment. Because the combined groups have 

different treatments at different stages of refusal conversion, the response rates for the groups confound 

the treatment effects. 

Instead of response rates, either “ever refusal” rates or the percentage of cases that complete the 

interview at a specific stage of the process are often provided. Ever refusal rates are used primarily to 

evaluate the effect of a treatment in the first stage, prior to any refusal conversion. Ever refusal rates give 

the (weighted) number of cases in the group that refused the Screener divided by the total weighted 

number of cases that either completed or refused the Screener. The nonresponse cases that were never 

classified as refusals (such as those with language problems or that are not available in the field period) 

are not included in the computations of this rate. The ever refusal rate is the complement of the rate that 
Monetary Incentives and Mailing Procedures 

13 

AAPOR (2004) calls the cooperation rate. The term “ever refusal rate” is preferred because cooperation 

and response rates are often confused. 

In addition to measures related to improving response rates, estimates of the number of call 

attempts required to finalize the case or to complete a specific phase of the interview are produced. For 

example, table 4 provides the mean numbers of call attempts by the final disposition of the Screener. In 

other tables, statistics such as the mean numbers of call attempts needed to move from an initial refusal to 

a second refusal by experimental group are given. These numbers are used as an indication of whether 

there are obvious cost issues associated with an experimental group beyond the direct costs of the 

mailings and incentives. 

The results from the Screener experiment are given below, and the adult education survey 

experiment is discussed in the next section. All of the analyses were done with the appropriate survey 

weights using WesVar software (Westat 2000) to calculate jackknife variance estimates. Tests of 

hypotheses were done using two-tailed t-tests with a 5 percent type I error rate. Bonferroni adjustments 

were used to compensate for multiple comparisons to ensure that the overall significance level associated 

with all of the comparisons remained at or below the specified level of 5 percent. Given the problem of 

falling response rates, relatively small increases in response rates associated with a particular 

experimental condition are considered noteworthy in this paper. Differences that are not statistically 

significant and statistically significant differences of less than one percentage point are not discussed. 

Table 5 gives the Screener response rates and ever refusal rates for each of the 10 treatment 

groups. Since nonresidential telephone numbers are not used in computing response rates, the 30,725 

nonresidential numbers identified through prefield procedures, the 12,259 nonresidential mailable 

numbers, and the 9,968 nonresidential nonmailable numbers (see table 4) are excluded, and the estimates 

in the table are based on the remaining 47,106 mailable and 9,742 nonmailable numbers. The 

corresponding rates for the cases without addresses are also presented for comparison purposes. The 

response rates in this table and the percentage of sampled cases that were completed given in table 3 

highlight the efficiency considerations that led to oversampling telephone numbers with addresses in the 

survey. Figure 2 contains a graphical display of the Screener response rates and ever refusal rates by 

incentive group. 

Advance Mailing

The first topic considered is the effect of including the brochure in the advance mailing. This is 

the starting point because if the brochure does improve cooperation, then it would not be useful to 

combine group 1 with other groups that have no monetary incentive in the advance mailing. If including 

the brochure were effective, the ever refusal rate for group 1 should be lower than the ever refusal rates 

for the other groups with no money in the advance letter (groups 2, 3, 4, and 5). The results in table 6 

indicate that the inclusion of the brochure in the advance letter does not increase cooperation at the initial 

stage of the process. In fact, the initial cooperation rate for the brochure group (group 1) was lower than 

the rate for the other groups with no advance incentive combined. 

Table 5 illustrates that the ever refusal rates show the type of pattern expected, with rates for those 

with no incentive higher than those with $2 or $5 in the advance letter. The size of the differences by 

incentive amount is discussed in connection with the next table (table 6). Table 5 also gives the response 

rates for the 10 experimental groups. Keeping in mind the caveat given earlier about the potential effects 

of the second refusal conversion Priority Mailing (sent to subsamples of groups 1, 2, 3, 6, 9, and 10), 

these rates are examined. The unit response rates for the two groups with no incentive (groups 1 and 4) 

are generally lower than the rates for the other groups. Ordering the groups from those sent no monetary 
Monetary Incentives and Mailing Procedures 

14 

incentive to those sent either $4 or $5 (groups 8, 9, and 10), it can be seen that the higher incentives do 

result in higher response rates (table 5), but with diminishing returns. Figure 3 illustrates this trend. The 

effect of the refusal conversion mailings is discussed later. 

Table 5. Screener response rates and ever refusal rates, by incentive group: 2003 

Incentive group Sample size1 Response rate, percent Ever refused rate, percent

 Total ............................... 56,848 — —

Total mailable cases................. 47,106 — —

 1 - ($0 brochure/1st $0)........ 4,574 64.1 (0.67) 48.8 (0.76)

 2 - ($0 /1st $2)...................... 4,524 67.3 (0.71) 47.1 (0.84)

 3 - ($0 /1st $5)...................... 4,518 69.5 (0.64) 45.4 (0.74)

 4 - ($0 /Priority $0) .............. 5,422 63.7 (0.69) 46.6 (0.69)

 5 - ($0 /Priority $2) .............. 4,543 66.7 (0.71) 45.2 (0.83)

 6 - ($2 /1st $0)...................... 5,424 67.9 (0.67) 40.9 (0.67)

 7 - ($2 /Priority $0) .............. 4,558 68.9 (0.72) 42.1 (0.69)

 8 - ($2 /Priority $2) .............. 4,498 69.1 (0.76) 42.4 (0.81)

 9 - ($5 /1st $0)...................... 4,505 69.7 (0.70) 38.9 (0.76)

 10 - ($2 /1st $2).................... 4,540 69.9 (0.56) 42.1 (0.70)

Total nonmailable cases........... 9,742 49.3 (0.59) 55.9 (0.64)

— Not reported. 

1 The sample size is the number of sampled telephone numbers excluding those numbers that were classified as nonresidential. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. Nonresidential telephone numbers are excluded from the table. The sample sizes are 

unweighted. The rates are calculated after weighting cases in the sample for differential probabilities of selection; therefore, rates 

cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 
Monetary Incentives and Mailing Procedures 

15 

Figure 2. Screener response rates and ever refusal rates, by incentive group: 2003 

42.1

38.9

42.4

42.1

40.9

45.2

46.6

45.4

47.1

69.9

69.7

69.1

68.9

67.9

66.7

63.7

69.5

67.3

64.1

48.8

30 35 40 45 50 55 60 65 70 75

10 - ($2/1st $2)

9 - ($5/1st $0)

8 - ($2/Priority $2)

7 - ($2/Priority $0)

6 - ($2/1st $0)

5 - ($0/Priority $2)

4 - ($0/Priority $0)

3 - ($0/1st $5)

2 - ($0/1st $2)

1 - ($0 brochure/

1st $0)

Incentive group

Rate (%)

Unit response

rate (%)

Ever refused 

rate (%)

NOTE: The incentive group descriptions give the advance mailing condition before the / mark and the initial refusal condition 

after the / mark. After the / mark, 1st means the initial refusal materials were sent by first class mail. After the / mark, Priority 

means the initial refusal materials were sent by Priority Mail. All advance letters were sent by first class mail in a U.S. 

Department of Education business envelope, as were initial refusal letters in conditions specifying first class mail. The letters sent 

by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters included a colored NHES project 

brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

0 
Monetary Incentives and Mailing Procedures 

16 

Table 6 combines groups with a similar advance mailing treatment. group 1 is included with the 

other groups with no incentive. The ever refusal rates are the most important rates in this table since the 

response rates combine groups with different subsequent treatments. The ever refusal rates for the three 

groups differ, with the highest rates associated with no incentive and the lowest rates for the $5 incentive 

group.1

 One way of thinking about the relationship is that adding the $2 incentive reduces the ever refusal 

rate by 2.4 percentage points for each dollar relative to no incentive, while adding $5 reduces the refusal 

rate by 1.5 percentage points for each dollar relative to no incentive. Thus, while the larger incentive is 

more effective in the experiment, the relative effectiveness decreases when more money is sent, at least 

for the dollar values considered here. 

Table 6. Screener response rates and ever refusal rates, by advance letter incentive: 2003 

Advance letter 

incentive Sample size1 Response rate, percent Ever refused rate, percent

$0 ................................... 23,581 66.1 (0.31) 46.6 (0.30)

$2 ................................... 19,020 68.9 (0.36) 41.8 (0.37)

$5 ................................... 4,505 69.7 (0.70) 38.9 (0.76)

1 The sample size is the number of sampled telephone numbers excluding those numbers that were classified as nonresidential. 

NOTE: Standard error shown in parenthesis. The sample sizes are unweighted. The rates are calculated after weighting cases in 

the sample for differential probabilities of selection; therefore, rates cannot be calculated directly from the sample sizes 

presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Refusal Conversion 

In the previous section, advance mailing conditions were evaluated. Here, treatments involving 

experimental conditions for Screener refusal conversion are considered. These treatments (refusal 

conversion mailings and possible further monetary incentives) were applied only if a household refused to 

complete the Screener. Before discussing the effects of the refusal conversion treatments, it is important 

to remember that most households that completed the Screener never refused (table 7). To help put refusal 

conversion strategies into perspective, analysis is first conducted to show how cases that never refused 

behaved by the kind of incentive they received in the advance mailing (table 8). Next, cases that ever 

refused are studied (table 9). It should be kept in mind that cases that ever refused include those that 

refused more than once. 

 

1

 The ever refusal rate higher for the brochure group than 2 of the other 4 groups that received no initial monetary incentives. However, all groups 

that received some initial monetary incentive (6-10) had lower ever refusal rates than all groups that received no initial monetary incentives (1-5). 

For this reason, groups that received no initial monetary incentives were combined to facilitate comparisons.
Monetary Incentives and Mailing Procedures 

17 

Figure 3. Screener response rates by total incentive amount and incentive group: 2003 

69.7

69.5

69.9

69.1

68.9

67.9

66.7

67.3

63.7

64.1

30 35 40 45 50 55 60 65 70 75

$5 (Group 9)

$5 (Group 3)

$4 (Group 10)

$4 (Group 8)

$2 (Group 7)

$2 (Group 6)

$2 (Group 5)

$2 (Group 2)

$0 (Group 4)

$0 (Group 1)

Incentive group

Unit response rate (%)

NOTE: All advance letters were sent by first class mail in a U.S. Department of Education business envelope, as were initial 

refusal letters in conditions not specifying Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service 

Priority Mail envelopes. Refusal letters included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

0 
Monetary Incentives and Mailing Procedures 

18 

Table 7. Percentage distribution of Screener completes for mailable cases that completed the 

Screener, by refusal conversion stage: 2003 

Stage Sample size1 Percentage distribution

 Total ........................................... 29,196 100.0 (†) 

Never refused................................. 20,501 70.3 (0.28)

First refusal conversion.................. 6,740 23.1 (0.27)

Second refusal conversion ............. 1,955 6.7 (0.14)

† Not applicable. 

1

 The sample size is the number of mailable telephone numbers with completed Screeners. 

NOTE: Nonresidential telephone numbers are excluded from the table. Percents may not sum to 100 because of rounding. The 

sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for differential probabilities of 

selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Table 8 shows that, among cases that never refused to complete the Screener, the percent that 

completed the Screener was lower for cases that received no incentive in the advance mailing than it was 

for cases that did. The cases that did not complete the Screener could be nonresidential cases that never 

answered the phone, or other nonresponse.2

Table 8. Percentage of cases with completed Screeners that never refused, by advance letter 

incentive: 2003 

Advance letter 

incentive 

Sample 

size1

 Percent completed Mean total calls2

Mean calls to 

complete2

$0 ............ 18,927 57.0 (0.37) 5.5 (0.06) 3.8 (0.05)

$2 ............ 16,158 60.2 (0.41) 5.2 (0.05) 3.7 (0.04)

$5 ............ 3,955 61.7 (0.73) 4.9 (0.12) 3.6 (0.11)

1

 Sample size is the number of telephone numbers at which a refusal was never received, minus numbers identified as 

nonresidential prior to calling (i.e., through prefield activities). 2

 The total calls estimates include calls to all telephone numbers in the subgroup, while the calls to complete estimates include 

calls to only the telephone numbers for which completed Screeners were obtained. 

NOTE: Standard error shown in parenthesis. The $0 advance letter incentive category combines incentive groups 1, 2, 3, 4, and 5. 

The $2 advance letter incentive category combines incentive groups 6, 7, 8, and 10. The $5 advance letter incentive category is

incentive group 9. The sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for 

differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

 

2 There were 16,731 nonmailable cases that never refused and 15.6 percent of these completed the Screener (s.e. = 0.32 percent). The mean 

numbers of calls for these cases were 4.6 for all cases and 4.3 for completed cases (with standard errors of 0.04 and 0.11, respectively). 
Monetary Incentives and Mailing Procedures 

19 

Table 9 includes all cases that ever refused to complete the Screener; thus, it excludes all the cases 

shown in table 8. The percentage completed (those cases that eventually completed a Screener) is an 

overall measure of the joint effectiveness of the first and second refusal conversion attempts. Several of 

the groups with larger completion rates in the table are those that had Priority Mailings that were sent to 

75 percent of the second refusals (groups 1, 2, 3, 6, 9, and 10 [data not shown]). Because of the 

confounding of the two refusal conversion treatments, these results are analyzed separately. 

Table 9. Percentage of cases that ever refused to complete the Screener and final disposition, by 

incentive group: 2003 

Final disposition Incentive group Sample size1

Percent completed Percent refused

 Total ........................................... 23,230 — —

Total mailable cases............................. 20,259 — —

 1 - ($0 brochure/1st $0).................... 2,166 42.9 (1.17) 50.6 (1.05)

 2 - ($0 /1st $2).................................. 2,088 47.2 (1.14) 45.6 (1.08)

 3 - ($0 /1st $5).................................. 2,023 49.1 (1.16) 43.8 (1.15)

 4 - ($0 /Priority $0) .......................... 2,486 39.5 (1.17) 53.7 (1.16)

 5 - ($0 /Priority $2) .......................... 2,007 44.2 (1.16) 48.8 (1.18)

 6 - ($2 /1st $0).................................. 2,188 40.0 (1.07) 52.0 (1.08)

 7 - ($2 /Priority $0) .......................... 1,850 44.3 (1.25) 49.7 (1.25)

 8 - ($2 /Priority $2) .......................... 1,839 44.8 (1.28) 49.3 (1.38)

 9 - ($5 /1st $0).................................. 1,734 40.8 (1.32) 51.7 (1.25)

 10 - ($2 /1st $2)................................ 1,878 45.1 (1.08) 47.3 (1.23)

Total nonmailable cases....................... 2,971 28.2 (0.89) 55.9 (1.02)

— Not reported. 

1

 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. Details do not sum to 100 percent because of the exclusion of the “Other 

nonresponse” category. The sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for 

differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Table 10 gives the mean number of calls needed to reach the final disposition status for Screener 

refusal conversion cases by the experimental groups. The values in the table show little variation by 

experimental group in the mean total calls, the mean calls to obtain a completed Screener, and the mean 

calls to obtain a final refusal. Table 11 gives more details on the mean number of call attempts, dividing 

the total call attempts into those used to obtain the first refusal and those used to finalize the initial refusal 

case as either a completed Screener or final refusal. Even with the added detail, the mean numbers of calls 

generally did not vary across experimental groups. 
Monetary Incentives and Mailing Procedures 

20 

Table 10. Mean number of calls to final status for Screeners that ever refused, by incentive group: 

2003 

Incentive group 

Sample 

size1 Mean total calls

Mean calls to 

complete 

Mean calls to 

refusal

1 - ($0 brochure /1st $0)......... 2,166 10.1 (0.17) 7.4 (0.18) 12.5 (0.24)

2 - ($0 /1st $2)........................ 2,088 10.0 (0.20) 7.1 (0.20) 13.0 (0.31)

3 - ($0 /1st $5)........................ 2,023 9.8 (0.18) 6.8 (0.17) 13.2 (0.26)

4 - ($0 /Priority $0) ................ 2,486 10.1 (0.15) 7.0 (0.16) 12.5 (0.23)

5 - ($0 /Priority $2) ................ 2,007 10.3 (0.20) 7.2 (0.18) 13.1 (0.32)

6 - ($2 /1st $0)........................ 2,188 10.0 (0.18) 7.0 (0.19) 12.7 (0.24)

7 - ($2 /Priority $0) ................ 1,850 10.0 (0.19) 7.1 (0.18) 12.6 (0.29)

8 - ($2 /Priority $2) ................ 1,839 10.1 (0.21) 6.8 (0.18) 13.3 (0.33)

9 - ($5 /1st $0)........................ 1,734 10.5 (0.18) 7.0 (0.22) 13.4 (0.28)

10 - ($2 /1st $2)...................... 1,878 10.1 (0.19) 7.0 (0.20) 13.1 (0.30) 1

 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Table 11. Mean number of calls for initial refusal Screener cases, by outcome of call and incentive 

group: 2003 

Mean number of calls 

Incentive group 

Sample 

size1

To initial refusal

From refusal to 

complete 

From first to second 

refusal

1 - ($0 brochure/1st $0)........ 2,166 3.9 (0.10) 4.1 (0.12) 3.3 (0.10)

2 - ($0 /1st $2)...................... 2,088 3.9 (0.11) 3.9 (0.14) 3.7 (0.13)

3 - ($0 /1st $5)...................... 2,023 3.8 (0.10) 3.7 (0.12) 3.5 (0.11)

4 - ($0 /Priority $0) .............. 2,486 3.8 (0.08) 3.8 (0.11) 3.5 (0.10)

5 - ($0 /Priority $2) .............. 2,007 4.0 (0.12) 4.0 (0.13) 3.6 (0.12)

6 - ($2 /1st $0)...................... 2,188 3.7 (0.10) 4.0 (0.12) 3.6 (0.12)

7 - ($2 /Priority $0) .............. 1,850 3.9 (0.10) 3.9 (0.14) 3.7 (0.13)

8 - ($2 /Priority $2) .............. 1,839 4.0 (0.12) 3.6 (0.13) 3.6 (0.12)

9 - ($5 /1st $0)...................... 1,734 3.9 (0.10) 3.9 (0.16) 3.7 (0.14)

10 - ($2 /1st $2).................... 1,878 4.0 (0.12) 3.7 (0.12) 3.6 (0.12) 1

 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 
Monetary Incentives and Mailing Procedures 

21 

Table 12 shows the percentage of cases that refused that were completed after the first refusal 

efforts (called the first refusal conversion rate). These rates vary among the experimental groups from 29 

percent to 39 percent. If the use of Priority Mail at this stage is ignored, the 10 incentive groups can be 

classified into two groups that can be compared by the amount of incentive given in the conversion letter 

($0 or $2). The mean first conversion rate for those in the $0 group (groups 1, 4, 6, 7) is 31.8 percent (s.e. 

= 0.58 percent), and the mean for those in the $2 group (Groups 2, 5, 8, 10) is 35.7 percent (s.e. = 0.54 

percent). The groups with $2 conversion incentives have significantly higher first conversion rates than 

the groups with $0 conversion incentives (Groups 1, 4, 6, 7, and 9). While this analysis does not explicitly 

account for the incentive amount in the initial mailing, the balance of the design supports the conclusion 

that these results are the effect of the refusal conversion incentive amount.3

 “Balance” refers to the fact 

that, excluding incentive groups 3 and 9, half of the four groups with an initial incentive of $0 had a $2 

conversion incentive and half of the four groups with an initial incentive of $2 had a $2 conversion 

incentive. 

Table 12. Screener response rates after first conversion attempt, by incentive group: 2003 

Response rate after first refusal 

conversion1 First refusal conversion rate 

Incentive group Sample size2 Percent Sample size3

 Percent

 Total ................................ 56,848 — 23,230 —

Total mailable cases.................. 47,106 — 20,259 —

 1 - ($0 brochure/1st $0)......... 4,574 57.9 (0.78) 2,166 30.7 (1.20)

 2 - ($0 /1st $2)....................... 4,524 61.9 (0.71) 2,088 36.3 (1.05)

 3 - ($0 /1st $5)....................... 4,518 64.6 (0.76) 2,023 38.8 (1.23)

 4 - ($0 /Priority $0) ............... 5,422 59.9 (0.70) 2,486 31.8 (1.11)

 5 - ($0 /Priority $2) ............... 4,543 62.4 (0.68) 2,007 35.1 (1.06)

 6 - ($2 /1st $0)....................... 5,424 63.4 (0.69) 2,188 29.5 (1.06)

 7 - ($2 /Priority $0) ............... 4,558 65.0 (0.78) 1,850 35.1 (1.26)

 8 - ($2 /Priority $2) ............... 4,498 65.6 (0.73) 1,839 36.7 (1.10)

 9 - ($5 /1st $0)....................... 4,505 65.3 (0.74) 1,734 29.9 (1.30)

 10 - ($2 /1st $2)..................... 4,540 65.4 (0.64) 1,878 34.9 (1.10)

Total nonmailable cases............ 9,742 45.6 (0.63) 2,971 21.1 (0.85) 1 Screeners that were converted on second refusal conversion attempts are counted as nonrespondents in this table. 2

 For mailable cases, the sample size is the number of telephone numbers assigned to each experimental condition, minus 

numbers identified as nonresidential. For nonmailable cases, the sample size is the total number of telephone numbers, minus 

numbers identified as nonresidential. 

3

 Sample size is the number of telephone numbers at which a refusal was received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. The sample sizes are unweighted. The rates are calculated after weighting cases in the 

sample for differential probabilities of selection; therefore, rates cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

 

3

 A regression analysis, discussed in appendix B, also supports this conclusion. 
Monetary Incentives and Mailing Procedures 

22 

Table 12 also shows the response rates computed after the first refusal conversion. (If a Screener 

was completed in the second refusal conversion attempt, it is classified as a refusal for this table in order 

to assess the results of the initial refusal treatment.) This table provides support for the combined effects 

of the advance letter mailing treatments and the first refusal conversion treatments, that is, without the 

effect of the second refusal conversion treatment. An examination of the response rates by group given in 

table 12 reveals many of the same patterns discussed earlier in conjunction with table 6. The two groups 

that did not include any payment generally have the lowest response rates.4

 The response rates for groups 

3, 7, 8, 9, and 10 are all close to 65 percent. Of these five groups, only group 7 involved sending the 

household less than $4 and group 7 had a Priority Mailing at refusal conversion. These findings are 

consistent with previous research in that they show these monetary incentives improve response rates and 

that higher levels of payment ($4 and $5 are still very modest payments) are typically more effective than 

$2. 

Priority Mail 

Refusing households were sent a refusal conversion letter using Priority Mail after they initially 

refused the screener in several of the groups with the hope that Priority Mail would distinguish the letter 

from other types of mail and convey the importance attached to responding to the survey. If a household 

refused twice and was in a group that had no previous Priority Mailing (groups 1, 2, 3, 6, 9, and 10), then 

75 percent of the households in the group were randomly sampled and these households were sent a third 

mailing using Priority Mail. The remaining 25 percent (like the households in the other groups) were not 

sent any mailing. A discussion of the effectiveness of the initial refusal Priority Mailing is provided 

below, followed by an analysis of the second refusal conversion Priority Mailing in NHES:2003. 

The response rates in table 12 can be used to examine the effectiveness of Priority Mail in the first 

refusal conversion attempt. For example, groups 2 and 5 have the same advance mailing treatment ($0 in 

a first class mail) and the same monetary incentive ($2) for the refusal conversion attempt. They differ 

only in that the refusal conversion letter for group 2 was sent by first class while for group 5 it was sent 

by Priority Mail. The four valid comparisons are: group 1 versus group 4, group 2 versus group 5, group 6 

versus group 7, and group 8 versus group 10. No differences in unit response rates for these contrasts 

were detected. Thus, the experiment shows that using Priority Mail in the initial refusal conversion letter 

does not improve response rates in NHES, holding the other conditions constant. 

Priority Mail was used for the second refusal conversion in order to examine whether its use 

improved the conversion rate. At the second refusal conversion stage, letters were sent by Priority Mail to 

a 75 percent random sample of cases in the groups that had not previously been sent Priority Mail. Those 

that had received a letter by Priority Mail during initial refusal conversion were not sent another letter at 

this stage. Table 13 shows the percentage of the second refusal conversions that were completed for each 

of the six groups by whether the case was in the 75 percent sample sent a Priority Mail letter. Among the 

six groups, the maximum difference in the completion percentage for households sent the Priority Mail 

letter and those not sent any letter was 15 percentage points. When the treated groups are aggregated into 

one category, the difference in the completion percentage between those sent a letter by Priority Mail and 

those not sent any mail in the second refusal conversion is statistically significant. 

 

4

 There was no detectable difference in the rate for group 4 compared to groups 2 and 5 after adjusting for multiple comparisons. 
Monetary Incentives and Mailing Procedures 

23 

Table 13. Percentage of cases that completed Screeners after second refusal conversion, by 

Priority Mail and incentive group: 2003 

No Priority Mail Priority Mail 

Incentive group Sample size1 Percent Sample size1

 Percent

1 - ($0 brochure/1st $0)............ 286 10.5 (1.77) 915 25.2 (1.31)

2 - ($0 /1st $2).......................... 247 18.9 (2.87) 772 23.1 (1.75)

3 - ($0 /1st $5).......................... 241 16.8 (2.71) 694 23.7 (1.54)

6 - ($2 /1st $0).......................... 303 13.3 (1.97) 877 20.8 (1.51)

9 - ($5 /1st $0).......................... 214 15.6 (2.88) 710 21.1 (1.55)

10 - ($2 /1st $2)........................ 215 16.9 (2.78) 714 21.4 (1.74)

Total nonmailable cases........... 1,695 12.4 (0.83) † †

† Not applicable. 

1

 The sample size is the number of sampled telephone numbers that refused two times. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. The sample sizes are unweighted. The percentages are calculated after weighting 

cases in the sample for differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample 

sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Thus, sending a priority letter after a second Screener refusal conversion does improve response 

rates in NHES, but the difference is not uniformly large and depends on previous treatments. Also, notice 

that the difference is between sending a Priority Mail letter and sending no mail, where in all the other 

situations considered the differences were between two different types of mailings. The data do not 

indicate whether sending Priority Mail at this refusal conversion stage would be more effective than 

sending a first class letter.5

 Since only about 7 percent of the completes for those with an address match 

are obtained in the second refusal conversion attempt, the evidence suggests that using Priority Mail at the 

second refusal conversion if there are no previous Priority Mailings seems likely to improve response 

rates, but not by very much. The primary reason the improvement in response rates is expected to be 

small is that the subgroup of cases who refuse twice and are mailable is small; additionally, the 

differences in second refusal conversion rates among all of the experimental groups, though statistically 

significant in some cases, are small. Because no first class letters were sent at the second refusal stage, it 

is not possible to ascertain whether a first class mailing would have affected the conversion rates in the 

same way. 

Tables 14 and 15 present more data on the dispositions and mean number of calls for Screener 

cases after second refusal conversion attempts, showing all the experimental groups. Table 14 shows the 

percentage of the second refusal cases that were completed and refused for each group (a small 

percentage fell into other disposition categories and are not shown). When considering the results in table 

14, it is important to keep in mind that Priority Mail letters were sent after the second refusal only to 

households in those incentive groups that had not previously been sent a Priority Mail letter. Although 

some of the groups do have statistically significantly different percentages completed, none of the 

 

5

 Since all households with mailable addresses had already been sent at least a first class letter, it was believed that after the second refusal, it 

would generally be necessary to do something more to capture their attention, and the use of Priority Mail was the “attention-getter” that was 

tested. 
Monetary Incentives and Mailing Procedures 

24 

differences in the table would have a large effect on overall response rates. The mean number of calls 

given in table 15 are not statistically different and do not suggest any major differences in the level of 

effort for the second refusal Screener cases. 

Table 14. Percentage of Screeners that refused twice, by final disposition and incentive group: 

2003 

Incentive group Sample size1 Percent completed Percent refused

1 - ($0 brochure/1st $0).............. 1,245 21.2 (1.10) 74.7 (1.16)

2 - ($0 /1st $2)........................... 1,062 21.5 (1.41) 73.9 (1.59)

3 - ($0 /1st $5)........................... 982 21.4 (1.26) 75.3 (1.35)

4 - ($0 /Priority $0) ................... 1,349 14.0 (1.08) 83.2 (1.18)

5 - ($0 /Priority $2) ................... 1,019 18.2 (1.15) 79.6 (1.26)

6 - ($2 /1st $0)........................... 1,228 18.7 (1.22) 78.2 (1.30)

7 - ($2 /Priority $0) ................... 977 17.1 (1.30) 80.3 (1.20)

8 - ($2 /Priority $2) ................... 909 16.3 (1.42) 81.1 (1.59)

9 - ($5 /1st $0)........................... 959 19.3 (1.37) 76.3 (1.53)

10 - ($2 /1st $2)......................... 970 19.8 (1.39) 76.3 (1.40) 1 Sample size includes telephone numbers at which two refusals were received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. groups 1, 2, 3, 6, 9, and 10 were sent a Priority Mail letter prior to second refusal

conversion efforts. Details do not sum to 100 percent because of the exclusion of the “Other nonresponse” category. The sample 

sizes are unweighted. The percentages are calculated after weighting cases in the sample for differential probabilities of selection; 

therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 
Monetary Incentives and Mailing Procedures 

25 

Table 15. Mean number of calls between refusals for second Screener refusal cases, by incentive 

group: 2003 

Mean number of calls 

Incentive group 

Sample 

size1

From 

first to second 

refusal

From second refusal 

to complete 

From second to 

final refusal

1 - ($0 brochure/1st $0)........ 1,245 3.3 (0.10) 2.5 (0.10) 3.2 (0.08)

2 - ($0 /1st $2)...................... 1,062 3.7 (0.13) 2.6 (0.12) 3.2 (0.10)

3 - ($0 /1st $5)...................... 982 3.5 (0.11) 2.5 (0.14) 3.4 (0.11)

4 - ($0 /Priority $0) .............. 1,349 3.5 (0.10) 2.5 (0.15) 3.2 (0.10)

5 - ($0 /Priority $2) .............. 1,019 3.6 (0.12) 2.7 (0.16) 3.2 (0.10)

6 - ($2 /1st $0)...................... 1,228 3.6 (0.12) 2.3 (0.11) 3.4 (0.08)

7 - ($2 /Priority $0) .............. 977 3.7 (0.13) 2.4 (0.15) 3.2 (0.10)

8 - ($2 /Priority $2) .............. 909 3.6 (0.12) 2.5 (0.15) 3.2 (0.11)

9 - ($5 /1st $0)...................... 959 3.7 (0.14) 2.4 (0.15) 3.3 (0.09)

10 - ($2 /1st $2)..................... 970 3.6 (0.12) 2.5 (0.14) 3.3 (0.11) 1

 Sample size includes telephone numbers at which two refusals were received. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Carryover Effect of Screener Treatments on Extended Interview Cooperation 

The last issue to be considered in connection with the Screener experiment is whether the 

treatments at the Screener level influenced cooperation at the extended interview level for PFI or AEWR. 

Singer, Groves, and Corning (1999) discuss the hypothesis that once an incentive has been paid, it 

continues to have an effect on the respondents for other survey-related tasks such as completing the 

extended interview. 

To examine the carryover effect the Screener incentive has on extended interview cooperation, 

table 16 shows the percentage of PFI and AEWR sampled persons that completed the extended interview 

for each of the Screener experimental groups and for cases without addresses. The percent complete for 

the PFI are more informative because the adult education survey percent completes are confounded with 

the experimental treatments applied in the adult education survey sample. The unit response rates for PFI 

vary somewhat by the experimental group, from a low of 80 percent for group 5 to a high of 89 percent 

for group 8. The adult education survey unit response rates are not measurably different; they are all 

within 5 percentage points of each other. When the groups are ranked by the percentage completed, the 

groups with higher extended unit response rates are predominately those groups in which the household 

received greater incentives, but the pattern is not consistent. See figure 4 for an illustration. For example, 

group 6 has the third lowest percentage completed for PFI and the third highest percentage completed for 

adult education survey. The relatively small differences and lack of a pattern relative to the payment 

amount across most of the groups does not lend much support to the hypothesis that incentives at the 

Screener will result in significantly higher unit response rates for the extended interview. 
Monetary Incentives and Mailing Procedures 

26 

Table 16. Percentage of completed PFI and AEWR interviews, by incentive group: 2003 

PFI AEWR 

Incentive group Sample size1

Percent 

complete Sample size2

Percent 

complete

 Total ............................ 14,805 — 15,800 —

Total mailable cases.............. 13,306 84.4 (0.44) 14,487 77.2 (0.47)

 1 - ($0 brochure/1st $0)..... 1,183 82.3 (1.58) 1,392 76.7 (1.50)

 2 - ($0 /1st $2)................... 1,320 85.6 (1.28) 1,382 75.9 (1.39)

 3 - ($0 /1st $5)................... 1,283 86.9 (1.23) 1,409 78.7 (1.43)

 4 - ($0 /Priority $0) ........... 1,501 84.2 (1.37) 1,547 74.6 (1.47)

 5 - ($0 /Priority $2) ........... 1,347 79.6 (1.38) 1,354 75.7 (1.64)

 6 - ($2 /1st $0)................... 1,497 84.0 (1.37) 1,664 79.1 (1.25)

 7 - ($2 /Priority $0) ........... 1,350 85.0 (1.25) 1,413 76.4 (1.47)

 8 - ($2 /Priority $2) ........... 1,215 89.4 (1.24) 1,459 79.3 (1.45)

 9 - ($5 /1st $0)................... 1,331 84.2 (1.49) 1,439 76.7 (1.60)

 10 - ($2 /1st $2)................. 1,279 85.6 (1.44) 1,428 79.2 (1.45)

Total nonmailable cases........ 1,499 78.0 (1.35) 1,313 70.4 (1.65)

— Not reported. 

1

 Sample size is the number of children sampled for PFI, by Screener experimental group. 2 Sample size is the number of adults sampled for AEWR, by Screener experimental group. 

NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the advance 

mailing condition (before the / mark) and the initial refusal condition (after the / mark). All advance letters were sent by first 

class mail in a U.S. Department of Education business envelope, as were initial refusal letters in conditions not specifying 

Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters 

included a colored NHES project brochure. PFI is the Parent and Family Involvement in Education Survey of the National 

Household Education Surveys Program, 2003. AEWR is the Adult Education for Work-Related Reasons Survey of the National 

Household Education Surveys Program, 2003. The sample sizes are unweighted. The percentages are calculated after weighting 

cases in the sample for differential probabilities of selection; therefore, percentages cannot be calculated directly from the sample 

sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Parent and Family Involvement in Education 

Survey (PFI) and the Adult Education for Work-Related Reasons Survey (AEWR) of the National Household Education Surveys 

Program, 2003. 
Monetary Incentives and Mailing Procedures 

27 

Figure 4. Percentage of completed PFI and AEWR interviews, by incentive group: 2003 

84.4

85.6

84.2

85

84

79.6

84.2

85.6

82.3

77.2

79.2

76.7

79.3

76.4

79.1

75.7

74.6

78.7

75.9

76.7

86.9

89.4

55 60 65 70 75 80 85 90

Total 

10 - ($2 /1st $2) 

9 - ($5 /1st $0) 

8 - ($2 /Priority $2) 

7 - ($2 /Priority $0) 

6 - ($2 /1st $0) 

5 - ($0 /Priority $2) 

4 - ($0 /Priority $0) 

3 - ($0 /1st $5) 

2 - ($0 /1st $2) 

1 - ($0 brochure/

1st $0) 

Incentive group

Unit response rate (%)

AEWR

PFI

NOTE: The incentive group descriptions give the advance mailing condition before the / mark and the initial refusal condition 

after the / mark. After the / mark, 1st means the initial refusal materials were sent by first class mail. After the / mark, Priority 

means the initial refusal materials were sent by Priority Mail. All advance letters were sent by first class mail in a U.S. 

Department of Education business envelope, as were initial refusal letters in conditions specifying first class mail. The letters sent 

by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. Refusal letters included a colored NHES project 

brochure. PFI is the Parent and Family Involvement in Education Survey of the National Household Education Surveys Program, 

2003. AEWR is the Adult Education for Work-Related Reasons Survey of the National Household Education Surveys Program, 

2003. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Parent and Family Involvement in Education 

Survey (PFI) and the Adult Education for Work-Related Reasons Survey (AEWR) of the National Household Education Surveys 

Program, 2003. 

0 
Monetary Incentives and Mailing Procedures 

28 

4. Adult Education for Work-Related Reasons Survey Experiment 

A second experiment was designed and implemented to examine the effect of incentives and 

mailing strategies in improving response rates for the adult education interview. Once the NHES Screener 

was completed, the interviewer immediately asked to speak with a person sampled for an extended 

interview. As noted earlier, this experiment covered only persons sampled for the adult education survey 

because response rates for this survey were expected to be considerably lower than for the PFI. The 

sampled adults were included in the experiment if they either refused or were classified as having reached 

maximum call status.1

 Like the Screener experiment, the adult education survey experiment was restricted 

to persons with addresses because the treatments required sending a letter to the household; for the adult 

education survey it was further restricted to those cases that did not respond (specifically refused or were 

maximum calls). 

The treatment varied the type of letter mailed and the monetary incentive. Table 17 shows the 

number of adults assigned to each of the three different incentive groups by whether they were included 

because they refused or reached maximum call status. The sample sizes were developed assuming a 

baseline refusal conversion rate of 20 percent, based on results from the 2001 survey. This rate would 

yield groups large enough to detect differences in conversion rates of 5 percentage points. This would 

allow for examination of the effect of a $2 incentive and the effect of Priority Mail on refusal conversion 

and the completion of maximum call cases. 

As shown in table 17, a total of 3,123 adults sampled for the adult education survey with addresses 

were included in the experiment, with 1,975 who were initial refusals and 1,148 who were maximum 

calls. (Overall, 16,004 adults were sampled for the adult education survey.) Both initial refusals and 

maximum calls were randomly assigned to one of the three groups. Cases with any other status were not 

included in the incentive experiment. The random assignment was not implemented until January 29, 

2003, a few weeks after the start of the field period. As a result, 202 adult education survey cases were 

not assigned to an incentive group because they were called before January 29. These cases are excluded 

from all analysis. Table 5 also gives the estimated percent of persons in each of the incentive groups. The 

estimate is weighted by the base weight of the sampled adult that includes the household level weight and 

the inverse of the probability of selecting the adult within the household. This weight is used to produce 

the estimates for all the adult education survey experimental findings. 

Like the Screener, second refusal conversions were attempted for adults who refused the adult 

education survey a second time. In an effort to improve the survey unit response rate, the cases that had 

not been sent Priority Mail previously were sent Priority Mail with a refusal conversion letter. If the 

household had been sent Priority Mail at the Screener level, then it was excluded from the subgroup sent a 

Priority Mail at the second refusal conversion stage in the adult education survey. Thus, the experiment 

only evaluates the results of the first refusal conversion attempt. The goal of this special mailing was to 

increase the adult education survey response rate. 

A total of 12,725 adult education interviews were completed in NHES:2003. In most of these 

interviews (11,554), the respondent never refused to participate. The remaining 1,171 adult education 

interviews were completed on either the first refusal conversion attempt (869) or the second refusal 

conversion attempt (302). 

The adult education survey experiment involved only those cases that ever refused or those that had 

so many call attempts that they were classified into the maximum call disposition and then released for 

 

1 The number of calls required to classify an interview as a maximum call is 9, but some cases do not get classified until at least 14 calls 

depending on the specific outcomes of the case. 
Monetary Incentives and Mailing Procedures 

29 

more call attempts. Table 18 shows there was a total of 3,123 sampled persons included in the experiment, 

with 1,975 entering because of refusals and 1,148 because of the maximum call disposition. 

Table 17. Sample sizes and weighted percents for Adult Education for Work-Related Reasons 

experimental conditions: 2003 

Sample size1

Incentive group Total Refusals

Maximum 

calls 

Weighted 

percent of total

Total mailable cases.................................... 3,123 1,975 1,148 100.0

 A – First class letter, $0 incentive........... 1,078 703 375 34.2

 B – First class letter, $2 incentive........... 1,045 642 403 32.4

 C – Priority Mail letter, $0 incentive ...... 1,000 630 370 33.4

1 The sample size is the number of AEWR cases that received initial refusals or reached maximum call status. 

NOTE: For the AEWR incentive experiment, group A is the control group and groups B and C are two experimental groups. 

AEWR is the Adult Education for Work-Related Reasons Survey of the National Household Education Surveys Program, 2003. 

The sample sizes are unweighted. The percentages are calculated after weighting cases in the sample for differential probabilities 

of selection; therefore, percentages cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Table 18 gives the final disposition (complete, other nonresponse, and refusal) that accounts for 

nearly all sampled adults in the experiment. (A total of 17 adults who were out of scope are not in one of 

these three categories.) Using the percent completed as a measure of effectiveness, the table shows none 

of the three treatments resulted in a significant increase in completion rates, whether examining the 

differences overall or within the refusal and maximum call categories separately. Thus, contrary to the 

effects shown above for the Screener, even the $2 incentive in the refusal conversion did not result in a 

completion rate that was detectably higher. Table 19 shows that the mean number of call attempts is also 

not significantly affected by the treatment for either the refusals or the maximum call sampled adults. 

As described earlier in this section, all sampled adults in households that had never been sent a 

Priority Mailing and refused the adult education survey a second time were sent a letter by Priority Mail. 

Adults who were not eligible for a Priority Mailing in the adult education survey were in households that 

previously refused at the Screener, otherwise they would have been eligible for the adult education survey 

special mailing. Adults who were eligible for a Priority Mailing in the adult education survey were a mix 

of households that had not refused the Screener and those that had refused the Screener but were in 

Screener experimental groups that did not have a Priority Mail treatment. Adults in households that did 

not refuse the Screener are probably more likely to complete the extended interview. The adults in the 

two groups are not comparable and were not randomly assigned to treatments. Consequently, examination 

of results on the use of Priority Mail in the second refusal conversion attempt of the adult education 

survey has limited value. However, analysis of outcomes for the two groups may suggest some additional 

experimentation. 

Table 20 shows the results of the second refusal conversion attempt for the three groups by 

whether they were sent Priority Mail. Since treatment group C had all been sent Priority Mail in the adult 

education survey experiment, the outcome for all these are on one row of the table. The first two rows are 

the estimates for the adults who had been mailed only a first class letter in the first refusal conversion 

attempt. The second set of rows contains estimates for those who had been mailed a first class letter with 

$2 in the first refusal conversion attempt. The percentage complete for adults sent Priority Mail is at least 
Monetary Incentives and Mailing Procedures 

30 

10 percentage points higher than it is for the adults who were not sent any letter at this stage. These 

results suggest it might be useful to test whether a first class letter at the second refusal conversion stage 

is useful. This outcome would be consistent with the findings from the Screener incentive experiment. 

Table 18. Percentage distribution of AEWR final dispositions, by type of nonresponse and 

incentive group: 2003 

Incentive group Sample size1

Percent

complete

Percent other 

nonresponse2

Percent

refusal

 Total ..................... 3,123 40.7 (1.03) 26.3 (0.95) 32.9 (0.94)

A – (1st, $0) ................... 1,078 40.2 (1.94) 26.9 (1.73) 32.9 (1.73)

B – (1st, $2) ................... 1,045 42.8 (1.87) 26.1 (1.58) 31.1 (1.48)

C – (Priority, $0)............ 1,000 39.3 (1.88) 26.0 (1.86) 34.8 (2.02)

Refusals.......................... 1,975 38.2 (1.40) 9.9 (0.94) 51.9 (1.26)

 A – (1st, $0) ............... 703 38.4 (2.37) 10.6 (1.62) 51.0 (2.34)

 B – (1st, $2)................ 642 39.2 (2.35) 9.3 (1.49) 51.5 (2.20)

 C – (Priority, $0)........ 630 37.1 (2.48) 9.6 (1.36) 53.3 (2.64)

Maximum calls............... 1,148 45.0 (1.65) 54.2 (1.66) 0.8 (0.46)

 A – (1st, $0) ............... 375 43.4 (3.17) 56.0 (3.13) 0.6 (0.30)

 B – (1st, $2)................ 403 48.3 (3.14) 51.5 (3.15) 0.2 (0.20)

 C – (Priority, $0)........ 370 43.2 (3.44) 55.1 (3.43) 1.7 (1.19)

1 Sample size includes persons sampled for AEWR who refused or reached maximum call status. 2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent 

(maximum call status), the respondent being unavailable during the field period, or the telephone number being disconnected or 

reassigned before the AEWR interview could be completed. 

NOTE: Standard error shown in parenthesis. Percents may not sum to 100 due to rounding. The parenthetical descriptions of the 

experimental conditions give the type of mail and the incentive. First class mail was sent in a U.S. Department of Education 

business envelope in conditions not specifying Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service 

Priority Mail envelopes. All refusal letters included a colored NHES project brochure. AEWR is the Adult Education for WorkRelated Reasons Survey of the National Household Education Surveys Program, 2003. The sample sizes are unweighted. The 

percentages are calculated after weighting cases in the sample for differential probabilities of selection; therefore, percentages 

cannot be calculated directly from the sample sizes presented. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Education for Work-Related Reasons 

Survey (AEWR) of the National Household Education Surveys Program, 2003.
31

Table 19. Number of calls to final disposition for AEWR initial refusal and maximum call cases, by type of nonrespons

2003 

Incentive group Sample size1

Mean calls to

first refusal

Mean calls to 

complete 

Mean calls to

other nonresponse2

Refusals............................. 1,975 — — —  A – (1st, $0) .................. 703 9.8 (0.42) 6.2 (0.36) 10.6 (1.03)  B – (1st, $2) .................. 642 10.3 (0.40) 5.4 (0.36) 11.0 (1.47)  C – (Priority, $0)........... 630 9.8 (0.37) 6.6 (0.39) 10.2 (1.27) Maximum calls ................. 1,148 † — —  A – (1st, $0) .................. 375 † 21.6 (0.78) 30.6 (0.92)  B – (1st, $2) .................. 403 † 22.1 (0.63) 30.9 (0.63)  C – (Priority, $0)........... 370 † 21.6 (0.71) 30.9 (0.48) — Not reported. † Not applicable. 1 There were nine adults included in the experiment because of their maximum call disposition who eventually refused, but they are not included in this tabu

sample size. 

2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent (maximum call status), the res

the field period, or the telephone number being disconnected or reassigned before the AEWR interview could be completed. NOTE: Standard error shown in parenthesis. The parenthetical descriptions of the experimental conditions give the type of mail and the incentive for the initmail was sent in a U.S. Department of Education business envelope in conditions not specifying Priority Mail. The letters sent by Priority Mail were sent in 

Mail envelopes. All refusal letters included a colored NHES project brochure. AEWR is the Adult Education for Work-Related Reasons Survey of the Natio

Surveys Program, 2003. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Education for Work-Related Reasons Survey (AEWR) of the Nati

Surveys Program, 2003. 
32 Table 20. Number of AEWR second refusal cases, by final disposition, incentive group, and Priority Mail: 2003 

1 The sample size is the number of persons sampled for AEWR who refused twice. 2 Other nonresponse includes cases that finalized due to language problems, the inability to make contact with the respondent (maximum call status), the 

unavailable during the field period, or the telephone number being disconnected or reassigned before the AEWR interview could be completed. 

NOTE: Standard error shown in parenthesis. Percents may not sum to 100 because of rounding. The parenthetical descriptions of the experimental condmailing condition (before the / mark) and the second refusal condition (after the / mark). First class mail was sent in a U.S. Department of Education bus

not specifying Priority Mail. The letters sent by Priority Mail were sent in the U.S. Postal Service Priority Mail envelopes. All refusal letters included a cbrochure. AEWR is the Adult Education for Work-Related Reasons Survey of the National Household Education Surveys Program, 2003. The sample s

percentages are calculated after weighting cases in the sample for differential probabilities of selection; therefore, percentages cannot be calculated direcpresented. SOURCE: U.S. Department of Education, National Center for Education Statistics, Adult Education for Work-Related Reasons Survey (AEWR) of the 

Education Surveys Program, 2003.

Incentive group Sample size1 Percent complete Percent other nonresponse2  A – (1st, $0/no Priority)................ 175 9.1 (3.08) 7.6 (2.13)  A – (1st, $0/Priority)..................... 202 25.7 (3.58) 4.4 (1.66)  B – (1st, $2/no Priority)................ 166 12.0 (2.46) 4.0 (1.40)  B – (1st, $2/Priority)..................... 157 23.8 (3.44) 6.1 (2.35)  C – (Priority, $0/no Priority) ........ 336 14.9 (2.38) 6.1 (1.53) 
Monetary Incentives and Mailing Procedures 

33 

5. Summary 

It has become increasingly difficult to achieve high response rates in sample surveys in recent 

years, even as the level of effort in obtaining completed interviews has increased. Previous experience in 

NHES had demonstrated the effectiveness of an advance mailing (Hagedorn et al., 2003), contrary to the 

findings of Singer, Van Hoewyk, and Maher (2000). One possible reason the advance mailing is effective 

in NHES is that the U.S. Department of Education envelope and letterhead convey a level of legitimacy 

and salience that other forms of mailings may not. Singer, Van Hoewyk, and Maher (2000) used 

University of Michigan mailing envelopes and stationery.) Despite the effectiveness of the advance letter, 

however, Screener unit response rates have fallen over the years in NHES. These declining Screener unit 

response rates in NHES led to the design and execution of an experiment examining the effects of 

respondent incentives as a means to improve response. 

The Screener incentive experiment implemented in the 2003 NHES administration included 10 

conditions with varied combinations of mailing procedures (first class and Priority Mail) and respondent 

incentive amounts ($0, $2, and $5). The experimental conditions were used to study advance mailing 

strategies and refusal conversion strategies. An experiment was also conducted in the Adult Education 

Survey, one of two topical surveys conducted in NHES:2003. Three experimental conditions (first class 

letter advance mailing, $0 incentive; first class letter advance mailing, $2 incentive; and Priority Mail 

advance mailing, $0 incentive) were applied in adult education survey cases where the respondent 

initially refused or had not responded after a large number of calls. 

The results of the Screener incentive experiment indicate that small cash incentives are effective in 

improving NHES Screener unit response. In NHES:2003, refusals occurred less often among those who 

received advance incentives of $2 or $5 than among those who received no incentive; the benefit of the 

incentive in the initial refusal rate was 5 to 7 percentage points, depending on the incentive amount. 

While the rate of refusal was lower for those who received an advance incentive of $5 compared with 

those who received $2, a diminishing effect per dollar of incentive was observed. Additionally, when an 

advance letter with no incentive and a refusal conversion letter with a monetary incentive are used 

(experimental groups 2 and 3), subsampling of refusals for conversion efforts is effective in gaining 

additional response. 

Singer, Van Hoewyk, and Maher (2000) reported a 10 percentage point response rate benefit of an 

advance incentive. This effect is larger than that observed in the NHES:2003 experiment. One reason for 

the difference is that the 10 percentage point effect is only for the cases with addresses, while NHES 

gives the effect over the full sample. Another possible reason is the difference in the saliency and 

sponsorship of the surveys. 

The NHES:2003 experiment also showed respondent incentives were effective in obtaining 

cooperation at the refusal conversion stage for the Screener. Experimental treatment groups that included 

payments were more likely to respond at the first refusal conversion stage compared with those who had 

not received payments. In addition, the advance incentive treatments and refusal incentive treatments 

combined to yield higher response rates after the first refusal conversion stage for those who had received 

payments of $4 (two payments of $2) and those who had received $5 (in either one of the payments). The 

use of Priority Mail did not significantly improve initial refusal conversion rates, contrary to expectations. 

This may reflect the efficacy of conducting first class mailings in U.S. Department of Education 

envelopes. 

At the second refusal conversion stage for the Screener, a letter was sent by Priority Mail to 

households that had refused twice and had not previously been sent a Priority Mail letter. This strategy 

improved the rates of refusal conversion for these groups slightly, but the overall effect on the response 
Monetary Incentives and Mailing Procedures 

34 

rate is marginal because only 7 percent of completed Screeners result from second refusal conversion 

efforts. 

The effect of the Screener incentives on extended interview unit response rates was also 

examined. While, in general, PFI and AEWR unit response rates were higher when households had 

received a Screener incentive, the pattern was not consistent and the effects were not large. The findings 

do not support a hypothesis that the Screener incentives improve extended interview response in NHES. 

An examination of the effect of the incentive treatments on the numbers of calls required to finalize a 

Screener showed no effects that would have implications for the rest of the data collection. 

The Adult Education for Work-Related Reasons Survey experiment showed that the use of a $2 

incentive and the use of Priority Mail did not significantly increase the completion rates for initial refusal 

conversion or for cases with large numbers of contact attempts. In addition, no differences were detected 

in the numbers of calls required to finalize the cases. The use of Priority Mail was found to be effective in 

the second refusal conversion stage for the adult education survey, improving the unit response rate for 

cases that had refused twice by about 10 percentage points. The effect on the unit response rate for the 

entire adult education survey sample, however, was very small. 

 

In addition to the findings presented in this report, additional questions for investigation were 

identified. A test of the relative refusal conversion rates realized with FedEx (rather than Priority Mail) 

versus first class would be useful. It may be that neither special mail procedure is superior to the first 

class mailing in a U.S. Department of Education envelope, but this has yet to be established for FedEx in 

a concurrent experiment. Another question as to mailing procedures is the efficacy of first class mail at 

the second refusal conversion stage. Dillman (2000) suggests that varying mail contacts with the 

household, rather than using the same approach in each contact, is more effective. It may be that a first 

class letter at the second refusal conversion stage would not be effective compared with Priority Mail or 

FedEx, but this has yet to be examined within the context of this survey program. Finally, researchers’ 

experiences with gaining respondent cooperation are not static, and ongoing investigation into the 

efficacy of various methods over time is necessary. 
Monetary Incentives and Mailing Procedures 

35 

References 

The American Association for Public Opinion Research (AAPOR). (2004). Standard Definitions: Final 

Dispositions of Case Codes and Outcome Rates for Surveys. 3rd edition. Lenexa, KS: AAPOR. 

Atrostic, B.K., Bates, N., Burt, G., and Silberstein, A. (2001). Nonresponse in U.S. Government 

Household Surveys: Consistent Measures, Recent Trends, and New Insights. Journal of Official 

Statistics, 17: 209-226. 

Berk, M.L., Mathiowetz, N.A., Ward, E.P., and White, A.A. (1987). The Effect of Prepaid and Promised 

Incentives: Results of a Controlled Experiment. Journal of Official Statistics, 3: 449-457. 

Brick, J.M., and Collins, M. (1997a). An Overview of Response Rates in the National Household 

Education Survey: 1991, 1993, 1995, and 1996. U.S. Department of Education, National Center for 

Education Statistics. NCES 1997-948. Washington, DC. 

Brick, J.M., and Collins, M. (1997b). An Experiment in Random-Digit-Dial Screening. U.S. Department 

of Education, National Center for Education Statistics. NCES 1998-255. Washington, DC. 

Brick, J.M., Montaquila, J., and Scheuren, F. (2002). Estimating Residency Rates for Undetermined 

Telephone Numbers. Public Opinion Quarterly, 66(1): 18–39. 

Church, A.H. (1993). Estimating the Effect of Incentives on Mail Surveys: A Meta-Analysis. Public 

Opinion Quarterly, 57: 62-79. 

Curtin, R., Presser, S., and Singer, E. (2000). The Effects of Response Rate Changes on the Index of 

Consumer Sentiment. Public Opinion Quarterly, 64: 413-428. 

Curtin, R., Presser, S., and Singer, E. (2005). Changes in Telephone Survey Nonresponse Over the Past 

Quarter Century. Public Opinion Quarterly, 69: 87-98. 

Dillman, D.A. (2000). Mail and Internet Surveys: The Tailored Design Method. New York: John Wiley 

& Sons. 

Estrella, A. (1998). A New Measure of Fit for Equations with Dichotomous Dependent Variables. 

Journal of Business and Economic Statistics, 16, 198–205. 

Gelman, A., Stevens, M., and Chan, V. (2003). Regression Modeling and Meta-Analysis for Decision 

Making: Cost-Benefit Analysis of Incentives in Telephone Surveys. Journal of Business & 

Economic Statistics, 21: 213-225. 

Groves, R.M., and Couper, M.P. (1996). Contact-level Influences on Cooperation in Face-to-Face 

Surveys. Journal of Official Statistics, 12:63-83. 

Groves, R.M., and Couper, M.P. (1998). Nonresponse in Household Interview Surveys. New York: John 

Wiley & Sons. 

Groves, R.M., Singer, E., Corning, A.D., and Bowers, A. (1999). A Laboratory Approach to Measuring 

the Effects on Survey Participation of Interview Length, Incentives, Differential Incentives, and 

Refusal Conversion. Journal of Official Statistics, 15: 251-268. 
Monetary Incentives and Mailing Procedures 

36 

Groves, R.M., Singer, E., and Corning, A. (2000). Leverage-Saliency Theory of Survey Participation. 

Public Opinion Quarterly, 64: 299-308. 

Hagedorn, M., Montaquila, J., Kim, K., Li, L., Vaden-Kiernan, N., and Chapman, C. (forthcoming). 

Household Education Surveys Program: 2003: Methodology Report. U.S. Department of 

Education, National Center for Education Statistics, Washington, DC. 

Hagedorn, M., Montaquila, J., Nolin, M.J., Kim, K., Kleiner, B., Waits, T., Chapman, C., and Chandler, 

K. (2003). National Household Education Surveys of 2001: Data File User’s Manual, Volume I.

U.S. Department of Education, National Center for Education Statistics. NCES 2003-079. 

Washington, DC. 

Hagedorn, M., Montaquila, J., Vaden-Kiernan, N., Kim, K., and Chapman, C. (forthcoming). National 

Household Education Surveys of 2003: Data File User’s Manual, Volume I. U.S. Department of 

Education, National Center for Education Statistics. Washington, DC. 

Martin, E. Abreu, D., and Winters, F. (2001). Money and Motive: Effects of Incentives on Panel Attrition 

in the Survey of Income and Program Participation. Journal of Official Statistics, 17: 267-284. 

Maynard, D.W., and Schaeffer, N.C. (1997). Keeping the Gate: Declinations of the Request to Participate 

in a Telephone Survey Interview. Sociological Methods & Research, 26:34-79. 

Nolin, M.J., Montaquila, J., Lennon, J., Kim, K., Kleiner, Chapman, C., Chandler, K., Creighton, S., and 

Bielick, S. (2000). National Household Education Surveys of 1999: Data File User’s Manual, 

Volume I. U.S. Department of Education, National Center for Education Statistics. NCES 2000-

076. Washington, DC. 

Shapiro, G., Battaglia, M., Camburn, D., Massey, J., and Tompkins, L. (1995). Calling Local Telephone 

Company Business Offices to Determine the Residential Status of a Wide Class of Unresolved 

Telephone Numbers in a Random-Digit-Dialing Sample. Proceedings of the Survey Research 

Methods Section of the American Statistical Association, 975–980. 

Shettle, C. and Mooney, G. (1999). Monetary Incentives in U.S. Government Surveys. Journal of Official 

Statistics, 15: 231-250. 

Singer, E. (2002). The Use of Incentives to Reduce Nonresponse in Household Surveys, Chapter 11 in 

Nonresponse in Household Interview Surveys, Eds. (Groves, Dillman, Eltinge, Little), New York: 

John Wiley & Sons. 

Singer, E., Groves, R.M., and Corning, A.D. (1999). Differential Incentives: Beliefs about Practices, 

Perceptions of Equity, and Effects on Survey Participation. Public Opinion Quarterly, 63: 251-260. 

Singer, E., Van Hoewyk, J., Gebler, N., Raghunathan, T., and McGonagle, K. (1999). The Effect of 

Incentives on Response Rates in Interviewer-Mediated Surveys. Journal of Official Statistics, 15: 

217-230. 

Singer, E., Van Hoewyk, J., and Maher, M.P. (2000). Experiments with Incentives on Telephone Surveys. 

Public Opinion Quarterly, 64: 171-188. 

Steeh, C., Kirgis, N., Cannon, B., and DeWitt, J. (2001). Are They Really as Bad as They Seem? 

Nonresponse Rates at the End of the Twentieth Century. Journal of Official Statistics, 17: 227-247. 
Monetary Incentives and Mailing Procedures 

37 

Trussell, N., and Lavrakas, P.J. (2004). The Influence of Incremental Increases in Token Cash Incentives 

on Mail Survey Response: Is There an Optimal Amount? Public Opinion Quarterly, 68(3): 349-

367. 

Westat (2000). WesVar 4.0 User’s Guide. Rockville, MD: Westat. 

Appendix A 

Response Rate Calculation Methods 

Monetary Incentives and Mailing Procedures 

A-3 

Response Rate Calculation Methods 

This appendix describes the calculation of unit response rates for the National Household 

Education Surveys Program (NHES:2003). A unit response rate is the ratio of the number of units with 

completed interviews (e.g., the units could be telephone numbers, households, or persons) to the number 

of units sampled and eligible for the interview. In some cases, these rates are easily defined and 

computed, while in other cases the numerator or denominator of the ratio must be estimated. 

For reporting the results from NHES:2003, the overall unit response rate indicates the percentage of 

possible interviews that were completed taking all survey stages into account, while the unit response rate 

measures the percentage of interviews that were completed for a specific stage of the survey. For example, 

household members were identified for interviews in a two-stage process. Screener interviews were 

conducted to enumerate and sample household members, and then questionnaires were administered for the 

sampled members. If the responding household member failed to complete the first-stage Screener, no 

members could be sampled for other interviews. Under this design, the unit response rate for the second 

stage is the percentage of sampled persons who completed the PFI or AEWR interview. The overall unit 

response rate is the product of the first- and second-stage unit response rates. 

Unit response rates can be either unweighted or weighted. The unweighted rate, computed using 

the raw number of cases, provides a useful description of the success of the operational aspects of the 

survey. The weighted rate, computed by summing the weights (usually the base weights, which are the 

reciprocals of the probability of selecting the units) for both the numerator and denominator, gives a 

better description of the success of the survey with respect to the sampled population since the weights 

allow for inference of the sample data (including response status) to the population. Both rates are usually 

similar unless the probabilities of selection and the response rates in the categories with different selection 

probabilities vary considerably. All of the unit response rates discussed in this report are weighted unless 

noted specifically in the text, since the main purpose of this report is to describe the results of the 

experiment with respect to the survey population. 

In computing unit response rates for an RDD survey, the first concern is to classify each sampled 

telephone number according to its residential status. The three major categories of residential status are 

those identified as numbers for residential households, those identified as nonresidential numbers 

(primarily nonworking and business telephone numbers), and those numbers that, despite numerous 

attempts, could not be classified as either residential or nonresidential. Calculation of unit response rates 

is complex because of the possible ways residential status can be assigned to the latter set of numbers. 

There are various approaches for computing RDD Screener unit response rates. Each of these 

approaches uses the same numerator, the number of households (weighted or unweighted, for the 

weighted and unweighted rates, respectively) that completed the Screener. Variability in the estimates 

arises because it is not possible to identify precisely the residential status for each telephone number. The 

difference among the rates is in the allocation of the numbers in the unknown residential status category 

that is used in the calculation of the denominator of the unit response rate. 

The survival approach uses information about cases for which no answer was obtained in the 

estimation of their residency rate. Specifically, for NHES:2003, the listed status, interviewers’ codings of 

answering machine call attempts, and the total number of call attempts are used in the estimation of the 

residency rate based on survival analysis methods. Estimates based on the survival method suggest that 

19.7 percent of telephone numbers with unresolved residency status in NHES:2003 are residential. 

Therefore, the denominator of the unit response rate based on the survival method is all the telephone 

numbers that were known to be residences plus 19.7 percent of the numbers with an unresolved 

residential status. Because the survival method uses more information about the telephone numbers and 
Monetary Incentives and Mailing Procedures 

A-4 

their call histories, it is a more accurate approach for estimating unit response rates in random digit 

dialing (RDD) surveys. See Brick, Montaquila, and Scheuren (2002) for further details about the survival 

method. 

The business office method derives its name from the technique used to estimate the denominator 

of the rate. A random sample of 350 telephone numbers with unresolved residency status were selected in 

the NHES:1995 and the numbers were classified by local telephone companies. The telephone companies 

were asked to first classify the numbers as working or not working. The companies were asked to further 

identify working numbers as residential or business numbers. As a result of this process, it was estimated 

that 40.5 percent of the numbers were residential. This percentage is nearly identical to the result from a 

study conducted at the end of the NHES:1991. Therefore, the denominator of the unit response rate based 

on the business office method is all the telephone numbers that were known to be residences plus 40.5 

percent of the numbers with an unresolved residential status. Some research suggests that the business 

office approach may be inaccurate due to the reporting practices of the phone companies (Shapiro et al. 

1995). 

Three other unit response rates may be computed by allocating different proportions of the 

numbers with unknown residency status into the residential category. The Council of American Survey 

Research Organizations (CASRO) rate is computed by allocating the numbers with unknown residential 

status in the same proportion observed in the numbers with known residential status, which, in the 

NHES:2003, was 40.4 percent. Evidence from the study of the business office method described earlier 

suggests that the residency rate for numbers with unknown residential status is lower than implied by the 

CASRO rate calculation. Therefore, the CASRO rate is not recommended for unit response rate 

calculations for NHES. 

The conservative and liberal unit response rates define the lower and upper bounds of the unit 

response rate. The conservative unit response rate is computed assuming that all of the numbers with 

unknown residential status are actually residential numbers. The liberal rate is computed assuming that all 

of the numbers with unknown residential status are nonresidential. As noted earlier, the variability in the 

estimates arises because it is not possible to identify precisely the residential status for each telephone 

number. 
Appendix B 

Logistic Regression Analysis
Monetary Incentives and Mailing Procedures 
Monetary Incentives and Mailing Procedures 

B-3 

Logistic Regression Analysis 

To refine the analyses of the incentive experiment results, the initial cooperation and refusal 

conversion rates were examined using logistic regression, where the outcome for a sampled number was 

considered a “success” if the household completed the screener at that stage. After examining many 

variables unrelated to the experiments that were available for all telephone numbers,1

 two good predictors 

of initial cooperation were identified: region of the country, and metropolitan status. Table B-1 shows the 

final model for the initial cooperation rates. The model is parameterized so that the last level of each 

predictor is the reference cell and the parameter for that level is set equal to zero. In general, the 

significance tests for the effects of the predictor variables in the logistic regression model control for the 

effects of other variables in the model. 

In table B-1, the estimated regression coefficients for the $0 and $2 incentive amounts indicate 

that, after controlling for other predictor variables in the model, households receiving these treatments 

have a lower probability of responding initially to the screener than those receiving the $5 incentive. The 

odds ratios show that compared to a household in the $5 incentive group, a household in the $2 incentive 

group is 0.87 times as likely to respond initially, and a household in the $0 incentive group is only 0.69 

times as likely to respond initially. A contrast test shows that the effect of the $0 incentive is statistically 

different from the effect of the $2 incentive. These results are consistent with the results on initial 

cooperation rates given in table 6. 

Table B-1. Logistic regression estimates for initial cooperation 

Parameter Estimate p Odds ratio

Intercept .................................. 0.56 (0.049) 0.000 †

Region (1) ............................... –0.20 (0.020) 0.000 0.82

Metro status (1)....................... –0.13 (0.035) 0.001 0.88

Metro status (2)....................... –0.13 (0.031) 0.000 0.87

Metro status (3)....................... –0.12 (0.038) 0.003 0.89

Metro status (4)....................... –0.29 (0.050) 0.000 0.75

Initial incentive ($0)................ –0.37 (0.034) 0.000 0.69

Initial incentive ($2)................ –0.14 (0.034) 0.000 0.87

† Not applicable. 

NOTE: Standard error shown in parenthesis. The sample size is 39,298 residential telephone numbers, excluding numbers that 

were never completed or refused, such as language problem cases. The modified likelihood ratio statistic proposed by Estrella 

(1998) as a goodness of fit measure for logistic regression models is .008. p is the p-value for the two-sided test that the 

parameter is equal to 0. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

Next, logistic regression was used to examine the first refusal conversion rates. In this analysis, 

predictor variables were included for the two first refusal conversion experimental treatments: whether 

 

1

 The following characteristics were examined: census region (combined Northeast and South/combined Midwest and West, with Midwest and 

West as the reference group), interview language (English/Spanish), presence of person under 21 in household, home ownership 

(own/rented/other arrangement), median home value in telephone exchange, median income in telephone exchange, Metropolitan Statistical Area 

(MSA) status [(1) in county in central city, (2) in county not in central city, (3) subcounty of MSA, (4) MSA in its own county, and (5) not MSA, 

minority stratum, percent of households with incomes of $75,000 to $100,000, percent of households with income above $100,000, percent Asian 

in exchange, percent Black in exchange, percent Hispanic in exchange, percent college graduates in exchange, and percent homeowners in 

exchange. Although all of the characteristics listed were considered, only those found to be significant were included in the final model. 
Monetary Incentives and Mailing Procedures 

B-4 

Priority Mail was sent (no/yes) and the refusal incentive amount ($0/$2/$5). Other models were 

considered (e.g., including an interaction between Priority Mail and a level of the monetary incentive), 

but these are not discussed because they added little to the predictive power of the model. Table B-2 gives 

the estimated regression coefficients and odds ratios. There are three important results from this analysis. 

First, the estimated coefficient for the Priority Mail treatment is not significantly different from zero. 

Second, the monetary incentives at the refusal conversion stage do increase the conversion rates. The $2 

incentive results in a higher conversion rate than no incentive, and $5 is more effective than $2. Third, the 

monetary incentives in the initial mailing do not have a statistically significant effect on the refusal 

conversion rates. In other words, after controlling for other variables in the model, the only treatment at 

the first refusal conversion stage that results in higher rates of completing the screener is the amount of 

money sent prior to the refusal conversion attempt. This analysis supports statements made earlier in the 

tabular analysis. 

Table B-2. Logistic regression estimates for first refusal conversion 

Parameter Estimate p Odds ratio

Intercept .......................................... –0.21 (0.102) 0.041 †

Region (1) ....................................... –0.08 (0.036) 0.032 0.92

Metro status (1)............................... –0.11 (0.044) 0.015 0.90

Metro status (2)............................... –0.01 (0.050) 0.850 0.99

Metro status (3)............................... –0.03 (0.038) 0.416 0.97

Metro status (4)............................... –0.11 (0.078) 0.149 0.89

Initial incentive ($0)........................ 0.04 (0.069) 0.604 1.04

Initial incentive ($2)........................ 0.06 (0.070) 0.411 1.06

First refusal Priority Mail (no)........ –0.07 (0.035) 0.066 0.94

Refusal incentive ($0)..................... –0.37 (0.063) 0.000 0.69

Refusal incentive ($2)..................... –0.18 (0.063) 0.005 0.83

† Not applicable. 

NOTE: Standard error shown in parenthesis. The sample size is 18,797 households that initially refused the screener and either 

completed or refused the conversion. The Estrella goodness of fit statistic is .005. p is the p-value for the two-sided test that the 

parameter is equal to 0. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003. 

The final logistic regression analysis examines the effectiveness of sending Priority Mail at the 

second refusal conversion stage. Since only second refusal households that had not been sent a Priority 

Mail in the first refusal conversion stage were treated, the sample for this analysis only includes groups 1, 

2, 3, 6, 9, and 10. The predictor variable for using Priority Mail at the first conversion stage is dropped. 

Table B-3 gives the estimated regression coefficients and odds ratios that show the use of Priority Mail is 

effective at this stage. A household that was not sent a Priority Mail letter at this stage was only 0.6 times 

as likely to respond as a household that was sent a Priority Mail second refusal conversion letter. 

Interestingly, the incentives at the initial and first refusal conversion stages are not statistically significant, 

again suggesting that there is little or no carry-over effect from incentives at earlier stages. 
Monetary Incentives and Mailing Procedures 

B-5 

Table B-3. Logistic regression estimates for second refusal conversion 

Parameter Estimate p Odds ratio

Intercept .......................................... –0.85 (0.172) 0.000 †

Region(1) ........................................ –0.04 (0.067) 0.599 0.96

Metro status (1)............................... –0.53 (0.089) 0.000 0.59

Metro status (2)............................... –0.30 (0.115) 0.010 0.74

Metro status (3)............................... –0.33 (0.098) 0.001 0.72

Metro status (4)............................... –0.37 (0.172) 0.035 0.69

Initial incentive ($0)........................ 0.10 (0.120) 0.421 1.10

Initial incentive ($2)........................ –0.06 (0.126) 0.640 0.94

Refusal incentive ($0)..................... –0.03 (0.102) 0.774 0.97

Refusal incentive ($2)..................... 0.04 (0.107) 0.740 1.04

Second refusal Priority Mail (no).... –0.51 (0.084) 0.000 0.60

† Not applicable. 

NOTE: Standard error shown in parenthesis. The sample size is 5,951 households that refused the screener twice and had not 

been sent Priority Mail previously and completed or refused on the last attempt. The Estrella (1998) goodness of fit statistic is 

.014. p is the p-value for the two-sided test that the parameter is equal to 0. 

SOURCE: U.S. Department of Education, National Center for Education Statistics, National Household Education Surveys 

Program, 2003.