SIPP 2008 Incentive Analysis 

Stephen M. Clark and Stephen P. Mack 

U.S. Census Bureau, Demographic Statistical Methods Division, Washington, D.C. 20233 

Contact Author: stephen.clark@census.gov 

Abstract: In an effort to determine an effective incentive for increasing response rates in the Survey of Income and Program 

Participation (SIPP), we implemented a randomized experiment in the first and second waves of the 2008 SIPP Panel. To 

contain the costs for the experiment, 50% of the sample received no incentive. Twenty five percent were sent a $20 debit 

card with an advance letter introducing them to the survey. The remaining 25% were eligible for a $40 debit card conditional 

on their participation in the survey. The field staff were given enough $40 debit cards to cover 15% of their workload. For 

the complete first and second waves, we computed and compared the weighted response rates for overall effectiveness for 

each of the individual incentive groups as well as found their associated errors. We also computed the ratios of promised to 

available $40 incentives to try to explain any irregularities within these response rates as well as computed the weighted 

response rates by stratum and incentive. 

Key Words: incentives, nonresponse 

Disclaimer: This paper is released to inform interested parties of ongoing research and to encourage discussion of work in 

progress. Any views expressed are those of the authors and not necessarily those of the U. S. Census Bureau. 

I. Background on the SIPP 

The U.S. Census Bureau sponsors the Survey of Income and Program Participation (SIPP), which surveys households to 

gather information on things like poverty, income, and participation in government programs (for example, food stamps and 

Temporary Assistance for Needy Families (TANF)) to better understand the overall population of the United States of 

America. 

The U.S. Census Bureau has twelve regional offices to coordinate the data collection of various surveys, including SIPP, 

across the United States of America. These are: Boston, New York, Philadelphia, Detroit, Chicago, Kansas City, Seattle, 

Charlotte, Atlanta, Dallas, Denver, and Los Angeles. 

In SIPP, we can have up to 13 waves for a given panel. A panel year usually denotes when the start of the survey happens 

(i.e., the 2008 SIPP Panel started in 2008). Each wave has four rotations. We randomly assign households to a rotation and 

interview the household about the four months before their interview month. This means the 2008 SIPP Wave 1 covered 

interviews conducted from September 2008 to December 2008 and 2008 SIPP Wave 2 covered interviews conducted from 

January 2009 to April 2009. We asked households interviewed in September 2008 about the timeframe of May 2008 through 

August 2008. We asked households interviewed in October 2008 about the timeframe of June 2008 through September 2008, 

those in November 2008 about July 2008 through October 2008, those in December 2008 about August 2008 through 

November 2008, etc. We found an 80.64% overall weighted response rate for households in SIPP 2008 Wave 1 and 74.07% 

for households in SIPP 2008 Wave 2. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

II. Background on Past SIPP Incentives 

A large amount of literature is available from past SIPP Panels about the use of incentives and their effectiveness. 

In SIPP 1996 Wave 1, Rotations 2 through 4, we compared a $10 incentive to a $20 incentive, both paid in advance 

(unconditional) to households. The finding was that the $20 incentive increased response rates for key SIPP respondents (i.e. 

those tending towards poverty) by 3.4% to 6.0%. [James, 1997] [Flanagan, 2007] [Killion, 2008] The $20 incentive also 

worked for Black households where they reduced household, person, and item nonresponse rates in the initial interview by 

about 1.5%. The $10 incentive did not significantly reduce nonresponse. [Mack, et al., 1998] Also, the $20 incentive 

strongly improved attrition of households in the high poverty stratum by reducing the nonresponse rate from 9.32% to 5.94%. 

[James, 1997] 

In SIPP 2001, we compared the control group to a group that would receive a $40 discretionary conditional incentive based 

on completion of an interview. We limited these incentives to only 10% of the eligible households. However, in the early 

waves few discretionary incentives were given out (for example, 1.94% in the first wave) which resulted in an increase of 

only 0.9% to 1.9% in response rates over the control group in six of the first eight waves (all but waves 1 and 4) where the 

incentive effectively worked at the 10% significance level. We saw no significant differences found in waves 1 and 4. 

[Killion, 2008] [Lewis, et al., 2005] 

In SIPP 2004, incentives became standard rather than an experiment, so we have no results as to the effectiveness of the 

incentives for this panel. However, in SIPP 2004, the field staff had enough $40 debit cards to cover approximately 20% of 

their workload to persuade reluctant respondents to participate. [Creighton, 2003] 

III. Motivation and Introduction of the Current Incentive Experiment 

To run the current incentive experiment for SIPP 2008, we needed to look at what happened with the 2001 SIPP Panel 

because that is where we introduced discretionary incentives. In the 2001 SIPP Panel, the regional offices concluded the 

discretionary incentives are justified due to lower costs. [Creighton, 2003] 

We therefore felt the need to address two key questions for SIPP 2008: 

1. Do incentives increase overall response rates? 

2. What type of incentive gives the most increase in overall response rates (is more effective): conditional or 

unconditional? [Killion, 2008] 

We implemented a randomized experiment starting in the 2008 SIPP Panel Wave 1. To control the costs for this experiment, 

we assigned 50% of our sample so they received no incentive and thus were the control group. We sent 25% of the sample a 

$20 debit card, unconditionally, with an advance letter introducing them to the survey. The remaining 25% of the sample 

were eligible, at the field representative’s discretion, to receive a $40 debit card upon completion of the survey. The field 

representatives in the regional offices were given enough $40 debit cards to cover 15% of their overall workload, so they 

could not offer the $40 incentive to all of the eligible cases. 

IV. Results of the SIPP 2008 Incentive Experiment 

The following results are available for the randomized experiment in the first and second waves of the 2008 SIPP Panel (note 

that all response rates are weighted): 

A. Wave 1 Response Rates by Regional Office [RO] and Incentive 

B. Wave 2 Response Rates by Regional Office [RO] and Incentive 

C. Wave 1 Response Rates by Stratum and Incentive 

D. Wave 2 Response Rates by Stratum and Incentive 

E. Costs of the Incentives 

F. Ratios of Promised to Available $40 Incentives 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

A. Wave 1 Response Rates by Regional Office [RO] and Incentive 

In this section we present weighted response rates by regional office (RO) and incentive for SIPP 2008 Wave 1: 

Table 1: Wave 1 Response Rates by Regional Office [RO] and Incentive 

RO Name $0 $20 $40 

21 Boston 78.53 79.45 77.2 

22 New York 73.03 75.84 71.49 

23 Philadelphia 78.33 76.5 75.86 

24 Detroit 83.9 82.38 84.15 

25 Chicago 83.74 84.61 84.13 

26 Kansas City 85.67 88.86** 86.27 

27 Seattle 79.42 85.15** 82.93*** 

28 Charlotte 81.99 82.99 80.59 

29 Atlanta 83.33 82.1 83.92 

30 Dallas 73.72 73.95 75.1 

31 Denver 83.07 82.73 83.35 

32 Los Angeles 76.42 79.61 78.9 

All ROs 80.31 81.37** 80.58 

* $20 and $40 significant difference noted. 

** $0 and $20 significant difference noted. 

*** $0 and $40 significant difference noted. 

(Tested at 10% significance level; higher response rate is starred.) 

In Table 1, we see the $20 unconditional incentive is effective compared to the control group across all regional offices. This 

table also shows the $20 unconditional incentive raised response rates by 1.06% and the $40 conditional incentive was 

effective only in Seattle. Note that Seattle had overpromised the $40 conditional incentive. The $20 incentive is effective 

only in two ROs: Kansas City and Seattle. The $40 conditional incentive is not more effective than the $20 unconditional 

incentive. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I I I I 

B. Wave 2 Response Rates by Regional Office [RO] and Incentive 

In this section we present weighted response rates by regional office (RO) and incentive for SIPP 2008 Wave 2: 

Table 2: Wave 2 Response Rates by Regional Office [RO] and Incentive 

RO Name $0 $20 $40 

21 Boston 71 72.31 70.56 

22 New York 67.11 68.42 64.57 

23 Philadelphia 70.27 69.05 68.02 

24 Detroit 78.07 76.04 78.42 

25 Chicago 77.38 79.74 78.39 

26 Kansas City 80.43 83.91** 80.8 

27 Seattle 71.79 77.92** 77.26*** 

28 Charlotte 75.06 76.2 75.66 

29 Atlanta 78.81 77.41 78.24 

30 Dallas 65.19 65.75 68.04 

31 Denver 77.38 77.86 78.42 

32 Los Angeles 67.36 70.53 70.22 

All ROs 73.53 74.84** 74.37 

* $20 and $40 significant difference noted. 

** $0 and $20 significant difference noted. 

*** $0 and $40 significant difference noted. 

(Tested at 10% significance level; higher response rate is starred.) 

In Table 2, we see the $20 unconditional incentive is still effective compared to the control group across all regional offices. 

The table also shows that the $20 unconditional incentive raised response rates by 1.31% and the $40 conditional incentive 

was still effective only in Seattle. Furthermore, the $20 incentive is still effective only in Kansas City and Seattle. 

Tables 1 and 2 reflect the same pattern for where the response rates turned out to be significant. This is because about two 

thirds of the households not interviewed in the first wave were still eligible for interviews in the second. Also, one expects 

the results to be similar from one wave to the next. 

C. Wave 1 Response Rates by Stratum and Incentive 

In this section we present weighted response rates by stratum and incentive for SIPP 2008 Wave 1: 

Table 3: Wave 1 Response Rates by Stratum and Incentive 

$0 $20 $40 $40 Promised* 

Low Income Stratum 81.54 82.87 81.35 13.29 

Non-Low Income Stratum 79.98 80.97 80.38 10.68 

* Percent of households promised a $40 incentive. 

The response rates are pretty uniform, within each individual stratum. Households with low income stratum respond at 81.35 

to 82.87%; others respond at 79.98 to 80.97%. 

We also see the low income households responded .97% (within the $40 incentive) to 1.9% (within the $20 incentive). 

Contrary to our expectations, these were not significant differences. The incentives tended to be given to those in financial 

need because the percent of households promised a $40 conditional incentive was higher in the low-income stratum than in 

the non-low income stratum, by 2.61%. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

I I I I 

I I I I 

D. Wave 2 Response Rates by Stratum and Incentive 

In this section we present weighted response rates by stratum and incentive for SIPP 2008 Wave 2: 

Table 4: Wave 2 Response Rates by Stratum and Incentive 

$0 $20 $40 $40 Promised* 

Low Income Stratum 73.73 75.26 75.08 19.75 

Non-Low Income Stratum 73.48 74.72 74.18 13.02 

* Percent of households promised a $40 incentive. 

The response rates are uniform by incentive and across income stratum. The low income stratum has response rates between 

73.73% and 75.26% whereas the non-low income stratum has response rates between 73.48% and 74.72%. Also, we see that 

the $40 promised rates for each individual stratum are higher in Wave 2 than Wave 1. This is because they used more $40 

conditional incentives in the second wave. 

E. Costs of the Incentives 

In this section we present the overall cost figures for the $20 unconditional incentive and the $40 conditional incentive. 

Table 5 shows the costs of the SIPP 2008 incentives: 

Table 5: Costs of the Incentives 

Incentive 

$20 $40 Wave 1 $40 Wave 2 

# Received 16,500 1,858 1,661 

Total Cost $330,000 $74,320 $66,440 

Please note that “# Received” means the number of households that received the particular incentive for the complete first 

wave. These figures do not reflect the administrative nor hidden fees associated with each of the individual incentives. 

In short, we see that the $20 unconditional incentive costs significantly more (by a factor of approximately 4.4) than does the 

$40 conditional incentive and that the costs of the $40 incentive decrease due to fewer of these incentives being offered. This 

is because the $20 incentive was given to everyone, but the $40 incentive was only given to 1,858 households in the first 

wave and 1,661 households in the second wave. 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

F. Ratios of Promised to Available $40 Incentives 

For the $40 conditional incentive, regional offices were directly responsible for overseeing the operations in their respective 

areas. In this section we present the ratios of promised to available $40 incentives, rounded to one decimal place: 

Table 6: Ratios of Promised to Available $40 Incentives 

Name Promised Available Percent 

Boston 77 206 37.4 

New York 142 244 58.2 

Philadelphia 57 255 22.4 

Detroit 22 131 16.8 

Chicago 285 207 137.7 

Kansas City 136 146 93.2 

Seattle 261 241 108.3 

Charlotte 323 295 109.5 

Atlanta 140 166 84.3 

Dallas 231 211 109.5 

Denver 31 163 19.0 

Los Angeles 153 172 89.0 

From Table 6 we see that there are several regional offices that severely underpromised the $40 conditional incentive, 

meaning the percent used is below 25. These were Philadelphia, Detroit, and Denver. 

Alternatively, we see that there are four regional offices, out of the twelve, who overpromised the $40 incentive. These were 

Chicago, Seattle, Charlotte, and Dallas. 

There were various reasons that led to the overpromising in these regional offices. In the regional office of Dallas, for 

instance, the interviewers had encountered trouble within the instrument whereby they could access the promised screen for 

the $20 incentive, when the promised screen was solely meant for $40 incentive respondents. This led to promising some of 

the $40 incentives to $20 incentive respondents. 

Another instance of overpromising came about in the case of the Seattle regional office. They assumed debit cards from 

other regions could replace their own stock when the time came for a new wave of SIPP interviews. 

V. Conclusions 

Overall we see that the $20 unconditional incentive proved effective compared to the control group whereas the $40 

conditional incentive did not. We also see that the weighted response rates appear to be uniform within most of the 

individual regional offices and also in individual stratum. The $20 unconditional incentive costs more than the $40 

conditional incentive. Finally, it appears persons are getting incentives who need them rather than for the purposes of 

lowering nonresponse. 

VI. Future Research Directions 

Some possible questions to answer involve the following: 

1. What causes the uniformity in response rates within most of the individual regional offices? Furthermore, how do 

individual regional offices differ in the administration of the SIPP interviews? 

2. What else can be done to reduce the nonresponse to the SIPP if incentives are not proving to be effective or they do 

not change the nonresponse rate to a satisfactory level? 
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

VII. References 

Creighton, Kathleen P. (2003). Letter from Creighton to Schechter (regarding use of monetary incentives in 2004 SIPP 

Panel). Census Bureau, April 8, 2003. 

Creighton, Kathleen P., King, Karen E., and Martin, Elizabeth A. (2001). The Use of Monetary Incentives in Census Bureau 

Longitudinal Surveys. Statistical Policy Working Paper 32: 2000 Seminar on Integrating Federal Statistical 

Information and Processes. Washington DC: Federal Committee on Statistical Methodology, Office of Management 

and Budget, April 2001, pp. 289-310. 

Flanagan, Patrick (2007). SIPP 2004: Incentive Analysis (ALYS-3). Internal Memorandum from Flanagan to Creighton. 

Census Bureau, March 27, 2007. 

Gelman, Andrew, Stevens, Matt, and Chan, Valerie (2003). Regression Modeling and Meta-Analysis for Decision Making: A 

Cost-Benefit Analysis of Incentives in Telephone Surveys. Journal of Business & Economic Statistics, Vol. 21. 

James, Tracy L. (1997). Results of the Wave 1 Incentive Experiment in the 1996 Survey of Income and Program 

Participation. ASA Proceedings of the Section on Survey Research Methods, 834-839. 

Killion, Ruth A. (2008). SIPP 2008: Incentives Test (ALYS-4). Internal Memorandum from Killion to Harris-Kojetin. 

Census Bureau, July 2, 2008. 

Lewis, Denise, and Creighton, Kathleen (2005). The Use of Monetary Incentives in the Survey of Income and Program 

Participation. Paper presented at the annual meeting of the American Association for Public Opinion Research, 

Fontainebleau Resort, Miami Beach, FL. 

Mack, Stephen, Huggins, Vicki, Keathley, Donald, and Sundukchi, Mahdi (1998). Do Monetary Incentives Improve 

Response Rates in the Survey of Income and Program Participation?. ASA Proceedings of the Section on Survey 

Research Methods, 529-534. 

Shettle, Carolyn (1996). Evaluation of Using Monetary Incentives in a Government Survey. American Statistical Association, 

Aug. 1996. 

Singer, Eleanor (2001). The Use of Incentives to Reduce Nonresponse in Household Surveys. Survey Nonresponse, eds. R. 

M. Groves, D. A. Dillman, J. L. Eltinge, and R. J. A. Little, New York: Wiley, pp. 163-177.