Journal of Economic Perspectives—Volume 29, Number 4—Fall 2015—Pages 1–29

Large and nationally representative surveys are arguably among the most 

important innovations in social science research of the last century. As the 

leadership of the Committee on National Statistics of the National Academy 

of Sciences wrote: “It is not an exaggeration to say that large-scale probability surveys 

were the 20th-century answer to the need for wider, deeper, quicker, better, cheaper, 

more relevant, and less burdensome official statistics” (Brown, Citro, House, 

Marton, and Mackie 2014). Household surveys are the source of official rates of 

unemployment, poverty, health insurance coverage, inflation, and other statistics 

that guide policy. They are also a primary source of data for economic research and 

are used to allocate government funds.

However, the quality of data from household surveys is in decline. Households 

have become increasingly less likely to answer surveys at all, which is the problem 

of unit nonresponse. Those that respond are less likely to answer certain questions, 

which is the problem of item nonresponse. When households do provide answers, 

they are less likely to be accurate, which is the problem of measurement error. We will 

Household Surveys in Crisis†

■ Bruce D. Meyer is the McCormick Foundation Professor, Irving B. Harris Graduate School 

of Public Policy Studies, University of Chicago, Chicago, Illinois. Wallace K. C. Mok is 

Assistant Professor of Economics, Chinese University of Hong Kong, Hong Kong. James X. 

Sullivan is the Rev. Thomas J. McDonagh, C.S.C., Associate Professor of Economics, University of Notre Dame, Notre Dame, Indiana. Their email addresses are bdmeyer@uchicago.edu, 

wallacemok@cuhk.edu.hk, and sullivan.197@nd.edu.

† For supplementary materials such as appendices, datasets, and author disclosure statements see the 

article page at

http://dx.doi.org/10.1257/jep.29.4.1 doi=10.1257/jep.29.4.1

Bruce D. Meyer, Wallace K. C. Mok, and 

James X. Sullivan

j_meyer_293.indd 1 7/30/15 10:01 AM
2 Journal of Economic Perspectives

document a noticeable rise in all three threats to survey quality in many of the most 

important datasets for social science research and government policy.1

Of course, if nonresponse arises randomly across the population, survey data 

would still lead to unbiased estimates of distributions. Thus, we also investigate what 

is known about the extent to which these problems create bias. However, it can be 

difficult to verify that nonresponse is independent of survey measures of interest. 

After all, we typically have very limited information on the characteristics of those 

who do not respond. The fundamental problem in assessing survey bias due to these 

problems is the lack of a benchmark measure of the true outcome.

One productive approach to measuring the degree of bias in household surveys, 

along with addressing potential bias, is to compare survey results with administrative data. In this paper, we focus on the accuracy of survey reporting of government 

transfers because reliable benchmarks for these programs exist from both aggregate 

and micro-level administrative data. For example, aggregate administrative data for 

Temporary Assistance for Needy Families (TANF) are available in annual reports 

provided by the US Department of Health and Human Services, Administration 

for Children and Families, and total Supplemental Nutrition Assistance Program 

(SNAP) payments are available from the US Department of Agriculture, Food, 

and Nutrition Service. Micro-level administrative data for these same program 

would typically come from a state welfare agency and take the form of records of 

monthly benefit payment amounts to individuals along with some information on 

each individual’s characteristics. Another advantage of focusing on transfers is that 

the questions about these programs are often clear and comparable in surveys and 

administrative sources. We examine the quality of household survey data through 

comparisons with administrative data from nine large programs that receive considerable attention from both the research and policy community. For example, we 

compare the total dollar value of food stamp benefits reported, by all respondents 

in a survey to the total dollar value of food stamp benefits awarded as recorded in 

US Department of Agriculture, Food and Nutrition Service administrative data.

Our results show a sharp rise in the downward bias in household survey estimates of receipt rates and dollars received for most programs. In recent years, more 

than half of welfare dollars and nearly half of food stamp dollars have been missed 

in several major surveys. In particular, this measurement error typically takes the 

form of underreporting resulting from true program recipients being recorded as 

nonrecipients. (Throughout this paper we use underreporting as a synonym for 

understatement or underrecording, since it is likely due to errors by both interviewers and interviewees.) We argue that although all three threats to survey quality 

are important, in the case of transfer program reporting and amounts, measurement 

error rather than unit nonresponse or item nonresponse appears to contribute the 

most bias.

1 In certain cases, additional measurement issues like coverage error and sampling error will be important. See Groves (2004) and Alwin (2007) for an exhaustive list of types of survey errors.

AQ1

j_meyer_293.indd 2 7/30/15 10:01 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 3

The underreporting of transfer income in surveys has profound implications 

for our understanding of the low-income population and the effect of government 

programs for the poor. We point to evidence from linked administrative and survey 

data that indicates that this underreporting leads to an understatement of incomes 

at the bottom, the rate of program receipt, and the poverty-reducing effects of 

government programs and to an overstatement of poverty and inequality.

The evidence on declining survey quality we present here is likely not unique 

to transfer income. While evidence comparing other survey variables to administrative benchmarks is scarce, there is evidence suggesting that survey biases in 

self-employment and pension income, education, pension contributions, and some 

categories of expenditures have also risen.

Our results call for more research into why survey quality has declined. Our 

preferred explanation is that households are overburdened by surveys, leading to a 

decline in many measures of survey cooperation and quality. The number and breadth 

of government surveys rose sharply between 1984 and 2004 (Presser and McCulloch 

2011), and the number of private surveys has been rising as well. We discuss the limited 

evidence concerning some alternative explanations including increasing concerns 

about privacy, a decline in public spirit, less leisure time, or the stigmatizing effect of 

giving certain answers to questions. We conclude by noting the need for research on 

ways to improve the quality of household surveys. In particular, more frequent linking 

of survey data with administrative microdata provides one potentially fruitful avenue 

for improving the quality of survey data.

Rising Unit Nonresponse Rates

Unit nonresponse, which occurs when a household in a sampling frame is not 

interviewed at all, has been rising in most surveys. Unit nonresponse rates rose by 

3–12 percentage points over the 1990s for six US Census Bureau surveys (Atrostic,

Bates, Burt, and Silberstein 2001). In non-Census surveys, the rise in unit nonresponse 

is also evident, and in some cases even sharper (Steeh, Kirgis, Cannon, and DeWitt 

2001; Curtin, Presser, and Singer 2005; Battaglia, Khare, Frankel, Murray, Buckley, 

and Peritz 2007; Brick and Williams 2013). The National Research Council (2013) 

provides a thorough summary for US surveys, but the pattern is apparent in surveys in 

other countries as well (de Leeuw and de Heer 2002).

Indeed, the problem of rising unit nonresponse in major surveys has been a 

heavily discussed topic in the survey research community. Unit nonresponse was the 

subject of two National Research Council reports and a special issue of a major social 

science journal (National Research Council 2011, 2013; Massey and Tourangeau 

2013). The Office of Management and Budget (2006) has set a target response 

rate for federal censuses and surveys, and recommends analysis of nonresponse bias 

when the unit response rate is less than 80 percent. At least one influential journal, 

the Journal of the American Medical Association, restricts publication of research using 

low-response-rate surveys (Davern 2013).

AQ2

j_meyer_293.indd 3 7/30/15 10:01 AM
4 Journal of Economic Perspectives

In Figure 1, we report the unit nonresponse rate for five prominent household surveys during the 1984–2013 period: the Current Population Survey Annual 

Demographic File/Annual Social and Economic Supplement (CPS), which is the 

source of the official US poverty rate and income distribution statistics; the Survey 

of Income and Program Participation (SIPP), which is the best source for the information needed to determine eligibility for and receipt of transfers; the Consumer 

Expenditure (CE) Survey, which is the main source of data on consumption and 

provides the weights that are put on price changes when calculating inflation as 

measured by the Consumer Price Index; the National Health Interview Survey 

(NHIS), which is the primary source for information on the health status of the 

US population; and the General Social Survey (GSS), which may be the most used 

dataset across the social sciences for social and attitudinal information. For other 

analyses in this paper, we also examine nonresponse rates in the American Community Survey (ACS), which replaced the Census long form, providing detailed 

F1

Figure 1 

Unit Nonresponse Rates of Major Household Surveys

Sources: For the Current Population Survey, see Appendix G of US Census Bureau (various years (a)). 

For the Survey of Income and Program Participation, see Source and Accuracy Statement of US Census 

Bureau (various years (b)). For the National Health Interview Survey, see Table 1 of US Department 

of Health and Human Services (2014). For the Consumer Expenditure Survey, see US Department of 

Labor (various years). For the General Social Survey, see Table A.6 of “Appendix A: Sampling Design and 

Weighting,” in Smith, Marsden, Hout, and Kim (2013).

Note: We report the unit nonresponse rate for five prominent household surveys during the 1984–2013 

period: the Current Population Survey Annual Demographic File/Annual Social and Economic 

Supplement (CPS), the Survey of Income and Program Participation (SIPP), the Consumer Expenditure 

(CE) Survey, the National Health Interview Survey (NHIS), and the General Social Survey (GSS).

0 

0.05 

0.10 

0.15 

0.20 

0.25 

0.30 

0.35 

1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 

Nonresponse rate 

Survey year 

CPS SIPP (Wave 1) NHIS CE Survey GSS 

j_meyer_293.indd 4 7/30/15 10:01 AM
Household Surveys in Crisis 5

small-area information annually, and the Panel Study of Income Dynamics (PSID), 

which is the longest-running longitudinal survey (and thus allows tracking specific 

households over time).

The surveys in Figure 1 show a pronounced increase in unit nonresponse over 

time, in recent years reaching 16 to 33 percent. Between 1997 and 2013, the unit 

nonresponse rate in the Current Population Survey rose from 16 to 20 percent while 

the rate in the National Health Interview Survey rose from 8 to 24 percent.2

 The 

National Research Council (2013) reports a general decline in response rates for a 

long list of surveys. The decline in response rates seems to be even more pronounced 

for public opinion surveys (Pew 2012). Interestingly, response rates appear to be 

much higher for surveys in developing countries. Mishra, Hong, and Khan (2008) 

report that recent demographic and health surveys from 14 different African countries all had unit response rates above 92 percent.

One of the few notable exceptions to high nonresponse rates for domestic 

surveys is the American Community Survey. The ACS’s low survey nonresponse rate 

(about 3 percent in recent years) is due in large part to the fact that the survey is 

mandatory. A US Census Bureau (2003) study showed that a change to a voluntary 

standard for the ACS led to a rise in nonresponse rates to the mail version of the 

survey of more than 20 percentage points. The ACS also contacts potential respondents through multiple modes including mail, telephone, and personal visits. Only 

about 66 percent of households in most years respond to the initial two modes 

of contact. A random subsample of nonrespondents at that point is selected for a 

personal home visit and their responses are given extra weight. Those not selected 

are given a weight of zero, so they do not contribute to the overall nonresponse 

rate. The ACS can affect community funding (Reamer 2010), which may also in part 

account for why the nonresponse rate is so low.

Of the problems with surveys, rising unit nonresponse has gotten the most attention. This emphasis is not surprising given that it is widespread, often easy to measure, 

and increases survey costs. However, the rate of unit nonresponse is not particularly 

informative about the accuracy of statistics from a survey. Unit nonresponse only leads 

to bias if it is nonrandom, with the exact requirement depending on the statistic in 

question and the weighting method. However, exploring whether unit nonresponse 

is random can be difficult because researchers typically have only limited information 

on the characteristics of nonresponders. Even if nonresponders look like responders 

based on a limited set of characteristics—say, age and geography—this does not mean 

that these groups are similar along other dimensions, such as willingness to participate in government programs. Evidence on the extent to which unit nonresponse 

leads to bias differs by survey and question. While there are examples of substantial bias, in other cases the resulting bias is small or can be mitigated by appropriate 

weighting—certain demographic variables in the survey are weighted to correspond 

2 Regression estimates of a linear time trend over the available years yield a positive coefficient on year 

for each of the surveys that is strongly significantly different from zero in four of the five cases, and weakly 

significant in the remaining case. For details, see Online Appendix Table 1.

j_meyer_293.indd 5 7/30/15 10:01 AM
6 Journal of Economic Perspectives

to the total population (National Research Council 2013, pp. 42–43). Even in public 

opinion surveys with response rates under 10 percent, researchers have argued that 

properly weighted responses are largely representative (Pew 2012). In their survey 

of bias estimates, Groves and Peytcheva (2008) found that bias magnitudes differed 

more across statistics (such as mean age or gender) within a survey than they did 

across surveys. Indeed, with the possibilities for weighting adjustments taken into 

account, unit nonresponse is probably not the main threat to the quality of household survey data.

Several methods have been proposed for improving unit nonresponse—for 

example, advance notification of the survey through the mail, increasing the number 

of times the potential respondent is contacted, improving the training of interviewers, 

or offering financial incentives for participation—but the evidence suggests only small 

effects from such efforts (National Research Council 2011). Even when such efforts 

increase response rates, they do not necessarily lead to a reduction in bias. Indeed, if 

they mainly encourage the groups that are already overrepresented in the survey or 

who are unmotivated to co-operate, they can even make the bias worse (Groves 2006; 

Groves and Peytcheva 2008; Tourangeau, Groves, and Redline 2010; Peytchev 2013; 

Kreuter, Müller, and Trappmann 2014). Also, a tradeoff can arise between different 

measures of survey accuracy: inducing participation from those who are initially 

reluctant to complete a survey may lead to greater problems with item nonresponse 

(missing answers to questions) or measurement error (inaccurate answers).

Rising Item Nonresponse

Even if a household agrees to participate in a survey, responses to key questions 

may not be obtained due to refusal or inability to answer, or failure of the interviewer to record the response. This item nonresponse is distinct from a respondent 

misreporting that he did not receive a certain type of transfer income, which would 

be considered measurement error. Most surveys (and all of those that we examine) 

typically impute a response in these cases of missing data. Many methods are used to 

impute, though the Census “hot-deck” procedure, where a missing value is imputed 

from a randomly selected similar record, is probably the most common (for more 

information, see Andridge and Little 2010). Surveys impute responses for all sorts 

of questions, including those related to demographic characteristics such as age and 

education, employment, and income. Nonresponse rates are typically low for most 

questions (Bollinger and Hirsch 2006), but they can be quite high for questions 

related to labor and nonlabor income. For transfer programs, surveys may impute 

“recipiency” (that is, whether or not a person received a given type of benefit at all) 

as well as the dollars received or the months of benefits received.

As evidence of the extent of item nonresponse and how it has changed over 

time, we present imputation rates for survey questions on receipt of transfer income. 

We calculate the share of dollars recorded in two major household surveys that is 

imputed for six large transfer programs that all receive considerable attention 

j_meyer_293.indd 6 7/30/15 10:01 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 7

from both the research and policy community: Aid to Families with Dependent 

Children/Temporary Assistance for Needy Families (AFDC/TANF); the Food Stamp 

Program/Supplemental Nutrition Assistance Program (FSP/SNAP); Supplemental 

Security Income (SSI); Social Security (OASDI) including both retirement and 

disability benefits; Unemployment Insurance (UI); and Workers’ Compensation 

(WC). These large national programs provide benefits to tens of millions of individuals—together they distributed almost $1 trillion in 2011. We present the imputation 

shares for the Current Population Survey in Figure 2A and for the Survey of Income 

and Program Participation in Figure 2B. These two surveys focus on income and 

program receipt, and they are a good indicator of the state of the art in survey collection over time. Although not reported here, we have also calculated similar imputation 

rates for the American Community Survey, the Consumer Expenditure Survey, and 

the Panel Study of Income Dynamics (Meyer, Mok, and Sullivan 2015).

The imputation rates are quite high, averaging about 25 percent. In 2013, the 

imputation shares in the Current Population Survey ranged from 24 percent of 

dollars recorded from Temporary Assistance for Needy Families and the Supplemental Nutrition Assistance Program to 36 percent of Social Security dollars. 

Overall, the Survey of Income and Program Participation has noticeably higher 

imputation rates than the Current Population Survey.3

Figures 2A and B also show an increase in imputation rates over the past two and 

a half decades. This rise is evident in all programs in both the Current Population 

Survey and Survey of Income and Program Participation. The estimates suggest, for 

example, that for AFDC/TANF, in the CPS the fraction of dollars imputed is rising 

by 0.4 percentage points each year.4 The imputation rates for months of receipt 

(not reported) are similar to those for dollars reported here. In recent years, at 

least 10 percent of months are imputed in the CPS for all four programs for which 

we have months. For the SIPP, month imputation shares are sometimes below 

10  percent, but are more typically between 10 and 20 percent. The shares have 

generally risen over time.

Transfer income may be imputed either when there is missing information on 

whether the household receives income from a given program, or when there is 

missing information on the number of dollars received. We have also calculated the 

3 Imputation procedures in the Survey of Income and Program Participation take advantage of information collected in previous waves. For example, beginning with the 1996 panel, missing data were imputed 

by using the respondent’s data in the previous wave (if available). Starting with wave 2 of the 2004 panel, 

the SIPP began to use “dependent interviewing” in which the interviewers use information from the 

prior wave to tackle item nonresponse during the actual interview. For the results in Figure 2B and 

Table 1 we do not include values imputed from prior wave information in our calculation of total dollars 

imputed. See Meyer, Mok, and Sullivan (2015), Chapter 4 of US Census Bureau (2001), and Pennell 

(1993) for more information.

4 We summarize the trends by regressing the imputation share on a constant and a time trend separately 

for each program and survey. As shown in Online Appendix Table 2, for all six programs and both 

surveys the coefficient on the time trend is positive. In the case of the Current Population Survey, the 

upward trend is statistically significant at the 1 percent level for four of the six programs, while the trend 

is significant in the Survey of Income and Program Participation for five of six programs.

F2

j_meyer_293.indd 7 7/30/15 10:01 AM
8 Journal of Economic Perspectives

share of total dollars reported attributable to those whose recipiency was imputed. 

In the Current Population Survey and the Survey of Income and Program Participation this share, is typically on the order of 10 percent but is frequently higher for 

programs covered by these surveys. There is substantial variation across programs 

and over time. For most of the years since 2000, recipiency imputation exceeds 

20 percent for AFDC/TANF. The rise in recipiency imputation over time is less 

pronounced than that for overall imputation, which includes not only recipiency 

imputation but also imputation of dollar amounts when receipt is reported but the 

dollar amount is not.

While imputation might improve quality assessments based on comparisons 

to aggregate amounts paid out, there are important limitations associated with 

imputed values. Studies using linked survey and administrative data show that 

the rates of false positive and false negative reporting of receipt are almost always 

much higher among the imputed observations than the nonimputed ones (Meyer, 

Goerge, and Mittag 2014; Celhay, Meyer, and Mittag 2015). Also, even in cases where 

imputations may reduce bias for statistics such as the mean, they may create greater 

bias in other statistics such as measures of inequality or covariances. For example, 

Bollinger and Hirsh (2006) show that including imputed values of earnings leads to 

biased coefficients in regressions of earnings on other attributes.

Figure 2 

Item Nonresponse Rates in Two Surveys for Various Transfer Programs 

Calculated as Share of Dollars Reported in the Survey that Is Imputed

0

0.05

0.10

0.15

0.20

0.25

0.30

0.35

0.40

0.45

19911992199319941995199619971998199920002001200220032004200520062007200820092010201120122013

Survey year

AFDC/TANF FSP/SNAP OASDI SSI UI WC

A: Current Population Survey (CPS)

j_meyer_293.indd 8 7/30/15 10:01 AM
Household Surveys in Crisis 9

Measurement Error and Estimates of Bias

Inaccurate responses given that there is a response are called measurement 

error, and can contribute to bias (the difference between an estimate and the true 

value) in common statistics calculated from survey data. One way to test for measurement error is to link survey data on the payments that individuals or households say 

they have received, with administrative microdata on the amounts actually provided 

to each household. Comparisons to administrative microdata on program receipt 

have been fairly limited in the literature. This approach has often been restricted 

to a single state, year, program, and dataset (Taeuber, Resnick, Love, Stavely, Wilde, 

and Larson 2004). Examples of studies that examine more than one program (but 

still a single dataset) include Moore, Marquis, and Bogen (1996), Sears and Rupp 

(2003), and Huynh, Rupp, and Sears (2002). A review of earlier studies can be 

found in Bound, Brown, and Mathiowetz (2001).

An alternative approach to comparisons to administrative microdata is to compare 

aggregate survey and administrative data. Comparisons to administrative aggregates 

have been used widely, but results are only available for a few years, for a few transfer 

Notes: We calculate the share of dollars recorded in the Current Population Survey (CPS) (Figure 2A) 

and the Survey of Income and Program Participation (SIPP) (Figure 2B) that is imputed, for six large 

transfer programs: Aid to Families with Dependent Children/Temporary Assistance for Needy Families 

(AFDC/TANF), the Food Stamp Program/Supplemental Nutrition Assistance Program (FSP/SNAP), 

Supplemental Security Income (SSI), and Social Security (OASDI) including both retirement and disability 

benefits, Unemployment Insurance (UI), and Workers’ Compensation (WC). For the SIPP, Figure 2B, the 

share of dollars reported that is imputed excludes imputation using previous wave information.

Figure 2 (continued)

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

1990 1991 1992 1993 1994 1995 1996 

1997 

1998 1999 2000 2001 2002 2003 2004 2005 2006 

2007 

2008 2009 2010 

2011 

2012 2013 

Survey year

B: Survey of Income and Program Participation (SIPP) 

AFDC/TANF FSP/SNAP OASDI SSI UI WC

j_meyer_293.indd 9 7/30/15 10:02 AM
10 Journal of Economic Perspectives

programs, and for some of the key datasets. Important papers include Duncan and 

Hill (1989), Coder and Scoon-Rogers (1996), Roemer (2000), and Wheaton (2007). 

These papers tend to find substantial underreporting that varies across programs.

To provide a more comprehensive look at the magnitude of measurement error 

in many surveys, across many years, and for several programs, we compare aggregate 

survey and aggregate administrative data. The aggregate administrative data are 

available through a variety of reports from government agencies. A complete list of 

sources for these administrative aggregates is available in Meyer, Mok, and Sullivan 

(2015). The administrative aggregate data that we use have been audited, so we 

expect bias in these data to be small. There might be some bias that results from 

different coverage or variable definitions between the administrative aggregates and 

the survey aggregates, but we make considerable effort to ensure these align closely.

Since administrative data sources are heterogeneous, they should not always 

be taken as accurate. For example, Abowd and Stinson (2013) model administrative and survey measures of earnings, treating both sources as error-ridden. Their 

approach is driven in part by conceptual differences between the survey and administrative measures of earnings that mean that there is not a single true earnings 

measure. This problem is not present when one examines transfers, because the 

survey and administrative concepts are generally the same.

Through comparisons to aggregate administrative data, we show that survey 

measures of whether an individual receives income and of how much income is 

received from major transfer programs are both sharply biased downward, and this 

bias has risen over time. Although these measures of bias include all three threats to 

survey quality—unit nonresponse, item nonresponse, and measurement error—in 

the following section we argue that the bias is largely due to measurement error.

Here, we will focus on two statistics, the mean receipt of certain transfer 

programs measured in dollars, and the mean receipt measured in months of 

receipt. Means of transfer receipt are important statistics. Reports of these means 

affect distributional calculations of inequality and poverty; calculations of the 

effects of programs on the income distribution; and estimates of what share of 

those eligible for a certain program receive support from the program. Our analyses focus on how underreporting has changed over time. For a more extensive 

discussion of how these findings differ across programs and datasets see Meyer, 

Mok, and Sullivan (2015).

Below we report estimates of the proportional bias in dollar reporting, which 

we call Dollar Bias, and in month reporting, which we call Month Bias. These biases 

can be defined as the net reporting rate minus 1, or more specifically,

Dollar Bias = dollars reported in survey, population weighted __________________________________________ dollars reported in administrative data − 1

and 

Month Bias = months reported in survey, population weighted __________________________________________ months reported in administrative data − 1.

j_meyer_293.indd 10 7/30/15 10:02 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 11

These expressions give us the proportional bias in the mean and therefore can be 

thought of as the proportional bias in the total dollars or months, or in per person 

dollars or months. Also note that the reporting rates in the above definitions are net 

rates reflecting underreporting by true recipients counterbalanced to some extent 

by overreporting by recipients and nonrecipients.

We calculate the bias in the mean receipt of transfer dollars for the same programs 

for which we reported imputation rates above, only now we are able to divide Old-Age, 

Survivors, and Disability Insurance (OASDI) into its retirement (OASI) and disability 

(SSDI) components.5

 We also calculate months-of-receipt reporting biases for seven 

programs. Months of receipt are not available in all cases, including Unemployment 

Insurance and Workers’ Compensation, but they are available for some programs 

for which we do not observe dollars, including the National School Lunch Program 

(NSLP) and the Special Supplemental Nutrition Program for Women, Infants, and 

Children (WIC). We do this for as many individual years as are available for five of the 

most important datasets for analyzing income, its distribution, and receipt of transfers: the Current Population Survey, the Survey of Income and Program Participation, 

the American Community Survey, the Consumer Expenditure Survey, and the Panel 

Study of Income Dynamics. If these datasets are understating transfers received in a 

substantial way—and we will show that they are—it has important implications for our 

understanding of the economic circumstances of the population and the effects of 

government programs. We should emphasize that all of the bias estimates we report 

include imputed values in the survey totals, so the bias calculated here understates 

the measurement problems. To put it another way, by providing values for households 

that do not report receipt of transfer income, imputations may lead to smaller estimates of bias in our approach even though these imputations introduce considerable 

measurement error due to the inaccuracy of imputed values.

The top panel of Table 1 presents the average Dollar Bias over the 2000–2012 

period for seven programs from five household surveys. With the single exception 

of Supplemental Security Income in the Survey of Income and Program Participation, the bias is negative, indicating underreporting of dollars of transfer income. 

The upward bias in reporting of SSI appears to be due to confusion among recipients between SSI, a Social Security Administration program aimed at low-income 

people who are blind, disabled, or elderly, and Old-Age and Survivor’s Insurance 

(OASI), which is what most people mean by Social Security (Huyhn, Rupp, and 

Sears 2002; Gathright and Crabb 2014). In most cases, the bias reported in Table 1 

is large. For our main cash welfare programs, Temporary Aid to Needy Families 

(combined with General Assistance in two cases), four of five surveys have a bias 

of 50 percent or more, meaning that less than half of the dollars given out are 

5 In several of the datasets, Social Security Disability benefits are in some cases combined with Social Security Retirement and Survivors benefits. To separate these programs, we use data from the Social Security 

Bulletin (US Social Security Administration, various years) to calculate for each year, age, school status, and 

gender cell, the proportions of total Social Security dollars that are paid to OASI and SSDI recipients. See 

Meyer, Mok, and Sullivan (2015) for more details on methodology.

T1

j_meyer_293.indd 11 7/30/15 10:02 AM
12 Journal of Economic Perspectives

captured in surveys. Even in the Survey of Income and Program Participation 

(SIPP), a survey especially designed to capture transfer program income, more than 

one-third of TANF dollars are missed. For the Food Stamp Program/Supplemental 

Nutrition Assistance Program (FSP/SNAP), the bias is at least 30 percent for four of 

the five surveys. The bias in dollar reporting of Unemployment Insurance (UI) and 

Workers’ Compensation (WC) is also pronounced: it is at least 32 percent for UI 

and 54 percent for WC in all surveys. The Social Security Administration programs 

(the retirement program OASI, the disability insurance program SSDI, and support 

for the low-income elderly, blind, and disabled through SSI) have much less bias, 

which may, in part, be due to the fact that receipt of these programs tends to be 

more regular or permanent.

The average Month Bias for this same period is reported in the bottom panel of 

Table 1. These biases are very similar to the corresponding dollar reporting biases 

in the top panel. In the case of the Food Stamp Program/Supplemental Nutrition 

Assistance Program (FSP/SNAP), the similarity is striking, with the bias in the two 

types of reporting never differing by more than 1.1 percentage points for the three 

datasets. This pattern suggests that individuals report about the right dollar amount 

on average, conditional on reporting. Or, put another way, most of the bias is due 

to not reporting at all, rather than reporting too little conditional on reporting. In 

Table 1

Proportional Bias in Survey Estimates of Mean Program Dollars and Months 

Received, by Program and Survey, 2000–2012

AFDC/

TANF

FSP/

SNAP

Social Security 

OASI SSDI SSI UI WC NSLP WIC

Dollars

ACS −0.519 −0.458 −0.165 −0.299 −0.046

CE −0.767 −0.587 −0.149 −0.214 −0.283 −0.583 −0.618

CPS −0.500 −0.417 −0.086 −0.187 −0.162 −0.325 −0.541

PSID −0.619 −0.308 −0.086 −0.176 −0.322 −0.360 −0.646

SIPP −0.357 −0.170 −0.070 −0.146 0.164 −0.388 −0.651

Months

ACS −0.154 −0.261 −0.372

CPS −0.453 −0.422 −0.147 −0.154 −0.397 −0.503 −0.341

PSID −0.574 −0.297 −0.114 −0.121 −0.502 −0.470 −0.180

SIPP −0.232 −0.165 −0.008 0.041 0.023 0.141 −0.197

Notes: Each cell reports the average dollars/months proportional bias for the specified program and survey 

in the 2000–2012 period. The transfer programs are: Aid to Families with Dependent Children/Temporary 

Assistance for Needy Families (AFDC/TANF), which is combined with General Assistance in the 

case of the Consumer Expenditure Survey and the American Community Survey; the Food Stamp 

Program/Supplemental Nutrition Assistance Program (FSP/SNAP); Social Security, including Old-Age 

and Survivors Insurance (OASI) and Social Security Disability Insurance (SSDI); Supplemental Security 

Income (SSI); Unemployment Insurance (UI); Workers’ Compensation (WC); National School Lunch 

Program (NSLP); and Women, Infants, and Children (WIC). The surveys are: American Community 

Survey (ACS), Consumer Expenditure (CE) Survey, Current Population Survey (CPS), Panel Study of 

Income Dynamics (PSID), and Survey of Income and Program Participation (SIPP).

j_meyer_293.indd 12 7/30/15 10:02 AM
Household Surveys in Crisis 13

Meyer, Mok, and Sullivan (2015), we report several sets of conditions under which 

equality of the dollar and month bias implies that dollar amounts are reported 

correctly on average. The Dollar Bias estimates are only slightly larger in absolute 

value than the Month Bias estimates, suggesting that there is a small amount of 

underreporting of dollars conditional on receipt, nevertheless. In the case of the 

CPS and the PSID, the evidence suggests that total dollars and months are understated by similar amounts, again suggesting that conditional on reporting receipt, 

the monthly benefits are reported about right on average.

For Old-Age and Survivors Insurance (OASI) and Social Security Disability 

Insurance (SSDI), we see similar biases for monthly receipt and dollar receipt, with 

the bias for dollar receipt being slightly larger (in absolute value), again suggesting 

that most of the downward bias results from failure to report receipt rather than 

underreporting the dollar amount of benefits conditional on reporting receipt. For 

Supplemental Security Income, the bias for dollar receipt is actually smaller in absolute value than the bias for monthly receipt for each survey other than the Survey of 

Income and Program Participation, where the bias is larger but positive, all of which 

suggests some overreporting of dollars conditional on reporting receipt.6

The average biases in monthly participation reporting for the National School 

Lunch Program (NSLP) and for the Special Supplemental Nutrition Program for 

Women, Infants, and Children (WIC) are also reported in the bottom panel of 

Table 1. Reporting of NSLP months is quite low for both the Panel Study of Income 

Dynamics (PSID) and the Current Population Survey (CPS), which both have an 

average bias of about 50 percent. In the Survey of Income and Program Participation (SIPP), on the other hand, the bias is positive, indicating that more months of 

participation are reported than we see in the administrative data. This result is likely 

due in part to our assumptions that all eligible family members (ages 5–18) receive 

lunches and that they do so for all four of the reference months of a given wave of 

the SIPP. WIC is also underreported significantly. The average bias for monthly WIC 

receipt in the CPS, PSID, and SIPP ranges from 19 to 34 percent.

This large bias in mean receipt of transfer programs has been increasing over 

time. Table 2 reports estimates from regressions of annual estimates of the proportional bias in dollar reporting on a constant and a time trend for various years from 

1967 to 2012 for the five surveys and seven programs. Most household reports of 

transfer programs in the Consumer Expenditure (CE) Survey, Current Population 

Survey (CPS), and Panel Study of Income Dynamics (PSID) show a significant increase 

in the downward bias—that is, a decline in dollar reporting over time. The downward 

bias in mean dollars reported of AFDC/TANF in the CPS, for example, increases by 

about one percentage point each year. The time trends in bias in the Survey of Income 

6 For Old-Age and Survivors Insurance, Social Security Disability Insurance, and Supplemental Security 

Income, the surveys other than the Survey of Income and Program Participation do not report monthly 

participation, only annual participation. Since our administrative numbers are for monthly participation, 

we use the relationship between average monthly and annual participation calculated in the SIPP to 

adjust the estimates from the other sources. This adjustment step likely induces some error that accounts 

for the weaker similarity between the bias for monthly and dollar receipt.

T2

j_meyer_293.indd 13 7/30/15 10:02 AM
14 Journal of Economic Perspectives

and Program Participation (SIPP) and the American Community Survey (ACS) are 

less pronounced. The exceptions to the general rise in bias are the programs Old-Age 

Survivors Insurance (OASI) and Supplemental Security Income (SSI), which have 

rising reporting rates in most cases. However, in the case of SSI in the SIPP, rising 

reporting leads to greater bias because the bias is always positive in recent years.7

The implication that measurement error in survey responses to government 

programs has grown over time is consistent with findings from Gathright and Crabb 

(2014), who calculate measurement error directly by linking Survey of Income 

and Program Participation data to Social Security Administration data for the 

7 Estimates consistent with those reported in Tables 1 and 2 are available in previous studies for some 

surveys for a subset of years and programs including: Coder and Scoon-Rogers (1996) for five of our 

programs for 1984 and 1990 for the Current Population Survey and the Survey of Income and Program 

Participation; Roemer (2000) for the same five programs for 1990–1996 for the CPS and the SIPP; 

Wheaton (2007) for four programs between 1993 and 2005 in the CPS and a shorter period in the 

SIPP; and Duncan and Hill (1989) for the CPS and Panel Study of Income Dynamics for earlier years.

Table 2

Trend in Proportional Bias in Mean Dollars Reported in Survey (Including Those 

Imputed), by Program and Survey

Social Security

AFDC/TANF FSP/SNAP OASI SSDI SSI UI WC

ACS −0.96 0.08 −0.68 3.50

(0.87) (0.07) (0.11)*** (1.11)**

12 12 12 12

CE −1.87 −1.1 0.07 −0.51 0.05 −0.74 −2.33

(0.43)*** (0.43)** (0.23) (0.23)** (0.27) (0.19)*** (0.38)***

33 33 33 33 33 33 33

CPS −0.71 −0.59 0.20 −0.61 0.41 −0.39 −0.71

(0.20)*** (0.09)*** (0.02)*** (0.08)*** (0.12)*** (0.19)* (0.16)***

37 34 45 45 38 26 25

PSID −1.04 −0.93 0.40 −0.62 −0.04 −0.47 −0.46

(0.12)*** (0.27)*** (0.10)*** (0.23)** (0.26) (0.16)*** (0.12)***

36 38 36 36 34 30 30

SIPP −0.46 −0.06 0.05 −0.33 1.52 −0.45 −0.50

(0.34) (0.15) (0.18) (0.49) (0.37)*** (0.22)* (0.10)***

29 30 30 30 30 30 29

Notes: For each cell, we report the year coefficient from a regression of the proportional bias in 

percentages on a constant and year, with its standard error underneath, followed by the sample 

size, where each observation is a year. The number of years varies across survey and program with as 

many 45 years for Old-Age and Survivors Insurance (OASI) in the Current Population Survey (CPS) 

(1967–2012, 1969 missing) and as few as 12 for the ACS (2000–2011). The regressions correct for 

first-order autocorrelation using the Prais–Winsten procedure. For titles of all the surveys and transfer 

programs, see the note to Table 1.

***, **, and *, indicate that the coefficient is statistically significantly different from zero at the 1, 5, and 

10 percent levels, respectively.

j_meyer_293.indd 14 7/30/15 10:02 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 15

Supplemental Security Income and the Old-Age Survivors and Disability Insurance 

programs. An added benefit of such linking is that one can identify false positives 

and false negatives. Their analysis shows that false positive and false negative rates for 

reported receipt and the mean absolute deviation of the reported benefit amount 

from the administrative amount increased between the 1996 and 2008 panels of 

the SIPP for both SSI and OASDI. During this period, the mean absolute error in the 

benefit amount increased by 70 percent for OASDI and by 60 percent for SSI.

The underreporting of transfer income in surveys has profound implications 

for our understanding of the low-income population and the effect of government 

programs for the poor. Accounting for underreporting of receipt, and substantial 

reporting and imputation error in amounts conditional on correctly reporting receipt, 

sharply changes what one learns from the survey data. Meyer and Mittag (2015) link 

data on four transfer programs (SNAP, TANF, General Assistance, and Housing Subsidies) to the New York data from the Current Population Survey over a four-year period 

from 2008–2011. In the survey data, 43 percent of SNAP recipients and 63 percent 

of public assistance recipients are not recorded as receiving benefits. Accounting for 

the survey errors more than doubles the estimate of the income of those who are 

reported to have income below half the poverty line. It leads the reported poverty 

rate to fall by 2.5 percentage points for the entire population and over 11 percentage 

points for single mothers. It nearly doubles the poverty-reducing effect of the four 

programs overall, and increases it by a factor of over 1.5 for single mothers. The share 

of single mothers with no earnings or program receipt is cut in half.

Is the declining quality of survey data unique to transfer income? One might 

argue that potential reasons for declining quality that might be unique to transfer 

income, such as rising stigma or less recognition of the general program names, 

make it a special case. But as we argue below, these reasons do not appear to explain 

the sharp rise in bias that we find.

The evidence on whether measurement error has grown over time for other 

outcomes is limited, but this evidence suggests the problem of declining survey 

quality goes well beyond transfer income.8

 Reporting of self-employment income 

has worsened, but there is no clear trend for wage and salary income and dividends 

in the Current Population Survey (CPS) and the Survey of Income and Program 

Participation (Coder and Scoon-Rogers 1996; Roemer 2000). Comparing earnings aggregates from the Survey of Consumer Finances with those computed using 

IRS Statistics of Income data, Johnson and Moore (2008) find that respondents 

overreport earnings, and this overreporting has worsened over time. They also 

find a sharp increase in pension income underreporting. Barrow and Davis (2012) 

compare reported postsecondary enrollment in the October CPS to Integrated 

Postsecondary Education Survey administrative data, showing that CPS reporting 

of type of college attended has gotten worse, though error in reporting of overall 

enrollment has remained stable. Other studies have shown that measurement 

8 Many studies document substantial bias in levels for other outcomes; for a summary, see Bound, Brown, 

and Mathiowetz (2001).

j_meyer_293.indd 15 7/30/15 10:02 AM
16 Journal of Economic Perspectives

error in pension contributions has grown over time (Dushi and Iams 2010). Also, 

Bee, Meyer, and Sullivan (2015) show that while reporting rates for some of the 

biggest components of consumption have remained stable over time, there have 

been noticeable declines for some categories such as food away from home, shoes 

and clothing, and alcoholic beverages. Future research on changes in bias in other 

outcomes would be a valuable extension to this literature.

Decomposing the Overall Bias

The bias estimates we present in Tables 1 and 2 are based on aggregate data, 

and for that reason they reflect not just measurement error but also coverage error 

(which arises when the sampling frame does not properly represent the underlying 

population) and error due to unit and item nonresponse. For several reasons, we 

argue that the most important source of the overall bias is measurement error.

Coverage error could explain some of the significant underreporting we 

find if the sampling frame for the surveys we examine (typically based on the 

noninstitutionalized Census population) does not capture the entire population 

that receives benefits. This argument about underweighting is essentially an argument about individuals being missed in the Census count.9

 Although we do not have 

undercount data for those who receive transfer income, estimates of the overall 

Census undercount are small, particularly relative to most of our bias estimates for 

transfer reports (Hogan 1993; Robinson, Amed, Das Gupta, and Woodrow 1993). 

Moreover, undercount estimates have declined over time, and the estimates for the 

2010 Census actually suggest an overcount (US Census 2012).

There are also reasons to believe that bias resulting from unit and item 

nonresponse might be small. While unit nonresponse is surely nonrandom with 

respect to receipt of transfer income, appropriate weighting may offset much of this 

bias. Similarly, item nonresponse also appears to be nonrandom, but will not lead 

to bias in mean reports if imputations are on average accurate.

Empirical studies relying on linked administrative and survey microdata 

support these arguments. Bee, Gathright, and Meyer (2015), for example, show that 

for income in the Current Population Survey, unit nonresponse leads to remarkably 

little bias in the distribution of income. The estimates of bias from studies linking 

survey and administrative microdata that are most comparable to ours using aggregate data come from Marquis and Moore (1990), which, we should point out, relies 

on survey data from 30 years ago. Their bias estimates for months of receipt are 

reported in the first column of Table 3. The second column reports our estimated 

9 We discuss issues related to the institutionalized population that receives transfers in the following section. 

As a check, for each survey and year, we have confirmed that our weighted population totals are close to 

Census population estimates. The sample weights in the Panel Study of Income Dynamics are not appropriate for weighting to the complete population in some years. We adjust them in a manner suggested by 

the PSID staff, and the appendix to Meyer, Mok, and Sullivan (2015) provides details.

T3

j_meyer_293.indd 16 7/30/15 10:02 AM
Household Surveys in Crisis 17

bias based on comparisons of aggregate survey and administrative data for the same 

year and survey as Marquis and Moore but not the same months or states. The 

bias we calculate in the second column is a function of sample weighting, coverage 

error, unit and item nonresponse, and measurement error, while the bias in the first 

column is only a function of item nonresponse and measurement error. Thus, if the 

biases in each of these columns are similar, then this suggests that the combination 

of sample weighting, coverage error, and unit nonresponse is not that important 

relative to the other sources of bias. The results in Table 3 suggest that the weights—

as well as unit nonresponse and coverage error—are not a substantial source of 

bias because the bias estimates from the linked microdata are fairly close to our 

estimates using comparisons to aggregates. Our estimates are particularly close (or 

higher) for the Food Stamp Program and for Supplemental Security Income, which 

are programs that target the poor—a group that perhaps is most plausibly thought 

to be underweighted or underrepresented.

Through linked survey and administrative microdata, one can decompose our 

bias estimates into three different sources of error: unit nonresponse (combined 

with coverage error and weighting), item nonresponse, and measurement error. In 

Table 4 we report this decomposition of our estimates of dollar bias for the Food 

Stamp Program (FSP) and Public Assistance (combining Temporary Assistance for 

Needy Families and General Assistance) in three of our surveys in recent years using 

New York state data for 2007–2012. We find that the bias due to the combination of 

coverage error, unit nonresponse, and weighting is substantial, the bias due to item 

nonresponse is small, and the bias due to measurement error is always larger than 

the other sources of bias combined. The combined coverage, unit nonresponse, 

and weighting bias varies from −0.049 to −0.096 for the FSP and −0.100 to −0.154 

for Public Assistance across the three surveys. The item nonresponse bias varies 

from −0.020 to −0.067 for the FSP and −0.022 to −0.057 for Public Assistance. The 

T4

Table 3

Proportional Bias Estimates from Microdata and Aggregate Data Compared

Transfer program

Microdata bias estimate 

due to unit nonresponse 

and measurement error

Aggregate data

bias estimate due to 

all sources of error

Aid to Families with Dependent Children (AFDC) −0.39 −0.21

Food Stamp Program (FSP) −0.13 −0.15

Old-Age, Survivors, and Disability Insurance (OASDI) 0.01 −0.06

Supplemental Security Income (SSI) −0.12 −0.14

Notes: The estimates using microdata are from Marquis and Moore (1990) and use data from the 

Survey of Income and Program Participation (SIPP) over June 1983 to May 1984 for months of 

receipt in Florida, New York (OASDI and SSI only), Pennsylvania, and Wisconsin. The aggregate data 

we use for the estimates in column 2 are averages of 1983 and 1984 average monthly participation 

for the entire United States. We also assume Old-Age, Survivors, and Disability Insurance (OASDI) 

participation is the sum of Old-Age and Survivors Insurance (OASI) and Social Security Disability 

Insurance (SSDI) participation.

j_meyer_293.indd 17 7/30/15 10:02 AM
18 Journal of Economic Perspectives

bias due to measurement error is substantial for the FSP, ranging from −0.121 to 

−0.267, and for Public Assistance it is even larger, ranging from −0.529 to −0.584.

Direct evidence of substantial measurement error is not restricted to these two 

programs. Through linked survey and administrative microdata, Gathright and 

Crabb (2014) document substantial measurement error in receipt and amounts of 

Supplemental Security Income and the Old-Age Survivors and Disability Insurance 

in the Survey of Income and Program Participation, and this measurement error is 

rising over time.

Methodological Issues when Comparing Aggregate Data

Comparing weighted microdata from surveys to administrative aggregates is an 

attractive approach for evaluating survey bias because it can be done easily for many 

years and across many surveys. However, this approach also has some important 

limitations including possible differences between the survey and administrative 

populations, and incomplete information on benefit receipt in some surveys. An 

additional concern that we will not discuss here is that by looking at net measures 

of bias, we are missing the extent to which a rise in false negative reports could be 

Table 4

Decomposition of Proportional Bias in Dollars Received into Its Sources Using 

Microdata

Survey Program

Bias due to 

combination of 

coverage, unit 

nonresponse, and 

weighting 

Bias due to item 

nonresponse

Bias due to 

measurement 

error

Total bias due to all 

sources of error

ACS Food Stamps −0.096 na* na* na*

Public Assistance −0.154 −0.022 −0.529 −0.705

CPS Food Stamps −0.049 −0.067 −0.267 −0.382

Public Assistance −0.106 −0.057 −0.563 −0.726

SIPP Food Stamps −0.056 −0.020 −0.121 −0.197

Public Assistance −0.100 −0.043 −0.584 −0.727

Source: Authors’ calculations based on New York State data for 2007–2012 from Celhay, Meyer, and 

Mittag (2015).

Notes: We calculate the bias due to the combination of errors in coverage, weighting, and unit nonresponse 

as the ratio of weighted administrative program dollars received by all linked households in the Current 

Population Survey to total administrative dollars paid out minus one. We calculate the bias due to item 

nonresponse as weighted dollars imputed to those not responding to the benefit question minus the 

dollars actually received by these households as a share of total dollars paid out. Finally, we calculate the bias 

due to measurement error as the dollars recorded by non-imputed respondents minus true dollars received 

as a share of total dollars paid out. The surveys are the American Community Survey (ACS), the Current 

Population Survey (CPS), and the Survey of Income and Program Participation (SIPP).

*Food stamp dollars received are not reported in these years of the ACS. 

j_meyer_293.indd 18 7/30/15 10:02 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 19

counterbalanced by a rise in false positive reports. Most of these problems are not 

present when linking microdata at the household level.

Survey and Administrative Data Populations Do Not Always Align

Our household survey totals do not include those living outside the 50 states and 

the District of Columbia, the institutionalized, or decedents. We make a number of 

adjustments in order to make the administrative and survey data totals comparable 

(for a full description, see Meyer, Mok, and Sullivan 2015). For example, we exclude 

from the administrative totals payments to those in US territories and those outside 

the United States. Where such information is not available, we subtract estimates 

of the share of such payments obtained from years when this information is available. 

For most programs, these adjustments are typically small, ranging from 0.02 percent 

(Supplemental Security Income) to about 3 percent (Social Security Disability Insurance). The notable exception is the Food Stamp Program, where dollars paid to US 

territories constituted about 10 percent of the total prior to 1982.

As another example, to adjust for the fact that the institutionalized can receive

some benefits in the Social Security-related programs, we rely on data from the 

Decennial Censuses (which include the institutionalized) and the 2006 American Community Survey to determine the share of dollars that are likely missed 

in household surveys that do not cover the institutionalized. That the surveys do 

not include decedents is a potential concern because recipients of transfers in 

one calendar year may subsequently die before being interviewed in a household 

survey the next year. We do not adjust for decedents, but assuming that the weights 

for extrapolating the household survey results to the population are well-chosen, 

we expect the lack of a specific adjustment for decedents to have little effect on our 

estimates in most cases.10

Often the reference period for the administrative data (typically a fiscal year) 

does not exactly align with that for the survey data. We convert fiscal year administrative data to a calendar basis by weighting the fiscal years. Another noncomparability 

is that administrative data for transfer income are based on awardees, while the 

survey data typically provide information on the person to whom the benefit is paid. 

Awardees and payees may be different people. For example, adults may receive Social 

Security and Supplemental Security benefits on behalf of their children. Most household surveys provide little information about exactly who is the true awardee of the 

10 Previous studies have adjusted for decedents by applying age-, gender-, and race-specific death 

rates to the data (Roemer 2000). However, if survey weights have previously been calculated to match 

survey-weighted population totals with universe population estimates by age, gender, and race, then 

such an adjustment is unwarranted. A case could be made for adjusting the data if these characteristics 

are nonstationary (but such an adjustment is likely to be small) or if the adjustments were based on 

additional individual characteristics that are not used to determine weights but are related to death, such 

as receipt of Social Security Disability Insurance or Supplemental Security Income or other programs, 

but we do not have this information. Consequently, our estimates of bias for SSDI and SSI are likely to 

be overstated somewhat, since recipients likely have a higher mortality rate than the average person of 

their age, gender, and race, and consequently are more likely to miss the interview the following year.

j_meyer_293.indd 19 7/30/15 10:02 AM
20 Journal of Economic Perspectives

benefit, although the Survey of Income and Program Participation does provide some 

partial information about who is the true awardee of Social Security benefits.

Some Surveys Provide Incomplete Information on the Receipt of Benefits

In certain years of the Panel Study of Income Dynamics, for example, we only 

have information about benefit receipt for the household head and the spouse. 

We address this issue by using the share of total benefits received by non-head, 

non-spouse family members in other years and scaling up the aggregates accordingly. This adjustment assumes that these shares change slowly over time. Non-head, 

non-spouse dollars received are typically under 10 percent of family dollars but 

exceed 20 percent for Supplemental Security Income in a few years.

Sometimes surveys do not distinguish between different types of benefits 

received. In some cases, we cannot distinguish between different types of Social 

Security income. In this situation, we apply the Old-Age Survivors and Disability 

Insurance dollar proportions from published totals to determine participation in 

these programs. Applying these proportions essentially assumes that an individual 

can only receive benefits from one of these programs, but not both. In practice, 

however, individuals can receive benefits from both programs in a year—most 

commonly those whose disability benefit switches automatically to an old-age benefit 

when they reach retirement age. This issue leads to a slight bias downward in our 

Social Security retirement and disability participation estimates.

Reasons for Nonresponse and Errors

Why are nonresponse and measurement error so prevalent? Why have 

these threats to survey quality grown over time? Regarding the high rate of unit 

nonresponse, disinterest or lack of time appear to be important factors. Based on 

data recorded by interviewers for two household surveys—the 1978 National Medical 

Care Expenditure Survey and the 2008 National Health Interview Survey—the most 

common reasons given for unit nonresponse include that potential respondents are 

not interested, do not want to be bothered, or are too busy, while privacy concerns 

also seem to be important (Brick and Williams 2013, p. 39; National Research 

Council 2013). Reasons for unit nonresponse are often divided into three categories: noncontact, refusals, and other reasons (such as language problems or poor 

health). Failure to contact has also been offered as a possibility by some who have 

noted the rise of gated communities and the decline of land-line phones, which 

could make door-to-door or phone surveys more difficult. However, the rise in 

nonresponse in household surveys has been primarily driven by refusals by those 

who are contacted (Brick and Williams 2013), and thus we will not emphasize these 

potential “technological” reasons for noncontact.

One might suspect that the reasons for item nonresponse and measurement 

error are closely related to those for unit nonresponse, though the literature on survey 

quality has tended to focus on unit nonresponse separately. One reason the three 

j_meyer_293.indd 20 7/30/15 10:02 AM
Household Surveys in Crisis 21

sources of error may be related would arise if some potential respondents are just less 

cooperative so that their participation is worse in many dimensions. Some research 

has examined this hypothesis. For example, Bollinger and David (2001) show that 

those who respond to all waves of a Survey of Income and Program Participation 

panel report participation in the Food Stamp Program more accurately than those 

who miss one or more waves. Similarly, Kreuter, Muller, and Trappmann (2014) show 

in a German survey that hard-to-recruit respondents provided less-accurate reports of 

welfare benefit receipt than those easy to recruit. The reasons for item nonresponse 

likely differ depending on the nature of the questions. In the case of earnings, Groves

and Couper (1998) suggest that the most important reason for nonresponse is 

concerns about confidentiality but that insufficient knowledge is also important.

The reasons for the underreporting of transfer benefits in household surveys 

have been catalogued by several authors; Marquis and Moore (1990) provide 

nice examples for the Survey of Income and Program Participation, while Bound, 

Brown, and Mathiowetz (2001) and Groves (2004) provide more general discussions. Interviewees may forget receipt or confuse the names of programs. They may 

misremember the timing of receipt or who are the true recipients of a program 

within a family. Errors may be due to a desire to shorten the time spent on the interview, the stigma of program participation, the sensitivity of income information, or 

changes in the characteristics of those who receive transfers. Survey and interviewer 

characteristics such as the interview mode (in person or by phone) and respondent 

type (self or proxy) may also matter for the degree of underreporting. Notice that 

all of these explanations may lead to item nonresponse, measurement error conditional on responding, or both.

Information on the extent of underreporting and how it varies across programs, 

surveys, and time should help in differentiating among the explanations for underreporting. For example, one standard explanation of underreporting is the stigma 

of reporting receipt of “welfare” programs, and the inclination to give “socially 

desirable” answers (Sudman and Bradburn 1974). This explanation is consistent 

with the low reporting rates of four of the programs most associated with “welfare” 

or idleness: Temporary Assistance to Needy Families, the Food Stamp Program, 

Unemployment Insurance, and the Special Supplemental Nutrition Program for 

Women, Infants, and Children. However, other patterns of reporting by program do 

not fit with a stigma explanation for underreporting. Workers’ Compensation has 

the greatest bias but is presumably not a program that greatly stigmatizes its recipients, given that the program is for those injured while working.

Another common explanation for underreporting is that interviewees forget 

receipt, misremember the timing of receipt, or confuse the names of programs. 

Such issues should arguably be less common for programs that are received regularly, such as Old-Age and Survivors Insurance, Social Security Disability Insurance, 

and Supplemental Security Income. And, as shown in Table 1, these three programs 

typically have smaller bias than the other transfer programs we examine. However, 

the estimates in Table 1 show that the proportional bias for these programs is still 

large, particularly for SSDI and SSI, although this could be due to greater stigma 

j_meyer_293.indd 21 7/30/15 10:02 AM
22 Journal of Economic Perspectives

for these two programs. Also, all three of these Social Security programs have item 

nonresponse rates that are no better than for some programs with less-regular 

receipt (see Figures 2A and B).

Why has survey quality deteriorated over time? Several studies have considered this question, mostly focusing on unit nonresponse. The traditional reasons 

proposed include increasing urbanization, a decline in public spirit, increasing time 

pressure, rising crime (this pattern reversed long ago), increasing concerns about 

privacy and confidentiality, and declining cooperation due to “oversurveyed” households (Groves and Couper 1998; Presser and McCullogh 2011; Brick and Williams 

2013). The continuing increase in survey nonresponse as urbanization has slowed 

and crime has fallen make these less likely explanations for present trends. Tests of 

the remaining hypotheses are weak, based largely on national time-series analyses 

with a handful of observations. Several of the hypotheses require measuring societal 

conditions that can be difficult to capture: the degree of public spirit, concern about 

confidentiality, and time pressure. The time pressure argument seems inconsistent 

with the trend toward greater leisure (Aguiar and Hurst 2007) and would suggest 

that those with higher incomes and less leisure should be less likely to respond to 

surveys—a pattern that is at best weakly present. We are unaware of strong evidence 

to support or refute a steady decline in public spirit or a rise in confidentiality 

concerns as a cause for declines in survey quality. Some of these hypotheses seem 

amenable to a geographically disaggregated time-series approach, but little work 

seems to have been done along those lines. Groves and Couper (1998) show that 

nonresponse rates differ across demographic groups; cooperation is lower among 

single person households and households without young children, for example. But 

more research is needed on whether changes in demographic characteristics such 

as these can account for declining survey quality.

Changes in survey procedures over time can also provide evidence on the 

reasons for changes in underreporting of receipt of government transfers. The 

reduction or elimination of in-person interviewing seems to have had little effect on 

reporting rates. For example, reporting rates do not change much after the 1996 

reduction of in-person interviewing in the Survey of Income and Program Participation. This result is consistent with the observation by Groves (2004) that there is 

no robust evidence of a difference in errors between in-person and phone interviewing. Reporting for transfer programs does not appear to be sensitive to whether 

or not the interviewer explicitly mentions the name of a program (Meyer, Mok, 

and Sullivan 2015). There is some evidence that adding “bracketed” responses—for 

example, starting in 2001, when a specific amount is not provided, the Consumer 

Expenditure Survey asks interviewees whether the amount falls within certain 

ranges—leads to increased reporting rates for some programs, but this evidence is 

not consistent across programs (Meyer, Mok, and Sullivan 2015).

Our own reading of the evidence supports the hypothesis that “oversurveyed” 

respondents are less cooperative resulting in greater nonresponse and measurement error. Presser and McCullogh (2011) document a sharp rise in the number 

of government surveys administered in the United States over the 1984–2004 

j_meyer_293.indd 22 7/30/15 10:02 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 23

period. They report that a series of random-digit-dial telephone surveys found that 

the share of Americans surveyed in the past year more than quadrupled between 

1978 and 2003 (Council for Marketing and Opinion Research 2003). They also 

note that real expenditures on commercial survey research increased by more than 

4 percent annually for the 16 years ending in 2004. We suspect that talking with 

interviewers, once a rare chance to tell someone about your life, is now crowded out 

by an annoying press of telemarketers and commercial surveyors. The decline in 

unit and item response rates may not fully reflect the secular decline in the willingness of households to cooperate because survey administrators have tried to offset 

a declining household willingness to be surveyed by altering their methods. For 

example, Groves and Couper (1998) note cases where the number of attempted 

contacts with respondents has increased in order to stem the rise in nonresponse.

Taken together, the existing evidence does not provide a complete explanation for why survey quality has deteriorated over time. That some households 

are oversurveyed seems to contribute to the problem, but other explanations are

likely important as well. There is a clear need for further research to fill the gaps 

in this literature.

The Future of Microdata

As the quality of conventional household survey data has declined, the availability of alternative data for research and policy analysis has increased. For empirical 

research, while the role of survey data has declined, that of administrative data has 

risen; Chetty (2012) reports that the share of nondevelopment-microdata-based 

articles in the “top four” general interest economics journals that relied on survey 

data fell from about 60 to 20 percent between 1980 and 2010, while the share of 

articles relying on administrative data rose from about 20 to 60 percent. A number 

of standard sources of administrative data have already been mentioned in this article; 

these include data from tax records and from transfer programs. In addition, the use 

of alternative forms of administrative data has been increasing. For example, the work 

surveyed in Einav and Levin (2014) offers examples like the use of administrative data 

on student test scores to measure teacher value added, or data on earnings to assess 

the effect of the spread of broadband Internet access into different areas.

Administrative data offer a bundle of advantages and disadvantages. The datasets 

often have large sample sizes and low measurement error, permitting the estimation 

of small effects and the testing of subtle hypotheses. The data often allow longitudinal 

measurement, which is not possible in cross-sectional household data and can be difficult in longitudinal surveys with substantial attrition. When changes occur in policy or 

practice, especially when those changes affect only certain populations or geographic 

areas, administrative data often enable the use of experimental or quasi-experimental 

research methods. On the other hand, administrative data sets are typically not 

designed for academic research, and they can be quite heterogeneous in origin, 

topic, and quality. Researchers can find it difficult to access these data, whether for 

j_meyer_293.indd 23 7/30/15 10:02 AM
24 Journal of Economic Perspectives

original research or for replication. Administrative data often cover only a limited 

set of characteristics of individuals, and these variables are often of low quality, especially for types of information that are collected but are not directly needed by the 

program for administration or other purposes. Also, administrative data sources often 

have incomplete coverage and are nonrepresentative, making the data unsuitable for 

drawing generalizable conclusions or examining population trends. 

The limitations of administrative data can potentially be addressed by linking 

to household survey data. Many recent reports by government agencies, advocacy groups, and politicians have pointed to the advantages of administrative data 

linked to survey data (for example, Burman et al. 2005; Brown et al. 2014; Office of 

Management and Budget 2014; US House of Representatives 2014). These reports 

have noted the usefulness of such data for a wide variety of policy analyses. President 

Obama’s 2016 budget calls for $10 million for the Census Bureau to “to accelerate 

the process of acquiring additional key datasets . . . ; expand and improve its infrastructure for processing and linking data; and improve its infrastructure for making 

data available to outside researchers” (White House 2015). A recent bipartisan bill 

would establish a commission to recommend the structure of a clearinghouse for 

administrative and survey data (US Senate 2015).

Linking administrative microdata to survey microdata may improve the quality 

of survey data by providing more accurate information for some variables or by 

shortening the interview length and reducing the burden on survey respondents 

who no longer need to be asked questions they might be reluctant to answer. Such 

data linking can also be useful for improving the existing stock of data. For example, 

Nicholas and Wiseman (2009, 2010) and Meyer and Mittag (2015) show how one 

can use linked data to correct for underreporting of transfer income when calculating poverty rates.

A number of examples already exist of linked survey and administrative data. 

The Health and Retirement Survey is linked to administrative data on Social Security 

earnings and claims, as well as to Medicaid data. The National Center for Health Statistics is currently linking several of its population-based surveys to administrative data. 

Many randomized experiments of welfare and training programs linked household 

survey instruments to Unemployment Insurance earnings records or other administrative datasets (Grogger and Karoly 2005). Ad hoc examples within government have 

also produced useful research such as the work by Scherpf, Newman, and Prell (2014) 

using administrative data on the Supplemental Nutrition Assistance Program.

Surveys have explored alternative methods to improve survey quality, including 

multi- or mixed-mode methods to collect information from respondents (Citro 

2014). Use of the Internet has become an increasingly more common mode. In 

addition to standard mail, telephone, and face-to-face interview modes, the American Community Survey now allows respondents to respond online. These new 

methods may, in some cases, reduce costs and have the potential to improve data 

quality, but whether these approaches reduce bias remains to be seen.

Much of what we know about the conditions of the American public and the 

information that is used for public policy formation comes from national survey 

j_meyer_293.indd 24 7/30/15 10:02 AM
Household Surveys in Crisis 25

data. The ongoing deterioration of household survey data documented in this 

paper seems unlikely to end, especially as surveying for commercial purposes and 

the feeling of being oversurveyed continues to grow. Without changes in data 

collection and availability, the information infrastructure to formulate and evaluate 

public policies and to test social science theories will degrade. Efforts to improve 

national survey data and to reduce nonresponse bias and measurement error are 

worthwhile. But perhaps the most productive step toward improving the quality of 

data available for social science research—rather than just seeking to slow the pace 

of erosion in the quality of that data—is to increase the availability of administrative datasets and to find additional ways to link them to household survey data and 

substitute administrative variables for survey questions in a timely fashion.

■ We are grateful for the assistance of many New York Office of Temporary and Disability 

Assistance (OTDA) and Census Bureau employees including George Falco, Dave Dlugolecki, 

Graton Gathright, Joey Morales, Amy O’Hara, and Frank Limehouse. The microdata analysis 

was conducted at the Chicago Census Research Data Center by researchers with Special Sworn 

Status and the results were reviewed to prevent the disclosure of confidential information. 

We would like to thank Pablo Celhay for excellent research assistance. We also thank Dan 

Black, Constance Citro, Michael Davern, Nikolas Mittag and participants at seminars at the 

American Enterprise Institute and the Federal Deposit Insurance Corporation for their helpful 

comments. The views expressed herein are those of the authors and do not necessarily reflect 

the views of the New York OTDA or the US Census Bureau.

References

Abowd, John, and Martha Stinson. 2013. 

“Estimating Measurement Error in Annual Job 

Earnings: A Comparison of Survey and Administrative Data.” Review of Economic Statistics 95(5): 

1451–67.

Aguiar, Mark, and Erik Hurst. 2007. “Measuring 

Trends in Leisure: The Allocation of Time over 

Five Decades.” Quarterly Journal of Economics 

122(3): 969–1006.

Alwin, Duane F. 2007. Margins of Error: A Study 

of Reliability in Survey Measurement. John Wiley & 

Sons: Hoboken, NJ.

Andridge, Rebecca R., and Roderick J. A. 

Little. 2010. “A Review of Hot Deck Imputation 

for Survey Non‐Response.” International Statistical 

Review 78(1): 40–64.

Atrostic, B. K., Nancy Bates, Geraldine Burt, 

and Adrian Silberstein. 2001. “Nonresponse in 

US Government Household Surveys: Consistent 

Measures, Recent Trends, and New Insights.” 

Journal of Official Statistics 17(2): 209–226.

Barrow, Lisa, and Jonathan Davis. 2012. “The 

Upside of Down: Postsecondary Enrollment in the 

Great Recession.” Economic Perspectives, Federal 

Reserve Bank of Chicago, 4Q: 117–29. 

Battaglia, Michael P., Mina Khare, Martin R. 

Frankel, Mary Cay Murray, Paul Buckley, and 

Saralyn Peritz. 2007. “Response Rates: How Have 

They Changed and Where Are They Headed?” 

In Advances in Telephone Survey Methodology

edited by James M. Lepkowski, Clyde Tucker, J. 

Michael Brick, Edith D. de Leeuw, Lilli Japec, 

j_meyer_293.indd 25 7/30/15 10:02 AM
26 Journal of Economic Perspectives

Paul J. Lavrakas, Michael W. Link, and Roberta L. 

Sangster, 529–60. New York, NY: Wiley.

Bee, C. Adam, Graton Gathright, and Bruce D. 

Meyer. 2015. “Bias from Unit Non-Response in the 

Measurement of Income in Household Surveys.” 

Unpublished paper, University of Chicago.

Bee, C. Adam, Bruce Meyer, and James Sullivan.

2015. “The Validity of Consumption Data: Are 

the Consumer Expenditure Interview and Diary 

Surveys Informative?” In Improving the Measurement 

of Consumer Expenditures, edited by Christopher 

Carroll, Thomas Crossley, and John Sabelhaus, 

204–240. University of Chicago Press.

Bollinger, Christopher, and Martin H. David.

2001. “Estimation with Response Error and 

Nonresponse: Food-Stamp Participation in the 

SIPP.” Journal of Business and Economic Statistics

19(2): 129–41.

Bollinger, Christopher R., and Barry Hirsch.

2006. “Match Bias from Earnings Imputation 

in the Current Population Survey: The Case of 

Imperfect Matching.” Journal of Labor Economics

24(3): 483–519.

Bound, John, Charles Brown, and Nancy 

Mathiowetz. 2001. “Measurement Error in Survey 

Data.” In Handbook of Econometrics, vol. 5, edited by 

James J. Heckman and Edward Leamer. Elsevier: 

Amsterdam.

Brick, Michael J., and Douglas Williams. 2013. 

“Explaining Rising Nonresponse Rates in CrossSectional Surveys.” Annals of the American Academy 

of Political and Social Science 645(1): 36–59.

Brown, Lawrence, Constance Citro, Carol 

House, Krisztina Marton, and Christopher Mackie.

2014. “The Past, Present, and Future of Federal 

Surveys: Observations from the Committee on 

National Statistics.” Conference proceedings 

from JSM ( Joint Statistical Meetings) 2014 - Social 

Statistics Section.

Burman, Len, Dan Feenberg, Austan Goolsbee, 

Charles Hulten, Bruce Meyer, John Karl Scholz, 

and Joel Slemrod. 2005. “Report on the State of 

Publicly Available Data and Statistics For the Study 

of Public Economics.” Report for the Committee 

on Economic Statistics of the American Economic 

Association. 

Celhay, Pablo, Bruce D. Meyer, and Nicholas 

Mittag. 2015. “Measurement Error in Program 

Participation.” Unpublished paper.

Chetty, Raj. 2012. “Time Trends in the Use 

of Administrative Data for Empirical Research.” 

NBER Summer Institute presentation. Available at 

the author’s website.

Citro, Constance F. 2014. “From Multiple Modes 

for Surveys to Multiple Data Sources for Estimates.” 

Survey Methodology 40(2): 137–61.

Council for Marketing and Opinion Research.

2003. “Respondent Cooperation and Industry 

Image Study.” Previously available at http://www.

cmor.org/rc/studies.cfm.

Coder, John, and Lydia Scoon-Rogers. 1996. 

“Evaluating the Quality of Income Data Collected 

in the Annual Supplement to the March Current 

Population Survey and the Survey of Income and 

Program Participation.” Housing and Household 

Economic Statistics Division, Bureau of the Census. 

https://www.census.gov/sipp/workpapr/wp215.pdf.

Curtin, Richard, Stanley Presser, and Elinor 

Singer. 2005. “Changes in Telephone Survey 

Nonresponse over the Past Quarter Century.” 

Public Opinion Quarterly 69(1): 87–98.

Davern, Michael. 2013. “Nonresponse Rates 

Are a Problematic Indicator of Nonresponse Bias 

in Survey Research.” Health Services Research 48(3): 

905–912.

de Leeuw, Edith, and Wim de Heer. 2002. 

“Trends in Households Survey Nonresponse: 

A  Longitudinal and International Comparison.” 

In Survey Nonresponse, edited by Robert M. Groves, 

Don A. Dillman, John Eltinge, and Roderick J.A. 

Little. Wiley.

Duncan, Greg J., and Daniel H. Hill. 1989. 

“Assessing the Quality of Household Panel Data: 

The Case of the Panel Study of Income Dynamics.” 

Journal of Business and Economic Statistics 7(4): 

441–52.

Dushi, Irena, and Howard Iams. 2010. “The 

Impact of Response Error on Participation Rates 

and Contributions to Defined Contribution 

Pension Plans.” Social Security Bulletin 70(1): 45–60. 

Einav, Liran, and Jonathan Levin. 2014. 

“Economics in the Age of Big Data.” Science 346 

(6210).

Gathright, Graton, and Tyler A. Crabb. 2014. 

“Reporting of SSA Program Participation in 

SIPP.” Working Paper, US Census Bureau. http://

www.eventscribe.com/2014/ASA-JSM/assets/

pdf/312130_92347.pdf. 

Grogger, Jeffrey, and Lynn A. Karoly. 2005. 

Welfare Reform: Effects of a Decade of Change.

Cambridge, MA: Harvard UP.

Groves, Robert M. 2004. Survey Errors and 

Survey Costs. Hoboken, NJ: Wiley.

Groves, Robert M. 2006. “Nonresponse Rates 

and Nonresponse Bias in Household Surveys.” 

Public Opinion Quarterly 70(5): 646–75.

Groves, Robert M., and Mick P. Couper. 1998. 

Nonresponse in Household Interview Surveys. New 

York, NY: Wiley.

Groves, Robert M., and Emilia Peytcheva. 2008. 

“The Impact of Nonresponse Rates on Nonresponse 

Bias.” Public Opinion Quarterly 72(2): 167–89.

Hogan, Howard. 1993. “The 1990 

Post-Enumeration Survey: Operations and 

j_meyer_293.indd 26 7/30/15 10:02 AM
Bruce D. Meyer, Wallace K. C. Mok, and James X. Sullivan 27

Results.” Journal of the American Statistical Association 88(423): 1047–60.

Huynh, Minh, Kalman Rupp, and James Sears.

2002. “The Assessment of Survey of Income and

Program Participation (SIPP) Benefit Data using 

Longitudinal Administrative Records.” Unpublished paper, Social Security Administration, 

http://www.oecd.org/std/36232630.pdf.

Johnson, Barry W., and Kevin Moore. 2008. 

“Differences in Income Estimates Derived from 

Survey and Tax Data.” 2008 SOI Paper Series.

Kreuter, Frauke, Gerrit Müller, and Mark 

Trappmann. 2014. “A Note on Mechanisms 

Leading to Lower Data Quality of Late or Reluctant Respondents.” Sociological Methods & Research

43(3): 452–64.

Marquis, Kent H., and Jeffrey C. Moore. 1990. 

“Measurement Errors in SIPP Program Reports.” 

In Proceedings of the 1990 Annual Research Conference, 

721–45. Washington, DC: US Bureau of the Census.

Massey, Douglas S., and Roger Tourangeau, 

eds. 2013. “The Nonresponse Challenge to Surveys 

and Statistics,” (special title of) volume 645, issue 1 

of Annals of the American Academy of Political and 

Social Science, pages 6–236.

Meyer, Bruce D., Robert Goerge, and Nicholas 

Mittag. 2014. “Errors in Survey Reporting and 

Imputation and Their Effects on Estimates of 

Food Stamp Program Participation.” Unpublished 

paper.

Meyer, Bruce D., and Nicholas Mittag. e: Implications for Poverty, Program Effectiveness and 

Holes in the Safety Net.” Unpublished paper, 

University of Chicago.

Meyer, Bruce D., Wallace K. C. Mok, and 

James X. Sullivan. 2015. “The Under-Reporting 

of Transfers in Household Surveys: Its Nature and 

Consequences.” NBER Working Paper 15181. 

Updated version June 2015.

Mishra, V. B. Barrere, R. Hong, and S. Khan.

2008. “Evaluation of Bias in HIV Seroprevalence 

Estimates from National Household Surveys.” Sexually Transmitted Infection 84(Supplement 1): i65-i70. 

Moore, Jeffrey C., Kent H. Marquis, and Karen 

Bogen. 1996. “The SIPP Cognitive Research 

Evaluation Experiment: Basic Results and Documentation.” Working Paper 212, Survey of Income 

and Program Participation, US Census Bureau.

National Research Council. 2011. The Future of 

Federal Household Surveys: Summary of a Workshop. K. 

Marton and J.C. Karberg, rapporteurs. Committee 

on National Statistics, Division of Behavioral and 

Social Sciences and Education. Washington, DC: 

The National Academies Press.

National Research Council. 2013. Nonresponse in 

Social Science Surveys: A Research Agenda. Edited by 

Roger Tourangeau and Thomas J. Plewes. Panel on 

a Research Agenda for the Future of Social Science 

Data Collection, Committee on National Statistics, Division of Behavioral and Social Sciences 

and Education, National Research Council. 

Washington, DC: The National Academies Press.

Nicholas, Joyce, and Michael Wiseman. 2009. 

“Elderly Poverty and Supplemental Security 

Income.” Social Security Bulletin 69(1): 45–73.

Nicholas, Joyce, and Michael Wiseman. 2010. 

“Elderly Poverty and Supplemental Security 

Income, 2002–2005.” Social Security Bulletin 70(2): 

1–30.

Office of Management and Budget. 2006. 

“Standards and Guidelines for Statistical Surveys.” 

September. Executive Office of the President, 

Washington, DC.

Office of Management and Budget. 2014. 

“Guidance for Providing and Using Administrative 

Data for Statistical Purposes.” Memorandum for 

the Heads of Executive Departments and Agencies, M-14-06, February. Executive Office of the 

President, Washington, DC.

Pennell, Steven G. 1993. “Cross-Sectional 

Imputation and Longitudinal Editing Procedures 

in the Survey of Income and Program Participation.” Prepared for the US Bureau of the Census. 

https://www.census.gov/sipp/workpapr/

wp9314.pdf.

Pew Research Center. 2012. “Assessing 

the Representativeness of Public Opinion 

Surveys,” May 15. http://www.people-press.org

/2012/05/15/assessing-the-representativeness

-of-public-opinion-surveys/.

Peytchev, Andrew. 2013. “Consequences of 

Survey Nonresponse.” Annals of the American 

Academy of Political and Social Science 645(1): 88–111.

Presser, Stanley, and Susan McCullogh. 2011. 

“The Growth of Survey Research in the United States: 

Government-Sponsored Surveys, 1984–2004.” Social 

Science Research 40(4): 1019–2024.

Reamer, Andrew D. 2010. “Surveying for 

Dollars: The Role of the American Community 

Survey in the Geographic Distribution of Federal 

Funds.” Brookings Report, July 2010.

Robinson, J. Gregory, Bashir Ahmed, Prithwis 

Das Gupta, and Karen A. Woodrow. 1993. “Estimation of Population Coverage in the 1990 United 

States Census Based on Demographic Analysis.” 

Journal of the American Statistical Association

88(423): 1061–71.

Roemer, Marc I. 2000. “Assessing the Quality 

of the March Current Population Survey and 

the Survey of Income and Program Participation 

Income Estimates, 1990–1996.” Working Papers, 

Census Bureau.

Scherpf, Erik, Constance Newman, and Mark 

Prell. 2014. “Targeting of Supplemental Nutrition 

j_meyer_293.indd 27 7/30/15 10:02 AM
28 Journal of Economic Perspectives

Assistance Program Benefits: Evidence from the 

ACS and NY SNAP Administrative Records.” Agricultural and Applied Economics Association 2014 

Annual Meeting, July 27–29, 2014, paper 174295.

Sears, James, and Kalman Rupp. 2003. 

“Exploring Social Security Payment History 

Matched with the Survey of Income and Program 

Participation.” Unpublished paper, Social 

Security Administration. http://www.oecd.org/

std/36232603.pdf.

Smith, Tom W., Peter V. Marsden, Michael 

Hout, Jibum Kim. 2013. General Social Surveys, 

1972–2012 [Cumulative File]. Codebook. ICPSR 

34802, Inter-University Consortium for Political 

and Social Research.

Steeh, Charlotte, Nicole Kirgis, Brian Cannon, 

and Jeff DeWitt. 2001. “Are They Really as Bad 

as They Seem? Nonresponse Rates at the End of 

the Twentieth Century.” Journal of Official Statistics

17(2): 227–47.

Sudman, Seymour, and Norman M. Bradburn.

1974. Response Effects in Surveys: A Review and 

Synthesis. Chicago: NORC/Aldine Publishing 

Company.

Taeuber, Cynthia, Dean M. Resnick, Susan 

P. Love, Jane Stavely, Parke Wilde, and Richard 

Larson. 2004. “Differences in Estimates of Food 

Stamp Program Participation Between Surveys 

and Administrative Records.” Working paper, US 

Census Bureau.

Tourangeau, Roger, Robert M. Groves, and 

Cleo D. Redline. 2010. “Sensitive Topics and Reluctant Respondents: Demonstrating a Link between 

Nonresponse Bias and Measurement Error.” Public 

Opinion Quarterly 74(3): 413–32.

US Census Bureau. 2001. “Survey of Income 

Program Participation Users’ Guide.” Third 

Edition. United States Department of Commerce, 

Bureau of the Census, Washington, DC.

US Census Bureau. 2003. “Meeting 21st Century 

Demographic Data Needs—Implementing the 

American Community Survey: Testing the Use of 

Voluntary Methods.” United States Department of 

Commerce, Bureau of the Census, Washington, DC.

US Census Bureau. 2012. “Census Bureau 

Releases Estimates of Undercount and Overcount 

in the 2010 Census.” United States Department of 

Commerce, Bureau of the Census, Washington, 

DC. http://www.census.gov/newsroom/releases/

archives/2010_census/cb12-95.html.

US Census Bureau. Various years (a). “Current 

Population Survey: Annual Social and Economic 

(ASEC) Survey Codebook.” United States Department of Commerce, Bureau of the Census, 

Washington, DC.

US Census Bureau. Various years (b). “Survey 

of Income and Program Participation Codebook.” 

United States Department of Commerce, Bureau 

of the Census, Washington, DC.

US Department of Health and Human Services.

2014. “National Health Interview Survey – Survey 

Description.” Centers for Disease Control and 

Prevention, Division of Health Interview Statistics, 

National Center for Health Statistics, Hyattsville, 

Maryland.

US Department of Labor (various years).

“Consumer Expenditure Interview Survey Public 

Use Microdata Documentation.” U.S. Department 

of Labor, Bureau of Labor Statistics, Division of 

Consumer Expenditure Surveys.

US House of Representatives. 2014. 

“Expanding Opportunity in America.” House 

Budget Committee Majority Staff, July 24.

US Senate. 2015. “Murray, Ryan Introduce 

Bill to Expand Data Use in Evaluating Federal 

Programs, Tax Expenditures.” News Release. 

http://www.murray.senate.gov/public/index.

cfm/2015/4/evidence-based-policy-murray-ryan

-introduce-bill-to-expand-data-use-in-evaluating

-federal-programs-tax-expenditures. 

US Social Security Administration. Various 

years. “Annual Statistical Supplement to the 

Social Security Bulletin.” US Social Security 

Administration, Office of Research, Evaluation 

and Statistics.

Wheaton, Laura. 2007. “Underreporting of 

Means-Tested Transfer Programs in the CPS and 

SIPP.” 2007 Proceedings of the American Statistical 

Association, Social Statistics Section.

White House. 2015. “The President’s Budget 

Fiscal Year 2016.” https://www.whitehouse.gov/

sites/default/files/omb/budget/fy2016/assets/

fact_sheets/building-and-using-evidence-to

-improve-results.pdf.

j_meyer_293.indd 28 7/30/15 10:02 AM
AUTHOR QUERIES 29

PLEASE ANSWER ALL AUTHOR QUERIES (numbered with “AQ” in the 

margin of the page). Please disregard all Editor Queries (numbered with 

“EQ” in the margins). They are reminders for the editorial staff.

AQ# Question Response

1. This sounds weird: “a sharp rise 

in the downward bias”. I would 

suggest going back to what you 

had: “Our results show a sharp 

rise in the bias in household 

survey estimates of receipt rates 

and dollars received for most 

programs, and it is a downward 

bias.”

2. I removed some “of’s” You had 

“of incomes at the bottom, of the 

rate of program receipt, and of 

the poverty-reducing effects . . .”

j_meyer_293.indd 29 7/30/15 10:02 AM