Analysis of Possible Nonresponse Bias in the 

National Crime and Victimization Survey 

Zhiwei Zhang1, Louise Woodburn2, Fritz Scheuren3

1 NORC/University of Chicago, 4350 East West Highway, Suite 800, Bethesda, Maryland zhang-zhiwei@norc.org

2

 Independent Consultant, Richmond, Virginia, r3louise@gmail.com 

3 NORC/University of Chicago, 4350 East West Highway, Suite 800, Bethesda, Maryland scheuren@aol.com 

1. Introduction

The measurement of crime and the validity and reliability of crime statistics have long been of concern to 

social scientists1

. For much of the twentieth century the Uniform Crime Reports (UCR) produced by the Federal 

Bureau of Investigation (FBI) were considered ―almost sacrosanct‖ as a source of official crime statistics in the 

United States2. However, by the late twentieth century there were a large number of studies questioning the extent to

which UCR statistics can be treated as an accurate and adequate measure of crime. 

To address these concerns, in 1973 the Bureau of Justice Statistics (BJS) introduced the National Crime

Victimization Survey (NCVS, formerly NCS), which is fielded by the US Census Bureau. The purpose of that 

survey was ―to learn more about crimes and the victims of crime [and] to measure crimes not reported to police as 

well as those that are reported‖

3

. Data are collected twice a year from a nationally representative sample to obtain 

information about incidents of crime, victimization, and trends involving victims 12 years of age and older and their 

households. The survey has long been considered a leader in making methodological advances (e.g., Scheuren, 

2000). The survey underwent an ―intensive methodological redesign‖ in 1993 to ―improve the questions used to 

uncover crime, update the survey methods, and broaden the scope of the crimes measured‖

4

. 

The UCR and the NCVS differ in that they ―are conducted for different purposes, use different methods, 

and focus on somewhat different aspects of crime‖5. So inevitably there are discrepancies between estimates derived 

from these two different measures of crime. Nonetheless, ―long-term [NCVS and UCR] trends can be brought into 

close concordance‖ by analysts familiar with the programs and data sets

6

. This is not surprising in that the NCVS 

was designed ―to complement the UCR program‖

7

. So while the NCVS and UCR programs each were designed to 

collect different data, each offers data that are criminologically relevant, and together they ―provide a more complete 

assessment of crime in the United States‖

8

. 

The conclusion that both programs are essential to the measurement of crime in the United States 

underscores the importance of the current request by BJS. In this research paper, however, we concentrate mainly on 

the NCVS. 

1

E.g., see: Biderman, A. (1967). Surveys of population samples for estimating crime incidence. Annals of the

American Academy of Political and Social Science, 374, 16-33. Biderman, A. (1981). Sources of data for 

victimology. The Journal of Criminal Law and Criminology, 72, 789-817. 

2

 E.g., see page 31 in Savitz, L. (1967). Dilemmas in Criminology. New York: McGraw Hill.

3

See page 11 in Bureau of Justice Statistics (1988). Report to the Nation on Crime and Justice. (2nd ed.) NCJ105506. Washington, DC: US Department of Justice.

4

E.g., see page 1 in Bureau of Justice Statistics. (2004). The Nation’s Two Crime Measures. NCJ-122705.

Washington, DC: US Department of Justice. 

5

 Ibid 

6

 Ibid, page 2 

7

 Ibid 

8

Lauritsen, J.L., and Schaum, R.J. (2005). Crime and Victimization in the Three Largest Metropolitan Areas, 1980-

98. NCJ 208075. Washington, DC: Bureau of Justice Statistics.

1 
This paper is based on our analytic work performed, with the onsite data access help from the Census 

Bureau, for the Bureau of Justice Statistics of the U.S. Department of Justice. This topic is one of four priority areas 

for methodological research on potential improvements to the NCVS selected by the Bureau of Justice Statistics 

(BJS). The four priority areas are based on a set of recommendations resulting from a review of the NCVS by the 

Committee on National Statistics and the Committee on Law and Justice of the National Research Council of the 

National Academies. 

In this study, we initiate and use a variety of strategies that follow OMB guidelines for measuring the

nonresponse bias. Although NCVS is still noted as having achieved a household response rate over 90 percent, 

response rates for most household surveys in the U.S. are declining – a cause of concern for the NCVS. Major 

consequences of increasing nonresponse rates include higher survey costs and potential biases in survey estimates. 

We are mindful that our study in this area is designed to permit integration with the others to support the broad goal 

and requirements of the NCVS redesign in particular and the contemporary challenges of survey research more 

generally.

2. A Capture/Recapture Analysis 

2.1 Introduction 

We examine a capture/recapture approach to estimating the fraction of the nonresponse that is potentially 

nonignorable. In each wave of the NCVS after the first, interviewers attempt to interview both prior nonrespondents 

and previously interviewed cases. Given this interview approach, we are then able to fit the following model. 

 Construct for each NCVS subgroup of interest 2x2 tables, with cell entries given by the values a, b, c, and d

– where the a cases had been interviewed twice, the entries b and c once each, and the entry d is for those 

not interviewed at all. 

Under the assumptions of the capture/recapture model -- assumptions equivalent to ignorability — we can 

estimate the capturable or ignorable portion of the d cell, denoted dI,as dI= bc/a. The remainder (d - dI) is 

then potentially nonignorable.9 

In NCVS wave 1? 

In NCVS wave 2? 

yes no

yes a b 

no c d 

This method, under a model, separates the occasional nonresponder from the chronic nonresponder, thereby 

making it possible to estimate the portion of nonresponse that is potentially nonignorable.10 The name 

―capture/recapture‖ comes from the famous and often used dual systems approach to estimating undercoverage in 

censuses. The application of the old dual systems idea was first described in 2001 but can be expanded to cover a 

survey, like the NCVS, that has 7 waves.11 Now, of course, there may be dependency across waves that would need 

 

9

 The nonrespondents can be further subdivided into refusals and noncontacts, but the simpler model is 

presented here to explain the concept. 

10 Only the Wave 1/Wave 2 example has been used. This method can be employed with each pair of adjacent 

waves and has been exemplified in Table A1. 

11 Scheuren, F. 2001. ―Macro and Micro Paradata for Survey Assessment,‖ in 1999 NSAF Collection of 

Papers, by Tamara Black et al. and J. Michael Brick et al., 2C-1 – 2C-15 Washington, D.C.: Urban Institute, 

http://anfdata.urban.org/nsaf/methodology_rpts/1999_Methodology_7.pdf. See also 

http://www.unece.org/stats/documents/2000/11/metis/crp.10.e.pdf (both accessed on October 2, 2009). 

Assessing the New Federalism Methodology Report No. 7. 

2 
to be modeled before the results were used. We do not believe, based on earlier applications12 that this will be an 

insurmountable barrier, if handled properly. 

What we are doing is treating those households13 that respond on some occasion(s) but not others as 

missing at random (MCAR or MAR), while the ―never responders‖ are more likely to be nonignorable (NMAR). 

The base and follow-up interviews for NCVS can, thus, be used under this model to estimate the portion of 

nonresponse that is potentially nonignorable. 14 Typically, in longitudinal surveys, and the NCVS would seem to be 

no different, attrition or chronic nonresponse becomes more and more common in later waves. In some longitudinal 

surveys, once a refusal occurred in an earlier wave, no further attempts were made in later waves. This is not the 

case with the NCVS, and we have used that fact in a manner similar to that used in Vaughan and Scheuren.15 

2.2 Types of Nonresponse 

Operationally, two major components of survey nonresponse are conventionally considered – nonresponse 

due to noncontact and nonresponse due to refusal. The literature demonstrates that both noncontact rates and refusal 

rates have been on the rise in the recent decade and that, in face-to-face surveys, refusals can now be a larger 

component of nonresponse than noncontacts.16 ―Uncorrectable‖ nonresponse bias may arise mainly from noncontact 

nonresponse, since typically in such settings-- like the first wave of the NCVS--we have very little to go on in 

adjusting for the nonresponse.17 Refusal nonresponse, on the other hand, often rises after a first contact, when some 

information is known about the respondents. What we know about the nonrespondents allows us to usefully 

distinguish among three models, first proposed by Rubin:18

Ignorable nonresponse: If the probability that a household or a within-household individual selected for the 

NCVS sample does not depend on the vector of information known about the sampling unit (such as 

geographic region, household income, race, gender, age, etc.), the response of interest (such as variables 

about victimization status), or the survey design, then the nonresponses are ignorable and can be treated as 

―missing completely at random‖ (MCAR). These nonresponses would be essentially selected at random 

from the sample and, therefore, can be ignored as a source of bias. They do, however, increase costs and

raise concerns about the credibility of survey estimates.19 

Conditional ignorable nonresponse: If the probability that a household or a within-household individual 

selected for the NCVS sample depends on the vector of information known about the sampling unit but not 

on the response of interest, the nonresponse can be treated as missing at random (MAR), given covariates. 

 

12 Scheuren, F. 2007. ―Paradata Inference Applications,‖ presented at the 56th Session of the International 

Statistical Institute, Lisbon, August 22-29. 

13 We do not know enough about the use of this model for the sampling of individuals within households, so we 

have not offered it for use here. A future study of this would be recommended, if enough resources were available. 

14 The fact that a household never responds does not mean that it is biasing and nonignorable. It could have 

characteristics very similar to those of respondents; hence we have characterized this group as only potentially 

nonignorable. Still, it is better that we use this unit nonresponse rate than a rate which treats all of the 

nonrespondents as potentially nonignorable. 

15 Vaughan, D. and Scheuren, F. 2002. ―Longitudinal Attrition in SIPP and SPD,‖ Proceedings of the Survey 

Research Methods Section, American Statistical Association (2002): 3559-3564.

16 See Atrostic, B. K. et al. 2001. ―Nonresponse in U.S. Government Household Surveys: Consistent 

Measures, Recent Trends, and New Insights,‖ Journal of Official Statistics 17: 209-226.

17 Also, in some surveys like the CPS, a household that was not at home may be an indicator that the household 

members could be working. Temporary absent nonresponders in the CPS might, on the other hand, be on vacation. 

18 Rubin, D. 1978. ―Multiple Imputations in Sample Surveys: A Phenomenological Bayesian Approach to 

Nonresponse,‖ Proceedings of the Survey Research Methods Section, American Statistical Association (1978): 20-

28. See also D. Rubin, ―Inference and Missing Data,‖ Biometrika 63, no. 3 (1976): 581-592.

19 It is important to note that so far we have been talking about the bias of a single univariate variable. We will 

continue to do so but caution that, as mentioned in Scheuren, F. 2005. ―Seven Model Motivated Rules of Thumb or 

Equations,” http://www.niss.org/sites/default/files/Scheuren.pdf (accessed on September 30, 2009, most of the time 

all forms of nonresponse are present, sometimes for different variables, sometimes for different time periods. 

3 
The nonresponse can be conditionally ignorable since we may use models to explain the nonresponse 

mechanism, and the nonresponse can be ignorable after the model accounts for it.20

Nonignorable nonresponse: If the probability of nonresponse depends on the value of a response variable 

such as victimization status and cannot be completely explained by the value of the vector of information 

known about the sampling units (household or individuals within a household), then the nonresponse is 

nonignorable or not missing at random (NMAR). Theoretically, by using additional covariates, perhaps 

from an augmented frame or from an earlier wave of the same survey, models can help in this situation.

Make no mistake about the NMAR case, though; it can seldom be dealt with satisfactorily for the entire 

vector of survey variables. There are many cases, however, where, relative to sampling error, the mean 

square error (MSE) increase over the sampling variance (VAR) is small, i.e., {MSE/VAR}1/2 lies within a 

narrow range not much larger than if there had been no nonresponse,21 and hence confidence intervals are 

not unduly lengthened. 

In the present paper we distinguish between the concerns about bias that a raw response rate might 

engender and measuring the bias arising from nonresponse after adjusting for it, using whatever is known about the 

selected units.22 Different survey approaches may lead to a higher response rate for a similar cost. As pointed out in 

Scheuren (2005), unit nonrespondents, m, can be divided up into three parts (MCAR, MAR, and NMAR), all usually 

present in any given survey; that is --

 m = mMCAR + mMAR + mNMAR. 

 

For our work with the NCVS, it is important to learn the size of m overall, and, conditional on that value, 

how to minimize mNMAR. 

Our efforts carried out so far have been confined to studies of unit nonresponse. Based on our prior work

23

we have working hypotheses on the relative sizes of the quantities mMCAR, mMAR, and especially mNMAR. Of course, 

we do not expect to test all of our working hypotheses but shall state them for the record in any case. 

2.3 NCVS Longitudinal Data and Interview Status across Waves 

Each month the U.S. Census Bureau selects respondents for the NCVS using a ―rotating panel‖ sample 

design. Households are randomly selected and all age-eligible individuals become part of the panel. Once in the 

sample, respondents are interviewed every six months for a total of seven interviews over a three-year period.24 For 

 

20 Obviously the more we know about the unit selected for study, perhaps from a strong frame or previous 

successful contacts, the more likely this form of nonresponse may be successfully modeled. 

21 This point is developed further in Scheuren, F, 2005. ―Seven Model Motivated Rules of Thumb or Equations.” 

http://www.niss.org/sites/default/files/Scheuren.pdf (accessed on September 30, 2009), in which the following 

related works are cited: W. G. Cochran, ―Sampling Techniques”, 3

rd ed. (New York: John Wiley, 1977); and M. H. 

Hansen, W. N. Hurwitz, and W. G. Madow, ―Sample Survey Methods and Theory”, 2 vols. (New York: Wiley, 

1953).

22 In our treatment here we have largely focused on unit nonresponse concerns, as distinct from item 

nonresponse. In a complex survey like the NCVS, the line between these two forms of missingness gets blurry. 

There is a gray area where methods like multiple imputation (Rubin, D. 1978. ―Multiple Imputations in Sample 

Surveys: A Phenomenological Bayesian Approach to Nonresponse,‖ Proceedings of the Survey Research Methods 

Section, American Statistical Association (1978): 20-28) that grew up mainly to handle item nonresponse can be 

used to handle unit nonresponse just as well or do even better than weighting approaches. For a discussion of this, 

see the exchange between Little (Little, R. J. A. 1988. ―Missing-Data Adjustment in Large Surveys,‖ Journal of 

Business & Economic Statistics 6, no. 3 (1988): 287-296) and Scheuren. Scheuren, F. 1988. ―Missing-Data 

Adjustments in Large Surveys: Comment,‖ Journal of Business & Economic Statistics 6, no. 3 (1988): 298-299.

23 Scheuren, F. 2007. ―Paradata Inference Applications‖ (presentation, International Statistical Institute, 56th

Session, Lisbon, August 22-29). 

24National Crime Victimization Survey, 2007 [Record-Type Files]: Codebook (Ann Arbor, MI: Inter-university 

Consortium for Political and Social Research, 2009), http://www.icpsr.umich.edu/cgibin/bob/archive2?study=25141&path=NACJD&docsonly=yes (accessed on October 5, 2009). 

4 
example, we constructed a longitudinal file for the households that came into the NCVS sample as the new incoming 

units to be interviewed for the first time in 2003. Two cohorts of NCVS households were setup, with first cohort 

containing households starting to be approached for interviews for the first time within the first six months of 2003, 

and the second cohort containing households starting to be interviewed for the first time within the second six 

months of 2003. Each of the households in these two cohorts can stay in the sample to be interviewed seven times 

for seven waves, till the first half of 2006 and the second half of 2006 respectively. 

Noninterviews may occur at any of the waves for any of the households approached for interviews. A 

sample unit for which an interview could not be obtained is classified as one of three non-interview types, namely, 

Type A, Type B, and Type C noninterviews25. 

Tables 1 and 2 summarize the statuses of the households in the two cohorts across the seven waves starting 

from 2003. Take table 1 for example, among the 9,363 ―incoming‖ households in the first cohort of 2003; there were 

6,898 interviewed in the first wave, 1,372 were Type B non-interviews, 416 were Type C non-interviews, and the 

rest were Type A non-interviews (336 refusals, 236 with no one at home, and 105 for other Type A reasons). In each 

of the subsequent waves, some households were not linked for reasons such as their moving out of the sample. 

These, so called ―not matched‖ cases were excluded from this analysis and excluded in the paired 2x2 

capture/recapture analysis. 

Table 1: Summary of Interview Status of Households Starting in the First Six-Months of 2003 

Wave 

Not 

Matched Interviewed 

Type A 

Refused No One Home Other Type B Type C Total 

1 . 6,898 336 236 105 1,372 416 9,363 

2 641 6,806 330 205 104 1,230 47 9,363 

3 667 6,789 363 181 91 1,245 27 9,363 

4 703 6,783 383 164 87 1,224 19 9,363 

5 1,276 6,226 423 169 87 1,169 13 9,363 

6 1,662 5,903 385 155 65 1,185 8 9,363 

7 4,266 4,043 250 117 37 643 7 9,363 

Note: The period from Wave 1 to Wave 7 spans from 2003Q1Q2 to 2006 Q1Q2.

Source: NCVS 2003-2006

Table 2: Summary of Interview Status of Households Starting in the Second Six-Months of 2003 

Wave 

Not 

Matched Interviewed 

Type A 

Refused No One Home Other Type B Type C Total 

1 . 6,924 339 275 108 1,383 468 9,497 

2 740 6,881 306 183 92 1,250 45 9,497 

3 803 6,748 352 192 86 1,287 29 9,497 

4 1,306 6,307 370 216 73 1,201 24 9,497 

5 1,694 5,964 359 174 84 1,199 23 9,497 

6 4,485 3,861 290 121 35 692 13 9,497 

7 4,446 3,964 232 82 55 698 20 9,497 

Note: The period from Wave 1 to Wave 7 spans from 2003Q3Q4 to 2006 Q3Q4. Source: NCVS 2003-2006

 

25 Type A non-interviews consist of households occupied by persons eligible for interviews but from whom non 

interviews were obtained because, for example, no one was found at home in spite of repeated visits, the household 

refused to give any information, the unit cannot be reached due to Type B non-interviews are for units which are 

unoccupied or which are occupied solely by persons who have a usual residence elsewhere (URE). Type C cases are 

ineligible addresses arising because of impassable roads, serious illness or death in the family, or the interviewer is 

unable to locate the sample unit. Because Type A non-interviews are considered avoidable, every effort is made to 

convert them to interviews. The ―every effort‖ is extremely conservative and expensive strategy, especially given 

that much of the missingness may be ignorable. 

5 
2.4 Fraction of Nonresponse That Is Ignorable 

A key promising feature of the capture-recapture method for NCVS nonresponse analysis is its capacity to 

estimate the fraction of nonresponse that is ignorable and how the fractions of ignorable nonresponse can vary for 

various subgroups. To test the fraction of nonresponse that is ignorable, we examined the interview statuses for the 

whole range of the pairs of 2x2 waves, with the current wave tabulated by each of all the subsequent waves. 

Table 3 and Table 4 show the capture-recapture analysis results on the interview status across waves among 

cohort 1 and cohort 2 households respectively. The last columns under [u/(b+c+d)]*100 calculate the fractions of 

nonresponses that may not be ignorable. For any of the 2x2 pair of the waves, the fraction of nonresponse that is not 

ignorable falls into the range between about 10% to slightly less than 40%. That is, the majority of the nonresponses 

can be treated as ignorable. The results also reveal that the farther apart the two waves were the proportion of 

nonignorable nonresponses would be smaller. 

The capture/recapture approach separates nonresponse cases into two forms of missingness -- ignorable and 

potentially nonignorable. This is, of course, under an independence model. The ignorable portion, by definition, is 

not biasing but does increase the sampling error because the number of respondents is reduced. It also raises the 

average cost per usable respondent too. The balance of the missingness is only potentially nonignorable. The 

balance, too, could be ignorable, if a more refined model were used. The interpretation of the capture/recapture 

results is based on the notion that some nonresponse is chronic, coming from units that never respond and some 

nonresponse is or behaves as if it were ―random,‖ coming from units that would respond or even do respond another 

time. In our treatment here we are using the model results as a lower bound on the ignorable nonresponse. 

2.5 Ignorable Nonresponses and Returning Interviews by Subgroups 

As an extension of the capture/recapture method, we divide respondents at one wave between those who 

continued to remain respondents and those who later became nonrespondents. The panel data of NCVS have 

considerable information about nonrespondents who participated in some earlier wave. There are data available on 

demographic and victimization characteristics; therefore, it is possible to discern differences between these 

individuals and those who continued to respond. In addition, study of later wave nonrespondents helps not only to 

develop nonresponse weighting adjustments26 but also to gain an understanding of the causes of panel attrition27

Tables B1 presents the capture recapture analysis on all household respondents (detail tables are available also by 

gender, race, and age upon request). For each group, the summarized percentage of nonresponse that is ignorable is 

calculated. The extent of the returning interviews was also assessed. 

A summary of the fraction of nonresponse that is ignorable is in Table 5. Overall, more than 80 percent of 

the nonresponses in NCVS can be regarded as 'ignorable." Proportionately, more nonresponses by male, black, and 

young (age 25 or less) eligible interviewees are ignorable. The largest of variation occur for the race/ethnicity, with 

eligible black interviewees having proportionately more ignorable nonresponses (84.81% vs. 80.43%). 

2.6 Discussion 

 Survey practice regarding nonresponse, including in the NCVS, continues to use methods that grew up in

an era of low unit and item nonresponse (the 1940/50s). These methods need now to be augmented. Organizations, 

like the US Census Bureau, that pioneered these earlier approaches, notably the application of implicit quasirandomization methods28 have stayed with them too long. Costs of attempting to patch these older approaches (e.g., 

as by refusal conversion) have continued to grow and with no satisfactory way of measurably assessing whether or 

not they remain effective. 

 

26 Oh, L. and Scheuren, F. 1983. ―Weighting Adjustment for Unit Nonresponse,‖ in Incomplete Data in Sample 

Surveys: Vol. 2, Theory and Bibliographies, eds. W. G. Madow, I. Olkin, and D. B. Rubin (New York: Academic 

Press. 1983). 

27 Kalton, G. et al.. 1992. ―Characteristics of Second Wave Nonrespondents in a Panel Survey,‖ Proceedings of 

the Survey Research Methods Section, American Statistical Association: 462-467.

28 Oh, H.L. and Scheuren, F. 1983. ―Weighting Adjustment for Unit Nonresponse,‖ in Incomplete Data in 

Sample Surveys: Vol. 2, Theory and Bibliographies, eds. W. G. Madow, I. Olkin, and D. B. Rubin (New York: 

Academic Press). 

6 
Table 3 Capture-Recapture Analyses of Household Cohort 1

1

-- Interviews Across Waves 

Wave by Wave A b c D di=bc/a u=(d-di) [u /d]*100 2[u /(a+b+c+d)]*100 [u/(b+c+d)]*100 

ic12 6,156 304 294 275 14.52 260.48 94.7% 3.706% 29.838% 

ic13 6,013 355 344 218 20.31 197.69 90.7% 2.853% 21.558% 

ic14 5,912 404 384 178 26.24 151.76 85.3% 2.206% 15.710% 

ic15 5,419 452 350 163 29.19 133.81 82.1% 2.096% 13.866% 

ic16 5,086 411 354 143 28.61 114.39 80.0% 1.908% 12.598% 

ic17 3,473 288 264 80 21.89 58.11 72.6% 1.416% 9.194% 

ic23 6,134 289 291 279 13.71 265.29 95.1% 3.794% 30.884% 

ic24 6,008 358 321 236 19.13 216.87 91.9% 3.133% 23.702% 

ic25 5,457 420 304 201 23.40 177.60 88.4% 2.783% 19.200% 

ic26 5,111 390 332 169 25.33 143.67 85.0% 2.394% 16.124% 

ic27 3,491 256 242 109 17.75 91.25 83.7% 2.227% 15.034% 

ic34 6,142 288 268 291 12.57 278.43 95.7% 3.984% 32.873% 

ic35 5,559 382 272 237 18.69 218.31 92.1% 3.385% 24.502% 

ic36 5,183 354 294 192 20.08 171.92 89.5% 2.854% 20.467% 

ic37 3,548 238 216 130 14.49 115.51 88.9% 2.796% 19.779% 

ic45 5,685 329 214 298 12.38 285.62 95.8% 4.377% 33.961% 

ic46 5,276 313 240 246 14.24 231.76 94.2% 3.815% 29.007% 

ic47 3,589 217 186 150 11.25 138.75 92.5% 3.350% 25.091% 

ic56 5,250 251 275 297 13.15 283.85 95.6% 4.674% 34.490% 

ic57 3,571 187 218 176 11.42 164.58 93.5% 3.964% 28.328% 

ic67 3,646 156 163 206 6.97 199.03 96.6% 4.772% 37.910% 

Note: a. Count of households interviewed in both designated waves 

b. Count of households interviewed in the first designated wave but not in the second designated wave 

c. Count of households interviewed in the second designated wave but in the first designated wave 

d. Count of eligible households not interviewed in both designated waves 

1Based on households in cohort 1 with the first rotation in the sample in the first 6 month in 2003 

2

Percentages in this column denote the percentages of potentially nonignorable missing households 

Source: NCVS 2003-2006 Longitudinal File 
Table 4: Capture-Recapture Analyses of Household Cohort 21

-- Interviews Across Waves 

Wave by Wave A b c D di=bc/d u=(d-di) [u/d]*100 2[u /(a+b+c+d)]*100 [u/(b+c+d)]*100 

ic12 6,214 251 313 284 12.64 271.36 95.5% 3.842% 32.000% 

ic13 5,937 376 380 207 24.07 182.93 88.4% 2.651% 18.996% 

ic14 5,468 429 378 185 29.66 155.34 84.0% 2.405% 15.660% 

ic15 5,122 422 377 152 31.06 120.94 79.6% 1.991% 12.717% 

ic16 3,298 316 251 95 24.05 70.95 74.7% 1.792% 10.718% 

ic17 3,392 251 265 88 19.61 68.39 77.7% 1.711% 11.323% 

ic23 6,147 307 243 264 12.14 251.86 95.4% 3.618% 30.942% 

ic24 5,644 401 269 203 19.11 183.89 90.6% 2.822% 21.064% 

ic25 5,245 410 291 150 22.75 127.25 84.8% 2.087% 14.953% 

ic26 3,358 309 197 104 18.13 85.87 82.6% 2.164% 14.077% 

ic27 3,442 250 218 85 15.83 69.17 81.4% 1.731% 12.507% 

ic34 5,690 330 262 264 15.20 248.80 94.2% 3.801% 29.066% 

ic35 5,260 344 275 222 17.98 204.02 91.9% 3.344% 24.259% 

ic36 3,370 281 214 133 17.84 115.16 86.6% 2.880% 18.337% 

ic37 3,438 218 230 114 14.58 99.42 87.2% 2.485% 17.690% 

ic45 5,324 290 273 281 14.87 266.13 94.7% 4.315% 31.532% 

ic46 3,372 236 210 174 14.70 159.30 91.6% 3.991% 25.694% 

ic47 3,421 181 236 149 12.49 136.51 91.6% 3.424% 24.119% 

ic56 3,470 200 178 212 10.26 201.74 95.2% 4.969% 34.193% 

ic57 3,487 162 217 164 10.08 153.92 93.9% 3.819% 28.346% 

ic67 3,532 126 207 205 7.38 197.62 96.4% 4.855% 36.732% 

Note: a. Count of households interviewed in both designated waves 

b. Count of households interviewed in the first designated wave but not in the second designated wave 

c. Count of households interviewed in the second designated wave but in the first designated wave 

d. Count of eligible households not interviewed in both designated waves 

1Based on households in cohort 2 with the first rotation in the sample in the second 6 month in 2003 

2

Percentages in this column denote the percentages of potentially nonignorable missing households 

Source: NCVS 2003-2006 Longitudinal File 
Table 5 Ignorable Nonresponses by Subgroups 

Percent of Nonresponses that 

are Ignorable

Total Counts of Ignorable 

Nonresponses

All 81.10 2762

Male 84.04 1327

Female 83.43 1435

Black 84.81 469

Other 80.43 2294

Age 25 or Younger 84.11 323

Age 26 or Older 83.74 2441

 

 We cannot offer a modeling approach to nonresponse, without reminding the reader that we are not 

believers in the notion that a ―best model exists and can be found. ― The results from our capture-recapture analyses 

show that the vast proportion of the nonresponses can be deemed as ignorable when the nonresponse pattern of 

individuals across time is examined along with its covariates. 

The NCVS has many aspects that offer ―handles‖ to pull existing Census practice up to a more cost 

effective and inferentially supportive paradigm. The Census Bureau has really not used the excellent longitudinal 

structure of the NCVS to improve cross-section estimates, which seem to be the main focus currently for BJS. The 

longitudinal approach has been regarded as essential to study the performance of the justice system as a whole and it 

has been recommended that strategies for improving longitudinal structures, including improving the linkage 

capacity of existing data to fielding panel surveys of crime victims.29 We heartily concur, as we found at many 

points in our analyses where some research objectives had to be accomplished only indirectly, if at all. 

The capture-recapture method proposed for NCVS has implications for the survey sponsor in that it can test 

whether there is evidence for a potentially serious nonresponse bias arising from the unobserved fraction of the 

refusals. It also has implications for the expensive refusal conversion process and the extent to which that process 

should be pursued based on its seemingly small bias reduction potential. Finally, the raw weighted nonresponse rate 

measure in NCVS could be recalibrated to reflect only the potentially nonignorable portion of the nonresponse. Like 

most surveys, the raw NCVS nonresponse rate continues to be used as a quality and credibility measure when, in 

fact, matters are far more nuanced. This one simple change could allow BJS to focus resources elsewhere, for 

example at the fall-off in reported crime incidences as the survey proceeds, wave by wave.

3. Response Analysis of Early vs. Late Responders and Key Subgroups 

3.1 Introduction 

In this study, the second intended method to examine bias due to nonresponse would use a level-of-effort 

approach by contrasting respondents with different levels of recruitment effort. NORC has applied this approach in 

nonresponse bias analysis30 and has found it effective in estimating the direction and the size of nonresponse bias. 

For the NCVS, we had proposed to compare survey data for 1) respondents who required less than three contact 

attempts/visits vs. respondents who required three or more visits to complete the survey, and 2) respondents who 

answered the survey request readily without refusal conversion effort vs. respondents who required refusal 

conversion effort. Unfortunately, the number of attempts to obtain an interview is not a data field readily available 

for use – nor is the amount of effort required to convert an initial refusal. These data may be available on a raw 

 

29 Groves, R.M. and Cork, D.L. 2009. ―Ensuring the Quality, Credibility, and Relevance of U.S. Justice 

Statistics.” Washington, D.C.: National Academies Press.

30 See Skalland, B. et al. 2006. “A Non-Response Bias Analysis to Inform the Use of Incentives in Multistage 

RDD Telephone Surveys,‖ Proceedings of the Survey Research Methods Section, American Statistical Association: 

3705-3712. 

9 
audit file kept by Census on a sample of the interviews. NORC did ultimately receive a copy of a Raw Audit File, 

but the amount of effort to decipher the variables and their meanings did not fit in with the requirements for this 

study. Thus, as a proxy, we use differences in estimates between respondents who were amenable and did not refuse 

the survey request and those who refused the survey request at least once but were converted in a later wave. 

Several years of data are used to examine stability and trends of the patterns, more details are included in 

the Appendix in our report to the BJS. Overall, the household and person level public use files for 2002-2006, and 

2007, as well as the linked household internally created file for 2002-2006 are used. Due to the longitudinal nature 

of the data collection, previous responses can be used in the same way as frame data to make nonresponse or 

missing data adjustments. 

In this section, results of logistic regression models are presented. We make no claim that the model results 

are any ―best‖ predictors of nonresponse; instead, the purpose of the logistic models is threefold: (1) determining 

pockets or particular interactions of characteristics that correlate with response, (2) investigating the correlation of 

crime victimization estimates and response patterns, (3) comparing response patterns across longitudinal data versus 

annual collection efforts to build on the natural structure of the data. 

3.2 Early vs. Late and Easy vs. Hard Responder Comparisons 

The Census Bureau employs a rotating panel longitudinal sample to use for the NCVS interviews. Each 

selected household is included in the sample seven times over a period of three and a half years. Until 2006, the first 

interview was used as a bounding interview and not released on the public use file. Beginning in 2006, the first 

 ̳unbounded‘ interviews were phased in and included for release. NORC was given access to the internal files, and 

created two household level longitudinal cohort files for years 2002-2006 -- including the first or unbounded 

interview. Employing these data, we look at the frequency of response, by analyzing the distribution of wave 

response by key demographic variables. In particular, our exploratory analysis focuses on the panel survey response 

issue of continued response and dropout issues – that is, that initial respondents do not continue to respond through 

all waves of the survey. There are two issues to address – (1) which initial respondents are most likely to drop out 

and (2) after all data are collected, what is the best way to adjust for the non-response. The exploratory analysis 

focuses on singling out characteristics of drop outs. Using the cohort file NORC created, we looked at initial 

responding households that entered the survey in the second half of 2002 and computed how many waves they 

participated in. 

Table 6: Number and Percent of Responding Households by Number of 

Waves Participation 

Number of Waves 

Response 

Number of 

Responders 

Percent of Initial 

Responders 

7 (all) 3722 53

6 1425 20

5 940 14

4 388 6 

3 207 3 

2 130 2 

1 (only wave 1) 148 2 

Total 6960 100

There is much literature about the differences in response rate by age31. Figure 1 is a stacked line graph 

that shows the percent of respondents in the age group that participated – shown is the number of waves they 

participated in, given that they participated in the first wave. The deep blue color shows the percent of respondents 

that participated in all 7 waves. The red (warning!) color shows those respondents that only participated in one 

wave. The percent adds up to 100 for each age group. It is clear that the younger age groups are less likely to 

 

31 Ibid.

10
continue responding. However even for the youngest age group, nearly 80% of the respondents did participate in at 

least 5 of the survey waves. Similar Charts for Educational Attainment and Reported Income are included in

Appendix of our study report. 

Figure 1: Percent of Responding Households in Age Group 

by Number of Waves Participation, Internal Cohort File 2002-2006 

Figure 2 below, contains the stacked chart for different categories of household structure. Response appears higher 

for households with couples, versus households without couples. 

Figure 2: Percent of Responding Households by Household Structure 

by Number of Waves Participation, Internal Cohort File 2002-2006 

11
For this particular analytic purpose, in order to also investigate at the individual response level, Public Use 

Files (PUF) at the individual level were downloaded from the ICPSR site managed by University of Michigan.32

These person level files were merged together in order to look at person level cohorts beginning in the first half of 

2002. Since the first bounding interview is not included in the Public Use Files, the analysis here focuses on the 

results from Waves 2 through 7 for both the person level and household cohorts. By focusing on the panel/rotation 

group that was initially interviewed in the first half of 2002 (panel/rotation in 13,23,33,43,53,63), we are able to 

include all possible responses from that group for the remaining waves. The patterns are similar for the households 

and individual characteristics we examined. Figure 3 below is a double chart that compares household and person 

level stacked number of waves responded to. Similar charts for Education Attained, Hispanic Origin, and Race are 

available. 

Figure 3: Percent of Responding Households and Individuals in Age Group by Number of Waves 

Participation, for PUF 2002-2006

3.3 Modeling Continued Response and Characteristics of Drop Outs 

The descriptive charts are informative re overall trends, but we also developed logistic regression models to 

explore interactions between the variables. For this exercise, we use the household cohort files, representing the 

cohorts beginning in the second half of 2002. As in the household graphs above, we only use records that responded 

to the first, bounding wave, and include their continued response. For prediction variables, indicators and grouped 

variables were developed for the following variables of interest. Also, interactions for race and Hispanic origin with 

the other variable groups were introduced.33 

 

32 NCVS public use data and documentation are available at http://www.icpsr.umich.edu/NACJD/NCVS/

(accessed on June - September, 2009). 

33 Since not all units responded to the first wave, the value used for the independent variable was taken from the 

earliest wave response available. 

12
Table 7: Variable Groups Input to Logistic Model of Response/Drop 

Out 

Gender Rural/Urban 

Race: Black or Asian Region 

Hispanic Origin Homeowner 

Age MSA Status 

Marital Status Family Structure 

Education Number of Crime Incidents 

Household Income 

Two models were developed looking at the extremes of response, first we modeled continued response, or

those households that responded to at least 6 waves. Correspondingly, we also developed a model to explore drop 

outs – that is, those that only responded to 3 waves or less. The logistic models were run with a stepwise procedure 

with the cut-off SLS=0.02. The model variables and their ranking are shown in the table below, the direction is also 

indicated. The specific logistic results are included in our report to the BJS. The concordance for both models was 

around 65%. Homeowner showed as the most important variable in both models. The interaction of 

―Race=Black‖, with at least 1 crime incidence reported was significant for both models. This is something that 

should be investigated further. Income and Age came in with the expected direction of correlation. That is, age and 

income are both positively correlated with response. There was a good amount of overlap for the variables that 

showed up significant for the two models. 

Table 8: Model Variables Shown in Order of Importance for the Logistic Continued Response/Drop Out 

Models 

Drop Out 

(3 or Less Wave Responses) 

Continued Response 

(6 or More Wave Responses) 

Homeowner -1 +1

Married -2 +3

black*Incidence Reported +3 -9 

Rural +4

Age Bounded (20,50) -5 +2

Asian*Married +6

Rank of Household Income -7 +8

South +8

Family Structure = Male w/others -7 

Hispanic Origin +4

Midwest +5

Post College +6

4. Differential Response Rates and Dispositions by Subgroup 

4.1 Introduction 

Inferences about differences in response rates rely on the assumption that survey errors are comparable 

across groups.34 Studies indicate that nonresponse is not randomly distributed across the population, but tends to be 

 

34 If this assumption fails and sampling or nonsampling errors (of coverage, non-response and measurement) differ, 

then any differences detected between groups may be artifacts of the data (e.g., Blom, 2008). Blom, A. (2008). 

Decomposing the Processes Leading to Differential Nonresponse and Nonresponse Bias. Presented at the 63rd

Annual Conference of the American Association for Public Opinion Research, New Orleans, LA, May 15. 

13
higher among those at both ends of the income distribution--among the elderly, for men, and for those with limited 

English proficiency35. There exist geographic variations in deviant behavioral measures36, 37, 38and measurement 

errors39. 

The lifestyle-routine activity theory posits that certain demographic characteristics increase the risk of 

victimization, because role expectations are related to a lifestyle that places suitable targets in proximity to 

motivated offenders without appropriate societal constraints40. Housing units in the central city of SMSAs, for 

example, have higher risk of burglary than units elsewhere. Units in multi-unit dwellings are at greater risk than 

single family units. Changes in household structure (household size) are significantly related to risk of burglary41, as 

larger households more often tend to have someone home. 

 Also pertinent to nonresponse analysis is the conjectured relationship between a tendency to survey 

nonresponse with either offender recidivism42 or an individual‘s victimization for some crimes.43 Such relationships 

may cause bias in survey estimation.44 

The third method compares response rates and disposition codes (outcomes) of key subgroups of the target 

population for domains where response rates are available. The respondent distribution of geography, age, sex, 

race/ethnicity, education, employment status, household income, and household size among reference households 

and household members can be compared to the population distribution. If the population proportion of, say, 

Hispanics, is 10%, whereas the unweighted sample proportion is only 5%, there are reasons to be concerned about 

nonresponse bias in estimates for Hispanics. This analysis uncovers population domains that are at greater risk of 

nonresponse bias, even if it is possible to post-stratify them successfully. 

If response rates are much lower in some strata or subgroups than in others, there exist at least two 

concerns. First, strata or subgroups with lower response rates might require a larger weight to compensate for 

nonresponse, which inflates the variance of the national estimates of interest. Second, even if overall national 

estimates are not biased, there is still the danger that a stratum or subgroup with a much lower response rate suffers 

from nonresponse bias for particular subgroups that might be of analytic interest to some users. 

 

35 Bradburn, N.M. (1992). A response to the nonresponse problem. Public Opinion Quarterly, 56, 3, 391-397.

36 Osborn, D.R., Trickett, A., and Elder, R. (1992). Area characteristics and regional variates as determinants of area 

property crime levels. Journal of Quantitative Criminology, 8, 265-285.

37 Trickett, A., Osborn, D., Seymour, J., and Pease, K. (1992). What is different about high crime areas? British 

Journal of Criminology, 35, 343-359.

38 Wright, D., and Zhang, Z. (1998). Hierarchical Modeling in National Household Survey on Drug Abuse, Pp. 756-

762 in 1998 Proceedings of the Section on Survey Research Method, Alexandria, VA: American Statistical 

Association . 

39 Zhang, Z. and Gerstein, D.R. (2003a). "Geographic and Other Variations in Measuring Drug Use: Implications of 

Research Data for Understanding the Impact of Drug Use on Crime and the Criminal Justice System.‖ Presented at 

the 40th Annual Meeting of Academy of Criminal Justice Sciences, Boston, March 5, 2003. 

40 Meier, R.F., and Miethe, T.D. (1993). Understanding theories of criminal victimization. Pp. 459-499 in Tonry, M. 

(eds), Crime and Justice: A Review of Research, Vol. 17. Chicago: University of Chicago Press.

41 Lynch, J.P., Berbaum, M.L., Planty, M. (1998.) Investigating Repeated Victimization with the NCVS. Final Report 

for National Institute of Justice. 

42 Zhang, Z. and Gerstein, D.R. (2003b). ―A Multi-site Assessment of the Extent and Correlates of Arrest 

Recidivism and Its Impact on Arrestee Drug Abuse Prevalence and Pattern Estimations.‖ Paper presented at the 

163rd Annual Joint Statistical Meetings, San Francisco, California, August 5, 2003.

43 The propensity to report victimization may vary by type of crime. Victims of certain types of crime, e.g., hate 

crimes, rape, etc. may have quite different propensities to respond to victimization surveys than victims of other 

types of crimes (Lauritsen, 2005). In addition, the propensity of being victimized repeatedly may also be related to 

the propensity to respond to victimization surveys (Lehnen and Reiss, 1978). 

44 Population structure may have compositional effect on crime, and crime can also affect demography (South and 

Messner, 2000).

14
Counts of Households, by Sample Dispositions with the Table 9. NCVS 2005 Sample Frame 

Not nonresponse Overview 

Units that field investigation proves do not exist (n=15,509) 

Unfit/demo 410

vacant-regular (type B non-interview) 11,372 

vacant-storage 853

unoccupied site 387

Type B other 255

demolished (type C non-interview) 107

condemned (type C non-interview) 10

unused line list (type C non-interview) 10

Outside segment 1 

permit granted 72

permit abandoned/other 37

under construct 376

convert perm 27

Merged 37

see codebook? 9 

temp occupied 1,294 

convert to temp 252

Nonresponse45 (n=7,911)

language problems 63

house/trailer moved 63

refused 4,659 

temp absent 481

Type A other occupied 338

No one home 2,307 

Interviewed household 77,224 

TOTAL 100,644 

Source: National Crime Victimization Survey, 2005

 

45 The unit nonresponse considered in this table arises because the household at a particular address could 

not be contacted or declined to participate at all. 

15
Furthermore, variation in the mix of disposition codes (corresponding to survey outcomes) among 

subgroups might also indicate the potential for nonresponse bias. Noncontacts and refusals are expected to 

be different types of nonrespondents, with Wave 1 noncontacts being less likely to be ignorable.46 Table 9 

shows the various dispositions for the 2005 NCVS. As shown in Table 9, the reasons for non-interviews are 

complex but can be grouped into two categories – those noninterviews that were not nonresponses and 

those that were nonresponses.47 As in Table 10, the dispositions can be further summarized by geography 

or other variables available in the public use data file. In Table 10, note the higher percentage of refusals in 

the West, and the higher percentage of vacant housing units in the South. 

Table 10. Reason for Noninterview by Region 

% within REGION 

REASON FOR NONINTERVIEW

 

REGION Total 

Northeast Midwest South West 

Language problems .1% .0% .1% .1% .1% 

 No one home 3.6% 1.6% 2.1% 2.1% 2.3% 

 Temp absent .6% .3% .4% .7% .5% 

 Refused 4.9% 4.1% 4.0% 5.9% 4.6% 

 Type A other occupied .5% .3% .3% .3% .3% 

 Temp occupied 1.5% 1.1% 1.3% 1.3% 1.3% 

 Vacant-regular 11.9% 10.4% 13.1% 8.6% 11.3% 

 Vacant-storage .4% .9% 1.0% .8% .8% 

 Unfit/demo .2% .4% .5% .3% .4% 

 Under construct .2% .3% .4% .5% .4% 

 Convert to temp .1% .2% .4% .2% .3% 

 Unoccupied site .2% .4% .5% .4% .4% 

 Permit granted .1% .1% .1% .0% .1% 

 Type B other .2% .1% .5% .1% .3% 

 Demolished .1% .1% .2% .0% .1% 

 House/trlr moved .0% .0% .1% .0% .1% 

 Outside segment .0% .0% 

 Convert perm .0% .0% .0% .0% .0% 

 Merged .1% .0% .0% .0% .0% 

 Condemned .0% .0% .0% .0% .0% 

 See codebook .0% .0% .0% .0% .0% 

 Unused line list .0% .0% .0% .0% .0% 

 Permit abandoned/other .0% .0% .0% .0% .0% 

 Interviewed hhld 75.2% 79.4% 74.8% 78.4% 76.7% 

Total 100.0% 100.0% 100.0% 100.0% 100.0% 

 

46 The reason that the Wave 1 noncontacts are likely to be nonignorably missing is that there will be little or 

no information on which to condition in attempting to adjust out some of the missingness. Put another way, 

some of the mNCAR can be made mMAR if the right covariates are present.

47 The nonresponse rate we calculated here, 7911/(7911+77224) =90.7%, is consistent with the published 

statistics on NCVS 2005 (Catalano, 2006). Catalano, S. (2006) Criminal Victimization, 2005. Bureau of 

Justice Statistics Bulletin, NCJ 214644, Office of Justice Programs, U.S. Department of Justice. 

16
4.2 Differential Response Rates and Dispositions by Subgroups 

Although, the NCVS data collection is based on a longitudinal sample design with the possibility 

of responding to the survey seven times in three and a half years, the NCVS releases estimates and public

use files with an annual focus. To reflect this we too focus on annual response patterns. In particular, we 

investigate the data collected during 2002 and, for a more recent comparison, 2007. Instead of focusing 

only on one cohort, which is basically one-sixth of the total sample, we are able to include much more 

data. For the annual estimates, the selected units have the possibility of responding during January to 

June, and then separately again during July to December. For analysis of patterns of disposition 

outcomes, the entire annual data file is used. We also use the entire file for general patterns of 

geographic48 and race for the Type A refusal nonresponse analysis. For the more detailed socio-crime 

related analysis which includes more detailed data collected for the survey, we investigate the response 

pattern of those responding Jan-June and/or July-December, for this analysis we only include the four 

cohorts that have the opportunity to respond in both periods.49 

We analyze the differential response by beginning at the top examining the disposition patterns of 

sampled households and tunneling through to the detailed analysis of individual respondents. At the top of 

the analyses is the detailing of the disposition codes by the available geographic data – region, MSA/not 

MSA, place size, type of living quarters and land use (rural/urban). The first level of response is at the 

sampled household. As a benchmark, the resulting dispositions are compared for year 2002 and 2007 in 

terms of percent of total sampled units during January through December of the respective year. There is 

about a 4% decrease in overall percentage of interviewed household, almost half of this is due to an 

increase in the percent of vacant sampled units. There were also small 0.5% increases in Type A reasons –

No One at Home, Refusals and Other. Overall the results appear fairly consistent for the two years. The 

detailed data is included as Appendix Table in our report to the BJS. Delving a bit deeper, we looked at 

disposition across geographic characteristics available on all sampled household units: region, land use, 

msa status, place size code, type of living quarters. Disposition code has been collapsed to the main 

categories. The results for urban/rural are shown in Table 11 below. There is a pattern of higher refusals in 

urban areas, and more vacant units in rural areas. 

Table 11: Major Disposition Outcomes for Sampled Units, by Urban/Rural 

 Year 2002 Year 2007 

 Urban Rural Urban Rural Type A 

No one home 2.06% 0.90% 2.33% 1.42% 

Refused 4.11% 2.86% 5.01% 3.37% 

Other Type A 1.05% 0.63% 1.39% 0.84% 

Type B 

Vacant-regular 

8.60% 14.60% 10.52% 15.95% 

Other Type B 2.84% 6.73% 3.75% 6.62% 

Type C 

Demolished, converted to business 0.27% 0.58% 0.62% 1.11% 

 Interviewed Household 81.07% 73.70% 76.38% 70.68% 

 

48 Region, MSA status, size of area, living quarters.

 

49 That is, we omit the cohort that is finishing up in the Jan-June time frame, and the cohort that has its first 

 interview in the July-Dec timeframe. 

17
Dropping out the Type B and Type C units, we focus on responders and Type A non responders. 

We are able to look at non response reason & responder results by these same geographic variables with the 

addition of race (black/non-black). The overall results are shown in Table 12. Overall, blacks appear less 

responsive, with more ―No One Home‖ and ―Refusals‖. 

Table 12: Response Outcomes for Black and Non Black for Year 2002 and 2007 

 Year 2002 Year 2007 

 Non Black Black Non Black Black 

Duplicate or Language problems 0% 0% 0.08% 0.08% 

No one home 1.9% 3.2% 2.4% 3.5% 

Temporarily absent 0.6% 0.6% 0.4% 0.3% 

Refused 4.4% 5.0% 5.5% 5.9% 

Other occupied 0.5% 0.5% 1.1% 1.3% 

Respond 92.6% 90.7% 90.5% 88.8% 

Total 100% 100% 100% 100%

The response rates are shown separately for Region/black/nonblack in Table 13. Note there is a 

lower response rate for blacks in the North East and West for the year 2002, whereas the black response 

rate decreases for the Midwest region for 2007. 

Table 13: Response Outcomes for Black/Non Black, by Region 

Response Rate 

2002 2007

North East Black 85% 85%

Non-black 90% 87%

Midwest Black 92% 86%

Non-black 94% 93%

South Black 93% 92%

Non-black 94% 92%

West Black 85% 83%

Non-black 91% 89%

The lower response rate for the blacks in the Northeast and Midwest appears to be mainly due to 

low response in urban areas for those regions, as shown in Figure 4 below where response rate is graphed 

against percent of sample. Each point represents a group identified by Region, Urban/Rural, and 

Black/nonblack. The two points in the lower left corner show the much lower response rate obtained for 

Black respondents in the Northeast and West urban areas. 

18
Figure 4: Response Rate by Percent of Sample, 2002 

4.3 More on Responder Differences 

We now turn to look at the differences in responders, where we have more detailed data as well as 

survey outcomes that allow a more intense view of the impacts of differential nonresponse. The question at 

this point becomes, what differential not missing at random non response remains that can be accounted for

with models or other factors based on prior waves response. 

The Public Use Files are structured to allow analysts to compute annual estimates, either in a 

collection year, or as the data year. We are working with the two waves that are put together to compute

estimates for a collection year. Sampled units have an option of responding to either the first or second, or 

preferably, to both waves in a given year. To get a feeling for the patterns, we first examine patterns of 

responding households for the data collection year. Response pattern per wave 1 and wave 2 by income is 

shown below in Figure 5, the corresponding graph by Education is included in the report. 

Figure 5: Percent Responding Households by Income, 2002 

19
One method to examine the response impact, is to compute the restricted estimates by the response 

pattern (Jan-June only, both Jan-June & July-Dec, and July – Dec only) results, shown in Table 14 below, 

are based only on those households with the possibility of responding in both Jan-June and July-Dec 2002. 

That is, like the above graphs, the panels that were being rotated out or rotated in are not included.50 There 

is not a noticeable difference in the restricted estimates for the different groups of responders. 

Table 14: Restricted Results for Annual 2002 Estimates: Proportion of 

Households Reporting Crime Incident 

Nonresponse Respond 

July-Dec July-Dec

Nonresponse % population 2%

Jan-June 

Crime Incident 0.0867 

Response 3% 94%

Jan-June 

0.0920 0.0842 

Using the more detailed data on the responders, we develop logistic regression models to predict 

nonresponse. In this situation, we separate the annual file into responders (responded in both time periods) 

and nonresponders (did not respond in one time period). We develop models for both 2002 and 2007. The

results are similar as those where we used all of the wave responses to predict drop outs, or loyal 

responders. The concordance for the 2002 model is 62.7, for the 2007 model it is slightly higher at 68.7. 

One must note that there are 8% nonresponders in the 2002 data, and 14.5% for 2007. This difference is 

because the first (the unbounded) interview is included for analysis on the later public use file.51 The 

logistic regression results are shown in Appendix Tables in our report to the BJS52. 

Stepping back from the detailed file, we consider broader patterns of nonresponse, including the 

Type A refusals, and their relationship to victimization estimates. The pattern in Figure 6 suggests 

something we already saw in our modeling work in Section 2; that it is plausible to believe that much of the 

nonresponse is not biasing. In Section 2 we assessed this from a process perspective. Here we are looking 

at refusal rates by crime rates and see little pattern. Again we caution against overpromising relative to low 

bias for the NCVS but consider the outcome encouraging. One last point: The nonresponse rate from the 

first round is not included for the 2002 results, but in the later public use files (e.g., for 2007) the crime rate 

estimates shown are cumulative of all rounds.53 Similar plots are included in the Appendix in our full 

report, along with the table data. 

 

50 The population percentages, and the proportion of crime reported are weighted estimates, using the 

collection year weight available on the public use file. 

51 Beginning in 2006, the first bounding interviews are included on the Public Use Files. 

52 Another possible method for addressing nonresponse is to impute missing units using their prior 

survey data. Such an analysis was performed, the results are available upon request. 

53 Beginning in 2006, the first bounding interviews are included on the Public Use Files. 

20
Figure 6: Refusal Rate vs. Crime Rate in Groups Defined by Region, 

Place Size & Race (black/non-black) [only groups with at least 

50 individuals included in graph], Year 2002 

5. An Analysis of the NCVS and UCR Crime Statistics at the County-Level, 2003-2006 

5.1 Introduction 

The U.S. Department of Justice administers two statistical programs to measure the magnitude, 

nature, and impact of crime in the Nation: the Uniform Crime Reporting (UCR) Program and the National 

Crime Victimization Survey (NCVS). The UCR and the NCVS differ in that they ―are conducted for 

different purposes, use different methods, and focus on somewhat different aspects of crime.‖

54 So 

inevitably there are discrepancies between estimates derived from these two different measures of crime. 

Nonetheless, ―long-term [NCVS and UCR] trends can be brought into close concordance‖ by analysts 

familiar with the programs and data sets55 that the NCVS was designed ―to complement the UCR 

program.‖

56 So while the NCVS and UCR programs each were designed to collect different data, each 

offers data that are criminologically relevant, and together they ―provide a more complete assessment of

crime in the United States‖ 57 than either could produce alone.58 

The conclusion that both surveys are essential to the measurement of crime in the United States 

underscores the importance of the current request by BJS for proposals to conduct methodological research 

to support a present-day redesign of the NCVS.59 More broadly, these are challenging times for survey 

 

54 BJS 2004:1. Bureau of Justice Statistics. 2007. National Crime Victimization Survey: MSA Data, 

1979-2004 [Computer file]. Ann Arbor, MI: Inter-University Consortium for Political and Social 

Research. 

55 BJS 2004-2

56 ibid.

57 Lauritsen, J.L. and Schaum, R.J. 2005. ―Crime and Victimization in the Three Largest Metropolitan 

Areas, 1980-98.” Washington, D.C.: Bureau of Justice Statistics, 

http://www.ojp.usdoj.gov/bjs/pub/pdf/cv3lma98.pdf (accessed September 30, 2009). 

58

 Rand, M. R. 2009. ―Criminal Victimization, 2008.‖ Washington, D.C.: U.S. Bureau of Justice 

Statistics, http://www.ojp.usdoj.gov/bjs/pub/pdf/cv08.pdf (accessed on October 4, 2009). 

59 Federal Bureau of Investigation. 2008. ―The Nation’s Two Crime Measures. Uniform Crime Report, 

Crime in the United States, 2007.” Washington, D.C.: U.S., 

http://www.fbi.gov/ucr/cius2007/documents/crime_measures.pdf (accessed on October 4, 2009).

21
research generally given dramatic and fast-paced technological, social, and cultural change. It is also 

challenging how the UCR data may facilitate in improving the NCVS estimation counts at the local level.60

In order to better understand and utilize the relationship between the NCVS and UCR at the subnational level, we examined the NCVS crime victimization estimates and the UCR arrest. Specifically, we 

attempted to estimate the victimization totals at the county level and compare all the NCVS county 

estimates with the count records from the UCR. For illustration, we focused on the 2003-2006 period, used 

four-year pooled NCVS and UCR, and examined summated measures of victimizations and crimes so that 

the NCVS and UCR measures can be better comparable. 

The National Crime Victimization Survey (NCVS) Series, previously called the National Crime 

Surveys (NCS), has been collecting data on personal and household victimization through an ongoing 

survey of a nationally-representative sample of residential addresses since 1973. During the 2003-2006, 

household residents from all the 50 states plus the Districtrict of Columbia participated in the surveys. Not 

all counties participated and there were wide variations in terms of the numbers of the counties that were in 

the NCVS samples in this period. The top five states with the largest number of counties involved in the 

NCVS data collections were Texas (52 Counties), Virginia (47 counties), Ohio (44 counties), Georgia (39 

counties), and New York (37 counties). Only one county within the following states had residents 

participating in NCVS during 2003-2006: Hawaii, New Hampshire, Vermont, and Wyoming. 

5.2 Data Sources 

This analysis examined the differences and the relationships at the county level between the 

National Crime Victimization Surveys and the Uniform Crime Reports (UCR) in the period of 2003-2006. 

New weights were developed for this analyses so that the county-level annual NCVS estimations of the 

totals can be produced. UCR information were retrieved from the annualized county-level UCR data only 

for those counties in the NCVS samples in the same year. 

Because the BJS designed the NCVS to complement the UCR Program, the two programs share 

many similarities. As much as their different collection methods permit, the two measure the same subset

of serious crimes, defined alike. Both programs cover rape, robbery, aggravated assault, burglary, theft, and 

motor vehicle theft. Rape, robbery, theft, and motor vehicle theft are defined virtually identically by both 

the UCR and the NCVS. 

There are significant differences between the two programs: (1) the two programs were created to 

serve different purposes; (2) the two programs measure an overlapping but nonidentical set of crimes; (3) 

The NCVS includes crimes both reported and not reported to law enforcement. The NCVS excludes, but 

the UCR includes, homicide, arson, commercial crimes, and crimes against children under age 12. The 

UCR captures crimes reported to law enforcement but collects only arrest data for simple assault and sexual

assault other than forcible rape. (3) the NCVS and UCR definitions of some crime differ. For example, the 

UCR defines burglary as the unlawful entry or attempted entry of a structure to commit a felony or theft. 

The NCVS, not wanting to ask victims to ascertain offender motives, defines burglary as the entry or 

attempted entry of a residence by a person who had no right to be there. 61 Although rape is defined 

analogously, the UCR Program measures the crime against women only, and the NCVS measures it against 

both sexes. 

 

60 McDowall, D. and C. Loftin, C. 2007. ―What Is Convergence and What Do We Know About It?‖ in 

Understanding Crime Statistics: Revisiting the Divergence of the NCVS and UCR, eds. J. P. Lynch and L. 

A. Addingtion. New York: Cambridge University Press.

61 Federal Bureau of Investigation. 2008. ―Crime in the United States, 2008 .‖ Washington, D.C.: U.S. 

Federal Bureau of Investigation, 2008, http://www.fbi.gov/ucr/cius2008/about/index.html (accessed on 

October 4, 2009). 

22
5.3 Measurement 

The National Crime Victimization Survey covers all of the index offenses covered by the Uniform 

Crime Reports, except for homicide and arson. Therefore, when comparing the total counts of crime 

victimizations and arrests, we exclude murder and arson from the UCR total count measure. 

Due to skewed distributions of the untransformed raw counts and ―outliers‖ found in the 

scatterplots, separate alternative scatterplots were made using the logarithm transformations of the crime 

totals (log(counts +1)). Further scatterplots were shown with some peculiar counties (i.e., counties with no 

crime victimization reported, that is, NCVS county level crime incident count=0, and counties with no 

arrest reported, that is, UCR county level arrest count=0 for the 2003-2006 period) excluded. 

In this analysis, the crimes included in the totals from the NCVS included: Rape, Robbery, 

Assault, Burglary, Motor Vehicle Theft, Purse Snatching, and Theft; and the crimes included in the totals

from the UCR included: Rape, Robbery, Assault, Burglary, Motor Vehicle Theft, and Larceny. 

During 2003-2006, of all the counties where NCVS data were collected, a total of 46 counties 

showed zero number of arrests. All these 46 counties had considerable large amount of crime victimization 

incident reports in the same time period. A total of 56 counties had zero crime victimization incidents 

reported during 2003-2006, although many of them made many arrests for criminal offenses. 

5.4 Results

 Estimations and counts were obtained for each of the four years in 2003-2006. The combined totals at 

the county level were thereafter obtained through the summations of the year-specific totals in NCVS and 

UCR respectively. Only the results for the combined 2003-2006 are shown here. The year-specific scatter 

plots are also available in the NORC work papers and appendices. 

Figure 7 shows the scatter plot of the total victimizations in NCVS by the total crimes in UCR. 

Significant positive relationship was observed. The R2 of the linear regression model was 0.80. 

Figure 7 

As the distribution of the victimization counts at the county level appeared to be skewed, we made 

a logarithmic transformation on the outcomes without dropping any cases. Figure 8 shows the scatter plot. 

23
Figure 8 

Because of the logarithm transformations of the crime totals (log(counts +1)), counties with zero 

count of victimizations could still be shown; actually, the scatter plot in Figure 8 demonstrated that there 

were quite a few zero-type of counties from both NCVS and UCR. Not surprisingly, the R2 as a fit statistics 

of the regression model dropped dramatically due to these outliers. 

5.5 Outliers 

The counties with either victimization counts being zero or crime arrest counts being zero – were 

carefully examined next. Of course, these zero-counties are only an example of the data problems that a 

careful analysis might find 

1. UCR “zero-type “counties. Among all the counties where NCVS data were collected during 

2003-2006, a total of 46 counties were found to have ―zero‖ number of arrests for any of the 

six major index crimes (murder was excluded). As shown in Table 15, 3/5 of these counties 

were located in the State of Florida, and 1/3 of these counties were located in the State of 

Illinois. Minnesota and Virginia each had one ―zero-type‖ county. 

2. NCVS “zero-type” counties. During 2003-2006, there existed 55 counties where NCVS data 

were collected but there were no victimization incidents reported. Virginia had the largest 

number of ―zero-type‖ of counties (n=12), followed by Texas (n=6), Louisiana (n=4). Table 

4.2 list all states which had at least one ―zero-type‖ county. 

Table 15: Distribution of Counties Where UCR Crime Counts During 

2003 – 2006 Were Zero 

State Frequency Percent 

Florida 28 61

Illinois 16 35

Minnesota 1 2 

Virginia 1 2 

ALL 46 100

24
Table 16: Number of Counties Where NCVS Crime Counts During 2003 – 2006 Were Zero, by State 

State Frequency Number of Total 

Counties 

Virginia 12 12

Texas 6 6 

Louisiana 4 4 

Colorado, Georgia, Missouri, Tennessee 3 12

Iowa, Kentucky, Mississippi, Wisconsin 2 8 

Alabama, Illinois, Indiana, Michigan, Minnesota, Nebraska, Nevada, New 

Mexico, New York, North Carolina, Oklahoma, Pennsylvania, Utah 1 14

All 56

Did the UCR ―zero-type‖ counties have larger than 0 amount of victimization incidents reported in 

NCVS? or vice versa? The answer is yes to both. Details, including the counties involved are shown in the 

Appendix Tables in our report. Whereas the inconsistencies found between the UCR and NCVS may need 

further investigations, we excluded these ―zero-type‖ counties from the subsequent analyses. 

5.6 Relationship between the NCVS and UCR 

Figure 9 shows the scatter plot of the total victimizations in NCVS by the total crimes in UCR 

among the counties which had non-zero amount of victimization incidents and criminal offense arrests.

Figure 9: Scatterplot of the Total Crime Counts, NCVS by UCR, for Counties 

Which Had At Least One Victimmization Incident and One Official Arrest, 

at the County Level, 2003-2006 

Figure 9a: Raw Totals Figure 9b: Logarithm of Raw Total

25
Figure 10: Logarithms of Total Counts of Crime Incidents – NCVS by UCR, in 2003 – 2006, 

By Region, excluding counties where total victimization incident count =0 or arrest count =0 

Figure 10 shows the scatter plots of the log transformations of the NCVS victimization incident 

count by UCR arrest count for each of the four regions separately.62 Strong positive significant 

relationships were observed for each of the four regions. 

Table 17: R-squares in the regression analysis of the Arrests Reported by UCR and the Crime 

Victimizations Captured by the NCVS 

Crimes ALL Region 

Northeast Midwest South West 

Logarithm of total crime counts with 

restrictions to Total Victimization and Crime 

> 0 

0.828 0.815 0.827 0.765 0.945

Note: In the regression models depicted by the scatter plots, the square root of R-square is the same as the 

correlation coefficients. Overall, and across each of the census regions, the correlations (r) between the 

NCVS estimates and the UCR estimates are very high. The R2 was 0.828 (r=.9+) at the national level, and 

ranged from 0.765 (r= +.8) to 0.945 (r= +.95) at the regional level.

 

62 Region-specific scatter plots on raw totals, region-specific scatter plots on raw totals with zero-type 

outliers excluded, and region-specific scatter plots on log transformations with raw totals are listed in the 

Appendix of our study report. 

26
Table 17 shows a summary of the R-squares in the linear regression models with the weighted 

estimations of the total counts of the crime victimization incidents reported in NCVS as the dependent 

variable and the total counts of arrests reported by the county-level UCR as the independent variable. 

There are variations across the four census regions in terms of the extent the magnitudes of the 

UCR arrest counts can explain the variability of the crime victimizations reported by householders. 

Regardless whether we transformed the crime and victimization counts or whether we eliminated the 

outliers such as those counties which had no or extremely high level of victimizations, the West Region had 

the highest level of R2. (i.e., R2 = .929 before any transformation and truncation; R2 = .945 after the 

exclusion of outliers and the transformation). 

In the past, the UCR and the NCVS have been used at the national level to assess their correlations 

on specific index crimes63,64. Both high and low correlations have been observed. A high correlation 

between UCR and NCVS trends would suggest that either data series would serve as a reasonable proxy for 

some analytical purposes65. In addition to definitional difference on certain crimes66, there are conjectures 

on what would make the UCR and NCVS differ such as the matters concerning the public‘s willingness to 

report crime to the police and the way police departments record crime, how these factors may vary across 

regions or other geographic units remains an important questions that shall need further investigation which 

is beyond the scope this study. 

7. Discussion 

In the event of possible decline in response rates and increasing nonresponse rates, while we may 

not have the resources to get high response rates across the board, we can allocate the data collection 

resources in a more targeted manner to learn more about the possible bias arising from a low response rate 

or the deviation of the respondent-based statistics from the full sample statistics. This general strategy is of 

special importance for the NCVS given the likely continuing falling response rates with attendant 

increasing field costs to avoid their decline. As Bradburn (1992) indicated and it is still true today, there is 

considerable room in our practice for increasing our understanding on nonresponse without great increases 

in cost. The methods proposed in this study focus more on understanding the nonresponders and using this 

information to adjust the data more intelligently. 

8. Recommendations for Immediate Action 

While a great deal has been learned in our study of the NCVS recommendations cannot yet be 

made. Unquestionably, though, there do seem to be some major consequences due to the continuing decline 

in NCVS response rates. Among these is an increase in survey costs associated with the greater difficulty in 

attempting to complete interviews and the possible introduction of biases in survey estimates associated 

with high nonresponse rates for some population subgroups. Our research has already increased our 

understanding of the nonresponders and using this information we are now testing methods that might 

allow us to adjust the data more intelligently. The goal of this increasing understanding of nonresponse is to 

alter survey practice, so as to achieve better results without great increases in cost. 

 

63 Lauritsen, J.L., and Schaum, R.J. (2005). Crime and Victimization in the Three Largest Metropolitan 

Areas, 1980-98. NCJ 208075. Washington, DC: Bureau of Justice Statistics. 

64 McDowall D., and Loftin, C. (1992). Comparing the UCR and NCS over time. Criminology, 30, 125-32.

65 E.g., see page 72 in National Research Council (2008). Survey Victims: Options for Conducting the 

National Crime Victimization Survey. Panel to review the Programs of the Bureau of Justice Statistics.

Robert M. Grove and Daniel L. Cork, eds. Committee on National Statistics and Committee on Law and 

Justice, division of Behavioral and Social Sciences and education. Washington, D.C.: The National 

Academy Press. 

66 Federal Bureau of Investigation (FBI) (2008). Crime in the United States, 2007. Washington, DC: US 

Department of Justice. 

27
We believe that the cost-effective decisions can be made if we can validate and utilize at least two 

sources of knowledge: (1) the fact that nonresponse rates are not equal across subpopulations and (2) the 

fact that differential nonresponse does not automatically translate into bias. Therefore, intentionally 

ignoring the nonresponse from certain subpopulations may be both statistically justifiable and also 

economically viable, provided that the balance of the response error and bias can be accounted for. 

We have repeatedly expressed concerns about the first round being potentially biasing. A 

discussion of this and two other process recommendations are highlighted below. 

 Nonresponse during first attempted contact. The literature on panel surveys cited earlier 

suggests that the first round is where the potential for nonresponse bias is the most severe, 

largely because there are so few covariates to model and adjust with.67 Doing more here in 

the NCVS, especially adding to the frame seems an obvious action step. Bringing forward 

additional data from the UCR or the previous census would be good. A close examination of 

the paradata picked up when there is a noncontact or a refusal in the first round outcome 

might be made. In NORC‘s Survey of Consumer Finances, for example, neighborhood 

information is obtained. Some pairing of cases ahead of time, e.g., having two linked 

interviews in the same ultimate cluster could be a sensible precaution for household, person, 

and item nonresponse. 

 Reinterviews to check on response quality and nonresponse bias. The scope of the NORC 

proposal kept us from looking at the Census Bureau‘s reinterview program. We would 

recommend time be spent studying how successful this effort is and whether it could be 

harnessed to study a small sample of nonresponse cases from each round of the NCVS, 

especially but not exclusively the first round. Since the focus will be on bias examination a 

very high response rate will be needed for these reinterviews, making this an expensive 

undertaking in time and money. To limit the effort, a real-time MIS might be set up and 

results posted routinely. Stopping rules could be developed after the program started and after 

efforts to optimize resources were attempted. 

 Imputation Experiments. We stated more detailed ideas in the report to the Bureau of Justice 

Statistics about how to plan and carryout nonresponse adjustments that were mixtures of 

reweighting and imputation. These seem to offer the best general approach to NCVS 

missingness, whether of whole households, persons or individual items. This too should be 

tried in a limited way. 

Acknowledgement 

This study was supported by the Bureau of Justice Statistics of the U.S. Department of Justice (2008-BJCX-K062). The authors thank Jeremy Schimer, David Watt, Laura Flores, and Stephen Ash at the Census 

Bureau who provided help on site. The authors also thank Chet Bowie who provided corporate oversight 

throughout this project at the National Opinion Research Center (NORC). Any opinions expressed in this

paper are those of the authors and do not necessarily represent the official positions of the BJS, the Census 

Bureau, or NORC.

 

67 With only a limited number of covariates the nonresponse may, ceteris paribus, be more often 

nonignorable. 

28