Bureau of Justice Statistics Research and Development Series

EVALUATION OF DIRECT VARIANCE ESTIMATION, 

ESTIMATE RELIABILITY, AND CONFIDENCE INTERVALS 

FOR THE NATIONAL CRIME VICTIMIZATION SURVEY 

Rick Williams, PhD

David Heller, MS

Lance Couzens, BS

Bonnie Shook-Sa, MAS

Marcus Berzofsky, DrPH

Hope Smiley-McDonald, PhD

Chris Krebs, PhD

RTI International

NCJ 249242

R&DP-2015: 02

NCJ 249242 

December 28, 2015 

Research papers have been reviewed by the Bureau of Justice sStatistics (BJS) to ensure the 

general accuracy of information presented and adherence to confidentiality and disclosure 

standards. This paper is released to inform interested parties of research and to encourage 

discussion. Any opinions and conclusions expressed herein are those of the author(s) and do 

not necessarily represent the views of BJS and the U.S. Department of Justice. 
This document was prepared using Federal funds provided by the U.S. Department of Justice 

under Cooperative Agreement number 2011-NV-CX-K068. The BJS Project Managers were 

Lynn Langton, BJS Statistician, and Michael Planty, Victimization Unit Chief.
iii

CONTENTS

Section Page

Section 1. Data Structure ...........................................................................................................1

Section 2. Variance Estimation..................................................................................................3

2.1 National Crime Victimization Survey Design .........................................................3

2.2 Variance Estimation Options...................................................................................4

2.2.1 Generalized Variance Functions..................................................................7

2.2.2 Taylor Series Linearization..........................................................................8

2.2.3 Balance Repeated Replication ...................................................................10

2.3 Single-Year Estimates............................................................................................11

2.4 Pooled-Year Estimates...........................................................................................15

2.5 Cross-Single-Year Comparisons............................................................................16

2.6 Cross-Pooled-Year Comparisons...........................................................................17

Section 3. Estimate Reliability.................................................................................................19

3.1 Introduction............................................................................................................19

3.2 Potential Reliability Flagging Rules......................................................................19

3.3 Reliability Flagging Rules for Victimization Rates and Totals.............................21

3.4 Reliability Flagging Rules for Percentages of Victimizations...............................22

Section 4. Confidence Intervals...............................................................................................41

4.1 Introduction............................................................................................................41

4.2 Confidence Intervals for Victimization Rates........................................................42

4.3 Confidence Intervals for Victimization Totals ......................................................42

4.4 Confidence Intervals for Percentages of Victimizations .......................................44

Section 5. Findings and Recommendations.............................................................................46

Section 6. References...............................................................................................................49
iv

LIST OF EXHIBITS

Number Page

2.1A. Example of Sample Rotation and Phase-in and Phase-out of Decennial 

Samples...........................................................................................................................5

2.2A. Grouping of Years by Decennial Census and Number of Strata by Year.......................6

2.2.2A. Cross Year Strata and PSUs............................................................................................9

2.3A. Percent RSEs for Selected Crime Victimization Rates for Single Years from 

2001 through 2011 ........................................................................................................13

2.3B. Percent RSEs for Selected Crime Victimization Rates for Single Years from 

2001 through 2011 ........................................................................................................14

2.4A. Percent Relative Standard Errors (RSEs) for Selected Crime Victimization 

Rates for Pooled-Year Estimates from 2001–2003, 2002–2004, 2005–2007, 

2008–2010, and 2009–2011..........................................................................................16

2.5A. P-values for Comparisons between Single-Year Victimization Estimates...................17

2.6A. P-values for Comparisons between Pooled-Year Victimization Estimates..................18

3A. Summary of Criteria for Flagging Unreliable Estimates Used by Various 

National Surveys...........................................................................................................20

3B. Unreliable Estimate Flagging Rules Considered ..........................................................21

3C. Number and Percentage of Single-Year Victimization Rates Flagged.........................24

3D. Number and Percentage of Pooled-Year Victimization Rates Flagged........................28

3E. Number and Percentage of Single-Year Victimization Totals Flagged........................32

3F. Number and Percentage of Pooled-Year Victimization Totals Flagged.......................35

3G. Number and Percentage of Victimization Percentages Flagged...................................40

4A. Confidence Interval Summary for Single-Year and Pooled-Year Victimization 

Rates..............................................................................................................................43

4B. Confidence Interval Summary for Single-Year and Pooled-Year Victimization 

Totals.............................................................................................................................43

4C. Confidence Interval Summary for Single-Year and Pooled-Year Percentages of 

Victimizations...............................................................................................................45

5. Findings and Recommendations for Variance Estimation, Estimate Reliability, 

and Confidence Intervals for the National Crime Victimization Survey......................46
1

SECTION 1. DATA STRUCTURE

The National Crime Victimization Survey (NCVS), sponsored by the Bureau of Justice 

Statistics (BJS), estimates the incidence and describes the characteristics of criminal 

victimization in the United States. When calculating NCVS estimates, researchers must take into 

account the complex stratified, four-stage sample design and analysis weights. Stratification, 

clustering, and variation in analysis weights all affect the variances of survey parameters, and not 

appropriately accounting for these factors during estimation can lead to invalid results (Cochran,

1977).

Two broad methods exist for calculating variances of estimates from complex sample 

designs: generalized variance functions (GVFs) and direct variance estimation. GVFs model the 

design-consistent variances for multiple survey estimates to obtain GVF parameters. Using the 

formulas and parameters from the GVF models, users are able to calculate approximations of 

variances without knowledge of the sample design. Direct variance estimation uses software that 

accounts for complex sample designs. Two direct variance techniques are Taylor series 

linearization (TSL) and balanced repeated replication (BRR).

Currently, BJS uses GVFs to calculate variances of NCVS estimates. However, the GVFs 

developed for the NCVS do not allow for complex analyses such as regression modeling, are 

cumbersome when multiple estimates are produced, and produce GVF estimates for outcomes 

not included in developing the GVF parameters that are of unknown accuracy. Use of GVFs 

requires knowledge about the correct GVF parameters and formulas to use, and these decisions 

are dependent on the outcome of interest.

Direct variance estimation has not been used for the NCVS because two weights are 

needed for the calculation of key NCVS estimates (victimization rates): a population weight and 

a victimization weight. The population weight represents the number of persons or households in 

a domain of interest. The victimization weight represents the number of victimizations 

experienced by the person or household. To properly calculate the variance of a rate, an analyst 
2

needs both weights. However, currently, no software package allows for two weight values to be 

used in the calculation of the variance, making it difficult to use direct variance estimation.

This report examines the feasibility of using direct variance estimation for the NCVS. It 

compares GVF estimates to two direct variance estimation methods (TSL and BRR). 

Furthermore, it provides recommendations regarding the calculation of confidence intervals

(CIs) and identification of unstable estimates. A companion user’s guide has been developed to 

describe how to implement the direct variance techniques detailed in this report using different 

software packages (Shook-Sa, Couzens, & Berzofsky, 2014).

When comparing direct variance estimation to the current GVF approach, the following 

areas will be addressed:

1. Single-year estimation

2. Pooled-year estimation

3. Cross-single-year estimation

4. Cross-pooled-year estimation

5. Identifying estimates with low reliability

6. CIs

All comparisons will use “with-series” victimization rates and the newly available “with 

series” GVF parameters, as further discussed in Section 2.3.
3

SECTION 2. VARIANCE ESTIMATION

2.1 National Crime Victimization Survey Design

As described in the NCVS codebook for 2010 (BJS, 2010), the NCVS sample consists of 

approximately 50,000 sample housing units selected each year with a stratified, multistage 

cluster design. The primary sampling units (PSUs) composing the first stage of the sample 

include counties, groups of counties, or large metropolitan areas. PSUs are further grouped into 

strata. Large PSUs are included in the sample automatically and each is assigned its own stratum. 

These PSUs are considered to be self-representing (SR) because all of them were selected. The 

remaining PSUs, called non-self-representing (NSR) because only a subset of them was selected, 

were combined into strata by grouping PSUs with similar geographic and demographic 

characteristics, as determined by the decennial census used to design the sample. A sample of 

one NSR PSU was selected from each stratum. For analytic purposes, the SR PSUs were each 

separated into two pseudo-PSUs and labeled as coming from the same pseudo-stratum. Each 

NSR PSU was paired with a second NSR PSU selected from a similar stratum and labeled as two 

pseudo-PSUs coming from the same pseudo-stratum. The pseudo-PSUs and pseudo-strata are 

important concepts for the variance estimation methods described below and are used to describe 

the sample design when analyzing the data.

The NCVS sample of PSUs is drawn every 10 years from the decennial census and used 

until the next decennial census is available when a new sample of PSUs is selected. At 

approximately mid-decade, sample selection from the most recent census is phased in; before

that, sample selection is based on the census before the most recent one. For example, before

1995, the sample was drawn from the 1980 decennial census. From January 1995 until December 

1997, the sample drawn from the 1990 census was phased in. From January 1998 until 

approximately 2005, the complete NCVS sample was drawn from the 1990 census. From 2005 

through 2007, samples from the 2000 census were phased in.

Because of the continuing nature of the NCVS, a rotation scheme is used to avoid 

interviewing the same household indefinitely. The sample of housing units is divided into six 

rotation groups, and each group is interviewed every 6 months for a period of 3 years. Within 
4

each of the six rotation groups, six panels are designated. A different panel is interviewed each 

month during the 6-month period.

The combined process of phasing out and phasing in samples from the 1990 and 2000 

censuses, along with rotation groups, is shown in Exhibit 2.1A. In year 2004, the sample was

derived entirely from the 1990 census. Years 2005, 2006, and 2007 were a so-called phasein/phase-out period in which the sample included elements from both the 1990 and the 2000 

censuses. This was followed by a period in which the sample was derived entirely from the 2000 

census. As will be shown, the transition between decennial PSU samples is important when 

implementing direct variance estimation.

2.2 Variance Estimation Options

Multistage sample designs, including the design of the NCVS, complicate the analysis of 

the data because the individual observations are not independent (Wolter, 1985). The 

observations are correlated because of having been selected from geographic clusters of 

observations of likely related units (e.g., within PSUs or housing units). Also, using the same 

sample of PSUs for a 10-year period, combined with repeated interviews of the same housing 

units over rotating 3-year periods, causes estimates from years using the same PSU sample to be 

correlated.

In the sections that follow, three methods for variance estimation are discussed and 

compared. The first is GVFs, which have been available for use with the NCVS public use data 

for many years. The other two, TSL and BRR, are two direct variance estimation methods that 

are being explored in this report as alternative methods for use with the NCVS public use data.

Direct variance estimation methods use statistical software designed to calculate the 

variance of an estimate directly from the full dataset. To implement direct variance estimation, 

users must organize and code the data so that each observation is associated with the strata and 

PSU from which it was selected. To this end, the public use data files include the following two 

variables:
5

Exhibit 2.1A. Example of Sample Rotation and Phase-in and Phase-out of Decennial 

Samples
6

Pseudo-stratum (V2117): The variable designating the pseudo-stratum code associated 

with each observation created from the sampling strata used to select the PSUs.

Half-sample (V2118): The variable designating the pseudo-PSU code associated with 

each observation created from the sampling PSUs selected into the sample. The term 

“half-sample” is used because each pseudo-stratum includes two pseudo-PSUs that

approximately divide the sample in half.

The terms “strata” and “PSU” will be used throughout this document to refer to the 

variables pseudo-stratum (V2117) and half-sample (V2118).

Exhibit 2.2A presents the number of 

strata included on the NCVS public use files 

from 1993 through 2011, with each stratum 

containing two PSUs. The exhibit also 

presents the grouping of years for which 

decennial census data were used to select the 

sample of PSUs contributing to the data for 

the years in each group. Except for issues 

arising from the phase-in/phase-out periods, 

the PSUs used to select the data within a 

Year Group are the same for each year, 

whereas for the between-Year Groups the 

samples of PSUs are different. Thus, the data 

between Year Groups are assumed to be 

independent, but the data within a Year 

Group are assumed to be cluster correlated 

within the PSUs across years. These 

assumptions will be used for direct variance 

estimation. Although these assumptions are 

only approximately true because of the phase-in/phase-out process, the assumptions are 

necessary because the public use data files do not contain the level of detail needed to separately 

account for the overlap of PSUs during the phase-in/phase-out period. The approximations will, 

however, support appropriate direct variance estimation.

Exhibit 2.2A. Grouping of Years by 

Decennial Census and 

Number of Strata by Year

Grouping of Years by 

Decennial Census Year

Number 

of Strata

Year Group 1

Primary sampling unit (PSU)

sample primarily from the 1980 

decennial census

1993 164

1994 164

1995 164

1996 164

Year Group 2

PSU sample primarily from the 

1990 decennial census

1997 143

1998 143

1999 143

2000 143

2001 143

2002 143

2003 143

2004 143

2005 144

Year Group 3

PSU sample primarily from the 

2000 decennial census

2006 160

2007 160

2008 160

2009 160

2010 160

2011 160
7

2.2.1 Generalized Variance Functions

Within the NCVS, GVFs are formulae estimated by the U.S. Census Bureau that 

approximate the variance of an estimate as a function of readily available information about the 

estimate. The process starts by selecting a set of NCVS estimates and calculating their associated 

variances. Over the years, the Census Bureau has estimated the variances using different direct 

variance estimation methods, including TSL, jackknife, BRR, and successive difference 

replication. The first three methods are widely used (Wolter, 1985), but the latter is a more 

specialized method described in Fay and Train (1995) and Ash (2010). Modeling methods like 

those described in Wolter (1985, Chapter 5) are then used to model the variance as a function of 

such values as the estimate, the sample size or the population size, or other characteristics related 

to the sample design (such as location of urban or rural) or to the respondent (such as age, race, 

or marital status). It is also common that separate models are required for various types of 

estimates—for example, victimization rates, totals, or percentages. The resulting models are 

called GVFs.

Although GVFs are easy to use, they are limited to the specific situations for which they 

are designed. Moreover, separate GVFs are needed for different victimization types and for each 

year. Thus, when conducting a large analysis spanning several years and victimization types, 

many different GVFs are needed, which makes it difficult to manage the analysis.

To appropriately calculate GVF estimates, analysts must decide whether to include or 

exclude series or repeat victimizations. Series victimizations occur when an NCVS respondent 

recalls at least six criminal incidents of a similar nature but cannot recall the dates and other 

details of the individual incidents well enough to report them separately. In these cases, the 

respondent can report the number of victimizations and the details of only the most recent event. 

Until recently, BJS reported crime statistics excluding series victimizations, but BJS now reports 

crime victimization statistics including series victimizations (Lauritsen, Owens, Planty, Rand, & 

Truman, 2012). Up until this change, the U.S. Census Bureau created GVFs for estimates 

excluding series victimizations. In July 2013 and February 2014, the Census Bureau issued 

updated GVFs for estimates including series victimizations for years 1993–2012. Because BJS 

recommends including series victimizations in the calculation of victimization rates, this report 

will use data including series victimizations. However, because the inclusion of series 
8

victimizations dramatically affects estimated variances, if series victimizations are excluded, the 

appropriate “without-series” GVF parameters should be used in estimation. The importance of 

using the appropriate set of GVF parameters is demonstrated in Section 2.3.1.

2.2.2 Taylor Series Linearization

For a stratified multistage cluster sample like the one used for the NCVS, there is an 

unbiased variance estimator for a linear statistic. An example of a linear statistic is the estimated 

total number of victimizations for a year given by = ∑ wwjjyyjj nn

jj=1 , where wwjj and yyjj are the analysis 

weight and the number of victimizations incurred by the j

th participant in the survey, 

respectively. The variance estimator is based on the commonly used assumption that the PSUs in 

a multistage sample were selected with replacement. Although replacement PSU selection is 

almost never done, it is a good approximating assumption when the sampling fraction (i.e., the 

ratio of the number of PSUs selected to the total number of PSUs in the stratum) among the 

PSUs is small. For the NCVS, only two PSUs per stratum are selected out of a large number of 

PSUs available per stratum, so the replacement PSU sampling assumption is appropriate. The 

variance estimation formula is

VVTTTTTT(YY) = � nnh

nnh − 1�(YYhii − YY�

h)2

nnh

ii=1

HH

h=1

,

where YYhii = ∑ wwhii yyhiijj

mmhii

jj=1 and YY�

h = ∑ yyhii

nnh

ii=1 /nnh. The subscripts have been expanded to include 

strata (h), PSUs (ii), and respondents (jj), and with nnh being the number of PSUs in a stratum and 

mmhii the number of respondents in a PSU. This variance estimator has been shown to be unbiased 

for linear statistics (Särndal, Swensson, & Wretman, 1992; Williams, 2000).

When a nonlinear statistic is being considered, the TSL method replaces the nonlinear 

statistic with a first-order Taylor series linear approximation and then uses the above variance 

estimator with the linear approximation data to estimate the variance of the nonlinear statistic. 

The resulting variance estimate is a consistent estimate of the variance of the nonlinear statistic. 

For example, the victimization rate is estimated by RR = YY/XX, where YY is the estimated total 

number of victimizations as just described and XX = ∑ wwjj nn

jj=1 is the estimated total number of 
9

people in the population. Following the descriptions in Wolter (1985, Section 6.5) or Williams 

(2008), it can be shown that the linearized values for a ratio are zzjj = (yyjj − RRxxjj)/XX.

The TSL method is widely implemented in statistical analysis software packages, such as 

SUDAAN, SAS, Stata, and SPSS (complex samples package). All of these analysis packages 

automatically determine the linearized values for a wide range of statistics without the need for 

user input. However, the analysis packages require the user to specify the strata and PSUs used 

to select the sample so that the variance can be estimated appropriately. For an estimate based 

upon data from a single year, the 

variables pseudo-stratum (V2117) and 

half-sample (V2118) are the variables 

that specify the strata and PSUs to the 

analysis package. The situation is 

slightly more complex when analyzing 

data across years because of the use of 

the same PSUs across 10-year intervals 

and the repeated interviewing of the 

same households over 3 years. In this 

situation, the same strata and PSUs are 

used across years within the Year 

Groups shown in Exhibit 2.2A. The key 

is to group data across the years by the 

strata and PSUs used to select the data. 

Thus, Exhibit 2.2A illustrates how to 

create cross-year strata so that data 

within the same Year Group use the 

same strata and PSUs in the variance 

calculation, which will capture the 

statistical correlation among these data. On the other hand, the cross-year strata will separate the 

data from two different Year Groups in the variance calculation and treat the different Year 

Exhibit 2.2.2A. Cross Year Strata and PSUs

Cross-Year Strata PSUs

Years of 

Data

Year 

Group

Pseudo-stratum 

(V2117)

Half-sample 

(V2118)

1 1 1 1993–1996

1 1 2 1993–1996

1 2 1 1993–1996

1 2 2 1993–1996

⁞ ⁞ ⁞ ⁞

1 164 1 1993–1996

1 164 2 1993–1996

2 1 1 1997–2005

2 1 2 1997–2005

2 2 1 1997–2005

2 2 2 1997–2005

⁞ ⁞ ⁞ ⁞

2 144 1 1997–2005

2 144 2 1997–2005

3 1 1 2006–2011

3 1 2 2006–2011

3 2 1 2006–2011

3 2 2 2006–2011

⁞ ⁞ ⁞ ⁞

3 160 1 2006–2011

3 160 2 2006–2011
10

Groups as statistically independent. This process is more fully described in the User’s Guide 

(Shook-Sa, Couzens, & Berzofsky, 2014).

2.2.3 Balance Repeated Replication

BRR is another commonly used direct variance estimate method for complex sample 

surveys (Lumley, 2008). Like the TSL method, BRR takes advantage of the with-replacement 

sampling assumption of the PSU sample. BRR is most easily implemented for a stratified sample 

with two PSUs selected per stratum like the pseudo-strata and pseudo-PSUs of the NCVS. The 

NCVS is separated into half-samples created by selecting one PSU from each stratum. The 

weights of observations in the selected half-sample are doubled, and the weights for the

remaining observations are set to zero. A half-sample estimate of a statistic (victimization total, 

rate, or percent) is then obtained from the half-sample data. A large number of half-samples are 

generated along with a corresponding set of half-sample estimates denoted as θθ1, ... , θθGG, where G

is the total number of half-samples created. The variance is then estimated by

VV(θθ) = ��θθgg − θθ�

2

GG

gg=1

/GG,

where θθ is the estimated statistic from the full NCVS sample. The set of half-samples is usually 

selected so that they are in full orthogonal balance, in which case an efficient and consistent 

estimate of the variance is obtained. The conditions and methods for creating half-samples with 

full orthogonal balance are described by Wolter (1985, Chapter 3).

Similar to the TSL method, special consideration is needed to account for the overlap in 

strata and PSUs within a Year Group. The same cross-year strata and PSUs presented in 

Exhibit 2.2A can be used when forming the BRR half-samples. When data from a single Year 

Group are analyzed, the strata and PSUs specific to that Year Group are used to form the halfsamples for BRR estimation. Once formed, the same half-samples are used for all years within 

the Year Group. For example, Year Group 1 has 164 strata, each with two PSUs for all the years 

of data in Year Group 1; the half-samples would be formed from these strata and PSUs. For 

analyses using data from two Year Groups, half-samples are needed using the strata and PSUs 

from both Year Groups. For example, if data were being compared across Year Groups 1 and 2, 

say pooled data from 1993 through1996 compared with those from 1997 through 1999, then 
11

half-samples would be created from the combined 208 (164 + 144 = 208) strata from Year 

Groups 1 and 2. Finally, if all three Year Groups were included in the analysis, half-samples 

would be created from all 368 strata (164 + 144 + 160 = 368). In any of these cases, the data 

within a Year Group would be included or excluded from the same half-samples so as to capture 

the correlations due to sharing the same PSUs in a Year Group.

2.3 Single-Year Estimates

This section explores single-year victimization rate and total estimates and compares the 

GVF, TSL, and BRR variance estimation approaches. It also demonstrates why choosing the 

correct set of parameters (i.e., parameters including or excluding series victimizations) is 

essential when calculating GVF estimates. The following victimization types are included:

Personal Victimization Types

 Rape/sexual assault

 Robbery

 Aggravated assault

 Simple assault

 Personal theft

Property Victimization Types

 Household burglary

 Motor vehicle theft

 Theft

For each of these victimization types, estimates were produced for the following subpopulations:

Personal Victimization Subpopulations

 Sex

 Race

 Age category

 Region

 Rural/urban

 Metropolitan Statistical Area 

(MSA) status

Property Victimization Subpopulations

 Household income

 Region

 Rural/urban

 MSA status

To study the relationships among these variance estimates, the percent relative standard 

error (RSE) was used. The percent RSE, the square root of the variance of an estimate divided by 

the estimate, is expressed as a percentage �100 × �VVVVVV(YY)/YY�. The percent RSE removes the 

scale of the estimate and allows comparisons to be made across multiple types of estimates with 

different scales (e.g., totals versus rates).
12

Exhibit 2.3A presents three figures summarizing the results for crime victimization rates 

for single-year estimates from 2001 through 2011. Series victimizations are included in the 

estimates and the GVFs. The figures display the relationship between the three variance 

estimation methods—TSL vs. GVF, BRR vs. GVF, and TSL vs. BRR—by plotting percent RSE 

from one method along the horizontal (x) axis and the alternative method along the vertical (y) 

axis. If two methods produce consistent results then the bulk of the RSE comparisons would fall 

along the 45° line of equality between the two methods with some estimates varying slightly 

above or below the line. Figures were also produced for crime victimization totals, but they were 

almost identical to the victimization rate figures and are therefore, not presented herein.

The first item of note is that both the TSL and the BRR methods match the GVF method 

well. The RSEs in both Figures 1 and 2 of Exhibit 2.3A are centered on the 45° line, indicating 

congruence between the two methods. When the RSEs are less than 30%, they are tightly 

clustered around the 45° line, whereas a wider spread is found for the estimates with RSEs 

greater than 30%. An estimate with a large RSE is not reliably estimated and will have a wide CI

no matter which variance estimation method is used. Figures 1 and 2 provide confidence that the 

TSL and BRR methods applied to the public use data files are matching the methods used by the 

Census Bureau when producing the GVFs.

A second item of note is that the TSL and BRR methods yield almost exactly the same 

results as shown in Figure 3 in Exhibit 2.3A. All plotted values are extremely close to the 45°

line. In addition, the relationship between TSL and BRR was explored for pooled-year estimates 

and for comparison tests between years. All of these situations also showed that TSL and BRR 

variance estimates and tests of differences were almost exactly the same for the NCVS public 

use data. In addition, as described in Section 2.2.3, the BRR method requires a much more 

complex data setup than the TSL method does to account for the phase-in/phase-out of PSUs 

across the Year Groups. Furthermore, although several analysis packages support both TSL and 

BRR methods, one of the most widely used by NCVS researchers is SPSS, which does not 

support BRR variance estimation. For these reasons, BRR direct variance estimation was not 

examined further, and the remainder of this report will focus on TSL direct variance estimation.
13

Exhibit 2.3A. Percent RSEs for Selected Crime Victimization Rates for Single Years from 2001 through 2011

Figure 1. TSL vs. GVF Figure 2. BRR vs. GVF Figure 3. TSL vs. BRR

Note: Series victimizations are included in both the estimates and in the GVFs.
14

As stated earlier, the U.S. Census Bureau has prepared GVFs for estimates in which 

series victimization reports are both included and excluded, and the appropriate parameters must 

be used to obtain unbiased variance estimates. To demonstrate the importance of using the 

correct GVF parameters, single-year estimates were prepared for the years 2001 through 2011

including series victimizations. Direct TSL variances were calculated for all of these estimates. 

GVF estimates were calculated two ways: using the “with series” parameters (as appropriate 

when series victimizations were included) and using the without-series parameters. The results 

are summarized in Exhibit 2.3B, in which the percent RSEs from the TSL method are compared 

with the percent RSEs from the GVFs.

Exhibit 2.3B. Percent RSEs for Selected Crime Victimization Rates for Single Years from 

2001 through 2011

Figure 1. GVFs Excluding Series Victimizations Figure 2. GVFs Including Series 

Victimizations

Note: Series victimizations are included in the estimates for both Figures 1 and 2.

When the estimates include series victimizations but the without-series GVF parameters 

are used, as shown in Figure 1 of Exhibit 2.3B, the majority of the plotted values are above the 

45° line of equality, which means that most of the TSL percent RSEs are greater than the GVF 

percent RSEs. This result is likely because the GVFs for these years were developed excluding 

series victimizations and the GVF RSEs are too small because they do not account for the added 

variability that arises from including series victimizations. Additional evidence for this inference 
15

is shown in Figure 2 of Exhibit 2.3B, in which the estimates and GVF parameters include series

victimizations. In this situation, the TSL and the GVF methods closely align, as shown by the 

clustering of the plotted RSEs around the 45° line of equality. As with Exhibit 2.3A, estimates 

are more tightly clustered around the 45° line when the RSEs are less than 30%, whereas a wider 

spread is found for the estimates with RSEs greater than 30%. This figure shows that TSL and 

GVF estimates track well when series victimizations are included and the appropriate GVF 

parameters are used. To confirm that the without-series parameters track well with direct 

variance estimates when series victimizations are excluded, annual victimization rates were 

calculated excluding series victimizations, and percent RSEs for TSL estimates were compared 

to percent RSEs for GVF estimates (using the without-series parameters). The results were very 

consistent, clustering around the 45° line (figures not included). These results provide further 

evidence that TSL and GVF estimates track well regardless of whether series victimizations are 

included or excluded, provided that the appropriate GVF parameters are used. The remainder of 

this report will focus solely on estimates including series victimizations and the appropriate GVF 

parameters.

2.4 Pooled-Year Estimates

Because many types of victimization occur at very low rates, it is often necessary to pool 

several years of data together in order to obtain enough cases to support an analysis. This section 

considers estimates from data pooled across 3-year periods for the same victimization types and 

subpopulations listed in Section 2.3.

Exhibit 2.4A presents a comparison of the TSL percent RSEs and the GVF percent RSEs 

for pooled estimates from five different Year Groups: 2001–2003, 2002–2004, 2005–2007,

2008–2010, and 2009–2011 (similar to what was presented in Exhibit 2.3A). The results for 

crime victimization totals were nearly identical to the rates and, thus, are not included. The GVF 

and TSL variance methods correspond closely for pooled-year estimates, as demonstrated by the 

plotted values, which are clustered around the 45° line of equality for the two methods. This 

reinforces the earlier conclusion that the TSL direct variance estimation method has been 

properly specified for use with the NCVS public use data.
16

Exhibit 2.4A. Percent Relative Standard Errors (RSEs) for Selected Crime Victimization 

Rates for Pooled-Year Estimates from 2001–2003, 2002–2004, 2005–2007, 

2008–2010, and 2009–2011

2.5 Cross-Single-Year Comparisons

This section considers tests of differences, or comparisons, between estimates from two 

years. The hypothesis tested is Ho: RRuu = RRss vs. Ha: RRuu ≠ RRss, where RRuu and RRss are the 

victimization rates for two different years, uu and ss. The test statistic is zz =

| uu − ss|⁄�VVVVVV( uu − ss), where uu and ss are the estimated values of the two victimization rates 

being compared. The test statistic is considered to follow a standard normal distribution. 

Likewise, comparisons of victimization totals can also be tested by substituting totals for rates in 

the preceding hypothesis and test statistic. For this analysis, the same victimization types and 

subpopulations listed in Section 2.3 were included for three sets of single-year comparisons: 

2004 against 2005, 2005 against 2006, and 2001 against 2011. These comparisons include 

examples of years in the same Year Group drawn from the same PSUs (2004 against 2005), 

years spanning the phase-in/phase-out period (2005 against 2006), and years in different Year 

Groups consisting of independent samples of PSUs (2001 against 2011).
17

Exhibit 2.5A presents the p-values associated with the tests of the cross-year comparisons 

computed using either the TSL or the GVF method to estimate the variance of the difference 

between 2 years. Similar to previous exhibits, the TSL and GVF p-values are compared by 

plotting the GVF p-values along the horizontal (x) axis and the TSL p-values along the vertical 

(y) axis. For both victimization rates and totals, the p-values are well aligned along the 45° line 

of equality, which shows that the two methods yield similar results.

Exhibit 2.5A. P-values for Comparisons between Single-Year Victimization Estimates

Note: Comparisons between 2004 and 2005 estimates, 2005 and 2006 estimates, and 2001 and 2011 estimates.

2.6 Cross-Pooled-Year Comparisons

As was noted in Section 2.4, it is often necessary to pool several years of data together in 

order to obtain enough cases to support an analysis. This section extends the discussion in 

Section 2.5 to the test of differences, or comparisons, between estimates from two different 

poolings of years. The same hypothesis and test statistic from Section 2.5 are considered here

and comparisons are made between two sets of pooled-year estimates (2002–2004 against 2005–

2007 and 2001–2003 against 2009–2011) for the same victimization types and subpopulations 

listed in Section 2.3. Like the single-year comparisons, these years were selected to span 

multiple Year Groups and to include years during a phase-in/phase-out period.
18

Exhibit 2.6A presents the p-values associated with tests of the cross-pooled-year

comparisons using either the TSL or the GVF variance estimation methods in the same way as 

was done in Exhibit 2.5A. Again, the TSL and the GVF methods yield similar results for both 

victimization rates and totals, with the p-values well aligned along the 45° line of equality.

Exhibit 2.6A. P-values for Comparisons between Pooled-Year Victimization Estimates

Note: Comparisons between pooled 2002–2004 estimates and pooled 2005–2007 estimates, and pooled 2001–2003 

estimates and pooled 2009–2011 estimates.
19

SECTION 3. ESTIMATE RELIABILITY

3.1 Introduction

This section considers reliability standards for NCVS estimates and rules for flagging 

estimates with low reliability. In general terms, it is desirable that a NCVS estimate neither vary

widely because of sampling variation nor be overly influenced by a small number of 

observations. When neither situation occurs, an estimate is said to be reliable because the users 

can have confidence that conclusions drawn from the estimate are likely not due to random 

chance. On the other hand, it is difficult to interpret an unreliable estimate because the observed 

value may be due to random chance or may not represent the target population well. It is 

common for large national surveys like the NCVS to provide guidance to users concerning 

unreliable estimates. This is often done by establishing rules for flagging estimates in reports that 

potentially have low reliability. This chapter presents an evaluation of such rules using the TSL 

variance estimation approach.

3.2 Potential Reliability Flagging Rules

The two dimensions for measuring reliability considered here are

• variance, or RSE, and

• sample size.

The NCVS has historically flagged estimates of all types (rates, totals, and percentages) 

as unreliable when the numerator sample size is less than or equal to 10 or the percent RSE is 

greater than 50%. These flagging criteria can be compared with the review conducted by Klein, 

Proctor, Boudreault, and Turczyn (2002) of the rules used by various national surveys to flag, or 

suppress, unreliable estimates. A summary of their findings is presented in Exhibit 3A.
20

Exhibit 3A. Summary of Criteria for Flagging Unreliable Estimates Used by Various 

National Surveys

Survey Title Criteria for Flagging Unreliable Estimates

Behavioral Risk Factor Surveillance Survey (BRFSS) Denominator < 50

Medical Expenditure Survey (MEPS) Denominator < 70 or RSE > 30%

National Ambulatory Medical Care Survey (NAMCS) Numerator < 30 or RSE > 30%

National Health Interview Survey (NHIS) Denominator < 50 or RSE > 30%

National Health and Nutrition Examination Survey (NHANES) Denominator < 30 or RSE > 30%

National Hospital Discharge Survey (NHDS) Numerator < 30 or RSE > 30%

National Survey of Family Growth (NSFG) Denominator < 50 or RSE > 30%

Youth Risk Behavior Surveillance Survey (YRBSS) Denominator < 100 events

Note: Numerator and denominator are sample sizes. RSE, relative standard error.

Source: Klein, R. J., Proctor, S. E., Boudreault, M. A., & Turczyn, K. M. (2002). Healthy People 2010 criteria for 

data suppression (Healthy People 2010 Statistical Notes No. 24). Atlanta, GA: Centers for Disease Control 

and Prevention. Retrieved from http://www.cdc.gov/nchs/data/statnt/statnt24.pdf.

On the basis of the previous NCVS experience and the rules summarized in Exhibit 3A, 

the flagging rules presented in Exhibit 3B were considered as alternatives for use with the 

NCVS. For each estimate type, all combinations of sample size and percent RSE values from

Exhibit 3B were considered and applied to a selected set of estimates—for example, for rates,

one such set of flagging criteria was numerator ≤ 30 or RSE > 30%. The number and percentage

of estimates that would be flagged for each combination of criteria were tabulated and are 

discussed below. All RSEs were computed using the TSL variance estimation method.
21

Exhibit 3B. Unreliable Estimate Flagging Rules Considered

Estimate Type and Rule Set Event Sample Size Percent RSE

Rate (per 1,000 persons) a. Numerator ≤ 10;

b. Numerator ≤ 20; or

c. Numerator ≤ 30

a. RSE > 30%; or

b. RSE > 50%

Total a. Count ≤ 10;

b. Count ≤ 20; or

c. Count ≤ 30

a. RSE > 30%; or

b. RSE > 50%

Percentage-1 a. Denominator ≤ 10;

b. Denominator ≤ 20; or

c. Denominator ≤ 30

a. RSE > 30%; or

b. RSE > 50%

Percentage-2 a. Numerator ≤ 10;

b. Numerator ≤ 20; or

c. Numerator ≤ 30

a. RSE > 30%; or

b. RSE > 50%

Percentage-3 a. Denominator ≤ 10;

b. Denominator ≤ 20; or

c. Denominator ≤ 30

a. RSE log(p or 100−p) > 30%; or

b. RSE log(p or 100−p) > 50%

Percentage-4 a. Numerator ≤ 10;

b. Numerator ≤ 20; or

c. Numerator ≤ 30

a. RSE log(p or 100−p) > 30%; or

b. RSE log(p or 100−p) > 50%

3.3 Reliability Flagging Rules for Victimization Rates and Totals

The reliability flagging rules for victimization rates and totals are related because a total 

estimate is the numerator for the corresponding rate estimate. Consequently, rates and totals will 

be considered jointly in this section. In addition, as has been done in previous sections of this 

report, single-year and pooled-year estimates will be considered. Estimates were created for 

combinations of the victimizations types and subpopulations listed in Section 2.3. Single-year

estimates are for calendar years 2008–2011, whereas pooled-year estimates are for 2006–2008 

and 2009–2011.

The combinations of possible rules in Exhibit 3B generate a very large number of 

possibilities, each of which was reviewed but is not presented here. From this review, 

recommended rules for flagging unreliable estimates were determined. These are presented 

below, followed by a summary of the characteristics of the rules.
22

For victimization rates and totals, the following rules are recommended.

Victimization Rates

Flag an estimate as unreliable if

RSE > 30% or

numerator sample size ≤ 10.

Victimization Totals

Flag an estimate as unreliable if

RSE > 30% or

count sample size ≤ 10.

These rules for flagging unreliable estimates are recommended because they continue 

several features that have been used by the NCVS for many years while tightening the RSE 

criteria to be consistent with those of many other major national surveys.

The percentage of estimates recommended to be flagged as unreliable is summarized in 

Exhibits 3C, 3D, 3E, and 3F for single-year rates, pooled-year rates, single-year totals, and 

pooled-year totals, respectively. The percentage of estimates recommended to be flagged across 

domains is similar, but not identical, for rates and totals. However, the overall percentages of 

estimates recommended to be flagged are the same for rates and totals. Overall, 27.2% of singleyear rates and totals would be flagged (Exhibits 3C and 3E), which drops to 6.7% (Exhibits 3D

and 3F) for pooled-year rates and totals. For combinations of other criteria, the percentage of 

estimates flagged would be between 17.2% and 35.2% for single-year rates and totals and 1.3% 

and 15.0% for pooled-year rates and totals (results not shown).

3.4 Reliability Flagging Rules for Percentages of Victimizations

Percentages of victimizations are another type of estimate often created with NCVS data. 

Examples include the percentages of robberies committed against women or the percentage of 

household burglaries committed during daylight hours. A special consideration is that the RSE 

can become inflated for small percentages. The percent RSE for a percentage is 100 ×

�VVVVVV(PP)/PP, where PP is the percentage of victimizations under study. When PP is near 0%, the 

RSE will become large and can understate the precision of the estimated percentage. In addition, 

it is desirable that a reliability criterion for a percentage treat similarly the situations where PP
23

approaches either 0% or 100% because the variance of PP and 100−PP are the same; however, the 

RSE for PP and 100−PP are not the same. One option is to apply the reliability criterion to either 

the RSE of the log(PP) or log(100−PP), as is done in the National Survey on Drug Use and Health 

(Substance Abuse and Mental Health Services Administration, 2011). When the percentage is 

less than or equal to 50%, then the RSE of log(PP) is considered, whereas the RSE of log(100−PP) 

is considered when the percentage is greater than 50%. The log transformation lessens the impact 

of PP being near 0%, whereas the switch between PP and 100−PP makes the rule symmetric. This is 

the criterion included in the rows labeled Percentage-3 and Percentage-4 in Exhibit 3B. The 

percent RSE for the log(PP) is 100 × �VVVVVV(PP)�[PPabs(log(PP/100))] and, for the log(100−PP), 

the percent RSE is 100 × �VVVVVV(PP)�[(100 − PP)abs(log(1 − PP/100))].
24

Exhibit 3C. Number and Percentage of Single-Year Victimization Rates Flagged

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number Flagged Percent Flagged Number Flagged Percent Flagged Number Flagged Percent Flagged

All Crime Types and 

Subgroups 600 157 26.17 85 14.17 163 27.17

Personal Crimes 420 155 36.91 85 20.24 161 38.33

Rape/sexual assault 84 63 75.00 25 29.76 66 78.57

Robbery 84 24 28.57 7 8.33 24 28.57

Aggravated assault 84 9 10.71 4 4.76 9 10.71

Simple assault 84 4 4.76 0 0.00 4 4.76

Personal theft 84 55 65.48 49 58.33 58 69.05

Property Crimes 180 2 1.11 0 0.00 2 1.11

Household burglary 60 0 0.00 0 0.00 0 0.00

Motor vehicle theft 60 2 3.33 0 0.00 2 3.33

Theft 60 0 0.00 0 0.00 0 0.00

Estimate Range

0.00–0.25 18 16 88.89 18 100.00 18 100.00

0.25–0.50 34 28 82.35 28 82.35 29 85.29

0.5–1.0 70 40 57.14 28 40.00 43 61.43

1.0–5.0 217 69 31.80 11 5.07 69 31.80

5.0–75.0 207 4 1.93 0 0.00 4 1.93

75.0+ 54 0 0.00 0 0.00 0 0.00

(continued)
25

Exhibit 3C. Number and Percentage of Single-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number Flagged Percent Flagged Number Flagged Percent Flagged Number Flagged Percent Flagged

Personal Crimes by 

Subgroups

Sex

Male 20 6 30.00 4 20.00 6 30.00

Female 20 1 5.00 0 0.00 1 5.00

Race/Hispanic origin

White 20 2 10.00 0 0.00 2 10.00

Black 20 9 45.00 7 35.00 9 45.00

Hispanic 20 10 50.00 6 30.00 10 50.00

Other 20 13 65.00 9 45.00 15 75.00

Age

12–17 20 10 50.00 5 25.00 10 50.00

18–24 20 7 35.00 3 15.00 7 35.00

25–34 20 8 40.00 5 25.00 8 40.00

35–49 20 8 40.00 3 15.00 8 40.00

50–64 20 7 35.00 8 40.00 9 45.00

65 or older 20 19 95.00 15 75.00 19 95.00

(continued)
26

Exhibit 3C. Number and Percentage of Single-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number Flagged Percent Flagged Number Flagged Percent Flagged Number Flagged Percent Flagged

Region

Northeast 20 9 45.00 2 10.00 9 45.00

Midwest 20 8 40.00 2 10.00 9 45.00

South 20 4 20.00 2 10.00 4 20.00

West 20 7 35.00 1 5.00 8 40.00

Rural/urban

Urban 20 1 5.00 0 0.00 1 5.00

Rural 20 12 60.00 6 30.00 12 60.00

MSA status

Yes 20 1 5.00 0 0.00 1 5.00

No 20 12 60.00 7 35.00 12 60.00

Property Crimes by 

Subgroups

Household income

Less than $15,000 12 0 0.00 0 0.00 0 0.00

$15,000–$24,999 12 1 8.33 0 0.00 1 8.33

$25,000–$49,999 12 0 0.00 0 0.00 0 0.00

$50,000–$74,999 12 0 0.00 0 0.00 0 0.00

$75,000 or more 12 0 0.00 0 0.00 0 0.00

Unknown 12 0 0.00 0 0.00 0 0.00

(continued)
27

Exhibit 3C. Number and Percentage of Single-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number Flagged Percent Flagged Number Flagged Percent Flagged Number Flagged Percent Flagged

Region

Northeast 12 0 0.00 0 0.00 0 0.00

Midwest 12 0 0.00 0 0.00 0 0.00

South 12 0 0.00 0 0.00 0 0.00

West 12 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 12 0 0.00 0 0.00 0 0.00

Rural 12 0 0.00 0 0.00 0 0.00

MSA status

Yes 12 0 0.00 0 0.00 0 0.00

No 12 1 8.33 0 0.00 1 8.33

Note. MSA, Metropolitan Statistical Area; RSE, relative standard error.
28

Exhibit 3D. Number and Percentage of Pooled-Year Victimization Rates Flagged

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

All Crime Types and 

Subgroups 300 20 6.67 3 1.00 20 6.67

Personal Crimes 210 20 9.52 3 1.43 20 9.52

Rape/sexual assault 42 14 33.33 2 4.76 14 33.33

Robbery 42 1 2.38 0 0.00 1 2.38

Aggravated assault 42 0 0.00 0 0.00 0 0.00

Simple assault 42 0 0.00 0 0.00 0 0.00

Personal theft 42 5 11.91 1 2.38 5 11.91

Property Crimes 90 0 0.00 0 0.00 0 0.00

Household burglary 30 0 0.00 0 0.00 0 0.00

Motor vehicle theft 30 0 0.00 0 0.00 0 0.00

Theft 30 0 0.00 0 0.00 0 0.00

Estimate Range

0.00–0.25 5 4 80.00 3 60.00 4 80.00

0.25–0.50 12 3 25.00 0 0.00 3 25.00

0.5–1.0 40 6 15.00 0 0.00 6 15.00

1.0–5.0 103 7 6.80 0 0.00 7 6.80

5.0–75.0 111 0 0.00 0 0.00 0 0.00

75.0+ 29 0 0.00 0 0.00 0 0.00

(continued)
29

Exhibit 3D. Number and Percentage of Pooled-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Personal Crimes by 

Subgroups

Sex

Male 10 1 10.00 0 0.00 1 10.00

Female 10 0 0.00 0 0.00 0 0.00

Race/Hispanic origin

White 10 0 0.00 0 0.00 0 0.00

Black 10 1 10.00 0 0.00 1 10.00

Hispanic 10 1 10.00 0 0.00 1 10.00

Other 10 3 30.00 1 10.00 3 30.00

Age

12–17 10 1 10.00 0 0.00 1 10.00

18–24 10 1 10.00 0 0.00 1 10.00

25–34 10 0 0.00 0 0.00 0 0.00

35–49 10 1 10.00 0 0.00 1 10.00

50–64 10 4 40.00 0 0.00 4 40.00

65 or older 10 2 20.00 1 10.00 2 20.00

(continued)
30

Exhibit 3D. Number and Percentage of Pooled-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Region

Northeast 10 0 0.00 0 0.00 0 0.00

Midwest 10 1 10.00 0 0.00 1 10.00

South 10 0 0.00 0 0.00 0 0.00

West 10 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 10 0 0.00 0 0.00 0 0.00

Rural 10 2 20.00 1 10.00 2 20.00

MSA status

Yes 10 0 0.00 0 0.00 0 0.00

No 10 2 20.00 0 0.00 2 20.00

Property Crimes by 

Subgroups

Household income

Less than $15,000 6 0 0.00 0 0.00 0 0.00

$15,000–$24,999 6 0 0.00 0 0.00 0 0.00

$25,000–$49,999 6 0 0.00 0 0.00 0 0.00

$50,000–$74,999 6 0 0.00 0 0.00 0 0.00

$75,000 or more 6 0 0.00 0 0.00 0 0.00

Unknown 6 0 0.00 0 0.00 0 0.00

(continued)
31

Exhibit 3D. Number and Percentage of Pooled-Year Victimization Rates Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Number 

Flagged Percent Flagged

Region

Northeast 6 0 0.00 0 0.00 0 0.00

Midwest 6 0 0.00 0 0.00 0 0.00

South 6 0 0.00 0 0.00 0 0.00

West 6 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 6 0 0.00 0 0.00 0 0.00

Rural 6 0 0.00 0 0.00 0 0.00

MSA status

Yes 6 0 0.00 0 0.00 0 0.00

No 6 0 0.00 0 0.00 0 0.00

Note. MSA, Metropolitan Statistical Area; RSE, relative standard error.
32

Exhibit 3E. Number and Percentage of Single-Year Victimization Totals Flagged

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

All Crime Types and Subgroups 600 157 26.17 85 14.17 163 27.17

Personal Crimes 420 154 36.67 85 20.24 160 38.10

Rape/sexual assault 84 64 76.19 25 29.76 67 79.76

Robbery 84 23 27.38 7 8.33 23 27.38

Aggravated assault 84 9 10.71 4 4.76 9 10.71

Simple assault 84 4 4.76 0 0.00 4 4.76

Personal theft 84 54 64.29 49 58.33 57 67.86

Property Crimes 180 3 1.67 0 0.00 3 1.67

Household burglary 60 0 0.00 0 0.00 0 0.00

Motor vehicle theft 60 3 5.00 0 0.00 3 5.00

Theft 60 0 0.00 0 0.00 0 0.00

Personal Crimes by Subgroups

Sex

Male 20 6 30.00 4 20.00 6 30.00

Female 20 1 5.00 0 0.00 1 5.00

Race/Hispanic origin

White 20 2 10.00 0 0.00 2 10.00

Black 20 9 45.00 7 35.00 9 45.00

Hispanic 20 10 50.00 6 30.00 10 50.00

Other 20 12 60.00 9 45.00 14 70.00

(continued)
33

Exhibit 3E. Number and Percentage of Single-Year Victimization Totals Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Age

12–17 20 10 50.00 5 25.00 10 50.00

18–24 20 7 35.00 3 15.00 7 35.00

25–34 20 8 40.00 5 25.00 8 40.00

35–49 20 8 40.00 3 15.00 8 40.00

50–64 20 7 35.00 8 40.00 9 45.00

65 or older 20 19 95.00 15 75.00 19 95.00

Region

Northeast 20 9 45.00 2 10.00 9 45.00

Midwest 20 8 40.00 2 10.00 9 45.00

South 20 5 25.00 2 10.00 5 25.00

West 20 6 30.00 1 5.00 7 35.00

Rural/urban

Urban 20 1 5.00 0 0.00 1 5.00

Rural 20 12 60.00 6 30.00 12 60.00

MSA status

Yes 20 1 5.00 0 0.00 1 5.00

No 20 12 60.00 7 35.00 12 60.00

(continued)
34

Exhibit 3E. Number and Percentage of Single-Year Victimization Totals Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Property Crimes by Subgroups

Household income

Less than $15,000 12 0 0.00 0 0.00 0 0.00

$15,000–$24,999 12 1 8.33 0 0.00 1 8.33

$25,000–$49,999 12 0 0.00 0 0.00 0 0.00

$50,000–$74,999 12 0 0.00 0 0.00 0 0.00

$75,000 or more 12 0 0.00 0 0.00 0 0.00

Unknown 12 0 0.00 0 0.00 0 0.00

Region

Northeast 12 0 0.00 0 0.00 0 0.00

Midwest 12 0 0.00 0 0.00 0 0.00

South 12 0 0.00 0 0.00 0 0.00

West 12 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 12 0 0.00 0 0.00 0 0.00

Rural 12 0 0.00 0 0.00 0 0.00

MSA status

Yes 12 0 0.00 0 0.00 0 0.00

No 12 2 16.67 0 0.00 2 16.67

Note. MSA, Metropolitan Statistical Area; RSE, relative standard error.
35

Exhibit 3F. Number and Percentage of Pooled-Year Victimization Totals Flagged

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

All Crime Types and Subgroups 300 20 6.67 3 1.00 20 6.67

Personal Crimes 210 20 9.52 3 1.43 20 9.52

Rape/sexual assault 42 14 33.33 2 4.76 14 33.33

Robbery 42 1 2.38 0 0.00 1 2.38

Aggravated assault 42 0 0.00 0 0.00 0 0.00

Simple assault 42 0 0.00 0 0.00 0 0.00

Personal theft 42 5 11.91 1 2.38 5 11.91

Property Crimes 90 0 0.00 0 0.00 0 0.00

Household burglary 30 0 0.00 0 0.00 0 0.00

Motor vehicle theft 30 0 0.00 0 0.00 0 0.00

Theft 30 0 0.00 0 0.00 0 0.00

Personal Crimes by Subgroups

Sex

Male 10 1 10.00 0 0.00 1 10.00

Female 10 0 0.00 0 0.00 0 0.00

Race/Hispanic origin

White 10 0 0.00 0 0.00 0 0.00

Black 10 1 10.00 0 0.00 1 10.00

Hispanic 10 1 10.00 0 0.00 1 10.00

Other 10 3 30.00 1 10.00 3 30.00

(continued)
36

Exhibit 3F. Number and Percentage of Pooled-Year Victimization Totals Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Age

12–17 10 1 10.00 0 0.00 1 10.00

18–24 10 1 10.00 0 0.00 1 10.00

25–34 10 0 0.00 0 0.00 0 0.00

35–49 10 1 10.00 0 0.00 1 10.00

50–64 10 4 40.00 0 0.00 4 40.00

65 or older 10 2 20.00 1 10.00 2 20.00

Region

Northeast 10 0 0.00 0 0.00 0 0.00

Midwest 10 1 10.00 0 0.00 1 10.00

South 10 0 0.00 0 0.00 0 0.00

West 10 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 10 0 0.00 0 0.00 0 0.00

Rural 10 2 20.00 1 10.00 2 20.00

MSA status

Yes 10 0 0.00 0 0.00 0 0.00

No 10 2 20.00 0 0.00 2 20.00

(continued)
37

Exhibit 3F. Number and Percentage of Pooled-Year Victimization Totals Flagged (continued)

Total No. of 

Estimates

Percent RSE > 30% Victimization Count ≤ 10

Percent RSE > 30%

Victimization Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Property Crimes by Subgroups

Household income

Less than $15,000 6 0 0.00 0 0.00 0 0.00

$15,000–$24,999 6 0 0.00 0 0.00 0 0.00

$25,000–$49,999 6 0 0.00 0 0.00 0 0.00

$50,000–$74,999 6 0 0.00 0 0.00 0 0.00

$75,000 or more 6 0 0.00 0 0.00 0 0.00

Unknown 6 0 0.00 0 0.00 0 0.00

Region

Northeast 6 0 0.00 0 0.00 0 0.00

Midwest 6 0 0.00 0 0.00 0 0.00

South 6 0 0.00 0 0.00 0 0.00

West 6 0 0.00 0 0.00 0 0.00

Rural/urban

Urban 6 0 0.00 0 0.00 0 0.00

Rural 6 0 0.00 0 0.00 0 0.00

MSA status

Yes 6 0 0.00 0 0.00 0 0.00

No 6 0 0.00 0 0.00 0 0.00

Note. MSA, Metropolitan Statistical Area; RSE, relative standard error.
38

The recommended rule is to flag as unreliable percentages of victimizations if

 the RSE > 30%, and

o the percentage is ≤ 50%, then use the RSE of log(PP)

o the percentage is > 50%, then use the RSE of log(100−PP); or

 denominator sample size is ≤ 10.

The denominator sample size is recommended here because it corresponds to the sample 

size, or count, of victimizations used in calculating the percentage.

The various flagging rules for victimization percentages were applied to estimates for the 

following situations.

 Type of victimization among all victimizations

 Gender percentages among individual types of violent victimizations

 Racial percentages among individual types of violent victimizations

 Age category percentages among individual types of violent victimizations

 Regional percentages among individual types of violent and property 

victimizations

 Rural/urban percentages among individual types of violent and property 

victimizations

 MSA status percentages among individual types of violent and property 

victimizations

 Income category percentages among individual types of property victimizations

 Time of day category percentages among all types and among individual types of 

violent and property victimizations, by gender, race, and region

 Police notified category percentages among all types and among individual types 

of violent and property victimizations, by gender, race, and region

 Victim relationship category percentages among all types and among individual 

types of violent victimizations, by gender, race, and region

 Weapon involvement category percentages among all types and among individual 

types of violent victimizations, by gender, race, and region

 Time of day category percentages among individual types of property 

victimizations, by MSA status
39

 Police notified category percentages among individual types of property 

victimizations, by MSA status

Exhibit 3G presents the number and percentage of estimates recommended to be flagged 

as unreliable. For single-year victimization percentages, 20.5% of estimates are recommended 

for flagging, with most of these resulting from the RSE criterion. The increased sample sizes 

associated with pooling years of data reduce the estimates recommended for flagging to 7.3%,

with all of these resulting from the RSE criterion. Flagging criteria based upon the RSE of PP, 

rather than the RSE of the log of PP or 100−PP, would have flagged many more estimates—over 

40.1% and 25.7% for single-year and pooled-years estimates, respectively, for a percent RSE 

> 30%.
40

Exhibit 3G. Number and Percentage of Victimization Percentages Flagged

Total No. 

of 

Estimates

RSE of the Log > 30% Denominator Count ≤ 10

RSE of the Log > 30% 

Denominator Count ≤ 10

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Number 

Flagged

Percent 

Flagged

Single-Year Percentages

All Crime Types and Subgroups 4,329 877 20.26 307 7.09 887 20.49

Percentage Range

0.00–0.25 6 0 0.00 0 0.00 0 0.00

0.25–0.50 16 0 0.00 0 0.00 0 0.00

0.50–1.00 96 0 0.00 0 0.00 0 0.00

1.00–5.00 459 38 8.28 1 0.22 38 8.28

5.00–75.00 3,452 759 21.99 255 7.39 764 22.13

75.00+ 300 80 26.67 51 17.00 85 28.33

Pooled-Year Percentages

All Crime Types and Subgroups 2,336 171 7.32 18 0.77 171 7.32

Percentage Range

0.00–0.25 3 0 0 0 0.00 0 0.00

0.25–0.50 25 0 0.00 0 0.00 0 0.00

0.50–1.00 48 0 0.00 0 0.00 0 0.00

1.00–5.00 326 13 3.99 0 0.00 13 3.99

5.00–75.00 1,807 150 8.30 16 0.89 150 8.30

75.00+ 127 8 6.30 2 1.57 8 6.30

Note. RSE, relative standard error.
41

SECTION 4. CONFIDENCE INTERVALS

4.1 Introduction

A CI is used to convey the precision with which a value has been estimated, along with 

the associated level of confidence, expressed as a percentage. For example, a 95% CI is said to 

have a 95% chance of including the true population value under study. A 95% standard normal 

CI for an estimate YY is constructed as YY ± 1.96 × SSSS(YY), where SSSS(YY) is the standard error of YY

given by SSSS(YY) = �VVVVVV(YY). A complication for victimization rates and totals is that the lower 

end point of the CI might be negative, or fall below zero, whereas a rate or total cannot be 

negative. Likewise, a victimization percentage is constrained to be between 0% and 100%,

whereas its standard normal CI could include values outside of this range. To avoid such 

situations, range-preserving transformations are used to first transform an estimate to a different 

scale. A CI is then calculated on the new scale and then transformed back to the original scale of 

the estimate. For victimization rates and totals, a natural logarithm, or log, transformation will be 

considered, and a logit transformation will be considered for percentages.

A log-transform CI for either a rate or total starts by transforming the original value YY to 

ZZ = log(YY) and its standard error to SSSS(ZZ) = SSSS(YY)/YY. Then, the lower and upper end points of 

the 95% CI for ZZ are ZZll = ZZ − 1.96 × SSSS(ZZ) and ZZuu = ZZ + 1.96 × SSSS(ZZ), respectively. 

Transforming back to the scale of either the rate or total, the lower and upper end points of the 

95% CI for YY are YYll = exp(ZZll) and YYuu = exp(ZZuu), respectively. The resulting CI for a rate or 

total will not include any negative values.

The similar process for a victimization percentage is slightly more complicated because it 

must be bounded to be between 0% and 100%. The logit transformation for a proportion will be 

used to achieve this property for the CI of a percentage. First, the percentage, PP, is changed to its 

corresponding proportion as pp = PP/100 along with SSSS(pp) = SSSS(PP)/100. The logit 

transformation of pp is zz = log[pp⁄(1 − pp)] and its standard error is SSSS(zz) = SSSS(pp)⁄[pp(1 − pp)]. 

The lower and upper end points of the 95% CI for zz are zzll = zz − 1.96 × SSSS(zz) and zzuu = zz +

1.96 × SSSS(zz), respectively. Then, the lower and upper end points of the 95% CI for PP are PPll =

100 × eezzll (1 + eezzll ⁄ ) and PPuu = 100 × eezzuu (1 + eezzuu ⁄ ).
42

For the same set of estimates used in Section 3 to study estimate reliability, 95% CIs 

were created using both standard normal and transformed CIs based on the TSL variance 

estimation approach. These are discussed in the following sections.

4.2 Confidence Intervals for Victimization Rates

Exhibit 4A displays a summary of the CIs for single-year and pooled-year victimization 

rates. For single-year rates, the upper portion of Exhibit 4A, 40 out of 630 of the standard normal 

95% CIs include lower end points that were negative, whereas, by design, none of the logtransform CIs include a negative lower end point. In addition, the median lengths of the standard 

normal and the log-transform CIs are very similar. The median percentage of the length of the CI 

to the left of the estimate is presented as a measure of asymmetry of the log-transform CIs. A 

symmetrical CI would have 50% of its length to the left of the estimate. As expected, the smaller 

rates are more asymmetric with a lower percentage to the left of the estimate, but the percentage 

steadily increases toward 50% as the rate increases. The pooled-year rates, the lower portion of 

Exhibit 4A, display a very similar pattern but with far fewer negative lower end points and 

shorter CI lengths because of the increased sample size and precisions associated with pooling 

data across years.

4.3 Confidence Intervals for Victimization Totals

A summary of CIs for single-year and pooled-year victimization totals is given in 

Exhibit 4B. Forty-two out of 630 of the standard normal CIs include a negative lower end point 

for single-year totals, whereas two lower end points are negative for pooled-year totals. As 

before, the log-transform CIs are all positive and are largely symmetrical, with the median 

percentages to the left of the estimate being over 40% for both single-year and pooled-year

totals. Finally, the median lengths of the standard normal and the log-transform CIs are close,

with the log-transform CIs being slightly larger.
43

Exhibit 4A. Confidence Interval Summary for Single-Year and Pooled-Year

Victimization Rates

All 

Rates

Rates

0.00–

0.25

Rates

0.25–

0.50

Rates

0.50–

1.00

Rates

1.00–

5.00

Rates

5.00–

75.00

Rates

75.0+

Single-Year Rates

Number of Confidence Intervals (CIs) 630 16 34 75 229 218 58

Standard Normal Method

Number of negative lower end points 40 3 5 6 16 0 0

Median length of the CI 2.8 0.4 0.6 0.8 2.1 6.5 20.4

Log Transform Method

Number of negative lower end points 0 0 0 0 0 0 0

Median length of the CI 3.1 0.5 0.7 0.9 2.2 6.6 20.5

Median % left of estimate 41.3 23.7 29.3 35.0 39.2 44.0 47.5

Pooled-Year Rates

Number of CIs 316 5 12 42 108 118 31

Standard Normal Method

Number of negative lower end points 2 1 1 0 0 0 0

Median length of the CI 2.0 0.3 0.3 0.6 1.4 3.7 14.3

Log Transform Method

Number of negative lower end points 0 0 0 0 0 0 0

Median length of the CI 2.0 0.4 0.3 0.6 1.4 3.7 14.3

Median % left of estimate 44.7 33.1 37.8 39.8 43.3 46.3 48.4

Exhibit 4B. Confidence Interval Summary for Single-Year and Pooled-Year

Victimization Totals

Single-Year Totals Pooled-Year Totals

Number of Confidence Intervals (CIs) 630 316

Standard Normal Method

Number of negative lower end points 42 2

Median length of the CI 175,302 347,344

Log Transform Method

Number of negative lower end points 0 0

Median length of the CI 187,279 353,294

Median % left of estimate 41.4 44.6
44

4.4 Confidence Intervals for Percentages of Victimizations

Exhibit 4C presents a summary of the CIs for single-year and pooled-year percentages of 

victimizations. The end points of standard normal CIs can be less than 0% or exceed 100% for 

percentages. For single-year percentages, this occurred over 1,000 times for the data considered 

here, with smaller percentages making up the majority of the cases. The logit-transform CIs 

remove this problem while making the median CI lengths very slightly longer than those for the 

standard normal CIs. Considering the median percentage of the length of the CI to the left of the 

estimate, the CIs for percentages with values around 50% are approximately symmetrical, with 

approximately the same amount of the CI to the left of the percentage (49%) as to the right (51% 

= 100%–49%). The CIs become more asymmetrical as the value of the percentages moves

toward 0% or 100%. This is expected as the logit-transformation keeps the CIs bounded between 

0% and 100%. For percentages of victimizations from pooled years, the results are very similar; 

however, both the numbers of CIs extending outside of 0% to 100% and the median lengths of 

the CIs are reduced, which results from the increase in precision when pooling years to increase 

the sample size.
45

Exhibit 4C. Confidence Interval Summary for Single-Year and Pooled-Year Percentages of Victimizations

All 

Percentages

Percentages 

0–5

Percentages 

5–10

Percentages 

10–25

Percentages 

25–75

Percentages 

75–90

Percentages 

90–95

Percentages 

95–100

Single-Year Percentages

Number of Confidence Intervals (CIs) 4,329 577 542 1,022 1,888 206 78 16

Standard Normal Method

Number of end points outside of 0% to 

100%

1,002 355 214 196 137 60 31 9

Median length of the CI 17 5 12 17 27 24 12 8

Logit Transform Method

Number of end points outside of 0% to 

100%

0 0 0 0 0 0 0 0

Median length of the CI 18 7 13 18 27 24 14 9

Median % left of estimate 44.9 23.5 32.3 40.5 49.0 64.2 68.4 76.5

Pooled-Year Percentages

Number of Confidence Intervals 2,336 402 283 500 1,024 80 43 4

Standard Normal Method

Number of end points outside of 0% to 

100%

326 198 56 43 13 10 4 2

Median length of the CI 10 4 7 10 18 13 7 5

Logit Transform Method

Number of end points outside of 0% to 

100%

0 0 0 0 0 0 0 0

Median length of the CI 10 5 7 10 18 13 8 6

Median % left of estimate 46.3 27.9 39.0 43.9 49.3 58.5 60.9 70.5
46

SECTION 5. FINDINGS AND RECOMMENDATIONS

The findings and recommendations of this report are presented in Exhibit 5.

Exhibit 5. Findings and Recommendations for Variance Estimation, Estimate 

Reliability, and Confidence Intervals for the National Crime Victimization 

Survey

Findings Recommendations

Variance Estimation: Generalized Variance Function (GVF) Method

• GVF method works well for situations for which is 

was developed. Evaluated method with rates and 

totals for

– estimates from single years and pooled years; and

– tests of difference between single years and 

pooled years.

• GVF method consistent with Taylor series 

linearization (TSL) method for the situation 

evaluated.

• Method not appropriate for situations not considered 

when creating the GVFs.

• Care must be taken when determining which version 

of the GVFs to use (e.g., based on estimate type, 

inclusion/exclusion of series victimizations).

• Simple to use when a small number of estimates are to 

be processed. Cumbersome when a large number of 

estimates are involved.

• GVF method can be used with victimization 

rates and totals in the following situations:

– for estimates from single years and pooled 

years; and

– for tests of difference between single years 

and pooled years.

• Carefully select the correct GVF version 

corresponding to the application under 

consideration.

Variance Estimation: Taylor Series Linearization (TSL) Method

• Adequate information and variables are available on 

the National Crime Victimization Survey (NCVS)

public use files to support TSL direct variance 

estimation. Evaluated method with rates, totals, and 

percentages for

– estimates from single years and pooled years; and

– tests of differences between single years and 

pooled years (rates and totals only).

• TSL method able to capture complex design features,

including primary sampling unit (PSU) sampling once 

every 10 years.

• Statistical theory supports use of TSL method for a 

wide set of analysis methods beyond rates, totals, and 

percentages.

• TSL method requires merging of person and 

household data files with victimization data files. 

Special merging procedures required.

• TSL method recommended for direct variance 

estimation with NCVS public use data.

• Follow instructions given in Users Guide 

(Shook-Sa, Couzens, & Berzofsky, 2014) to 

implement TSL method due to special 

procedures required when merging person and 

household data files with victimization data 

files to construct analysis files.
47

• TSL method supported by statistical analysis 

packages including SPSS, SUDAAN, SAS, and 

STATA (complex samples package).

(continued)

Exhibit 5. Findings and Recommendations for Variance Estimation, Estimate 

Reliability, and Confidence Intervals for the National Crime Victimization 

Survey (continued)

Findings Recommendations

Variance Estimation: Balanced Repeated Replication (BRR) Method

• Adequate information and variables are available on 

the NCVS public use files to support BRR direct 

variance estimation for single-year estimates. 

Evaluated method for rates and totals from single 

years.

• For single-year estimates, BRR method produced 

variance estimates nearly identical to those of the TSL 

method.

• BRR method requires a very complex data 

configuration to capture changes in PSU sampling 

every 10 years, especially for pooled-year estimates 

and across-year comparisons.

• BRR replicate weights had to be created from pseudostratum and half-sample variables (recently, BRR 

replicate weights have been made available for some 

years).

• BRR method requires merging of person and 

household data files with victimization data files. 

Special merging procedures required.

• BRR method supported by statistical analysis 

packages including SUDAAN, SAS, and STATA. 

BRR not supported by SPSS.

• BRR method not recommended as primary 

direct variance estimation method with NCVS 

public use data.

• BRR method is suited for single-year estimates.

Flagging and Suppression of Unreliable Estimates 

• Criteria from other major national surveys included

– most flagged or suppressed if percent relative 

standard error (RSE) > 30%; and

– most had additional criteria involving minimum 

numerator or denominator sample sizes.

• Evaluated many rules combining criteria across

– maximum allowable RSE and

– minimum sample sizes.

• Percentages of victimizations required special 

treatment using the RSE of the natural logarithm of 

the percentages.

• Rules for victimization rates: flag as unreliable 

if

o RSE > 30% or

o numerator sample size ≤ 10.

• Rules for victimization totals: flag as unreliable 

if

o RSE > 30% or

o count sample size ≤ 10.

• Rules for percentages of victimizations: flag as 

unreliable if

o RSE of log(P) > 30% when P ≤ 50%;

o RSE of log(100−P) > 30% when P > 

50%; or

o denominator sample size ≤ 10.

(continued)
48

Exhibit 5. Findings and Recommendations for Variance Estimation, Estimate 

Reliability, and Confidence Intervals for the National Crime Victimization 

Survey (continued)

Findings Recommendations

Confidence Intervals (CIs)

• CIs for victimization rates and totals:

– standard normal CIs yield negative lower end 

points for some rates and totals;

– log-transform CIs range solely over positive 

values; and

– median lengths of standard normal and logtransform CIs are similar, with the log-transform 

CIs being slightly longer.

• CIs for percentages of victimizations:

– standard normal CIs yield end points outside the 

range of 0% to 100%;

– logit-transform CIs bounded between 0% and 

100%; and

– median lengths of standard normal and logittransform CIs are similar, with the logittransform CIs being slightly longer.

• Log-transform CIs recommended for 

victimization rates and totals.

• Logit-transform CIs recommended for 

percentages of victimizations.
49

SECTION 6. REFERENCES

Ash, S. (2010). Using successive difference replication for estimating variances. In Proceedings 

of the Survey Research Methods Section, American Statistical Association (pp. 3534–

3548). Alexandria, VA: American Statistical Association. Retrieved from 

http://www.amstat.org/sections/srms/proceedings/y2011/Files/302108_67867.pdf

Bureau of Justice Statistics (BJS), Office of Justice Programs, U.S. Department of Justice. 

(2010). National Crime Victimization Survey codebook, 2010 (ICPSR31202-v2). Ann 

Arbor, MI: Inter-university Consortium for Political and Social Research [distributor], 

2012-06-22. Available at http://doi.org/10.3886/ICPSR31202.v2

Cochran, W. G. (1977). Sampling techniques. New York, NY: Wiley.

Fay, R. E., & Train, G. F. (1995). Aspects of survey and model-based postcensal estimation of 

income and poverty characteristics for states and counties. In Proceedings of the 

Government Statistics Section, American Statistical Association (pp. 154–159).

Alexandria, VA: American Statistical Association.

Klein, R. J., Proctor, S. E., Boudreault, M. A., & Turczyn, K. M. (2002). Healthy People 2010 

criteria for data suppression (Healthy People 2010 Statistical Notes No. 24). Atlanta, 

GA: Centers for Disease Control and Prevention. Retrieved from 

http://www.cdc.gov/nchs/data/statnt/statnt24.pdf

Lauritsen, J. L., Owens, J. G., Planty, M. R., & Truman, J. L. (2012). Methods for counting highfrequency repeat victimizations in the National Crime Victimization Survey (NCJ No. 

237308). Washington, DC: Bureau of Justice Statistics. Retrieved from 

http://www.bjs.gov/content/pub/pdf/mchfrv.pdf

Lumley, T. (2008). Balanced repeated replication (BRR). In P. J. Lavrakas (Ed.), Encyclopedia 

of survey research methods. Newbury Park, CA: Sage.

Särndal, C. E., Swensson, B., & Wretman, J. (1992), Model assisted survey sampling. New York, 

NY: Springer-Verlag.

Shook-Sa, B., Couzens, G. L, & Berzofsky, M. (2014). National Crime Victimization Survey 

(NCVS) direct variance user’s guide. Prepared for the Bureau of Justice Statistics, 

Washington, DC.

Substance Abuse and Mental Health Services Administration. (2011). Results from the 2010 

National Survey on Drug Use and Health: Summary of national findings (NSDUH Series 

H-41, HHS Publication No. (SMA) 11-4658). Rockville, MD: Author.

Williams, R. L. (2000). A note on robust variance estimation for cluster-correlated data. 

Biometrics, 56, 218–219.
50

Williams, R. L. (2008). Taylor series linearization. In P. J. Lavrakas (Ed.), Encyclopedia of 

survey research methods. Newbury Park, CA: Sage.

Wolter, K. M. (1985). Introduction to variance estimation. New York, NY: Springer-Verlag.
The Bureau of Justice Statistics of the U.S. Department of Justice is the principal 

federal agency responsible for measuring crime, criminal victimization, criminal 

offenders, victims of crime, correlates of crime, and the operation of criminal 

and civil justice systems at the federal, state, tribal, and local levels. BJS collects, 

analyzes, and disseminates reliable and valid statistics on crime and justice 

systems in the United States, supports improvements to state and local criminal 

justice information systems, and participates with national and international 

organizations to develop and recommend national standards for justice statistics. 

William J. Sabol is director.

Office of Justice Programs

Innovation • Partnerships • Safer Neighborhoods

www.ojp.usdoj.gov

NCJ249242