The views presented in this paper are those of the author(s) and do not represent the official views of any Federal Government agency/department or Westat.

2018 Federal Committee on Statistical Methodology • March 7, 2018

Bureau of Justice Statistics

Implications of Repeated Sampling in a 

Crime Survey

J. Michael Brick, Westat

Grace Kena, BJS

Pam Broene, Westat
Outline

• Describe NCVS Companion Survey (NCVS-CS)

– The Bureau of Justice Statistics’ National Crime 

Victimization Survey (NCVS)—the nation’s primary source of 

information on criminal victimization

– CS developed as a low-cost approach for producing localarea victimization and community/policing estimates 

• Evaluate effect of overlap on

– Response rates

– Outcomes

• Examine effect of overlap on estimates of change

• Discuss conclusions on usefulness of overlap

2018 FCSM 2
Reasons for Repeated Sampling

• Key reasons for doing panel or longitudinal study include:

– Examine spells, durations, and gross changes over time

– Reduce sampling error for estimates of change over time

– Reduce sampling costs

• Sampling error of change is reduced when retained units 

exhibit positive correlation

• The NCVS retains sample addresses for 7 rounds; the 

NCVS-CS retained address over 2 years

2018 FCSM 3
NCVS–CS Design

• Low-cost probability sample of households designed to 

estimate crime victimization and community views on 

policing and safety

• Conducted in 2015 (Year 1) and 2016 (Year 2) by mail in 

40 largest metropolitan areas (CBSAs)

• Randomized block design with experiments: overlap, 

instrument (ILS/PLS), incentive amount, mailing method

– ILS=Incident-level survey; PLS=Person-level survey

– Also, Form A (community questions first); Form B (last)

• Focus of this presentation is on the overlap experiment 

where some addresses were sampled for both years

2018 FCSM 4
Overlap Design

• Overlap experiment restricted to those with common 

experimental condition in Year 1=Form A (50 percent of 

all Year 1 addresses)

• Randomly assigned 1/2 of cases (25 percent of all Year 1 

addresses) to be retained regardless of Year 1 response

• All retained cases kept Year 1 instrument (ILS or PLS)

• Randomized within each CBSA so that overlap equally 

distributed by geography

2018 FCSM 5
Response Rates

• Year 1 AAPOR RR1 = 44.2% (n=229,475)

• Year 2 AAPOR RR1 = 35.6% (n=217,250)

– The lower rate in Year 2 is due to the testing of various cost 

reducing methods (e.g. lower incentives and mail protocols)

• Overlap treatment response rates for Year 2 

– No overlap RR1 = 37.3% (n=146,225)

– Overlap RR1 = 33.2% (n=48,655)

• Overlap depressed response rates in Year 2 by 4.1

percentage points (statistically significant)

2018 FCSM 6
Overlap Effect by Reporting Crime in Year 1

• Condition on those households responding in Year 1, 

examine Touched By Crime (TBC) reporting differences

– TBC=HH or 1+ person experienced 1+ crime in prior year

• Households who were TBC in Year 1 were less likely to 

respond to the CS in Year 2 than those not TBC

– Property crime: 7 percentage point lower Year 2 response 

rate for TBC than non-TBC

– Violent crime: 14 percentage point lower Year 2 response 

rate for TBC than non-TBC

– Small sample sizes (TBC is rare) limit power of this analysis, 

but in 80 percent of the CBSAs households, TBC in Year 1 

had lower Year 2 response rates

2018 FCSM 7
Overlap Effect on Reporting in Year 2

• The instrument structure also affected outcomes so 

results reported separately by instrument (ILS/PLS)

• Analyzed outcomes using paired t-test where difference 

between overlap and non-overlap sample computed for 

both instruments in each CBSA

• Computed differences for

– Crime measures: TBC Property, TBC Violent, and TBC 

Serious Violent crime

– Community measures: How safe is community, How often 

does fear of crime prevent you from doing things, Rate job 

police are doing 

2018 FCSM 8
Paired t-Test for TBC by Overlap in Year 2

TBC Difference

Incident-Level Survey

Property -2.5*

Violent -0.5*

Serious Violent -0.4*

Person-Level Survey

Property -2.8*

Violent -0.5*

Serious Violent -0.4*

2018 FCSM 9

Legend: Property=TBC Property Crime; 

Violent=TBC Violent Crime; 

Serious Violent=TBC Serious Violent Crime

p* <0.05.
Paired t-Test for Community Items by Overlap in 

Year 2

Item Difference

Incident-Level Survey

Safe 1.1*

Fear 1.2~

Rate 1.1~

Person-Level Survey

Safe 1.5*

Fear 1.1~

Rate 1.1~

2018 FCSM 10

Legend: Safe=How safe is community; 

Fear=How often does fear of crime prevent you from doing things; 

Rate=Rate job police are doing

p* <0.05.

p~ <0.1.
Overlap Effect on Estimates

• Compared to new sampled addresses, retained addresses 

generally reported: 

– Fewer victimizations

– Feeling safer 

– Higher ratings of local police actions

• These reporting patterns are consistent because people 

and households that have not been victimized tend to 

rate police higher than those who have been victimized

2018 FCSM 11
Estimates of Change

• Computed difference from Year 1 to Year 2 by CBSA and 

averaged over the CBSAs to take advantage of the 

randomized block design

• Again, the instruments performed differently so we 

present the data separately by the ILS/PLS

• Estimates of change presented simply to show that 

changes between years were small; change estimates 

were not statistically significant

2018 FCSM 12
Average Year 1 to Year 2 Change by Instrument

TBC Change

Incident-Level Survey

Property 0.5

Violent 0.3

Serious Violent 0.3

Person-Level Survey

Property -2.0

Violent 0.7

Serious Violent 0.2

2018 FCSM 13

Item Change

Incident-Level Survey

Safe 0.1

Fear 0.1

Rate 3.7

Person-Level Survey

Safe -0.1

Fear -1.0

Rate 4.9

Legend: Property=TBC Property Crime; 

Violent=TBC Violent Crime; 

Serious Violent=TBC Serious Violent Crime

Safe=How safe is community;

Fear=How often does fear of crime prevent you from doing things;

Rate=Rate job police are doing
Correlations—Touched by Crime and Community Items

TBC Correlation P value

Incident-Level Survey

Property 0.81 <.01

Violent 0.17 .30

Serious Violent 0.13 .41

Person-Level Survey

Property 0.89 <.01

Violent -0.0 .98

Serious Violent 0.02 .93

2018 FCSM 14

Comm/Pol Item Correlation P value

Incident-Level Survey

Safe 0.73 <.01

Fear 0.81 <.01

Rate 0.45 <.01

Person-Level Survey

Safe 0.74 <.01

Fear 0.88 <.01

Rare 0.54 <.01

Legend: Property=TBC Property Crime; 

Violent=TBC Violent Crime; 

Serious Violent=TBC Serious Violent Crime

Safe=How safe is community;

Fear=How often does fear of crime prevent you from doing things;

Rate=Rate job police are doing
Variance of Estimated Change Due to Overlap

• Effect of overlap computed using a ratio of variances

– Variance of estimate based on overlap sample to expected 

variance if the samples were independent

• Ratios close to 1 indicate that the overlap did not 

improve the precision of the estimate very much. Small 

ratios imply overlap reduced variance of estimated 

change.

2018 FCSM 15
Variance Ratios of Change Estimates

25th percentile Median 75th percentile

TBC

Property 0.91 0.99 1.02

Violent 0.96 0.99 1.04

Serious Violent 0.96 1.00 1.02

Community

Safe 0.97 1.04 1.10

Fear 0.92 0.99 1.07

Rate 0.83 1.01 1.07

2018 FCSM 16

• Most ratios are very close to 1, indicating that:

• The overlap did not improve the precision of the estimates very much.

• There was generally little variation within and across measures.
Explaining the Effect of the Overlap

• The table shows that there is no advantage to retaining 

sample addresses for estimating TBC or even the 

community items. This result, especially for the 

community items, may be surprising given the relatively 

high correlation.

• The main explanation is that the expected variance for 

the independent sample is based on a larger sample 

because the response rate for the new addresses is 

considerably larger (14%) and this more than 

compensates for the positive correlation.

2018 FCSM 17
Summary

• Retaining addresses in the sample from Year 1 to Year 2

– Significantly reduces response rates

– Differentially reduces responses from those who reported a 

victimization in Year 1

– Results in lower reported victimizations in Year 2

– Does not reduce the variance of estimates of change

• Retaining addresses in the sample for the NCVS-CS is not 

recommended

2018 FCSM 18
The views presented in this paper are those of the author(s) and do not represent the official views of any Federal Government agency/department or Westat.

2018 Federal Committee on Statistical Methodology • March 7, 2018

Bureau of Justice Statistics

Thanks

Contact: Grace.Kena @usdoj.gov