Page 1 GAO-03-867R COPS Evaluation 

United States General Accounting Office

Washington, DC 20548 

June 13, 2003 

The Honorable F. James Sensenbrenner 

Chairman 

Committee on the Judiciary 

House of Representatives 

Subject: Technical Assessment of Zhao and Thurman's 2001 Evaluation of the 

Effects of COPS Grants on Crime 

Dear Mr. Chairman: 

Community Oriented Policing Services (COPS) is a federal public safety program 

whose goals are to add officer positions to the streets of communities nationwide and 

to promote community policing. Since the program’s inception in 1994, local law 

enforcement agencies have received billions of dollars in grants to hire additional 

officers, acquire technology and civilian personnel, and implement innovative crimeprevention programs. To receive COPS grants, agencies are expected to implement or 

enhance community policing strategies illustrating community partnerships, problem 

solving, and organizational commitment. Given the large expenditures of funds, it is 

important for policy makers, among others, to have sound information on the 

effectiveness of the COPS program in reducing crime. You asked us to review one 

evaluation of the effectiveness of the COPS program—by Zhao and Thurman1—and to 

render an assessment of its quality. In this report, we provide information on the 

extent to which this particular study’s conclusions are supported by the data the 

researchers used and the analyses they conducted. GAO statisticians and 

methodology specialists reviewed the study using standard and widely accepted 

statistical and social science research principles. 

Our assessment of Zhao and Thurman’s work cannot be construed to be an 

assessment of the COPS program itself. Since we have not reviewed the quality of any 

other COPS evaluation or conducted an independent evaluation of the program, we 

have no basis to judge whether or not the program has been effective in achieving its 

stated goals. It is also important to note that these types of aggregate level analyses 

that are intended to assess program effectiveness are extremely difficult to execute 

successfully, in part, because direct measures of important variables are not always 

available. 

 

1

Zhao, J. and Thurman, Q. A National Evaluation of the Effect of COPS Grants on Crime from 1994 

to 1999 (Dec. 2001). 
Page 2 GAO-03-867R COPS Evaluation

We conducted our review of Zhao and Thurman’s study during a 3-week period in 

May 2003. In addition to reviewing Zhao and Thurman’s December 2001 report, we 

reviewed a November 2002 journal article by Zhao, Scheider, and Thurman based on 

the same study,2 reviewed a May 2003 draft of an updated COPS study by the same 

authors, and discussed data and statistical issues with these researchers in a 

telephone call on May 27. In this report, we focus the majority of our comments on 

Zhao et al.’s earlier COPS study (reported in December 2001 and November 2002). We 

discuss differences between the earlier study and the May 2003 follow-up study in a 

section at the end of this report. For ease of presentation, we refer to their original 

work as the “2001 study.” 

Background 

The Public Safety Partnership and Community Policing Act of 19943

 authorized $8.8 

billion in grants to be awarded to law enforcement agencies for fiscal years 1995 to 

2000. Focused on crime-prevention, the act required, among other things, that half the 

grants go to law enforcement agencies serving populations of 150,000 or less. The act 

also required that grantees not supplant state and local funding, but rather use the 

federal funds for additional law enforcement beyond what would have been available 

without a grant. The Attorney General created the Office of Community Oriented 

Policing Services to administer the grant programs and advance community policing 

across the country. 

The COPS office is tasked with promoting community policing through a variety of 

types of grants, including: 

• Hiring grants, which are used to fund the hiring of additional police officers. 

Through its Universal Hiring Program, the COPS program provides funding 

directly to local, state, and tribal jurisdictions. The funding provides up to 75 

percent of the salaries and benefits for new officers for 3 years up to a maximum 

of $75,000 per officer. According to the COPS Office, 71,192 officers were funded 

and 63,592 officers were hired through hiring grants as of July 26, 2002. The COPS 

Office estimated that hiring grant awards totaled about $5.6 billion as of June 3, 

2003. 

• Making Officer Redeployment Effective (MORE) grants, which are used to fund 

up to 75 percent of the total cost of acquiring new technologies and equipment 

and the hiring of civilians for 1 year. These are intended to allow police to spend 

more time patrolling the streets instead of on administrative and support tasks. 

According to the COPS Office, 24,436 full-time equivalent staff were redeployed 

through MORE grants as of July 26, 2002. The COPS Office estimated that MORE 

grant awards totaled about $1.3 billion as of June 3, 2003. 

• Innovative grants, which are used to promote innovative approaches to solving 

crime in specific areas such as domestic violence and drug abuse. The COPS 

 

2

Zhao, J., Scheider, C. and Thurman, Q. Funding Community Policing to Reduce Crime: Have COPS 

Grants Made a Difference? Journal of Criminology & Public Policy, Nov. 2002 (vol. 2, no. 1). 3

P.L. 103-322. 
Page 3 GAO-03-867R COPS Evaluation

Office estimated that innovative grant awards totaled $820 million as of June 3, 

2003. 

Results in Brief 

Our review of the 2001 study on the effects of COPS grants on crime rates indicated 

that the results of their study should be viewed as inconclusive. We believe that the 

study’s limitations in data and methods are significant and preclude meaningful 

interpretation of the results. We cannot agree with Zhao et al. that their 2001 study 

shows that some COPS grants (hiring and innovative) significantly reduced crime 

because, among other things, important variables were omitted from their analyses, 

the analytic models were misspecified, and the sample of cities included in the study 

was limited. Further, we have concerns about the use of outdated census data for 

control variables. Aside from concerns about data and methods, we question whether 

the statistically significant crime reductions that Zhao et al. found are significant in a 

practical sense. 

While we cannot agree with the Zhao et al.’s conclusions, we also cannot say that 

COPS grants are ineffective in reducing crime. A program’s effects and researchers’ 

ability to design studies that will accurately measure those effects are two different 

things. Other studies, which we have not reviewed, may have taken a more rigorous 

approach to assessing the effects of COPS grants on crime. We believe that a more 

rigorous study would incorporate, among other things, more reliable, valid, and 

complete measures; a more complete and generalizable sample of cities; and wellspecified analytic models. 

In written comments on a draft of this report, the Department of Justice’s COPS 

Office and Zhao and Thurman generally disagreed with our findings. The comments 

reflected the view that our standards for critiquing Zhao et al.’s work were too 

stringent, that we were incorrect in concluding that their statistical models were 

misspecified, and that the statistical controls incorporated into their analytic models 

were sufficient to account for the types of missing data we identified as limitations of 

the study. In our response, we address why we continue to believe that these 

limitations render the findings of this particular study inconclusive. 

Summary of Analysis and Results of the 2001 Study 

The 2001 study presented a statistical analysis of the effects of three types of grants—

hiring, MORE, and innovation—on the reported rates of violent and property crimes 

over a 5-year period across 6,100 U.S. cities that received COPS grants. The analysis, 

which looked separately at cities with populations greater than 10,000 and those with 

populations less than 10,000, sought to determine how the reported crime rates 

varied as a function of the amount of COPS funds received. 

The variables used in the 2001 study are presented in table 1, along with the averages 

and standard deviations for these variables across all cities included in the analysis. 
Page 4 GAO-03-867R COPS Evaluation

Table 1: Averages Across All Cities from 1994 to 1999 

Average Standard deviation

Dependent variables (1995-1999) 

Violent crime rate (per 100,000 population) 769.63 674.50 

Property crime rate (per 100,000 population) 5,016.39 2,820.74 

Independent variables (1994–1998) 

Hiring grants (per resident) $2.38 3.72 

Innovative grants (per resident) $0.42 2.45 

MORE grants (per resident) $0.65 1.45 

Demographic control variables 

% unemployment (1994-1998) 4.97 2.17 

% minority (1990 census) 30.40 23.32 

% single parent households (1990 census) 10.59 4.09 

% young people ages 15-24 (1990 census) 15.43 4.59 

% home owners (1990 census) 56.92 14.62 

% people in same household for 5 or more years (1990 census) 50.66 10.03 

Note: Zhao et al. used weighted averages to estimate the means of COPS grants and control variables. 

Source: Zhao et al., December 2001 and November 2002. 

Zhao et al. found that hiring grants significantly reduced reported violent and 

property crimes in larger cities, but significantly increased those rates in smaller 

cities. They speculated that the addition of police officers in smaller cities could 

produce an increase in reported crime because, among other things, the increased 

interaction between police and the community can help residents feel more 

comfortable and willing to report crimes. Innovative grants also significantly reduced 

the reported violent and property crime rates in larger cities, but had no significant 

effect in smaller cities. MORE grants had no discernable effect in larger cities, or on 

reported violent crimes in smaller cities, but they significantly increased the rates of 

reported property crimes in the smaller cities. Zhao et al. concluded that innovative 

programs, which are targeted at specific crime problems or jurisdictions, had the 

strongest effect on reducing reported crime rates. They also observed that “crime 

reduction in the United States is not a unitary phenomenon” in light of the different 

effects found in large versus smaller cities. 

Our Review Indicated Several Problems with the 2001 Study 

Our review revealed several problems with the 2001 study that cast doubt on the 

validity of the conclusions about the effectiveness of COPS grants. The problems we 

identified pertain to Zhao et al.’s interpretation of their findings, omission of 

important variables from the analysis, misspecifications in the analytic models used, 

and sample selection issues. We also had some concerns about the outdated nature of 

census data used as control variables in the 2001 study. 

The Meaning of the Study’s Findings Can Be Interpreted Differently

The finding that COPS grants exerted different effects on crime patterns in large 

versus small cities led the researchers to observe that crime reduction is not a unitary 

phenomenon. While this may be the case, one can also conclude that the study’s 

findings are equivocal, inconsistent, and inconclusive. 

Further, while the crime-reducing effects that Zhao et al. found for hiring and 

innovative grants may have been statistically significant, they could also be 
Page 5 GAO-03-867R COPS Evaluation

characterized as quite small in a practical sense.4 Table 2 demonstrates this point by 

presenting a summary of Zhao et al.’s estimates. 

Table 2: Estimates of the Effects of Three Types of COPS Grants, from Zhao et al. (2001) 

City 

type 

Crime 

type 

Hiring 

Grants 

Innovative 

grants 

MORE 

grants 

>10,000 Violent -5.26* -12.93* -0.11 

 Property -21.63* -45.53* -1.52 

1,000 - 10,000 Violent 0.83* 1.06 2.48 

 Property 8.97* 11.98 31.20* 

All cities 1000+ Violent -1.86 -12.26* 0.28 

 Property -10.44 -43.85* -0.28 

Note: An asterisk (*) denotes that the estimated effect was statistically significant. 

Source: GAO summary of Zhao et al. 2001 data. 

The coefficients in table 2 indicate how much each grant dollar spent per person in 

each city affected the rates of reported violent and property crimes; in other words, 

how much of a change in the reported violent and property crime rates we might 

expect if funding were increased by one dollar per resident. As shown in table 1, the 

average annual COPS innovative grant across all cities amounted to $0.42 per person, 

and the average rates of reported violent and property crimes, respectively, were 

about 770 and 5,016 per 100,000. These coefficients imply that if COPS funding in 

larger cities for innovative grants were doubled (from $0.42 to $0.84 per person), we 

would expect the violent crime rate to go down by 0.7 of 1 percent (from 770 to 765 

per 100,000).5

 We would expect the reported property crime rate to go down by 0.4 of 

1 percent (from 5,016 to 4,997 per 100,000).6

 As small as the effects are, there are 

reasons to question whether they accurately represent the expected returns on such 

an investment, and these reasons are listed below in general order of importance. 

Important Variables Were Omitted from the Analysis

While dummy variables were used in the 2001 study to control for unmeasured 

differences across counties, the only city-level variables in the analysis that were 

measured and explicitly controlled in the models of estimated COPS grant effects 

were (1) the 1994 crime rate and (2) the six demographic variables shown in table 1. 

Most conspicuously absent from these models is a measure of expenditures on police 

that were not derived from COPS grants. The researchers told us they did not include 

 

4

Statistical significance means that the observed effect does not result from chance alone. The number 

of observations in a sample can be an important determinant of statistical significance, with larger 

sample sizes frequently being associated with statistically significant findings. Zhao et al.’s 2001 study 

consisted of 36,605 observations, making it possible that statistically significant effects could have 

been found even when they were small on a practical level. 5

This is calculated as follows: From table 2, we see that in cities larger than 10,000, each dollar of 

innovative grant funding was associated with a decrease of 12.93 violent crimes. $0.42 is 42 percent of 

1 dollar, and 42 percent of 12.93 crimes equals 5.4. This represents the decrease in the expected crime 

rate as innovative grant funding increased by $0.42 per person. If the violent crime rate were 770 per 

100,000 population, doubling the $0.42 innovative grant expenditure per person would reduce the 

violent crime rate by 5, or to about 765 per 100,000 population.

6

The mean offered in Zhao and Thurman is a weighted average for all cities and only approximates the 

mean for large cities. Because of that, and the severe skew in the distribution of average grant amounts 

across cities (note the standard deviations in table 1), this may not be a very accurate way to estimate 

the effect size. The skew in the distribution of grant amounts also suggests that it might have been 

preferable to transform (using logarithms) those amounts prior to the analyses. 
Page 6 GAO-03-867R COPS Evaluation

non-COPS funded police expenditures because such data are not available. Because 

the COPS program supports only a portion of police agency budgets, however, we 

believe the absence of any control for state and local expenditure to be a serious 

weakness. 

Police departments that received COPS grants may have also received grants from 

other programs (such as Byrne grants). These amounts could be correlated with

COPS funding amounts. For example, if a department is proficient in getting COPS

funding, it may be proficient in getting other funding, as well. Without separating

COPS funding from other types of funding that police agencies receive, we cannot be

sure how much of an effect COPS grants by themselves have on crime reduction.

The study also lacked any measure of city size beyond the dichotomy (i.e., population 

smaller or larger than 10,000) used to split the sample of cities prior to model 

estimation. Other omitted measures include such socioeconomic variables as per 

capita income and percent male. County dummy variables controlled for some of the 

problems associated with omitted variables, but they would not control effectively 

for variables that differed across cities within counties, or variables that changed 

within counties over time. For example, if state and local expenditures on police 

varied across cities in a given county, using dummy variables to represent counties 

would not take these differences or changes into account in estimating the 

independent effect of COPS grants. 

Misspecifications in the Analytic Models

The models employed in Zhao et al.’s analyses are two-factor fixed effects models 

that employ 2,674 dummy variables representing the counties and 5 dummy variables 

representing the years included in the analysis. These dummy variables controlled for 

unmeasured variability across counties and over time, and they supplemented the 

controls for prior rate of crime and the 6 demographic variables described above. 

These models and the estimation procedures they involve are fairly sophisticated, but 

since the data on crime rates and COPS funds were measured at the city level, we 

believe that unmeasured variability would have been more effectively controlled had 

dummy variables been used to distinguish cities, instead of the counties in which the 

cities were located.7 With dummy variables representing counties, any unmeasured 

and systematic variability across cities within the same county remained 

uncontrolled and a potential source of bias in the parameters representing the effects 

of the COPS grants estimated in the models. 

Sample Selection Limited

Zhao et al.’s analysis is focused only on COPS program grants used to fund local city 

police departments. Their report indicates that other law enforcement agencies, such 

as state and county police agencies; sheriffs’ offices; campus police; and special 

purpose law enforcement agencies such as court, forest, and park police, among 

others, were excluded from their study. Since these other agencies accounted for 

 

7

A footnote in the 2001 study indicates that the researchers conducted initial analyses using city dummy 

variables. However, they ultimately decided to use county dummy variables, and all the report findings 

are derived from statistical models that included county rather than city dummy variables. 
Page 7 GAO-03-867R COPS Evaluation

4,891 (or 40 percent) of the 12,070 law enforcement agencies receiving COPS grant 

awards from 1994 to 1998, Zhao et al.’s study omitted a large portion of COPS grant 

recipients. Further, there is likely to be considerable overlap across jurisdictions 

receiving COPS grants (cities within counties, campus police within city 

jurisdictions).8

According to Zhao et al., the sample of cities included in their study represented a 

subset of 6,100 of the 7,179 cities whose local city police departments received COPS 

grants at some point during the period from 1994 to 1998.9 The researchers deleted 

535 cities with populations less than 1,000, and 544 cities that lacked Uniform Crime 

Reports (UCR) data.10,11 Four states (Delaware, Illinois, Kansas, and Montana) 

contributed only 8 cities between them owing to missing UCR data. These omissions 

may have affected the study’s results. Of greater concern, however, is the omission 

of the potentially large number of cities that received no COPS funding at all. We 

believe that cities with no COPS funding should have been included in the analyses in 

order avoid sample selection problems and ensure that the results were generalizable 

across all cities.12

 

Concerns about Measures of Demographic Variables. 

While the rates of violent crimes and property crimes were measured and allowed to 

vary in each of the 5 years from 1994 to 1998, in the 2001 study at least 5 of the 6 

demographic variables were derived from the 1990 census and fixed at their 1990 

levels. We believe the 1990 figures would be a poor basis for estimates because in 

many cities, the demographic characteristics of residents in 1990 would be expected 

to be quite different from those in the mid- to late-1990s; and in all cities, these timeinvariant estimates would fail to account for the significant demographic changes 

 

8

 For example, if the city of College Park, MD, received a COPS grant and the University of Maryland 

campus police (located in College Park) received a separate COPS grant, their joint impact on the 

city’s crime rates would not be included in this analysis. 

9

 In the analysis, the crime rates from 1995 to 1999 were intentionally lagged a year to allow these 

agencies to receive and deploy these funds. 

10 UCR is a nationwide database of police statistics consisting of crime data voluntarily reported to the 

Federal Bureau of Investigation by nearly 17,000 city, county, and state law enforcement agencies. 

UCR data form the basis for a Crime Index, which is used to gauge fluctuations in the nation’s overall 

volume and rate of crime. The offenses included in the "violent crime" category are murder and 

nonnegligent manslaughter, forcible rape, robbery, and aggravated assault. The offenses included in 

the "property crime" category are burglary, larceny-theft, and motor vehicle theft, and arson.

11 In personal discussions with the researchers, we learned that their 2001 published study contained 

an error related to missing data. Specifically, the researchers had intended to eliminate cities from 

their analysis if crime data were missing for even a single month of the year. However, the dataset they 

obtained did not uniformly distinguish between missing data and "zero" reported crimes. In those 

cases, the analysis would have produced an underestimate of the 12-month crime rate. After publishing 

their results, the researchers corrected these data errors and reanalyzed the dataset. They told us that 

the revised results did not differ substantially from those published. Time limitations prevented us 

from assessing the revised results. 

12In a November 2002 publication in the Journal of Criminology and Public Policy, Zhao et al. 

explained that their analyses omitted cities without COPS grants because of concern that including 

these cities would produce a downward bias in their estimation of COPS program effects. They said 

this is because crime was decreasing across the board between 1994 and 1998 in cities with and 

without COPS grants. We disagree with their rationale. Since Zhao and Thurman controlled for the 

baseline rate of crime by including the 1994 rate in their model as a control variable, cities with COPS 

grants would presumably have a higher rate of decrease than cities without COPS grants. We continue 

to believe that Zhao and Thurman’s estimates of COPS program effects were biased as a result of 

omitting cities that did not receive COPS grants. 
Page 8 GAO-03-867R COPS Evaluation

that may have occurred over time. It was not entirely clear to us how the 

unemployment data derived from the U.S. Department of Labor Statistics for the 

years 1994 to 1998 were used in these models. However, it too represented a 

potentially poor measure of unemployment in many cities. This is because data are 

not available for cities with populations less than 25,000, and county-level rates were 

used for those cities instead. 

Comments on Zhao et al.’s Draft Updated COPS Study 

Our previous comments pertain to the unpublished 2001 study by Zhao and Thurman 

and the 2002 publication by Zhao, Scheider, and Thurman which resulted from the 

study and which was virtually identical to the unpublished study in terms of the 

primary results that were reported. That study, as we noted previously, relied on data 

from 6,100 cities for which COPS grant data for the years 1994 to 1998, and UCR 

crime data for the years 1994 to 1999, could be obtained. After reviewing that work, 

we received a draft updated report from those authors that re-estimated the effects of 

COPS grants on crime rates using data from an additional year (e.g., COPS grant data 

for 1994-1999 and UCR data for 1994-2000)13 and models that incorporated updated 

2000 census data and allowed the demographic characteristics to vary over time. 

While these newer estimates, like those in the 2001 and 2002 reports, were derived 

from models that used county dummy variables, we also received from the 

researchers additional information that showed how results compared when they 

used dummy variables representing cities in place of the county dummy variables. 

These updated results are shown in table 3, along with the results from the 

researchers’ prior study. The researchers have asserted, both in the draft updated 

report and in their conversations with us, that these updated results are largely 

consistent with the previously published results, and in a general sense we agree with 

this. That is, with or without the newer data, regardless whether demographic factors 

are allowed to vary, and regardless whether county or city or dummy variables are 

used, both studies found (1) no evidence that COPS grants have diminished the crime 

rates in cities with populations less than 10,000, and (2) some evidence that they have 

done so in larger cities. Apart from this general observation, however, the results of 

the two studies are inconsistent in that the size and significance of some of the 

estimated effects of COPS grants differed under alternative specifications. For 

example, when updated data and the time varying covariates were used, the 

estimated effects of innovative grants on violent and property crimes in large cities 

declined in size to less than half of the prior estimates, while the effects of MORE 

grants increased more than 10-fold, and became statistically significant in the case of 

property crimes.14

 

 

13One difference in the crime rates analyzed in the two studies was that arson was included as a 

property crime in the newer study, but not in the 2001 study. 

14The authors provided us with additional information from their follow-up study on the analytic 

results obtained when they used dummy variables to represent cities instead of counties. They found 

that in large cities, the estimated effects of hiring grants on violent crimes doubled, the estimated 

effects of MORE grants doubled and became statistically significant, and the effect of innovative grants 

became statistically not significant. The effect of MORE grants on property crimes remained 

significant in large cities when city dummy variables were used, but diminished to half the size that 

was estimated by a model that used county dummy variables. 
Page 9 GAO-03-867R COPS Evaluation

Since these newer results have not been finalized, it is premature for us to make a 

final determination of their validity and usefulness. The researchers are to be 

commended for the considerable effort they made to determine how reliable and 

robust the estimated effects of the different COPS grants were over time, and under 

alternative specifications. Nonetheless, the newer study that we reviewed had some 

of the same limitations as the 2001 study. Specifically, the newer study (1) omitted 

important variables, including measures of expenditures on police apart from COPS 

grants, (2) omitted a large number of cities that did not receive COPS grants, and (3) 

did not control for the effect of city size on crime in a more refined fashion than 

dichotomizing city populations. Our review of the results of the newer analyses has 

not fundamentally altered our view that the estimated effects of COPS grants on 

reported violent and property crimes were small in a practical sense. Again, it is 

important to note that this does not imply that COPS grants do not have positive 

effects in reducing crime; only that it is hard to reach firm conclusions about their 

effects from the particular studies we reviewed. Our technical assessment of Zhao et 

al.’s work is not a commentary on the effectiveness of the COPS program. 

Table 3: Zhao et al.’s Estimates of the Effects of Three Types of COPS Grants with Dummy Variables 

Representing Counties in (a) 2001 Study Using 1994-1999 Data and (b) Draft Updated Study Using 1995-

2000 Data 

City 

type 

Crime 

type 

COPS study Hiring 

grants 

Innovative 

grants 

MORE 

grants 

>10,000 Violent 2001 study -5.26* -12.93* -0.11 

 Draft updated study -5.49* -5.31* -2.00 

 

 Property 2001 study -21.63* -45.53* -1.52 

 Draft updated study -25.22* -20.65* -21.47* 

1,000 - 10,000 Violent 2001 study 0.83* 1.06 2.48 

 Draft updated study 1.47* 0.60 2.92 

 Property 2001 study 8.97* 11.98 31.20* 

 Draft updated study 7.91* 1.30 30.51* 

Note: An asterisk (*) denotes that the estimated effect was statistically significant. 

Source: GAO summary of Zhao et al.’s 2001 and updated studies. 

Agency Comments and Our Evaluation 

The Acting Deputy Director of the COPS Office and Professors Zhao and Thurman 

provided us with written comments on a draft of this report. Their comments 

contained a number of points that disagreed with the limitations we identified in our 

assessment. The comments reflected the view that we (1) applied an overly stringent 

standard to the study’s design and failed to consider the fact that this study was 

better and more comprehensive than previous research on the subject; (2) were 

incorrect in concluding that their statistical models were misspecified and did not 

control for the effect of missing police expenditure data; (3) were ill-advised in 

stating that including data on cities’ access to grants other than COPS grants would 

have improved the estimates of COPS grant effects; (4) were ill-advised in stating that 

including data on such socioeconomic variables as percentage of the population that 

is male would have improved the estimates of COPS grant effects; (5) were incorrect 

in stating that including data on COPS-funded jurisdictions within cities, such as 
Page 10 GAO-03-867R COPS Evaluation

university police, would have improved the estimates of COPS grant effects; and (6) 

were ill-advised in stating that including police departments in cities that did not 

receive COPS funding would have improved the estimates of COPS grant effects. We 

continue to disagree with the researchers on these key points and discuss our 

reasons below. 

First, with respect to the assertion that our standards were too high and that we did 

not consider the advances made by this study, we would reiterate that the purpose of 

our assessment was to determine the extent to which the conclusions of this 

particular COPS study were supported by the data used and analyses conducted. 

Because we were asked to review this single study and did not have time to review 

any others, we cannot comment on whether and how this study’s approach to 

evaluating the effectiveness of the COPS program may have been an incremental 

improvement over other similar efforts. We acknowledge in the introduction to this 

report that it is extremely difficult to assess program effectiveness via aggregate level 

analyses. We also believe that the researchers should be commended for their efforts, 

which involved merging data on more than 6,000 towns and cities over a multi-year 

period from four different sources and using sophisticated methods to analyze those 

data under a variety of specifications. But, in our estimation, the problems that we 

identified with the research make the results more suggestive than conclusive. 

Second, with respect to the assertion that the statistical models were both correctly 

specified and sufficiently controlled for the effect of missing data on police 

expenditures, we do not believe this was the case. Zhao et al. believe that we are 

unjustifiably critical of their having used county rather than city dummy variables in 

estimating the effects of COPS grants on crime rates. They point out that they ran 

both their 2001 and 2003 analyses using both city and county dummy variables, and 

the results of the two types of analyses did not differ substantially.15 While the 

models incorporating county or city dummy variables do, as the authors assert, 

explain a sizable portion (between 64 percent and 86 percent) of the variation in 

reported crime rates across cities over time, this is not surprising and is largely 

attributable to the very large number of dummy variables included in their models. 

The proportion of variance explained, however, does not necessarily imply that the 

estimates of the effects of COPS grants were unbiased. The authors, in our opinion, 

are mistaken in their claim that the use of dummy variables controls for the effects of 

all unmeasured differences between cities and over time. That is, the county dummy 

variables do not control for unmeasured differences between cities within counties, 

and even the combination of city and year dummy variables do not control for 

differences within cities over time, unless the changes in all cities are similar. Crime 

rates in cities did not show similar changes over time,16 however, and there are many 

 

15The researchers noted that they recently collected original police expenditure data from 55 of the 

largest police departments and found that including these data in the statistical models showed that 

they had virtually no effect on their estimates of the effects of COPS funding. We appreciate the 

difficulty of obtaining police expenditure data for large cities and endorse efforts to marshal 

supporting evidence from a sample of those cities. However, we have not seen the results of these 

analyses and have no basis to judge how representative these 55 cities are of large cities in general, or 

whether the estimated effects of COPS grants from the 55 cities are generalizable to larger cities 

generally. 

16Bureau of Justice Statistics data on 62 local police departments serving cities with a population of 

250,000 or more revealed a high degree of change in violent and property crime rates within the same 

city over time. For example, New York’s reported violent crime rate dropped by 57 percent between 
Page 11 GAO-03-867R COPS Evaluation

factors that can change in cities from one year to the next in ways that might affect 

crime rates. For example, fluctuations in local, state, and other expenditures on 

police could produce changes in crime rates within cities over time, and the failure to 

control for such factors can seriously bias the estimates of the effects of COPS 

grants. 

Third, with respect to the assertion that omitting data on cities’ access to grants other 

than COPS grants probably did not affect the results, we are not convinced. We agree 

that data on grants that cities receive are not readily available. However, we believe 

that information on at least major grant programs could be obtained from the Office 

of Justice Programs. To the extent that cities that receive COPS grants may be more 

likely to receive other types of grants, omitting consideration of other grants that are 

also targeted at reducing crime may lead to an overestimation of the effects of COPS 

grants. By restricting their attention to COPS grants awarded to city and local police, 

the researchers investigated the effects of only a portion of all COPS grants. They 

ignored the effects of other grants and of state and local expenditures, generally, and 

therefore increased the potential for obtaining biased estimates of COPS grant 

effects. 

Fourth, with respect to the assertion that the study’s results were not impaired by the 

omission of such socioeconomic variables as percentage of the city population that is 

male, we disagree with the researchers that this is not problematic. They assert that 

(1) the dummy variables in their statistical models controlled for the effects of 

socioeconomic variables other than those in their analyses, (2) a city’s male 

population should not significantly affect the estimated effects of COPS grants on 

crime, (3) the socioeconomic variables included in the analyses were sufficient and 

grounded in widely accepted social disorganization theory, and (4) problems of 

multicollinearity17 could have arisen had they included additional socioeconomic 

variables. As with police expenditures, we maintain that data on factors affecting 

crime rates that vary across cities and over time should be included in analyses, and 

may not be sufficiently controlled by statistical models that use dummy variables to 

control for unmeasured differences. While we do not know whether and how COPS 

grant amounts to cities may be associated with the socioeconomic characteristics of 

city residents, the literature indicating a gender difference in crime is extensive.18 To 

the extent that socioeconomic characteristics affect crime rates, and to the extent 

that cities that received COPS grants may have different socioeconomic 

characteristics, we believe it would be wise to incorporate such variables into models 

to lessen any potential bias in the estimates of the COPS grants on crime. Since this 

study was intended to be an evaluation of the effects of COPS grants on crime and 

 1990 and 2000, while Nashville’s rate increased by 29 percent during that same time period. Similarly, 

New York’s reported property crime rate dropped by 60 percent, while Nashville’s rate increased by 22 

percent. (Police Departments in Large Cities, 1990-2000. Department of Justice, Bureau of Justice 

Statistics, May 2002). 

17Multicollinearity means that the independent variables are highly correlated. If this occurs, it is 

impossible to distinguish between them in estimating their effects on the dependent variable. 18For example, a Bureau of Justice Statistics study indicated that men comprised 93 of the state prison 

population in 2001; 93 percent of the federal prison population in 1997; and 90 percent of the local jail 

population in 1996 (http://www.ojp.usdoj.gov/bjs/crimoff.htm#findings). Another study reported that 

in 1960, 1975, and 1990, men were arrested at much higher rates than women for all crime categories 

except prostitution (Steffenmeier, D. and Allen, E., “Gender and Crime: Toward a Gendered Theory of 

Female Offending,” Annual Review of Sociology, 1996, vol. 22, pp. 459-87). 
Page 12 GAO-03-867R COPS Evaluation

not as a test of social disorganization theory, we do not believe that limiting the 

socioeconomic control variables to those dictated by this particular theory was 

warranted. Finally, with 36,000 observations in their study, we do not believe that 

multicollinearity would have been a problem had additional socioeconomic variables 

been included in the analyses. 

Fifth, with respect to the assertion that including data on COPS-funded agencies 

within cities would not have improved the estimates of COPS grant effects, we 

continue to believe that this cannot be known. Zhao and Thurman state that there is 

no meaningful way to include such agencies—for example, park and university 

police—in their statistical models because the jurisdictions overlap. They note that it 

was neither necessary nor possible to estimate the effects of such agencies on crime 

rates because they report crime incident data to the Federal Bureau of Investigation 

separately and because census data for them are not readily available. We maintain 

that by restricting their attention to crimes reported to local and city police 

departments, the researchers are investigating the effect of only a portion of all COPS 

grants and are looking at only a subset of all crimes reported. Again, we do not know 

whether these restrictions result in an overestimate or underestimate of the effect of 

COPS grants on crimes, but they can potentially bias their estimates. We 

acknowledge that data may not be readily available for such an analysis, but that does 

not mean they cannot be collected or that they are unimportant. 

Sixth, with respect to the assertion that including data on police departments in nonCOPS funded cities would not have improved the estimates of COPS grant effects, we 

continue to disagree. Zhao and Thurman note that because small cities are more 

likely than large cities to not receive COPS funding, including nonfunded agencies in 

their analysis could bias the findings towards showing an effect of COPS grants. It is 

our view that missing cases, except when they are missing at random, should be 

regarded as problematic. The 6,100 agencies that Zhao et al. analyzed represented 

about 85 percent of the COPS-funded city and local police departments, 51 percent of 

the total number of COPS-funded agencies, and 36 percent of the agencies that 

participate in the UCR system. Some of these exclusions may have been unavoidable, 

but their cumulative impact is likely to be non-negligible. 

We do not know how or to what extent the findings that Zhao et al. obtained would 

change if the limitations that we identified in our assessment were successfully 

resolved. We do know, however, that while Zhao et al. may have performed the most 

sophisticated and advanced research on the topic, drawing inferences or making 

policy decisions about COPS grant effects from this work are unwarranted at this 

time. Indeed, Zhao and Thurman are themselves continuing this work, an indication 

that they also believe refinements are needed. 

The comments from the COPS office and the researchers are reproduced in the 

enclosure to this report. The COPS Office also provided us with technical comments, 

which we incorporated in the report as appropriate. 

-- -- -- -- -- -- 

As agreed with your office, unless you publicly announce the contents of this report 

earlier, we plan no further distribution of it until 30 days from the date of this report. 
Page 13 GAO-03-867R COPS Evaluation

We will then send copies of the report to the Attorney General and will make copies 

available to others upon request. In addition, the report will be available at no charge 

on GAO’s web site at http://www.gao.gov. 

If you have any questions about this report, please contact me at (202) 512-8777. The 

key contributors to this report were David Alexander, Carl Barden, Evi Rezmovic, 

and Douglas Sloane. 

Sincerely yours, 

Laurie E. Ekstrand 

Director, Homeland Security 

 and Justice Issues 

Nancy Kingsbury 

Managing Director, Applied Research 

 and Methods Issues 

Enclosures - 2 
Enclosure I

Page 14 GAO-03-867R COPS Evaluation 

Enclosure I 
Enclosure I

Page 15 GAO-03-867R COPS Evaluation 
Enclosure I

Page 16 GAO-03-867R COPS Evaluation 
Enclosure II

Page 17 GAO-03-867R COPS Evaluation 

Enclosure II 
Enclosure II

Page 18 GAO-03-867R COPS Evaluation 
Enclosure II

Page 19 GAO-03-867R COPS Evaluation 
Enclosure II

Page 20 GAO-03-867R COPS Evaluation 
Enclosure II

Page 21 GAO-03-867R COPS Evaluation 
Enclosure II

Page 22 GAO-03-867R COPS Evaluation 
Enclosure II

Page 23 GAO-03-867R COPS Evaluation 

(440213)