U.S. Department of Justice
Office of Community Oriented Policing Services

Measuring Excellence: 
Planning and Managing Evaluations 
of Law Enforcement Initiatives

Kristin Ward | Susan Chibnall | Robyn Harris


ICF International, a global professional services firm, 
partners with government and commercial clients to deliver 
consulting services and technology solutions in energy, 
climate change, environment, transportation, social programs, 
health, defense, and emergency management. In October 
2005, ICF acquired Caliber Associates, an organization that 
provides full life-cycle program consulting and support 
in research and evaluation, communications, training and 
technical assistance, and information technology. Together, 
ICF and Caliber have expanded capabilities to assist both 
civilian and national security organizations to develop more 
effective and innovative human resource, education, and 
social service programs.

Prior to the acquisition, Caliber had been awarded a 2-year 
contract from the U.S. Department of Justice Office of 
Community Oriented Policing Services (the COPS Office) 
to develop Measuring Excellence: Planning and Managing 
Evaluations of Law Enforcement Initiatives. Tailored specifically 
for the law enforcement community, the guide emphasizes 
the importance of program evaluation to law enforcement. 
It is in a user-friendly format that assists law enforcement 
administrators throughout the evaluation management 
processfrom the initial stages of thinking about and 
planning a program evaluation, through selecting, designing, 
and implementing program evaluations, and using the results 
to meet the needs of their communities and stakeholders. 


Measuring Excellence: 
Planning and Managing 
Evaluations of Law 
Enforcement Initiatives

Kristin Ward | Susan Chibnall | Robyn HarrisJune 2007

This project was supported by cooperative agreement 
#2003-CK-WX-K052 awarded by the Office of 
Community Oriented Policing Services, U.S. Department 
of Justice. The opinions contained herein are those of 
the authors and do not necessarily represent the official 
position of the U.S. Department of Justice. References 
to specific companies, products, or services should not 
be considered an endorsement by the authors or the 
U.S. Department of Justice. Rather, the references are 
illustrations to supplement discussion of the issues.


Table Of Contents

Foreword . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii

Chapter I: Why Evaluate? . . . . . . . . . . . . . . . . . . . . . . 1

Purpose of the Guide . . . . . . . . . . . . . . . . . . . . . . . . 3

Why Evaluate? The Rationale for Program Evaluation . . . . . . . . . 3

Real-World Examples of Evaluation in Policing . . . . . . . . . . . 10

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Chapter II: Key Concepts in Evaluation  . . . . . . . . . . . . . . 15

The Different Types of Evaluation . . . . . . . . . . . . . . . . . 17

Evaluation Designs  . . . . . . . . . . . . . . . . . . . . . . . . 29

The Continuous Feedback Loop of Program Planning and 
Evaluation and the Key Steps in the Evaluation Process  . . . . . . 40

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 43

Chapter III: Conducting an Internal Evaluation  . . . . . . . . . . 45 

Steps in the Evaluation Process . . . . . . . . . . . . . . . . . . 49

Step One: SpecifyEvaluation Planning . . . . . . . . . . . . . . . 50

Step Two: CollectData Collection . . . . . . . . . . . . . . . . . 58

Step Three: CompareData Analysis . . . . . . . . . . . . . . . . . 69

Chapter Summary . . . . . . . . . . . . . . . . . . . . . . . . . . 75

Chapter IV: Managing Evaluations  . . . . . . . . . . . . . . . . . 77

How Do I Manage an Evaluation?  . . . . . . . . . . . . . . . . . . 79

What Do I Need to Do During the Evaluation? . . . . . . . . . . . . 88

What Do I Need to Do After the Evaluation?  . . . . . . . . . . . . 89

Challenges to Managing Evaluations  . . . . . . . . . . . . . . . . 92

Should I Hire an External Evaluator?  . . . . . . . . . . . . . . . 93

Chapter V: Emerging Topics in Evaluation Management  . . . . . . . 101

Collaborative Evaluation   . . . . . . . . . . . . . . . . . . . . 103

Participatory Evaluation . . . . . . . . . . . . . . . . . . . . . 107

Performance Measurement  . . . . . . . . . . . . . . . . . . . . . 110

Chapter Summary  . . . . . . . . . . . . . . . . . . . . . . . . . 113



Exhibit I-1: Six Myths About Evaluation  . . . . . . . . . . . . . . 6

Exhibit II-1: Examples of Questions Addressed in a 
Needs Assessment  . . . . . . . . . . . . . . . . . . . . . . . . . 19

Exhibit II-2: Examples of Questions Addressed in a 
Process Evaluation  . . . . . . . . . . . . . . . . . . . . . . . . 21

Exhibit II-3: Examples of Questions Addressed in an 
Outcome Evaluation  . . . . . . . . . . . . . . . . . . . . . . . . 23

Exhibit II-4: Examples of Questions Addressed in an 
Impact Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . 26

Exhibit II-5: Examples of Questions Addressed in a 
Cost-Benefit Evaluation . . . . . . . . . . . . . . . . . . . . . . 27

Exhibit II-6: Overview of Evaluation Types  . . . . . . . . . . . . 29

Exhibit II-7: Strengths and Weaknesses of Different 
Types of Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 36

Exhibit II-8: Common Threats to the Validity of 
Evaluation Results  . . . . . . . . . . . . . . . . . . . . . . . . 39

Exhibit II-9: The Program Planning and Implementation Cycle . . . . 40

Exhibit II-10: The Evaluation Cycle . . . . . . . . . . . . . . . . 41

Exhibit II-11: Integration of the Planning and Evaluation Cycles  . 42

Exhibit III-1: Difference Between Academic Research and 
Internal Evaluation . . . . . . . . . . . . . . . . . . . . . . . . 48

Exhibit III-2: Tips for Developing Goals and Objectives . . . . . . 54

Exhibit III-3: Sample Logic Model . . . . . . . . . . . . . . . . . 56

Exhibit III-4: Tips for Developing Surveys and Survey Questions . . 62

Exhibit III-5: Tips for Conducting Surveys  . . . . . . . . . . . . 64

Exhibit III-6: Tips for Planning and Facilitating Focus Groups  . . 66

Exhibit III-7: Data Collection Methods at a Glance  . . . . . . . . 67

Exhibit III-8: Options for Organizing and Reporting 
Qualitative Data  . . . . . . . . . . . . . . . . . . . . . . . . . 72


Exhibit IV-1: Assessing Evaluator Capabilities  . . . . . . . . . . 94

Exhibit IV-2: Advantages of Hiring an External Evaluator 
Compared to Using Internal Staff  . . . . . . . . . . . . . . . . . 95

Exhibit IV-3: Sources for Finding an External Evaluator . . . . . . 98

Exhibit V-1: Difference between Participatory Evaluation 
and Traditional Evaluation . . . . . . . . . . . . . . . . . . . . 109

Exhibit V-2: Differences between Performance Measurement 
and Program Evaluation . . . . . . . . . . . . . . . . . . . . . . 110

Appendixes

Appendix A: Sample Needs Assessment Action Plan Template . . . . . 117

Appendix B: Preparing Measurable Goals and Objectives  . . . . . . 118

Appendix C: Planning Model Template  . . . . . . . . . . . . . . . 121

Appendix D: Community Satisfaction Survey  . . . . . . . . . . . . 123

Appendix E: Sample Focus Group Checklist . . . . . . . . . . . . . 126

Appendix F: Instrument Development Checklist . . . . . . . . . . . 128

Appendix G: Topics to Cover in an Informed Consent Statement 
for Research Participants  . . . . . . . . . . . . . . . . . . . . 130

Appendix H: Sample Evaluation Plan Template  . . . . . . . . . . . 132

Appendix I: Sample Interview Questions Checklist . . . . . . . . . 133

Appendix J: Glossary . . . . . . . . . . . . . . . . . . . . . . . 134

Appendix K: Sample Job Announcement  . . . . . . . . . . . . . . . 139

Appendix L: Evaluation Resources . . . . . . . . . . . . . . . . . 141

References  . . . . . . . . . . . . . . . . . . . . . . . . . . . .153



Foreword

The primary purpose of this guide is to provide you with information for conducting and 
managing program evaluations in your law enforcement agency. However, during the 
needs assessment conducted to develop this guide, during which input was solicited from 
law enforcement personnel nationwide through focus groups and telephone interviews, it 
became clear that many police administrators and managers would benefit from an enhanced 
understanding of evaluation. From the needs assessment, five key factors arose that shaped the 
development of this guide:

 
1. Many of you would benefit from exploring the value, importance, and potential uses 
of evaluation information  Chapter I: Why Evaluate? 

2. Many of you also would benefit from learning more about evaluation basics, so you 
could better manage and conduct a program evaluation  Chapter II: Key Concepts 
in Evaluation. 

3. In many law enforcement agencies, the person responsible for managing evaluation 
activities also conducts them  Chapter III: Conducting an Internal Evaluation. 

4. Law enforcement agencies around the country are conducting their own program 
evaluations of program processes, outcomes, and impacts. In addition, for large or 
complex community initiatives, law enforcement agencies are conducting external 
evaluations with the help of evaluation experts outside their department. Many of 
you could use tips on managing both types of evaluative endeavors  Chapter IV: 
Managing Evaluations. 

5. Law enforcement agencies are increasingly engaging in new types of evaluation 
methodologies, including participatory evaluations, collaborative evaluations, 
and performance measurement activities (i.e., monitoring and reporting program 
accomplishments)  Chapter V: Emerging Topics in Evaluation Management.

This guide presents the basic concepts and guidance for planning, implementing, and 
managing an evaluation, including understanding evaluation concepts and activities. 
Where more information might be needed or desired, an appendix is provided with references 
to other, more detailed, sources. 

This guide is designed to help law enforcement agencies that plan to begin conducting evaluations 
as well as those that want to refine and improve their evaluation management techniques. 

The authors thank Bonnie Bucqueroux for reviewing the report. Through her suggestions, she 
made important contributions to the work.


Chapter I: 
Why Evaluate?


Why Evaluate?

Purpose of the Guide

Modern policing continues to place new demands on police leadership. Policing 
today not only addresses crime, fear of crime, and social and physical disorder, 
but also requires the police to be involved in collaborative problem solvingthe 
only government agency patrolling neighborhoods around the clock. 

As the scope of police work broadens, so too does its measurement, extending 
beyond numbers of arrests made and response time and making defining success 
an increasingly complex challenge. Whose views matter? The police themselves? 
Elected officials, the business community, schools, the faith community, 
the media, average citizens? Particularly in an environment of strict budget 
considerations, police are compelled to move beyond mere opinion and justify 
their work through comprehensive, quality evaluations. 

Persuading police to embrace evaluation prompts the question, Whats in it for 
us? Also, for evaluation to gain widespread acceptance, police administrators 
need the tools to conduct evaluations and do them well. Therefore, this guide 
begins with an explanation of why evaluation must become an integral part of 
police work, especially as police administrators repeatedly find themselves asked 
to do more with less. 

The guide also provides practical information and advice on conducting quality 
evaluations that recognize the unique challenges inherent in police work
evaluations that can withstand rigorous scrutiny and help build community trust 
and support. This guide is a user-friendly resource that is supplemented with 
appendixes that take you through several practical exercises and direct you to 
more detailed information from other sources.

Why Evaluate? The Rationale for Program Evaluation

There are many kinds of evaluation, but the most common is program evaluation, 
in which the goal is to discover how well a particular program is working. 
Program evaluation is defined as a systematic process of gathering and analyzing 
information for the purposes of program assessment, program improvement, 
and, in a broader sense, strategic management. It provides a structured way to 
verify, document, and quantify program activities and their effects. 

Done well, evaluation provides meaningful results to help you make informed 
judgments about the effectiveness and efficiency of your strategies and programs. 



You should be able to demonstrate that you are committing resources to efforts 
that produce tangible benefits for the entire community. Evaluation can help you 
in these ways:

Compare alternatives. What works best? Framed correctly, evaluation can let 
you compare programs and strategies so you can prioritize where to put scarce 
resources.

Determine appropriateness. Is the program the right way to address the problem? 
Does it enjoy not only the support but also the participation of the community? 
Policing is about both means and ends and evaluation can help you determine 
whether the program is the best and most inclusive way to deal with the problem. 

 
Clarify program objectives. What are you trying to accomplish? How will you 
define success? A well-planned evaluation requires you to clarify assumptions 
about the links between your target population, the program activities, and the 
immediate, intermediate, and long-term outcomes you expect. There is wisdom in 
the saying: If you dont know where you are going, any path will take you there. 
Evaluation helps you decide where you are going and how to determine whether 
you are taking the best path to get there. 

 
Advocate for your efforts. How can you energize supporters and persuade critics 
to come on board? Perhaps the biggest virtue of an evaluation is that it has 
more validity than mere opinion. It is easy to say that you know something is 
working and provide anecdotes that support that view. However, as we grow 
more sophisticated in our decision-making, it becomes increasingly important to 
conduct outcome-based evaluations.

Describe the program and educate the community. Do key stakeholders 
understand the initiative and how it works? Do average citizens know what the police 
are doing and why? In the rush to do good things, police often fail to explain their 
actions. Evaluation can require documenting not only the program but also the 
underlying philosophy and the results. People outside police circles often believe 
in the myth of the easy answer or the quick fix to crime and social disorder 
problems. A thoughtful evaluation can help stakeholders and citizens understand 
better the challenges of policing.

Solidify support. How can we keep supporters engaged? Positive results from a 
well-planned evaluation can be used to justify a programs existence, maintain 
commitments from program administrators or the community, and leverage 
additional resources from funders, community partners, and other stakeholders.

 
Address program cost. Are you operating within budget? Are there opportunities 
for cost savings? Do the results justify the investment? Evaluation is a process 
that provides feedback to help you make adjustments and can give valuable 
information about ways to save money and whether the program delivered 
sufficient bang for the buck.


Make informed program improvements. Do you need to make midcourse 
corrections? Are there changes that should be made before the program is allowed to 
continue or is replicated elsewhere? The information collected during an evaluation 
can provide the information you need to make intelligent changes likely to 
improve opportunities for success.

Monitor program integrity. Are you really doing what you said you would do? 
Tracking the number and type of activities you offer, the number and type of 
participants involved, and your activity-related expenses can help determine if you 
actually are implementing the program as intended and promised. 

 
Provide a blueprint for peers. How will newcomers learn what to do? Will others 
inside the agency be able to implement the program or strategy? A quality evaluation 
provides a road map for others to follow, with the potential barriers and pitfalls 
identified.

Recognize and capture unanticipated or unintended program 
effects. Were there side effects that need to be documented and understood? 
Evaluation often uncovers positive or negative program effects that were neither 
anticipated nor intendedconsequences that can be lost if no evaluation process 
is in place.

 
Inform internal systems. Can the findings be used to improve functioning inside 
the department? Evaluation also can be used more holistically and broadly as a 
tool for internal strategic planning and management. If, for example, you find 
that a new initiatives success correlates to the communication skills of the officers 
implementing the program, this has implications for your human resources 
department and its recruiting guidelines. 

Assess overall effectiveness. Did the program or effort work as intended? Data 
collected during an evaluation will help you document and demonstrate what is 
working and what is not. 

 
Pilot test innovations. How can we identify new ideas for the future? Policing must 
never be static and managers must be willing to take reasonable risks on new 
ideas. But the challenge is to conduct quality evaluations to determine whether 
promising ideas might pan out in practice.

 
Contribute to the field. Does this evaluation help our understanding of future 
directions? The lessons that you learn from your evaluation can be shared with 
other law enforcement agencies and communities. Sharing evaluation findings 
regularly will contribute to creating a much-needed body of verifiable law 
enforcement knowledge.


Evaluation in the Law Enforcement Context

Formal evaluation is rarely, if ever, required to maintain general funding of police 
department operations. Often, it is not even required of state and federal law 
enforcement grants. Consequently, many police chiefs and other law enforcement 
administrators often do not make evaluation a priority. In part, this reluctance 
to integrate and implement evaluation might stem from not wanting to invest in 
learning bad news. Common myths and misconceptions about evaluation, as well 
as a lack of understanding about the place of evaluation in the policing context, 
play a role in the scarcity of evaluation activity in police departments across the 
country. Six myths about evaluation are presented in Exhibit I-1.

Exhibit I-1: 

Six Myths About Evaluation

The Myth
Police leaders dont value 
evaluation; lack of internal 
support.

The Reality

Todays police leaders must be able to prove that they are 
spending tax dollars wisely. Many police departments today 
are conducting and managing meaningful evaluations of 
their efforts.

The Myth
Police departments do not 
need to do evaluation (not 
required by law or grants).

The Reality

Quality evaluations are for YOU, not THEM. Evaluation 
provides a framework that assists you in doing the best job 
possible within existing resources. 

The Myth
Police departments lack 
the resources to do proper 
or meaningful evaluation 
studies.

The Reality

Program evaluation need not be costly or time-consuming. 
Valuable results can be obtained by asking targeted 
questions or using existing data. Do what you can with what 
you have now and build from there.

The Myth
Police staff lack the skills 
for evaluation design and 
analysis.

The Reality

Program evaluation is not just for academics. The best 
evaluations are based on a common-sense approach to 
gathering and analyzing information.

The Myth
Police departments lack 
relevant data to evaluate 
their efforts.

The Reality

The data will be there if you weave your evaluation strategy 
into program planning. If you did not capture appropriate 
data this time, make sure to craft a better plan for the future.

The Myth
Police departments fear 
negative results.

The Reality

The politics of policing can make top command skittish 
about investing in efforts that may generate bad news. But 
evaluation is really about improving police effectiveness 
and prioritizing scarce resources. An evaluation is not a 
gotcha, but a blueprint for improvement.


Evaluation informs spending decision

In addition to the reasons listed in Exhibit I-1, police officials need to embrace 
evaluation to maintain public trust. As stewards of public funds, police departments 
are accountable for ensuring that taxpayer dollars are used appropriately, effectively, 
and efficiently. Program evaluation not only addresses those concerns, but also can 
provide a cost-benefit analysis, which is a tool that can help administrators decide 
where to invest limited funds. While there can be risks involved in evaluating 
programs and strategies (e.g., discovering that a popular 
or expensive strategy is not working), most administrators 
would rather know the truth as soon as possible. With that 
knowledge, decision-makers can make adjustments based 
on informed feedback, or abandon a misguided effort before 
more money is wasted.

Evaluation supports community policing and 
problem-oriented policing

Program evaluation is essential for, and compatible with, cutting-edge policing 
approaches such as community policing and problem-oriented policing. 
For instance, those familiar with the SARA (Scanning, Analysis, Responseand 
Assessment) model will recognize that evaluation is an integral component. 
Problem-oriented policing requires recognition (scanning) and analysis of a problem, 
a strategically planned response to address it, and an assessment (i.e., an evaluation) of 
how well the strategy solved or reduced it. Evaluationsespecially those that include 
stakeholder participationcan further the objectives of community policing. By using 
strategies that capture and analyze findings from the community and participating 
professionals, evaluation offers insights into opportunities for enhancement and 
improvement. Also, evaluating community-based initiatives with the help of citizen-
participants can bolster credibility, buy-in, partnership, and accountability, while 
contributing to long-term citizen satisfaction and neighborhood development and 
improvement.

Moreover, law enforcement already is data-driven. In fact, it could be said that 
informal evaluation occurs all the time, as police administrators consult a variety 
of data to make decisionscrime data, calls for service, clearance rates, discussions 
with officers. The challenge is to move from an informal approach that might not 
be rigorous or comprehensive, or that has gaps, to a more 
informed and institutionalized process that contributes 
to better decision-making. As problem-oriented policing 
and community policing increasingly become the ways in 
which most police agencies deliver service, comprehensive 
evaluation provides a way to move beyond the narrow 
and incomplete measures of police performance of the 
past, such as crime rates and response times. This guide 
provides both a rationale for quality evaluation and the 
tools you will need to do the job.

 What you dont know CAN hurt 
you. Evaluation lets you know 
what works and what doesnt. 
 Chief Tony Kleibecker, Muskegon 
(Michigan) Police Department

 We need to evaluate the 
short-term temporary gains 
associated with reactive policing 
so that we can effectively plan 
for long-term neighborhood 
improvements. 
 Chief Harry Dolan, Grand Rapids (Minnesota) 
Police Department


Examples of How Law Enforcement Agencies Nationwide
Use Evaluation

Despite the obstacles to conducting formal evaluation in the policing context, 
many police agencies are successfully incorporating program evaluation into their 
daily practice. The Community Policing Consortium offered advice regarding 
what to evaluate in the policing context: 

 
Internal structures and systems: 

Departmental organization

Recruiting

Training

Performance evaluation

Promotion

Formal and informal rewards

Internal communication

Morale

 
Strategic plans:

Assess the process of developing a strategic plan

Monitor implementation of the strategic plan

 
Resource allocation: 

Assess the decision-making process for resource allocation

Monitor resources in terms of what is being spent, where, why, and 
whether money could be spent more wisely

 
Pilot projects: 

Determine the benefits and drawbacks of wider implementation

Effectiveness of community policing efforts/strategies

Assess outcomes from the perspectives of different stakeholders

 
Quality of police services:

Customer satisfaction

Community perceptions of the police

(Community Policing Consortium Resources: Strategic Planning for Community 
Policing, n.d.)

 We dont have 
enough resources, 
time, or energy to 
waste on things that 
dont work  How 
can we prove that 
the millions of dollars 
spent on us are 
actually having some 
effect?
 Chief Jim Bueerman, 
Redlands (California) Police 
Department 


Evaluations vary from large-scale, expensive, external, and experimental efforts to 
small-scale, low-budget, internal assessments of local problem-solving initiatives. 
Their purposes range from discrete, one-time evaluations of specific activities to 
using evaluation as a long-term strategic development and management tool. 

The following are examples of successful evaluations that law enforcement 
agencies across the country have conducted:

Evaluation of the effectiveness of a blended housing and recreation 
initiative to lower risk factors for, and incidence of, crime in 
neighborhoods, using a citywide experimental design (Redlands, 
California: 2004).

Evaluation of a 12-officer bike patrol program, using neighborhood 
satisfaction surveys and process/implementation evaluation to understand 
whether the program was being implemented as intended and whether 
funds were well spent (Muskegon, Michigan: 2003).

Evaluation of the effectiveness of multidisciplinary collaboration among 
city groups and agencies to identify drug dealers and increase the number 
of narcotics raids (Muskegon, Michigan: 2003). 

Evaluation of youth violence-prevention efforts (e.g., partnerships 
between police departments and schools to prevent and decrease 
bullying), using pre- and post-program surveys with students, parents, 
and teachers, as well as observations (Chula Vista, California: 2003).

Evaluation of collaborative efforts to reduce residential burglaries in a 
new housing development area, using interviews with victims, street-view 
environmental assessments, review of incident reports, and tracking and 
comparing trends in theft data in various neighborhoods 
(Chula Vista, California: 2003).

Evaluation of efforts to address disorder in budget motels using 
pre- and post-program interviews with motel managers (Chula Vista, 
California: 2003).


Real-World Examples of Evaluation in Policing

This guide will discuss how you can manage and use evaluation to measure and 
improve programs and direct your strategic vision. Throughout the guide, we will 
reference real-world examples, describing how your colleagues in law enforcement 
have planned, implemented, and ultimately learned from evaluation in their 
work. We begin in this chapter with two examples that illustrate the range of 
evaluation activity and purpose in the policing context. We hope they will inspire 
and encourage you to learn more about what your department can achieve 
through evaluation.

The first example is the Minneapolis Police Departments Hawthorne Huddle, 
which clarifies how evaluation can help you gauge the effectiveness of community 
policing strategies and collaborations with community agencies and citizens to 
improve neighborhoods. The Hawthorne Huddle also illustrates how evaluation 
can assist in improving or complementing the strategies used to build police/
community partnerships. 

The second case study, the Colorado Springs Police Departments Police 
Accountability and Service Standards (PASS) model, exemplifies how evaluation 
can be utilized for developing and assessing the strategic vision, goals, and 
objectives of law enforcement agencies.


The Hawthorne Huddle

A collaborative and action-oriented program evaluation model

Conditions in the Hawthorne neighborhood of North Minneapolis declined into the 
mid-1990s. The area was plagued by high crime, poor rental property management, 
and a deteriorating housing stock. Drug dealing and other crime contributed to 
the disorder in the neighborhood, and residents felt fearful, disenfranchised, and 
abandoned by what they perceived as a lack of effort on the part of city officials to 
improve the situation. Police responsible for patrolling the area felt frustrated by the 
fact that they saw no sign of the bad times abating in the area. 

The Minneapolis Police Department (MPD) decided to collaborate with other 
city and county agencies, nonprofit organizations, a major corporate sponsor 
(the General Mills Foundation), and community residents to develop a plan to 
deal with crime and disorder in the area. MPD used crime mapping and analysis 
of police reports to define and target the biggest problems in the area. Next, 
they organized neighborhood meetings and encouraged citizen participation 
in block clubs to garner support and assistance with problem-solving efforts. 
Residents quickly accepted the block club meetings, which grew over time. To 
this day, agency and community members gather at monthly meetings called the 
Hawthorne Huddle.

Additional strategies were aimed at reducing the number of criminal narcotics 
arrests at specific addresses, as well as the number of vandalism incidents and 
burglaries in the neighborhood. To assess the effectiveness of these strategies, 
several evaluation techniques were ultimately used. 

Evaluation strategies were collaborative and participatory in that MPD was not 
solely, or even predominantly, responsible for evaluation of the Hawthorne Huddle. 
Instead, the agencies involved in the project worked together to define goals and 
objectives and decided how success would be measured. Participating agencies and 
organizations volunteered to collect various kinds of relevant data. MPD gathered 
and analyzed crime statistics for the area while other organizations conducted 
surveys and focus groups to measure residents feelings of safety and satisfaction, 
and gauge ongoing concerns. This collaboration helped keep Hawthorne Huddle 
program activities grounded in real community issues and concerns.

This collaborative and action-oriented evaluation model relies on stakeholders 
working together to identify the problem, develop goals and objectives, define 
success, decide how and what to measure, and determine how to use results 
to continue improving the community. Evaluation results are released to the 
community at monthly Huddle meetings. Stakeholders use the information to 
make decisions about future efforts (reflecting the values of community policing 
and its emphasis on building connections to local leaders), work together to identify 
community needs, and leverage evaluation results to improve the community.


The Police Accountability and Service 
Standards (PASS) Model

A strategic process, program, and performance evaluation model

Under the leadership of Chief Luis Velez, the Colorado Springs Police 
Department (CSPD) developed the PASS model as a structured way to gauge the 
departments progress in infusing the community policing philosophy throughout 
the organization. PASS is an ambitious effort that includes a commitment 
to evaluate how new programs are designed and implemented, recognizing 
that community policing cares about means and ends. It includes qualitative 
evaluation strategies that go beyond traditional quantitative measures of police 
activity such as response time and numbers of arrests. PASS serves as a model of 
how police agencies can develop a comprehensive evaluation strategy to ensure 
that all police activities reflect the philosophy, vision, mission, and values of the 
department. 

Colorado Springs is one of a handful of jurisdictions nationwide that have 
designed a prototype for identifying and measuring police service standards. The 
PASS model is a holistic strategic planning and evaluation design that requires an 
annual assessment of the past years performance, development of a strategic plan 
in partnership with citizens, implementation of that strategic vision, measurement 
of both processes and outcomes associated with police service delivery, assessment 
of overall organizational performance, and demonstration of police accountability 
and stewardship to community. It involves conducting individual, unit, and 
division performance evaluations to enhance the alignment of goals and objectives 
throughout the police organization (which are linked to the citys strategic plan). 

Evaluation expert Mora L. Fiedler assisted in developing the user-friendly PASS 
model, which calls for collecting both process and outcome data. Analysis of 
process data answers questions about how efforts are conducted, with what 
resources, with whom, and how the police department partners with others in 
the community. Outcome data are collected to measure the effectiveness of the 
service standards that have been defined with citizen input to the PASS process. 
For example, CSPD collects measurement data about response times, officer 
deployment, traffic, drug and vice activity, neighborhood policing, and citizen 
satisfaction with police services. CSPD follows various evaluation methods to 
collect and analyze these data. Examples will be cited throughout this guide. 

Source: Fiedler, M. The Colorado Springs Police Departments police accountability and service standard (PASS) 
model. Colorado Springs: Management Services Division, Planning Section, Colorado Springs Police 
Department, 2002.


Chapter Summary

Evaluation is important in modern policing, particularly in light of limited 
budgets and the adoption of problem-oriented policing and community policing 
strategies. Police leaders need to know what works and what does not, both inside 
and outside of the department. This chapter outlined the benefits of evaluation 
and discussed several common myths that can prevent law enforcement agencies 
from pursuing evaluations. However, despite both real and imagined barriers, 
police departments nationwide are beginning to incorporate evaluation into their 
practice. Suggested areas for evaluation within policing and several contemporary 
examples of successful evaluations in United States police departments were 
presented. 

Evaluation in law enforcement holds tremendous potential and is not a dangerous 
process designed to document failure or generate bad news. Instead, it is a 
valuable tool that agencies can use to enhance, prioritize, and justify activities. 
The following chapters go further in dispelling the myths that evaluation is too 
demanding, technical, and expensive for police departments to implement. This 
guide will help demystify the process of program evaluation and will provide 
practical advice for starting or enhancing evaluation activities. Refer to this guide 
often for ideas and inspiration. 


Chapter II: 
Key Concepts in Evaluation


17

Measuring Excellence

Key Concepts in Evaluation

The previous chapter explained the reasons why program evaluation is pertinent 
to policing and how police departments across the country have used evaluation 
to improve program and agency performance. Grasping the fundamentals of 
evaluation requires learning the terminology, understanding the purposes and 
requirements of various models, and thinking through how the advantages and 
disadvantages of certain approaches apply in specific situations. 

This chapter introduces several key concepts, with which you should be familiar, 
to effectively manage evaluations in your police department. The different types of 
evaluations are defined, followed by an overview of common evaluation designs. 
Information about evaluation types and designs can help you understand how to 
select the best and most appropriate approach for your purposes. Different types of 
evaluation data that can be collected, the benefits and drawbacks of each, and the 
importance of data validity and reliability also are discussed. The chapter concludes 
by introducing the continuous feedback loop of program planning and evaluation, 
and outlining key steps in the evaluation process.

Managing or conducting evaluations in your police department requires a 
basic knowledge of evaluation types, designs, and methodologies and a general 
understanding of what certain types of evaluation can and cannot accomplish. This 
is not to say that you have to be an expert, know how to conduct such evaluations 
yourself, or that evaluations have to be complicated or academic. Rather, conducting 
and managing practical evaluations successfully depends on keeping the process 
organized, of reasonable scope, and directly related to evaluative questions. In other 
words, do not try to do too much and keep things straightforward and realistic. This 
section will educate you about the core concepts and strategies of evaluative inquiry.

The Different Types of Evaluation

Different types of evaluation will be more or less applicable, depending on your 
evaluation questions and needs. The type of evaluation you are going to manage 
or conduct will guide your design and the data you need to collect. 

The first step in deciding which type of evaluation is best suited for your needs is 
to ask yourself what it is you want to know as a result of the evaluation. Generally, 
you can ask one of these four questions: 

1. What is the nature and extent of the problem of interest? 

2. What is my program doing? 

3. Is my program achieving its goals and objectives for program participants?

4. Are our efforts effecting change in the population in which the problem 
initially documented?

The question you choose will direct you to data-collection methods and sources 
that can determine the answer.



There are four main types of evaluative inquiry: needs assessment, process 
evaluation, outcome evaluation, and impact evaluation. The first is associated 
with assessing the nature and extent of the problems that your community (or 
geographic area or jurisdiction) is experiencing. The second focuses on improving 
a program and its activities, and the latter two are linked more closely with 
proving that a program works (or achieves the desired and anticipated results). 

Following are brief descriptions of the four kinds of evaluation as well as a 
cost analysis. Each is illustrated by an example that shows how these kinds 
of evaluations could be applied to a new police department initiative aimed 
at domestic violence. We chose a domestic violence example because this is a 
complex and often hidden crime, and one that demands a sophisticated and 
comprehensive response. Showing how evaluations could be constructed to 
deal with these challenges highlights the options that can be applied to a broad 
spectrum of situations. 

Needs Assessment 

This type of evaluation helps an agency, organization, or coalition determine the 
nature and extent of specific problems in the community and how a problem 
is perceived among diverse groups. A careful, thorough needs assessment is 
the road map for change that provides the basis for a strategic plan to address 
specific problems in the community. Needs assessment includes reviewing local 
archival data to establish baseline measures of problem behaviors (e.g., break-
ins, vandalism, violent crime) and comparing those data to state or national 
data to determine the extent to which a communitys problems are above or 
below state or national levels. Data from surrounding communities also can be 
used for comparison. Needs assessments also should include data from surveys 
and questionnaires and key informant interviews. The goal is to prioritize those 
behaviors that are most problematic in your community and make them the focus 
of your efforts. Examples of questions addressed in a needs assessment are shown 
in Exhibit II-1.


Domestic Violence Needs Assessment

A well-designed and comprehensive needs assessment that focuses on domestic 
violence in your jurisdiction could include gathering the following information:

Determine the extent of the problem. You can use official police 
department data (calls for service, arrests) to establish a baseline to 
monitor over time. Also work with prosecutors, courts, and corrections 
to gather data on how many cases go to trial, average sentences, and 
number of repeat offenders. Although only a fraction of domestic 
violence incidents are reported to police, a thorough needs assessment 
should explore other strategies to get a truer picture of the problem. Such 
strategies might include gathering data from focus groups, surveys, and 
interviews of beat cops who are closest to the community, professionals 
and volunteers who work with victims at community hot lines, personnel 
at domestic violence shelters and hospital emergency rooms, and those 
who work with batterers. Strategies also might include targeted outreach 
to professional womens groups because women from middle- and 
upper-income families often are more reluctant to report problems 
to police. Other data that can aid in understanding the extent of the 
problem would be total deaths and injuries, hospital costs, and workdays 
lost that are attributable to domestic violence. 

Compare rates to other jurisdictions. In addition to comparing 
your data to state and local statistics, you can compare them to nearby 
jurisdictions.

Exhibit II-1

Examples of Questions Addressed in a Needs 
Assessment

What is the nature and extent of the problem(s) in our community? 

How does our community compare with other local communities 
and state and national rates of the problem? 

According to the data, which behaviors are most problematic in our 
community?

What resources are already in place to deal with these problems? 

What strategies or activities are available that have demonstrated 
effectiveness in changing these problem behaviors? 

What is the scope of our efforts to work on this problem? 

end exhibit II-1


Assess trends over time. Do the data show whether the problem is 
getting better or worse? What factors could be influencing those changes 
over time? 

Explore the underlying dynamics. Domestic violence advocates 
might have demographic data about victims and offenders, such as 
age, education, income, cultural and language barriers, and geographic 
incidence. You may also want to gather data on the prevalence of 
substance abuse, including alcohol and illicit drugs. In addition to 
statistics, interviews with experts will increase understanding of prevailing 
theories, such as how the cycle of power and control results in eruptions 
of domestic violence and why victims are often so reluctant to leave. 
Individual focus groups with victims and offenders also could paint a 
better portrait of the problem of domestic violence in your community. 

Describe current responses. Document ongoing police and community 
efforts to deal with the problem, analyzing gaps, shortfalls, and obstacles 
to success. This step allows you to document the policies, practices, 
and procedures within the department that influence or hinder police 
response to domestic violence, and the prevailing laws that might dictate 
arrest policies.

Identify potential partners. In looking for data and candidates to 
interview or involve in focus groups, you are likely to identify community 
members who might be willing to work with you to implement solutions. 
Some to approach are domestic violence shelters, womens groups, 
schools, the faith community, hospitals, physicians and nurses groups, 
and the business community (who suffer economic consequences 
associated with lost employee productivity of domestic violence victims). 

Gather information about opportunities to make a positive 
difference. While a needs assessment is not designed to explore solutions, 
nevertheless it makes sense to provide a mechanism for capturing good 
ideas as you proceed. 

To help you organize a needs assessment in your department, see Appendix A for 
a Sample Needs Assessment Action Plan Template.

Process Evaluation

This type of evaluation answers the question, What does my program look like? 
Data about the activities and tasks that characterize your program or strategy 
are collected primarily through observations of program activities, document 
review (e.g., meeting minutes, call logs, attendance records), and interviews with 
program staff, program participants, and other relevant stakeholders. See Exhibit 
II-2 for some questions that might be included in a process evaluation.


Domestic Violence Process Evaluation 

Imagine that one of the top findings from the needs assessment outlined above 
was that many domestic violence victims said they did not know where and how 
to apply for Personal Protection Orders (PPO) or they felt too intimidated by 
the system to ask for help. The focus groups uncovered victims who did not call 
police because past experience or community gossip made them worry about the 
risk of dual arrest. Domestic violence victims, particularly those with children, 
hesitate to reach out to police if they fear they could be removed from the home. 

The police department decided to launch the first phase of an initiative to 
improve its response to domestic violence, including a public awareness and 
assistance campaign to educate victims about PPOs and make it easier for them 
to apply. The plan included developing brochures and videos about the benefits of 
PPOs and partnering with emergency rooms, schools, Neighborhood Watch, and 
local womens groups to spread the word. The department also would recruit and 
train volunteers to help domestic violence victims complete the paperwork, with 
volunteers working from various community locations so victims being stalked 
would not be easily identified as seeking a court order. 

Exhibit II-2

Examples of Questions Addressed in a Process 
Evaluation

What are the critical components/activities of this project 
(both explicit and implicit)?

What aspects of the implementation process are facilitating success 
or acting as stumbling blocks for the project?

To what extent does the project look and act like the one originally 
planned?

Are the differences between planned and actual implementation 
based on what made sense for the stakeholders and goals of the 
project?

What project strengths can be built upon to improve the program?

Where are the gaps in services/program activities, and how can the 
program be improved to meet those unmet needs?

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 
W.K. Kellogg Foundation, 1998.

end exhibit II-2


To address concerns about police response, the department elected to develop 
and deliver specialized training to dispatchers and line-level officers on how to 
respond appropriately to domestic violence calls. While complaint records did 
not show response to be a serious problem, the chief and training staff felt the 
department would benefit from training in how to encourage victims to come 
forward and to help officers avoid overuse of dual arrest. 

Through observation, interviews, activity forms, and surveys, the process 
evaluation would address questions about the three main components of the 
program:

1. Public Awareness Campaign Process Evaluation. Were the brochures 
and videos developed, tested, revised, and distributed in sufficient 
numbers as promised to the partnering groups? Did the materials 
address language barriers and cultural concerns? Did the partnering 
organizations participate as promised? How many brochures were 
distributed within the time frame and to whom? How many times were 
the videos shown and to whom? What worked, what did not, and what 
could be improved? Were there unexpected obstacles or concerns and 
how were they addressed? Did the initiative identify other materials 
(posters, articles in church newsletters, public service announcements) 
that should be considered? 

2. Personalized Assistance Process Evaluation. Were sufficient numbers 
of volunteers recruited and trained? Was the training adequate (by what 
criteria and according to whom)? Did schedules and locations require 
adjustment? How were those decisions made? How many people were 
served during the evaluation period? Where did participants learn about 
the program? Did they receive a brochure or view the video, or both? If 
not, how did they find out about the effort? What seemed to work well 
and what did not? 

3. Police Training Evaluation. Was an appropriate training curriculum 
developed and delivered? How many dispatchers and police officers 
were trained? Were participants asked to evaluate the training and, if 
so, what were the results? How often did dispatchers and officers have 
the opportunity to apply what they learned in the target area? What did 
police officers do differently during calls for service? 

Additional questions to be answered for all program components are: Were all 
of the activities accomplished in accordance with the proposed timetable and 
budget? If not, why not? Also, have issues or concerns surfaced that must be 
addressed before the program is expanded or replicated? 

The process evaluation should gather all relevant products or written records
copies of the brochures and videos, curriculum and support materials used to 
train volunteers and the police, meeting minutes, calls for service records, and 
officer activity formsto document how activities are implemented and help 
you facilitate program revisions and replication.


Outcome Evaluation

This type of evaluation addresses the questions, What is my program 
accomplishing in the short term? and Am I meeting my objectives? Data 
to measure where you are in relation to program goals and objectives can be 
collected in many ways, but common methods include testing, surveying, 
interviewing, and observing participants. Questions in an outcome evaluation 
might include those listed in Exhibit II-3.

Domestic Violence Outcome Evaluation

For our domestic violence example, three major outcomes are to be evaluated: (1) 
Public Awareness  Did the brochure and video outreach succeed in raising public 
awareness about PPOs and how to apply for one? (2) PPO Assistance  Did this 
initiative increase the number of domestic violence victims seeking a PPO? 3) Police 
Training  Did the training improve police response to domestic violence calls? 

Public Awareness and PPO Assistance Outcome Evaluation. The 
success of the public awareness and PPO assistance hinges on how many 
more people applied for PPOs as a result. We already know the number 
of people who received brochures and saw the video, but the marker for 
success is whether that prompted the desired actiondomestic violence 
victims securing a PPO. 

Exhibit II-3

Examples of Questions Addressed in an 
Outcome Evaluation

What effect is the project having on its stakeholders or participants 
(e.g., change in knowledge, attitudes, or behavior)?

What unexpected outcomes, if any, have resulted from the program?

What can be modified to make the program more effective?

Is there any evidence showing that funders should continue to 
support this program?

Source: National Center for the Advancement of Prevention. Achieving outcomes: A practitioners guide to 
effective prevention. Washington, D.C.: Department of Health and Human Services, Substance Abuse and 
Mental Health Services Administration, and the Center for Substance Abuse Prevention, 2002. 

W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: W.K. Kellogg Foundation, 1998.

end exhibit II-3


Determining program outcomes in this case requires establishing a baseline 
for the number of PPOs that people applied for prior to the intervention and 
comparing that figure to the number requested during an equivalent time 
period after public awareness and PPO assistance activities were launched. Also 
important is identifying other external factors that could influence the outcome 
and consider these when interpreting your findings. For example, since economic 
stress can trigger domestic violence, if there were massive layoffs, plant closings, 
or if the local economy experienced a boom during your studys time frame, 
you would want to note this when interpreting your findings. You would not be 
expected to introduce these factors into your measurement model, but simply 
to think about the extent to which they may have influenced the outcomes you 
are seeing. Also find out whether there might be new prevention activities or 
programs that might affect overall rates of domestic violence in the community. 

Perhaps the number of PPO applications increased 35 percent during the first 6 
months of the new initiative, compared to the same period the previous year, and 
there were no other significant external events that appeared to account for the 
increase. An additional challenge is to determine whether the increase is a modest 
or great success and, as we will see when we examine cost effectiveness, whether 
the benefits justify the cost.

- Police Training Outcome Evaluation. Evaluating the outcome of the 
police training is more challenging. The training was developed to address 
concerns raised in the victim focus groups; however, the extent of the 
problem was never quantified, giving no statistical baseline for easy 
comparison. In fact, the decision to train police was not made because 
problems with dual arrest were verified, but because of a notion that a 
new training component would make sense. To establish a baseline, the 
department might survey domestic violence victims with PPOs who 
placed a call for service before the training took place and compare the 
results to those from a similar group who placed calls after the training. 
If this approach is cost-prohibitive, the evaluation simply might survey 
victims after the training has taken place to determine if perceptions 
about police response improve over time. 

Ideally, the survey would be administered face-to-face by trained interviewers, 
including those who speak the language of the person being interviewed, and 
repeat visits would be scheduled to increase the response rate by developing 
rapport and trust with the interview subject. However, the issues of cost, time, 
and talent must be considered and there often is a tradeoff between conducting 
an ideal evaluation and a more modest one that fits your budget. You must decide 
whether available money might be invested more wisely in delivering a Cadillac 
program with a serviceable, but more modest, Chevy evaluation so that the cost 
of the evaluation does not detract from the program itself. 


Impact Evaluation

This type of evaluation answers the questions, Is my program producing 
long-term, global changes? and Am I meeting my long-term goals? Data 
are collected about the long- or wide-reaching impact of the program. Many 
people confuse outcome and impact evaluation or are not clear about the 
differences between the two. While there are differences in such issues as the 
unit of analysis, the key difference is that an outcome evaluation documents 
short-term or immediate outcomes, while an impact evaluation is focused on 
long-term, more global changes. Using our domestic violence example again, the 
outcome evaluation is focused on increasing the numbers of PPOs applied for by 
individuals. On the other hand, the impact evaluation focuses on reducing overall 
domestic violence in the county, which will take much longer to achieve than 
increasing the number of PPOs. 

Domestic Violence Impact Evaluation

Assuming the three-component program of our example will encourage victims to 
secure a PPO and improve police emergency response to domestic violence calls, 
an impact evaluation will assess whether the new initiative had a positive impact 
on reducing domestic violence in the community. Sample questions that can be 
addressed in an impact evaluation are presented in Exhibit II-4. 

The simplest indicator of intervention success is a reduction in the total number 
of domestic violence incidents. (Success criteria also could include extending 
the time between repeat incidents or a reduction in the severity of individual 
incidents, but those outcomes will not be addressed in this example.) 

While the evaluation strategy appears straightforward and relatively easy 
to accomplish, you cannot rely on a simple comparison of the number of 
domestic violence incidents reported to police before and after the program is 
implemented. Issuing more PPOs and improving police emergency response by 
themselves might not be enough to make a dent in this chronic problem. Success 
also depends on others, such as prosecutors and courts, having an impact, too.

Also, although only a fraction of domestic violence incidents are reported 
to police, the public awareness component of the new initiative might bring 
previously hidden incidents to the attention of the police, as victims become 
empowered to seek help and as concerned neighbors recognize the importance 
of reporting suspected crimes. In fact, the number of reported domestic violence 
incidents actually could rise if the program succeeds in raising public awareness.

The challenge is in how to pinpoint any impact that this focused but relatively 
limited police initiative might have on a complex, chronic, and entrenched 
problem like domestic violence. While the evaluation strategy still would include 
gathering data on the number of incidents reported to police before and after 
intervention, more must be done to analyze what those numbers really mean. 


Incident data could be supplemented with surveys or interviews of professionals 
who work with victims (e.g., domestic violence shelter workers, emergency room 
nurses) to learn whether they witness fewer suspicious incidents. Combining 
those findings with feedback from victims through follow-up interviews or 
periodic repetition of victim focus groups, will supplement statistical data with 
relevant qualitative case studies. 

A combination of skill, creativity, and experience is required to craft powerful 
impact evaluation strategies. If necessary, leave the more difficult strategies to 
professional researchers or at least include them in the evaluation team. 

Cost-Effectiveness Evaluation and Cost-Benefit Analysis

This type of evaluation, which can be part of one of the evaluation types discussed 
above, answers the questions, What does the program cost? and/or What do 
the changes caused by the program cost in relation to the changes that occur 
as a result of program implementation? To answer the first question, data are 
collected about program activities and costs (e.g., salaries, materials, space). To 
answer the second question, data are collected about program outcomes or impact 
and costs. Cost data commonly are collected from accounting or audit records, 
when available. If these data are not available, you can itemize all the resources 
used by the program and estimate costs for each. These and other questions are 
listed in Exhibit II-5.

While it is helpful to compare the costs of program services with costs of services 
otherwise received, you also can compare costs of program outcomes with costs of 
outcomes anticipated if there were no program. 

Exhibit II-4

Examples of Questions Addressed in an Impact Evaluation

What effect is the project having on our long-term goals 
(e.g., change in the number of reported incidents or in rates 
associated with the problem)?

What effect did the intervention activity have on components of the 
system in which the activity was targeted?

Were there any negative outcomes? Are they the result of implementation 
failure or some aspect of the intervention itself?

What degree of confidence is there that the outcomes can be attributed to 
the intervention itself? 

end exhibit II-4


While cost-benefit analysis has been around for a long time, especially in the 
environmental, technology, and physical sciences, it has been applied only 
very recently to social programs like the ones you are implementing. It also is 
a very complex analytic technique that can be difficult to generalize for small, 
or even large, social programs. As a result, you may find it difficult to develop 
a cost-benefit component for your program evaluation without soliciting input 
from a consultant or other outside expert who is well-schooled in cost-benefit 
procedures and techniques. You might be able to do some simple cost estimates 
and then determine whether those costs are offset by the programs benefits 
(e.g., outcomes). To conduct a full cost-benefit study of your program, however, 
particularly a program that is large or has multiple components and target 
audiences, your best bet may be to identify an expert to assist in its development 
and implementation. 

Domestic Violence Cost Evaluation 

This evaluation examines whether the program delivered enough bang for the 
buck. The first step involves calculating costs and comparing them to outcomes 
and impact. Attempting to place a value on human life and human suffering is 
controversial; however, we must remember that relatively modest investments in 
effective programs can have a profound economic as well as humanitarian benefit 
to the community. 

Research shows that real costs are associated with domestic violence, including 
hospital and treatment costs and lost wages for the victim, lost productivity 
experienced by employers, and costs to taxpayers for dealing with batterers. 

Exhibit II-5

Examples of Questions Addressed in a 
Cost-Benefit Evaluation

What does the program cost?

What are the cost savings associated with program implementation 
(e.g., the return on investment)?

Do the costs outweigh the benefits?

Do the benefits outweigh the costs?

Will the cost savings increase with ongoing implementation or will 
they level off? 

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 
W.K. Kellogg Foundation, 1998. 

end exhibit II-5


In 1996, the National Institute of Justice attempted to quantify the costs of crime 
victimization. It calculated, for example, that just one murder cost society $1.03 
million in tangible costs and $1.91 million in intangible costs.

Cost-benefit analysis is more common in business where the focus is on profits 
but when the goal is achieving justice, such tools also can help decision-making 
about investing scarce resources.

Discussion 

These brief examples underscore how crafting a comprehensive evaluation strategy 
for any police program or intervention requires careful thought and planning. 
Each type of evaluation can inform any others you might conductbaselines 
identified in the needs assessment can provide the basis for comparison in process 
and outcome evaluations, and the findings in outcome evaluations can enrich 
the analysis of impact evaluations. Moreover, all evaluations can illuminate 
opportunities and obstacles and identify potential new partners. A summary of 
the evaluation types discussed above is in Exhibit II-6.


Evaluation Designs

The type of evaluation you select, based on the questions you want answered 
about your program, will determine what evaluation design you will usewhat 
you will measure and how often or when you will measure it. The evaluation 
design is all-important because it affects what you are able to know about your 
program and its impact. 

As we introduce some of the most common evaluation designs, remember that 
some will be more feasible than others in the policing context. In fact, program 
evaluation has its roots in the fields of medicine (curing disease) and psychology 
(changing behavior), in which treatments are applied to groups of individuals, 
allowing researchers to identify statistically valid differences in outcomes. 
Evaluating police treatments (interventions) can mean adapting evaluation 
techniques to identify whether a program has had an impact on a geographic 
area (reducing drug dealing in a target neighborhood) or a problem (reducing 
physical disorder by organizing a cleanup) rather than on individuals within a 
group. Consequently, your internal evaluations are most likely to use post-test-
only designs and least likely to use quasi-experimental or experimental designs. 
Nevertheless, in this section we have included information on all designs to offer 

Exhibit II-6 

Overview of Evaluation Types table

Table column headers: Evaluation Type, Process, Outcome, Impact, Cost

Questions 
Answered

Process -
What is the program 
doing?

Outcome -
Is the program 
achieving its goals 
and objectives?

Impact -
Is the program 
effecting change?

Cost -
What does the 
program cost? 
What do the 
changes caused by 
the program cost?

Sample Variables

Process- 
Number of 
brochures 
distributed to ERs 
and schools

Number of 
volunteers 
recruited and 
trained; number 
of police officers 
trained

Number and 
content of 
training session 
topics

Training style

Outcome -
Increase in 
the number of 
applications for 
PPOs

Increase in public 
awareness about 
how to apply for 
a PPO

Improved 
police response 
to domestic 
violence calls

Impact -
Decrease in 
the number 
of domestic 
violence 
incidents

Increase in time 
elapsed between 
repeat incidents 

Cost -
Cost per response

Cost per PPO filed

end exhibit II-6

you a broader understanding of the benefits and limitations of each. 
This information will help you appreciate what you can and cannot learn about 
your programs and activities, depending on which design strategy you follow.

Five common evaluation designspost-test only, pre- and post-test, pre- post- 
post-test, quasi-experimental, and experimentalare described below and 
presented in order of complexity and expertise required, from the simplest to 
the most rigorous and elaborate (Shadish, et al., 2002). Classic evaluation design 
tables are read from left to right, first noting how many groups are involved in the 
research and continuing by noting the order in which measurements are taken 
and treatments applied. Keep in mind that in program evaluation, a pre-test or 
a post-test refers to a measurement (either through an interview, survey, or other 
data-collection method) used to document change in knowledge, attitude, or 
behavior. 

For the post-test-only evaluation design, data are collected only once from the 
program or from people who live in the target area. Using this design within 
a program delivered to the target group, participants are measured after they 
have gone through the program. Post-test-only measures are the least helpful 
for evaluating the impact of a program because there often is no valid baseline 
to which the post-test results can be compared. To assess change, you must be 
able to compare post-program data with a baseline measure. Although this is 
the weakest form of evaluation design, it can be used when it is important (or 
feasible) to know if participants have reached an identified outcome, rather 
than to measure degree of change. Also, if you have limited resources (included 
limited access to program participants), this could be the best or only option. 
For example, consider a grant that funds a multi-agency federal, state, and local 
task force to collaborate in a yearlong effort to take illegal guns off the street. At 
the end of that year, the task force might conduct a post-test-only evaluation that 
shows total number of guns seized, with breakdowns about kinds of weapons and 
the places where they were seized. Such an evaluation also would include data 
on the number of cases investigated, number of arrests made, and a profile of a 
typical offender.

Post-Test Only Design table

Group - Treatment Group

Pre-measure - null

Treatment - X

Post-measure - O

X = program intervention is applied

O = measurement is taken


The pre- and post-test design allows you to measure an individual or groups of 
individuals at two different timesbefore and after the interventionenabling 
you to assess change by comparing the two measurements. This evaluation 
design has the benefit of being relatively easy to implement because you simply 
administer the same measure twicebefore and after the intervention. You must 
carefully determine when to take the post-program measure, making sure to allow 
enough time for your program to have an effect.

While the pre- and post-test design is an improvement over the post-test-only 
design, it is not perfect. Even if your evaluation reveals a change in what was 
measured, you cannot be absolutely sure that your program is responsible for 
that change (i.e., you cannot determine causality). Many other factors that were 
not accounted for in your design could be at play, and it might be one of those 
factors, not your program, that had an impact. Without a well-defined program 
and sample, this type of evaluation design cannot be accomplished. 

To illustrate, consider the issues involved in evaluating a new police effort to 
reduce prostitution in a specific neighborhood where residents have complained 
that the problem is escalating out of control. A pre- and post-test design requires 
establishing baselines that will be measured after the intervention so, in this case, 
indicators likely would include the number of citizen complaints and number of 
arrests made, with the expectation that success would be reflected in a decline in 
complaints and a rise in arrests. 

As mentioned above, the challenge is making sure there are not other factors 
operating in the community that influence results. In other words, how do you 
know that it was your efforts, and your efforts alone, that caused any changes 
that are measured at post-test? Maybe there was a dramatic decline in complaints 
because the people who run prostitution in the neighborhood began intimidating 
residents who then feared talking to police. Or perhaps local public health 
officials received a grant to launch a comprehensive effort to reduce prostitution 
through an outreach that offers substance abuse treatment and job training. These 
circumstances can be addressed only through experimental and, to a lesser degree, 
quasi-experimental designs (described below). 

Pre- and Post- Test Only Design table 

Group - Treatment Group

Pre-measure - O

Treatment - X

Post-measure - O

X = program intervention is applied

O = measurement is taken


Among the benefits of a community policing approach to chronic problems like 
prostitution is not only that police will be more likely to find out about other 
community-based efforts, but also that collaborating to address the problem 
offers the best chance of success. By partnering with other agencies such as public 
health, the police also might enjoy benefits from being able to tap into their 
evaluation capabilities, and this may result in a stronger evaluation design and a 
greater understanding of whether police efforts were responsible for any changes 
measured at post-test.

Pre- Post- Post-Test Design table
Group - Treatment Group

Pre-measure - O

Treatment - X

Post-measure - O O

X = program intervention is applied

O = measurement is taken

The pre- post- post-test design provides measures at three distinct points in time. 
A pre-test measure is taken before any exposure to the program. The first post-test 
measure is taken immediately at the end of treatment, and the second (or follow-
up) post-test measure is taken some time (e.g., 3, 6, or 12 months) after program 
completion. The advantage of this design is that it allows you to understand if the 
program treatment had any sustained impact or to know if program participants 
retained knowledge after a period of time. 

This design has the same weaknesses as the pre-post design in its ability to 
determine causality (i.e., to show conclusively that the program treatment is 
responsible for any changes measured). In addition, finding people to participate 
in the second post-test can be difficult and costly. Furthermore, people who 
are measured a third time could become bored or annoyed with the same 
measurement instrument, thereby affecting their responses. However, if we 
think of the prostitution example again, it might be valuable to examine citizen 
complaint and prostitution arrest data again 6 months after the intervention ends 
to see whether the department should renew its effort before the problem escalates 
again. This approach represents a practical way for police departments to achieve 
more longitudinal data collection and evaluation. 


The quasi-experimental design comes one step closer to achieving the 
confidence level whereby you know that it was your program that caused change. 
In this design, you compare your target (or treatment) group (the group, 
neighborhood, or community that is the target of the policing effort or program) 
with another similar group that did not receive the program (i.e., a comparison 
group). Using quasi-experimental design, you would pre-test both groups before 
the program begins then deliver the program to only one group. You would 
administer a post-test to both groups, analyze the changes, and compare the 
results from both groups.

The challenge of the quasi-experimental design is to find two or more groups that 
are sufficiently similar in demographics (i.e., gender, ethnicity, socioeconomic 
status, education) and situation (i.e., both groups are adolescent boys at risk 
for juvenile delinquency) to be truly comparable. Your ability to say for certain 
that those who went through the program were the same as those who did not is 
limited, affecting your ability to say your program is responsible for differences 
that might exist between the groups after program intervention. The more alike 
the two groups are, the more confidence you can have that your program was 
responsible for positive outcomes.

The quasi-experimental design is frequently the best option for conducting 
outcome evaluations in the social services context, as it is often the only practical 
and ethical way to study two or more groups who vary in the programming they 
receive, because it does not rely on random assignment (see the experimental 
design example that follows). Additionally, to maintain and study a control 
group can be costly, which is even more significant because that group does not 
benefit from program treatment and already limited resources are diverted from 
programming. The quasi-experimental design averts these problems by using 
groups that already exist or that are not assigned randomly. An example of a 
comparison group is in a school setting where one class that participates in a 
program is compared to another class that does not participate. 

Quasi-Experimental Design table

Group - Treatment Group

Pre-measure - O

Treatment - X

Post-measure - O

Group - Comparison Group

Pre-measure - O

Treatment - null

Post-measure - O

X = program intervention is applied

O = measurement is taken


This kind of research design is used more commonly by criminal justice 
researchers, often in concert with police, rather than by police alone. 

A jurisdiction moving toward a preferred or mandatory arrest (pro-arrest) policy 
in domestic violence situations could conceivably apply the policy in one area 
or sector and not another and then compare results. This would be an example 
of a quasi-experiment. Obviously, there are challenges involved in training and 
supervision to ensure that officers are applying the policies appropriately. Also 
of concern is that such policies require the support of the county prosecutor or 
district attorneys office, since the point of pro-arrest polices is to send the message 
that domestic violence will be prosecuted with or without the full participation 
of the victim. It also goes without saying that, if pro-arrest policies do indeed save 
lives and reduce injuries, there are ethical concerns about whether it is or is not 
appropriate to withhold the intervention to the victims who happen to live in the 
area where the new policies do not apply.

This example qualifies as a quasi-experimental design because there could be 
differences between the geographic areas being compared that could affect the 
results of the evaluation. For example, arrest might carry more of a stigma in 
one area than another, so the impact of a pro-arrest strategy might show greater 
promise in an area where arrest induces shame than in an area where arrest is a 
more common occurrence. 

Experimental Design table

Group - Randomly SelectedTreatment Group 

Pre-measure - O

Treatment - X

Post-measure - O

Group - Randomly Selected Control Group

Pre-measure - O

Treatment - X

Post-measure - O

X = program intervention is applied

O = measurement is taken


In experimental design, people from the same target population are assigned 
randomly either to a treatment group or a control group. Random assignment 
means that each person in the target population has an equal chance of being 
selected for either group. This randomness is crucial, because it is the only way 
to ensure that two groups are similar at the beginning of the intervention period. 
With this design, as with quasi-experimental design, both groups are pre-tested, 
only one receives the program intervention, and then both groups are post-tested.

Experimental design offers the best opportunity to know (and claim to others) 
that your program is responsible for changes in outcomes. It is used most 
often by researchers in the physical sciences because they can control their lab 
environments, repeat experiments, and determine causality. For obvious reasons, 
it is much more difficult to implement valid experimental designs in the policing 
context. Ethical concerns regarding experimental designs with humans often 
render experimental evaluation of human services unfeasible. Additionally, it 
might cost more to implement an experimental design and, despite the extra 
expense, several threats to the validity of the study would still exist. Threats to 
validity are presented in Exhibit II-8. Such threats are nearly impossible to control 
in real-life settings.

If we think of the pro-arrest example above, the logistics involved in randomly 
assigning officers to use a pro-arrest strategy on one call and then not on the next 
pose serious logistical and ethical questions. We also know, however, that pro-
arrest policies can result in problems with dual arrest, where frustrated officers 
find it easier to arrest both individuals involved in the altercation than to try to 
differentiate between victim and offender. Training half the force in strategies that 
are designed to reduce the problem of dual arrest might allow for the randomness 
required by the experimental design. Then, if the results prove positive, it would 
justify investing in training the rest of the force.

To select the design that is right for your department, consider your needs and 
resources and pick the most rigorous design possible, given those restraints. Each 
type of design has its advantages and disadvantages, so be realistic about what 
kind of design you can implement. A post-test-only design might not produce the 
information you require, but quasi-experimental and experimental design studies 
could be too complex to perform without outside help. To manage evaluation 
effectively in your department, do not waste time and resources committing 
to a design that is beyond your capabilities or will not provide the information 
that you need in the end. In most cases, you should be able to find a happy 
medium. The different types of evaluations and their strengths and weaknesses are 
summarized in Exhibit II-7.


Exhibit II-7

Strengths and Weaknesses of Different Types of Evaluation table

Table Headers: Design, Characteristics, Advantages, Disadvantages, Required 
Expertises

Post-test only

Measures one 
group at one time 
onlyafter the 
treatment.

Can document 
the reduction of a 
problem; requires 
access to only one 
group at only one 
time.

There is often no 
valid baseline 
measure for 
comparison. Does 
not allow one to 
assess change from 
baseline.

Low

Pre- and Post-test

Measures 
one group at 
two different 
timesbefore 
and after program 
completion

Provides a baseline 
measure; requires 
access to only one 
group.

Cannot know for 
certain that your 
program produced 
the change.

Moderate

Pre- Post- Post-test

Measures a group 
at three different 
timesonce before 
the program and 
twice after program 
completion.

Enables you to 
determine if there 
are sustained effects 
from your program.

Does not show 
conclusively that your 
program produced 
the change; may be 
difficult to follow up 
with everyone.

Moderate

Quasi-experimental

Measures two 
groups at two 
different times
before and after 
the program; 
one group does 
not receive the 
treatment program.

Offers a decent 
level of confidence 
that your program 
caused the observed 
change.

May be difficult to 
find two groups that 
are similar enough 
to be compared; is 
more costly to have 
comparison groups.

Moderate to High

Experimental

groups whose 
members are 
randomly assigned 
at two different 
timesbefore and 
after the program; 
one group does 
not receive the 
treatment program.

Provides the highest 
level of confidence 
that your program 
caused the observed 
change.

Difficult to 
implement in a 
real-life setting; there 
are ethical issues 
involved with the 
random withholding 
of a beneficial 
program from one 
group.

High


Different types of data

Once an evaluation design has been chosen, focus on what types of data you will 
collect to answer your research questions and the data-gathering methods you 
will use. Quantitative data measure something that can be counted and assigned 
a number, and can be collected using several methods, including simple counts, 
surveys, and tests. Police departments often rely on quantitative data such as calls 
for service, arrests made, and citations issued. You might also take a simple count 
of how many people attended a neighborhood community meeting or calculate 
the percentage of survey respondents who are satisfied with police services. 
Quantitative measurement often assesses the extent and direction of change that 
has taken place as a result of the implementation of a strategy or program, and 
statistics often are used in quantitative data analysis. Quantitative measurement 
has several advantages. First, it yields data that can be standardized across a large 
number of people, can be replicated easily, is conducive to making comparisons, 
and can be less expensive and time-consuming to collect than qualitative data. 
However, quantitative data sometimes give a simplistic representation of reality 
and have been criticized for lack of attention to participant input. In addition, 
survey questions must be developed carefully to ensure they will capture the 
information you intend to measure.

One common problem that police face is that easily quantifiable data may not 
be kept in a form that suits their needs. For example, imagine that a department 
is planning a new effort to reduce the number of commercial burglaries. You 
find, however, that your call for service data does not make a distinction between 
commercial and residential burglaries and that there is no easy way to disaggregate 
the data. We know instinctively that there are many more home burglaries than 
commercial burglaries, so even if the new program is very successful, the data 
might not show enough of a reduction in the total number of burglaries to verify 
success. Do you delay the intervention until you can persuade dispatch to begin 
capturing burglary data in the format you need? Or do you pay someone to check 
the address for each burglary call to see whether it is a commercial or residential 
address? 

Qualitative data are collected through activities such as observations, interviews, 
focus groups, and even conversations. They describe the attributes of an object 
without referring to quantity and are expressed in words, not numbers. Instead 
of measuring the extent and direction of change or program impact, qualitative 
data are better suited to describing the nature of the change or impact that has 
taken place. Qualitative data are used frequently in process evaluation to describe 
how a program functions. In outcome evaluations, qualitative data are useful 
for providing context to and humanizing the numbers. Qualitative methods 
capture nuances and insights that statistical analysis of survey instruments 
cannot measure. Qualitative data provide context and can be more interesting 
and informative for a lay audience than a technical report filled with numbers 
and statistics. However, qualitative data can be overwhelming and cumbersome 


to collect and analyze, and because data collection generally is more time-
intensive, it is more expensive. Moreover, qualitative data lend themselves less to 
comparison and generalization than quantitative data.

In 2003, the Michigan Bureau of Juvenile Justice launched its yearlong Building 
Restorative Communities (BRC) initiative in five counties, designed to help 
build the capacity to identify and implement evidence-based, data-driven, and 
outcome-focused interventions to reduce juvenile crime. The core group of 
police, prosecutors, courts, and juvenile justice officials who work with at-risk 
youth helped mobilize a cross section of community groups to partner with them 
on developing a comprehensive 5-year plan. A grant allowed each county to hire a 
coordinator and a part-time evaluator for the project. 

One of the six work groups designated for the project was tasked with hosting 
separate focus groups of juvenile offenders in detention and victims of juvenile 
crime. Focus group facilitators were given a script with questions for each group, 
and a list of themes they could use to sort participant comments into useful 
categories. One question for the juveniles was, What could have helped you stay 
out of trouble? The findings would assist the steering committee work group 
to prioritize its final recommendations. This project confirmed that qualitative 
research produces significant amounts of important information but that it is 
not as easy to convey (or convey quickly). In Monroe County, with only 15 
participants in the juvenile offender focus group and two participants in the 
victim focus group, the final reports totaled more than 15 pages.

Both types of data have advantages and disadvantages, therefore, it is now widely 
accepted that using multiple methods and mixing the methods is the best 
strategy for gaining the most complete understanding of a program or strategy 
and its effects. In fact, the use of mixed methods increasingly is considered a 
prerequisite for quality data.

A discussion of the different methods that can be used to collect quantitative and 
qualitative data is presented in Chapter III: Conducting an Internal Evaluation. 
Whichever data instruments you choose, you must consider the ability or 
suitability of the instrument to capture the most valid and reliable data possible. 
The validity of data collected hinges on whether the data collection instrument 
measures what it intends or claims to measure. For example, crime statistics are 
not a valid measure of citizen satisfaction with police services. The reliability 
of data collected is a term used to describe an instruments ability to record 
consistently what is being studied. For an instrument to be reliable, it must yield 
the same (or similar) results even when administered in different places or at 
different times. Instruments must be both valid and reliable for you to claim that 
you have collected quality data. Exhibit II-8 summarizes several common threats 
to the validity of evaluation results.


Exhibit II-8

Common Threats to the Validity of Evaluation Results

History: Any event that happens between the time of your first 
measurement (pre-test) and your second (post-test) that can affect the 
measurement. 

Maturation: The normal processes of development that occur over time 
could affect your outcomes, independent of your intervention. For 
example, as children grow and mature, they are likely to develop longer 
attention spans, according to available research on child development. 
This naturally occurring phase of development may occur while the child 
is enrolled in your program, making it difficult to separate program effects 
from normal developmental growth.

Testing: These are the effects that test taking have on the study results. 
Taking a pre-test may influence the behavior of your subjects as much or 
more than your intervention does. You cannot be sure that the effects you 
see are caused by your intervention and not by the administration of a 
pre-test.

Instrumentation: Any flaws in your measurement device that can skew 
your results. For example, if you collect information from an observer, he 
or she may be less attentive to collecting data on one occasion than on 
another.

Statistical Regression: A general tendency for extreme scores on any 
measure to move toward a less extreme score at a different time. If you 
measure something at the lowest possible amount, there is no way for it 
to change except to increase. Statistical regression occurs when you select 
study participants based on their extreme scores (e.g., if you decide to 
study only very depressed adolescents). When you remeasure, there is a 
tendency for these extreme scores to move back toward the overall group 
average regardless of the effects of any intervention provided.

Selection Bias: Any determination, except random placement, by which 
you place some subjects in a treatment group and other subjects in a 
control group is a type of selection bias. Subjects assigned to groups in any 
way other than random placement opens the possibility that the groups 
differ in important ways that can affect the results of your study. For 
example, students who volunteer to attend a tutoring session may be more 
academically motivated than students who do not volunteer to attend this 
group. Different levels of motivation may affect these two groups as much 
as or more than the tutoring groups did. Random assignment to groups is 
the best way to protect against the effects of selection bias.

Source: Mertinko, E., L.C. Novotney, T. Baker, T., and J. Lange. Evaluating your program: A beginners 
self-evaluation workbook for mentoring programs. Washington, D.C.: Office of Juvenile Justice and 
Delinquency Prevention, Office of Justice Programs, and the United States Department of Justice, 2002. 

end exhibit II-8


The Continuous Feedback Loop of Program 
Planning and Evaluation and the Key Steps in the 
Evaluation process

In managing or conducting an evaluation, it is critical to know where evaluation 
fits into the larger scheme of implementing police programs and strategies. 
Ideally, programs, projects, initiatives, and strategies are planned and carried out 
with a continuous feedback loop, of which evaluation is an integral part. Exhibit 
II-9 illustrates how evaluation is part of a continuous cycle of program planning, 
implementation, and refinement, and emphasizes the importance of connecting 
evaluation with program improvement for maximum results.

Exhibit II-9

The Program Planning and Implementation Cycle

Evaluate 
the program/initiative

Compare 
and analyze results against 
baseline and/or objectives

Specify (Plan)
what is to be collected 
and how

Collect 
information with 
appropriate tools

end exhibi II-9

The specific steps in evaluation also can be detailed, as shown in Exhibit II-10. 


The evaluation cycle itself is composed of several steps, which can be followed 
without much evaluation expertise; however, there may be times when you need 
outside assistance in developing or implementing an evaluation step, depending 
on the experience of your team and the complexity of the evaluation. Whether 
you are conducting an internal evaluation or managing an outside one, you 
will need some specific information about all the evaluation cycle components: 
evaluation planning, data collection, data analysis, and using and disseminating 
evaluation findings. Chapters III and IV in this guide provide more details about 
these components.

Exhibit II-11 highlights the integration of the planning and evaluation cycles.

Exhibit II-10

The Evaluation Cycle

Plan 
the program/initiative

Refine
the program based 
on evaluation results

Implement
the program/initiative

Evaluate 
the program/initiative

end exhibit II-10


Exhibit II-11:

Integration of the Planning and Evaluation Cycles

Plan 
the program/initiative

Refine
the program based 
on evaluation results

Implement
the program/initiative

Evaluate 
the program/initiative

Compare
analyze results against 
baseline and/or objectives

Specify
what is to be collected and how

Collect 
information with 
appropriate tools

end exhibit II-11


Chapter Summary

Chapter II presented an overview of evaluation types and designs and described 
evaluations place in the cycle of program implementation and improvement. 
This chapter also covered the distinction between process, outcome, and impact 
evaluation and the pros and cons of several evaluation designs. Understanding 
these key evaluation concepts and the steps in the evaluation process will 
prepare you to conduct and manage valuable evaluation research in your police 
department. 

Evaluation is a powerful tool and the process is logical and relatively easy to 
grasp, particularly when you follow the logic model approach and other useful 
tips outlined in the next chapter. The rest of this guide concentrates on particular 
steps in the evaluation cycle, including specifics on how to conduct internal 
evaluations and manage them successfully so that results are disseminated and 
used effectively for program refinement and improvement. Chapter III provides 
details and guidelines, including examples and worksheets, for conducting an 
internal evaluation, while Chapter IV offers practical tools and instructions to 
help you manage evaluative research in your police agency.


Chapter III:
Conducting an Internal Evaluation



Conducting an Internal Evaluation

An internal evaluation is one in which the police agency relies primarily on 
people within the department to conduct an evaluation of policing efforts and 
to disseminate results. Internal evaluations are designed to help police agencies 
learn more about the operation, performance, and perceptions of all or part of 
their departments. Evaluations can vary from a process evaluation to determine 
whether a new call management system has been implemented as directed, to an 
outcome evaluation of a narcotics task force, to a community satisfaction survey 
as one indicator of the long-term impact of community policing on attitudes 
about police.

The following are some benefits of an internal evaluation:

The evaluation will reflect staff knowledge about the program.

The evaluation may be more successful in getting support from police 
administration and other staff.

The evaluation may be less expensive than other evaluation options.

While an internal evaluation might meet the needs of the program staff and 
administrators better, the following are some potential criticisms of internal 
evaluations:

The evaluation can be perceived as less objective and less credible.

The evaluation might not be adequate for answering the evaluation 
questions.

Perhaps the biggest misunderstanding about internal evaluation is that it must 
follow an academic research model, which focuses on testing hypotheses. Most 
law enforcement agencies should focus instead on answering practical program 
and policy questions. While academic research requires a more controlled 
approach with comparison and control groups, this is seldom realistic for a law 
enforcement agency operating in a real community with many uncontrolled 
variables and where using a control group is not feasible or is unwise. For an 
internal evaluation designed to improve programs and practices, you might not 
need such control or sophisticated statistical analyses. The methodology for 
practical research should be based on identifying program or effort that is being 
evaluated, the type of data that can be collected, and the availability of resources 
for data collection and analysis. 

Some of the primary differences in purpose, method, and statistics between 
academic research and internal evaluation are presented in Exhibit III-1



If evaluation results are intended mainly for internal use, the purpose, method, 
and statistics of academic research often are too costly for most law enforcement 
agencies or are beyond their scope. Nevertheless, employing evaluation procedures 
that are explicit, formal, and justifiable is important so that police departments can 
have confidence in the validity of their findings. Law enforcement agencies, therefore, 
should do the best evaluation they can with available resources, carefully noting the 
limitations of findings. Ultimately, the standard by which your evaluation should be 
judged is whether the collected information was used to make decisions or 
improve the program (Loe, 2001). 

If you have decided, based on your evaluation needs and resources, 
that conducting an internal evaluation is your next step, the rest of this 
chapter provides an overview of how to conduct an evaluation, including 
developing an evaluation plan, determining data-collection strategies, 
and conducting data analysis. Several tips and practical worksheets are 
included throughout the chapter and in the appendices. The resource list 
in Appendix L directs you to more evaluation references. 

Exhibit III-1

Difference Between Academic Research and 
Internal Evaluation table

Table Headers: Academic Research, Internal Evaluation

Purpose

Test hypotheses; scientifically 
test a program model; advance 
the overall literature and field of 
law enforcement

Improve program and practice; 
address the needs of a single 
law enforcement agency or a 
group of agencies

Method

Controlled environment

Context-sensitive

Statistics

Sophisticated

Simple

Source: The University of Texas-Houston Health Science Center School of Public Health, and The Texas Department 
of Health. Practical evaluation of public health programs. Atlanta, Georgia: The Centers for Disease Control and 
Prevention and The Association of Schools of Public Health Cooperative Agreement, 1998.

end exhibit III-1

How We Did It: 

Colorado Springs Police Department

The Colorado Springs Police Departments 
(CSPD) Police Accountability and Service 
Standards (PASS) model demonstrates how 
internal evaluations can produce valuable 
information to both the law enforcement 
agency and the community. CSPD uses 
internal data collection and analysis of 
its operations and programs to improve 
policing efforts and ensure that they are 
meeting community expectations.


Steps in the Evaluation Process

The four main steps in the evaluation process were introduced in Chapter II: 
SPECIFYevaluation planning, COLLECTdata collection, COMPARE
data analysis, and COMMUNICATING and using the results. The first three 
steps are the focus of this chapter. The final step is a management task and is 
discussed in greater depth in Chapter IV.

Step 1: SPECIFYEvaluation Planning. An evaluation plan provides a 
framework that will help you specify what to evaluate and how. A clear 
evaluation plan facilitates not only evaluation implementation but also 
its management. Evaluation planning steps that will be discussed in 
this chapter include the following: assessing whether your program is 
ready to be evaluated, assessing stakeholders commitment to evaluation 
and planning for their involvement, understanding and defining your 
program goals and objectives, developing a logic model, determining your 
evaluation questions, and selecting an evaluation design based on the 
careful planning you have done.

Step 2: COLLECTData Collection. Your evaluation data will be 
collected through surveys, interviews, focus groups, document review, or 
other methods, depending on what has been specified in your evaluation 
plan. After determining what data already exist and will be useful, you 
might still need to develop your own data- collection instruments. Tips 
for creating and implementing surveys, interviews, and focus groups are 
included in this chapter.


Step 3: COMPAREData Analysis. The data you collect will be analyzed 
according to your evaluation plan and the data-collection instruments 
that have been used. The purpose of analysis is to reduce, categorize, and 
summarize your data in a way that helps you and others understand what 
is happening with your policing programs and activities and/or what is 
changing as a result of those efforts. Evaluation data often are analyzed 
using simple counts, statistics, and content analysis. This chapter gives a 
brief overview of each.

Step 1: SPECIFYEvaluation Planning

Preparing for any evaluation requires an investment of time and thought. 
Attention to planning each evaluation step will increase the likelihood of 
producing useful results. Evaluation planning helps you carry out activities 
efficiently and effectively and allows someone outside the organization to 
understand what and how you are evaluating. 

Many police agencies make the mistake of delaying their thinking about evaluation 
as long as possible, often until after the program is underway. Treating evaluation 
as an afterthought not only jeopardizes its quality, but also robs the department of 
the benefits that evaluation can deliver to planning and implementation. Thinking 
through the evaluation plan while the program or intervention is being developed 
helps clarify what should be done and how. Identifying the various measurements 
that will be used also focuses participants on what it will take to make the initiative 
a success. Delaying evaluation planning also might make it more difficult to gather 
the data you need or to put it in the format that best suits your purposes than if you 
had explored various options earlier. 


Perhaps the biggest advantage of early evaluation planning is that it can allow 
police to make course corrections along the way, increasing the odds of success. 
Police executives who worry that evaluation could trumpet bad news should 
understand that it is in their interest to engage in evaluation planning in advance, 
incorporating opportunities to monitor progress so that appropriate changes and 
improvements can be made quickly. 

Evaluability Assessment

Not all programs can or should be evaluated because some are too small, too 
short, or are ill-conceived. For others, it might be too early in their development 
to conduct an outcome or impact evaluation. As part of planning, it can be 
helpful as a first step to identify whether your program is ready to be evaluated. 
To do so, you can conduct an informal evaluability assessment, which lets you 
verify whether the program has clearly specified activities and well-articulated and 
measurable goals and objectives that lend themselves to identifying data that can 
be collected. This process includes reviewing the program history, design, and 
operation, perhaps by observing the program in action; determining the programs 
capacity for data collection; and assessing the likelihood the program will meet 
its goals and objectives based on implemented activities. While the concept of an 
evaluability assessment might be unfamiliar, this type of preassessment increases 
the likelihood that a full-blown evaluation will provide timely, relevant, and 
responsive findings (Juvenile Justice Evaluation Center, 2003). Poorly planned 
or implemented programs should not be evaluated because findings will be 
inconclusive at best and misleading at worst. Therefore, it is often in a programs 
best interest to invest the resources necessary for either a formal or informal 
evaluability assessment (Juvenile Justice Evaluation Center, 2003). If your 
evaluability assessment results show a program is not ready for an evaluation, an 
outside evaluator might be able to work with the program manager and staff to 
bring the program to the appropriate stage. 

In addition to helping determine whether to conduct an evaluation, data collected 
from an evaluability assessment can describe the sequence of critical elements 
that influence program design, implementation, and effectiveness, and they can 
direct you toward developing suitable measures and research designs. This often is 
graphically represented in a logic model, which is described in greater detail later 
in this chapter.

Planning Stakeholder Involvement and Assessing 
Stakeholder Commitment

Another step in evaluation planning is to identify key stakeholders and assess how 
committed they are to an evaluation. In addition to your police department, an 
evaluation stakeholder could be the agency or organization that funds the grant 
under which your program operates. For example, if you are receiving funds 


from the Department of Justice to implement a gang-prevention program, you may want to 
talk with your federal program manager to see if he or she would supportive an evaluation. 
Other important stakeholders might include local government, members of the business 
community, schools, and nonprofit agencies, as well as community residents who have been 
involved in identifying problems and finding solutions. The following questions will guide 
you in thinking about the level of stakeholders commitment and the ways in which they 
might facilitate your evaluation:

What might be the concerns stakeholders have about conducting evaluation at this 
time? How might obstacles be overcome?

What are the reasons that stakeholders might support an evaluation at this time? 
What questions do they have about the implementation or effectiveness of policing 
efforts?

What information do the stakeholders need about your program? How will key 
stakeholders use information from this evaluation?

What information or expertise can stakeholders contribute to your evaluation? What 
are stakeholders willing to contribute to an evaluation at this time?

How can evaluation help make a persuasive argument for community support?

Understanding stakeholders concernsand what they might be most enthusiastic about
can help in planning a more effective evaluation strategy. At the same time, addressing their 
concerns and interests generates buy-in. Stakeholder commitment often can be increased by 
establishing an advisory group or evaluation team. A well-constituted advisory group, made 
up of the evaluator, representatives from the police agency, program participants, and/or 
interested citizens, can be an asset for formulating evaluation questions, providing additional 
evaluation resources, and being advocates and spokespeople for the dissemination of results 
(Center for Substance Abuse Prevention, 2003). 

The Evaluation Plan

An evaluation plan documents a system for collecting evidence of 
program effectiveness and will help ensure that you are collecting the 
information you need in the most meaningful way. Developing your 
plan means defining carefully what to evaluate and how to measure 
success, formulating evaluation questions, selecting the appropriate 
evaluation design, and developing data-collection methods. During 
evaluation planning, you also should think through a project timeline 
and budget and a strategy for using and disseminating your findings. 
These management issues are discussed in the chapter Managing 
Evaluations. 

Key Steps in Evaluation 
Planning

Determine whether the program is 
ready for evaluation. 

Assess stakeholder commitment to 
evaluation.

Define program goals and objectives.

Develop a program logic model.

Determine evaluation questions.

Choose an evaluation design.

Plan for data collection and analysis.


Determine What Is Going To Be Evaluated

Although it might seem obvious, determining what is going to be evaluated could 
be difficult if your program was not well defined at the outset or if activities 
were not designed and implemented so they could achieve the stated goals or 
produce meaningful benefits. Before conducting an evaluation, be clear about 
the programs purpose, its goals and objectives, and the activities that have been 
put in place to meet them. Understanding the relationship among them also 
might help determine whether you want to examine the whole program or just a 
particular component. For instance, one program component might be selected 
if you think it has the strongest effect on participants, or if it requires the greatest 
proportion of your resources and, therefore, you are particularly concerned about 
its effectiveness. 

Define Program Goals and Objectives

Understanding your programs mission, goals, and objectives will give you ideas 
about how success can be measured in your evaluation. Before evaluation begins, 
therefore, become familiar with your programs goals and objectives and know 
whether they are measurable. If goals and objectives have not been articulated 
or written, you must work with your evaluation team and stakeholders to do 
so to ensure you do not undermine your results. See Exhibit III-2 for tips for 
developing goals and objectives.

A goal is a measurable statement of the desired longer term, or global, impact of 
the program. Goals typically address change. For example, recalling our domestic 
violence example from Chapter II, a goal might have been, By 2006, our 
program will reduce domestic violence in our county by 10 percent. 

An objective is a specific, measurable statement of the desired immediate or direct 
outcomes of the program that support the accomplishment of a goal. Again, using 
our domestic violence example from Chapter II, an objective could be, Our 
program will increase the number of personal protection orders (PPOs requested 
in our county by 50 percent. 

Note that objectives state the desired immediate or direct outcomes of 
programmatic efforts; therefore, it is useful to distinguish between outcomes and 
outputs. Outcomes refer to the changes that result from a program or its activities 
(e.g., increase the number of PPOs requested). Outputs are units of service 
that are provided by the program (e.g., the number of police officers who went 
through domestic violence/PPO training).

Program goals and objectives should be tangible and measurable. Use the exercises 
in Appendix B to guide you in developing program goals and objectives or in 
adopting those you feel accurately reflect your policing mission. 


Develop a Program Logic Model

Determining exactly what to evaluate can be facilitated by using a logic model, 
which is a graphic representation of the logical relationships among program 
conditions, activities, outcomes (objectives), and impacts (goals). A program logic 
model for an evaluation plan provides explicit insight into how the program is 
supposed to work to achieve the desired benefits. A logic model incorporates a 
series of if-then changes that the program aims to achieve through its activities, 
outputs, and outcomes, and which can help you evaluate whether the program 
has met its goals and objectives. A logic model does the following:

Requires program staff to think through goals, objectives, and strategies 
and to develop a realistic idea of what the program can accomplish

Clarifies assumptions and relationships between program efforts and 
expected results

Provides a useful framework for examining program outcomes 
and impacts

Helps specify what to measure through evaluation.

Logic models usually are diagrammed as a series of boxes representing the 
conditions (needs or problems), activities, outputs, outcomes, and impacts. 
Each box in the logic model will give you a reference point for comparing your 
programs progress toward achieving the desired goals and objectives (note: 
outcomes and impacts are the measure of your objectives and goals, respectively). 
Obtaining stakeholder input into the development of the program logic model 
can be a strong foundation for planning and prioritizing evaluation activities. 

Exhibit III-2

Tips for Developing Goals and Objectives

Be careful about defining goals too narrowly so that they appear 
to be outcomes.

Beware of stating activities as goals or objectives.

Be sure not to write compound goals and objectives.

Be realistic.

Source: W.K. Kellogg Foundation. Evaluation handbook. Battle Creek, Michigan: 
W.K. Kellogg Foundation, 1998.

end exhibit III-2


A logic model can guide all types of evaluations and help you specify what needs 
to be collected. For process evaluations, the logic model identifies expectations for 
how the program should work so you can see whether your program has derailed 
or is on track. For outcome and impact evaluations, the logic model displays how 
and for whom certain services or activities are expected to create change.

A logic model with law enforcement examples is provided in Exhibit III-3. The 
corresponding worksheet and template in Appendix C will walk you through the 
steps of developing a logic model pertinent to your evaluation needs. 


Exhibit III-3

Sample Logic Model

Conditions

(problems or needs)

What problem or 
condition is your 
program designed 
to change? 

(e.g., need to reduce 
domestic violence; 
need to increase 
citizen participation)

Activities

What are the 
strategies, 
techniques, or 
types of services 
that comprise a 
programs service? 

(e.g., police officer 
training; block 
watch programs)

Outputs

What are the 
direct products of 
program activities? 

(e.g., number of 
police officers 
trained; number 
of people who 
attended block 
watch programs)

Outcomes

(short-term changes 
tied to program 
objectives) 

What are the 
immediate benefits 
of the program 
activities? 

(e.g., increase 
in number of 
PPOs requested; 
increased citizen 
participation)

Impacts

(long-term changes 
tied to program 
goals)

What changes have 
been achieved 
in the original 
conditions? 

(e.g., reduced 
domestic violence; 
improved relations 
with community) 


Develop Your Evaluation Questions

The next step in evaluation planning is determining what you and other key 
stakeholders want to know about the program through your evaluation inquiry. 
As noted in Chapter II, there are four major types of evaluation: needs assessment; 
process evaluation that documents whether a program is being implemented as 
planned; outcome evaluation that documents short-term outcomes and often is 
used for ongoing program improvement; and impact evaluation, which tells you 
whether the program achieves longer term goals. 

The evaluation questions you choose will be tied to your program goals and 
objectives as well as the type of evaluation you plan to implement. Evaluation 
questions should be developed according to what you want to learn about your 
program. The following are some typical evaluation questions:

What is the nature of the problem (e.g., delinquency, graffiti)? 

Where is the problem? How widespread is it? Whom does it affect?

What is the program or intervention? 

Is the program being implemented as intended? 

Is the program reaching its target population?

What are the barriers to implementing the program?

What approaches to overcoming implementation obstacles have been 
effective?

How much does the program intervention cost?

Is the program achieving its objectives? 

What are the actual outcomes?

Is the program cost-effective?

Choose an Evaluation Design

Understanding your program goals and objectives, creating your logic model, 
and delineating your evaluation research questions will help point you to the 
appropriate evaluation design. Five of the most common evaluation designs, 
post-test only, pre- and post-test, pre- post- post-test, quasi-experimental, 
and experimental, were discussed at length with specific examples from the 
law enforcement context in the chapter Key Concepts in Evaluation. Please 
reference the chapter to refresh your memory and to help select the design that 
is most appropriate for your needs and resources. The evaluation design that you 
choose can help direct you to the appropriate data-collection sources, which in 
turn will influence the analytic procedures you use to make sense of your data and 
answer your evaluation questions. Data collection and analyses are the focus of 
the remaining section in this chapter.


Step 2: COLLECTData Collection

Once you have determined your evaluation purpose and design, you can begin 
preparing for data collection. The data-collection methods you choose will 
reflect the type of evaluation you decide to conduct, the evaluation design that 
you have chosen, the evaluation research questions on which you plan to focus 
your activities, and the specific data needed to answer those questions. Since 
data collection can be resource-intensive (both expensive and time-consuming), 
plan your data collection carefully and limit it to only the data that answer the 
evaluation questions you have specified. Your data-collection strategies can be 
relatively simple and straightforward if you plan wisely. In fact, data collection 
can be incorporated easily into routine program activities and often is something 
police officers already do, perhaps without realizing it. 

Key Steps in Data 
Collection

Review existing data and data sources. 

Develop any necessary new data-
collection protocols.

Obtain the necessary clearance and 
permission.

Collect and track data.


Review Existing Data and Data Sources

Prior to selecting new data-collection strategies, you should first consider and 
review existing data and data sources. For example, traditional measures of police 
performance and crime, such as Uniform Crime Report (UCR) data, provide 
a nationwide view of crime based on statistics from law enforcement agencies 
throughout the country. The UCR program routinely collects data on known 
offenses and persons arrested by police departments for murder and non-negligent 
manslaughter, forcible rape, robbery, aggravated assault, burglary, larceny-theft, 
motor vehicle theft, and arson. These data may be appropriate to use if you are 
conducting a needs assessment (to determine the nature and extent of criminal 
activity in your jurisdiction) or an impact evaluation (to determine the extent 
to which your programs, strategies, and initiatives are effecting change in longer 
term criminal activity in your jurisdiction). However, they might not help you 
understand much about any single program or specific outcomes related to your 
targeted activities. For that, you would need to collect outcome data, which are 
tied to your shorter term objectives, and which typically are gathered directly 
from program participants or stakeholders (e.g., citizens, businesses, or police 
officers who go through special training) before and after implementation of your 
program or strategies. These outcome data will help you assess the extent to which 
your efforts are meeting defined objectives; impact data will show you whether 
you have met your longer term goals. 

Data collection can be one of the most resource-intensive steps in the evaluation 
cycle. To find and use existing data for your evaluation, we encourage you 
to explore other less traditional data sources that might inform an outcome 
evaluation, including calls for service, medical admissions (e.g., gunshots, 
overdoses), vandalism and graffiti costs from parks and recreation, and insurance 
information on stolen cars and burglaries.

Develop Necessary Data-Collection Protocols

Reviewing your existing data and data sources will allow you to decide what data 
you need to collect that you are not capturing now. Once you have done this, 
make a list of topics about which you need to know more. Prioritize among the 
items and finalize the list based on the relative importance and usefulness of the 
information as well as the cost and ease of collecting those specific data. Now 
you are ready to develop data-collection protocols you might need to help you 
obtain the information on that list. Quantitative and qualitative approaches to 
data collection are reviewed briefly in this section (also see Chapter II), along 
with an overview of specific data-collection methods and tips for creating surveys, 
planning and conducting focus groups, and developing interview guides.


Quantitative Data

Quantitative methods capture data that can be counted, measured, compared, 
or expressed in numerical terms. Surveys, questionnaires, and archival data are 
commonly used quantitative data sources. Archival data already have been 
collected by your department or by another organization, such as a government or 
nonprofit agency. These documents might include program records or materials 
such as proposals, annual or monthly progress reports, budgets, organization 
charts, memoranda, policies and procedures, operations handbooks, and 
training materials. Archival data are analyzed through document review. These 
data can be the simplest and cheapest kind to gather, because someone else has 
collected them. Additional benefits to using archival data are that the review of 
program documents can provide an idea of how the program functions without 
interrupting program staff or activities and document review requires relatively 
little expertise to conduct.

Surveys and questionnaires involve collecting data directly from individuals. 
Through self-administered or face-to-face surveys, telephone or mail surveys, 
questionnaires, and checklists, you can ask program stakeholders directly and in 
a standardized way whether and how your efforts are making a difference (Salant 
and Dillman, 1994). Surveys and questionnaires can be especially useful in 
evaluation if the same set of questions is asked at the beginning of a program (i.e., 
a pre-test used for baseline information) and again when the program is over (i.e., 
a post-test to document any change over time). 

In law enforcement, surveys can help you collect information about criminal 
victimization, community views about crime, and residents willingness to report 
crime to the police. Since such data provide a measure of police performance, 
they can be used to analyze the way police deliver services and possibly change 
resource allocation, if needed. Some community surveys provide detailed 
information about specific problems affecting parts of a city, which help to focus 
police activities (Weisel, 1999). 

Law enforcement agencies that implement community policing strategies often 
conduct citizen surveys to gauge community satisfaction and identify citizen 
concerns. Such surveys can help identify target areas and document progress 
and change from year to year. An example of a community satisfaction survey is 
found in Appendix D. You might find it beneficial and cost-effective to adopt or 
modify existing surveys that have been used in the policing context or in other 
relevant research. Data-collection instruments often are free, although some are 
copyrighted and must be purchased. For example, the Bureau of Justice Statistics 
and the COPS Office developed a software program for localities to conduct 
their own telephone surveys of residents to collect data on crime victimization, 
attitudes toward policing, and other community related issues. This survey can 
produce information similar to that published in Criminal Victimization and 
Perception of Community Safety, Twelve Cities. This software has recently been 
developed into an on-line application called Justice Survey Software and can be 
found at www.bjsjss.org 


Surveys and questionnaires have limitations, in general terms and when used 
in the policing context. In general, written surveys can preclude participation 
by people who speak other languages, who are illiterate, or who have learning 
disabilities, while telephone surveys can omit people who do not have telephone 
service. When using surveys to evaluate policing efforts, there are special concerns 
and barriers inherent in surveying victims and offenders. Many crimes occur in 
high-crime areas where people often are transient, so a disproportionately high 
number of potential survey subjects are likely to be harder to reach by telephone 
or written survey. Residents in high-crime neighborhoods also are more likely 
to be undereducated and under- or unemployed, which makes them harder to 
reach and potentially less likely to have the literacy skills required to respond to 
a written survey. Traumatized victims could be reluctant to share information 
over the telephone to strangers they cannot see, for fear of retaliation and 
because of the personal nature of information they might be asked to share. 
Honesty and candor are obvious concerns in surveying offenders and there are 
often insurmountable hurdles in identifying potential offenders who have not 
yet committed a crime. Ensuring that surveys are the appropriate means of 
data collection for your evaluation requires thinking through these potential 
barriers. Should you decide to use surveys in your research, some tips for survey 
development are presented in Exhibit III-4.


Exhibit III-4:

Tips for Developing Surveys and Survey Questions

Keep your survey short, ideally no more than three or four pages.

Make it easy to useparticipants will be more likely to complete it.

State the purpose of the survey on the first page, and also address the 
matter of confidentiality. Include necessary directions, explanations, or 
definitions.

Indicate the approximate time it should take to complete the survey, 
where the respondent should return it, the due date, and contact 
information for answers to survey related questions.

Keep your survey instrument simple, with short questions and clear 
answer categories.

Look for other survey instruments with similar content that have been 
used in the field. Often, data-collection instruments are free, although 
others are copyrighted and have to be purchased. When relevant and 
appropriate, use existing questions from these surveys.

Avoid double-barreled questions (i.e., questions that ask two things 
at one time). For example, Did you find the training useful and 
timely? The respondent may have found the training timely but not 
useful, so by combining what are two separate ideas into one question, 
you will not get an accurate answer.

Use appropriate language for the audience. Questions should be 
devoid of vague qualifiers, abstract and biased terms, and jargon.

Group your questions thoughtfully and logically. In general, begin 
with easier questions and progress into more sensitive or complicated 
topics.

Request demographic information.

Always include a statement at the bottom of your instrument thanking 
respondents for their time.

Conduct a pilot test of your instrument.

For written or e-mail surveys, send reminders to respondents to 
increase your response rate.

end exhibit III-4


Qualitative Data

Qualitative methods allow you to capture information that is difficult to measure, 
count, or express in numerical terms. Four qualitative data-collection methods 
and several tips for conducting interviews and focus groups are described on the 
following pages.

Interviews

Qualitative interviews are used to gather detailed information on a topic. The 
researcher asks open-ended questions to allow respondents to express in their 
own words their attitudes about, and experiences with, the program. Interviews 
typically ask people to describe or explain particular program practices or issues in 
relation to their personal experiences with them. Interviews can be a primary data 
source, a first step in developing other data-collection strategies (e.g., survey or 
focus group questions), or a complement to quantitative data collection. 

Interviews can be conducted by telephone or in person. Among the strengths 
of interviews is that they let you gather information you cannot learn through 
document review or observation. While interviews can provide more detail and 
depth than surveys can about stakeholders experiences and opinions, one of their 
disadvantages is that they are time-consuming. Interviewing tips are offered in 
Exhibit III-5.


Exhibit III-5:

Tips for Conducting Interviews

Prepare your questions. 

Use interview questions that are open-ended, clearly worded, 
and neutral. 

Organize broad questions around the general themes in which you 
are interested, then you can probe to gain more information based 
on participant responses. 

Ask how and why questions and make sure you prompt 
respondents to elaborate on yes or no answers.

Use appropriate language to which respondents will relate.

Organize questions so they flow naturally and easily. Begin by 
asking nonthreatening questions that put respondents at ease and 
end the interview by asking if there is anything important you 
forgot to ask.

Choose an appropriate setting. Interviewees should feel comfortable 
and not be distracted by other activities, lights, or noises.

Explain the purpose and format of the interview to respondents. 
This explanation also should cover terms of confidentiality, 
approximate length of the interview, and securing permission to record 
or take notes during the interview.

Practice active listening skills so participants are fully aware that 
you are paying close attention to their responses. Use silence to allow 
for elaboration.

Probe for clarification. Answers to questions are not always direct 
and might require clarification or follow-up. Use probes to steer 
respondents toward the issue at hand.

Tape-record the interviews, with respondents permission.

Always thank interview participants for their time.

end exhibit III-5


Focus Groups

A focus group is a small-group discussion guided by a trained leader, which 
explores a topic in depth to steer future action. Focus groups are a quick way to 
gain impressions and opinions from various stakeholders and are appropriate 
when action should be guided by public opinion. Some law enforcement agencies 
use focus groups when securing community input on a critical issue before 
moving forward. 

As with all data-collection strategies, successful implementation depends on 
thorough planning. To prepare for a focus group, ask yourself, What do I want 
to learn? and Why am I conducting a focus group? Advice for preparing for 
a focus group is offered in Exhibit III-6 and a focus group checklist with more 
detailed information is in Appendix E.


Exhibit III-6

Tips for Planning and Facilitating Focus Groups

Find a good focus group leader. A good leader should be 
experienced, understand the focus group topic, and relate well to 
participants. While you might be tempted to facilitate the focus group 
yourself, it would be more appropriate to ask someone else within 
your agency to facilitate or hire an outside facilitator who will be 
familiar with focus group procedures. These include asking appropriate 
questions, reviewing the goals of the meeting with participants, setting 
participation ground rules, ensuring that all opinions are heard, 
concluding the meeting on time, and providing participants with 
information about the next steps in the process.

Prepare your questions. Before the focus group starts, you should 
prepare a list of topics and questions that you want to ask. Word and 
organize them strategically. For example:

Choose easy and nonthreatening opening questions designed to 
break the ice and make participants feel comfortable.

Use transition questions to get the data you need.

Prioritize your key questions and make sure they are answered 
before you finish the focus group.

Include a final question that summarizes the discussion and 
gathers missing information.

Find a recorder. A focus group requires someone to take notes similar 
to taking minutes at a meeting. You also can tape-record focus groups 
with the groups permission. It can be useful to have an audio record in 
case you need to enhance your written notes. 

Decide about incentives. Sometimes focus group members get paid, 
even a small amount. If you cant afford it, other possible incentives 
include refreshments, public recognition, or something to take home. 

Recruit your members. Ideally, those invited should be a 
representative sample of those whose opinions you value. Keep the 
focus group small, usually six to eight people, to encourage active 
participation by all group members. 

end exhibit III-6


An overview of the data-collection methods described in this section, including a 
summary of the advantages and disadvantages of each, is provided in Exhibit III-
7. Because each data-collection method has its pros and cons, you should consider 
using a mix of both quantitative and qualitative methods in your evaluation. 
In addition to describing data-collection methods, this section offered practical 
advice to develop data-collection protocols to meet your evaluation needs. Refer 
to the Instrument Development Checklist in Appendix F for more guidance as 
you create your own evaluation tools. 

Exhibit III-7

Data-Collection Methods at a Glance table

Table Headers: Methods, Pros, Cons, Costs, Time To 
Complete, Expertise Needed

Document 
Review

Objective; least 
obtrusive

Access to data 
may be tricky; data 
can be difficult 
to interpret; 
data may be 
incomplete

Inexpensive 
for an internal 
evaluation

Depending on the 
volume of data, 
may require a lot 
of time

Some expertise 
required to access 
and interpret the 
data

Interviews

Gather in-depth, 
detailed info; data 
can be used to 
generate survey 
or focus group 
questions

Takes time and 
expertise to 
conduct and 
analyze

CInexpensive 
for an internal 
evaluation

About 
45 
minutes 
per interview; 
analysis can be 
lengthy

Good interview/
conversation skills; 
qualitative analysis 
skills

Surveys

Can be conducted 
anonymously; 
standardized data 
collection

Selecting an 
appropriate 
sample may be 
difficult

Moderate

Moderate, depends 
on method of 
distribution (i.e., 
mail, telephone, or 
in-person)

Some training 
needed to 
systematically 
administer and 
collect survey data

Focus Groups

Can get 
information 
quickly about 
needs, attitudes, 
and opinions

Can be difficult 
to recruit 
participants, 
facilitate the 
group, and 
analyze data

Inexpensive 
for an internal 
evaluation

About 
90 minutes

Good group 
facilitation/ 
conversation skills; 
qualitative analysis 
skills

end exhibit III-7


Obtain the Necessary Clearance

Depending on the nature of your program and the evaluation you have designed, 
you might need approval from your agency leadership before moving ahead 
with data collection. Invasion of privacy and confidentiality issues are a serious 
concern, which is why academic researchers almost always send their survey 
protocols for pre-implementation review to an independent review board to 
certify that the research will not have an adverse impact on any human subject. 

Evaluation data often are very sensitive, and in some cases you might need to 
protect the confidentiality of program participants and obtain their informed 
consent. Informed consent allows participants to decide voluntarily whether to 
participate in the evaluation. Securing informed consent can involve describing 
the benefits that participants reasonably might expect to encounter, explaining 
alternatives to participating in the research, and telling participants the extent to 
which their personal information will be confidential. An example of an informed 
consent statement that you can modify for your evaluation needs is in Appendix 
G. To ensure that data are handled appropriately as they are collected, limit the 
use of data that personally identifies an individual (such as name and address) and 
keep evaluation data (e.g., notes, papers, or cassette tapes) under lock and key.

Plan for Collecting and Tracking Your Data

Once you have finalized your data-collection protocols and procedures and are 
ready to begin, you will need a way to manage data collection and organize 
the data you gather. A data-collection plan should be written and included as a 
subset of your overall evaluation plan. The plan should list a schedule of required 
data-collection activities matched to the appropriate data-collection instrument 
and person responsible for each task. An estimate of the level of effort necessary 
to complete each task also might be included. Strategies for data analysis can be 
linked to each data-collection protocol or strategy, as well (see Appendix H).

In addition, you should think through processes for entering, tracking, storing, 
and securing your data. As you collect data, there are several procedures that will 
allow you to manipulate easily and understand your data once you begin analysis. 
Several tips for ongoing data collection and management areas follows:

Data tracking. It is important to track carefully the data you collect. For 
instance, you can create a chart that includes the title of the instrument, 
the source of the data (e.g., program participant, community resident, 
program document), how the instrument was administered (e.g., focus 
group, mailed survey), and the period of data collection. 


Data entry and management. You will need to create a system for 
entering and managing your data. A database system will allow you to 
know at a glance how many people have responded, when, and if there 
are any missing participant data. Your database also will be used to carry 
out more complicated analysis. Before entering data into a computer from 
paper documents, be sure to make a copy of the form and you can make 
edits or comments on the copies. If the data are in electric form to begin 
with, make a backup copy on your hard drive or keep separate copies of 
your database on floppy disks or compact discs (CD).

Data storage and security. Think about where and how you will store 
and secure the data you collect. If you collect data in hard copy form, you 
will need to decide where to keep the forms. Store written surveys and 
program documents in a place safe from damage or loss. For electronic 
data such as databases or interview transcripts, back up hard drives or 
keep separate copies of your documents on a floppy disk or CD.

By following these tips while you collect information, you will be well organized 
to analyze your data and learn about your program. Data analysis is the focus of 
the next section in this chapter.

Step 
3: COMPAREData Analysis 

Once your data have been collected, analyze them following procedures 
that reduce and categorize the information to make it easier to manipulate, 
understand, and report. Always begin your analysis by reviewing your evaluation 
purpose and specific research questions, then proceeding to analyze data to 
yield results that are easily understandable. To be useful to stakeholders and 
decision-makers, data analysis should lead to straightforward, readily interpretable 
information about program processes and effectiveness. 


Techniques for analyzing data range from basic counts, to content analysis, 
to complicated statistical analyses. Databases, spreadsheets, and statistical or 
qualitative analysis software often can facilitate steps in data analysis, including 
creating a computer database file to enter data, coding and entering them, 
cleaning data, using a computer package to analyze them, and interpreting results. 
Depending on the type of data you collected, you might conduct quantitative 
data analysis, qualitative data analysis, or both. Simple procedures for analyzing 
qualitative and quantitative data are discussed in this section.

Quantitative Analysis

Using statistical analysis to examine quantitative data (i.e., data that can be 
counted or assigned a number) can be difficult. Law enforcement agencies that 
have been successful with complex statistical analysis report having sophisticated 
data-tracking systems and using standard analysis techniques and evaluation tools. 
For law enforcement agencies that do not have the expertise or resources for data 
analysis, or when sophisticated statistical analyses are needed, consider hiring a 
consultant for guidance. 

Simple frequencies, percentages, and cross tabulations often are all that is needed. 
The following a few tips for analyzing quantitative data that do not require 
complicated statistical procedures: 

Clean your data (i.e., review data for missing or incorrect responses). 

Tabulate the information by counting the frequency of its appearance. 
For example, if you want to know how many people have ever 
attended a neighborhood watch meeting, simply count the number 
of yes responses to the survey question, Have you ever attended a 
neighborhood watch meeting in your area?

If it is ratings and rankings you are interested in, you can compute the 
mean (average), median, or mode. You can find the mean, or average, by 
dividing the sum of all responses to a particular question by the number 
of responses to that question. You can generate the median score by 
arranging the responses in a list. The middle number is the median. If 
there are two middle numbers, then add the two numbers and average 
them for a median. Last, you can tabulate the mode by looking for the 
number or value that occurs most often.

During analysis, keep in mind the following questions:

What do the raw data tell me?

Are the results low, average, or high?

Are there any red flags or extreme values to be aware of? 
What do they mean?


Qualitative Analysis

As with quantitative data analysis, qualitative data analysis is a process for 
reducing the amount of data that you have as well as aggregating and organizing 
individual responses into themes that allow you to identify data trends. 
Qualitative data include information from interviews, focus groups, written 
documents, observations, and open-ended survey questions. While qualitative 
data analysis can seem less straightforward than quantitative analysis, there are 
ways to analyze qualitative information systematically. Much of your analysis will 
be focused on providing a mechanism for organizing and describing the themes, 
patterns, activities, and content of the study rather than on illustrating causal 
linkages between processes and outcomes (Patton, 1990). 

While it is not within the scope of this guide to delve deeply into the strategies 
and nuances of qualitative data analysis, many handbooks and manuals exist for 
those who are interested in learning more. 
Basic steps in qualitative analysis are 
as follows:

Review all the data (e.g., interview transcripts or focus group notes; 
program documents).

Organize and label (code) responses into similar categories or themes. 

Try to identify patterns or associations among the data; look for 
similarities and differences across respondents (e.g., all law enforcement 
officers who attended training have a positive attitude toward the 
program, but officers who did not attend training think the program is a 
waste of time).

Identify the meaning that has emerged from the data (i.e., ask yourself, 
What are respondents telling me through these data?).

Interpret the data in relation to the issue of interest (i.e., ask yourself, 
What does this mean in light of my evaluation questions?).

Description is the foundation of all qualitative data reporting. Several options for 
organizing and reporting descriptive findings (Patton, 2002) are summarized in 
Exhibit III-8.

Footnote:
 For more information on qualitative analysis, we encourage you to consult Miles, M.B. and A. M. Huberman. 
Qualitative data analysis. Sage: Thousand Oaks, California, 1994. Another excellent reference is Patton, M.Q. 
Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002.


Analysis of Program Costs and Benefits

The need to link criminal justice program costs to outcome measures has become 
critical as policy makers increasingly require decision-making tools to allocate 
scarce funding. Systematic economic valuation of program costs and dollar 
benefits improves the basis for making funding decisions. 

Exhibit III-8

Options for Organizing and Reporting Qualitative Data

Analytic Framework Approaches

Processes: Qualitative data can be organized to describe important 
processes. For example, if the department stations an officer in a public 
housing project to reduce youth violence, describing the process of 
how the officer begins interacting with the community, especially 
young people, might become an important theme that could be used 
to frame evaluation questions or analyze the data gathered.

Issues: An analysis can be organized to illuminate key issues, often the 
equivalent of the primary evaluation questions; for example, variations 
in how participants changed as a result of the program. With the 
public housing example above, building trust is an important issue. 
The evaluation could explore various tactics that the officer used to 
promote trust and how perceptions and behaviors changed on both 
sides. 

Questions: Responses to interviews can be organized question 
by question, especially where a standardized interviewing format 
was used. For example, the evaluator might want to ask the same 
questions of the officer and community residents, and then organize 
the responses by question, allowing readers to compare police and 
community responses.

Sensitizing Concepts: When sensitizing concepts such as trust 
versus mistrust might have played an important role in directing 
data collection, the data can then be organized and described through 
these concepts. 

Source: Patton, M.Q. Qualitative research and evaluation methods. Sage: Thousand Oaks, California, 2002.

end exhibit III-8


Cost Analysis

Cost analysis consists of a thorough examination of the type and amount of all 
resources used to support a law enforcement agencys activities. Determining the 
cost figures for the agency as a whole, or in its component parts, is a basic form 
of cost analysis. Tangible program items such as salaries and equipment are given 
a dollar value. Costs typically are provided at several levels, from the total cost of 
the agency to the costs associated with each area of responsibility.

Cost-Effectiveness and Cost-Benefit Analyses

There are two levels of cost analysis evaluation: cost-effectiveness and cost-benefit. 
The former assesses whether the money for a program was well spent to reach 
the program outcomes, while the latter assesses costs of achieving the intended 
impact.

Cost-effectiveness is the relationship between agency costs and its effectiveness 
in providing community safety. Cost-effectiveness is a technique in which costs 
are measured as dollars spent, whereas effectiveness is measured in changes 
in outcomes (alternatively, outcome counts can be used). The overall cost-
effectiveness of an agency is improved by first determining the activities of the 
agency that contribute most to effectiveness and then determining which of 
those activities have the lowest cost. For example, the cost-effectiveness of a law 
enforcement agency might be measured as the number of offenders apprehended 
in one month versus the costs of various programs that an agency uses to 
accomplish this (e.g., community outreach, officer training, crime mapping).

Cost-benefit analysis compares costs and benefits using the same monetary 
units. Determining the dollar value of a benefit is the primary challenge in 
cost-benefit analysis. The benefits of law enforcement activities are many and 
include decreased social disorder, decreased fear of crime, and better or more 
public services to respond to crime. Another prime benefit is decreased crime, 
resulting in decreased justice costs and lower costs to potential victims and society. 
The criminal justice literature provides estimates for such of benefits. 

During cost-benefit analysis, the present value of future benefits must be 
calculated and compared to present investment costs. Economists use discounting 
when benefits will be received in the future to make future monetary benefits 
comparable to current dollar values. To accomplish this, future benefits are 
discounted by some factorthe higher the rate, the lower value of future benefits. 
The discount rate is important because offender benefits might not be realized for 
years. Generally a rate between 3 and 5 percent is used. To determine a return on 
investment, the cost-benefit analysis will present results for benefit-cost ratios and 
net present values.

 
An example can be found in Cohen, M.A. Measuring the Costs and Benefits of Crime and Justice (pgs. 
263315). In Measurement and Analysis of Crime and Justice. Washington, D.C.: U.S. Department of Justice, 2000. 


If we think of the domestic violence example used in Chapter II, both approaches 
require totaling program costs, often a combination of grant funding and regular 
operating costs. A cost-effectiveness analysis would explore whether the money 
invested was justified by the major outcomesthe number of PPOs issued and 
the number of police personnel trained. A cost-benefit analysis would attempt to 
answer whether money expended on the program was justified by a reduction in 
domestic violence cases, including deaths and injuries. As this example suggests, it 
is easier to conduct a cost-effectiveness analysis than a cost-benefit analysis, in part 
because it is difficult to quantify deaths and injuries prevented. Ascribing a dollar 
amount to those costs also is problematic.


Chapter Summary

While the part-time or full-time participation of an evaluation 
consultant could enhance the strength and objectivity of evaluations of 
policing programs, many departments can and should perform more 
evaluation activities by relying on their internal resources. This chapter 
detailed the major steps in the evaluation cycle and provided practical 
information and advice to help you conduct an internal evaluation of 
your policing strategies. 

In the first section of this chapter, the myriad components of 
evaluation planning were discussed, including evaluability assessment, 
stakeholder involvement in the evaluation process, defining program 
goals and objectives, and determining the logical relationship among 
program inputs, outputs, and outcomes. Tips were offered for 
developing evaluation questions and selecting an appropriate design 
based on evaluation planning activities.

This chapter also provided guidance about ways to identify and 
implement data-collection strategies beyond traditional police 
measures. Material presented in the section on data collection included 
a brief summary of the pros and cons of different data-collection 
methods as well as tips for developing data-collection procedures and 
protocols such as surveys, interview guides, and focus group guides. 
The importance of obtaining necessary clearances and permissions to 
collect data from program participants was underscored. Last, simple 
procedures for making sense of your data were discussed in the section 
on data analysis. 

Now that you are familiar with all the steps involved in planning 
and implementing an evaluation, you should be able to create your 
own evaluation plan. A sample evaluation planning template can 
be reviewed in Appendix H and used to assist you in developing 
and documenting your general evaluation plans. The following 
chapter provides useful strategies for managing internal and external 
evaluations of policing activities. 


Chapter IV:
Managing Evaluations



Managing Evaluations

There are two types of evaluations in which law enforcement agencies are likely 
to be involvedinternal evaluations and external evaluations. As noted in 
Chapter III, internal evaluations are those in which police agencies rely primarily 
on employees within the department to conduct evaluations and disseminate 
results. In some instances, agencies have research or evaluation divisions (or 
staff) specifically to support the development and implementation of evaluation 
within the agency. On the other hand, external evaluations are those for which 
the agency hires an outside expert to conduct the research. Generally, external 
evaluators are hired when the agency does not possess the expertise or resources 
to conduct evaluations internally or when a particular policing program, 
intervention, or strategy requires an evaluation design that is more complex than 
the internal evaluation staff are prepared to handle. 

While internal and external evaluations are conducted for different reasons, many 
of the steps required in planning and managing them are very similar; although 
each has its own unique set of challenges as well. This chapter first outlines the 
steps required to plan and manage an internal or external evaluation; the section 
that follows outlines some of the challenges unique to managing each type of 
evaluation. Finally, this chapter provides important information about how to 
hire an external evaluator, including assessing evaluator capacities, what to look 
for in an external evaluator, and sources for finding external evaluators. 

How Do I Manage an Evaluation?

The planning and ongoing management of evaluation activities, whether internal 
or external, can be as important as the methodological aspects. While there is no 
single recipe for managing an evaluation, law enforcement administrators who 
do must possess a basic understanding of evaluation for appreciating the work to 
be conducted, assisting in its planning, and identifying weaknesses in the design 
or data-collection plan. In addition, there are management issues to consider at 
critical junctures prior to, during, and following an evaluation. These are outlined 
in more detail below. 

What do I need to consider prior to the evaluation?

If you are assigned to manage the evaluation, you might consider obtaining 
input from several colleagues, particularly those with an evaluation background 
or experience with the program or strategy the evaluation will target. If possible, 
also involve groups that are in some way connected with the program or strategy, 
but are not affiliated with the law enforcement agency. These might include 
community members (in the case of community outreach activities), stakeholders 
(those supporting or funding the policing activity or strategies), and program 
staff. Finally, involving the leadership within your agency will help ensure support 
at the highest level. 



Before conducting an evaluation, determine the evaluation requirements, 
develop the evaluation plan, and set up administrative and monitoring tools. 
These activities are described in the section that follows. 

Determining the Evaluation Requirements

Before anything else, think about the reasons for your evaluation. Are you 
considering it because you want to use the information to improve a particular 
program or strategy? Do you want to know if something is producing its intended 
outcomes? Is the evaluation a requirement of a grant or other contractual 
relationship? Understanding the reasons will help you determine the best 
approach. For example, if the goal is to modify or improve a program or strategy, 
you might consider an evaluation that is focused on process rather than outcome. 
Or, if the evaluation is a grant requirement, you must develop an evaluation 
plan that meets the information needs of your contract with the funding source. 
Most important, determine if an evaluation even is feasible by considering several 
issues: 

Are the data you need to answer the research questions available? 
If available, can you access them? For example, school records may be 
available but school rules or consent procedures may prohibit you from 
accessing them. 

Do you have the financial and other resources to conduct an 
evaluation? If internal, this includes having staff with the time and 
expertise necessary to conduct the evaluation. If external, this includes 
having access to an experienced evaluator who can work within time and 
budget constraints. In either case, financial resources must be available to 
support evaluation costs. 

Do you have buy-in and support from your agency administrators? 
If the leaders in your agency (e.g., your chief or executive) do not 
support and approve of the evaluation, it may preclude you from fully 
implementing or funding it. 

Once you have determined that an evaluation is both necessary and feasible, you 
can proceed to the next stepdeveloping the evaluation plan. 

Developing the Evaluation Plan 

Developing an evaluation plan, whether you are managing an internal or external 
evaluation, requires several important steps. Some of these are outlined in 
detail in Chapter III and will not be covered here, including determining what 
will be evaluated, establishing goals and objectives, developing logic models, 
developing evaluation questions, and determining the appropriate design. 
Other management tasks that are an important part of an evaluation plan will 
be explored here: determining the evaluation budget and timeline, identifying 
resources available to conduct the evaluation, and devising a plan for using and 
disseminating evaluation results.


Budgets and Timelines

Developing budgets and timelines are very different tasks. Because your budget 
will have an impact on the duration of your evaluation, and vice versa, developing 
one without consideration of the other is not recommended. On the other 
hand, each one also has unique circumstances that require consideration during 
planning. With this in mind, we present each issue separately. The final paragraph 
ties the two together. 

Budgets 

When thinking about your evaluation budget, take into account several categories 
of expenses. Labor typically is the most costly evaluation expense, whether your 
evaluation is internal or external. Internal labor expenses include staff time and, 
for full-time staff, associated costs such as overhead (e.g., electricity and rent) 
and fringe benefits (e.g., insurance, retirement, and vacation time). If you use 
an external evaluator, you will not need to estimate overhead or fringe benefits, 
but you will likely require more financial resources than if you were using 
internal staff because external evaluators, especially academic or other highly 
qualified professional evaluators, generally are more expensive than internal 
staff. Depending on the extent of the evaluation, an external evaluator may cost 
you approximately 20 percent of your overall budget. For example, you might 
consider setting aside $20,000 if your full budget is $100,000. 

A more efficient method of estimating external costs is to solicit cost proposals 
from several evaluators. Cost proposals require that each bidder describe in detail 
the tasks to be completed, the number of hours each task will take, and the 
costs associated with each, including labor. You should receive several bids that 
are approximately the same. If you receive bids that are very different from the 
rest, you can assume they either under- or overbid the work. In all cases, your 
resources are finite so be sure you have considered all possible costs and that they 
are fully addressed in the cost proposal of the bidder you choose. Unanticipated 
costs can derail an evaluation quickly, particularly if the costs are associated with 
activities that are critical to the evaluation (e.g., data collection or analysis).

In addition to labor, evaluation budgets also can include incentives or rewards 
for participants, if appropriate, and other indirect costs (e.g., postage, telephone, 
duplication). Travel expenses (e.g., airline tickets, rental cars, per diem) also must 
be estimated; be careful, because extended travel expenses can add up quickly. 
In addition, you can never fully anticipate costs associated with cancelled and 
rescheduled flights, and stays in expensive hotels when the preferred (and less 
expensive) ones are full or cannot accommodate a government rate. If data can 
be collected locally rather than off site, your travel expenses can be reduced 
drastically. Finally, consider whether the evaluation will require any special 
equipment, varying from standardized instruments (e.g., existing surveys and 
questionnaires available for purchase) and tape recorders for data collection, to 
data analysis and management tools such as software and computer upgrades. 


As you estimate evaluation expenses, be certain you have the resources to fund 
your evaluation before you take any steps toward implementation. There typically 
are several options for funding evaluation activities, including the following:

Agency budgets. Many law enforcement agencies have a line item in the 
budget to cover evaluation activities. If this is the case in your agency, be 
clear about how much money is in that budget, the types of activities it is 
designed to fund (some set-asides are earmarked for specific categories of 
expenses such as labor or data collection), and if other officers intend to 
use the money for their own evaluations. 

Grant funds. Many grants require the grantee to set aside a certain 
percentage of the grant funds (most often 10 percent) to cover evaluation 
expenses. Unfortunately, these funds often are insufficient to cover all 
related evaluation expenses. As a result, if you are using grant funds for 
your evaluation, you may need to consider additional sources of funding. 

Outside funding sources. Government, private, and some nonprofit 
agencies and foundations offer grant funds specifically for evaluation 
activities. These funds are offered most often in the context of advancing 
knowledge in a particular area or to support the implementation and 
evaluation of best or promising practices. Therefore, you may have 
to search to find an evaluation grant program focused on funding 
evaluations in your program area. 

Timelines

The requirements of your evaluation most often determine its duration. For 
evaluations that are designed to meet funding requirements, the funder and the 
duration of the grant generally determine the evaluation timeline. For example, 
if you are awarded a 3-year grant to implement a community policing strategy 
in a local neighborhood, you might need to measure and report outcomes of 
interest to the funder at the end of the 3 years. In the interim, the funder could 
require that you document certain components of the strategy, and/or measure 
short-term or intermediate outcomes and submit the findings in quarterly or 
semiannual reports. In this situation, the evaluation as funded is expected to 
terminate at the end of the grant period and will continue only if you choose to 
and have resources to support ongoing evaluative efforts. 

For evaluations that are not bound by funding requirements (e.g., those that are 
funded using agency dollars or are designed to monitor program progress), it 
could be up to you to determine the timeline. If this is the case for your program 
or strategy, there are several issues to consider. First, think about what outcomes 
the program or strategy is designed to achieve and when those outcomes 
realistically might be realized. 


For example, if you are implementing a substance abuse awareness program in 
the local high school, you might expect knowledge about the dangers of drug use 
to be evident in the participants immediately after they complete the program. 
Lower substance use rates in the high school population, however, could take 
longer to achieve; in fact, they might not be evident for years. Determine which 
outcomes your evaluation is intended to achieve and when they will occur, and 
develop your timeline accordingly. If you conduct an impact evaluation (e.g., 
demonstrating long-term outcomes), extend the evaluation timeline outward to a 
point at which these outcomes can be realized. On the other hand, if you conduct 
an outcome evaluation (e.g., demonstrating immediate, short-term outcomes), 
your evaluation timeline will be more closely related to the duration of your 
program or strategy. For example, if you implement a program or strategy that is 
10 weeks long, you should gather valuation data right before (pre-test) and right 
after (post-test) program implementation. 

Despite their differences, budgets and timelines should be developed 
simultaneously. The money and other resources that are available for the 
evaluation will have implications for how long it can continue. Whether internal 
staff conduct your evaluation or you hire external evaluators, you must know the 
extent and duration of available resources. You could have ample resources for 
evaluation one year, but as agency priorities shift, funds might be reallocated in 
future years. In addition, you might not start new evaluation projects if you know 
the chief is retiring or the governor is running for reelection. Changes in political 
structure sometimes mean law enforcement priorities change and budgets can be 
realigned to reflect those changes. You do not want to discover midway through 
your evaluation that funds have been exhausted. Be certain from the outset what 
funds are available and for how long so you can implement your evaluation 
completely and successfully. 

Evaluation Resources

Even if your organization maintains a research and evaluation division through 
the annual budget, you might need to hire an external evaluator or outside 
consultants to assist internal staff with complicated evaluations or those that 
exceed your in-house capacity. Therefore, you could require evaluation funds in 
addition to those in your existing budget. Or, your organization could be one of 
the many law enforcement agencies that lack not only an evaluation unit but also 
an evaluation budget. Fortunately, regardless of your situation, several evaluation 
resources are available, including the following: 

Writing a grant proposal to request evaluation funds. Increasingly, 
government and foundations are willing to support thoughtful evaluations 
that have the potential to identify and foster promising practices.


Building your agencys internal evaluation capacity. Training and 
technical assistance (TA) often is available for staff development on 
evaluation-related topics. Providers include trained evaluators, academics, 
and individuals employed by federal agencies or professional evaluation 
associations. Because TA providers often are supported by state or 
federal grants, many of these resources are available free of charge or 
for a nominal fee. Check with agencies from which you currently 
receive program or other funding to learn if they offer evaluation TA 
opportunities to their grantees. See Appendix L for a list of technical 
assistance resources.

Reviewing existing law enforcement evaluation reports for best 
practices and lessons learned. An inexpensive way to learn about 
evaluation is by reading evaluation reports and related materials. 
Evaluation reports often are available to the public and can be a good 
source of information regarding evaluation strategies that have been 
implemented in your field and those that have been particularly 
successful. In addition, hundreds of evaluation resources are available 
on the Internet, including manuals, step-by-step guides, and technical 
reports. Most of these materials are either free of charge or can be 
obtained for a nominal fee. See Appendix L for a list of evaluation reports 
related to policing.

Other law enforcement agencies. Some law enforcement agencies 
have research or evaluation divisions staffed with competent evaluators 
who also are law enforcement agents. These individuals might be willing 
to assist you and your staff to plan and conduct evaluations, share 
information about what has worked for them, or provide technical 
assistance to build the internal capacity of your agency. 

Graduate students. Graduate students often are required to conduct a 
research study as part of their graduation requirements. Masters degree 
students must complete a thesis, while doctoral students are required 
to prepare a dissertation, and they often will take on an evaluation or 
other research study without being paid. In addition, because the work is 
being conducted to fulfill graduation requirements, a team of academics 
generally is assigned to oversee and monitor the project from planning 
through completion, ensuring a high-quality and rigorous evaluation. 


Utilization and Dissemination Plan

The likelihood that evaluation findings will be used increases with 
comprehensive planning, preparation, and follow-up. An evaluation 
might meet standards for rigorous design, careful data collection and 
analysis, and a well-written report, but if it does not have practical 
implications and is not used, it has failed. When an evaluation proves 
to be useful, it usually succeeds in answering a specific, clearly defined 
question that someone wanted answered. As evaluation manager, you 
must make sure the appropriate evaluation questions are asked. Soliciting 
stakeholder input about what they need from the evaluation and what 
would be most important for them to know about the program or 
strategy ensures that evaluation results will not simply sit on a shelf 
somewhere. 

As evaluation manager, you also should have an information 
dissemination strategy to fit stakeholder needs, using a variety of 
communication methods, which might include formal reports and 
briefings as well as informal strategies, if appropriate. The dissemination 
strategy could identify the purposes and stakeholder audiences, which 
might include program staff and supports, community members, 
funders, policymakers, and the media. 

Developing Monitoring Plans

During planning, establish procedures to ensure that you can monitor 
both the evaluation work and budget effectively, regardless of whether 
the evaluation is conducted internally or externally. For example, how 
and when will your external evaluator submit invoices for payment? 
What means will you use to monitor the work 
of your internal evaluation staff? These issues are 
discussed in detail in the section that follows. 

Monitoring Procedures

Monitoring your evaluation effectively requires 
you to supervise your staff, their work, and the 
budget. The procedures you use, however, depend 
on whether you are managing an internal or 
external evaluation. 

Questions to Guide Your 
Internal Monitoring Plan

How will you maintain frequent contact 
with your evaluation staff?

How will you require your evaluation 
team to document its progress?

How will you learn of problems 
promptly?

Do you need a project manager or 
primary staff person to oversee the 
evaluation?

Internal Evaluations

Having frequent and regularly scheduled 
meetings with internal staff is a good way 
to monitor the extent to which they are 
implementing the work plan as intended and 


are staying on track with the evaluation timeline. Staff meetings also provide 
opportunities for identifying potential problems and solving them before they 
grow. Frequent contact with your evaluation staff also allows you to monitor 
projected labor hours. If your staff are spending more hours than you have 
allocated for them (or fewer), you may need to modify your work plan and/or 
find additional resources (e.g., money or staff) to assist with implementation. 
You also can monitor internal staff through e-mails and telephone conversations. 
These methods can be used in between staff meetings to keep abreast of progress 
and to solve problems routinely. 

Also keep in mind how your internal team will document progress, including 
challenges to implementation that might disrupt your timeline. You might set 
aside time during regularly scheduled staff meetings for oral progress reports 
that are structured in content, refer to a specific period (the 2 weeks prior to the 
meeting, for example), and might cover major activities, problems or challenges, 
action steps taken, and future activities. If you plan to have staff report orally, be 
sure to have someone document what is said, perhaps the staff person responsible 
for taking meeting minutes. A written record of what is happening in your 
evaluation is important as a reference if your funder or chief has questions about 
the evaluation or if problems arise. 

In developing a plan for monitoring internal staff, decide if you want to designate 
someone other than yourself as the evaluation manager or primary staff person 
(e.g., this might be the most senior person on the evaluation team or the one with 
the most evaluation experience). Designating someone as the project manager not 
only provides a point of contact your staff can turn to when you are not available, 
but also identifies one person whom you (or your chief or funder) can call when 
questions or problems arise. You do not want to waste time figuring out which 
staff member to call when your chief or a federal project officer is on the line 
with a question about the evaluation. In addition, a project manager is someone 
to whom you can delegate certain responsibilities to assist you to do your job 
more effectively. For example, you can ask your project manager to facilitate staff 
meetings and report to you about progress and problems.

External Evaluations

Compared to an internal evaluation, managing an external evaluation can 
demand more effort in advance to select and hire a qualified evaluator, to work 
collaboratively with the evaluator throughout the evaluation, and afterwards to 
review the report and disseminate findings. Many of the techniques for managing 
internal evaluations, however, also apply to external evaluations. 

In-person meetings and frequent telephone and e-mail contact also are good 
methods for monitoring external evaluators. Monitoring an external evaluator, 
however, also poses a unique set of challenges. 


First, external evaluators are just thatexternal. You do not have the same 
opportunities to meet with and monitor their progress as you would with internal 
staff. Nonetheless, you can offset this problem by making regularly scheduled 
meetings a part of the contract with the evaluator. You might stipulate that 
monthly (or weekly or biweekly) evaluation meetings will take place either in-
person or by telephone, and that these meetings will be used to discuss progress 
against the work plan, problems with or challenges to implementation, and 
budgetary matters. In the contract, also outline the consequences if certain 
requirements are not met. For example, you can reserve the right either to cancel 
the contract or withhold payment until requirements meet with your satisfaction. 
Should the evaluator fail to appear for meetings or be otherwise unavailable to 
you throughout the evaluation, you can initiate the agreed-on consequences. We 
also recommend that you require some type of progress report from your external 
team. Written progress reports are a standard monitoring method and, as with 
invoicing, you will want to determine how, and how frequently, you will require 
that your evaluator submit these. In the contract with your external team, you 
also should state clearly the purpose, content, and timeline for these reports. 

Determine how you will monitor the evaluation budget of an external evaluator. 
Will you require monthly or quarterly invoicing? More frequent invoicing might 
allow you to monitor your budget more closely but, if your budget is small, less 
frequent invoicing may be acceptable. Also consider 
what details you want each invoice to contain. Some 
contracts require little detail about how the money was 
spent or expenses were incurred, while others specify 
all items must be clearly identified and documented. In 
addition, you should know if your agency has standard 
invoicing procedures for an outside contract so that 
you can adhere to them. 

In managing an external evaluation, your contract 
is your leverage; therefore, it must state clearly each 
requirement and how it will be monitored. Be sure 
your evaluator or team understands the contract 
requirements before they sign it. If, like many large 
law enforcement agencies, your agency has a contracts 
department, consider involving the department in 
contract development and negotiations. Contracts are 
legally binding documents and can be challenged in a 
court of law; therefore, make sure your contract meets 
the standards for contracts of its kind. These steps will 
contribute to the leverage and documentation you 
need to monitor your external evaluator effectively. 

Questions to Guide Your 
External Monitoring Plan 

Have you contacted the evaluators 
references? 

Do you have a context for his or her 
previous work?

Does the contract stipulate: How 
often and by what means meetings 
will take place? What consequences 
will be implemented if the contract is 
violated? 

Do you have a plan for monitoring the 
budget? Does it allow you to identify 
problems immediately? 

How will you learn of problems 
promptly so that they can be resolved 
efficiently and immediately? 

Have you identified a primary contact 
person with whom you can work in 
the principal investigators absence? 


You can use administrative procedures such as those suggested above to keep 
your external evaluator in check. However, nothing beats having a positive 
working relationship, which facilitates communication between you and your 
evaluator and, in turn, ensures that critical information is available to you in a 
timely manner for smoother process monitoring. You can increase the likelihood 
of a positive relationship with your evaluator by hiring an individual or agency 
you know, that you (or someone you know) have worked with in the past, or 
is referred to you by someone you know and trust. If this is not possible, do 
not hesitate to ask potential hires for past evaluation reports and references. 
Past reports let you see the kind of work done previously and identify agencies 
or individuals who have contracted with the person for similar work. If these 
agencies or individuals are not given as references, do not hesitate to contact 
them yourself. Also, be sure to contact the references that are provided, not only 
asking questions about the candidates work, but also about work style. You also 
might ask for additional references to others for whom the candidate has worked. 
Contacting these individuals or agencies will give you additional information 
about the credibility and integrity of the potential evaluator. 

What Do I Need to Do During the Evaluation?

For the most part, what you do during the evaluation is to implement the 
monitoring techniques you developed during the planning phase. For internal 
evaluations, this includes conducting regular staff meetings, conducting 
e-mail and telephone communication with your evaluation staff, and reviewing 
project expenditures. If you have an internal project manager, that person can 
handle many of these activities and report to you. For external evaluations, hold 
regular meetings with your evaluator either in person or by 
telephone; review progress reports and invoices, and follow 
up on any issues that emerge. During the evaluation, be sure 
you are meeting other demands as well. For example, if you 
are required to give periodic progress updates to your chief or 
funding agency, you must plan for these activities, especially if 
they must be in writing. 

During the evaluation, find ways to provide feedback to the 
team (and those individuals and agencies involved with the 
program or strategy under study) about how things are going 
and what initial findings are emerging. As evaluation manager, 
you will have the most comprehensive information about the 
study and the outcomes being achieved. Keeping your team 
informed is important because, in completing day-to-day 
evaluation activities, staff can lose sight of the big picture. 
Providing staff, whether internal or external, with frequent 
updates about the evaluation not only provides you with an 
opportunity to acknowledge their hard work, but also helps 
them remain focused on overall project goals. 

During the Evaluation

Conduct frequent meetings with your evaluation 
team.

Review project expenditures.

Follow up immediately on problems or other 
concerns. 

Provide feedback about the evaluation to 
the team and your other stakeholders.

Be prepared to make program modifications 
if early findings are not promising.


Despite your best intentions, a variety of things can go wrong during an 
evaluation, whether it is internal or external. Staff turnover at the program 
site (if your data are being collected by program staff or those responsible for 
implementing your strategy directly) and on the evaluation team, low program 
participation rates and poor program implementation can derail even the best 
evaluation plans. If you can anticipate these events, you can respond to them 
appropriately and quickly. However, some issues, such as staff turnover, are 
difficult to anticipate. Generally, staff do not make public their intentions to leave 
a position, at least not with enough time to plan for their departure. Therefore, 
in addition to building a relationship with your evaluator, you should maintain 
contact with those close to the program or strategy being evaluated (e.g., a project 
director or program staff person). These relationships help you learn important 
information about events you might otherwise not obtain (e.g., a staff person is 
leaving the agency). 

What Do I Need to Do After the Evalaution? 

Your evaluation responsibilities do not end with submission of the final report. 
You will need to implement several important activities after the evaluation ends 
to ensure that your evaluation has resulted in the type of information you need to 
understand. This means not only how effective your program or project was but 
also what improvements need to be made to increase the likelihood of achieving 
positive results in the future. These activities are outlined in detail in the sections 
that follow. 

Review the Evaluation Report

The evaluation manager has primary responsibility for responding to and using 
evaluation results. This responsibility encompasses reviewing the report to 
determine that it is a clear accounting of the evaluation process and results and 
that it meets the reporting requirements set out in the contract. Specifically, the 
evaluation manager should review the evaluations key findings, conclusions, and 
recommendations. Most well-written evaluation reports include six sections: 

1. Executive summary. An executive summary should contain a very brief 
description of the evaluations purpose, questions, procedures, most 
important findings, implications, and recommendations. It should not be 
more than a few pages in length.

2. Introduction. The introduction should contain the purpose of the 
evaluation and the overview of the report contents. The introduction 
sets the stage for the rest of the report by outlining basic evaluation 
purposes and introducing the report contents that follow. It often 
states the intended audiences for the report and cautions the reader 
about limitations that might have affected the collection, analysis, or 
interpretation of information.


3. Description of the evaluation focus. The third section, the focus of the 
evaluation, should contain a description of the program being evaluated 
and the evaluation questions. This section typically discusses the rationale 
for the evaluation, who it is intended to benefit or serve, activities and 
strategies followed in implementing the evaluation, settings, and resources 
devoted to the program. Also in this section are the specific questions that 
shape the evaluation. You might want to include a copy of your program 
logic model here.

4. Evaluation plan and procedures. The evaluation plan and procedures 
section contains a description of the evaluation design, data-collection 
strategies and instruments, and an overview of data analysis. This is 
the section in which the evaluator presents the methodological and 
technical information to establish evaluation adequacy and credibility. It 
is helpful to outline the information being collected to answer each of the 
evaluation questions and describe the procedures for data collection and 
analyses and include any instruments as appendices.

5. Presentation of evaluation results. The presentation of evaluation 
results section should include a summary of evaluation findings and an 
interpretation of them. More extensive details related to the findings can 
be included in supporting appendices. The specific criteria being used to 
judge the results should be stated explicitly so they can be viewed in light 
of these standards. For example, if the standard you are using to judge 
effectiveness is citizen satisfaction for your area, data should indicate 
whether citizen satisfaction has increased, decreased, or stayed the same. 
Interpretation of the findings should reflect a careful process of synthesis 
and analysis that supports judgments and subsequent recommendations. 
It is helpful to relate the findings clearly to the specific evaluation 
questions.

6. Conclusions and recommendations. The conclusions and 
recommendations section should outline the implications of the 
evaluation findings for your program and specific recommendations 
for program modifications or improvements. Organize conclusions 
by strengths and weaknesses that focus attention on both positive and 
negative judgments. Descriptions of strengths and weaknesses must 
contain sufficient detail to give stakeholders an understanding of the 
rationale and to serve as the basis for recommendations.

Disseminate Evaluation Findings

After evaluation data have been collected and analyzed, communicating the 
results is critical if they are to be used effectively. When overseeing an evaluation, 
the evaluation manager should seek opportunities to share results with 
stakeholders and program managers throughout the process, but the final results 
are often what stakeholders want.


Specific strategies for developing appropriate evaluation reports for many 
stakeholder groups are presented below.

Program staff and supporters

Usually, the people who are involved in implementing your program believe that 
it works well and brings about positive results. They will be most interested in 
what the evaluation can tell them about ways to improve the programs function 
and about how participants feel about the program. Reports that include detailed 
information on these topics will be most appropriate for this audience.

Funders. Funders want to be sure a program is making good use of 
the resources they provide and the program is accomplishing its goals. 
Therefore, when reporting evaluation findings to funders, present 
information regarding the implementation and specific outcomes of your 
program. 

Policymakers. These people make strategic decisions about the program 
itself and tend to be interested in the big picture rather than being 
concerned with details. A brief document highlighting the findings that 
address program effectiveness is appropriate for them.

Media. One of your main objectives in delivering findings to the media 
might be to gain visibility for your program. Media outlets tend to run very 
brief, straightforward stories and probably will not be interested in detailed, 
extensive reports. Your best bet in working with the media is to provide a 
short, concise document that includes an overview of your program, the 
particular findings you want to highlight, and the implications of those 
findings for your community and future policing efforts.

How you use your findings depends greatly on your audience. While written 
reports and briefings are common strategies for communicating evaluation results, 
you should also consider using other techniques. The most useful evaluation 
products are often presentations to decision makers or community groups, but 
a brief summary report can be distributed to a wider audience, press releases 
can provide more exposure to evaluation findings, and web sites can include 
visual supplements such as photos and color charts. Web sites, e-mail messages, 
question-and-answer statements, press releases, op-ed items in newspapers, 
speeches, written testimony, newsletters, articles in professional journals, brown-
bag lunches, videotapes, or computerized evaluation presentations are increasingly 
being used as communication tools and should be considered when deciding how 
to disseminate evaluation findings (USAID Center for Development Information 
and Evaluation, 1997). 


Apply the Evaluation Results

Evaluation findings can be used in several ways, depending on your organization 
and what evaluation stakeholders deem appropriate. Generally, findings can be 
used in the following ways: 

For internal improvement. Evaluation results can be used in internal 
agency decision-making to improve ongoing implementation of a 
program or refine strategies based on lessons learned about challenges and 
successes.

For accountability. Findings can demonstrate to stakeholders that the 
program is having a positive impact or is particularly cost-effective. 
Evaluations often are used to secure grants or convince supporters that 
the program is functioning as planned. For example, to maintain or 
secure funding, a granting agency might ask for proof that your program 
is achieving its stated goals. Findings can show that the program is 
successful in meeting its outcomes. If you conduct a process evaluation, 
the report also can document how funds or resources are being used.

For outreach. Evaluation results can be shared with the media and 
directly with the community to increase visibility for your initiatives. 
Strategies include releasing findings to the media, organizing town hall 
meetings to discuss results, providing information to school and faith 
community newsletters, and building an information web site. 

Challenges to Managing Evaluations 

While most evaluations run smoothly if well planned and monitored, there are 
challenges associated with managing an evaluation, whether internal or external. 
These include the following: 

Time. Be clear about how much time you need for management activities 
and how your time can be spread across the various tasks for which you 
are responsible, including those not associated with the evaluation. As 
suggested earlier, if your time is limited, consider appointing a project 
manager to whom you can delegate a portion of the management 
activities. In addition to your time, also consider staff time. Do you have 
staff with time to implement the evaluation effectively? Generally, law 
enforcement officers are very busy individuals with little time to spare. If 
you do not have the internal capacity to conduct the evaluation, consider 
hiring someone from the outside. 

Resources. When budgets are prepared each year, every available dollar 
is assigned to some activity or function. If evaluation is not part of your 
agencys annual budget already, funding your research will be the most 
challenging aspect of your management duties. In addition, you might 
find that funding the evaluation fully is not a one-time event, but requires 
attention throughout the study. 


Competence. Whether managing internal or external evaluation staff, 
be confident they have the necessary skills to implement the evaluation 
effectively. Assessing the teams competence is an issue not only before 
the evaluation begins, but also as it progresses. Evaluation staff require 
methodological and analytic skills and also must possess the capability 
to work cooperatively with other people, groups, and agencies. You 
might have a clear understanding of their evaluation skills in advance, 
but their ability to collaborate with others most likely will emerge over 
time; therefore, have mechanisms in place to monitor evaluation staff 
competence. In addition, you need good management skills to handle 
the evaluation appropriately. Have you ever managed a budget before? 
Supervised staff? If not, consider appointing someone to manage the 
evaluation for you. 

Support. In law enforcement agencies, any activity outside the realm of 
law enforcement calls for support from the administration. If your chief is 
not convinced of the value of evaluation, you will be hard-pressed to find 
the resources you need to implement your research. If evaluation is not an 
agency priority, you can promote it by taking advantage of opportunities 
to educate the administration and other officers about the benefits of 
evaluation. Be sure you have done your homework and be prepared 
with examples showing how evaluation findings increased an agency or 
programs visibility or generated additional funds for the agency. 

While these challenges exist to some extent in most law enforcement agencies, 
they can be overcome or at least managed effectively. Think through these issues 
and ways to resolve them when planning the evaluation. If you do not plan for 
them, these challenges are more likely to interfere with your ability to garner the 
support you need to carry out your evaluation. 

Should I Hire an External Evaluator?

If you have decided to conduct an evaluation but your agency does not have an 
internal evaluation unit (or it does have such a unit, but the staff do not possess 
the necessary expertise), you might need to hire an external evaluator. This section 
of the guide offers tips for decision-making and selection when hiring an external 
evaluator and addresses four fundamental questions: 

1. How do I know if I should hire an external evaluator?

2. Why should I hire an external evaluator?

3. What do I look for in an external evaluator?

4. How do I find an external evaluator?


How Do I Know if I Should Hire an External Evaluator?

During the planning stage, assess the evaluation capabilities within your agency to 
determine if you can use internal staff or if you need to hire an external evaluator. 
Exhibit IV-1 presents questions to help you decide if it is in your agencys best 
interest to hire an external evaluator for a specific project. 

If the majority of the answers to these questions are no, then you should 
consider hiring an external evaluator to assist with planning and implementation. 
Even if you answered yes, review the sections in this guide that detail the 
competencies, roles, and responsibilities of the evaluator to identify the expertise 
within your internal team.

Why Should I Hire an External Evaluator?

To assist you further in making the decision, this section, including Exhibit 
IV-2, compares the advantages of hiring an external evaluator to the potential 
drawbacks of relying on internal staff. 

Exhibit IV-1

Assessing Evaluator Capabilities Circle Yes or No

Capability Questions

Does your agency have a person on staff who 
knows how to conduct needs assessments and/or 
evaluations? 
Yes No

Does a staff person from your agency know enough 
about evaluation to conduct one with the support of 
resources (e.g., manuals, toolkits, textbooks, guides, 
informal partners) that are available in the field?
Yes No

Do staff have enough time to devote to conducting a 
meaningful evaluation?
Yes No

Does your administration (e.g., chief or executive) 
support using internal staff to conduct an 
evaluation staff?
Yes No

end exhibit IV-1


Despite the advantages, there are potential challenges associated with hiring 
an external evaluator. Among these, expense and fit are discussed here along 
with suggestions for dealing with these possible pitfalls.

Expense. Conducting a thorough evaluation can be costly and 
time-consuming, especially for an evaluator who is not familiar with 
your community or program. To alleviate the start-up factor for an 
external evaluator, consider hiring someone from your community 
or field who has a basic understanding of the issues associated with 
law enforcement and your program or strategy. The 10 percent 
set-aside for evaluation that often is associated with grant awards can 

Exhibit IV-2

Advantages of Hiring an External Evaluator Compared to 
Using Internal Staff table

Table column headers: Advantage, External Evaluator, Internal Staff

Specialized 
Knowledge

Comes with specialized knowledge and 
ability; understands how to assess the 
needs of a community, document program 
outcomes, and collect and analyze. 

Often does not have expertise 
in the scientific approach to 
evaluation.

Reduces 
Burden

Reduces the burden on the internal staff 
who have limited time to devote to a 
meaningful evaluation.

May be focused on law 
enforcement activities and 
not have time to conduct a 
meaningful evaluation also. 

Objective 
Viewpoint

Remains unbiased about the program 
being studied; a good evaluator will point 
out both positive and negative aspects of a 
program, offering suggestions on how best 
to achieve the programs goals.

Can be committed to the work 
and believe unequivocally 
in the effectiveness of the 
program, thus running the 
risk of looking at evaluation 
findings in a biased way.

Higher Level of 
Credibility

More knowledgeable and objective about 
the evaluation process; conclusions and 
recommendations tend to carry more 
weight with funding institutions, the field, 
and the public.

Not as knowledgeable about 
evaluation; findings often 
viewed as less credible to the 
field.

end exhibit IV-2


be used to help cover the expense. In the long run, using the 10 percent 
set-aside to hire an external evaluator with specific expertise could be 
less expensive and produce better results than trying to do the work with 
existing staff.

Poor Match. Hiring an unqualified evaluator, or one who does not work 
well with your organization, can produce potentially damaging results 
such as alienating staff, intruding on the community and staff-community 
relationships, and drawing incorrect conclusions about the community 
and/or program. To find a proper match, take the time necessary to 
select the best evaluator for the job to ensure that the evaluation is not 
burdensome and that evaluation findings address research questions about 
how the program and services are working.

Carefully weigh all these factors when deciding to hire an external evaluator or 
rely on evaluation capabilities within your agency or partnering organizations to 
develop and carry out the evaluation plan.

What Do I Look for in an External Evaluator?

After careful deliberation, if you decide to hire an external evaluator, it often is 
helpful to organize a hiring committee. The committee should be large enough 
to address the issue under study and the dynamics of your agency adequately, but 
small enough so decision-making is efficient and not overly troublesome. The 
hiring committee should report to the appropriate parties (via periodic e-mails, 
newsletters, meetings, etc.) to keep everyone informed of the selection process.

The following sections provide guidance for considering the evaluators 
competencies, roles, and responsibilities. These also can be applied to internal staff 
when determining their competencies to carry out internal evaluation work.

Evaluator Competencies

Hiring committee members should become familiar with the qualifications 
necessary for an external evaluator. Evaluators should possess the following skills:

Identify data that can be used for planning, managing, and evaluating 
program policies and practices

Use multiple research methods and collect various types of data from a 
variety of sources

Establish data-collection and analysis priorities that are informed by the 
experiences, capabilities, and resources of program staff and participants

Produce reliable interpretation of data

Link data to program development and management.


In addition to these basic skills, evaluators must be willing to work within the 
realities of everyday program operations. Strong skills in communication, team 
building, group processes, and negotiation are vital to a successful relationship 
between the evaluator and your agency. Given the focus of your evaluation, it 
also is important to consider the following competencies when hiring an external 
evaluator:

Proficiency in facilitating an evaluation process that includes the 
experiences of the participants

Ability to build capacity in program staff to use evaluation findings

Willingness to develop and maintain a collaborative relationship based on 
cooperation and trust with program staff, managers, and stakeholders

Willingness to involve all partners in the evaluation 

Ability to design and implement methods to provide evaluation 
information to the program for use in planning, monitoring, and 
improvement

Experience in evaluating the type of programs, activities, or strategies that 
your agency is implementing 

Experience in conducting evaluations with law enforcement agencies 

Experience in working in culturally diverse settings with individuals of 
varying backgrounds and languages.

Find an evaluator with aptitude in as many of these competencies as possible. 
While evaluation success is not contingent on finding an evaluator with a high 
level of expertise in each of these competencies, it will benefit the project overall 
if you take time to secure an evaluator with the skills that meet your needs. Doing 
so will increase the probability of a meaningful evaluation. 

 How Do I Find an External Evaluator?

Now that you have a better understanding of the type of external evaluator who 
would be right for your project, you are ready to search for the person to fill that 
position. This section of the guide provides information on where to look and 
what to include in a job announcement.

Where to Look for an External Evaluator

When hiring an external evaluator, you might consider an individual or 
organization that either is referred to you or that you have worked with in the 
past. Alternatively, there are several sources you can consult to find a qualified 
evaluator. Potential sources, and the advantages and disadvantages of using each, 
are summarized in Exhibit IV-3.


Exhibit IV-3

Sources for Finding an External Evaluator table

Table column headers: Evaluator Source, Pros, Cons

College or 
university

Educational institutions offer the expertise of faculty 
members or graduate students at low to moderate 
overhead costs. Many small, not-for-profit organizations 
have found low-cost data-collection assistance through 
colleges or universities. With proper training, graduate 
students (the more inexpensive option) can create 
instruments and collect and analyze data. Contact college 
or university behavioral/social science, criminal justice, 
social work, or public health departments to learn if they 
have faculty and/or graduate students to assist you.

Independent researchers not working 
within a larger organization might 
be challenged to meet the variety of 
expertise and time commitments to 
complete the job within a grant cycle.

Independent 
researcher

Researchers who work alone or with minimal staff can 
provide high-level expertise at low overhead costs. They 
commonly have the flexibility to be located on site, 
enabling frequent interaction with initiative participants.

May be focused on law enforcement 
activities and not have time to 
conduct a meaningful evaluation. 

Research and 
evaluation 
organization

Organizations with research and evaluation departments 
offer a range of expertise. They are able to meet the 
multiple needs of an initiative by offering experts across 
disciplines and having state-of-the-art technology in 
house.

These organizations typically are 
associated with higher overhead 
costs.

end exhibit IV-3


What to Include in a Job Announcement

The evaluator job announcement should include an overview of your project, 
the roles and responsibilities of the evaluator, suggested timeline for completion 
of work, estimated budget information, delineation of preferred skills, and a 
request for applicant background and education information. Preparing the job 
announcement should be the responsibility of the hiring committee or at least it 
should be reviewed and approved by those involved in developing and managing 
the evaluation. A sample job announcement is provided in Appendix K.

Following the application submission deadline, the hiring committee should 
meet to review applications and select candidates to interview. Appendix I offers 
examples of interview questions. After committee members select a candidate they 
believe is right for the job, they should solicit feedback on their recommendation 
from all partners. Scheduling follow-up interviews to assess the personal style and 
characteristics of the potential evaluator helps ensure that the individual hired 
is acceptable to program staff, administrators, and key stakeholders. Interview 
more than one applicant in case your first choice is unavailable or does not accept 
your offer. Once an offer is accepted, engage the evaluator as soon as possible in 
evaluation planning. If you already have developed an evaluation plan, review 
the document with the external evaluator and make any changes necessary. 
Once your plan is finalized, you and your evaluator are ready to proceed with 
implementation.

Additional Resources

Two additional resources for finding an external evaluator are the American 
Evaluation Association (AEA) and the Evaluation Center at Western Michigan 
University. See also Appendix L.

American Evaluation Association (www.eval.org) 

The AEA is an international professional association of evaluators devoted to 
the application and exploration of program evaluation, personnel evaluation, 
technology, and many other forms of evaluation. AEA provides links to help you 
find an evaluator and information on training opportunities.

The Evaluation Center at Western Michigan University
(www.wmich.edu/evalctr)

The Evaluation Centers mission is to advance the theory, practice, and utilization 
of evaluation. The centers principal activities include research, development, 
dissemination, instruction, and national and international leadership in 
evaluation. The center provides service to a selected group of clientsincluding 
school districts, nonprofit agencies, government agencies, foundations, and 
collegesto evaluate their programs. The center also provides evaluation training 
to the field.


Chapter V:
Emerging Topics in 
Evaluation Mangement


Emerging Topics in Evaluation 
Management

Three emerging topics in evaluation and program accountability are presented in 
this section: 

1. Collaborative evaluation. An evaluation conducted jointly by more than 
one office, agency, or partner. 

2. Participatory evaluation. An evaluation that features active involvement 
by all stakeholders.

3. Performance measurement. An ongoing process of documenting 
indicators of progress toward goal achievement. 

Collaborative and participatory evaluations have similarities, particularly because 
both involve stakeholders in the process. Performance evaluation provides an 
opportunity for police departments to apply program evaluation results to 
internal decision-making on critical issues such as staffing.

Collaborative Evaluation

If the process of evaluation is carried out regularly and openly, with all group 
members participating, the answers they arrive at are in themselves not so 
important as what is learned from the discussion and from the process of reaching 
consensus on what questions should be used to evaluate outcomes and capacity. 
(Uphoff, 1991) 

Many of the barriers to evaluation, including a lack of skills, tight budgets, and 
limited access to data, can be addressed through a collaborative approach. Such 
an approach entails forming a team that shares decisions about the purpose 
of evaluation, how it will be conducted, and how results will be interpreted 
and used. Generally, the team includes individuals, agencies, or organizations 
that have some vested interest in the program, strategy, or activity that will be 
evaluated. For example, if your law enforcement agency is planning to implement 
a drug use prevention program in schools, your collaborative evaluation team 
might include representatives from your agency and the funding source in 
addition to representatives from the local school district (administrators and 
teachers), parents, substance abuse service providers, and other groups that might 
be involved in planning or implementing the evaluation. In addition, if you do 
not possess the expertise to facilitate evaluation planning and implementation, 
be sure to include an evaluation person. The key to the collaborative approach to 
evaluation is that no single person, agency, or organization owns the evaluation; it 
belongs to the team of individuals who have come together to support it. 


In developing this guide, we asked law enforcement representatives if they 
had used collaborative approaches to evaluation before and, if so, what their 
experiences had been. Most reported using this approach with positive results. 
In fact, they claimed several benefits, including access to more financial and staff 
resources (across agencies), the capacity to conduct more sophisticated evaluations 
(because of combined talents), and the opportunity to develop relationships with 
various organizations, agencies, and community and neighborhood groups. In 
addition to those reported by local law enforcement agencies, a collaborative 
approach to evaluation could have other benefits. According to Practical 
Evaluation of Public Health Programs (The University of Texas-Houston Health 
Science Center, 2001), a collaborative approach to evaluation accomplishes the 
following: 

Reduces suspicion and fear. Including stakeholders in decision-making 
about what to evaluate and how the evaluation will be conducted is likely 
to prompt support, whereas those who do not participate can become 
suspicious about the project and fearful of the outcome. They might not 
trust the results or those involved in the evaluation. People often fear that 
evaluation will result in termination of the program or their jobs, but 
involving them helps them understand that the focus of the evaluation is 
to improve the program.

Increases awareness and commitment. Through a collaborative 
approach, participants become more aware of, and committed to, the 
evaluation process. If people are involved, they have an interest in the 
results. A candid presentation and discussion of the programs data can 
spur development of a consensus about interpretation and follow-up 
action.

Increases the possibility of achieving objectives. If people understand 
what is being evaluated and why, they are more likely to work toward 
improving those elements of the program. The collaborative approach 
promotes ownership of the process and responsibility for the outcomes.

Broadens knowledge. A collaborative approach draws on broader 
knowledge, skills, and experience in evaluation. In choosing people for the 
evaluation team, the role each person will play and the knowledge, skills, 
and experience they bring to the evaluation should be considered. This is 
a good opportunity to look both inside and outside the organization to 
tap resources needed for the type of evaluation planned. The team should 
count people from a variety of backgrounds, including those with front-line 
service experience, statistical and epidemiological expertise, management 
and policy perspectives, and planning skills. All these skills might not be 
available within the program or even the organization, so it is often helpful 
to rely on a community college or university in your area for assistance 
either from faculty or student interns.


Teaches evaluation skills. When people work together, they share ideas, 
knowledge, skills, and abilities. Evaluation team members can learn 
about program objectives, data-collection methods, making evaluation 
decisions, and even how to work on a team. Team members learn by 
doing and come away from the process with an enhanced set of skills. 

Teaches stakeholders. A collaborative approach can teach community 
stakeholders about policing efforts. Involving people from outside the 
program increases their knowledge about your program and its role in 
law enforcement. This is an excellent way to involve the community in a 
discussion of law enforcement problems and possible interventions. For 
example, you might solicit participation from local elected officials and 
representatives of community-based organizations, all of whom would 
finish the evaluation with a better understanding of your programs 
strengths, the constraints under which law enforcement programs 
operate, what is needed for improvement, and how the program relates to 
their interests.

Increases the possibility that findings will be used. Involving 
stakeholders increases the possibility that the findings will be used 
and implemented. When a variety of staff are involved in conducting 
evaluations, identifying problems, and determining solutions to problems, 
they are more likely to accept implementation of the solutions decided on 
by the group.

Allows for different perspectives. Recalling the old adages, two heads 
are better than one and the whole is greater than the sum of its parts, 
the collaborative approach includes people from outside the program who 
have unique points of view.


The box below provides an example of a collaborative evaluation. As explained in 
this case study of the Hawthorne Huddle in Minneapolis, collaborative evaluation 
helped this community initiative demonstrate results, facilitate decision-making, 
and sustain its efforts. The evaluation managers stated that the participation 
of the Minneapolis Police Department in providing data throughout the 
implementation of the Huddle evaluation helped keep it focused on the needs 
of the community and the primary reason the Huddle was formedto reduce 
community crime and violence. In addition, participation of the Minneapolis 
Police Department in Huddle activities and the evaluation helped strengthen 
working relationships among all participating agencies. These are only some of the 
benefits a law enforcement agency can gain 
from a collaborative evaluation. 
A more detailed description of the 
Hawthorne Huddle is on page 11. 

How They Did It: 

Minneapolis Police Department

The Hawthorne Huddle, a 1999 
winner of the Herman 
Goldstein Award for Excellence in Problem-Oriented 
Policing, is a community-based initiative in which citizens; 
the faith community; the Minneapolis Police Department; 
city, county, and federal agencies; nonprofit organizations; 
and the General Mills Foundation worked together 
to develop a strategy to combat crime and improve 
neighborhood conditions. The Hawthorne Huddle used 
evaluation strategies throughout the initiative to identify 
(e.g., reviewing crime statistics and community concerns) 
and analyze (e.g., ongoing review of crime statistics and 
other community indicators) the problem, and develop a 
response. Many of the evaluation activities were conducted 
by Huddle staff and volunteers in a collaborative manner. 
In the fifth year, an external evaluation was conducted that 
was instrumental in documenting the accomplishments, 
lessons learned, and future goals of the initiative. The 
evaluation of this community initiative was managed by the 
General Mills Foundation, rather than by the Minneapolis 
Police Department. However, the results were used by all 
participating organizations, including the Minneapolis 
Police Department, to improve Huddle meetings, decide 
future strategic direction, and transition to new leadership.


Participatory Evaluation 

When conducting [evaluation] in a collaborative mode, professionals and 
nonprofessionals become co-evaluatorsGenuinely participatory approaches 
to evaluation require power sharing. One of the negative connotations often 
associated with evaluation is that it is something done to people. Participatory 
evaluation, in contrast, involves working with people. Instead of being research 
subjects, the people in the evaluation become co-investigators. The process is 
facilitated by the evaluator, but is controlled by the people in the program or 
community. (Patton, 2002, pg. 183)

Participatory evaluation necessitates active involvement by all stakeholders and 
typically has several characteristics that set it apart from traditional evaluation 
approaches:

Participant focus and ownership. Participatory evaluations are oriented 
primarily to the information needs of program stakeholders rather 
than the law enforcement agency. The agency simply helps participants 
conduct their own evaluations, thus building their ownership and 
commitment to the results and facilitating follow-up action.

Flexible design. While some preliminary planning for the evaluation 
might be necessary, design issues are decided in the participatory process. 
Generally, evaluation questions and data-collection and analysis methods 
are determined by participants, not outside evaluators.

Participant negotiations. Participating groups meet to communicate 
and negotiate to reach a consensus on evaluation findings, solve problems, 
and make plans to improve performance.

Use of facilitators. Participants actually conduct the evaluation; however, 
external evaluators can provide a supporting role as mentors, group 
processors, or methodologists.

Learning process. Participatory evaluation emphasizes identifying 
lessons learned to help participants improve program operations and 
assess whether targets were achieved (USAID Center for Development 
Information and Evaluation, 1996).

Each type of evaluation has its own strengths and limitations. Exhibit V-1 
presents some of the major differences between traditional and participatory 
approaches to evaluation. Some may be more appropriate than others, given 
circumstances and information needs. For example, if objectivity and credibility 
are key requirements, a more traditional external evaluation might be the right 
choice, with a hired evaluator who develops the plan and implements it with little 
input from you or your staff. If stakeholder ownership and program improvement 
are priorities, more collaborative or participatory approaches usually are better 
(USAID Center for Development Information and Evaluation, 1997). 


Some people criticize participatory evaluations as being less objective than more 
traditional approaches, while others suggest that a participatory approach can 
interfere with rigor when the evaluation requires complex analysis. Finally, some 
evaluators believe these approaches require more resources than more traditional 
approaches, because of the time and effort associated with bringing all 
stakeholders into the process. Regardless of the criticisms, your agency should 
determine the best approach to meet its information needs. 

How They Did It: 

Boston Police Department

 The emergence of crack cocaine in the mid-1980s fueled 
an escalating crisis of youth violence in urban centers such 
as Boston. The Boston Gun Project was one of almost 20 
community-based prevention initiatives designed to reduce 
youth violence that were evaluated by a participant team from 
Harvard University. While other initiatives focused on issues 
ranging from drug treatment to job placement, the Boston Gun 
Project involved a law enforcement problem-solving approach 
that specifically targeted guns. The initiative used a crackdown 
approach (pulling every [legal] lever) to focus police attention 
on illicit gun sellers and the children who bought those 
weapons. The strategy included quickly arresting high-profile 
gangsters in the targeted high-crime district, even for trivial gun 
violations, making this crackdown a hot topic of conversation 
on the street. The idea was to send a loud and clear message 
that the only way to get the police off your back was to disarm. 
Operation Ceasefire officially began in May 1996 
and within weeks there was a dramatic decline in youth violenceoverall 
a 63 percent drop in the monthly number of youth homicides, 
a 25 percent decrease in the monthly number of citywide all-
age gun assault incidents, and a 44 percent reduction in the 
monthly number of youth gun assault incidents in the target 
district. At roughly the same time, the city adopted a strategic 
plan that included a neighborhood police initiative called 
Same Cop, Same Neighborhood. According to the Harvard 
researchers, in a 29-month period ending in January 1998, there 
had not been a single teenage homicide, something virtually 
unheard of in a major U.S. city at the time.



Exhibit V-1

Difference between Participatory Evaluation and 
Traditional Evaluation table

Participatory Evaluation

Participant focus and ownership of the 
evaluation

Broad range of stakeholders participate

Focus is on learning

Flexible design

Evaluators are facilitators

Tradition Evaluation

Agency focus and ownership of the 
evaluation

Stakeholders have limited participation

Focus is on accountability

Predetermined design

Evaluators conduct the evaluation

end exhibit V-1

An example of the participatory approach to evaluation comes from 
Harvard Universitys David M. Kennedy, Anthony A. Braga, and 
Anne M. Piehl, who developed and implemented Operation Ceasefire 
in Boston. Elin J. Waring conducted the evaluation. Their report, 
Reducing Gun Violence: The Boston Gun Projects Operation Ceasefire 
(NCJ 188741, 2001), demonstrates the benefits of having the research 
team immersed in the initiative from the beginning. Instead of reading 
like a dry research report, the publication tells a magnificent story of 
how the police department became an integral and essential part of a 
citywide effort to save childrens lives.

The researchers helped design the intervention so they could determine 
whether any drop in gun violence was the result of Operation Ceasefire 
or another factor. The project launch was immediate and intense to 
ensure that the target community got the message. The official start date 
of Operation Ceasefire was May 15, 1996, the post-implementation 
phase began only 2 weeks later, on June 1, when the data already showed 
a dramatic drop in youth gun violence (see sidebar). The researchers 
identified and controlled for rival causal factors such as changes in the 
unemployment rate or the number of people classified as youth, as 
well as citywide trends in violent crime, homicides among older (age 
25+) residents, and youth involvement in street-level drug dealing (as 
measured by Boston Police Department arrests). Research participants 
credit this new kind of police-researcher partnership with enriching the 
entire process, from planning through evaluation. 


Performance Measurement

Performance measurement is the ongoing monitoring and reporting of program 
accomplishments, particularly progress toward preestablished goals, and is 
typically conducted by program or agency management. Performance measures 
may address the type of level of program activities conducted (process), the 
direct products and services delivered by a program (outputs), and the results of 
those products or services (outcomes). A program may be any activity, project, 
function, or policy that has an identifiable purpose or set of objectives (Artley 
and Stroh, 2001).

Performance measurement is an ongoing process of assessing performance 
indicators to determine whether a program, activity, or strategy is achieving 
its objectives. It differs from program evaluation in several important ways, 
which are presented in Exhibit V-2. The main difference between performance 
measurement and evaluation is that the former is management-focused, while the 
latter is research-focused. Each is designed to answer different questions and is 
implemented for different reasons. 

Exhibit V-2

Difference between Performance Measurement 
and Program Evaluation table

Performance Measurement

Management-focused activity

Continuous activity, tracking many projects 
across multiple fiscal years

Complexity at a minimal cost

General impact of grant program efforts

Evaluators are facilitators

Program Evaluation

Research-based activity

Discrete activity, focusing on fewer projects for 
a limited period

Complexity and cost are not major issues

Specific impact of grant program efforts

Evaluators conduct the evaluation

end exhibit V-2


For example, a performance measurement system for a demand reduction 
education program in which law enforcement officers participate might include 
indicators such as number of juveniles in programs, number of adults in 
programs, and number of officers providing demand reduction training. Similarly, 
for multijurisdictional task force programs that integrate federal, state, and/or 
local drug law enforcement agencies and prosecutors to enhance interagency 
coordination and intelligence and facilitate multijurisdictional investigations, 
performance indicators might include number of offenders arrested, number of 
offenders prosecuted, number of drug seizures, quantity by weight (e.g., ounces, 
grams, dose units), drug type, and total value of funds and assets forfeited. 
Finally, programs designed to target the domestic sources of controlled and illegal 
substances, such as precursor chemicals, diverted pharmaceuticals, clandestine 
laboratories, and cannabis cultivation might have performance indicators such as 
number of offenders arrested, number of offenders prosecuted, number of labs 
eliminated, number of drug seizures, quantity of seizure by weight (e.g., ounces, 
grams, dose units), and drug type. 

The Colorado Springs Police Departments (CSPD) performance monitoring 
system is based on the PASS (Police Accountability and Service Standards) 
model. Police service standards parallel the high-level organizational outcome 
goals for police service that the Colorado Springs community expects. CSPD 
service standards reflect the citys strategic plan and the police departments vision, 
mission, and values. Police service standards are described in terms of quality, 
quantity, cost, or time, and the seven categories are as follows:

1. Response times (e.g., response to emergency calls, answering telephone 
calls for police service). 

2. Officer deployment (e.g., scheduling and deployment efficiency). 

3. Traffic (e.g., crashes at top 25 targeted intersections). 

4. Clearance rates (e.g., violent and property crimes compared with national 
average). 

5. Drug and vice activity (e.g., response to methamphetamine labs, response 
to narcotics tips). 

6. Neighborhood policing (e.g., time consumed in proactive policing, 
partnerships). 

7. Citizen satisfaction with police services (e.g., customer satisfaction with 
officer contact, community perceptions of fear, safety, and disorder).

As noted by the chief of police and the citys planning manager: 

To achieve true accountability to the community, the police and the 
community must define police services and the level at which the services 
are provided. They must also identify measures of success for those services 
as viewed by their local communities. And finally, the police must be open 


to being held accountable by the community for the measures of success 
agreed upon. Such an open dialogue between the police and the community 
will help gain the trust and confidence of all of our stakeholders, police and 
citizens alike (Velez and Paine, 2002).

Based on data gathered through the performance system, informed decisions 
are made about service delivery, staffing levels, support personnel, manpower 
allocation, and deployment. The performance standards should reflect and guide 
realistic community expectations for police services. In commenting on the PASS 
model, the CSPD says the following:

The other thing that this model (PASS) does is it keeps the pulse of the 
community. . . I think departments that dont are missing out on a huge 
benefit, not only on monitoring their performance internally but also using 
that information to quantify why they need more officers, or that they need 
more police cars. 

Performance measurement cannot replace data on program costs, political 
judgments about priorities, creative solutions, or common sense. A major purpose 
of performance measurement is to raise fundamental questions; the measures, by 
themselves, provide definitive answers.

Characteristics of a good performance measurement system include the following:

Quality over quantity. Performance goals should be relevant to the core 
mission of the program and the result the program is intended to achieve.

Importance to budget decisions. Achievement of performance goals 
should inform funding decisions.

Public clarity. Performance goals should be understandable to the users of 
what is being measured.

Feasibility. Performance goals should be feasible and based on relevancy 
of the outcomes.

Collaboration. Often agencies must work together to achieve a single 
outcome.


Chapter Summary

Particularly as police departments embrace community policing, the benefits of 
collaborative and participatory evaluations become obvious. These approaches 
involve strategies that engage the community in the evaluation process, building 
opportunities to improve communication and promote trust, in addition 
to improving the scope and quality of evaluations themselves. Performance 
measurement is proving to be a valuable tool to tie program evaluation to 
municipal and police department decision-making about critical issues such as 
budgeting and staffing. Police departments that learn about and experiment with 
these techniques now will be fully prepared if evaluations are mandated in the 
future.

Police departments face mounting pressure to conduct quality evaluations and 
many federal agencies and private foundations want a strong commitment 
to evaluation demonstrated in competitive grant applications. While few 
municipalities so far tie funding to evaluation, this could change in the future. 
Police agencies that embrace evaluation now and experiment with a variety of 
techniques ensure that they will be ready for the future, while learning valuable 
lessons today. 


Appendixes



Appendix A

Sample Needs Assessment Action Plan Template

1. Who needs to be included?

Chief of police

Upper and middle management law enforcement 
personnel

Front-line police officers

Prosecutors

Court staff

Corrections officers

Womens advocacy groups.

2. What do we want to learn 
about our community?

Rate of domestic violence within a jurisdiction.

3. What do we already know 
about our community?

Official police data including number of calls for service and 
arrests.

4. Do we need to collect new data?

Sources: Who will have these data?

Front-line police officers who work closely with the 
community, professionals and volunteers who work with 
victims at community hot lines, personnel at domestic 
violence shelters and hospital emergency rooms, and 
batterer intervention providers. 

Methods: How will we gather the information?

Focus groups, surveys, and interviews with beat cops. 
Strategies also might include targeted outreach to 
professional womens groups. 

Instruments: What tools shall we use?

Focus groups, surveys, and interview protocols.

5. How will the data be analyzed?

Assess trends over time: Use data from calls for service and 
arrest as well as information collected from focus groups, 
surveys, and interviews with beat cops to determine 
whether the problem is getting better or worse. Use these 
data to examine what factors could be influencing changes 
over time. 

6. How will the findings be reported?

Create a brief write-up summarizing major findings. 
Communicate findings to community leaders, police 
personnel, and domestic violence advocacy groups. 

7. How will the data be used?

Gain insight into factors that could be influencing the rate 
of domestic violence within a community. 

Needs Assessment Action Plan table

Table Column headers: Task/Activity, Persons Responsible, Due Date,
Anticipated Result/Deliverable

Ex: Identify needs 
assessment questions

Ex: Law enforcement 
middle manager

Ex: October 15, 2006

Ex: List of needs 
assessment questions

Anticipated Needs Assessment Budget $__________



Appendix B

Preparing Measurable Goals and Objectives

The ABCDEs of writing measurable goals and objectives cover the who, what, to 
what degree, and by when information of your program.

AudienceWho?

The population/target audience for whom the desired outcome is intended.

BehaviorWhat? What is to happen?

A clear statement of the behavior change/results expected.

ConditionBy when?

The conditions under which measurements will be made. This may refer to 
the time frame and/or implementation of a specific intervention.

DegreeBy how much?

The quantification of, or the level of, results expected. This often involves 
measuring change compared to an identified baseline.

EvidenceAs measured by?

The definition of the method of measuring expected change. The degree 
of change (set forth above) will be measured using a specific instrument or 
criterion.

Using the ABCDE method, these are the steps for developing goals and objectives: 

To develop measurable goals

Step 1: Identify the longer term, global outcomes you wish to achieve.

Step 2: Identify each element (A, B, C, D, E).

Step 3: Formulate the goal statement using each necessary element. 

To develop measurable objectives

Step 1: 
Identify the shorter term, more immediate outcomes you 
wish to achieve.

Step 2: Identify each element (A, B, C, D, E). 

Step 3: Formulate the objective statement using each necessary element.


Sample Worksheet for Writing 
Measurable Program Goals and Objectives

Purpose: 

To develop measurable goals and objectives using the ABCDE method.

Directions: 

1. Identify the outcomes/impacts you wish to achieve. 
2. Identify each element of your goal and objective (A, B, C, D, E)
3. Formulate the goal and objective statements using each necessary 
element. 

GOAL: The longer term, global effects you wish to achieve.

Identify the elements:

A. Audience is Citizens within a community.

B. Behavior to be changed is Community response to domestic violence.

C. Conditions under which change will occur is (are) Greater 
communication and collaboration among community organizations that 
work with domestic violence victims and offenders.

D. Degree of change expected is Enhanced safety of domestic violence 
victims and their families.

E. Evidence of change is Decreased domestic violence mortality rate 
within a community.

Goal Statement: 
To implement a Domestic Violence Enhanced Response Team (DVERT) to 
improve the safety of citizens within a community. 

OBJECTIVES: The more immediate, direct results you wish to achieve.

Identify the elements:

A. 
Audience is Domestic violence victims and offenders and the agencies 
that work with these populations.

B. 
Behavior to be changed is  Community response to high-risk-for-
lethality domestic violence victims.


C. 
Conditions under which change will occur is (are) Enhanced 
coordinated community response to reports of domestic violence by 
creating a collaborative made up of agencies that work with domestic 
violence victims and offenders.

D. 
Degree of change expected is Improve response time to reports of 
domestic violence and increase communication and collaboration 
among partners.

E. 
Evidence of change is Appropriate containment of high-risk domestic 
violence offenders, improved response time to reports of domestic 
violence, faster intervention by domestic violence service providers.

Objective Statements: 

To increase communication and coordination between law enforcement, domestic 
violence service providers, batterer intervention service providers, the courts, 
probation and parole, and the district attorneys office.

To produce greater batterer accountability by increasing batterer enrollment and 
attendance in offender treatment.

To connect domestic violence victims with services and resources immediately 
following a domestic violence incident.


Appendix C

Planning Model Template

Draw seven columns on a sheet of paper and name them Column 
A, B, C, D, E, F, G.

Step 1: In Column A, list all background factors and conditions you think 
might influence the relationship between your program activities and goals. For 
example, lack of knowledge about the rights and needs of domestic violence 
victims would be a background factor. 

Step 2: In Column B, list program inputs. For example, the amount of funds 
that will go toward increasing the knowledge of the rights and needs of domestic 
violence victims would be an input. 

Step 3: In Column C, list program activities. For example, training law 
enforcement officers would be an activity.

Step 4: In Column D, list program outputs. For example, the number of law 
enforcement officers trained on the rights and needs of domestic violence 
victims would be an output. 

Steps 5 and 6: In Columns E and F, list all events occurring during or after your 
program activities that could affect how or whether you accomplish your goals. 
For example, an immediate outcome of the law enforcement officer training 
would be a better understanding of domestic violence. An intermediate outcome 
among law enforcement would be improved ability to assist domestic violence 
victims in obtaining a personal protection order.

Step 7: In Column G, list program goals. Listing program goals helps you see 
the overall effects of your inputs as well as the changes resulting from them. For 
example, one goal might be to decrease the incidence of domestic violence in your 
county. It is best to start with short-term goals because it often is difficult to measure 
and document long-term ones. 


Sample Planning Model Template

Conditions
Problem or need:
Improve community 
response to 
incidents of 
domestic violence

Inputs
What we invest: 
Time, money, and 
staff.

Activities
What strategies or 
services comprise 
our program?
Collaborative made 
up of police officers 
and victim/offender 
service providers.
Training for police 
officers on how to 
respond to incidents 
of domestic violence.

Outputs
What are the direct 
products of our 
program activities?
DVERT collaborative 
meetings held two 
times/month.
150 police officers 
trained on ways to 
respond to reports of 
domestic violence.

Outcomes  Impacts
Immediate, Intermediate, Long-term

Short-term 
changes we 
expect:
Increased 
knowledge among 
police officers on 
ways to respond to 
domestic violence. 

Medium-term 
changes we 
expect:
Increased 
communication 
and collaboration 
between partners 
involved with the 
project (e.g., police 
officers and victim 
and offender 
service providers).

Long-term 
changes we 
expect:
Increasing batterer 
enrollment and 
attendance 
in offender 
treatment.

Connecting 
domestic 
violence victims 
with services 
and resources 
immediately 
following a 
domestic violence 
incident.

Contextual Factors 
1. Agency budgets.

2. Staffing.

3. Political factors.

4.Community interest or media 
attention to this issue.

5. Legislation.


Appendix D

Community Satisfaction Survey

The Fictitious Police Department wants to know about your attitudes, opinions, and 
experiences regarding our performance. All responses will remain confidential. Thank 
you for your time and support.

Please circle the answer closest to your views:

1. 
Do you feel the Fictitious Police Department has improved its police service 
during the past 5 years? 
Yes No NA

2. 
How would you rate the competence of the Fictitious Police Department?
Excellent Good Fair Poor NA

3. 
Do you feel that Fictitious police officers act in a professional and courteous 
manner?
Yes No NA

4. 
Are Fictitious police officers actively involved in working with citizens to 
make neighborhoods safer?
Yes No NA

5. 
How satisfied are you overall with Fictitious police service?
Very satisfied 
Somewhat satisfied 
Neither satisfied nor unsatisfied 
Somewhat unsatisfied 
Very unsatisfied 
NA


6. How safe do you feel in your home?
Very safe
Somewhat safe
Neither safe nor unsafe
Somewhat unsafe
Very unsafe 

7. How safe do you feel on the street?
Very safe
Somewhat safe
Neither safe nor unsafe
Somewhat unsafe
Very unsafe 

8. Have you or a member of your family ever been a victim of a violent crime?
Yes No NA

9. Have you or a member of your family ever been a victim of a property crime?
Yes No NA

10. Have you interacted with a Fictitious police officer in the past year?
Yes No NA
If yes, please check all that apply:

_ Victim of crime
_ Traffic stop
_ I called police because of an emergency/crime
_ Others called police to my home
_ Neighborhood/school meeting
_ Other (please explain) _________________________________


11. Are you? 

Male
Female

12. Please circle your age category: 

Under 18 18-24 25-34 35-44 45-54 55-64 65 and up

13. How would you characterize your neighborhood?

High crime 
Low crime
Average level of crime
NA

14. 
Would you be willing to volunteer your time and talents to work with the 
Fictitious Police Department on community-based efforts to make our 
neighborhoods safer?

Yes No NA

If yes, please call xxx-xxx-xxxx or visit our web site at www.fictitiouspd.gov to find 
out about community meetings in your area and to volunteer.


Appendix E

Sample Focus Group Checklist

Focus Group Checklist

Consider the following items when you plan focus groups and develop focus 
group guides.

Advance Notice

_ Contact participants 1 to 2 weeks before the session.

_ Send each participant a letter of invitation.

_ Give the participants a reminder prior to the session. 

_ Over-recruit by three to five participants the number of participants 
necessary for each focus group.

Logistics

Make sure the room is satisfactory (size, tables, comfort, lighting, temperataure, etc.)

_ Arrive early to make necessary changes. 

_ Eliminate background noise as much as possible.

_ Bring nametags for participants.

_ Bring extra supplies like tapes, batteries, note pads, and extension cords.

_ Seat loud/disruptive participants next to the moderator.

_ Seat shy and quiet participants directly across from the moderator.

_ Serve refreshments before discussions begin.

_ Bring enough copies of handouts and/or visual aids. 

_ Set up a tape recorder and make sure that it works.

Questions

Introductory questions should be answered quickly and not identify personal 
characteristics of participants.

_ Questions should flow in a logical sequence.

_ Key questions should focus on the critical issues of concern.

_ Consider probe or follow-up questions.

_ Use open-ended questions.

_ Provide a summary of the discussion and invite comments.


Moderator Skills

_ Be well rested and alert for the focus group session.

_ Practice the introduction without referring to notes.

_ Make sure participants are comfortable and relaxed.

_ Ask questions with minimal reference to notes.

_ Be careful to avoid head nodding (i.e., showing support or favoritism 
for answers).

_ Avoid comments that signal approval (i.e., Excellent, Great, Wonderful)

_ Avoid giving personal opinions. 

Immediately After the Session

_ Prepare a brief written summary of key points as soon as possible.

_ Check to see if the tape recorder captured the comments. 


Appendix F

Instrument Development Checklist

Consider the following items when you develop your evaluation instrument.

Instrument Title

_ Use clear and concise words. 

_ State the program name. 

_ Indicate the type of service provided (e.g., counseling, shelter). 

_ Reflect the instrument method and content (e.g., survey, interview). 

Introductory Statement

_ Include information about the instruments purpose. 

_ Include information about how the data will be used. 

_ Include information about the level of confidentiality that will be 
arranged (e.g., who will see their responses, how responses will be 
reported). Always provide the confidentiality you promise. 

Demographics

_ Include questions that ask respondents for relevant information about 
themselves and their backgrounds (e.g., victims name, country of origin, 
and language). 

_ If necessary, include questions that ask about the person administering 
the instrument, if not you or program staff (e.g., observers name). 

_ If appropriate, identify the length of respondent participation in the 
program. 

Directions

_ Include general directions on how to complete the instrument itself (e.g., 
when, where, and how to return the instrument). 

_ Include specific directions on how to complete each section of the 
instrument. 

_ Make sure specific directions appear before each section. 


Questions

_ Use language that respondents understand (e.g., way to teach vs. 
pedagogical). 

_ Avoid double-barreled questions (e.g., Has there been an increase in 
program resources and the number of program participants?). 

_ Allow enough space for respondents to write when using open-ended 
questions. 

_ Avoid biased and value-laden words or phrases. 

_ Include only questions asking for needed information. 

_ Keep question and answer options on the same page. 

_ Allow space for comments, concerns, or suggestions. 

Format

_ Use icons or graphics as clarifiers (e.g., Please place a check in the 
appropriate box.). 

_ Use a clearly legible font (e.g., Arial, Times New Roman). 

_ Lay out text and graphics using an entire page. 

_ Allow enough space between questions. 

_ Develop an instrument that it is pleasing to the eye (e.g., not busy). 

_ Indicate the date of instrument administration. 

_ Identify whether it is a pre-, post-, or ongoing survey. 

_ Note the name of the program/organization that developed the instrument in 
the footer (at the bottom of the page). 

_ Include a computer file location path of where to find a copy of the 
instrument in the footer (e.g., X:\Trafficking Victim Survey), if appropriate. 

_ Include the date of each new version in the header/footer. 

Pilot Testing

_ Clearly label a draft instrument DRAFT. 

_ Be mindful that advance permission to conduct the pilot test might be 
necessary. 

_ Arrange for pilot test participants and conditions to be as close to actual 
administration conditions as possible (e.g., time of day, location, methods, 
respondents). 

Source: The University of Texas at Arlington, School of Social Work. (n.d.). Instrument development checklist. 
Retrieved June 10, 2004 from www2.uta.edu/sswmindel/S6324/Class%20Materials/measurement/instrument 
development_checklist.htm


Appendix G

Topics to Cover in an Informed Consent 
Statement for Research Participants

Invitation to Participate and Description of Project

_ Introduce yourself and describe the scope and purpose of your research.

_ Explain what agencies or groups are involved in funding and/or 
conducting the evaluation.

_ Tell the reader who is being targeted for participation in the research 
project.

Risks and Inconveniences

_ Describe the risks associated with participating in the research, if any.

_ Tell participants how much of their time will be required if they choose to 
participate.

Benefits

_ Explain why the research is important.

_ Note the benefits of participating in this research. 

Description of Procedures

_ Tell respondents what to expect if they agree to participate (e.g., how 
many interviews will be conducted with each person, where and when the 
focus group will be held; how much time it will take, whether any follow-
up will be necessary).

Voluntary Participation and Withdrawal

_ Make sure respondents know that participation is voluntary and they can 
withdraw at any time without penalty. 

Your participation is completely voluntary. You are free to choose not to 
participate in the study or to stop your participation at any time. You dont have 
to answer any questions that you dont want to, and you can withdraw your 
participation at any time without consequences or penalties. 


Confidentiality

_ Explain the steps you will take to protect the confidentiality of respondents.

The information you provide will be kept confidential and will be used only for 
research purposes. We will not connect your name with your responses to the research 
questions. Research team members will not release any of the information you 
provide on this survey outside the confines of this research project. Your responses to 
the interview questions will be kept in a locked cabinet and will be seen only by the 
research team members. Your answers will be grouped with the other responses and 
summarized later in a report. 

Also, we would like your permission for the interview to be audiotaped. Please 
understand that the transcripts will be used only for research purposes and that no 
names will be associated with your transcript or audiotape.

Questions

Please feel free to ask any questions and to think about whether you want to 
participate before you make a decision.

Do you agree to participate in this research study? 
0. No 
1. Yes

Can we audiotape this interview? 
0. No 
1. Yes

If you have further questions about this project or if you have a research-related problem, 
you may contact


Appendix H

Sample Evaluation Plan Template

1. Focus

What Will We 
Evaluate (Which 
Program Or Aspect 
Of A Program)? 

Examine 
whether a police 
departments 
Domestic Violence 
Enhanced 
Response Team 
(DVERT) is 
enhancing the 
safety of high-
risk-for-lethality 
victims.

2. Questions
What Do You Want
To Know?

1. Is DVERT 
improving safety 
of children and 
families?

2. Is DVERT 
helping to ensure 
appropriate 
containment 
of high-risk 
offenders?

3. Indicators-
Evidence

How will we 
know it? 

1. a. Client 
satisfaction 
surveys

b. Interviews with 
DVERT partner 
agency staff (e.g., 
child welfare and 
domestic violence 
agencies) and 
focus groups with 
clients.

 
2. a. Case records 
from District 
Attorneys Office

b. Interviews with 
DVERT partner 
agency staff (e.g., 
child welfare and 
domestic violence 
agencies) and 
focus groups with 
clients.

4. Timing

When should we 
collect data?
 
Work with 
DVERT program 
management 
to determine a 
timeline for data 
collection that will 
not interfere with 
program activities.

5. Data Collection

Sources

Who will have this 
information?

Meet with District 
Attorneys Office 
to determine 
accessibility of case 
records.

Methods

How will we gather 
the information? 

Client satisfaction 
surveys

Interviews with 
DVERT partner 
agencies

District Attorneys 
Office Case Record 
Review

Focus groups with 
DVERT clients.

Sample

Who will we 
question?

Work with 
DVERT program 
management 
to identify staff 
to participate in 
interviews and 
staff who will be 
responsible for 
administering 
surveys to clients.

Instruments

What tools shall we 
use?

Develop client 
satisfaction survey 
protocol, interview 
protocol, and focus 
group protocol.

6. How will the data be 
analyzed?

Data from case records and 
satisfaction surveys will be analyzed 
using quantitative methods (e.g., 
descriptive analysis such as frequencies 
distributions and cross-tabulations).

Data from interviews and focus groups 
will be analyzed using qualitative 
methods (e.g., descriptive analysis such 
as thematic coding).

7. How will the data 
interpreted?

Compare analyses from surveys, 
interviews, focus groups, and case 
reviews.

8. How will the results be communicated?

To Whom?

DVERT program manager

High-level law enforcement 
administrators

Beat cops.

When? Where? How?

Create a brief summary of findings to be 
discussed during role calls and during DVERT 
management meetings.


Appendix I

Sample Interview Questions Checklist

The following checklist contains a series of questions to help you interview and select 
an external evaluator. The questions are designed to help you judge the qualifications 
and characteristics of prospective evaluators, including basic training in evaluation, 
previous evaluation experiences, and some personal characteristics that fit your 
initiative and the community in which you will be implementing your program. In 
addition to the checklist, it will be important to look at work samples and contact 
references for any prospective evaluator.

Interview Questions

Evaluator appears to be: (Check one for each item)

Well-Qualified

Not Well-
Qualified

Cannot 
Determine

What formal training have you had in conducting needs 
assessment studies? What practical experience (ask for 
examples of needs assessment studies they have led or 
directed)?

What formal training have you had in conducting evaluation 
studies? What practical experience (ask for examples of 
studies they have led or directed)?

What experience do you have in evaluating policing 
programs?

What experience do you have in conducting evaluations of 
community initiatives?

What experience do you have in integrating evaluation 
activities with community planning processes?

What experience do you have in conducting evaluations 
with law enforcement (you may also ask about their 
experience working with any other agencies with whom 
you are partnering and with whom they would have to 
work)?

How familiar are you with the issue of _____ (insert the 
program/strategy area you will be evaluating, e.g., domestic 
violence)?

What indicators would you use for this type of evaluation? 

Source: Adapted from the W. K. Kellogg Foundation Evaluation Handbook and Evaluation 
Handbook for Projects Funded by S.T.O.P. Formula Grants.


Appendix J

Glossary

Archival data  Information that has already been collected by a department 
or organization, such as a government or nonprofit agency. These documents 
may include program records or materials such as proposals, annual or monthly 
progress reports, budgets, organization charts, memoranda, policies and 
procedures, operations handbooks, and training materials.

Baseline  Initial data on program participants or other program aspects 
collected prior to receipt of services or program intervention. Baseline data are 
often gathered through intake interviews and observations and are used later for 
comparing measures that determine changes in participants or a program.

Causality A cause and effect relationship. The causality of two events describes 
the extent to which one event is caused by the other. When there is causality, 
there is a measure of predictability between the two events. 

Cleaning data  The process by which quantitative data are reviewed to identify 
missing or incorrect responses.

Comparison group  Individuals whose characteristics are similar to those of 
program participants or the treatment group. The individuals in the comparison 
group may not receive any services, or they may receive a different set of services, 
activities, or products. The experimental (or treatment) group and the comparison 
group are assessed to determine which type of services, activities, or products 
provided by a program produced the expected changes. 

Control group  A group of individuals whose characteristics are similar to those 
of program participants or an experimental group, but do not receive the program 
being evaluated. Participants are randomly assigned to either the treatment (or 
program) group or the control group. A control group is used to assess the effect 
of a program on participants compared to similar individuals not receiving the 
activities being evaluated. 

Cost benefit  An assessment of whether program costs are achieving the 
intended program impact. This analysis compares costs and benefits using the 
same monetary units. Determining the dollar value of a benefit is arguably the 
primary challenge in cost-benefit analysis.

Cost effectiveness  An assessment of whether money expended on a program 
was well-spent in reaching the program outcomes. This is the relationship 
between agency costs and its effectiveness in its mission. 

Data  Information or facts that are collected to answer specific research 
questions.


Document review  A process of data collection that involves examining existing 
records or documents. 

Evaluability assessment  A process used to identify whether a program is ready to 
be evaluated. This is an assessment of whether or not a program has clearly defined 
activities and measurable goals and objectives that lend themselves to identifying data 
that can be collected. This process includes reviewing the program history, design, 
and operation, perhaps by watching the program in action; determining the programs 
capacity for data collection; and assessing the likelihood that the program will meet 
its goals and objectives based on the activities that are actually being implemented. 

Executive summary  A condensed version of a longer reports content. 
Executive summaries are written for someone who most likely does not have time 
to read a lengthy report. An executive summary may be called an abstract when it 
accompanies a scholarly document.

Experimental design  Individuals are randomly assigned to a test and control 
group and both are measured before and after the intervention, then the results are 
compared and analyzed.

External evaluation  An external is conducted by an agency or individuals not 
directly involved in or responsible for the program or activities being evaluated.

Evaluation design  The overall plan and approach for an evaluation. A description of 
how you plan to measure program components and how you plan to use the resulting 
measurements. The evaluation design is determined by the type of evaluation that is 
used and on the basic questions you want answered about your program.

Evaluation plan  A description of the overall approach or design used to guide 
an evaluation. The evaluation plan should include information on what evaluation 
activities will take place, a methodology for executing these activities, a timeline for 
completing planned evaluation tasks, a list of the individuals who will be responsible 
for completing each evaluation task, and a description of the purpose of the 
evaluation.

Formal evaluation  Rigorous and comprehensive assessments of data that are used to 
make decisions within a department, agency, or organization.

Frequency  A measure used to determine the number of times a given response 
occurs. This measure can be expressed using whole numbers or percentages.

Goal  A measurable statement of the desired long-term impact of the program.

History  Any event that happens between the time of your first measurement (pre-
test) and your second (post-test) that can affect the measurement. This can pose a 
threat to the validity of evaluation results. 

Impact evaluation  This evaluation is designed to assess whether a program or 
intervention had a long-term impact on a specified goal.


Informal evaluation  Less rigorous or comprehensive assessments of data that are 
used to make decisions within a department, agency, or organization.

Instrumentation  Changes over time can take place in the instruments used 
to measure participants performance. Changes caused by instrumentation 
can threaten internal validity if they cannot be separated from the effect of the 
treatment. Any flaws in a measurement device can skew evaluation results. 

Internal evaluation  An internal is conducted by the agency or individuals 
who are directly involved with or responsible for the program or activities being 
evaluated.

Logic model  A graphic representation that clearly identifies and lays out the 
logical relationships among program conditions, activities, outcomes (objectives), 
and impacts (goals).

Maturation  This describes the normal processes of development that occur 
over time that could affect evaluation outcomes independent of the evaluators 
intervention, and can pose a threat to the validity of evaluation results.

Mean  An average score that is calculated by dividing the sum of all responses to 
a particular question by the number of responses to that question.

Median  The midpoint or value below which half the values in a distribution 
fall. This score can be generated by arranging all responses in a list. The middle 
number is the median. If there are two middle numbers, then add the two 
numbers and average them for a median. 

Mixing methods  An approach that consists of intentionally combining different 
evaluation tools and techniques, not only to observe and to gather quantitative 
and qualitative information, but also to structure and analyze this information 

Mode  the number or value that occurs most often in a distribution of values.

Multiple methods  See definition of mixing methods.

Needs assessment  This evaluation is designed to assess problems in the 
community and perceptions about why they occur and persist.

Objective  A specific, measurable statement of the desired immediate or direct 
outcomes of the program that support the accomplishment of a goal.

Outcome  Refers to a change that results from a program or its activities.

Outcome evaluation  This evaluation is designed to assess whether the 
intervention achieved its stated short-term goals and objectives.

Outcome data  Information that is collected to measure the effectiveness 
of a program.

Output  Units of service that are provided by the program.


Post-test only design  A type of evaluation design in which the target group is 
measured only after the intervention and those results are analyzed.

Pre- and post-test design  A type of evaluation design in which the target group 
is measured before and after the intervention and those results are compared and 
analyzed.

Pre- post- post-test design  A type of evaluation design in which the target group is 
measured before the intervention, immediately afterward, and then at a later time and 
those results are compared and analyzed.

Process data  This evaluation is designed to assess whether the intervention was 
implemented as intended and, if not, why.

Process evaluation  This evaluation is designed to assess whether an intervention was 
implemented as intended and, if not, why.

Program assessment  A disciplined way of assessing the merit, value, and worth of 
projects and programs.

Qualitative data  Data that are collected through activities such as observations, 
interviews, focus groups, and conversations. These data describe the attributes 
of an object without referring to quantity and are expressed in words. Instead of 
measuring the extent and direction of change or program impact, qualitative data are 
better suited to describing the nature of the change or impact that has taken place. 
Qualitative data are used frequently in process evaluations to describe how a program 
functions. 

Quantitative data  Data that are a measurement of something that can be counted 
and assigned a number. These data can be collected by using several methods, 
including simple counts, surveys, and tests. 

Quasi-experimental design  An evaluation design in which the target group and a 
similar group used as a control are measured before and after the intervention, then 
the results are compared and analyzed.

Questionnaire  See the definition for survey.

Random assignment  The assignment of individuals in the pool of all potential 
participants to either the experimental (treatment) or control group in such a manner 
that their assignment to a group is determined entirely by chance. 

Reliable data  The extent to which a measurement produces consistent results 
over repeated observations or administrations of the instrument under the same 
conditions each time. 


Research questions  A data-collection tool used to gather information on the 
focus of an evaluation effort.

Selection bias  Any determination, except random placement, by which you 
place some subjects in a treatment group, and other subjects in a control group. 
Subjects assigned to groups in any way other than random placement opens the 
possibility that the groups differ in important ways that can affect the results of a 
study. 

Stakeholder  People who will be affected by a project or can influence it but who 
are not directly involved with doing the project work. 

Statistical regression  A general tendency for extreme scores on any measure to 
move toward a less extreme score at a different time. Statistical regression occurs 
when study participants are selected based on their extreme scores. When an 
evaluator re-measures, there is a tendency for these extreme scores to move back 
toward the overall group average regardless of the effects of any intervention 
provided.

Strategic management  The process of specifying an organizations objectives, 
developing policies and plans to achieve these objectives, and allocating resources 
to implement these plans.

Survey  A data-collection method that involves collecting information directly 
from individuals. Surveys can be self-administered, conducted face-to-face, over 
the telephone, or mailed. 

Target group  Also called a treatment group, a target group is composed of 
individuals receiving the activities or interventions that you are evaluating.

Testing  The process of test taking can pose a threat to the validity of evaluation 
results. Taking a pre-test may influence the behavior of subjects as much or more 
than the intervention does. The evaluator cannot be certain that the effects they 
see are because of the intervention and not caused by the administration of a pre-
test.

Treatment group  Also called a target group, a treatment group is composed of 
individuals receiving the activities or interventions that you are evaluating. 

Valid data  Data that are collected using a measurement instrument or test that 
accurately measures what they are supposed to measure. 


Appendix K

Sample Job Announcement

Overview of Project

The Fictitious Police Department is seeking the services of a qualified evaluator to 
assist in an assessment of its 1-year project to reduce drug dealing in the citys public 
housing developments. The department has been awarded a grant from the U.S. 
Department of Justices Bureau of Justice Administration and the U.S. Department 
of Housing and Urban Developments Office of Resident Initiatives to explore 
how police can engage community coalitions in targeted efforts to reduce the open 
distribution and sale of illicit drugs. The grant includes funding for an evaluation 
designed to assess the impact of various strategies on reducing the sale of drugs to 
residents of the complex and also how those strategies affect sales to outsiders who 
come there to purchase drugs.

Evaluator Roles/Responsibilities

The evaluator is responsible for developing and implementing the evaluation plan, 
including the methodology, types of data be gather and analyze, and the management 
plan for completing the evaluation according to the guidelines set forth in the grant 
agreement. The evaluator will work closely with top command in charge of the 
project to develop a task list and timeline for the departments participation in the 
evaluation process.

Tasks and Timeline

Applicants must outline an evaluation plan designed to produce the final report by 
the end of the 1-year grant-reporting period, as well as fulfill the quarterly reporting 
requirement specified in the grant agreement. Applicants must describe how they will 
meet the grant requirements, including a detailed discussion of the tasks and who will 
carry them out within the timeline detailed in the grant agreement.

Management Plan and Key Personnel

Applicants also must provide a proposed management plan designed to ensure that a 
comprehensive and valid evaluation will be completed within the timeline. The plan 
also must include a description of the roles and responsibilities of key personnel, as 
well as the role and responsibilities of the evaluator in oversight.


Preferred Skills and Experience

Applicants must demonstrate appropriate skills and experience in developing and 
implementing an evaluation strategy to assess a police initiative in a real-world 
setting. Applicants must demonstrate strong program evaluation and impact 
evaluation skills, in addition to the management skills required to complete a 
complex and multifaceted assessment within a specified time.

Background/Education

Applicants must submit a current curriculum vita or rsum for the lead evaluator 
and additional key personnel, as well as a brief description of relevant experience 
in police evaluation. Preferred candidates will exhibit educational credentials that 
certify broad experience in evaluation methodology and implementation, as well 
as the practical skills required to work alongside police in a diverse environment.

Complete applications along with references and examples of relevant work should 
be submitted to [Contact Person] by [Due Date] at [Address of the Agency].


Appendix L

Evaluation Resources

Evaluation resources with practical applications for people with little or 
no evaluation experience

Eck, John E. Assessing Responses to Problems: An Introductory Guide for Police Problem-
Solvers. Washington, D.C.: U.S. Department of Justice Office of Community 
Oriented Policing Services, 2002. www.cops.usdoj.gov/txt/pop/e08064504.txt

Eck, John E. Learning from experience in problem-oriented policing and situational 
prevention: The Positive Functions of Weak Evaluations and the Negative 
Functions of Strong Ones. In Evaluation for Crime Prevention, Nick Tilley, ed. 
Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. 

Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.). 
 
Beverly Hills, 
California: Sage Publications, 1990.

Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). Newbury 
Park, California: Sage Publications, 1993. 

Salant, P. and D. Dillman, D. How to Conduct Your Own Survey. New York: Wiley 
Publishers, 1994.

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement 
Aagencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice 
Statistics and the Office of Community Oriented Policing Services, 1999. 
www.ojp.usdoj.gov/bjs/pdf/ccspglea.pdf

Web sites that offer useful information on evaluation resources, 
tools, and articles:

American Evaluation Association: The American Evaluation Association is an 
international professional association of evaluators devoted to the application 
and exploration of program evaluation, personnel evaluation, technology, and 
many other forms of evaluation. Evaluation involves assessing the strengths and 
weaknesses of programs, policies, personnel, products, and organizations to 
improve their effectiveness. www.eval.org

Bureau of Justice Assistance (BJA) Evaluation Web Site: The evaluation web site, 
produced by the BJA Center for Program Evaluation, provides state 
administrative agency staff, criminal justice planners, researchers and evaluators, 
and local practitioners with a variety of resources for evaluating criminal justice 
programs. http://www.ojp.usdoj.gov/BJA/evaluation/index.html


Center for Problem Oriented Policing: The Center for Problem-Oriented Policing 
is a nonprofit organization composed of police practitioners, researchers, and 
universities dedicated to advancing the concept and practice of problem-oriented 
policing in open and democratic societies. www.popcenter.org

COPS Office: The Office of Community Oriented Policing Services 
(the COPS Office) was created as a result of the Violent Crime Control and 
Law Enforcement Act of 1994. As a component of the Justice Department, the 
mission of the COPS Office is to advance community policing in jurisdictions 
of all sizes across the country. COPS provides grants to tribal, state, and local 
law enforcement agencies to hire and train community policing professionals, 
acquire and deploy cutting-edge crime-fighting technologies, and develop and 
test innovative policing strategies. The COPS Office also offers publications, 
CD-ROMs, videos, toolkits, and training resources to law enforcement agencies 
nationwide. www.cops.usdoj.gov

International Association of Chiefs of Police (IACP): The IACP is the worlds 
oldest and largest nonprofit membership organization of police executives, with 
more than 19,000 members in more than 89 different countries. Its goals are to 
advance the science and art of police services; develop and disseminate improved 
administrative, technical, and operational practices; foster police cooperation 
and exchange of information; bring about recruitment and training of qualified 
persons in the police profession; and encourage adherence of all police officers to 
high professional standards of performance and conduct. www.theiacp.org/

Juvenile Justice Evaluation Center Online (JJEC): The JJEC, a segment of 
the Justice Research and Statistics Association, is a tool for assisting juvenile 
justice practitioners, policymakers, and state agency administrators with the 
assessment and evaluation of programs and initiatives. JJEC Online is composed 
of several sections, including JJEC Information, State Information, Juvenile 
Justice Evaluation Program Areas, Evaluation Resources, State Information, and 
Technical Assistance and Training. www.jrsa.org/jjec/index.html

Police Executive Research Forum (PERF): PERF, a national organization of 
progressive police executives, is dedicated to improving policing and advancing 
professionalism through research, public policy debate, provision of management 
services and executive development, training, and publishing. www.policeforum.org


Police Foundation: Since its founding in 1970, the Police Foundation has conducted 
seminal research in police behavior, policy, and procedure. The foundation has 
established and refined the capacity to define, design, conduct, and evaluate 
controlled experiments testing ways to improve the delivery of police services.

www.policefoundation.org/

Technical assistance resources:

The American Evaluation Associations (AEA) web site contains a list of individuals and 
firms that are available for evaluation consulting. www.ovcttac.org/aboutus.cfm

The Office for Victims of Crime Training and Technical Assistance Centers (OVC 
TTAC) training and technical assistance activities are coordinated through three core 
functions: 

Needs assessment. OVC TTAC uses a variety of measurement toolssurveys, 
stakeholder discussions, and literature reviewsto assess the needs of key 
constituencies and to identify available technical assistance and training 
resources to support their needs. 

Capacity building. By developing and delivering training and technical 
assistance resources, OVC TTAC helps organizations foster professional 
development of their staff, enhance services to their communities, and reach 
unserved and underserved victims of crime. 

Evaluation. OVC TTAC monitors customer satisfaction and measures the 
resulting changes of its training and TA activities over time. 

www.ovcttac.org/aboutus.cfm

The What Works Clearinghouse provides a Registry of Outcome Evaluators 
that allows web site users the opportunity to search for evaluators (both 
individuals and organizations) who conduct research on the effects of 
educational interventions. The Registry allows users to search by geographic 
region, individual/organization, content area, and target populations. 
www.whatworks.ed.gov/technicalassistance/EvlSearch.asp

Examples of evaluation reports related to policing:

Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: The Boston 
Gun Projects Operation Ceasefire. Washington, D.C.: U.S. Department of 
Justice, National Institute of Justice, 2001.

Lurigio, A.J. and W.G. Skogan. Winning the Hearts and Minds of Police Officers: 
An Assessment of Staff Perceptions for Community Policing in Chicago, Crime 
& Delinquency, 40 (3) (1994): 315330.

Independent Commission on Policing for Northern Ireland. (1999). A New 
Beginning: Policing in Northern Ireland. Retrieved November 24, 2004 from 
www.belfast.org.uk/report.htm


Evaluation journals:

American Journal of Evaluation

The Evaluation Exchange

Evaluation and Program Planning, ISSN 0149-7189

The Evaluation Review

New Directions for Evaluation

Practical Assessment Research and Evaluation, a peer-reviewed electronic journal, 
ISSN 1531-7714

Technical reports and articles:

Burt, M.R., et al. Evaluation Guidebook. Washington DC: National Institute 
of Justice, Office of Justice Programs, U.S, Department of Justice, Urban 
Institute, 1997.

Clark, R.V. and J. Eck. Becoming a Problem Solving Crime Analyst in Fifty-Five 
Small Steps. Jill Dando Institute of Crime Science. Ad Hoc Publications, 2002. 

Coldren, J.R., T. Bynum, and J. Thome. Evaluating Juvenile Justice Programs: 
A Design Monograph for State Planners. Office of Juvenile Justice and 
Delinquency Prevention, 1991. 
www.jp.usdoj.gov/BJA/evaluation/guide/documents/documents/documentg.html 

The Compiler. (1995). Evaluation as an Important Tool in Criminal Justice 
Planning. Retrieved January 20, 2004.

Eck, J. Assessing Responses to Problems: An Introductory Guide for Police 
Problem-Solvers. Washington, D.C.: U.S. Department of Justice Office of 
Community Oriented Policing Services, 2002. 
www.cops.usdoj.gov/txt/pop/e08064504.txt /www.cops.usdoj.gov

Joppe, M. The Research Process. Ontario Hostelry Institute, AMEX Canada, and 
Ryerson University, n.d.. Retrieved January 19, 2004, from 
www.ryerson.ca/~mjoppe/ResearchProcess/RequestforProposal.htm

Juvenile Justice Evaluation Center. Hiring and Working with an Evaluator: 
Program Evaluation Briefing Series No. 2. Washington, D.C.: Justice 
Research & Statistics Association, 2001.

Juvenile Justice Evaluation Center, Justice Research and Statistics Association. 
Evaluability Assessment: Examining the Readiness of a Program for Evaluation. 
Program Evaluation Briefing Series No. 6. Washington, D.C.: Office of 
Juvenile Justice and Delinquency Prevention, 2003. 
www.jrsa.org/jjec/about/briefing_evaluability-assessment.html


Melvin M., G.T. Henry, and G. Julnes. Evaluation: An Integrated Framework for 
Understanding, Guiding, and Improving Public and Nonprofit Policies and 
Programs. San Francisco: Jossey-Bass, 2002. 

Muraskin, L.D. Understanding Evaluation: The Way to Better Prevention Programs. 
Washington, D. C.: U.S. Department of Education, 1993. 
www.jrsa.org/jjec/resources/publications/westat-handbook.pdf

Novotney, L. C., E. Mertinko, J. Lange, and T. K. Baker. Juvenile Mentoring 
Program: A Progress Review, Juvenile Justice Bulletin. Washington, DC: U.S. 
Department of Justice, Office of Justice Programs, Office of Juvenile Justice and 
Delinquency Prevention, 2000.

Priest, S. A Program Evaluation Primer, Journal of Experiential Education, 24 (1) 
(2001), 3440.

Sechrest, L., and A.J. Figuerdo. Program Evaluation. Evaluation Group for Analysis 
of Data, Department of Psychology, University of Arizona. Annual Review of 
Psychology. 44 (1993), 64574.

Shadish, Jr., W.R., T.d. Cook, and L.C. Leviton. Foundations of Program Evaluation, 
Theories of Practice. London: Sage Publications, 1995. 

Taylor-Powell, E., S. Steele, and M. Douglah. Planning a Program Evaluation 
(G-3658-1). Madison: University of Wisconsin-Extension, Cooperative 
Extension, 1996.

Todevski, D. Public Administration Evaluation. Ministry of Finance, Republic of 
Macedonia, 2001. 

United States Air Force Family Advocacy Program. A guide to family intervention 
and prevention program evaluation: Conducting an evaluability assessment. In G. 
K. Kantor, K. Kendall-Tackett, C. M. Allen, eds. Durham: University of New 
Hampshire, 2000. 

United States Department of Health and Human Services, Administration for 
Children & Families. The Program Managers Guide to Evaluation. 

United States Department of Health and Human Services, Substance Abuse and 
Mental Health Services Administration, Center for Substance Abuse Prevention. 
Achieving Outcomes: A Practitioners Guide to Effective Prevention. Washington, 
DC, 2002. www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf 

Public Technology, Inc. Program Evaluation and Analysis: A Technical Guide for State 
and Local Governments. Washington, D.C.: U.S. Department of Housing and 
Urban Development, Office of Policy Development and Research, 1978.

United States Department of Housing and Urban Development, Office of Policy 
Development and Research. A Guide to Evaluating Crime Control of Programs in 
Public Housing. Prepared for the United States Department of Housing and Urban 
Development by KRA Corporation. Washington, D.C., 1997.


United States General Accounting Office. Designing Evaluations. Washington, 
D.C.: United States General Accounting Office, Program Evaluation and 
Methodology Division, 1991.

Weidman, D. R., et al. Intensive Evaluation for Criminal Justice Planning Agencies. 
Washington, D.C.: U.S. Department of Justice, 1975.

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement 
Agencies. U.S. Department of Justice, Bureau of Justice Statistics and the 
Office of Community Oriented Policing Services. 
www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf

Wholey, J.S., H.P. Harty, and K. E. Newcomer. The Handbook of Practical 
Program Evaluation. San Francisco, California: Jossey-Bass, 1994.

W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. 
Kellogg Foundation, 1998.

W.K. Kellogg Foundation. Logic Model Development Guide: Using Logic Models 
to Bring Together Planning, Evaluation, and Action. Battle Creek, Michigan: 
W.K. Kellogg Foundation, 2001.

Evaluation Books

Berk, R.A. and P. H. Rossi. Thinking About Evaluation (2nd ed.). Newbury Park, 
California: Sage Publications, 1999. 

Bickman, L., ed. Using Program Theory in Evaluation. San Francisco, California: 
Jossey-Bass, 1987. 

Boruch, R. F., P.M. Wortman, and D.S. Cordray, eds. Reanalyzing Program 
Evaluations. San Francisco: Jossey-Bass, 1981. 

Boyle, R. and D. Lemaire. Building Effective Evaluation Capacity: Lessons from 
Practice. New Brunswick, New Jersey: Transaction Publishers, 1999. 

Bradley, V. J. Assessing and Enhancing the Quality of Services: A Guide for the 
Human Services Field. Boston: Human Services Research Institute, 1984. 

Braybrooke, C.B., and C.B. Lindblom. A Strategy of Decision: Policy Evaluation as 
a Social Process. New York: Free Press, 1963. 

Brinkerhoff, R.O., D.M. Brethower, T. Hluchyj, and J.R. Nowakowski. 
Program Evaluation: A Practitioners Guide for Trainers and Educators. Boston:
Kluwer-Nijhoff, 1983. 

Brinkerhoff, R.O., ed. Evaluating Training Programs in Business and Industry, 
No.44, Winter. San Francisco: Jossey-Bass, 1989. 


Bryk, A.S., ed. Stakeholder-Based Evaluation. San Francisco, California: Jossey-Bass, 1983.

Campbell, D.T., and J.C. Stanley. Experimental and Quasi-Experimental Designs for 
Research. Chicago: Rand McNally, 1963. 

Caro, F.C., ed., Readings in Evaluation Research. New York: Russell Sage Foundation, 1971. 

Chambers, D.E., K.R. Wedel, and M.K. Rodwell. Evaluating Social Programs. Boston: 
Allyn & Bacon, 1992. 

Chen, H. Theory-Driven Evaluations. Newbury Park, CA: Sage Publications, 1990. 

Conner, R., and M. Hendricks, eds. International Innovations in Evaluation 
Methodology. San Francisco: Jossey-Bass, 1989. 

Cook, T.D., and D.T. Campbell. Quasi-Experimentation: Design and Analysis Issues for 
Field Settings. Chicago: Rand McNally, 1979. 

Cronbach, L.J. Designing Evaluations of Educational and Social Programs. 
San Francisco: Jossey-Bass, 1982.

Datta, L., ed. Evaluation in Change: Meeting New Government Needs. Newbury Park, 
California: Sage Publications, 1981. 

Denzin, N.K., and Y.S. Lincoln, eds. Handbook of Qualitative Research. Thousand 
Oaks, California: Sage Publications, 1994. 

Eck, J. Learning from Experience in Problem-Oriented Policing and Situational 
Prevention: The Positive Functions of Weak Evaluations and the Negative 
Functions of Strong Ones. In Evaluation for Crime Prevention, Nick Tilley, ed. 
Crime Prevention Studies, 14. Monsey, New York: Criminal Justice Press, 2002. 

Eisner, E.W. The Enlightened Eye: Qualitative Inquiry and the Enhancement of 
Educational Practice. Upper Saddle River, New Jersey: Prentice Hall, 1998. 

Fetterman, D.M., S. J. Kaftarian, and A. Wandersman, eds. Empowerment Evaluation: 
Knowledge and Tools for Self-Assessment and Accountability. Thousand Oaks, 
California: Sage Publications, 1996. 

Grinnell, R.M. Social Work Research and Evaluation (4th ed.). Itasca, Illinois: F.E. 
Peacock, 1993. 

Guba, E.C. and Y.S. Lincoln. Effective Evaluation: Improving the Usefulness of 
Evaluation Results through Responsive and Naturalistic Approaches. 
San Francisco: Jossey-Bass, 1981. 

Guba, E.G. and Y.S. Lincoln. Fourth Generation Evaluation. Newbury Park, 
California: Sage Publications, 1989. 

Guttentag, M. and E.L. Struening, eds. Handbook of Evaluation Research. Newbury 
Park, California: Sage Publications, 1975.


Hausman, J. and D. Wise, eds. Social Experimentation. Chicago: University of 
Chicago Press, 1985. 

Herman, J.L., L.L. Morris, and C.T. Fitz-Gibbon. Evaluators Handbook. 
Newbury Park, California: Sage Publications, 1987. 

House, E.R., ed. School Evaluation: The Politics and Process. Berkeley, California: 
McCutchan, 1973. 

House, E.R., ed New Directions in Educational Evaluation. London: Farmer, 1986.

King, J., L. Morris, and C.T. Fitz-Gibbon. How to Assess Program Implementation. 
Newbury Park, California: Sage Publications, 1987.

LaVigna, G.W., T. J. Willis, J.F. Shaull, M. Abedi, and M. Sweitzer. The Periodic 
Service Review: A Total Quality Assurance System for Human Services and 
Education. Baltimore: Paul H. Brookes, 1994. 

Levin, H.M. Cost-Effectiveness: A Primer. Newbury Park, California: Sage 
Publications, 1983. 

Levine, R.A., M.A. Solomon, C.M. Hellstern, and H. Wollmann. Evaluating 
Action Programs: Readings in Social Action and Education. Boston: Allyn & 
Bacon, 1981. 

Lincoln, Y.S. and E.C. Cuba. Naturalistic Inquiry. Newbury Park, California: Sage 
Publications, 1985. 

Lofland, J. and L.H. Lofland. Analyzing Social Settings: A Guide to Qualitative 
Observation and Analysis. Belmont, California: Wadsworth, 1984. 

Love, A.J. Internal Evaluation: Building Organizations from Within. Applied 
Social Research Methods Series, Vol. 24. Newbury Park, California: Sage 
Publications, 1991 

Madaus, C.F., M.J. Scriven, and D.L. Stufflebeam, eds. Evaluation Models: 
Viewpoints on Educational and Human Services Evaluation. Boston: Kluwer-
Nijhoff, 1983. 

Miles, M.B.and A.M. Huberman. Qualitative Data Analysis: A Sourcebook of New 
Methods. Newbury Park, California: Sage Publications, 1984. 

Mishan, E.J. Cost-Benefit Analysis (4th ed.). London: Unwin Hyman, 1988. 

Mohr, L.B. Impact Analysis for Program Evaluation (2nd ed.). Thousand Oaks, 
California: Sage Publications, 1995. 

Nathan, R.P. Social Science in Government: Uses and Misuses. New York: Basic 
Books, 1989. 

Nay, J.N. and P. Kay. Government Oversight and Evaluability Assessment. 
Lexington, Massachusetts: D.C. Heath, 1982.


Paine, S.C., G.T. Bellamy, and B.L. Wilcox, eds. Human Services that Work: From 
Innovation to Standard Practice. Baltimore, Maryland: Paul H. Brookes, 1984. 

Palumbo, D., ed. The Politics of Program Evaluation. Newbury Park, 
California: Sage Publications, 1991. 

Patton, M.Q. Practical Evaluation. Newbury Park, California: Sage Publications, 1982.

Patton, M.Q. Creative Evaluation (2nd ed.). Newbury Park, California: Sage 
Publications, 1987. 

Patton, M.Q. How to Use Qualitative Methods in Evaluation. Newbury Park, 
California: Sage Publications, 1987. 

Patton, M.Q. Qualitative Evaluation and Research Methods (2nd ed.). Beverly Hills, 
California: Sage Publications, 1990. 

Patton, M.Q. Utilization-Focused Evaluation (3rd ed.). Thousand Oaks, California: 
Sage Publications, 1997. 

Popham, I.W., ed. Evaluation. in Education: Current Applications. Berkeley, 
California: McCutchan, 1974. 

Pressman, I.L. and A. Wildavsky. Implementation (3rd ed.). Berkeley: University of 
California Press, 1984. 

Rossi, P.H., ed. Standards for Evaluation Practice. San Francisco: Jossey-Bass, 1982. 

Rossi, P.H. and H.E. Freeman. Evaluation: A Systematic Approach (5th ed.). 
Newbury Park, California: Sage Publications, 1993. 

Royce, D. . Program Evaluation. Chicago: Nelson Hall, 1992. 

Rutman, L., ed. Evaluation Research Methods: A Basic Guide. Newbury Park, 
California: Sage Publications, 1977.

Salant, P. and D. Dillman. How to Conduct Your Own Survey. New York: Wiley 
Publishers, 1994.

Saxe, L. and M. Fine. Social Experiments: Methods for Design and Evaluation. 
Newbury Park, California: Sage Publications, 1981. 

Schmidt, R.E., J.W. Scanlon, and J.B. Bell. Evaluability Assessment: Making Public 
Programs Work Better. Rockville, Maryland: U.S. Department of Health and 
Human Services, Project Share, 1979. 

Schulberg, H.C. and J.M. Jerrell, eds. The Evaluator and Management. Newbury Park, 
California: Sage Publications, 1979. 

Scriven, M. The Logic of Evaluation. Inverness, California: Edgepress, 1980. 

Scriven, M. Evaluation Thesaurus (4th ed). Newbury Park, California: Sage 
Publications, 1991. 


Sechrest, L., I. Bunker, I., and B. Perrin, eds. Improving Methods in Nonexperimental 
Research. Newbury Park, California: Sage Publications, 1991. 

Shadish, W., T. Cook, and L. Leviton. Foundations of Program Evaluation: Theories 
of Practice. Newbury Park, California: Sage Publications, 1991. 

Smith, N.L. ed. New Techniques for Evaluation. Newbury Park, California: Sage 
Publications, 1981. 

Smith, N.L., ed. Communication Strategies in Evaluation. Newbury Park, 
California: Sage Publications, 1982. 

Stake, R. B., ed. Evaluating the Arts in Education. A Responsive Approach. 
Columbus, Ohio: Merrill, 1975. 

Stufflebeam, D.L. and A.J. Shinkfield. Systematic Evaluation: A Self-Instructional 
Guide to Theory and Practice. Boston: Kluwer-Nijhoff, 1985. 

Suchman, B.A. Evaluative Research: Principles and Practice in Public Service and 
Social Action Programs. New York: Russell Sage Foundation, 1967. 

Thompson, M.S. Benefit-Cost Analysis for Program Evaluation. Newbury Park, 
California: Sage Publications, 1980. 

Trochim, W.M.K. Research Design for Program Evaluation: The Regression-
Discontinuity Approach. Newbury Park, California: Sage Publications, 1984.

Veney, J.E. and A.D. Kaluzny. Evaluation and Decision-Making for Health Services 
(3rd ed.). Ann Arbor, Michigan: Health Administration Press, 1998. 

Webb, B.J., D.T. Campbell, R.D. Schwartz, and L.B. Sechrest. Unobtrusive 
Measures: Nonreactive Research in the Social Sciences. Chicago: Rand 
McNally, 1966. 

Weiss C.H. Evaluation: Methods for Studying Programs and Policies (2nd ed.). 
Englewood Cliffs, New Jersey: Prentice Hall, 1997. 

Weiss, C.H. Evaluation Research: Methods for Assessing Program Effectiveness. 
Englewood Cliffs, New Jersey: Prentice-Hall, 1972. 

Weiss, H.B. and F. Jacobs. Evaluating Family Programs. New York: Aldine 
DeGruyter, 1988. 

Welch, W., ed. Case Study Methodology in Educational Evaluation. Minneapolis: 
Minnesota Research and Evaluation Center, 1981. 

Wholey, J.S. Evaluation and Effective Public Management. Boston: Little, 
Brown, 1983. 


Williams, W. The Implementation Perspective. Berkeley: University of California Press, 
1980.

Yin, R.K. Case Study Research: Design and Methods. Newbury Park, California: Sage 
Publications, 1984. 

Zweig, F.M., ed. Evaluation in Legislation. Newbury Park, California: Sage 
Publications, 1979. 

Zweig, F.M. and K.E. Martin, eds. Educating Policymakers for Evaluation. Newbury 
Park, California: Sage Publications, 1981 


References



References

Artley, A. and S. Stroh, eds. The Performance-Based Management Handbook: A Six-
Volume Compilation of Techniques and Tools for Implementing the Government 
Performance and Results Act of 
1993 (GPRA), Volume 2: Establishing an 
Integrated Performance Measurement System. Oak Ridge Institute for Science 
and Education, Training Resources and Data Exchange, The Performance-
Based Management Special Interest Group, 2001. 
www.orau.gov/pbm/pbmhandbook/Volume%202.pdf

Braga, A., D. Kennedy, A. Piehl, and E. Waring. Reducing Gun Violence: 
The Boston Gun Projects Operation Ceasefire. Washington, D.C.: U.S. 
Department of Justice, National Institute of Justice, 2001. 

Center for Substance Abuse Prevention (CSAP). Building a Successful Prevention 
Program: A Planning Tool for Substance APrevention Professionals. CSAP 
Western Center for the Application of Prevention Technologies (CAPT). 
Reno: University of Nevada, 2003. 

Community Policing Consortium Resources. Strategic Planning for Community 
Policing, Section 5: Evaluation. Washington, D.C.: Community Policing 
Consortium. www.communitypolicing.org/strategic/sec6.html

Fiedler, M. The Colorado Springs Police Departments Police Accountability and 
Service Standard (PASS) Mmodel. Colorado Springs: Management Services 
Division, Planning Section, Colorado Springs Police Department, 2002.

Juvenile Justice Evaluation Center, Justice Research and Statistics Association. 
Evaluability Assessment: Examining the Readiness of a Program for Evaluation. 
Program Evaluation Briefing Series No. 6. Washington, DC: Office of 
Juvenile Justice and Delinquency Prevention, 2006.
www.jrsa.org/jjec/about/briefing_evaluability-assessment.html

Loe, H., Jr. Practical Evaluation of Public Health Programs Workbook. Houston, 
TX: The University of Texas-Houston Health Science Center, School of 
Public Health and The Texas Department of Health, 2001.
www.phppo.cdc.gov/phtn/Pract-Eval/workbook.asp

Mertinko, E., L.C. Novotney, T. Baker, and J. Lange. Evaluating Your Pprogram: 
A Beginners Self-Evaluation Workbook for Mentoring Programs. Washington, 
D.C.: Office of Juvenile Justice and Delinquency Prevention, Office of 
Justice Programs, and the U.S. Department of Justice. 


Miles, M.B. and A.M. Huberman. Qualitative Data Analysis. Thousand Oaks, 
California: Sage Publications, 1994. 

Patton, M. Q. Qualitative Evaluation and Research Methods (2nd ed.). 
Beverly Hills, California: Sage Publications, 1990.

Patton, M.Q. Qualitative Research and Evaluation Methods. Thousand Oak,s 
California: Sage Publications, 2002.

Salant, P., D. Dillman. How to Conduct Your Own Survey. New York: Wiley 
Publishers, 1994.

Shadish, W. R., T.D. Cook, and D.T. Campbell. Experimental and Quasi-
Experimental Designs for Generalized Causal Inference. New York: Houghton 
Mifflin, 2002.

United States Department of Health and Human Services, Substance Abuse and 
Mental Health Services Administration (SAMHSA), Center for Substance 
Abuse Prevention. Achieving Outcomes: A Practitioners Guide to Effective 
Prevention. Washington, D.C.: SAMHSA, 2002. 
www.modelprograms.samhsa.gov/pdfs/AchievingOutcomes.pdf

The University of Texas at Arlington, School of Social Work. Instrument 
Development Checklist. www2.uta.edu/sswmindel/S6324/
Class%20Materials/measurement/instrument_development_checklist.htm

The University of Texas-Houston Health Science Center, School of Public 
Health and The Texas Department of Health. Practical Evaluation of Public 
Health Programs. Atlanta, Georgia: The Centers for Disease Control and 
Prevention and The Association of Schools of Public Health Cooperative 
Agreement, 1998.

University of Wisconsin-Extension: Program Development and Evaluation. 
Enhancing Program Performance with Logic Models. 
www.uwex.edu/ces/lmcourse/

Uphoff, N. A Field Methodology for Participatory Self-Evaluation, Community 
Development Journal, 26, (1991) 271286. 

Urban Institute. Evaluation Guidebook for Projects Funded by S.T.O.P. Formula 
Grants under the Violence Against Women Act. Washington, D.C.: National 
Institute of Justice, 1997.

USAID Center for Development Information and Evaluation. Performance 
Monitoring and Evaluation Tips: Conducting a Participatory Evaluation. 
Washington, D.C.: United States Agency for International Development, 1996.

USAID Center for Development Information and Evaluation. Performance 
Monitoring and Evaluation Tips: The Role of Evaluation in USAID. Washington, 
D.C.: United States Agency for International Development, 1997.


Velez, L. and T. Paine. Service Standards in the Colorado Springs Police Department. 
Retrieved May 1, 2007 from www.calea.org/Online/newsletter/No83/
servicestandards.htm

Weisel, D. Conducting Community Surveys: A Practical Guide for Law Enforcement 
Agencies. Washington, D.C.: U.S. Department of Justice, Bureau of Justice 
Statistics and the Office of Community Oriented Policing Services. 
www.ojp.usdoj.gov/bjs/pub/pdf/ccspglea.pdf 

W.K. Kellogg Foundation. Evaluation Handbook. Battle Creek, Michigan: W.K. 
Kellogg Foundation, 1998.


For More Information

U.S. Department of Justice
Office of Community Oriented Policing Services
1100 Vermont Avenue, N.W.
Washington, DC 20530

To obtain details on COPS programs, 
call the COPS Office Response Center at 800.421.6770

Visit COPS Online at the address listed below
www.cops.usdoj.gov

e06072996